Skip to contentSkip to conversation

btrfs: don't force DIO writes to be serialized

PATCH In review Mark Harmstone 3 participants 7 messages Apr 22 – Apr 28 Btrfs file system

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 btrfs: don't force DIO writes to be serialized

Before btrfs switched to the new mount API in 2023, we were setting SB_NOSEC in btrfs_mount_root(). This flag tells the VFS that the filesystem may have files which don't have security xattrs, enabling it to do some optimizations.

Unfortunately this was missed in the transition, meaning that IS_NOSEC will always return false for a btrfs inode. This means that btrfs_direct_write() calls will always get the inode lock exclusively, meaning that DIO writes to the same file will be serialized.

On my machine, this one-line change results in a ~59% improvement in DIO throughput:

Before patch:

test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=64
...
fio-3.39
Starting 32 processes
test: Laying out IO file (1 file / 1024MiB)
Jobs: 32 (f=32): [w(32)][100.0%][w=764MiB/s][w=195k IOPS][eta 00m:00s]
test: (groupid=0, jobs=32): err= 0: pid=586: Wed Apr 22 13:03:04 2026

    write: IOPS=202k, BW=787MiB/s (826MB/s)(46.1GiB/60012msec); 0 zone resets
     bw (  KiB/s): min=498714, max=1199892, per=100.00%, avg=806659.03, stdev=4229.94, samples=3808
     iops        : min=124677, max=299971, avg=201661.82, stdev=1057.49, samples=3808
    cpu          : usr=0.32%, sys=1.27%, ctx=8329204, majf=0, minf=1163
    IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
       issued rwts: total=0,12094328,0,0 short=0,0,0,0 dropped=0,0,0,0
       latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):

    WRITE: bw=787MiB/s (826MB/s), 787MiB/s-787MiB/s (826MB/s-826MB/s), io=46.1GiB (49.5GB), run=60012-60012msec

After patch:

test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=64
...
fio-3.39
Starting 32 processes
test: Laying out IO file (1 file / 1024MiB)
Jobs: 32 (f=32): [w(32)][100.0%][w=1255MiB/s][w=321k IOPS][eta 00m:00s]
test: (groupid=0, jobs=32): err= 0: pid=572: Wed Apr 22 13:13:46 2026

    write: IOPS=320k, BW=1250MiB/s (1311MB/s)(73.3GiB/60003msec); 0 zone resets
     bw (  MiB/s): min=  619, max= 2289, per=100.00%, avg=1251.28, stdev= 9.64, samples=3808
     iops        : min=158538, max=586025, avg=320320.80, stdev=2468.97, samples=3808
    cpu          : usr=0.35%, sys=11.50%, ctx=1584847, majf=0, minf=1160
    IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
       issued rwts: total=0,19203309,0,0 short=0,0,0,0 dropped=0,0,0,0
       latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):

    WRITE: bw=1250MiB/s (1311MB/s), 1250MiB/s-1250MiB/s (1311MB/s-1311MB/s), io=73.3GiB (78.7GB), run=60003-60003msec
Fixes ad21f15b0f79 ("btrfs: switch to the new mount API")
---
 fs/btrfs/super.c | 1 +
 1 file changed, 1 insertion(+)
fs/btrfs/super.c +1 -0
index a60bce413d33b5..fb15decb086189 100644
@@ -1872,6 +1872,7 @@ static int btrfs_get_tree_super(struct fs_context *fc)
 	fs_info->fs_devices = fs_devices;
 	mutex_unlock(&uuid_mutex);
 
+	fc->sb_flags |= SB_NOSEC;
 
 	sb = sget_fc(fc, btrfs_fc_test_super, set_anon_super_fc);
 	if (IS_ERR(sb)) {
Signature · 1 line
2.52.0
#1 lore raw
Quoted 5 lines · Mark Harmstone

Before btrfs switched to the new mount API in 2023, we were setting SB_NOSEC in btrfs_mount_root(). This flag tells the VFS that the filesystem may have files which don't have security xattrs, enabling it to do some optimizations.

Quoted 7 lines

Unfortunately this was missed in the transition, meaning that IS_NOSEC will always return false for a btrfs inode. This means that btrfs_direct_write() calls will always get the inode lock exclusively, meaning that DIO writes to the same file will be serialized.

On my machine, this one-line change results in a ~59% improvement in DIO throughput:

That's quite an improvement. What's the actual fio script you've used? Also the DIO depends on the block group profile wrt the buffered fallback so that would be good to know too.

#2 lore raw
Quoted 11 lines · David Sterba
Quoted 10 lines · Mark Harmstone

Before btrfs switched to the new mount API in 2023, we were setting SB_NOSEC in btrfs_mount_root(). This flag tells the VFS that the filesystem may have files which don't have security xattrs, enabling it to do some optimizations.

Unfortunately this was missed in the transition, meaning that IS_NOSEC will always return false for a btrfs inode. This means that btrfs_direct_write() calls will always get the inode lock exclusively, meaning that DIO writes to the same file will be serialized.

Quoted 6 lines
Quoted 2 lines

On my machine, this one-line change results in a ~59% improvement in DIO throughput:

That's quite an improvement. What's the actual fio script you've used? Also the DIO depends on the block group profile wrt the buffered fallback so that would be good to know too.

It is. There's a big dropoff in DIO write performance in 6.8 that we never recovered from. I'm going to look into some sort of automated performance so this kind of thing can't happen casually.

This was on a VM with 8 cores and 8GB of RAM, with a real NVMe exposed through PCI passthrough. The figures for XFS and ext4 in comparison are both about ~3GB/s.

# cat go
#!/bin/bash
mkfs.btrfs -f /dev/nvme0n1
mount /dev/nvme0n1 /mnt/test
mkdir /mnt/test/nocow
chattr +C /mnt/test/nocow
fio /root/test.fio
# cat /root/test.fio
[global]
rw=randwrite
ioengine=io_uring
iodepth=64
size=1g
direct=1
startdelay=20
force_async=4
ramp_time=5
runtime=60
group_reporting=1
numjobs=32
time_based
disk_util=0
clat_percentiles=0
disable_lat=1
disable_clat=1
disable_slat=1
filename=/mnt/test/nocow/fiofile
[test]
name=test
bs=4k
stonewall
#3 lore raw
Quoted 12 lines · Mark Harmstone
Quoted 11 lines · David Sterba
Quoted 10 lines · Mark Harmstone

Before btrfs switched to the new mount API in 2023, we were setting SB_NOSEC in btrfs_mount_root(). This flag tells the VFS that the filesystem may have files which don't have security xattrs, enabling it to do some optimizations.

Unfortunately this was missed in the transition, meaning that IS_NOSEC will always return false for a btrfs inode. This means that btrfs_direct_write() calls will always get the inode lock exclusively, meaning that DIO writes to the same file will be serialized.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

On my machine, this one-line change results in a ~59% improvement in DIO throughput:

That's quite an improvement. What's the actual fio script you've used? Also the DIO depends on the block group profile wrt the buffered fallback so that would be good to know too.

It is. There's a big dropoff in DIO write performance in 6.8 that we never recovered from.

There is the bounded page solution from iomap already, which will no longer fallback to buffered IO but to use extra page copy to make sure the final bio won't change its content halfway.

IIRC it's one extra flag and remove the btrfs' specific fallback checks, but I haven't yet verified the behavior/code.

Thanks,
Qu

Quoted 38 lines

I'm going to look into some sort of automated performance so this kind of thing can't happen casually.

This was on a VM with 8 cores and 8GB of RAM, with a real NVMe exposed through PCI passthrough. The figures for XFS and ext4 in comparison are both about ~3GB/s.

# cat go
#!/bin/bash
mkfs.btrfs -f /dev/nvme0n1
mount /dev/nvme0n1 /mnt/test
mkdir /mnt/test/nocow
chattr +C /mnt/test/nocow
fio /root/test.fio
# cat /root/test.fio
[global]
rw=randwrite
ioengine=io_uring
iodepth=64
size=1g
direct=1
startdelay=20
force_async=4
ramp_time=5
runtime=60
group_reporting=1
numjobs=32
time_based
disk_util=0
clat_percentiles=0
disable_lat=1
disable_clat=1
disable_slat=1
filename=/mnt/test/nocow/fiofile
[test]
name=test
bs=4k
stonewall
#4 lore raw
Quoted 30 lines · Qu Wenruo
Quoted 16 lines · Mark Harmstone
Quoted 15 lines · David Sterba
Quoted 13 lines · Mark Harmstone

Before btrfs switched to the new mount API in 2023, we were setting SB_NOSEC in btrfs_mount_root(). This flag tells the VFS that the filesystem may have files which don't have security xattrs, enabling it to do some optimizations.

Unfortunately this was missed in the transition, meaning that IS_NOSEC will always return false for a btrfs inode. This means that btrfs_direct_write() calls will always get the inode lock exclusively, meaning that DIO writes to the same file will be serialized.

On my machine, this one-line change results in a ~59% improvement in DIO
throughput:

Quoted 6 lines
Quoted 3 lines

That's quite an improvement. What's the actual fio script you've used? Also the DIO depends on the block group profile wrt the buffered fallback so that would be good to know too.

It is. There's a big dropoff in DIO write performance in 6.8 that we never recovered from.

There is the bounded page solution from iomap already, which will no longer fallback to buffered IO but to use extra page copy to make sure the final bio won't change its content halfway.

Quoted 5 lines

IIRC it's one extra flag and remove the btrfs' specific fallback checks, but I haven't yet verified the behavior/code.

Thanks,
Qu

That sounds like a different thing - this is just making it so that we're not forced to take the inode rwsem exclusively for each write.

Quoted 42 lines
Quoted 41 lines

I'm going to look into some sort of automated performance so this kind of thing can't happen casually.

This was on a VM with 8 cores and 8GB of RAM, with a real NVMe exposed through PCI passthrough. The figures for XFS and ext4 in comparison are both about ~3GB/s.

# cat go
#!/bin/bash
mkfs.btrfs -f /dev/nvme0n1
mount /dev/nvme0n1 /mnt/test
mkdir /mnt/test/nocow
chattr +C /mnt/test/nocow
fio /root/test.fio
# cat /root/test.fio
[global]
rw=randwrite
ioengine=io_uring
iodepth=64
size=1g
direct=1
startdelay=20
force_async=4
ramp_time=5
runtime=60
group_reporting=1
numjobs=32
time_based
disk_util=0
clat_percentiles=0
disable_lat=1
disable_clat=1
disable_slat=1
filename=/mnt/test/nocow/fiofile
[test]
name=test
bs=4k
stonewall
#5 lore raw
In reply to Mark Harmstone
Quoted 15 lines · Mark Harmstone
Quoted 14 lines · David Sterba
Quoted 12 lines · Mark Harmstone

Before btrfs switched to the new mount API in 2023, we were setting SB_NOSEC in btrfs_mount_root(). This flag tells the VFS that the filesystem may have files which don't have security xattrs, enabling it to do some optimizations.

Unfortunately this was missed in the transition, meaning that IS_NOSEC will always return false for a btrfs inode. This means that btrfs_direct_write() calls will always get the inode lock exclusively, meaning that DIO writes to the same file will be serialized.

On my machine, this one-line change results in a ~59% improvement in DIO throughput:

Quoted 7 lines
Quoted 3 lines

That's quite an improvement. What's the actual fio script you've used? Also the DIO depends on the block group profile wrt the buffered fallback so that would be good to know too.

It is. There's a big dropoff in DIO write performance in 6.8 that we never recovered from. I'm going to look into some sort of automated performance so this kind of thing can't happen casually.

As you've used fio for that, please add the test case to the fstests/tests/perf. Though the 'pref' tests haven't caught up I'd like to at least collect them in our btrfs/fstests repository.

Quoted 28 lines

This was on a VM with 8 cores and 8GB of RAM, with a real NVMe exposed through PCI passthrough. The figures for XFS and ext4 in comparison are both about ~3GB/s.

# cat go
#!/bin/bash
mkfs.btrfs -f /dev/nvme0n1
mount /dev/nvme0n1 /mnt/test
mkdir /mnt/test/nocow
chattr +C /mnt/test/nocow
fio /root/test.fio
# cat /root/test.fio
[global]
rw=randwrite
ioengine=io_uring
iodepth=64
size=1g
direct=1
startdelay=20
force_async=4
ramp_time=5
runtime=60
group_reporting=1
numjobs=32
time_based
disk_util=0
clat_percentiles=0
Quoted 8 lines
disable_lat=1
disable_clat=1
disable_slat=1
filename=/mnt/test/nocow/fiofile
[test]
name=test
bs=4k
stonewall

Thanks.

#6 lore raw
In reply to Mark Harmstone
Quoted 54 lines · Mark Harmstone

Before btrfs switched to the new mount API in 2023, we were setting SB_NOSEC in btrfs_mount_root(). This flag tells the VFS that the filesystem may have files which don't have security xattrs, enabling it to do some optimizations.

Unfortunately this was missed in the transition, meaning that IS_NOSEC will always return false for a btrfs inode. This means that btrfs_direct_write() calls will always get the inode lock exclusively, meaning that DIO writes to the same file will be serialized.

On my machine, this one-line change results in a ~59% improvement in DIO throughput:

Before patch:

test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=64
...
fio-3.39
Starting 32 processes
test: Laying out IO file (1 file / 1024MiB)
Jobs: 32 (f=32): [w(32)][100.0%][w=764MiB/s][w=195k IOPS][eta 00m:00s]
test: (groupid=0, jobs=32): err= 0: pid=586: Wed Apr 22 13:03:04 2026

    write: IOPS=202k, BW=787MiB/s (826MB/s)(46.1GiB/60012msec); 0 zone resets
     bw (  KiB/s): min=498714, max=1199892, per=100.00%, avg=806659.03, stdev=4229.94, samples=3808
     iops        : min=124677, max=299971, avg=201661.82, stdev=1057.49, samples=3808
    cpu          : usr=0.32%, sys=1.27%, ctx=8329204, majf=0, minf=1163
    IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
       issued rwts: total=0,12094328,0,0 short=0,0,0,0 dropped=0,0,0,0
       latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):

    WRITE: bw=787MiB/s (826MB/s), 787MiB/s-787MiB/s (826MB/s-826MB/s), io=46.1GiB (49.5GB), run=60012-60012msec

After patch:

test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=64
...
fio-3.39
Starting 32 processes
test: Laying out IO file (1 file / 1024MiB)
Jobs: 32 (f=32): [w(32)][100.0%][w=1255MiB/s][w=321k IOPS][eta 00m:00s]
test: (groupid=0, jobs=32): err= 0: pid=572: Wed Apr 22 13:13:46 2026

    write: IOPS=320k, BW=1250MiB/s (1311MB/s)(73.3GiB/60003msec); 0 zone resets
     bw (  MiB/s): min=  619, max= 2289, per=100.00%, avg=1251.28, stdev= 9.64, samples=3808
     iops        : min=158538, max=586025, avg=320320.80, stdev=2468.97, samples=3808
    cpu          : usr=0.35%, sys=11.50%, ctx=1584847, majf=0, minf=1160
    IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
       issued rwts: total=0,19203309,0,0 short=0,0,0,0 dropped=0,0,0,0
       latency   : target=0, window=0, percentile=100.00%, depth=64
Quoted 5 lines

Run status group 0 (all jobs):

    WRITE: bw=1250MiB/s (1311MB/s), 1250MiB/s-1250MiB/s (1311MB/s-1311MB/s), io=73.3GiB (78.7GB), run=60003-60003msec
Fixes ad21f15b0f79 ("btrfs: switch to the new mount API")

I've updated changelog with the fio script and added the patch to for-next. We want to get this backported to stable trees, ETA 2 weeks so we get some coverage.