balance failed with checksum error
This is just a test VM, but this error should not happen?
root@mux22:~# btrfs balance start / WARNING:
Full balance without filters requested. This operation is very intense and takes potentially very long. It is recommended to use the balance filters to narrow down the scope of balance. Use 'btrfs balance start --full-balance' option to skip this warning. The operation will start in 10 seconds. Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
ERROR: error during balancing '/': Input/output error
There may be more info in syslog - try dmesg | tail
root@mux22:~# dmesg | tail [22612.591075] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 806 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591095] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 805 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591115] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 257 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591184] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 810 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591204] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 809 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591224] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 808 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591244] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 807 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591264] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 806 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591283] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 805 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22613.689939] BTRFS info (device sda3): balance: ended with status: -5
root@mux22:~# btrfs version btrfs-progs v6.6.3
root@quak:~# sysinfo System: Linux quak 6.8.0-64-generic x86_64 Distribution: Linux Mint 22.1
root@mux22:~# df -TH / Filesystem Type Size Used Avail Use% Mounted on /dev/sda3 btrfs 32G 17G 16G 52% /
root@mux22:~# mount | grep sda3 /dev/sda3 on / type btrfs (rw,relatime,space_cache=v2,subvolid=256,subvol=/@)
root@mux22:~# btrfs filesystem show /
Label: none uuid: 13168cc9-1fde-4d4a-8af1-fc91da4f12db
Total devices 1 FS bytes used 14.15GiB
devid 1 size 29.76GiB used 17.06GiB path /dev/sda3root@mux22:~# btrfs filesystem df / Data, single: total=15.00GiB, used=13.39GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.00GiB, used=784.42MiB
root@mux22:~# btrfs filesystem usage /
Overall:
Device size: 29.76GiB
Device allocated: 17.06GiB
Device unallocated: 12.69GiB
Device missing: 0.00B
Device slack: 0.00B
Used: 14.92GiB
Free (estimated): 14.31GiB (min: 7.96GiB)
Free (statfs, df): 14.31GiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 37.58MiB (used: 0.00B)
Multiple profiles: noData,single: Size:15.00GiB, Used:13.39GiB (89.24%)
/dev/sda3 15.00GiB
Metadata,DUP: Size:1.00GiB, Used:784.42MiB (76.60%)
/dev/sda3 2.00GiB
System,DUP: Size:32.00MiB, Used:16.00KiB (0.05%)
/dev/sda3 64.00MiB
Unallocated:
/dev/sda3 12.69GiB
Signature · 6 lines
Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: [email protected] Allmandring 30a Tel: ++49-711-68565868 70569 Stuttgart (Germany) WWW: https://www.tik.uni-stuttgart.de/ REF:<[email protected]>
Quoted 17 lines · Ulli Horlacher
This is just a test VM, but this error should not happen?
root@mux22:~# btrfs balance start / WARNING:Full balance without filters requested. This operation is very intense and takes potentially very long. It is recommended to use the balance filters to narrow down the scope of balance. Use 'btrfs balance start --full-balance' option to skip this warning. The operation will start in 10 seconds. Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
ERROR: error during balancing '/': Input/output error
There may be more info in syslog - try dmesg | tail
Quoted 10 lines
root@mux22:~# dmesg | tail [22612.591075] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 806 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591095] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 805 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591115] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 257 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591184] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 810 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591204] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 809 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591224] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 808 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591244] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 807 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591264] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 806 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591283] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 805 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
So it's the same file with data checksum, but shared by tons of different subvolume.
Data checksum mismatch is known to happen for certain workload. E.g. direct IO with buffer modification during writeback.
Thus upstream commit 968f19c5b1b7 ("btrfs: always fallback to buffered write if the inode requires checksum") fix this hole, at the cost of performance for those direct IOs.
And that commit is only a preventive method, it won't fix any already corruption already on-disk.
Another possibility is, it's just a real corruption.
Either way the best solution is to remove all offending files.
Quoted 7 lines
[22613.689939] BTRFS info (device sda3): balance: ended with status: -5root@mux22:~# btrfs version btrfs-progs v6.6.3root@quak:~# sysinfo System: Linux quak 6.8.0-64-generic x86_64
This kernel is EOL, thus it will not that fix unless the vendor is actively doing backport.
Thanks,
Qu
Quoted 46 lines
Distribution: Linux Mint 22.1
root@mux22:~# df -TH / Filesystem Type Size Used Avail Use% Mounted on /dev/sda3 btrfs 32G 17G 16G 52% /root@mux22:~# mount | grep sda3 /dev/sda3 on / type btrfs (rw,relatime,space_cache=v2,subvolid=256,subvol=/@)root@mux22:~# btrfs filesystem show / Label: none uuid: 13168cc9-1fde-4d4a-8af1-fc91da4f12db Total devices 1 FS bytes used 14.15GiB devid 1 size 29.76GiB used 17.06GiB path /dev/sda3root@mux22:~# btrfs filesystem df / Data, single: total=15.00GiB, used=13.39GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.00GiB, used=784.42MiBroot@mux22:~# btrfs filesystem usage / Overall: Device size: 29.76GiB Device allocated: 17.06GiB Device unallocated: 12.69GiB Device missing: 0.00B Device slack: 0.00B Used: 14.92GiB Free (estimated): 14.31GiB (min: 7.96GiB) Free (statfs, df): 14.31GiB Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 37.58MiB (used: 0.00B) Multiple profiles: noData,single: Size:15.00GiB, Used:13.39GiB (89.24%)
/dev/sda3 15.00GiBMetadata,DUP: Size:1.00GiB, Used:784.42MiB (76.60%)
/dev/sda3 2.00GiBSystem,DUP: Size:32.00MiB, Used:16.00KiB (0.05%)
/dev/sda3 64.00MiBUnallocated:
/dev/sda3 12.69GiB
Quoted 1 line · Qu Wenruo
Quoted 1 line
root@mux22:~# btrfs balance start /
(...)
Quoted 10 lines
Quoted 7 lines
Starting balance without any filters.
ERROR: error during balancing '/': Input/output error There may be more info in syslog - try dmesg | tailroot@mux22:~# dmesg | tail [22612.591075] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 806 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index) [22612.591095] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 805 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)So it's the same file with data checksum, but shared by tons of different subvolume.
Tons? Only 6:
root@mux22:~# btrfs subvolume list / ID 256 gen 414 top level 5 path .snapshot/2025-07-27_0843--2025-07-30_0843.boot ID 257 gen 414 top level 5 path .snapshot/2025-07-27_0849--2025-07-30_0849.boot ID 258 gen 414 top level 5 path .snapshot/2025-07-27_0855--2025-07-30_0855.boot ID 259 gen 414 top level 5 path .snapshot/2025-07-27_0856--2025-07-30_0856.boot ID 260 gen 414 top level 5 path .snapshot/2025-07-27_0910--2025-07-30_0910.boot ID 261 gen 414 top level 5 path .snapshot/2025-07-27_1816--2025-07-30_1816.boot
Quoted 1 line
Another possibility is, it's just a real corruption.
You mean a hardware (disk) failure?
This is very unlikely, because my VM runs on a high-end NetApp RAID.
Quoted 4 lines
Either way the best solution is to remove all offending files.
Quoted 2 lines
[22613.689939] BTRFS info (device sda3): balance: ended with status: -5
Quoted 8 lines
Quoted 5 lines
root@mux22:~# btrfs version btrfs-progs v6.6.3root@quak:~# sysinfo System: Linux quak 6.8.0-64-generic x86_64This kernel is EOL, thus it will not that fix unless the vendor is actively doing backport.
Meanwhile I have upgraded the kernel:
root@mux22:~# uname -a Linux mux22 6.14.0-24-generic #24~24.04.3-Ubuntu SMP PREEMPT_DYNAMIC Mon Jul 7 16:39:17 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
And now there are no more error(s):
root@mux22:~# btrfs balance start / (...) Starting balance without any filters. Done, had to relocate 13 out of 13 chunks
root@mux22:~# dmesg | tail [ 163.838292] BTRFS info (device sda4): found 48242 extents, stage: update data pointers [ 164.806512] BTRFS info (device sda4): relocating block group 298844160 flags data [ 174.888393] BTRFS info (device sda4): found 2159 extents, stage: move data extents [ 175.554257] BTRFS info (device sda4): found 2159 extents, stage: update data pointers [ 176.008390] BTRFS info (device sda4): relocating block group 30408704 flags metadata|dup [ 179.472547] BTRFS info (device sda4): found 6326 extents, stage: move data extents [ 179.871047] BTRFS info (device sda4): relocating block group 13631488 flags data [ 180.078065] BTRFS info (device sda4): found 592 extents, stage: move data extents [ 180.226779] BTRFS info (device sda4): found 592 extents, stage: update data pointers [ 180.307905] BTRFS info (device sda4): balance: ended with status: 0
Signature · 6 lines
Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: [email protected] Allmandring 30a Tel: ++49-711-68565868 70569 Stuttgart (Germany) WWW: https://www.tik.uni-stuttgart.de/ REF:<[email protected]>
Hi,
Quoted 6 lines · Qu Wenruo
Data checksum mismatch is known to happen for certain workload. E.g. direct IO with buffer modification during writeback.
Thus upstream commit
968f19c5b1b7("btrfs: always fallback to buffered write if the inode requires checksum") fix this hole, at the cost of performance for those direct IOs.
6.12.40 is the latest LTS kernel, but it doesn't have this fix.
Quoted 2 lines · Ulli Horlacher
Quoted 2 lines
Quoted 2 lines
root@quak:~# sysinfo System: Linux quak 6.8.0-64-generic x86_64
[snip]
Quoted 6 lines
Meanwhile I have upgraded the kernel:
root@mux22:~# uname -a Linux mux22 6.14.0-24-generic #24~24.04.3-Ubuntu SMP PREEMPT_DYNAMIC Mon Jul 7 16:39:17 UTC 2 x86_64 x86_64 x86_64 GNU/LinuxAnd now there are no more error(s):
I guess this means the patch was applied to 6.14 as gregkh requested, and it just needs to be resent for 6.12.x?
Cheers,
Nicholas