Skip to contentSkip to conversation

balance failed with checksum error

DISCUSSION Ulli Horlacher 3 participants 4 messages Jul 26, 2025 – Jul 28, 2025 Btrfs file system

lore
Share & tools
lore
#0 lore raw

This is just a test VM, but this error should not happen?

root@mux22:~# btrfs balance start /
WARNING:

Full balance without filters requested. This operation is very intense and takes potentially very long. It is recommended to use the balance filters to narrow down the scope of balance. Use 'btrfs balance start --full-balance' option to skip this warning. The operation will start in 10 seconds. Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
ERROR: error during balancing '/': Input/output error
There may be more info in syslog - try dmesg | tail

root@mux22:~# dmesg | tail
[22612.591075] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 806 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591095] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 805 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591115] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 257 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591184] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 810 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591204] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 809 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591224] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 808 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591244] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 807 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591264] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 806 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591283] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 805 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22613.689939] BTRFS info (device sda3): balance: ended with status: -5
root@mux22:~# btrfs version
btrfs-progs v6.6.3
root@quak:~# sysinfo 
System:        Linux quak 6.8.0-64-generic x86_64
Distribution:  Linux Mint 22.1
root@mux22:~# df -TH /
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/sda3      btrfs   32G   17G   16G  52% /
root@mux22:~# mount | grep sda3
/dev/sda3 on / type btrfs (rw,relatime,space_cache=v2,subvolid=256,subvol=/@)
root@mux22:~# btrfs filesystem show /
Label: none  uuid: 13168cc9-1fde-4d4a-8af1-fc91da4f12db
        Total devices 1 FS bytes used 14.15GiB
        devid    1 size 29.76GiB used 17.06GiB path /dev/sda3
root@mux22:~# btrfs filesystem df /
Data, single: total=15.00GiB, used=13.39GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=784.42MiB
root@mux22:~# btrfs filesystem usage /
Overall:
    Device size:                  29.76GiB
    Device allocated:             17.06GiB
    Device unallocated:           12.69GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                         14.92GiB
    Free (estimated):             14.31GiB      (min: 7.96GiB)
    Free (statfs, df):            14.31GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               37.58MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:15.00GiB, Used:13.39GiB (89.24%)
/dev/sda3 15.00GiB

Metadata,DUP: Size:1.00GiB, Used:784.42MiB (76.60%)
/dev/sda3 2.00GiB

System,DUP: Size:32.00MiB, Used:16.00KiB (0.05%)
/dev/sda3 64.00MiB

Unallocated:
/dev/sda3 12.69GiB

Signature · 6 lines
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<[email protected]>
#1 lore raw
Quoted 17 lines · Ulli Horlacher

This is just a test VM, but this error should not happen?

root@mux22:~# btrfs balance start /
WARNING:

Full balance without filters requested. This operation is very intense and takes potentially very long. It is recommended to use the balance filters to narrow down the scope of balance. Use 'btrfs balance start --full-balance' option to skip this warning. The operation will start in 10 seconds. Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
ERROR: error during balancing '/': Input/output error
There may be more info in syslog - try dmesg | tail

Quoted 10 lines
root@mux22:~# dmesg | tail
[22612.591075] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 806 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591095] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 805 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591115] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 257 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591184] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 810 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591204] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 809 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591224] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 808 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591244] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 807 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591264] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 806 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591283] BTRFS warning (device sda3): checksum error at logical 3450912768 mirror 1 root 805 inode 1509 offset 299008 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)

So it's the same file with data checksum, but shared by tons of different subvolume.

Data checksum mismatch is known to happen for certain workload. E.g. direct IO with buffer modification during writeback.

Thus upstream commit 968f19c5b1b7 ("btrfs: always fallback to buffered write if the inode requires checksum") fix this hole, at the cost of performance for those direct IOs.

And that commit is only a preventive method, it won't fix any already corruption already on-disk.

Another possibility is, it's just a real corruption.

Either way the best solution is to remove all offending files.

Quoted 7 lines
[22613.689939] BTRFS info (device sda3): balance: ended with status: -5
root@mux22:~# btrfs version
btrfs-progs v6.6.3
root@quak:~# sysinfo
System:        Linux quak 6.8.0-64-generic x86_64

This kernel is EOL, thus it will not that fix unless the vendor is actively doing backport.

Thanks,
Qu

Quoted 46 lines

Distribution: Linux Mint 22.1

root@mux22:~# df -TH /
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/sda3      btrfs   32G   17G   16G  52% /
root@mux22:~# mount | grep sda3
/dev/sda3 on / type btrfs (rw,relatime,space_cache=v2,subvolid=256,subvol=/@)
root@mux22:~# btrfs filesystem show /
Label: none  uuid: 13168cc9-1fde-4d4a-8af1-fc91da4f12db
         Total devices 1 FS bytes used 14.15GiB
         devid    1 size 29.76GiB used 17.06GiB path /dev/sda3
root@mux22:~# btrfs filesystem df /
Data, single: total=15.00GiB, used=13.39GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=784.42MiB
root@mux22:~# btrfs filesystem usage /
Overall:
     Device size:                  29.76GiB
     Device allocated:             17.06GiB
     Device unallocated:           12.69GiB
     Device missing:                  0.00B
     Device slack:                    0.00B
     Used:                         14.92GiB
     Free (estimated):             14.31GiB      (min: 7.96GiB)
     Free (statfs, df):            14.31GiB
     Data ratio:                       1.00
     Metadata ratio:                   2.00
     Global reserve:               37.58MiB      (used: 0.00B)
     Multiple profiles:                  no

Data,single: Size:15.00GiB, Used:13.39GiB (89.24%)

    /dev/sda3      15.00GiB

Metadata,DUP: Size:1.00GiB, Used:784.42MiB (76.60%)

    /dev/sda3       2.00GiB

System,DUP: Size:32.00MiB, Used:16.00KiB (0.05%)

    /dev/sda3      64.00MiB

Unallocated:

    /dev/sda3      12.69GiB
#2 lore raw
Quoted 1 line · Qu Wenruo
Quoted 1 line
root@mux22:~# btrfs balance start /

(...)

Quoted 10 lines
Quoted 7 lines

Starting balance without any filters.
ERROR: error during balancing '/': Input/output error There may be more info in syslog - try dmesg | tail

root@mux22:~# dmesg | tail
[22612.591075] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 806 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)
[22612.591095] BTRFS warning (device sda3): checksum error at logical 3450908672 mirror 1 root 805 inode 1509 offset 294912 length 4096 links 1 (path: tux/.cache/mesa_shader_cache/index)

So it's the same file with data checksum, but shared by tons of different subvolume.

Tons? Only 6:

root@mux22:~# btrfs subvolume list /
ID 256 gen 414 top level 5 path .snapshot/2025-07-27_0843--2025-07-30_0843.boot
ID 257 gen 414 top level 5 path .snapshot/2025-07-27_0849--2025-07-30_0849.boot
ID 258 gen 414 top level 5 path .snapshot/2025-07-27_0855--2025-07-30_0855.boot
ID 259 gen 414 top level 5 path .snapshot/2025-07-27_0856--2025-07-30_0856.boot
ID 260 gen 414 top level 5 path .snapshot/2025-07-27_0910--2025-07-30_0910.boot
ID 261 gen 414 top level 5 path .snapshot/2025-07-27_1816--2025-07-30_1816.boot
Quoted 1 line

Another possibility is, it's just a real corruption.

You mean a hardware (disk) failure?
This is very unlikely, because my VM runs on a high-end NetApp RAID.

Quoted 4 lines

Either way the best solution is to remove all offending files.

Quoted 2 lines
[22613.689939] BTRFS info (device sda3): balance: ended with status: -5
Quoted 8 lines
Quoted 5 lines
root@mux22:~# btrfs version
btrfs-progs v6.6.3
root@quak:~# sysinfo
System:        Linux quak 6.8.0-64-generic x86_64

This kernel is EOL, thus it will not that fix unless the vendor is actively doing backport.

Meanwhile I have upgraded the kernel:

root@mux22:~# uname -a
Linux mux22 6.14.0-24-generic #24~24.04.3-Ubuntu SMP PREEMPT_DYNAMIC Mon Jul  7 16:39:17 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

And now there are no more error(s):

root@mux22:~# btrfs balance start /
(...)
Starting balance without any filters.
Done, had to relocate 13 out of 13 chunks
root@mux22:~# dmesg | tail
[  163.838292] BTRFS info (device sda4): found 48242 extents, stage: update data pointers
[  164.806512] BTRFS info (device sda4): relocating block group 298844160 flags data
[  174.888393] BTRFS info (device sda4): found 2159 extents, stage: move data extents
[  175.554257] BTRFS info (device sda4): found 2159 extents, stage: update data pointers
[  176.008390] BTRFS info (device sda4): relocating block group 30408704 flags metadata|dup
[  179.472547] BTRFS info (device sda4): found 6326 extents, stage: move data extents
[  179.871047] BTRFS info (device sda4): relocating block group 13631488 flags data
[  180.078065] BTRFS info (device sda4): found 592 extents, stage: move data extents
[  180.226779] BTRFS info (device sda4): found 592 extents, stage: update data pointers
[  180.307905] BTRFS info (device sda4): balance: ended with status: 0
Signature · 6 lines
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<[email protected]>
#3 lore raw
In reply to Qu Wenruo

Hi,

Quoted 6 lines · Qu Wenruo

Data checksum mismatch is known to happen for certain workload. E.g. direct IO with buffer modification during writeback.

Thus upstream commit 968f19c5b1b7 ("btrfs: always fallback to buffered write if the inode requires checksum") fix this hole, at the cost of performance for those direct IOs.

6.12.40 is the latest LTS kernel, but it doesn't have this fix.

Quoted 2 lines · Ulli Horlacher
Quoted 2 lines
Quoted 2 lines
root@quak:~# sysinfo
System:        Linux quak 6.8.0-64-generic x86_64

[snip]

Quoted 6 lines

Meanwhile I have upgraded the kernel:

root@mux22:~# uname -a
Linux mux22 6.14.0-24-generic #24~24.04.3-Ubuntu SMP PREEMPT_DYNAMIC Mon Jul  7 16:39:17 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

And now there are no more error(s):

I guess this means the patch was applied to 6.14 as gregkh requested, and it just needs to be resent for 6.12.x?

https://yhbt.net/lore/stable/968f19c5b1b7d5595423b0ac0020cc18dfed8cb5.1746665263.git.wqu@suse.com/T/#m6a70c86370f8a22ab6559704b4edbaed067763a0

Cheers,
Nicholas