scrub on low disk space leads to Transaction aborted
I ran scrub when disk space was low on a btrfs with some raid1 csum errors, on a system with kernel 5.11.0-rc2+. It lead to transaction aborted and rdonly FS.
[ 2365.314375] ------------[ cut here ]------------ [ 2365.314385] BTRFS: Transaction aborted (error -28) [ 2365.314470] WARNING: CPU: 2 PID: 4672 at fs/btrfs/inode.c:2823 btrfs_finish_ordered_io+0x8e6/0x950 [btrfs] <snap> [ 2365.314921] CPU: 2 PID: 4672 Comm: kworker/u8:11 Not tainted 5.11.0-rc2+ #3 [ 2365.314933] Hardware name: LENOVO 20FAS6PN00/20FAS6PN00, BIOS N1CET47W (1.15 ) 08/08/2016 [ 2365.314939] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs] [ 2365.315093] RIP: 0010:btrfs_finish_ordered_io+0x8e6/0x950 [btrfs] <snap> [ 2365.315290] Call Trace: [ 2365.315309] BTRFS warning (device sda6): btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left). [ 2365.315305] finish_ordered_fn+0x15/0x20 [btrfs] [ 2365.315532] btrfs_work_helper+0xcc/0x2f0 [btrfs] [ 2365.315763] process_one_work+0x1ee/0x390 [ 2365.315781] worker_thread+0x50/0x3d0 [ 2365.315797] kthread+0x114/0x150 [ 2365.315811] ? process_one_work+0x390/0x390 [ 2365.315824] ? kthread_park+0x90/0x90 [ 2365.315836] ret_from_fork+0x22/0x30 [ 2365.315861] ---[ end trace b0e289a2c3d424e8 ]--- [ 2365.315871] BTRFS warning (device sda6): btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left). [ 2365.316713] BTRFS warning (device sda6): btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left). [ 2365.316796] BTRFS warning (device sda6): Skipping commit of aborted transaction. [ 2365.316809] BTRFS: error (device sda6) in cleanup_transaction:1938: errno=-30 Readonly filesystem [ 2365.316824] BTRFS info (device sda6): forced readonly [ 2365.359195] BTRFS info (device sda6): scrub: not finished on devid 2 with status: -125
Quoted 5 lines · Anand Jain
I ran scrub when disk space was low on a btrfs with some raid1 csum errors, on a system with kernel 5.11.0-rc2+. It lead to transaction aborted and rdonly FS.
Would you please provide btrfs fi usage output?
There is a bug in recent kernel that global rsv is not increased properly so that we have a much higher chance to exhaust metadata space.
Thanks,
Qu
Quoted 35 lines
[ 2365.314375] ------------[ cut here ]------------ [ 2365.314385] BTRFS: Transaction aborted (error -28) [ 2365.314470] WARNING: CPU: 2 PID: 4672 at fs/btrfs/inode.c:2823 btrfs_finish_ordered_io+0x8e6/0x950 [btrfs] <snap> [ 2365.314921] CPU: 2 PID: 4672 Comm: kworker/u8:11 Not tainted 5.11.0-rc2+ #3 [ 2365.314933] Hardware name: LENOVO 20FAS6PN00/20FAS6PN00, BIOS N1CET47W (1.15 ) 08/08/2016 [ 2365.314939] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs] [ 2365.315093] RIP: 0010:btrfs_finish_ordered_io+0x8e6/0x950 [btrfs] <snap> [ 2365.315290] Call Trace: [ 2365.315309] BTRFS warning (device sda6): btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left). [ 2365.315305] finish_ordered_fn+0x15/0x20 [btrfs] [ 2365.315532] btrfs_work_helper+0xcc/0x2f0 [btrfs] [ 2365.315763] process_one_work+0x1ee/0x390 [ 2365.315781] worker_thread+0x50/0x3d0 [ 2365.315797] kthread+0x114/0x150 [ 2365.315811] ? process_one_work+0x390/0x390 [ 2365.315824] ? kthread_park+0x90/0x90 [ 2365.315836] ret_from_fork+0x22/0x30 [ 2365.315861] ---[ end trace b0e289a2c3d424e8 ]--- [ 2365.315871] BTRFS warning (device sda6): btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left). [ 2365.316713] BTRFS warning (device sda6): btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left). [ 2365.316796] BTRFS warning (device sda6): Skipping commit of aborted transaction. [ 2365.316809] BTRFS: error (device sda6) in cleanup_transaction:1938: errno=-30 Readonly filesystem [ 2365.316824] BTRFS info (device sda6): forced readonly [ 2365.359195] BTRFS info (device sda6): scrub: not finished on devid 2with status: -125
Quoted 9 lines · Qu Wenruo
Quoted 5 lines · Anand Jain
I ran scrub when disk space was low on a btrfs with some raid1 csum errors, on a system with kernel 5.11.0-rc2+. It lead to transaction aborted and rdonly FS.
Quoted 5 lines
Would you please provide
btrfs fi usageoutput?There is a bug in recent kernel that global rsv is not increased properly so that we have a much higher chance to exhaust metadata space.
Now I have made some space in the filesystem. So the usage below may not be relevant at the time of the problem.
# btrfs fi usage /Volumes/
Overall:
Device size: 154.97GiB
Device allocated: 153.12GiB
Device unallocated: 1.85GiB
Device missing: 76.56GiB
Used: 147.63GiB
Free (estimated): 3.35GiB (min: 3.35GiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 181.56MiB (used: 0.00B)Data,RAID1: Size:75.53GiB, Used:73.11GiB (96.79%)
/dev/sda6 75.53GiB
/dev/sdb2 75.53GiB
Metadata,RAID1: Size:1.00GiB, Used:724.14MiB (70.72%)
/dev/sda6 1.00GiB
/dev/sdb2 1.00GiB
System,RAID1: Size:32.00MiB, Used:16.00KiB (0.05%)
/dev/sda6 32.00MiB
/dev/sdb2 32.00MiB
Unallocated:
/dev/sda6 1.85GiB
/dev/sdb2 1.00MiB
Quoted 38 lines
Thanks,
QuQuoted 35 lines
[ 2365.314375] ------------[ cut here ]------------ [ 2365.314385] BTRFS: Transaction aborted (error -28) [ 2365.314470] WARNING: CPU: 2 PID: 4672 at fs/btrfs/inode.c:2823 btrfs_finish_ordered_io+0x8e6/0x950 [btrfs] <snap> [ 2365.314921] CPU: 2 PID: 4672 Comm: kworker/u8:11 Not tainted 5.11.0-rc2+ #3 [ 2365.314933] Hardware name: LENOVO 20FAS6PN00/20FAS6PN00, BIOS N1CET47W (1.15 ) 08/08/2016 [ 2365.314939] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs] [ 2365.315093] RIP: 0010:btrfs_finish_ordered_io+0x8e6/0x950 [btrfs] <snap> [ 2365.315290] Call Trace: [ 2365.315309] BTRFS warning (device sda6): btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left). [ 2365.315305] finish_ordered_fn+0x15/0x20 [btrfs] [ 2365.315532] btrfs_work_helper+0xcc/0x2f0 [btrfs] [ 2365.315763] process_one_work+0x1ee/0x390 [ 2365.315781] worker_thread+0x50/0x3d0 [ 2365.315797] kthread+0x114/0x150 [ 2365.315811] ? process_one_work+0x390/0x390 [ 2365.315824] ? kthread_park+0x90/0x90 [ 2365.315836] ret_from_fork+0x22/0x30 [ 2365.315861] ---[ end trace b0e289a2c3d424e8 ]--- [ 2365.315871] BTRFS warning (device sda6): btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left). [ 2365.316713] BTRFS warning (device sda6): btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left). [ 2365.316796] BTRFS warning (device sda6): Skipping commit of aborted transaction. [ 2365.316809] BTRFS: error (device sda6) in cleanup_transaction:1938: errno=-30 Readonly filesystem [ 2365.316824] BTRFS info (device sda6): forced readonly [ 2365.359195] BTRFS info (device sda6): scrub: not finished on devid 2with status: -125