Skip to contentSkip to conversation

scrub on low disk space leads to Transaction aborted

bug Anand Jain 2 participants 3 messages Jan 9, 2021 – Jan 11, 2021 Btrfs file system

lore
Share & tools
lore

Anand Jain

author
#0 lore raw

I ran scrub when disk space was low on a btrfs with some raid1 csum errors, on a system with kernel 5.11.0-rc2+. It lead to transaction aborted and rdonly FS.

[ 2365.314375] ------------[ cut here ]------------
[ 2365.314385] BTRFS: Transaction aborted (error -28)
[ 2365.314470] WARNING: CPU: 2 PID: 4672 at fs/btrfs/inode.c:2823 btrfs_finish_ordered_io+0x8e6/0x950 [btrfs]
<snap>
[ 2365.314921] CPU: 2 PID: 4672 Comm: kworker/u8:11 Not tainted 5.11.0-rc2+ #3
[ 2365.314933] Hardware name: LENOVO 20FAS6PN00/20FAS6PN00, BIOS N1CET47W (1.15 ) 08/08/2016
[ 2365.314939] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[ 2365.315093] RIP: 0010:btrfs_finish_ordered_io+0x8e6/0x950 [btrfs]
<snap>
[ 2365.315290] Call Trace:
[ 2365.315309] BTRFS warning (device sda6): btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left).
[ 2365.315305]  finish_ordered_fn+0x15/0x20 [btrfs]
[ 2365.315532]  btrfs_work_helper+0xcc/0x2f0 [btrfs]
[ 2365.315763]  process_one_work+0x1ee/0x390
[ 2365.315781]  worker_thread+0x50/0x3d0
[ 2365.315797]  kthread+0x114/0x150
[ 2365.315811]  ? process_one_work+0x390/0x390
[ 2365.315824]  ? kthread_park+0x90/0x90
[ 2365.315836]  ret_from_fork+0x22/0x30
[ 2365.315861] ---[ end trace b0e289a2c3d424e8 ]---
[ 2365.315871] BTRFS warning (device sda6): btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left).
[ 2365.316713] BTRFS warning (device sda6): btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left).
[ 2365.316796] BTRFS warning (device sda6): Skipping commit of aborted transaction.
[ 2365.316809] BTRFS: error (device sda6) in cleanup_transaction:1938: errno=-30 Readonly filesystem
[ 2365.316824] BTRFS info (device sda6): forced readonly
[ 2365.359195] BTRFS info (device sda6): scrub: not finished on devid 2 with status: -125
#1 lore raw
Quoted 5 lines · Anand Jain

 I ran scrub when disk space was low on a btrfs with some raid1 csum errors, on a system with kernel 5.11.0-rc2+. It lead to transaction aborted and rdonly FS.

Would you please provide btrfs fi usage output?

There is a bug in recent kernel that global rsv is not increased properly so that we have a much higher chance to exhaust metadata space.

Thanks,
Qu

Quoted 35 lines
[ 2365.314375] ------------[ cut here ]------------
[ 2365.314385] BTRFS: Transaction aborted (error -28)
[ 2365.314470] WARNING: CPU: 2 PID: 4672 at fs/btrfs/inode.c:2823
btrfs_finish_ordered_io+0x8e6/0x950 [btrfs]
<snap>
[ 2365.314921] CPU: 2 PID: 4672 Comm: kworker/u8:11 Not tainted
5.11.0-rc2+ #3
[ 2365.314933] Hardware name: LENOVO 20FAS6PN00/20FAS6PN00, BIOS
N1CET47W (1.15 ) 08/08/2016
[ 2365.314939] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[ 2365.315093] RIP: 0010:btrfs_finish_ordered_io+0x8e6/0x950 [btrfs]
<snap>
[ 2365.315290] Call Trace:
[ 2365.315309] BTRFS warning (device sda6):
btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left).
[ 2365.315305]  finish_ordered_fn+0x15/0x20 [btrfs]
[ 2365.315532]  btrfs_work_helper+0xcc/0x2f0 [btrfs]
[ 2365.315763]  process_one_work+0x1ee/0x390
[ 2365.315781]  worker_thread+0x50/0x3d0
[ 2365.315797]  kthread+0x114/0x150
[ 2365.315811]  ? process_one_work+0x390/0x390
[ 2365.315824]  ? kthread_park+0x90/0x90
[ 2365.315836]  ret_from_fork+0x22/0x30
[ 2365.315861] ---[ end trace b0e289a2c3d424e8 ]---
[ 2365.315871] BTRFS warning (device sda6):
btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left).
[ 2365.316713] BTRFS warning (device sda6):
btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left).
[ 2365.316796] BTRFS warning (device sda6): Skipping commit of aborted
transaction.
[ 2365.316809] BTRFS: error (device sda6) in cleanup_transaction:1938:
errno=-30 Readonly filesystem
[ 2365.316824] BTRFS info (device sda6): forced readonly
[ 2365.359195] BTRFS info (device sda6): scrub: not finished on devid 2

with status: -125

author
#2 lore raw
Quoted 9 lines · Qu Wenruo
Quoted 5 lines · Anand Jain

  I ran scrub when disk space was low on a btrfs with some raid1 csum errors, on a system with kernel 5.11.0-rc2+. It lead to transaction aborted and rdonly FS.

Quoted 5 lines

Would you please provide btrfs fi usage output?

There is a bug in recent kernel that global rsv is not increased properly so that we have a much higher chance to exhaust metadata space.

Now I have made some space in the filesystem. So the usage below may not be relevant at the time of the problem.

# btrfs fi usage /Volumes/
Overall:
    Device size:		 154.97GiB
    Device allocated:		 153.12GiB
    Device unallocated:		   1.85GiB
    Device missing:		  76.56GiB
    Used:			 147.63GiB
    Free (estimated):		   3.35GiB	(min: 3.35GiB)
    Data ratio:			      2.00
    Metadata ratio:		      2.00
    Global reserve:		 181.56MiB	(used: 0.00B)

Data,RAID1: Size:75.53GiB, Used:73.11GiB (96.79%)
/dev/sda6 75.53GiB
/dev/sdb2 75.53GiB

Metadata,RAID1: Size:1.00GiB, Used:724.14MiB (70.72%)
/dev/sda6 1.00GiB
/dev/sdb2 1.00GiB

System,RAID1: Size:32.00MiB, Used:16.00KiB (0.05%)
/dev/sda6 32.00MiB
/dev/sdb2 32.00MiB

Unallocated:
/dev/sda6 1.85GiB
/dev/sdb2 1.00MiB

Quoted 38 lines

Thanks,
Qu

Quoted 35 lines
[ 2365.314375] ------------[ cut here ]------------
[ 2365.314385] BTRFS: Transaction aborted (error -28)
[ 2365.314470] WARNING: CPU: 2 PID: 4672 at fs/btrfs/inode.c:2823
btrfs_finish_ordered_io+0x8e6/0x950 [btrfs]
<snap>
[ 2365.314921] CPU: 2 PID: 4672 Comm: kworker/u8:11 Not tainted
5.11.0-rc2+ #3
[ 2365.314933] Hardware name: LENOVO 20FAS6PN00/20FAS6PN00, BIOS
N1CET47W (1.15 ) 08/08/2016
[ 2365.314939] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[ 2365.315093] RIP: 0010:btrfs_finish_ordered_io+0x8e6/0x950 [btrfs]
<snap>
[ 2365.315290] Call Trace:
[ 2365.315309] BTRFS warning (device sda6):
btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left).
[ 2365.315305]  finish_ordered_fn+0x15/0x20 [btrfs]
[ 2365.315532]  btrfs_work_helper+0xcc/0x2f0 [btrfs]
[ 2365.315763]  process_one_work+0x1ee/0x390
[ 2365.315781]  worker_thread+0x50/0x3d0
[ 2365.315797]  kthread+0x114/0x150
[ 2365.315811]  ? process_one_work+0x390/0x390
[ 2365.315824]  ? kthread_park+0x90/0x90
[ 2365.315836]  ret_from_fork+0x22/0x30
[ 2365.315861] ---[ end trace b0e289a2c3d424e8 ]---
[ 2365.315871] BTRFS warning (device sda6):
btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left).
[ 2365.316713] BTRFS warning (device sda6):
btrfs_finish_ordered_io:2823: Aborting unused transaction(No space left).
[ 2365.316796] BTRFS warning (device sda6): Skipping commit of aborted
transaction.
[ 2365.316809] BTRFS: error (device sda6) in cleanup_transaction:1938:
errno=-30 Readonly filesystem
[ 2365.316824] BTRFS info (device sda6): forced readonly
[ 2365.359195] BTRFS info (device sda6): scrub: not finished on devid 2

with status: -125