Skip to contentSkip to conversation

Computer stalled, apparently from filesystem corruption

DISCUSSION Pierre Abbat 2 participants 4 messages Mar 10, 2022 – Apr 4, 2022 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Sending again, with less syslog. I wrote this on the 8th.

My computer was up for several months. On Saturday I came home from church and found it unresponsive except that I could move the mouse cursor and make the keyboard lights blink. I tried to shell in from another computer; I got a connection, but not a login prompt. I rebooted it with the power button and copied the syslog file that contains the incident. I saw "btrfs" several times in the section of log.

This morning, I was rsyncing some files and the same thing happened, except that the screen was stuck in the middle of flipping through windows with alt- tab and I was able to switch to a console, log in as root, and reboot. Again I found "btrfs" in syslog.

Running dmesg produces too much information. Here is the rest of the info:

root@puma:/home/phma# uname -a
Linux puma 5.15.15-76051515-generic #202201160435~1642693824~20.04~97db1bb-
Ubuntu SMP Fri Jan 21 14: x86_64 x86_64 x86_64 GNU/Linux
root@puma:/home/phma# btrfs --version
btrfs-progs v5.4.1 
root@puma:/home/phma# btrfs fi show
Label: none  uuid: d0137494-fe35-488f-b0da-fdbe075b8832
        Total devices 1 FS bytes used 72.88GiB
        devid    1 size 673.51GiB used 75.02GiB path /dev/mapper/concolor-big

Label: none uuid: 174643ac-cb44-46a9-a5a4-5fb5e9a4e79f
Total devices 1 FS bytes used 48.84GiB
devid 1 size 100.00GiB used 57.02GiB path /dev/mapper/concolor-
rootcopy

Label: none uuid: 155a20c7-2264-4923-b082-288a3c146384
Total devices 1 FS bytes used 101.49GiB
devid 1 size 158.00GiB used 134.02GiB path /dev/mapper/concolor-
cougar

Label: none uuid: 10c61748-efe7-4b9c-b1f7-041dc45d894b
Total devices 1 FS bytes used 39.38GiB
devid 1 size 127.98GiB used 53.04GiB path /dev/mapper/cougar-crypt

Label: none uuid: 1f5a6f23-a7ef-46c6-92b1-84fc2f684931
Total devices 1 FS bytes used 92.58GiB
devid 1 size 158.00GiB used 131.00GiB path /dev/mapper/puma-cougar

root@puma:/home/phma# btrfs fi df /big
Data, single: total=74.01GiB, used=72.78GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=1.01GiB, used=101.59MiB
GlobalReserve, single: total=92.41MiB, used=0.00B
root@puma:/home/phma# btrfs fi df /rootcopy
Data, single: total=55.01GiB, used=47.68GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=2.01GiB, used=1.16GiB
GlobalReserve, single: total=113.17MiB, used=0.00B
root@puma:/home/phma# btrfs fi df /home
Data, single: total=132.01GiB, used=100.23GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=2.01GiB, used=1.26GiB
GlobalReserve, single: total=321.31MiB, used=0.00B
root@puma:/home/phma# btrfs fi df /crypt
Data, single: total=52.01GiB, used=39.29GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.00GiB, used=91.14MiB
GlobalReserve, single: total=45.91MiB, used=0.00B
root@puma:/home/phma# btrfs fi df /olv
Data, single: total=52.01GiB, used=39.29GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.00GiB, used=91.14MiB
GlobalReserve, single: total=45.91MiB, used=0.00B
root@puma:/home/phma# btrfs fi df /backup
Data, single: total=52.01GiB, used=39.29GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.00GiB, used=91.14MiB
GlobalReserve, single: total=45.91MiB, used=0.00B

The kernel I was running on Saturday may be older.

Here are some excerpts from syslog:

Mar  5 08:31:52 puma kernel: [10359910.035946] INFO: task Indexed~ #56145:1182901 blocked for more than 120 seconds.
Mar  5 08:31:52 puma kernel: [10359910.035948]       Tainted: P           OE     5.13.0-7614-generic #14~1631647151~20.04~930e87c-Ubuntu
Mar  5 08:31:52 puma kernel: [10359910.035949] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  5 08:31:52 puma kernel: [10359910.035950] task:Indexed~ #56145 state:D stack:    0 pid:1182901 ppid:     1 flags:0x00000000
Mar  5 08:31:52 puma kernel: [10359910.035953] Call Trace:
Mar  5 08:31:52 puma kernel: [10359910.035954]  __schedule+0x2ee/0x900
Mar  5 08:31:52 puma kernel: [10359910.035956]  ? try_to_unlazy+0x55/0x90
Mar  5 08:31:52 puma kernel: [10359910.035959]  schedule+0x4f/0xc0
Mar  5 08:31:52 puma kernel: [10359910.035964]  wait_current_trans+0xd6/0x140 [btrfs]
Mar  5 08:31:52 puma kernel: [10359910.036043]  ? wait_woken+0x80/0x80
Mar  5 08:31:52 puma kernel: [10359910.036047]  start_transaction+0x4bb/0x5a0 [btrfs]
Mar  5 08:31:52 puma kernel: [10359910.036081]  btrfs_start_transaction_fallback_global_rsv+0x1b/0x20 [btrfs]
Mar  5 08:31:52 puma kernel: [10359910.036113]  btrfs_unlink+0x30/0xe0 [btrfs]
Mar  5 08:31:52 puma kernel: [10359910.036145]  vfs_unlink+0x114/0x200
Mar  5 08:31:52 puma kernel: [10359910.036147]  do_unlinkat+0x1a2/0x2d0
Mar  5 08:31:52 puma kernel: [10359910.036150]  __x64_sys_unlink+0x23/0x30
Mar  5 08:31:52 puma kernel: [10359910.036153]  do_syscall_64+0x61/0xb0
Mar  5 08:31:52 puma kernel: [10359910.036155]  ? do_syscall_64+0x6e/0xb0
Mar  5 08:31:52 puma kernel: [10359910.036157]  ? do_syscall_64+0x6e/0xb0
Mar  5 08:31:52 puma kernel: [10359910.036159]  ? do_syscall_64+0x6e/0xb0
Mar  5 08:31:52 puma kernel: [10359910.036160]  ? do_syscall_64+0x6e/0xb0
Mar  5 08:31:52 puma kernel: [10359910.036162]  ? do_syscall_64+0x6e/0xb0
Mar  5 08:31:52 puma kernel: [10359910.036164]  ? asm_sysvec_apic_timer_interrupt+0xa/0x20
Mar  5 08:31:52 puma kernel: [10359910.036166]  entry_SYSCALL_64_after_hwframe+0x44/0xae
Mar  5 08:31:52 puma kernel: [10359910.036167] RIP: 0033:0x7fc30923de3b
Mar  5 08:31:52 puma kernel: [10359910.036169] RSP: 002b:00007fc2c2bbd4d8 EFLAGS: 00000202 ORIG_RAX: 0000000000000057
Mar  5 08:31:52 puma kernel: [10359910.036170] RAX: ffffffffffffffda RBX: 00007fc274b75271 RCX: 00007fc30923de3b
Mar  5 08:31:52 puma kernel: [10359910.036171] RDX: 0000000000000000 RSI: 00007fc274b75271 RDI: 00007fc274b75271
Mar  5 08:31:52 puma kernel: [10359910.036173] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007fc272f01748
Mar  5 08:31:52 puma kernel: [10359910.036174] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc274b75271
Mar  5 08:31:52 puma kernel: [10359910.036175] R13: 00007fc2734b4400 R14: 0000000000000000 R15: 00007fc3085bd578
Mar  8 04:36:29 puma kernel: [39151.885472] INFO: task tracker-store:53596 blocked for more than 120 seconds.
Mar  8 04:36:29 puma kernel: [39151.885492]       Tainted: G           OE     5.15.15-76051515-generic #202201160435~1642693824~20.04~97db1bb-Ubuntu
Mar  8 04:36:29 puma kernel: [39151.885494] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  8 04:36:29 puma kernel: [39151.885496] task:tracker-store   state:D stack:    0 pid:53596 ppid:  3662 flags:0x00000000
Mar  8 04:36:29 puma kernel: [39151.885503] Call Trace:
Mar  8 04:36:29 puma kernel: [39151.885508]  <TASK>
Mar  8 04:36:29 puma kernel: [39151.885518]  __schedule+0x2cd/0x890
Mar  8 04:36:29 puma kernel: [39151.885530]  ? __cond_resched+0x19/0x30
Mar  8 04:36:29 puma kernel: [39151.885533]  schedule+0x4e/0xb0
Mar  8 04:36:29 puma kernel: [39151.885539]  btrfs_sync_log+0x178/0xef0 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885704]  ? __raw_callee_save___native_queued_spin_unlock+0x15/0x23
Mar  8 04:36:29 puma kernel: [39151.885717]  ? release_extent_buffer+0xbb/0xe0 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885753]  ? wait_woken+0x60/0x60
Mar  8 04:36:29 puma kernel: [39151.885758]  ? btrfs_free_path+0x27/0x30 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885787]  ? dput+0x62/0x320
Mar  8 04:36:29 puma kernel: [39151.885794]  ? log_all_new_ancestors+0x3bc/0x470 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885841]  ? btrfs_log_inode_parent+0x2db/0x890 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885872]  ? join_transaction+0x135/0x4c0 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885900]  ? start_transaction+0xd5/0x5b0 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885931]  ? dput+0x62/0x320
Mar  8 04:36:29 puma kernel: [39151.885933]  btrfs_sync_file+0x33f/0x460 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885961]  vfs_fsync_range+0x49/0x80
Mar  8 04:36:29 puma kernel: [39151.885965]  do_fsync+0x3d/0x70
Mar  8 04:36:29 puma kernel: [39151.885967]  __x64_sys_fsync+0x14/0x20
Mar  8 04:36:29 puma kernel: [39151.885968]  do_syscall_64+0x5c/0xc0
Mar  8 04:36:29 puma kernel: [39151.885971]  ? do_sys_openat2+0x1d3/0x320
Mar  8 04:36:29 puma kernel: [39151.885975]  ? exit_to_user_mode_prepare+0x3d/0x1c0
Mar  8 04:36:29 puma kernel: [39151.885980]  ? syscall_exit_to_user_mode+0x27/0x50
Mar  8 04:36:29 puma kernel: [39151.885983]  ? __x64_sys_openat+0x20/0x30
Mar  8 04:36:29 puma kernel: [39151.885986]  ? do_syscall_64+0x69/0xc0
Mar  8 04:36:29 puma kernel: [39151.885987]  ? do_syscall_64+0x69/0xc0
Mar  8 04:36:29 puma kernel: [39151.885988]  entry_SYSCALL_64_after_hwframe+0x44/0xae
Mar  8 04:36:29 puma kernel: [39151.885993] RIP: 0033:0x7f0eb970832b
Mar  8 04:36:29 puma kernel: [39151.885995] RSP: 002b:00007ffde79e29b0 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
Mar  8 04:36:29 puma kernel: [39151.885997] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0eb970832b
Mar  8 04:36:29 puma kernel: [39151.885999] RDX: 0000000000101441 RSI: 0000564c61177b50 RDI: 0000000000000008
Mar  8 04:36:29 puma kernel: [39151.886000] RBP: 0000000000000008 R08: 0000000000000000 R09: 0000000000000001
Mar  8 04:36:29 puma kernel: [39151.886001] R10: 00000000000001b0 R11: 0000000000000293 R12: 0000564c61176e50
Mar  8 04:36:29 puma kernel: [39151.886002] R13: 00007ffde79e2a00 R14: 00007ffde79e2be4 R15: 0000000000000000
Mar  8 04:36:29 puma kernel: [39151.886005]  </TASK>

Please copy me, I'm not on the mailing list.

Pierre

Signature · 2 lines
The gostak pelled at the fostin lutt for darfs for her martle plave.
The darfs had smibbed, the lutt was thale, and the pilter had nothing snave.
#1 lore raw
Quoted 150 lines · Pierre Abbat

Sending again, with less syslog. I wrote this on the 8th.

My computer was up for several months. On Saturday I came home from church and found it unresponsive except that I could move the mouse cursor and make the keyboard lights blink. I tried to shell in from another computer; I got a connection, but not a login prompt. I rebooted it with the power button and copied the syslog file that contains the incident. I saw "btrfs" several times in the section of log.

This morning, I was rsyncing some files and the same thing happened, except that the screen was stuck in the middle of flipping through windows with alt- tab and I was able to switch to a console, log in as root, and reboot. Again I found "btrfs" in syslog.

Running dmesg produces too much information. Here is the rest of the info:

root@puma:/home/phma# uname -a
Linux puma 5.15.15-76051515-generic #202201160435~1642693824~20.04~97db1bb-
Ubuntu SMP Fri Jan 21 14: x86_64 x86_64 x86_64 GNU/Linux
root@puma:/home/phma# btrfs --version
btrfs-progs v5.4.1 
root@puma:/home/phma# btrfs fi show
Label: none  uuid: d0137494-fe35-488f-b0da-fdbe075b8832
        Total devices 1 FS bytes used 72.88GiB
        devid    1 size 673.51GiB used 75.02GiB path /dev/mapper/concolor-big

Label: none uuid: 174643ac-cb44-46a9-a5a4-5fb5e9a4e79f
Total devices 1 FS bytes used 48.84GiB
devid 1 size 100.00GiB used 57.02GiB path /dev/mapper/concolor-
rootcopy

Label: none uuid: 155a20c7-2264-4923-b082-288a3c146384
Total devices 1 FS bytes used 101.49GiB
devid 1 size 158.00GiB used 134.02GiB path /dev/mapper/concolor-
cougar

Label: none uuid: 10c61748-efe7-4b9c-b1f7-041dc45d894b
Total devices 1 FS bytes used 39.38GiB
devid 1 size 127.98GiB used 53.04GiB path /dev/mapper/cougar-crypt

Label: none uuid: 1f5a6f23-a7ef-46c6-92b1-84fc2f684931
Total devices 1 FS bytes used 92.58GiB
devid 1 size 158.00GiB used 131.00GiB path /dev/mapper/puma-cougar

root@puma:/home/phma# btrfs fi df /big
Data, single: total=74.01GiB, used=72.78GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=1.01GiB, used=101.59MiB
GlobalReserve, single: total=92.41MiB, used=0.00B
root@puma:/home/phma# btrfs fi df /rootcopy
Data, single: total=55.01GiB, used=47.68GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=2.01GiB, used=1.16GiB
GlobalReserve, single: total=113.17MiB, used=0.00B
root@puma:/home/phma# btrfs fi df /home
Data, single: total=132.01GiB, used=100.23GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=2.01GiB, used=1.26GiB
GlobalReserve, single: total=321.31MiB, used=0.00B
root@puma:/home/phma# btrfs fi df /crypt
Data, single: total=52.01GiB, used=39.29GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.00GiB, used=91.14MiB
GlobalReserve, single: total=45.91MiB, used=0.00B
root@puma:/home/phma# btrfs fi df /olv
Data, single: total=52.01GiB, used=39.29GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.00GiB, used=91.14MiB
GlobalReserve, single: total=45.91MiB, used=0.00B
root@puma:/home/phma# btrfs fi df /backup
Data, single: total=52.01GiB, used=39.29GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.00GiB, used=91.14MiB
GlobalReserve, single: total=45.91MiB, used=0.00B

The kernel I was running on Saturday may be older.

Here are some excerpts from syslog:

Mar  5 08:31:52 puma kernel: [10359910.035946] INFO: task Indexed~ #56145:1182901 blocked for more than 120 seconds.
Mar  5 08:31:52 puma kernel: [10359910.035948]       Tainted: P           OE     5.13.0-7614-generic #14~1631647151~20.04~930e87c-Ubuntu
Mar  5 08:31:52 puma kernel: [10359910.035949] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  5 08:31:52 puma kernel: [10359910.035950] task:Indexed~ #56145 state:D stack:    0 pid:1182901 ppid:     1 flags:0x00000000
Mar  5 08:31:52 puma kernel: [10359910.035953] Call Trace:
Mar  5 08:31:52 puma kernel: [10359910.035954]  __schedule+0x2ee/0x900
Mar  5 08:31:52 puma kernel: [10359910.035956]  ? try_to_unlazy+0x55/0x90
Mar  5 08:31:52 puma kernel: [10359910.035959]  schedule+0x4f/0xc0
Mar  5 08:31:52 puma kernel: [10359910.035964]  wait_current_trans+0xd6/0x140 [btrfs]
Mar  5 08:31:52 puma kernel: [10359910.036043]  ? wait_woken+0x80/0x80
Mar  5 08:31:52 puma kernel: [10359910.036047]  start_transaction+0x4bb/0x5a0 [btrfs]
Mar  5 08:31:52 puma kernel: [10359910.036081]  btrfs_start_transaction_fallback_global_rsv+0x1b/0x20 [btrfs]
Mar  5 08:31:52 puma kernel: [10359910.036113]  btrfs_unlink+0x30/0xe0 [btrfs]
Mar  5 08:31:52 puma kernel: [10359910.036145]  vfs_unlink+0x114/0x200
Mar  5 08:31:52 puma kernel: [10359910.036147]  do_unlinkat+0x1a2/0x2d0
Mar  5 08:31:52 puma kernel: [10359910.036150]  __x64_sys_unlink+0x23/0x30
Mar  5 08:31:52 puma kernel: [10359910.036153]  do_syscall_64+0x61/0xb0
Mar  5 08:31:52 puma kernel: [10359910.036155]  ? do_syscall_64+0x6e/0xb0
Mar  5 08:31:52 puma kernel: [10359910.036157]  ? do_syscall_64+0x6e/0xb0
Mar  5 08:31:52 puma kernel: [10359910.036159]  ? do_syscall_64+0x6e/0xb0
Mar  5 08:31:52 puma kernel: [10359910.036160]  ? do_syscall_64+0x6e/0xb0
Mar  5 08:31:52 puma kernel: [10359910.036162]  ? do_syscall_64+0x6e/0xb0
Mar  5 08:31:52 puma kernel: [10359910.036164]  ? asm_sysvec_apic_timer_interrupt+0xa/0x20
Mar  5 08:31:52 puma kernel: [10359910.036166]  entry_SYSCALL_64_after_hwframe+0x44/0xae
Mar  5 08:31:52 puma kernel: [10359910.036167] RIP: 0033:0x7fc30923de3b
Mar  5 08:31:52 puma kernel: [10359910.036169] RSP: 002b:00007fc2c2bbd4d8 EFLAGS: 00000202 ORIG_RAX: 0000000000000057
Mar  5 08:31:52 puma kernel: [10359910.036170] RAX: ffffffffffffffda RBX: 00007fc274b75271 RCX: 00007fc30923de3b
Mar  5 08:31:52 puma kernel: [10359910.036171] RDX: 0000000000000000 RSI: 00007fc274b75271 RDI: 00007fc274b75271
Mar  5 08:31:52 puma kernel: [10359910.036173] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007fc272f01748
Mar  5 08:31:52 puma kernel: [10359910.036174] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fc274b75271
Mar  5 08:31:52 puma kernel: [10359910.036175] R13: 00007fc2734b4400 R14: 0000000000000000 R15: 00007fc3085bd578
Mar  8 04:36:29 puma kernel: [39151.885472] INFO: task tracker-store:53596 blocked for more than 120 seconds.
Mar  8 04:36:29 puma kernel: [39151.885492]       Tainted: G           OE     5.15.15-76051515-generic #202201160435~1642693824~20.04~97db1bb-Ubuntu
Mar  8 04:36:29 puma kernel: [39151.885494] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  8 04:36:29 puma kernel: [39151.885496] task:tracker-store   state:D stack:    0 pid:53596 ppid:  3662 flags:0x00000000
Mar  8 04:36:29 puma kernel: [39151.885503] Call Trace:
Mar  8 04:36:29 puma kernel: [39151.885508]  <TASK>
Mar  8 04:36:29 puma kernel: [39151.885518]  __schedule+0x2cd/0x890
Mar  8 04:36:29 puma kernel: [39151.885530]  ? __cond_resched+0x19/0x30
Mar  8 04:36:29 puma kernel: [39151.885533]  schedule+0x4e/0xb0
Mar  8 04:36:29 puma kernel: [39151.885539]  btrfs_sync_log+0x178/0xef0 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885704]  ? __raw_callee_save___native_queued_spin_unlock+0x15/0x23
Mar  8 04:36:29 puma kernel: [39151.885717]  ? release_extent_buffer+0xbb/0xe0 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885753]  ? wait_woken+0x60/0x60
Mar  8 04:36:29 puma kernel: [39151.885758]  ? btrfs_free_path+0x27/0x30 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885787]  ? dput+0x62/0x320
Mar  8 04:36:29 puma kernel: [39151.885794]  ? log_all_new_ancestors+0x3bc/0x470 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885841]  ? btrfs_log_inode_parent+0x2db/0x890 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885872]  ? join_transaction+0x135/0x4c0 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885900]  ? start_transaction+0xd5/0x5b0 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885931]  ? dput+0x62/0x320
Mar  8 04:36:29 puma kernel: [39151.885933]  btrfs_sync_file+0x33f/0x460 [btrfs]
Mar  8 04:36:29 puma kernel: [39151.885961]  vfs_fsync_range+0x49/0x80
Mar  8 04:36:29 puma kernel: [39151.885965]  do_fsync+0x3d/0x70
Mar  8 04:36:29 puma kernel: [39151.885967]  __x64_sys_fsync+0x14/0x20
Mar  8 04:36:29 puma kernel: [39151.885968]  do_syscall_64+0x5c/0xc0
Mar  8 04:36:29 puma kernel: [39151.885971]  ? do_sys_openat2+0x1d3/0x320
Mar  8 04:36:29 puma kernel: [39151.885975]  ? exit_to_user_mode_prepare+0x3d/0x1c0
Mar  8 04:36:29 puma kernel: [39151.885980]  ? syscall_exit_to_user_mode+0x27/0x50
Mar  8 04:36:29 puma kernel: [39151.885983]  ? __x64_sys_openat+0x20/0x30
Mar  8 04:36:29 puma kernel: [39151.885986]  ? do_syscall_64+0x69/0xc0
Mar  8 04:36:29 puma kernel: [39151.885987]  ? do_syscall_64+0x69/0xc0
Mar  8 04:36:29 puma kernel: [39151.885988]  entry_SYSCALL_64_after_hwframe+0x44/0xae
Mar  8 04:36:29 puma kernel: [39151.885993] RIP: 0033:0x7f0eb970832b
Mar  8 04:36:29 puma kernel: [39151.885995] RSP: 002b:00007ffde79e29b0 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
Mar  8 04:36:29 puma kernel: [39151.885997] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0eb970832b
Mar  8 04:36:29 puma kernel: [39151.885999] RDX: 0000000000101441 RSI: 0000564c61177b50 RDI: 0000000000000008
Mar  8 04:36:29 puma kernel: [39151.886000] RBP: 0000000000000008 R08: 0000000000000000 R09: 0000000000000001
Mar  8 04:36:29 puma kernel: [39151.886001] R10: 00000000000001b0 R11: 0000000000000293 R12: 0000564c61176e50
Mar  8 04:36:29 puma kernel: [39151.886002] R13: 00007ffde79e2a00 R14: 00007ffde79e2be4 R15: 0000000000000000
Mar  8 04:36:29 puma kernel: [39151.886005]  </TASK>
Quoted 1 line

Please copy me, I'm not on the mailing list.

There's no indication of corruption in those logs. Above the kernel is complaining that it's taking too long to finish transactions, which could be a btrfs problem, or a hardware problem, or even simply a large filesystem running normally on very slow disks. Not enough information to tell.

When posting logs, extract all lines with 'btrfs' on them, plus context lines, e.g.

grep -B9 -i btrfs /var/log/kern.log

or

dmesg | grep -B9 -i btrfs

If you can reproduce the hang, enable sysrq and do Alt-SysRq-W when it hangs (or run

echo w > /proc/sysrq-trigger

from a command line). This will provide stack traces of all blocked processes so we can see what the transaction is waiting for.

Quoted 7 lines

Pierre

Signature · 2 lines
The gostak pelled at the fostin lutt for darfs for her martle plave.
The darfs had smibbed, the lutt was thale, and the pilter had nothing snave.
#2 lore raw
Quoted 5 lines · EDT Zygo Blaxell

There's no indication of corruption in those logs. Above the kernel is complaining that it's taking too long to finish transactions, which could be a btrfs problem, or a hardware problem, or even simply a large filesystem running normally on very slow disks. Not enough information to tell.

It couldn't be a very slow disk, because all the drives are NVM or SSD.

Quoted 9 lines

When posting logs, extract all lines with 'btrfs' on them, plus context lines, e.g.

grep -B9 -i btrfs /var/log/kern.log

or

dmesg | grep -B9 -i btrfs

Quoted 7 lines

If you can reproduce the hang, enable sysrq and do Alt-SysRq-W when it hangs (or run

echo w > /proc/sysrq-trigger

from a command line). This will provide stack traces of all blocked processes so we can see what the transaction is waiting for.

Here are the whole sections of the logfiles from the first error until the computer hung, compressed. I'm afraid to run the rsync script again because it might hang the computer. Is there a way to find out if the filesystem access hung on a bad sector or something?

Pierre

Signature · 2 lines
Lanthanidia deliciosa: What the kiwifruit would be
if it weren't so radioactive.
#3 lore raw
Quoted 8 lines · Pierre Abbat
Quoted 5 lines · EDT Zygo Blaxell

There's no indication of corruption in those logs. Above the kernel is complaining that it's taking too long to finish transactions, which could be a btrfs problem, or a hardware problem, or even simply a large filesystem running normally on very slow disks. Not enough information to tell.

It couldn't be a very slow disk, because all the drives are NVM or SSD.

Those can be slow too. It's up to the firmware and the health of the underlying flash media, and some devices also implement throttling when over temperature.

e.g. Samsung 860 EVO has a firmware quirk that makes them significantly slower than spinning drives under continuous load. The drive drops to 2 iops every 5 seconds, an almost complete stop.

Quoted 12 lines
Quoted 12 lines

When posting logs, extract all lines with 'btrfs' on them, plus context lines, e.g.

grep -B9 -i btrfs /var/log/kern.log

or

dmesg | grep -B9 -i btrfs

If you can reproduce the hang, enable sysrq and do Alt-SysRq-W when it hangs (or run

Quoted 8 lines
Quoted 4 lines

echo w > /proc/sysrq-trigger

from a command line). This will provide stack traces of all blocked processes so we can see what the transaction is waiting for.

Here are the whole sections of the logfiles from the first error until the computer hung, compressed. I'm afraid to run the rsync script again because it might hang the computer.

It's probably not the rsync--I've retired quite a few btrfs bugs triggered by rsync, and your stack traces are pointing to system calls that rsync never uses (though rsync is caught up in the hang which affects everything writing the filesystem).

Quoted 2 lines

Is there a way to find out if the filesystem access hung on a bad sector or something?

Generally that would eventually trigger a timeout which would be reported by the device layer, but some device drivers fail to recover from those.

If the device is truly hung, then you wouldn't be able to access the device while the filesystem is locked up. Try this command (with "..." replaced with the device name):

dd if=/dev/... of=/dev/null bs=4k status=progress

If it shows progress as data is read from the device, then the device isn't hung. If it shows very slow progress (like <100K per 5 seconds) then the device isn't hung, but it is having some kind of problem that needs to be addressed (pick a different device model, add a heat sink, replace a failing device, replace a failing HBA, whatever).

This looks like a more conventional btrfs deadlock bug, though:
everything seems to be waiting for a fsync on a directory. You might try a different kernel, either the previous LTS (5.10) or the next one (5.15) or the current release (5.17) to see if it is already resolved.

If it still happens on a current release, try

echo w > /proc/sysrq-trigger
echo d > /proc/sysrq-trigger

and post the logs. Maybe someone can figure out what the deadlock is.

Quoted 5 lines

Pierre

Signature · 2 lines
Lanthanidia deliciosa: What the kiwifruit would be
if it weren't so radioactive.