Skip to contentSkip to conversation

fstests generic/774 hang again

bug report Shin'ichiro Kawasaki 5 participants 25 messages Sep 15 – Oct 22 XFS filesystem

lore
Share & tools
lore
#0 lore raw

I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB TCMU fileio devices. Actually I once reported this hang symptom in last October, and Darrick and John kindly took some actions [1]. Since then, the hang disappeared and I have not observed the hang almost one year. However, my test system started reporting the hang again last week.

[1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t

FYI, here I attache the kernel message [2]. I observed the hang last week with the kernel on the xfs/for-next branch at the commit 0ca15a1a1151 ("xfs: remove an extra cast in xfs_file_compat_ioctl"). Today, I tried again with the kernel at current xfs/for-next branch tip 4266ffdfd7cb ("xfs: remove an extra cast in xfs_file_compat_ioctl"), and observed the hang again. The hang can be recreated in stable manner by repeating the test case 20 times or so.

I also tried other test cases in atmoicwrites group, and observed no failure.

generic/765 [not run] write atomic not supported by this block device

 generic/767         11s
 generic/768         13s
 generic/769         13s
 generic/770         32s
 generic/773        [not run] write atomic not supported by this block device
 generic/774        (skipped)
 generic/775         293s
 generic/776        [notrun] write atomic not supported by this block device
 generic/778         48s
 xfs/838            [not run] External volumes not in use, skipped this test
 xfs/839            [not run] XFS error injection requires CONFIG_XFS_DEBUG
 xfs/840            [not run] write atomic not supported by this block device

Action for fix will be appreciated. If I can do anything on my test node, please let me know.

[2] kernel message during generic/774

[144539.456849][T1634092] run fstests generic/774 at 2026-09-09 16:51:53
[144539.691199][T3820710] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[144543.436776][T3868062] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[144543.648458][T3872722] XFS (sdg): Mounting V5 Filesystem 306fb2c4-6cd3-4cbd-b9ed-953c0bc0d3b8
[144543.674712][T3872722] XFS (sdg): Ending clean mount
[144544.328859][T3872774] XFS (sdg): Unmounting Filesystem 306fb2c4-6cd3-4cbd-b9ed-953c0bc0d3b8
[144545.743530][T3795425] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[144545.964812][T3872823] XFS (sdg): Mounting V5 Filesystem 91ee7431-7139-4d92-bea7-b7802c3f70d2
[144545.990603][T3872823] XFS (sdg): Ending clean mount
[144870.819650][  T172] INFO: task kworker/10:173:3785794 blocked for more than 122 seconds.
[144870.821440][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144870.823252][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144870.825054][  T172] task:kworker/10:173  state:D stack:0     pid:3785794 tgid:3785794 ppid:2      task_flags:0x4248060 flags:0x00080000
[144870.827098][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144870.828170][  T172] Call Trace:
[144870.829209][  T172]  <TASK>
[144870.830154][  T172]  __schedule+0xc86/0x1b10
[144870.831126][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144870.832166][  T172]  ? __pfx___schedule+0x10/0x10
[144870.833250][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.834400][  T172]  ? wq_worker_sleeping+0x213/0x280
[144870.835559][  T172]  ? find_held_lock+0x2b/0x80
[144870.836634][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144870.837699][  T172]  schedule+0xe5/0x2e0
[144870.838721][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.839915][  T172]  schedule_preempt_disabled+0x16/0x30
[144870.841017][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144870.842139][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144870.843236][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144870.844383][  T172]  ? __lock_acquire+0x691/0xd00
[144870.845598][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.846840][  T172]  ? lock_acquire+0x11a/0x140
[144870.848019][  T172]  down_write_nested+0x1c0/0x1f0
[144870.849121][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144870.850284][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144870.852895][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144870.855451][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144870.858004][  T172]  ? __lock_acquire+0x691/0xd00
[144870.859160][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144870.861775][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144870.864257][  T172]  iomap_dio_complete+0x140/0xb20
[144870.865378][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144870.866570][  T172]  iomap_dio_complete_work+0x58/0x90
[144870.867785][  T172]  process_one_work+0x947/0x1760
[144870.868995][  T172]  ? __pfx_process_one_work+0x10/0x10
[144870.870062][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.871227][  T172]  ? lock_is_held_type+0xb0/0x180
[144870.872361][  T172]  worker_thread+0x601/0xff0
[144870.873609][  T172]  ? __pfx_worker_thread+0x10/0x10
[144870.874813][  T172]  kthread+0x361/0x460
[144870.875873][  T172]  ? __pfx_kthread+0x10/0x10
[144870.876881][  T172]  ret_from_fork+0x560/0x860
[144870.878012][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144870.879036][  T172]  ? __switch_to+0x473/0xd50
[144870.880200][  T172]  ? __pfx_kthread+0x10/0x10
[144870.881232][  T172]  ret_from_fork_asm+0x1a/0x30
[144870.882353][  T172]  </TASK>
[144870.883486][  T172] INFO: task kworker/10:173:3785794 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144870.885498][  T172] INFO: task kworker/18:2:3812136 blocked for more than 122 seconds.
[144870.887038][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144870.888467][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144870.890050][  T172] task:kworker/18:2    state:D stack:0     pid:3812136 tgid:3812136 ppid:2      task_flags:0x4248060 flags:0x00080000
[144870.892033][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144870.893266][  T172] Call Trace:
[144870.894349][  T172]  <TASK>
[144870.895228][  T172]  __schedule+0xc86/0x1b10
[144870.896439][  T172]  ? __pfx___schedule+0x10/0x10
[144870.897500][  T172]  ? schedule+0x17e/0x2e0
[144870.898441][  T172]  ? wq_worker_sleeping+0x65/0x280
[144870.899427][  T172]  ? find_held_lock+0x2b/0x80
[144870.900392][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144870.901382][  T172]  schedule+0xe5/0x2e0
[144870.902249][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.903287][  T172]  schedule_preempt_disabled+0x16/0x30
[144870.904511][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144870.905518][  T172]  ? lock_release.part.0+0x1c/0x50
[144870.906482][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144870.907575][  T172]  ? __lock_acquire+0x691/0xd00
[144870.908506][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.939885][  T172]  ? lock_acquire+0x11a/0x140
[144870.940681][  T172]  down_write_nested+0x1c0/0x1f0
[144870.941553][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144870.942530][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144870.944492][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144870.946365][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144870.948153][  T172]  ? __lock_acquire+0x691/0xd00
[144870.949019][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144870.950793][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144870.952653][  T172]  iomap_dio_complete+0x140/0xb20
[144870.953439][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144870.954096][  T172]  iomap_dio_complete_work+0x58/0x90
[144870.954877][  T172]  process_one_work+0x947/0x1760
[144870.955644][  T172]  ? __pfx_process_one_work+0x10/0x10
[144870.956399][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.957121][  T172]  ? lock_is_held_type+0xb0/0x180
[144870.957949][  T172]  worker_thread+0x601/0xff0
[144870.958860][  T172]  ? __pfx_worker_thread+0x10/0x10
[144870.959586][  T172]  kthread+0x361/0x460
[144870.960273][  T172]  ? __pfx_kthread+0x10/0x10
[144870.960891][  T172]  ret_from_fork+0x560/0x860
[144870.961610][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144870.962503][  T172]  ? __switch_to+0x473/0xd50
[144870.963250][  T172]  ? __pfx_kthread+0x10/0x10
[144870.963930][  T172]  ret_from_fork_asm+0x1a/0x30
[144870.964760][  T172]  </TASK>
[144870.965444][  T172] INFO: task kworker/18:2:3812136 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144870.966764][  T172] INFO: task kworker/20:1:3814407 blocked for more than 123 seconds.
[144870.967875][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144870.968920][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144870.970043][  T172] task:kworker/20:1    state:D stack:0     pid:3814407 tgid:3814407 ppid:2      task_flags:0x4248060 flags:0x00080000
[144870.971395][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144870.972198][  T172] Call Trace:
[144870.972772][  T172]  <TASK>
[144870.973279][  T172]  __schedule+0xc86/0x1b10
[144870.973888][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144870.974596][  T172]  ? __pfx___schedule+0x10/0x10
[144870.975254][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.975899][  T172]  ? wq_worker_sleeping+0x213/0x280
[144870.976689][  T172]  ? find_held_lock+0x2b/0x80
[144870.977241][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144870.977916][  T172]  schedule+0xe5/0x2e0
[144870.978586][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.979182][  T172]  schedule_preempt_disabled+0x16/0x30
[144870.979886][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144870.980635][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144870.981287][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144870.982017][  T172]  ? __lock_acquire+0x691/0xd00
[144870.982700][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.983391][  T172]  ? lock_acquire+0x11a/0x140
[144870.984047][  T172]  down_write_nested+0x1c0/0x1f0
[144870.984778][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144870.985569][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144870.987084][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144870.988587][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144870.990093][  T172]  ? lock_is_held_type+0xb0/0x180
[144870.990804][  T172]  ? __lock_acquire+0x691/0xd00
[144870.991435][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144870.992815][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144870.994256][  T172]  iomap_dio_complete+0x140/0xb20
[144870.994897][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144870.995533][  T172]  iomap_dio_complete_work+0x58/0x90
[144870.996229][  T172]  process_one_work+0x947/0x1760
[144870.996919][  T172]  ? __pfx_process_one_work+0x10/0x10
[144870.997519][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.998095][  T172]  ? lock_is_held_type+0xb0/0x180
[144870.998755][  T172]  worker_thread+0x601/0xff0
[144870.999262][  T172]  ? __pfx_worker_thread+0x10/0x10
[144870.999854][  T172]  kthread+0x361/0x460
[144871.000422][  T172]  ? __pfx_kthread+0x10/0x10
[144871.000971][  T172]  ret_from_fork+0x560/0x860
[144871.001596][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.002150][  T172]  ? __switch_to+0x473/0xd50
[144871.002764][  T172]  ? __pfx_kthread+0x10/0x10
[144871.003268][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.003871][  T172]  </TASK>
[144871.006437][  T172] INFO: task kworker/20:1:3814407 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.007488][  T172] INFO: task kworker/16:2:3814415 blocked for more than 123 seconds.
[144871.008373][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.009098][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.009935][  T172] task:kworker/16:2    state:D stack:0     pid:3814415 tgid:3814415 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.010908][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.011529][  T172] Call Trace:
[144871.011921][  T172]  <TASK>
[144871.012338][  T172]  __schedule+0xc86/0x1b10
[144871.012801][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144871.013341][  T172]  ? __pfx___schedule+0x10/0x10
[144871.013912][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.014491][  T172]  ? wq_worker_sleeping+0x213/0x280
[144871.015122][  T172]  ? find_held_lock+0x2b/0x80
[144871.015734][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.016274][  T172]  schedule+0xe5/0x2e0
[144871.016820][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.017430][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.017981][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.018500][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144871.019066][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.019691][  T172]  ? __lock_acquire+0x691/0xd00
[144871.020216][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.020806][  T172]  ? lock_acquire+0x11a/0x140
[144871.021350][  T172]  down_write_nested+0x1c0/0x1f0
[144871.021847][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.022365][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.023499][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.024814][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.026064][  T172]  ? update_load_avg+0xb65/0x2c70
[144871.026726][  T172]  ? __lock_acquire+0x691/0xd00
[144871.027267][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.028538][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.029728][  T172]  iomap_dio_complete+0x140/0xb20
[144871.030224][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.030743][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.031190][  T172]  process_one_work+0x947/0x1760
[144871.031764][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.032294][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.032889][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.033502][  T172]  worker_thread+0x601/0xff0
[144871.034040][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.034640][  T172]  kthread+0x361/0x460
[144871.035071][  T172]  ? __pfx_kthread+0x10/0x10
[144871.035679][  T172]  ret_from_fork+0x560/0x860
[144871.036162][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.036758][  T172]  ? __switch_to+0x473/0xd50
[144871.037334][  T172]  ? __pfx_kthread+0x10/0x10
[144871.037914][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.038394][  T172]  </TASK>
[144871.038867][  T172] INFO: task kworker/16:2:3814415 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.039846][  T172] INFO: task kworker/1:2:3819893 blocked for more than 123 seconds.
[144871.040650][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.041422][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.042136][  T172] task:kworker/1:2     state:D stack:0     pid:3819893 tgid:3819893 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.043104][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.043712][  T172] Call Trace:
[144871.044164][  T172]  <TASK>
[144871.044651][  T172]  __schedule+0xc86/0x1b10
[144871.045172][  T172]  ? __pfx___schedule+0x10/0x10
[144871.045707][  T172]  ? schedule+0x17e/0x2e0
[144871.046192][  T172]  ? wq_worker_sleeping+0x65/0x280
[144871.046741][  T172]  ? find_held_lock+0x2b/0x80
[144871.047186][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.047796][  T172]  schedule+0xe5/0x2e0
[144871.048299][  T172]  xlog_grant_head_wait+0x175/0xac0 [xfs]
[144871.049471][  T172]  xlog_grant_head_check+0x312/0x3f0 [xfs]
[144871.050717][  T172]  ? __pfx_xlog_grant_head_check+0x10/0x10 [xfs]
[144871.051932][  T172]  xfs_log_regrant+0x380/0x7d0 [xfs]
[144871.053059][  T172]  ? __pfx_xfs_log_regrant+0x10/0x10 [xfs]
[144871.054192][  T172]  ? xfs_log_ticket_get+0x1b/0x30 [xfs]
[144871.055259][  T172]  ? xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
[144871.056414][  T172]  xfs_trans_roll+0x2d9/0x420 [xfs]
[144871.057517][  T172]  ? kmem_cache_alloc_noprof+0x45b/0x680
[144871.058100][  T172]  ? xfs_defer_finish_noroll+0x460/0xe70 [xfs]
[144871.059201][  T172]  xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
[144871.060341][  T172]  ? __pfx_xfs_defer_trans_roll+0x10/0x10 [xfs]
[144871.061364][  T172]  ? xfs_defer_relog+0x3e7/0xb50 [xfs]
[144871.062454][  T172]  xfs_defer_finish_noroll+0x460/0xe70 [xfs]
[144871.063491][  T172]  ? __pfx_xfs_reflink_end_cow_extent_locked+0x10/0x10 [xfs]
[144871.064782][  T172]  ? __pfx_xfs_defer_finish_noroll+0x10/0x10 [xfs]
[144871.065859][  T172]  ? lock_acquire+0x11a/0x140
[144871.066413][  T172]  xfs_trans_commit+0xfc/0x180 [xfs]
[144871.067509][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.068046][  T172]  ? __pfx_xfs_trans_commit+0x10/0x10 [xfs]
[144871.069143][  T172]  xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
[144871.070340][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.071419][  T172]  ? __lock_acquire+0x691/0xd00
[144871.089717][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.090837][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.091794][  T172]  iomap_dio_complete+0x140/0xb20
[144871.092194][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.092684][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.093251][  T172]  process_one_work+0x947/0x1760
[144871.093761][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.094406][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.095084][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.095679][  T172]  worker_thread+0x601/0xff0
[144871.096234][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.096820][  T172]  kthread+0x361/0x460
[144871.097360][  T172]  ? __pfx_kthread+0x10/0x10
[144871.098065][  T172]  ret_from_fork+0x560/0x860
[144871.098693][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.099175][  T172]  ? __switch_to+0x473/0xd50
[144871.099709][  T172]  ? __pfx_kthread+0x10/0x10
[144871.100275][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.100783][  T172]  </TASK>
[144871.101203][  T172] INFO: task kworker/20:2:3824669 blocked for more than 123 seconds.
[144871.102109][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.102813][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.103685][  T172] task:kworker/20:2    state:D stack:0     pid:3824669 tgid:3824669 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.104588][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.105360][  T172] Call Trace:
[144871.105868][  T172]  <TASK>
[144871.106339][  T172]  __schedule+0xc86/0x1b10
[144871.106949][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144871.107632][  T172]  ? __pfx___schedule+0x10/0x10
[144871.108170][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.108600][  T172]  ? wq_worker_sleeping+0x213/0x280
[144871.109207][  T172]  ? find_held_lock+0x2b/0x80
[144871.109728][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.110270][  T172]  schedule+0xe5/0x2e0
[144871.110762][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.111328][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.111898][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.112410][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144871.112989][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.113648][  T172]  ? __lock_acquire+0x691/0xd00
[144871.114157][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.114665][  T172]  ? lock_acquire+0x11a/0x140
[144871.115101][  T172]  down_write_nested+0x1c0/0x1f0
[144871.115664][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.116122][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.117105][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.118239][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.119389][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.119781][  T172]  ? __lock_acquire+0x691/0xd00
[144871.120177][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.121294][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.122391][  T172]  iomap_dio_complete+0x140/0xb20
[144871.122770][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.123425][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.124051][  T172]  process_one_work+0x947/0x1760
[144871.124445][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.125017][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.125580][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.126138][  T172]  worker_thread+0x601/0xff0
[144871.126680][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.127126][  T172]  kthread+0x361/0x460
[144871.127762][  T172]  ? __pfx_kthread+0x10/0x10
[144871.128234][  T172]  ret_from_fork+0x560/0x860
[144871.128772][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.129337][  T172]  ? __switch_to+0x473/0xd50
[144871.129842][  T172]  ? __pfx_kthread+0x10/0x10
[144871.130364][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.130892][  T172]  </TASK>
[144871.131250][  T172] INFO: task kworker/20:2:3824669 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.132261][  T172] INFO: task kworker/4:0:3824816 blocked for more than 123 seconds.
[144871.133177][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.133912][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.134827][  T172] task:kworker/4:0     state:D stack:0     pid:3824816 tgid:3824816 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.135724][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.136356][  T172] Call Trace:
[144871.136856][  T172]  <TASK>
[144871.137342][  T172]  __schedule+0xc86/0x1b10
[144871.137818][  T172]  ? __pfx___schedule+0x10/0x10
[144871.138410][  T172]  ? schedule+0x17e/0x2e0
[144871.138810][  T172]  ? wq_worker_sleeping+0x65/0x280
[144871.139356][  T172]  ? find_held_lock+0x2b/0x80
[144871.139817][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.140631][  T172]  schedule+0xe5/0x2e0
[144871.141220][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.141887][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.142473][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.143026][  T172]  ? lock_release.part.0+0x1c/0x50
[144871.143557][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.144101][  T172]  ? __lock_acquire+0x691/0xd00
[144871.144681][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.145057][  T172]  ? lock_acquire+0x11a/0x140
[144871.145709][  T172]  down_write_nested+0x1c0/0x1f0
[144871.146205][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.146885][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.147929][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.148978][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.150061][  T172]  ? __lock_acquire+0x691/0xd00
[144871.150641][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.151842][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.152998][  T172]  iomap_dio_complete+0x140/0xb20
[144871.153548][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.154099][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.154759][  T172]  process_one_work+0x947/0x1760
[144871.155210][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.155675][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.156336][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.156920][  T172]  worker_thread+0x601/0xff0
[144871.157437][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.157951][  T172]  kthread+0x361/0x460
[144871.158466][  T172]  ? __pfx_kthread+0x10/0x10
[144871.159012][  T172]  ret_from_fork+0x560/0x860
[144871.159502][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.160015][  T172]  ? __switch_to+0x473/0xd50
[144871.160539][  T172]  ? __pfx_kthread+0x10/0x10
[144871.161051][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.161685][  T172]  </TASK>
[144871.162077][  T172] INFO: task kworker/4:0:3824816 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.162900][  T172] INFO: task kworker/12:0:3826849 blocked for more than 123 seconds.
[144871.163697][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.164415][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.165277][  T172] task:kworker/12:0    state:D stack:0     pid:3826849 tgid:3826849 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.166221][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.166952][  T172] Call Trace:
[144871.167283][  T172]  <TASK>
[144871.167853][  T172]  __schedule+0xc86/0x1b10
[144871.168439][  T172]  ? __pfx___schedule+0x10/0x10
[144871.168914][  T172]  ? schedule+0x17e/0x2e0
[144871.169451][  T172]  ? wq_worker_sleeping+0x65/0x280
[144871.169927][  T172]  ? find_held_lock+0x2b/0x80
[144871.170288][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.170911][  T172]  schedule+0xe5/0x2e0
[144871.171387][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.171939][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.172496][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.173087][  T172]  ? lock_release.part.0+0x1c/0x50
[144871.173702][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.174226][  T172]  ? __lock_acquire+0x691/0xd00
[144871.174891][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.175466][  T172]  ? lock_acquire+0x11a/0x140
[144871.176164][  T172]  down_write_nested+0x1c0/0x1f0
[144871.176754][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.177531][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.178526][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.179662][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.180880][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.181342][  T172]  ? __lock_acquire+0x691/0xd00
[144871.181872][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.183071][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.184221][  T172]  iomap_dio_complete+0x140/0xb20
[144871.184745][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.185338][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.185864][  T172]  process_one_work+0x947/0x1760
[144871.186361][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.186920][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.187381][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.187924][  T172]  worker_thread+0x601/0xff0
[144871.188591][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.189109][  T172]  kthread+0x361/0x460
[144871.189647][  T172]  ? __pfx_kthread+0x10/0x10
[144871.190131][  T172]  ret_from_fork+0x560/0x860
[144871.190578][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.191172][  T172]  ? __switch_to+0x473/0xd50
[144871.191771][  T172]  ? __pfx_kthread+0x10/0x10
[144871.192247][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.192760][  T172]  </TASK>
[144871.193146][  T172] INFO: task kworker/12:0:3826849 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.202516][  T172] INFO: task kworker/21:1:3827095 blocked for more than 123 seconds.
[144871.203182][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.203953][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.204755][  T172] task:kworker/21:1    state:D stack:0     pid:3827095 tgid:3827095 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.205642][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.206251][  T172] Call Trace:
[144871.206793][  T172]  <TASK>
[144871.207382][  T172]  __schedule+0xc86/0x1b10
[144871.207815][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144871.208402][  T172]  ? __pfx___schedule+0x10/0x10
[144871.208871][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.209404][  T172]  ? wq_worker_sleeping+0x213/0x280
[144871.209925][  T172]  ? find_held_lock+0x2b/0x80
[144871.210492][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.211050][  T172]  schedule+0xe5/0x2e0
[144871.211527][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.212043][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.212786][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.213398][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144871.213975][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.214563][  T172]  ? __lock_acquire+0x691/0xd00
[144871.215009][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.215710][  T172]  ? lock_acquire+0x11a/0x140
[144871.216164][  T172]  down_write_nested+0x1c0/0x1f0
[144871.216688][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.217212][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.218301][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.219411][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.220508][  T172]  ? __lock_acquire+0x691/0xd00
[144871.221057][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.222191][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.223292][  T172]  iomap_dio_complete+0x140/0xb20
[144871.223820][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.224332][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.224792][  T172]  process_one_work+0x947/0x1760
[144871.225354][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.225861][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.226458][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.226982][  T172]  worker_thread+0x601/0xff0
[144871.227454][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.227993][  T172]  ? __kthread_parkme+0xbd/0x210
[144871.228532][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.229057][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.229619][  T172]  kthread+0x361/0x460
[144871.230083][  T172]  ? __pfx_kthread+0x10/0x10
[144871.230578][  T172]  ret_from_fork+0x560/0x860
[144871.231049][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.231633][  T172]  ? __switch_to+0x473/0xd50
[144871.232124][  T172]  ? __pfx_kthread+0x10/0x10
[144871.232544][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.233081][  T172]  </TASK>
[144871.233440][  T172] INFO: task kworker/21:1:3827095 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.234328][  T172] INFO: task kworker/10:0:3827669 blocked for more than 123 seconds.
[144871.235091][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.235858][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.236582][  T172] task:kworker/10:0    state:D stack:0     pid:3827669 tgid:3827669 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.237651][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.238173][  T172] Call Trace:
[144871.238946][  T172]  <TASK>
[144871.239469][  T172]  __schedule+0xc86/0x1b10
[144871.240084][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144871.240642][  T172]  ? __pfx___schedule+0x10/0x10
[144871.241088][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.241694][  T172]  ? wq_worker_sleeping+0x213/0x280
[144871.242326][  T172]  ? find_held_lock+0x2b/0x80
[144871.242813][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.243379][  T172]  schedule+0xe5/0x2e0
[144871.243881][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.244378][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.244920][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.245543][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144871.246066][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.246662][  T172]  ? __lock_acquire+0x691/0xd00
[144871.247131][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.247711][  T172]  ? lock_acquire+0x11a/0x140
[144871.248149][  T172]  down_write_nested+0x1c0/0x1f0
[144871.248724][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.249285][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.250297][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.251346][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.252414][  T172]  ? __lock_acquire+0x691/0xd00
[144871.252983][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.254104][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.255189][  T172]  iomap_dio_complete+0x140/0xb20
[144871.255756][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.256283][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.256814][  T172]  process_one_work+0x947/0x1760
[144871.257211][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.257962][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.258415][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.258907][  T172]  worker_thread+0x601/0xff0
[144871.259454][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.259949][  T172]  kthread+0x361/0x460
[144871.260364][  T172]  ? __pfx_kthread+0x10/0x10
[144871.260889][  T172]  ret_from_fork+0x560/0x860
[144871.261384][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.261926][  T172]  ? __switch_to+0x473/0xd50
[144871.262448][  T172]  ? __pfx_kthread+0x10/0x10
[144871.262968][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.263492][  T172]  </TASK>
[144871.263903][  T172] INFO: task kworker/10:0:3827669 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.264856][  T172] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
[144871.265760][  T172] 
[144871.265760][  T172] Showing all locks held in the system:
[144871.266766][  T172] locks held by pr/ttyS0/16: 2, last CPU#22:
[144871.267407][  T172]  #0: ffffffff8c320658 (console_srcu){....}-{0:0}, at: console_srcu_read_lock+0x49/0x70
[144871.268258][  T172]  #1: ffffffff9c3db638 (&port_lock_key){-.-.}-{3:3}, at: univ8250_console_device_lock+0x63/0xc0
[144871.269176][  T172] locks held by khungtaskd/172: 1, last CPU#20:
[144871.269752][  T172]  #0: ffffffff8c3369e0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[144871.270692][  T172] locks held by kworker/10:173/3785794: 4, on CPU#10:
[144871.271375][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.272341][  T172]  #1: ff11000109407d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.273284][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.274707][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.276050][  T172] locks held by kworker/18:2/3812136: 4, on CPU#18:
[144871.276811][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.277696][  T172]  #1: ff1100018e71fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.278634][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.280090][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.281626][  T172] locks held by kworker/20:1/3814407: 4, on CPU#20:
[144871.282267][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.283196][  T172]  #1: ff11000214267d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.284154][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.285620][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.287063][  T172] locks held by kworker/16:2/3814415: 4, on CPU#16:
[144871.287789][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.288637][  T172]  #1: ff1100019cc67d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.289792][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.291054][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.292462][  T172] locks held by kworker/1:2/3819893: 4, on CPU#1:
[144871.293085][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.294070][  T172]  #1: ff110001246d7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.294991][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.296348][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.297789][  T172] locks held by kworker/20:2/3824669: 4, on CPU#20:
[144871.298349][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.299205][  T172]  #1: ff1100022fc77d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.300225][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.301638][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.303077][  T172] locks held by kworker/4:0/3824816: 4, on CPU#4:
[144871.303744][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.312636][  T172]  #1: ff11000424997d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.313372][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.314586][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.315921][  T172] locks held by kworker/12:0/3826849: 4, on CPU#12:
[144871.316445][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.317222][  T172]  #1: ff110001e71bfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.318124][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.319340][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.320712][  T172] locks held by kworker/21:1/3827095: 4, on CPU#21:
[144871.321339][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.322330][  T172]  #1: ff11000172fefd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.323140][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.324642][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.326042][  T172] locks held by kworker/10:0/3827669: 4, on CPU#10:
[144871.326655][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.327575][  T172]  #1: ff11000122177d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.328366][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.329889][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.331476][  T172] locks held by kworker/17:1/3827670: 4, on CPU#17:
[144871.332105][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.332897][  T172]  #1: ff110001d0307d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.333703][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.334981][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.336354][  T172] locks held by kworker/16:1/3864132: 4, on CPU#16:
[144871.337033][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.337973][  T172]  #1: ff1100023211fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.338641][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.339966][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.341388][  T172] locks held by kworker/8:1/3868486: 4, on CPU#8:
[144871.342196][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.343143][  T172]  #1: ff110001a24d7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.344179][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.345576][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.347006][  T172] locks held by kworker/10:1/3872909: 4, on CPU#10:
[144871.347626][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.348518][  T172]  #1: ff110001606dfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.349516][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.350857][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.352433][  T172] locks held by kworker/17:2/3872910: 4, on CPU#17:
[144871.353061][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.354161][  T172]  #1: ff1100041c1b7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.354964][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.356302][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.357755][  T172] locks held by kworker/23:0/3872911: 4, on CPU#23:
[144871.358527][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.359329][  T172]  #1: ff110004b8c8fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.360223][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.361505][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.363110][  T172] locks held by kworker/7:3/3872919: 4, on CPU#7:
[144871.363803][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.364639][  T172]  #1: ff1100019af97d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.365488][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.366938][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.368364][  T172] locks held by kworker/20:3/3872924: 4, on CPU#20:
[144871.369008][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.369969][  T172]  #1: ff110001aa42fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.371118][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.372518][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.373983][  T172] locks held by kworker/3:2/3872925: 4, on CPU#3:
[144871.374629][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.375515][  T172]  #1: ff11000444b8fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.376358][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.377586][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.378950][  T172] locks held by kworker/16:3/3872926: 4, on CPU#16:
[144871.379685][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.380610][  T172]  #1: ff11000682cdfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.381567][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.382968][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.384263][  T172] locks held by kworker/17:3/3872927: 4, on CPU#17:
[144871.384833][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.385697][  T172]  #1: ff110001d3ca7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.386411][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.387778][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.389139][  T172] locks held by kworker/16:4/3872929: 4, on CPU#16:
[144871.389779][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.390582][  T172]  #1: ff110003cb7efd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.391402][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.392568][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.394019][  T172] locks held by kworker/19:2/3872930: 4, on CPU#19:
[144871.394685][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.395449][  T172]  #1: ff1100021374fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.396394][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.397664][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.399332][  T172] locks held by kworker/18:3/3872935: 3, on CPU#18:
[144871.400131][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.401110][  T172]  #1: ff110002862b7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.402243][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.403502][  T172] locks held by kworker/10:2/3872936: 4, on CPU#10:
[144871.404186][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.405139][  T172]  #1: ff110003d8e8fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.406011][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.407341][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.408729][  T172] locks held by kworker/8:3/3872941: 3, on CPU#8:
[144871.409141][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.410067][  T172]  #1: ff110003df1e7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.410799][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.411989][  T172] locks held by kworker/1:3/3872953: 4, on CPU#1:
[144871.412584][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.413531][  T172]  #1: ff1100043f1bfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.414464][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.424444][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.425808][  T172] locks held by kworker/4:4/3872955: 4, on CPU#4:
[144871.426603][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.427468][  T172]  #1: ff1100014de77d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.428336][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.429615][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.430975][  T172] locks held by fio/3872961: 3, on CPU#2:
[144871.431620][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.433089][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.434279][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.435797][  T172] locks held by fio/3872962: 3, on CPU#23:
[144871.436291][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.437618][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.438845][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.440124][  T172] locks held by fio/3872963: 3, on CPU#23:
[144871.440772][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.442297][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.443694][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.445025][  T172] locks held by fio/3872964: 3, on CPU#14:
[144871.445514][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.446890][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.448097][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.449466][  T172] locks held by fio/3872965: 3, on CPU#12:
[144871.450120][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.451359][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.452681][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.454164][  T172] locks held by fio/3872966: 3, on CPU#16:
[144871.454720][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.455979][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.457331][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.458737][  T172] locks held by fio/3872967: 3, on CPU#4:
[144871.459369][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.460809][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.461990][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.463126][  T172] locks held by fio/3872968: 3, on CPU#1:
[144871.463662][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.464985][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.466188][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.467438][  T172] locks held by fio/3872969: 3, on CPU#6:
[144871.468111][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.469612][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.471012][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.472352][  T172] locks held by fio/3872970: 3, on CPU#4:
[144871.473018][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.474395][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.475759][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.477331][  T172] locks held by fio/3872971: 3, on CPU#14:
[144871.477764][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.479087][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.480334][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.481548][  T172] locks held by fio/3872972: 2, on CPU#10:
[144871.482113][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.483336][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.484625][  T172] locks held by fio/3872973: 3, on CPU#10:
[144871.485350][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.486616][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.487949][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.489152][  T172] locks held by fio/3872974: 2, on CPU#18:
[144871.489572][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.491050][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.492416][  T172] locks held by fio/3872975: 3, on CPU#8:
[144871.493166][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.494401][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.495553][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.496938][  T172] locks held by fio/3872976: 3, on CPU#23:
[144871.497422][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.498613][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.499903][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.501249][  T172] locks held by fio/3872977: 3, on CPU#1:
[144871.501945][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.503238][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.504494][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.505963][  T172] locks held by fio/3872978: 3, on CPU#12:
[144871.506558][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.508041][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.509561][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.510975][  T172] locks held by fio/3872979: 3, on CPU#4:
[144871.511616][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.512986][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.514342][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.515561][  T172] locks held by fio/3872980: 2, on CPU#2:
[144871.516337][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.517588][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.518941][  T172] locks held by fio/3872981: 3, on CPU#5:
[144871.519544][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.520980][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.522436][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.523761][  T172] locks held by fio/3872982: 3, on CPU#7:
[144871.524345][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.525617][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.526995][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.528345][  T172] locks held by fio/3872983: 2, on CPU#18:
[144871.529013][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.530159][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.531658][  T172] locks held by fio/3872984: 3, on CPU#21:
[144871.532344][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.533662][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.535127][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.536354][  T172] locks held by fio/3872985: 3, on CPU#14:
[144871.536893][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.538193][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.539344][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.540565][  T172] locks held by fio/3872986: 3, on CPU#23:
[144871.549064][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.550342][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.551412][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.552644][  T172] locks held by fio/3872987: 3, on CPU#9:
[144871.553252][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.554617][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.555963][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.557289][  T172] locks held by fio/3872988: 2, on CPU#9:
[144871.557983][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.559251][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.560650][  T172] locks held by fio/3872989: 3, on CPU#13:
[144871.561148][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.562620][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.563966][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.565353][  T172] locks held by fio/3872990: 3, on CPU#22:
[144871.566014][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.567292][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.568639][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.569912][  T172] locks held by fio/3872991: 3, on CPU#15:
[144871.570643][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.572074][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.573526][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.574865][  T172] locks held by fio/3872992: 3, on CPU#19:
[144871.575385][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.576702][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.577974][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.579176][  T172] locks held by fio/3872993: 2, on CPU#8:
[144871.579776][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.581296][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.582823][  T172] locks held by fio/3872994: 2, on CPU#10:
[144871.583327][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.584824][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.586338][  T172] locks held by fio/3872995: 2, on CPU#0:
[144871.586746][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.588065][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.589592][  T172] locks held by fio/3872996: 3, on CPU#3:
[144871.590228][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.591624][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.592955][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.594411][  T172] locks held by fio/3872997: 2, on CPU#9:
[144871.594880][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.596102][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.597443][  T172] locks held by fio/3872998: 3, on CPU#11:
[144871.598214][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.599581][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.600877][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.602277][  T172] locks held by fio/3872999: 2, on CPU#15:
[144871.602933][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.604171][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.605502][  T172] locks held by fio/3873000: 2, on CPU#13:
[144871.606086][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.607446][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.608891][  T172] locks held by fio/3873001: 2, on CPU#20:
[144871.609629][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.610941][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.612536][  T172] locks held by fio/3873002: 2, on CPU#21:
[144871.613261][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.614548][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.615849][  T172] locks held by fio/3873003: 2, on CPU#7:
[144871.616466][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.617936][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.619512][  T172] locks held by fio/3873004: 3, on CPU#12:
[144871.620084][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.621559][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.623035][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.624371][  T172] locks held by fio/3873005: 3, on CPU#2:
[144871.624968][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.626326][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.627504][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.628825][  T172] locks held by fio/3873006: 2, on CPU#5:
[144871.629555][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.630924][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.632295][  T172] locks held by fio/3873007: 3, on CPU#22:
[144871.632986][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.634247][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.635569][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.636907][  T172] locks held by fio/3873008: 3, on CPU#16:
[144871.637391][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.638706][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.639974][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.641349][  T172] locks held by kworker/22:9/3873014: 4, on CPU#22:
[144871.641954][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.643046][  T172]  #1: ff1100075a3b7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.643896][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.645235][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.646870][  T172] locks held by kworker/10:3/3873017: 4, on CPU#10:
[144871.647514][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.648341][  T172]  #1: ff110002411cfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.649336][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.650672][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.652162][  T172] locks held by kworker/16:5/3873018: 4, on CPU#16:
[144871.652872][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.653694][  T172]  #1: ff1100019cfd7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.654690][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.656010][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.657334][  T172] locks held by kworker/16:6/3873019: 4, on CPU#16:
[144871.657807][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.658834][  T172]  #1: ff110005ec73fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.659987][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.661147][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.662392][  T172] locks held by kworker/16:7/3873020: 4, on CPU#16:
[144871.662945][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.663830][  T172]  #1: ff110004e58a7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.664843][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.674237][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.675488][  T172] locks held by kworker/16:8/3873021: 4, on CPU#16:
[144871.676270][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.676948][  T172]  #1: ff110001e6617d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.677872][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.679079][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.680423][  T172] locks held by kworker/16:9/3873022: 4, on CPU#16:
[144871.680913][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.681864][  T172]  #1: ff11000156bbfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.682805][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.684009][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.685395][  T172] locks held by kworker/16:10/3873023: 4, on CPU#16:
[144871.685974][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.686863][  T172]  #1: ff1100040bf57d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.687686][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.689001][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.690449][  T172] locks held by kworker/17:5/3873025: 4, on CPU#17:
[144871.691061][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.691966][  T172]  #1: ff1100013957fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.692935][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.694197][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.695674][  T172] locks held by kworker/12:3/3873030: 4, on CPU#12:
[144871.696301][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.697143][  T172]  #1: ff110007b3e6fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.698099][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.699455][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.700840][  T172] 
[144871.701294][  T172] =============================================
#1 lore raw
Quoted 7 lines · Shin'ichiro Kawasaki

I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB TCMU fileio devices. Actually I once reported this hang symptom in last October, and Darrick and John kindly took some actions [1]. Since then, the hang disappeared and I have not observed the hang almost one year. However, my test system started reporting the hang again last week.

[1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t

This was the series to resolve, I think:
https://lore.kernel.org/fstests/176279908967.605950.2192923313361120314.stgit@frogsfrogsfrogs/T/#t

As an experiment, can you reduce the file size further, like:

tests/generic/774 +1 -1
@@ -29,7 +29,7 @@ aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))")
 fsbsize=$(_get_block_size $SCRATCH_MNT)
 
 threads=$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100")
-filesize=$((aw_bsize * threads * 10))
+filesize=$((aw_bsize * threads))
 depth=$threads
 aw_io_size=$((filesize / threads))
 aw_io_inc=$aw_io_size
Quoted 2 lines

FYI, here I attache the kernel message [2].

Is mainline baseline hanging, i.e. v7.3-rc2?

Quoted 914 lines

I observed the hang last week with
the kernel on the xfs/for-next branch at the commit 0ca15a1a1151 ("xfs: remove an extra cast in xfs_file_compat_ioctl"). Today, I tried again with the kernel at current xfs/for-next branch tip 4266ffdfd7cb ("xfs: remove an extra cast in xfs_file_compat_ioctl"), and observed the hang again. The hang can be recreated in stable manner by repeating the test case 20 times or so.

I also tried other test cases in atmoicwrites group, and observed no failure.

generic/765 [not run] write atomic not supported by this block device

  generic/767         11s
  generic/768         13s
  generic/769         13s
  generic/770         32s
  generic/773        [not run] write atomic not supported by this block device
  generic/774        (skipped)
  generic/775         293s
  generic/776        [notrun] write atomic not supported by this block device
  generic/778         48s
  xfs/838            [not run] External volumes not in use, skipped this test
  xfs/839            [not run] XFS error injection requires CONFIG_XFS_DEBUG
  xfs/840            [not run] write atomic not supported by this block device

Action for fix will be appreciated. If I can do anything on my test node, please let me know.

[2] kernel message during generic/774

[144539.456849][T1634092] run fstests generic/774 at 2026-09-09 16:51:53
[144539.691199][T3820710] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[144543.436776][T3868062] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[144543.648458][T3872722] XFS (sdg): Mounting V5 Filesystem 306fb2c4-6cd3-4cbd-b9ed-953c0bc0d3b8
[144543.674712][T3872722] XFS (sdg): Ending clean mount
[144544.328859][T3872774] XFS (sdg): Unmounting Filesystem 306fb2c4-6cd3-4cbd-b9ed-953c0bc0d3b8
[144545.743530][T3795425] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[144545.964812][T3872823] XFS (sdg): Mounting V5 Filesystem 91ee7431-7139-4d92-bea7-b7802c3f70d2
[144545.990603][T3872823] XFS (sdg): Ending clean mount
[144870.819650][  T172] INFO: task kworker/10:173:3785794 blocked for more than 122 seconds.
[144870.821440][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144870.823252][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144870.825054][  T172] task:kworker/10:173  state:D stack:0     pid:3785794 tgid:3785794 ppid:2      task_flags:0x4248060 flags:0x00080000
[144870.827098][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144870.828170][  T172] Call Trace:
[144870.829209][  T172]  <TASK>
[144870.830154][  T172]  __schedule+0xc86/0x1b10
[144870.831126][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144870.832166][  T172]  ? __pfx___schedule+0x10/0x10
[144870.833250][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.834400][  T172]  ? wq_worker_sleeping+0x213/0x280
[144870.835559][  T172]  ? find_held_lock+0x2b/0x80
[144870.836634][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144870.837699][  T172]  schedule+0xe5/0x2e0
[144870.838721][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.839915][  T172]  schedule_preempt_disabled+0x16/0x30
[144870.841017][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144870.842139][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144870.843236][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144870.844383][  T172]  ? __lock_acquire+0x691/0xd00
[144870.845598][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.846840][  T172]  ? lock_acquire+0x11a/0x140
[144870.848019][  T172]  down_write_nested+0x1c0/0x1f0
[144870.849121][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144870.850284][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144870.852895][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144870.855451][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144870.858004][  T172]  ? __lock_acquire+0x691/0xd00
[144870.859160][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144870.861775][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144870.864257][  T172]  iomap_dio_complete+0x140/0xb20
[144870.865378][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144870.866570][  T172]  iomap_dio_complete_work+0x58/0x90
[144870.867785][  T172]  process_one_work+0x947/0x1760
[144870.868995][  T172]  ? __pfx_process_one_work+0x10/0x10
[144870.870062][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.871227][  T172]  ? lock_is_held_type+0xb0/0x180
[144870.872361][  T172]  worker_thread+0x601/0xff0
[144870.873609][  T172]  ? __pfx_worker_thread+0x10/0x10
[144870.874813][  T172]  kthread+0x361/0x460
[144870.875873][  T172]  ? __pfx_kthread+0x10/0x10
[144870.876881][  T172]  ret_from_fork+0x560/0x860
[144870.878012][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144870.879036][  T172]  ? __switch_to+0x473/0xd50
[144870.880200][  T172]  ? __pfx_kthread+0x10/0x10
[144870.881232][  T172]  ret_from_fork_asm+0x1a/0x30
[144870.882353][  T172]  </TASK>
[144870.883486][  T172] INFO: task kworker/10:173:3785794 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144870.885498][  T172] INFO: task kworker/18:2:3812136 blocked for more than 122 seconds.
[144870.887038][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144870.888467][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144870.890050][  T172] task:kworker/18:2    state:D stack:0     pid:3812136 tgid:3812136 ppid:2      task_flags:0x4248060 flags:0x00080000
[144870.892033][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144870.893266][  T172] Call Trace:
[144870.894349][  T172]  <TASK>
[144870.895228][  T172]  __schedule+0xc86/0x1b10
[144870.896439][  T172]  ? __pfx___schedule+0x10/0x10
[144870.897500][  T172]  ? schedule+0x17e/0x2e0
[144870.898441][  T172]  ? wq_worker_sleeping+0x65/0x280
[144870.899427][  T172]  ? find_held_lock+0x2b/0x80
[144870.900392][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144870.901382][  T172]  schedule+0xe5/0x2e0
[144870.902249][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.903287][  T172]  schedule_preempt_disabled+0x16/0x30
[144870.904511][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144870.905518][  T172]  ? lock_release.part.0+0x1c/0x50
[144870.906482][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144870.907575][  T172]  ? __lock_acquire+0x691/0xd00
[144870.908506][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.939885][  T172]  ? lock_acquire+0x11a/0x140
[144870.940681][  T172]  down_write_nested+0x1c0/0x1f0
[144870.941553][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144870.942530][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144870.944492][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144870.946365][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144870.948153][  T172]  ? __lock_acquire+0x691/0xd00
[144870.949019][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144870.950793][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144870.952653][  T172]  iomap_dio_complete+0x140/0xb20
[144870.953439][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144870.954096][  T172]  iomap_dio_complete_work+0x58/0x90
[144870.954877][  T172]  process_one_work+0x947/0x1760
[144870.955644][  T172]  ? __pfx_process_one_work+0x10/0x10
[144870.956399][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.957121][  T172]  ? lock_is_held_type+0xb0/0x180
[144870.957949][  T172]  worker_thread+0x601/0xff0
[144870.958860][  T172]  ? __pfx_worker_thread+0x10/0x10
[144870.959586][  T172]  kthread+0x361/0x460
[144870.960273][  T172]  ? __pfx_kthread+0x10/0x10
[144870.960891][  T172]  ret_from_fork+0x560/0x860
[144870.961610][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144870.962503][  T172]  ? __switch_to+0x473/0xd50
[144870.963250][  T172]  ? __pfx_kthread+0x10/0x10
[144870.963930][  T172]  ret_from_fork_asm+0x1a/0x30
[144870.964760][  T172]  </TASK>
[144870.965444][  T172] INFO: task kworker/18:2:3812136 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144870.966764][  T172] INFO: task kworker/20:1:3814407 blocked for more than 123 seconds.
[144870.967875][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144870.968920][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144870.970043][  T172] task:kworker/20:1    state:D stack:0     pid:3814407 tgid:3814407 ppid:2      task_flags:0x4248060 flags:0x00080000
[144870.971395][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144870.972198][  T172] Call Trace:
[144870.972772][  T172]  <TASK>
[144870.973279][  T172]  __schedule+0xc86/0x1b10
[144870.973888][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144870.974596][  T172]  ? __pfx___schedule+0x10/0x10
[144870.975254][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.975899][  T172]  ? wq_worker_sleeping+0x213/0x280
[144870.976689][  T172]  ? find_held_lock+0x2b/0x80
[144870.977241][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144870.977916][  T172]  schedule+0xe5/0x2e0
[144870.978586][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144870.979182][  T172]  schedule_preempt_disabled+0x16/0x30
[144870.979886][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144870.980635][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144870.981287][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144870.982017][  T172]  ? __lock_acquire+0x691/0xd00
[144870.982700][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.983391][  T172]  ? lock_acquire+0x11a/0x140
[144870.984047][  T172]  down_write_nested+0x1c0/0x1f0
[144870.984778][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144870.985569][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144870.987084][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144870.988587][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144870.990093][  T172]  ? lock_is_held_type+0xb0/0x180
[144870.990804][  T172]  ? __lock_acquire+0x691/0xd00
[144870.991435][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144870.992815][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144870.994256][  T172]  iomap_dio_complete+0x140/0xb20
[144870.994897][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144870.995533][  T172]  iomap_dio_complete_work+0x58/0x90
[144870.996229][  T172]  process_one_work+0x947/0x1760
[144870.996919][  T172]  ? __pfx_process_one_work+0x10/0x10
[144870.997519][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144870.998095][  T172]  ? lock_is_held_type+0xb0/0x180
[144870.998755][  T172]  worker_thread+0x601/0xff0
[144870.999262][  T172]  ? __pfx_worker_thread+0x10/0x10
[144870.999854][  T172]  kthread+0x361/0x460
[144871.000422][  T172]  ? __pfx_kthread+0x10/0x10
[144871.000971][  T172]  ret_from_fork+0x560/0x860
[144871.001596][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.002150][  T172]  ? __switch_to+0x473/0xd50
[144871.002764][  T172]  ? __pfx_kthread+0x10/0x10
[144871.003268][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.003871][  T172]  </TASK>
[144871.006437][  T172] INFO: task kworker/20:1:3814407 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.007488][  T172] INFO: task kworker/16:2:3814415 blocked for more than 123 seconds.
[144871.008373][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.009098][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.009935][  T172] task:kworker/16:2    state:D stack:0     pid:3814415 tgid:3814415 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.010908][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.011529][  T172] Call Trace:
[144871.011921][  T172]  <TASK>
[144871.012338][  T172]  __schedule+0xc86/0x1b10
[144871.012801][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144871.013341][  T172]  ? __pfx___schedule+0x10/0x10
[144871.013912][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.014491][  T172]  ? wq_worker_sleeping+0x213/0x280
[144871.015122][  T172]  ? find_held_lock+0x2b/0x80
[144871.015734][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.016274][  T172]  schedule+0xe5/0x2e0
[144871.016820][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.017430][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.017981][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.018500][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144871.019066][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.019691][  T172]  ? __lock_acquire+0x691/0xd00
[144871.020216][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.020806][  T172]  ? lock_acquire+0x11a/0x140
[144871.021350][  T172]  down_write_nested+0x1c0/0x1f0
[144871.021847][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.022365][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.023499][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.024814][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.026064][  T172]  ? update_load_avg+0xb65/0x2c70
[144871.026726][  T172]  ? __lock_acquire+0x691/0xd00
[144871.027267][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.028538][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.029728][  T172]  iomap_dio_complete+0x140/0xb20
[144871.030224][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.030743][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.031190][  T172]  process_one_work+0x947/0x1760
[144871.031764][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.032294][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.032889][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.033502][  T172]  worker_thread+0x601/0xff0
[144871.034040][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.034640][  T172]  kthread+0x361/0x460
[144871.035071][  T172]  ? __pfx_kthread+0x10/0x10
[144871.035679][  T172]  ret_from_fork+0x560/0x860
[144871.036162][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.036758][  T172]  ? __switch_to+0x473/0xd50
[144871.037334][  T172]  ? __pfx_kthread+0x10/0x10
[144871.037914][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.038394][  T172]  </TASK>
[144871.038867][  T172] INFO: task kworker/16:2:3814415 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.039846][  T172] INFO: task kworker/1:2:3819893 blocked for more than 123 seconds.
[144871.040650][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.041422][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.042136][  T172] task:kworker/1:2     state:D stack:0     pid:3819893 tgid:3819893 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.043104][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.043712][  T172] Call Trace:
[144871.044164][  T172]  <TASK>
[144871.044651][  T172]  __schedule+0xc86/0x1b10
[144871.045172][  T172]  ? __pfx___schedule+0x10/0x10
[144871.045707][  T172]  ? schedule+0x17e/0x2e0
[144871.046192][  T172]  ? wq_worker_sleeping+0x65/0x280
[144871.046741][  T172]  ? find_held_lock+0x2b/0x80
[144871.047186][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.047796][  T172]  schedule+0xe5/0x2e0
[144871.048299][  T172]  xlog_grant_head_wait+0x175/0xac0 [xfs]
[144871.049471][  T172]  xlog_grant_head_check+0x312/0x3f0 [xfs]
[144871.050717][  T172]  ? __pfx_xlog_grant_head_check+0x10/0x10 [xfs]
[144871.051932][  T172]  xfs_log_regrant+0x380/0x7d0 [xfs]
[144871.053059][  T172]  ? __pfx_xfs_log_regrant+0x10/0x10 [xfs]
[144871.054192][  T172]  ? xfs_log_ticket_get+0x1b/0x30 [xfs]
[144871.055259][  T172]  ? xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
[144871.056414][  T172]  xfs_trans_roll+0x2d9/0x420 [xfs]
[144871.057517][  T172]  ? kmem_cache_alloc_noprof+0x45b/0x680
[144871.058100][  T172]  ? xfs_defer_finish_noroll+0x460/0xe70 [xfs]
[144871.059201][  T172]  xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
[144871.060341][  T172]  ? __pfx_xfs_defer_trans_roll+0x10/0x10 [xfs]
[144871.061364][  T172]  ? xfs_defer_relog+0x3e7/0xb50 [xfs]
[144871.062454][  T172]  xfs_defer_finish_noroll+0x460/0xe70 [xfs]
[144871.063491][  T172]  ? __pfx_xfs_reflink_end_cow_extent_locked+0x10/0x10 [xfs]
[144871.064782][  T172]  ? __pfx_xfs_defer_finish_noroll+0x10/0x10 [xfs]
[144871.065859][  T172]  ? lock_acquire+0x11a/0x140
[144871.066413][  T172]  xfs_trans_commit+0xfc/0x180 [xfs]
[144871.067509][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.068046][  T172]  ? __pfx_xfs_trans_commit+0x10/0x10 [xfs]
[144871.069143][  T172]  xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
[144871.070340][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.071419][  T172]  ? __lock_acquire+0x691/0xd00
[144871.089717][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.090837][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.091794][  T172]  iomap_dio_complete+0x140/0xb20
[144871.092194][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.092684][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.093251][  T172]  process_one_work+0x947/0x1760
[144871.093761][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.094406][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.095084][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.095679][  T172]  worker_thread+0x601/0xff0
[144871.096234][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.096820][  T172]  kthread+0x361/0x460
[144871.097360][  T172]  ? __pfx_kthread+0x10/0x10
[144871.098065][  T172]  ret_from_fork+0x560/0x860
[144871.098693][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.099175][  T172]  ? __switch_to+0x473/0xd50
[144871.099709][  T172]  ? __pfx_kthread+0x10/0x10
[144871.100275][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.100783][  T172]  </TASK>
[144871.101203][  T172] INFO: task kworker/20:2:3824669 blocked for more than 123 seconds.
[144871.102109][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.102813][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.103685][  T172] task:kworker/20:2    state:D stack:0     pid:3824669 tgid:3824669 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.104588][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.105360][  T172] Call Trace:
[144871.105868][  T172]  <TASK>
[144871.106339][  T172]  __schedule+0xc86/0x1b10
[144871.106949][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144871.107632][  T172]  ? __pfx___schedule+0x10/0x10
[144871.108170][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.108600][  T172]  ? wq_worker_sleeping+0x213/0x280
[144871.109207][  T172]  ? find_held_lock+0x2b/0x80
[144871.109728][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.110270][  T172]  schedule+0xe5/0x2e0
[144871.110762][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.111328][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.111898][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.112410][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144871.112989][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.113648][  T172]  ? __lock_acquire+0x691/0xd00
[144871.114157][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.114665][  T172]  ? lock_acquire+0x11a/0x140
[144871.115101][  T172]  down_write_nested+0x1c0/0x1f0
[144871.115664][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.116122][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.117105][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.118239][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.119389][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.119781][  T172]  ? __lock_acquire+0x691/0xd00
[144871.120177][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.121294][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.122391][  T172]  iomap_dio_complete+0x140/0xb20
[144871.122770][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.123425][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.124051][  T172]  process_one_work+0x947/0x1760
[144871.124445][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.125017][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.125580][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.126138][  T172]  worker_thread+0x601/0xff0
[144871.126680][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.127126][  T172]  kthread+0x361/0x460
[144871.127762][  T172]  ? __pfx_kthread+0x10/0x10
[144871.128234][  T172]  ret_from_fork+0x560/0x860
[144871.128772][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.129337][  T172]  ? __switch_to+0x473/0xd50
[144871.129842][  T172]  ? __pfx_kthread+0x10/0x10
[144871.130364][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.130892][  T172]  </TASK>
[144871.131250][  T172] INFO: task kworker/20:2:3824669 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.132261][  T172] INFO: task kworker/4:0:3824816 blocked for more than 123 seconds.
[144871.133177][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.133912][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.134827][  T172] task:kworker/4:0     state:D stack:0     pid:3824816 tgid:3824816 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.135724][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.136356][  T172] Call Trace:
[144871.136856][  T172]  <TASK>
[144871.137342][  T172]  __schedule+0xc86/0x1b10
[144871.137818][  T172]  ? __pfx___schedule+0x10/0x10
[144871.138410][  T172]  ? schedule+0x17e/0x2e0
[144871.138810][  T172]  ? wq_worker_sleeping+0x65/0x280
[144871.139356][  T172]  ? find_held_lock+0x2b/0x80
[144871.139817][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.140631][  T172]  schedule+0xe5/0x2e0
[144871.141220][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.141887][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.142473][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.143026][  T172]  ? lock_release.part.0+0x1c/0x50
[144871.143557][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.144101][  T172]  ? __lock_acquire+0x691/0xd00
[144871.144681][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.145057][  T172]  ? lock_acquire+0x11a/0x140
[144871.145709][  T172]  down_write_nested+0x1c0/0x1f0
[144871.146205][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.146885][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.147929][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.148978][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.150061][  T172]  ? __lock_acquire+0x691/0xd00
[144871.150641][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.151842][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.152998][  T172]  iomap_dio_complete+0x140/0xb20
[144871.153548][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.154099][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.154759][  T172]  process_one_work+0x947/0x1760
[144871.155210][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.155675][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.156336][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.156920][  T172]  worker_thread+0x601/0xff0
[144871.157437][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.157951][  T172]  kthread+0x361/0x460
[144871.158466][  T172]  ? __pfx_kthread+0x10/0x10
[144871.159012][  T172]  ret_from_fork+0x560/0x860
[144871.159502][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.160015][  T172]  ? __switch_to+0x473/0xd50
[144871.160539][  T172]  ? __pfx_kthread+0x10/0x10
[144871.161051][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.161685][  T172]  </TASK>
[144871.162077][  T172] INFO: task kworker/4:0:3824816 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.162900][  T172] INFO: task kworker/12:0:3826849 blocked for more than 123 seconds.
[144871.163697][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.164415][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.165277][  T172] task:kworker/12:0    state:D stack:0     pid:3826849 tgid:3826849 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.166221][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.166952][  T172] Call Trace:
[144871.167283][  T172]  <TASK>
[144871.167853][  T172]  __schedule+0xc86/0x1b10
[144871.168439][  T172]  ? __pfx___schedule+0x10/0x10
[144871.168914][  T172]  ? schedule+0x17e/0x2e0
[144871.169451][  T172]  ? wq_worker_sleeping+0x65/0x280
[144871.169927][  T172]  ? find_held_lock+0x2b/0x80
[144871.170288][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.170911][  T172]  schedule+0xe5/0x2e0
[144871.171387][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.171939][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.172496][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.173087][  T172]  ? lock_release.part.0+0x1c/0x50
[144871.173702][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.174226][  T172]  ? __lock_acquire+0x691/0xd00
[144871.174891][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.175466][  T172]  ? lock_acquire+0x11a/0x140
[144871.176164][  T172]  down_write_nested+0x1c0/0x1f0
[144871.176754][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.177531][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.178526][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.179662][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.180880][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.181342][  T172]  ? __lock_acquire+0x691/0xd00
[144871.181872][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.183071][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.184221][  T172]  iomap_dio_complete+0x140/0xb20
[144871.184745][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.185338][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.185864][  T172]  process_one_work+0x947/0x1760
[144871.186361][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.186920][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.187381][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.187924][  T172]  worker_thread+0x601/0xff0
[144871.188591][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.189109][  T172]  kthread+0x361/0x460
[144871.189647][  T172]  ? __pfx_kthread+0x10/0x10
[144871.190131][  T172]  ret_from_fork+0x560/0x860
[144871.190578][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.191172][  T172]  ? __switch_to+0x473/0xd50
[144871.191771][  T172]  ? __pfx_kthread+0x10/0x10
[144871.192247][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.192760][  T172]  </TASK>
[144871.193146][  T172] INFO: task kworker/12:0:3826849 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.202516][  T172] INFO: task kworker/21:1:3827095 blocked for more than 123 seconds.
[144871.203182][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.203953][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.204755][  T172] task:kworker/21:1    state:D stack:0     pid:3827095 tgid:3827095 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.205642][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.206251][  T172] Call Trace:
[144871.206793][  T172]  <TASK>
[144871.207382][  T172]  __schedule+0xc86/0x1b10
[144871.207815][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144871.208402][  T172]  ? __pfx___schedule+0x10/0x10
[144871.208871][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.209404][  T172]  ? wq_worker_sleeping+0x213/0x280
[144871.209925][  T172]  ? find_held_lock+0x2b/0x80
[144871.210492][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.211050][  T172]  schedule+0xe5/0x2e0
[144871.211527][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.212043][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.212786][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.213398][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144871.213975][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.214563][  T172]  ? __lock_acquire+0x691/0xd00
[144871.215009][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.215710][  T172]  ? lock_acquire+0x11a/0x140
[144871.216164][  T172]  down_write_nested+0x1c0/0x1f0
[144871.216688][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.217212][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.218301][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.219411][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.220508][  T172]  ? __lock_acquire+0x691/0xd00
[144871.221057][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.222191][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.223292][  T172]  iomap_dio_complete+0x140/0xb20
[144871.223820][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.224332][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.224792][  T172]  process_one_work+0x947/0x1760
[144871.225354][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.225861][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.226458][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.226982][  T172]  worker_thread+0x601/0xff0
[144871.227454][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.227993][  T172]  ? __kthread_parkme+0xbd/0x210
[144871.228532][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.229057][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.229619][  T172]  kthread+0x361/0x460
[144871.230083][  T172]  ? __pfx_kthread+0x10/0x10
[144871.230578][  T172]  ret_from_fork+0x560/0x860
[144871.231049][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.231633][  T172]  ? __switch_to+0x473/0xd50
[144871.232124][  T172]  ? __pfx_kthread+0x10/0x10
[144871.232544][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.233081][  T172]  </TASK>
[144871.233440][  T172] INFO: task kworker/21:1:3827095 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.234328][  T172] INFO: task kworker/10:0:3827669 blocked for more than 123 seconds.
[144871.235091][  T172]       Tainted: G        W           7.3.0-rc2-kts-xfs-g0ca15a1a115+ #1
[144871.235858][  T172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[144871.236582][  T172] task:kworker/10:0    state:D stack:0     pid:3827669 tgid:3827669 ppid:2      task_flags:0x4248060 flags:0x00080000
[144871.237651][  T172] Workqueue: dio/sdg iomap_dio_complete_work
[144871.238173][  T172] Call Trace:
[144871.238946][  T172]  <TASK>
[144871.239469][  T172]  __schedule+0xc86/0x1b10
[144871.240084][  T172]  ? trace_hardirqs_on+0x18/0x1a0
[144871.240642][  T172]  ? __pfx___schedule+0x10/0x10
[144871.241088][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.241694][  T172]  ? wq_worker_sleeping+0x213/0x280
[144871.242326][  T172]  ? find_held_lock+0x2b/0x80
[144871.242813][  T172]  ? __lock_release.isra.0+0x69/0x1a0
[144871.243379][  T172]  schedule+0xe5/0x2e0
[144871.243881][  T172]  ? _raw_spin_unlock_irq+0x28/0x50
[144871.244378][  T172]  schedule_preempt_disabled+0x16/0x30
[144871.244920][  T172]  rwsem_down_write_slowpath+0x4b4/0x1460
[144871.245543][  T172]  ? __kasan_slab_alloc+0x7f/0x90
[144871.246066][  T172]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[144871.246662][  T172]  ? __lock_acquire+0x691/0xd00
[144871.247131][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.247711][  T172]  ? lock_acquire+0x11a/0x140
[144871.248149][  T172]  down_write_nested+0x1c0/0x1f0
[144871.248724][  T172]  ? __pfx_down_write_nested+0x10/0x10
[144871.249285][  T172]  ? xfs_ilock+0x1e7/0x310 [xfs]
[144871.250297][  T172]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.251346][  T172]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[144871.252414][  T172]  ? __lock_acquire+0x691/0xd00
[144871.252983][  T172]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[144871.254104][  T172]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[144871.255189][  T172]  iomap_dio_complete+0x140/0xb20
[144871.255756][  T172]  ? __pfx_aio_complete_rw+0x10/0x10
[144871.256283][  T172]  iomap_dio_complete_work+0x58/0x90
[144871.256814][  T172]  process_one_work+0x947/0x1760
[144871.257211][  T172]  ? __pfx_process_one_work+0x10/0x10
[144871.257962][  T172]  ? lock_acquire.part.0+0xc8/0x240
[144871.258415][  T172]  ? lock_is_held_type+0xb0/0x180
[144871.258907][  T172]  worker_thread+0x601/0xff0
[144871.259454][  T172]  ? __pfx_worker_thread+0x10/0x10
[144871.259949][  T172]  kthread+0x361/0x460
[144871.260364][  T172]  ? __pfx_kthread+0x10/0x10
[144871.260889][  T172]  ret_from_fork+0x560/0x860
[144871.261384][  T172]  ? __pfx_ret_from_fork+0x10/0x10
[144871.261926][  T172]  ? __switch_to+0x473/0xd50
[144871.262448][  T172]  ? __pfx_kthread+0x10/0x10
[144871.262968][  T172]  ret_from_fork_asm+0x1a/0x30
[144871.263492][  T172]  </TASK>
[144871.263903][  T172] INFO: task kworker/10:0:3827669 <writer> blocked on an rw-semaphore likely owned by task kworker/1:2:3819893 <writer>
[144871.264856][  T172] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
[144871.265760][  T172]
[144871.265760][  T172] Showing all locks held in the system:
[144871.266766][  T172] locks held by pr/ttyS0/16: 2, last CPU#22:
[144871.267407][  T172]  #0: ffffffff8c320658 (console_srcu){....}-{0:0}, at: console_srcu_read_lock+0x49/0x70
[144871.268258][  T172]  #1: ffffffff9c3db638 (&port_lock_key){-.-.}-{3:3}, at: univ8250_console_device_lock+0x63/0xc0
[144871.269176][  T172] locks held by khungtaskd/172: 1, last CPU#20:
[144871.269752][  T172]  #0: ffffffff8c3369e0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire.constprop.0+0x7/0x30
[144871.270692][  T172] locks held by kworker/10:173/3785794: 4, on CPU#10:
[144871.271375][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.272341][  T172]  #1: ff11000109407d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.273284][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.274707][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.276050][  T172] locks held by kworker/18:2/3812136: 4, on CPU#18:
[144871.276811][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.277696][  T172]  #1: ff1100018e71fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.278634][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.280090][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.281626][  T172] locks held by kworker/20:1/3814407: 4, on CPU#20:
[144871.282267][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.283196][  T172]  #1: ff11000214267d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.284154][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.285620][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.287063][  T172] locks held by kworker/16:2/3814415: 4, on CPU#16:
[144871.287789][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.288637][  T172]  #1: ff1100019cc67d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.289792][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.291054][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.292462][  T172] locks held by kworker/1:2/3819893: 4, on CPU#1:
[144871.293085][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.294070][  T172]  #1: ff110001246d7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.294991][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.296348][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.297789][  T172] locks held by kworker/20:2/3824669: 4, on CPU#20:
[144871.298349][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.299205][  T172]  #1: ff1100022fc77d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.300225][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.301638][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.303077][  T172] locks held by kworker/4:0/3824816: 4, on CPU#4:
[144871.303744][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.312636][  T172]  #1: ff11000424997d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.313372][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.314586][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.315921][  T172] locks held by kworker/12:0/3826849: 4, on CPU#12:
[144871.316445][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.317222][  T172]  #1: ff110001e71bfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.318124][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.319340][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.320712][  T172] locks held by kworker/21:1/3827095: 4, on CPU#21:
[144871.321339][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.322330][  T172]  #1: ff11000172fefd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.323140][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.324642][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.326042][  T172] locks held by kworker/10:0/3827669: 4, on CPU#10:
[144871.326655][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.327575][  T172]  #1: ff11000122177d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.328366][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.329889][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.331476][  T172] locks held by kworker/17:1/3827670: 4, on CPU#17:
[144871.332105][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.332897][  T172]  #1: ff110001d0307d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.333703][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.334981][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.336354][  T172] locks held by kworker/16:1/3864132: 4, on CPU#16:
[144871.337033][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.337973][  T172]  #1: ff1100023211fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.338641][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.339966][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.341388][  T172] locks held by kworker/8:1/3868486: 4, on CPU#8:
[144871.342196][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.343143][  T172]  #1: ff110001a24d7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.344179][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.345576][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.347006][  T172] locks held by kworker/10:1/3872909: 4, on CPU#10:
[144871.347626][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.348518][  T172]  #1: ff110001606dfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.349516][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.350857][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.352433][  T172] locks held by kworker/17:2/3872910: 4, on CPU#17:
[144871.353061][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.354161][  T172]  #1: ff1100041c1b7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.354964][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.356302][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.357755][  T172] locks held by kworker/23:0/3872911: 4, on CPU#23:
[144871.358527][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.359329][  T172]  #1: ff110004b8c8fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.360223][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.361505][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.363110][  T172] locks held by kworker/7:3/3872919: 4, on CPU#7:
[144871.363803][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.364639][  T172]  #1: ff1100019af97d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.365488][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.366938][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.368364][  T172] locks held by kworker/20:3/3872924: 4, on CPU#20:
[144871.369008][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.369969][  T172]  #1: ff110001aa42fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.371118][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.372518][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.373983][  T172] locks held by kworker/3:2/3872925: 4, on CPU#3:
[144871.374629][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.375515][  T172]  #1: ff11000444b8fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.376358][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.377586][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.378950][  T172] locks held by kworker/16:3/3872926: 4, on CPU#16:
[144871.379685][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.380610][  T172]  #1: ff11000682cdfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.381567][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.382968][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.384263][  T172] locks held by kworker/17:3/3872927: 4, on CPU#17:
[144871.384833][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.385697][  T172]  #1: ff110001d3ca7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.386411][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.387778][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.389139][  T172] locks held by kworker/16:4/3872929: 4, on CPU#16:
[144871.389779][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.390582][  T172]  #1: ff110003cb7efd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.391402][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.392568][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.394019][  T172] locks held by kworker/19:2/3872930: 4, on CPU#19:
[144871.394685][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.395449][  T172]  #1: ff1100021374fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.396394][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.397664][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.399332][  T172] locks held by kworker/18:3/3872935: 3, on CPU#18:
[144871.400131][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.401110][  T172]  #1: ff110002862b7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.402243][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.403502][  T172] locks held by kworker/10:2/3872936: 4, on CPU#10:
[144871.404186][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.405139][  T172]  #1: ff110003d8e8fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.406011][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.407341][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.408729][  T172] locks held by kworker/8:3/3872941: 3, on CPU#8:
[144871.409141][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.410067][  T172]  #1: ff110003df1e7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.410799][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.411989][  T172] locks held by kworker/1:3/3872953: 4, on CPU#1:
[144871.412584][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.413531][  T172]  #1: ff1100043f1bfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.414464][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.424444][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.425808][  T172] locks held by kworker/4:4/3872955: 4, on CPU#4:
[144871.426603][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.427468][  T172]  #1: ff1100014de77d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.428336][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.429615][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.430975][  T172] locks held by fio/3872961: 3, on CPU#2:
[144871.431620][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.433089][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.434279][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.435797][  T172] locks held by fio/3872962: 3, on CPU#23:
[144871.436291][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.437618][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.438845][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.440124][  T172] locks held by fio/3872963: 3, on CPU#23:
[144871.440772][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.442297][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.443694][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.445025][  T172] locks held by fio/3872964: 3, on CPU#14:
[144871.445514][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.446890][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.448097][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.449466][  T172] locks held by fio/3872965: 3, on CPU#12:
[144871.450120][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.451359][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.452681][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.454164][  T172] locks held by fio/3872966: 3, on CPU#16:
[144871.454720][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.455979][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.457331][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.458737][  T172] locks held by fio/3872967: 3, on CPU#4:
[144871.459369][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.460809][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.461990][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.463126][  T172] locks held by fio/3872968: 3, on CPU#1:
[144871.463662][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.464985][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.466188][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.467438][  T172] locks held by fio/3872969: 3, on CPU#6:
[144871.468111][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.469612][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.471012][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.472352][  T172] locks held by fio/3872970: 3, on CPU#4:
[144871.473018][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.474395][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.475759][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.477331][  T172] locks held by fio/3872971: 3, on CPU#14:
[144871.477764][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.479087][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.480334][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.481548][  T172] locks held by fio/3872972: 2, on CPU#10:
[144871.482113][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.483336][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.484625][  T172] locks held by fio/3872973: 3, on CPU#10:
[144871.485350][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.486616][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.487949][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.489152][  T172] locks held by fio/3872974: 2, on CPU#18:
[144871.489572][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.491050][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.492416][  T172] locks held by fio/3872975: 3, on CPU#8:
[144871.493166][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.494401][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.495553][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.496938][  T172] locks held by fio/3872976: 3, on CPU#23:
[144871.497422][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.498613][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.499903][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.501249][  T172] locks held by fio/3872977: 3, on CPU#1:
[144871.501945][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.503238][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.504494][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.505963][  T172] locks held by fio/3872978: 3, on CPU#12:
[144871.506558][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.508041][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.509561][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.510975][  T172] locks held by fio/3872979: 3, on CPU#4:
[144871.511616][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.512986][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.514342][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.515561][  T172] locks held by fio/3872980: 2, on CPU#2:
[144871.516337][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.517588][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.518941][  T172] locks held by fio/3872981: 3, on CPU#5:
[144871.519544][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.520980][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.522436][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.523761][  T172] locks held by fio/3872982: 3, on CPU#7:
[144871.524345][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.525617][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.526995][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.528345][  T172] locks held by fio/3872983: 2, on CPU#18:
[144871.529013][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.530159][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.531658][  T172] locks held by fio/3872984: 3, on CPU#21:
[144871.532344][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.533662][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.535127][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.536354][  T172] locks held by fio/3872985: 3, on CPU#14:
[144871.536893][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.538193][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.539344][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.540565][  T172] locks held by fio/3872986: 3, on CPU#23:
[144871.549064][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.550342][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.551412][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.552644][  T172] locks held by fio/3872987: 3, on CPU#9:
[144871.553252][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.554617][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.555963][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.557289][  T172] locks held by fio/3872988: 2, on CPU#9:
[144871.557983][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.559251][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.560650][  T172] locks held by fio/3872989: 3, on CPU#13:
[144871.561148][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.562620][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.563966][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.565353][  T172] locks held by fio/3872990: 3, on CPU#22:
[144871.566014][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.567292][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.568639][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.569912][  T172] locks held by fio/3872991: 3, on CPU#15:
[144871.570643][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.572074][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.573526][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.574865][  T172] locks held by fio/3872992: 3, on CPU#19:
[144871.575385][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.576702][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.577974][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.579176][  T172] locks held by fio/3872993: 2, on CPU#8:
[144871.579776][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.581296][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.582823][  T172] locks held by fio/3872994: 2, on CPU#10:
[144871.583327][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.584824][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.586338][  T172] locks held by fio/3872995: 2, on CPU#0:
[144871.586746][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.588065][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.589592][  T172] locks held by fio/3872996: 3, on CPU#3:
[144871.590228][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.591624][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.592955][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.594411][  T172] locks held by fio/3872997: 2, on CPU#9:
[144871.594880][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.596102][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.597443][  T172] locks held by fio/3872998: 3, on CPU#11:
[144871.598214][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.599581][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.600877][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.602277][  T172] locks held by fio/3872999: 2, on CPU#15:
[144871.602933][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.604171][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.605502][  T172] locks held by fio/3873000: 2, on CPU#13:
[144871.606086][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.607446][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.608891][  T172] locks held by fio/3873001: 2, on CPU#20:
[144871.609629][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.610941][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.612536][  T172] locks held by fio/3873002: 2, on CPU#21:
[144871.613261][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.614548][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.615849][  T172] locks held by fio/3873003: 2, on CPU#7:
[144871.616466][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.617936][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.619512][  T172] locks held by fio/3873004: 3, on CPU#12:
[144871.620084][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.621559][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.623035][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.624371][  T172] locks held by fio/3873005: 3, on CPU#2:
[144871.624968][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.626326][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.627504][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.628825][  T172] locks held by fio/3873006: 2, on CPU#5:
[144871.629555][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.630924][  T172]  #1: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_begin.isra.0+0x31d/0xb60 [xfs]
[144871.632295][  T172] locks held by fio/3873007: 3, on CPU#22:
[144871.632986][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.634247][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.635569][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x27f/0x490 [xfs]
[144871.636907][  T172] locks held by fio/3873008: 3, on CPU#16:
[144871.637391][  T172]  #0: ff11000100bb18d0 (&sb->s_type->i_mutex_key#28){++++}-{4:4}, at: xfs_ilock+0x181/0x310 [xfs]
[144871.638706][  T172]  #1: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.639974][  T172]  #2: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_trans_alloc_inode+0x17b/0x340 [xfs]
[144871.641349][  T172] locks held by kworker/22:9/3873014: 4, on CPU#22:
[144871.641954][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.643046][  T172]  #1: ff1100075a3b7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.643896][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.645235][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.646870][  T172] locks held by kworker/10:3/3873017: 4, on CPU#10:
[144871.647514][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.648341][  T172]  #1: ff110002411cfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.649336][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.650672][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.652162][  T172] locks held by kworker/16:5/3873018: 4, on CPU#16:
[144871.652872][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.653694][  T172]  #1: ff1100019cfd7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.654690][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.656010][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.657334][  T172] locks held by kworker/16:6/3873019: 4, on CPU#16:
[144871.657807][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.658834][  T172]  #1: ff110005ec73fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.659987][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.661147][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.662392][  T172] locks held by kworker/16:7/3873020: 4, on CPU#16:
[144871.662945][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.663830][  T172]  #1: ff110004e58a7d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.664843][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.674237][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.675488][  T172] locks held by kworker/16:8/3873021: 4, on CPU#16:
[144871.676270][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.676948][  T172]  #1: ff110001e6617d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.677872][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.679079][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.680423][  T172] locks held by kworker/16:9/3873022: 4, on CPU#16:
[144871.680913][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.681864][  T172]  #1: ff11000156bbfd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.682805][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.684009][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.685395][  T172] locks held by kworker/16:10/3873023: 4, on CPU#16:
[144871.685974][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.686863][  T172]  #1: ff1100040bf57d10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.687686][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.689001][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.690449][  T172] locks held by kworker/17:5/3873025: 4, on CPU#17:
[144871.691061][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.691966][  T172]  #1: ff1100013957fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.692935][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.694197][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.695674][  T172] locks held by kworker/12:3/3873030: 4, on CPU#12:
[144871.696301][  T172]  #0: ff1100015b4ee940 ((wq_completion)dio/sdg){+.+.}-{0:0}, at: process_one_work+0x1111/0x1760
[144871.697143][  T172]  #1: ff110007b3e6fd10 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_one_work+0x8c3/0x1760
[144871.698099][  T172]  #2: ff110002d3ad4650 (sb_internal#2){++++}-{0:0}, at: xfs_trans_alloc+0x7b/0x820 [xfs]
[144871.699455][  T172]  #3: ff11000100bb16c0 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[144871.700840][  T172]
[144871.701294][  T172] =============================================
#2 lore raw
Quoted 17 lines · John Garry
Quoted 7 lines · Shin'ichiro Kawasaki

I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB TCMU fileio devices. Actually I once reported this hang symptom in last October, and Darrick and John kindly took some actions [1]. Since then, the hang disappeared and I have not observed the hang almost one year. However, my test system started reporting the hang again last week.

[1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t

This was the series to resolve, I think:
https://lore.kernel.org/fstests/176279908967.605950.2192923313361120314.stgit@frogsfrogsfrogs/T/#t

As an experiment, can you reduce the file size further, like:

tests/generic/774 aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))") · line 35

Quoted 8 lines
tests/generic/774 +1 -1
@@ -29,7 +29,7 @@ aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))")
 fsbsize=$(_get_block_size $SCRATCH_MNT)
 
 threads=$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100")
-filesize=$((aw_bsize * threads * 10))
+filesize=$((aw_bsize * threads))
 depth=$threads
 aw_io_size=$((filesize / threads))
 aw_io_inc=$aw_io_size

Thanks for the response. I repeated the test case 100 times with the change above, and I did not observe the hang. The change looks avoiding the hang.

Quoted 5 lines
Quoted 3 lines

FYI, here I attache the kernel message [2].

Is mainline baseline hanging, i.e. v7.3-rc2?

I will try this out tomorrow.

#3 lore raw
Quoted 32 lines · Shin'ichiro Kawasaki
Quoted 17 lines · John Garry
Quoted 7 lines · Shin'ichiro Kawasaki

I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB TCMU fileio devices. Actually I once reported this hang symptom in last October, and Darrick and John kindly took some actions [1]. Since then, the hang disappeared and I have not observed the hang almost one year. However, my test system started reporting the hang again last week.

[1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t

This was the series to resolve, I think:
https://lore.kernel.org/fstests/176279908967.605950.2192923313361120314.stgit@frogsfrogsfrogs/T/#t

As an experiment, can you reduce the file size further, like:

tests/generic/774 aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))") · line 35

Quoted 8 lines
tests/generic/774 +1 -1
@@ -29,7 +29,7 @@ aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))")
  fsbsize=$(_get_block_size $SCRATCH_MNT)
 
  threads=$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100")
-filesize=$((aw_bsize * threads * 10))
+filesize=$((aw_bsize * threads))
  depth=$threads
  aw_io_size=$((filesize / threads))
  aw_io_inc=$aw_io_size

Thanks for the response. I repeated the test case 100 times with the change above, and I did not observe the hang. The change looks avoiding the hang.

Quoted 2 lines
Quoted 2 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

FYI, here I attache the kernel message [2].

Is mainline baseline hanging, i.e. v7.3-rc2?

I will try this out tomorrow.

thanks

About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for CoW-based atomics could help avoid this issue as we seem to be bogged down in lock contention.

#4 lore raw
Quoted 37 lines · John Garry
Quoted 36 lines · Shin'ichiro Kawasaki
Quoted 17 lines · John Garry
Quoted 7 lines · Shin'ichiro Kawasaki

I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB TCMU fileio devices. Actually I once reported this hang symptom in last October, and Darrick and John kindly took some actions [1]. Since then, the hang disappeared and I have not observed the hang almost one year. However, my test system started reporting the hang again last week.

[1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t

This was the series to resolve, I think:
https://lore.kernel.org/fstests/176279908967.605950.2192923313361120314.stgit@frogsfrogsfrogs/T/#t

As an experiment, can you reduce the file size further, like:

tests/generic/774 aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))") · line 35

Quoted 8 lines
tests/generic/774 +1 -1
@@ -29,7 +29,7 @@ aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))")
  fsbsize=$(_get_block_size $SCRATCH_MNT)
 
  threads=$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100")
-filesize=$((aw_bsize * threads * 10))
+filesize=$((aw_bsize * threads))
  depth=$threads
  aw_io_size=$((filesize / threads))
  aw_io_inc=$aw_io_size

Thanks for the response. I repeated the test case 100 times with the change above, and I did not observe the hang. The change looks avoiding the hang.

Quoted 5 lines
Quoted 3 lines

FYI, here I attache the kernel message [2].

Is mainline baseline hanging, i.e. v7.3-rc2?

Quoted 7 lines
Quoted 1 line

I will try this out tomorrow.

thanks

About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for CoW-based atomics could help avoid this issue as we seem to be bogged down in lock contention.

Which lock is being contended, anyway? It looks like the ILOCK?

--D

#5 lore raw
Quoted 6 lines · Darrick J. Wong
Quoted 5 lines

thanks

About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for CoW-based atomics could help avoid this issue as we seem to be bogged down in lock contention.

Which lock is being contended, anyway? It looks like the ILOCK?

Yeah, I think so Re. ilock. I had some perf data illustrating this from last year, which I can't seem to find, so I will re-generate it.

#11 lore raw
In reply to Shin'ichiro Kawasaki
Quoted 32 lines · Shin'ichiro Kawasaki
Quoted 17 lines · John Garry
Quoted 7 lines · Shin'ichiro Kawasaki

I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB TCMU fileio devices. Actually I once reported this hang symptom in last October, and Darrick and John kindly took some actions [1]. Since then, the hang disappeared and I have not observed the hang almost one year. However, my test system started reporting the hang again last week.

[1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t

This was the series to resolve, I think:
https://lore.kernel.org/fstests/176279908967.605950.2192923313361120314.stgit@frogsfrogsfrogs/T/#t

As an experiment, can you reduce the file size further, like:

tests/generic/774 aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))") · line 35

Quoted 8 lines
tests/generic/774 +1 -1
@@ -29,7 +29,7 @@ aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))")
 fsbsize=$(_get_block_size $SCRATCH_MNT)
 
 threads=$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100")
-filesize=$((aw_bsize * threads * 10))
+filesize=$((aw_bsize * threads))
 depth=$threads
 aw_io_size=$((filesize / threads))
 aw_io_inc=$aw_io_size

Thanks for the response. I repeated the test case 100 times with the change above, and I did not observe the hang. The change looks avoiding the hang.

Quoted 2 lines
Quoted 2 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

FYI, here I attache the kernel message [2].

Is mainline baseline hanging, i.e. v7.3-rc2?

I will try this out tomorrow.

I observed the hang with the mainline v7.3-rc2 kernel also:

[  516.678848] [   T5360] run fstests generic/774 at 2026-09-16 11:01:26
[  516.808855] [   T2710] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[  519.197067] [   T2825] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[  519.325438] [   T6378] XFS (sdh): Mounting V5 Filesystem f004c5a8-b5fd-4f05-9ea1-977c32c9fc76
[  519.342073] [   T6378] XFS (sdh): Ending clean mount
[  519.704393] [   T6429] XFS (sdh): Unmounting Filesystem f004c5a8-b5fd-4f05-9ea1-977c32c9fc76
[  520.341237] [   T2904] MODE SENSE: unimplemented page/subpage: 0x0a/0x05
[  520.461538] [   T6477] XFS (sdh): Mounting V5 Filesystem dcd73d62-bc80-42a1-9893-7815befb6fd1
[  520.480790] [   T6477] XFS (sdh): Ending clean mount
[  862.484216] [    T169] INFO: task kworker/0:0:9 blocked for more than 122 seconds.
[  862.486738] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.488497] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.491457] [    T169] task:kworker/0:0     state:D stack:0     pid:9     tgid:9     ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.493590] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.494696] [    T169] Call Trace:
[  862.496681] [    T169]  <TASK>
[  862.497741] [    T169]  __schedule+0xc86/0x1b10
[  862.499020] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  862.500434] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.501999] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.503638] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  862.505095] [    T169]  ? lock_acquire+0x11a/0x140
[  862.506540] [    T169]  schedule+0xe5/0x2e0
[  862.507546] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.508858] [    T169]  schedule_preempt_disabled+0x16/0x30
[  862.510463] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  862.511857] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  862.513526] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.514807] [    T169]  ? lock_acquire+0x11a/0x140
[  862.516272] [    T169]  down_write_nested+0x1c0/0x1f0
[  862.517706] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  862.519465] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  862.522491] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  862.525468] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.528537] [    T169]  ? lock_acquire+0x11a/0x140
[  862.529870] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.531028] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.533879] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.536645] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.537895] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.539424] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.540711] [    T169]  process_one_work+0x947/0x1760
[  862.542425] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.543881] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.545586] [    T169]  worker_thread+0x601/0xff0
[  862.547505] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.548953] [    T169]  kthread+0x361/0x460
[  862.550636] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.552067] [    T169]  ret_from_fork+0x560/0x860
[  862.598716] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.599886] [    T169]  ? __switch_to+0x473/0xd50
[  862.601344] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.602570] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.604391] [    T169]  </TASK>
[  862.605720] [    T169] INFO: task kworker/0:0:9 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  862.608060] [    T169] INFO: task kworker/10:0:95 blocked for more than 123 seconds.
[  862.610611] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.612670] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.615074] [    T169] task:kworker/10:0    state:D stack:0     pid:95    tgid:95    ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.617510] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.618863] [    T169] Call Trace:
[  862.619984] [    T169]  <TASK>
[  862.621425] [    T169]  __schedule+0xc86/0x1b10
[  862.622635] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  862.623900] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.625384] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.626539] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  862.627851] [    T169]  ? rcu_is_watching+0x69/0x90
[  862.629309] [    T169]  ? lock_acquire+0x11a/0x140
[  862.630501] [    T169]  schedule+0xe5/0x2e0
[  862.631786] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.633410] [    T169]  schedule_preempt_disabled+0x16/0x30
[  862.634482] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  862.635829] [    T169]  ? __kasan_slab_alloc+0x7f/0x90
[  862.637088] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  862.638545] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.639768] [    T169]  ? lock_acquire+0x11a/0x140
[  862.641378] [    T169]  down_write_nested+0x1c0/0x1f0
[  862.642616] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  862.644347] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  862.646648] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  862.649368] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.651640] [    T169]  ? update_load_avg+0xb65/0x2c70
[  862.652721] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.653736] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.656196] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.658537] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.659475] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.660451] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.661510] [    T169]  process_one_work+0x947/0x1760
[  862.662631] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.663652] [    T169]  ? __try_to_del_timer_sync+0xd3/0x130
[  862.664757] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.665845] [    T169]  worker_thread+0x601/0xff0
[  862.666850] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.667903] [    T169]  kthread+0x361/0x460
[  862.669351] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.670628] [    T169]  ret_from_fork+0x560/0x860
[  862.671393] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.672180] [    T169]  ? __switch_to+0x473/0xd50
[  862.673196] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.674012] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.675087] [    T169]  </TASK>
[  862.675904] [    T169] INFO: task kworker/10:0:95 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  862.677484] [    T169] INFO: task kworker/13:0:107 blocked for more than 123 seconds.
[  862.678751] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.679934] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.681454] [    T169] task:kworker/13:0    state:D stack:0     pid:107   tgid:107   ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.683148] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.684174] [    T169] Call Trace:
[  862.685025] [    T169]  <TASK>
[  862.685742] [    T169]  __schedule+0xc86/0x1b10
[  862.686690] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  862.687637] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.688543] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.689410] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  862.690380] [    T169]  ? lock_acquire+0x11a/0x140
[  862.691570] [    T169]  schedule+0xe5/0x2e0
[  862.692351] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.693301] [    T169]  schedule_preempt_disabled+0x16/0x30
[  862.694317] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  862.695571] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  862.696636] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.697620] [    T169]  ? lock_acquire+0x11a/0x140
[  862.698603] [    T169]  down_write_nested+0x1c0/0x1f0
[  862.699567] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  862.700843] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  862.702991] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  862.705330] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.707139] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.708286] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.709873] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.711607] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.712622] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.713758] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.714854] [    T169]  process_one_work+0x947/0x1760
[  862.715809] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.716900] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.718142] [    T169]  worker_thread+0x601/0xff0
[  862.719263] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.720613] [    T169]  kthread+0x361/0x460
[  862.721568] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.722681] [    T169]  ret_from_fork+0x560/0x860
[  862.723567] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.724692] [    T169]  ? __switch_to+0x473/0xd50
[  862.725643] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.726494] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.727727] [    T169]  </TASK>
[  862.728590] [    T169] INFO: task kworker/13:0:107 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  862.730308] [    T169] INFO: task kworker/18:0:131 blocked for more than 123 seconds.
[  862.731761] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.732954] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.734431] [    T169] task:kworker/18:0    state:D stack:0     pid:131   tgid:131   ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.735987] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.737180] [    T169] Call Trace:
[  862.737922] [    T169]  <TASK>
[  862.738784] [    T169]  __schedule+0xc86/0x1b10
[  862.739699] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  862.740958] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.742223] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.743837] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  862.745556] [    T169]  ? lock_acquire+0x11a/0x140
[  862.747573] [    T169]  schedule+0xe5/0x2e0
[  862.749001] [    T169]  xlog_grant_head_wait+0x175/0xac0 [xfs]
[  862.751614] [    T169]  xlog_grant_head_check+0x312/0x3f0 [xfs]
[  862.754331] [    T169]  ? __pfx_xlog_grant_head_check+0x10/0x10 [xfs]
[  862.757018] [    T169]  xfs_log_reserve+0x2b2/0x7a0 [xfs]
[  862.759361] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.760571] [    T169]  ? __pfx_xfs_log_reserve+0x10/0x10 [xfs]
[  862.763313] [    T169]  ? xfs_dec_freecounter+0x81/0x520 [xfs]
[  862.765660] [    T169]  xfs_trans_alloc+0x1a4/0x820 [xfs]
[  862.767935] [    T169]  xfs_reflink_end_atomic_cow+0x1b1/0x560 [xfs]
[  862.770604] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.773762] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.774868] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.777326] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.779795] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.780989] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.782460] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.783667] [    T169]  process_one_work+0x947/0x1760
[  862.784759] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.786260] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.787550] [    T169]  worker_thread+0x601/0xff0
[  862.788602] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.789757] [    T169]  kthread+0x361/0x460
[  862.790931] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.791965] [    T169]  ret_from_fork+0x560/0x860
[  862.793177] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.794419] [    T169]  ? __switch_to+0x473/0xd50
[  862.795543] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.796560] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.797661] [    T169]  </TASK>
[  862.798604] [    T169] INFO: task kworker/21:0:149 blocked for more than 123 seconds.
[  862.800419] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.801700] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.803425] [    T169] task:kworker/21:0    state:D stack:0     pid:149   tgid:149   ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.805354] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.806351] [    T169] Call Trace:
[  862.807571] [    T169]  <TASK>
[  862.808374] [    T169]  __schedule+0xc86/0x1b10
[  862.809575] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  862.810650] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.811595] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.812602] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  862.813639] [    T169]  ? rcu_is_watching+0x69/0x90
[  862.814471] [    T169]  ? lock_acquire+0x11a/0x140
[  862.815458] [    T169]  schedule+0xe5/0x2e0
[  862.816473] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.817491] [    T169]  schedule_preempt_disabled+0x16/0x30
[  862.818527] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  862.819664] [    T169]  ? __kasan_slab_alloc+0x7f/0x90
[  862.820690] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  862.821863] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.823035] [    T169]  ? lock_acquire+0x11a/0x140
[  862.882304] [    T169]  down_write_nested+0x1c0/0x1f0
[  862.882992] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  862.884022] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  862.885942] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  862.888021] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.890047] [    T169]  ? update_load_avg+0xb65/0x2c70
[  862.890723] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.891859] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.893613] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.895520] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.896394] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.897155] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.898492] [    T169]  process_one_work+0x947/0x1760
[  862.899478] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.900184] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.901237] [    T169]  worker_thread+0x601/0xff0
[  862.901968] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.903572] [    T169]  kthread+0x361/0x460
[  862.904365] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.905281] [    T169]  ret_from_fork+0x560/0x860
[  862.906378] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.907247] [    T169]  ? __switch_to+0x473/0xd50
[  862.908264] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.909171] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.910168] [    T169]  </TASK>
[  862.910906] [    T169] INFO: task kworker/21:0:149 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  862.912626] [    T169] INFO: task kworker/10:1:182 blocked for more than 123 seconds.
[  862.913741] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.914683] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.915828] [    T169] task:kworker/10:1    state:D stack:0     pid:182   tgid:182   ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.917294] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.918366] [    T169] Call Trace:
[  862.918970] [    T169]  <TASK>
[  862.919592] [    T169]  __schedule+0xc86/0x1b10
[  862.920361] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  862.921220] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.922009] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.922876] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  862.923772] [    T169]  ? lock_acquire+0x11a/0x140
[  862.924586] [    T169]  schedule+0xe5/0x2e0
[  862.925449] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.926194] [    T169]  schedule_preempt_disabled+0x16/0x30
[  862.927011] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  862.927988] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  862.928985] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.930050] [    T169]  ? lock_acquire+0x11a/0x140
[  862.930852] [    T169]  down_write_nested+0x1c0/0x1f0
[  862.931837] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  862.932707] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  862.934501] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  862.936477] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.938135] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.939145] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.940733] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.942374] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.943029] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.943836] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.944511] [    T169]  process_one_work+0x947/0x1760
[  862.945264] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.945902] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.946883] [    T169]  worker_thread+0x601/0xff0
[  862.947501] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.948411] [    T169]  ? __kthread_parkme+0xbd/0x210
[  862.949411] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.950258] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.950981] [    T169]  kthread+0x361/0x460
[  862.951742] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.952475] [    T169]  ret_from_fork+0x560/0x860
[  862.953085] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.953768] [    T169]  ? __switch_to+0x473/0xd50
[  862.954462] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.955329] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.956135] [    T169]  </TASK>
[  862.956743] [    T169] INFO: task kworker/10:1:182 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  862.958496] [    T169] INFO: task kworker/4:1:192 blocked for more than 123 seconds.
[  862.959780] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  862.960649] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  862.961646] [    T169] task:kworker/4:1     state:D stack:0     pid:192   tgid:192   ppid:2      task_flags:0x4248060 flags:0x00080000
[  862.962802] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  862.963656] [    T169] Call Trace:
[  862.964239] [    T169]  <TASK>
[  862.964876] [    T169]  __schedule+0xc86/0x1b10
[  862.965650] [    T169]  ? __pfx___schedule+0x10/0x10
[  862.966357] [    T169]  ? rwsem_optimistic_spin+0x1f0/0x480
[  862.967116] [    T169]  ? wq_worker_sleeping+0x65/0x280
[  862.967818] [    T169]  ? lock_acquire+0x11a/0x140
[  862.968455] [    T169]  schedule+0xe5/0x2e0
[  862.969236] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  862.969956] [    T169]  schedule_preempt_disabled+0x16/0x30
[  862.970745] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  862.971466] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  862.972318] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  862.973018] [    T169]  ? lock_acquire+0x11a/0x140
[  862.973929] [    T169]  down_write_nested+0x1c0/0x1f0
[  862.974668] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  862.975517] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  862.977132] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  862.978753] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  862.980434] [    T169]  ? lock_acquire+0x11a/0x140
[  862.981088] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  862.981891] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  862.983313] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  862.984866] [    T169]  iomap_dio_complete+0x140/0xb20
[  862.985470] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  862.986145] [    T169]  iomap_dio_complete_work+0x58/0x90
[  862.987165] [    T169]  process_one_work+0x947/0x1760
[  862.987782] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  862.988614] [    T169]  ? __try_to_del_timer_sync+0xd3/0x130
[  862.989478] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  862.990329] [    T169]  worker_thread+0x601/0xff0
[  862.991073] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  862.991879] [    T169]  kthread+0x361/0x460
[  862.992366] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.992943] [    T169]  ret_from_fork+0x560/0x860
[  862.993616] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  862.994429] [    T169]  ? __switch_to+0x473/0xd50
[  862.995061] [    T169]  ? __pfx_kthread+0x10/0x10
[  862.995764] [    T169]  ret_from_fork_asm+0x1a/0x30
[  862.996514] [    T169]  </TASK>
[  862.997177] [    T169] INFO: task kworker/4:1:192 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  862.998650] [    T169] INFO: task kworker/13:1:203 blocked for more than 123 seconds.
[  862.999569] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  863.000412] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  863.001516] [    T169] task:kworker/13:1    state:D stack:0     pid:203   tgid:203   ppid:2      task_flags:0x4248060 flags:0x00080000
[  863.002727] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  863.003433] [    T169] Call Trace:
[  863.003999] [    T169]  <TASK>
[  863.004720] [    T169]  __schedule+0xc86/0x1b10
[  863.005341] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  863.005926] [    T169]  ? __pfx___schedule+0x10/0x10
[  863.006767] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  863.007384] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  863.008172] [    T169]  ? rcu_is_watching+0x69/0x90
[  863.008798] [    T169]  ? lock_acquire+0x11a/0x140
[  863.009533] [    T169]  schedule+0xe5/0x2e0
[  863.010175] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  863.010753] [    T169]  schedule_preempt_disabled+0x16/0x30
[  863.011389] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  863.012000] [    T169]  ? __kasan_slab_alloc+0x7f/0x90
[  863.012699] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  863.013432] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  863.014201] [    T169]  ? lock_acquire+0x11a/0x140
[  863.014754] [    T169]  down_write_nested+0x1c0/0x1f0
[  863.015577] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  863.016221] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  863.017869] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  863.019128] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  863.020502] [    T169]  ? update_load_avg+0xb65/0x2c70
[  863.021124] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  863.021694] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  863.022983] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  863.024392] [    T169]  iomap_dio_complete+0x140/0xb20
[  863.024936] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  863.025750] [    T169]  iomap_dio_complete_work+0x58/0x90
[  863.026539] [    T169]  process_one_work+0x947/0x1760
[  863.027125] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  863.027748] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  863.028469] [    T169]  worker_thread+0x601/0xff0
[  863.029048] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  863.029602] [    T169]  ? __kthread_parkme+0xbd/0x210
[  863.030330] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  863.031083] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  863.031679] [    T169]  kthread+0x361/0x460
[  863.032432] [    T169]  ? __pfx_kthread+0x10/0x10
[  863.033138] [    T169]  ret_from_fork+0x560/0x860
[  863.033773] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  863.034384] [    T169]  ? __switch_to+0x473/0xd50
[  863.034950] [    T169]  ? __pfx_kthread+0x10/0x10
[  863.046680] [    T169]  ret_from_fork_asm+0x1a/0x30
[  863.047257] [    T169]  </TASK>
[  863.047797] [    T169] INFO: task kworker/13:1:203 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  863.048786] [    T169] INFO: task kworker/7:1:267 blocked for more than 123 seconds.
[  863.049651] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  863.050339] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  863.051394] [    T169] task:kworker/7:1     state:D stack:0     pid:267   tgid:267   ppid:2      task_flags:0x4248060 flags:0x00080000
[  863.052474] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  863.053233] [    T169] Call Trace:
[  863.053879] [    T169]  <TASK>
[  863.054380] [    T169]  __schedule+0xc86/0x1b10
[  863.055017] [    T169]  ? trace_hardirqs_on+0x18/0x1a0
[  863.055705] [    T169]  ? __pfx___schedule+0x10/0x10
[  863.056191] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  863.056776] [    T169]  ? wq_worker_sleeping+0x213/0x280
[  863.057382] [    T169]  ? rcu_is_watching+0x69/0x90
[  863.057888] [    T169]  ? lock_acquire+0x11a/0x140
[  863.058510] [    T169]  schedule+0xe5/0x2e0
[  863.059152] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  863.059634] [    T169]  schedule_preempt_disabled+0x16/0x30
[  863.060191] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  863.060660] [    T169]  ? __kasan_slab_alloc+0x7f/0x90
[  863.061305] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  863.061975] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  863.062852] [    T169]  ? lock_acquire+0x11a/0x140
[  863.063494] [    T169]  down_write_nested+0x1c0/0x1f0
[  863.063987] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  863.064651] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  863.065744] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  863.066867] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  863.068171] [    T169]  ? lock_acquire+0x11a/0x140
[  863.068608] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  863.069209] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  863.070280] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  863.071520] [    T169]  iomap_dio_complete+0x140/0xb20
[  863.072007] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  863.072614] [    T169]  iomap_dio_complete_work+0x58/0x90
[  863.073410] [    T169]  process_one_work+0x947/0x1760
[  863.073943] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  863.074647] [    T169]  ? __try_to_del_timer_sync+0xd3/0x130
[  863.075263] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  863.075717] [    T169]  worker_thread+0x601/0xff0
[  863.076457] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  863.076864] [    T169]  kthread+0x361/0x460
[  863.077206] [    T169]  ? __pfx_kthread+0x10/0x10
[  863.077635] [    T169]  ret_from_fork+0x560/0x860
[  863.077979] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  863.078467] [    T169]  ? __switch_to+0x473/0xd50
[  863.079013] [    T169]  ? __pfx_kthread+0x10/0x10
[  863.079724] [    T169]  ret_from_fork_asm+0x1a/0x30
[  863.080357] [    T169]  </TASK>
[  863.080782] [    T169] INFO: task kworker/7:1:267 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  863.081659] [    T169] INFO: task kworker/18:1:273 blocked for more than 123 seconds.
[  863.082544] [    T169]       Tainted: G        W           7.3.0-rc2-kts-xfs+ #3
[  863.083261] [    T169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  863.084177] [    T169] task:kworker/18:1    state:D stack:0     pid:273   tgid:273   ppid:2      task_flags:0x4248060 flags:0x00080000
[  863.085081] [    T169] Workqueue: dio/sdh iomap_dio_complete_work
[  863.085709] [    T169] Call Trace:
[  863.086190] [    T169]  <TASK>
[  863.086664] [    T169]  __schedule+0xc86/0x1b10
[  863.087350] [    T169]  ? __pfx___schedule+0x10/0x10
[  863.088018] [    T169]  ? rwsem_optimistic_spin+0x1f0/0x480
[  863.088839] [    T169]  ? wq_worker_sleeping+0x65/0x280
[  863.089320] [    T169]  ? lock_acquire+0x11a/0x140
[  863.089716] [    T169]  schedule+0xe5/0x2e0
[  863.090107] [    T169]  ? _raw_spin_unlock_irq+0x28/0x50
[  863.090475] [    T169]  schedule_preempt_disabled+0x16/0x30
[  863.090998] [    T169]  rwsem_down_write_slowpath+0x4b4/0x1460
[  863.091593] [    T169]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  863.092182] [    T169]  ? percpu_counter_add_batch+0x80/0x220
[  863.092592] [    T169]  ? lock_acquire+0x11a/0x140
[  863.092928] [    T169]  down_write_nested+0x1c0/0x1f0
[  863.093339] [    T169]  ? __pfx_down_write_nested+0x10/0x10
[  863.093878] [    T169]  ? xfs_ilock+0x1e7/0x310 [xfs]
[  863.094891] [    T169]  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
[  863.095987] [    T169]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10 [xfs]
[  863.097206] [    T169]  ? lock_acquire+0x11a/0x140
[  863.097778] [    T169]  ? update_cfs_rq_load_avg+0x4b/0x660
[  863.098326] [    T169]  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
[  863.099445] [    T169]  ? __pfx_xfs_dio_write_end_io+0x10/0x10 [xfs]
[  863.100619] [    T169]  iomap_dio_complete+0x140/0xb20
[  863.101449] [    T169]  ? __pfx_aio_complete_rw+0x10/0x10
[  863.102004] [    T169]  iomap_dio_complete_work+0x58/0x90
[  863.102595] [    T169]  process_one_work+0x947/0x1760
[  863.103268] [    T169]  ? __pfx_process_one_work+0x10/0x10
[  863.103864] [    T169]  ? __pfx_do_raw_spin_lock+0x10/0x10
[  863.104356] [    T169]  worker_thread+0x601/0xff0
[  863.104893] [    T169]  ? __pfx_worker_thread+0x10/0x10
[  863.105480] [    T169]  kthread+0x361/0x460
[  863.105995] [    T169]  ? __pfx_kthread+0x10/0x10
[  863.106607] [    T169]  ret_from_fork+0x560/0x860
[  863.107270] [    T169]  ? __pfx_ret_from_fork+0x10/0x10
[  863.107794] [    T169]  ? __switch_to+0x473/0xd50
[  863.108317] [    T169]  ? __pfx_kthread+0x10/0x10
[  863.108847] [    T169]  ret_from_fork_asm+0x1a/0x30
[  863.109354] [    T169]  </TASK>
[  863.109815] [    T169] INFO: task kworker/18:1:273 <writer> blocked on an rw-semaphore likely owned by task kworker/14:8:2880 <writer>
[  863.110690] [    T169] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
[  863.111555] [    T169] INFO: lockdep is turned off.
#6 lore raw
In reply to John Garry
Quoted 9 lines · John Garry
Quoted 6 lines · Darrick J. Wong
Quoted 5 lines

thanks

About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for CoW-based atomics could help avoid this issue as we seem to be bogged down in lock contention.

Which lock is being contended, anyway?  It looks like the ILOCK?

Yeah, I think so Re. ilock. I had some perf data illustrating this from last year, which I can't seem to find, so I will re-generate it.

Here is perf call graph snippet when running fio with 72x threads issuing 4KB atomic writes on 1MB file:

--50.46%--xfs_file_dio_write_atomic

|
|--37.19%--iomap_dio_rw
|          |
|      --37.14%--__iomap_dio_rw
|           |
|       --36.50%--iomap_iter
|            |
|             --36.49%--xfs_atomic_write_cow_iomap_next
|                  |
|                  |--27.34%--xfs_ilock
|                  |          |
|                  |      --27.34%--down_write
|                  |           |
|                  |       --27.28%--rwsem_down_write_slowpath
|                  |            |
|                  |            |--24.65%--osq_lock
|                  |            |
|                  |             --2.35%--rwsem_spin_on_owner
|                  |
|                  |--7.62%--xfs_trans_alloc_inode
|                  |          |
|                  |      --7.59%--xfs_ilock

Notice how much time we spend getting that ilock.

We could try something like this:

----8<------

[PATCH] xfs: serialize CoW-based atomic writes

We have had reports of system hangs when testing Cow-based atomic writes for large block sizes.

Analysis has shown large contention on the the per-inode ilock, superficially in creating the mapping in xfs_atomic_write_cow_iomap_begin().

Serialize atomic writes to reduce this contention. In some cases this may reduce performance, but high performance has not been a requirement so far for CoW-based atomic writes.

fs/xfs/xfs_file.c +7 -2
index d8202da15aca..ff48757cf0f0 100644
@@ -819,10 +819,12 @@ xfs_file_dio_write_atomic(
 	 * HW offload should be faster, so try that first if it is already
 	 * known that the write length is not too large.
 	 */
-	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max)
+	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max) {
 		dops = &xfs_atomic_write_cow_iomap_ops;
-	else
+		dio_flags |= IOMAP_DIO_FORCE_WAIT;
+	} else {
 		dops = &xfs_direct_write_iomap_ops;
+	}
 
 retry:
 	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
@@ -840,6 +842,8 @@ xfs_file_dio_write_atomic(
 	}
 
 	trace_xfs_file_direct_write(iocb, from);
+	if (dio_flags & IOMAP_DIO_FORCE_WAIT)
+		inode_dio_wait(VFS_I(ip));
 	if (mapping_stable_writes(iocb->ki_filp->f_mapping))
 		dio_flags |= IOMAP_DIO_BOUNCE;
 	ret = iomap_dio_rw(iocb, from, dops, &xfs_dio_write_ops, dio_flags,
@@ -853,6 +857,7 @@ xfs_file_dio_write_atomic(
 	 */
 	if (ret == -ENOPROTOOPT && dops == &xfs_direct_write_iomap_ops) {
 		xfs_iunlock(ip, iolock);
+		dio_flags |= IOMAP_DIO_FORCE_WAIT;
 		dops = &xfs_atomic_write_cow_iomap_ops;
 		goto retry;
 	}

---->8------

This is far from ideal, but I think that it should stops the hangs - having the kernel hang is intolerable.

I also notice that in xfs_atomic_write_cow_iomap_next() we drop the ilock for calling xfs_trans_alloc_inode() (which immediately grabs the same lock). It would seem that could be improved.

#12 lore raw
In reply to Shin'ichiro Kawasaki
Quoted 31 lines · Shin'ichiro Kawasaki

I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB TCMU fileio devices. Actually I once reported this hang symptom in last October, and Darrick and John kindly took some actions [1]. Since then, the hang disappeared and I have not observed the hang almost one year. However, my test system started reporting the hang again last week.

[1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t

FYI, here I attache the kernel message [2]. I observed the hang last week with the kernel on the xfs/for-next branch at the commit 0ca15a1a1151 ("xfs: remove an extra cast in xfs_file_compat_ioctl"). Today, I tried again with the kernel at current xfs/for-next branch tip 4266ffdfd7cb ("xfs: remove an extra cast in xfs_file_compat_ioctl"), and observed the hang again. The hang can be recreated in stable manner by repeating the test case 20 times or so.

I also tried other test cases in atmoicwrites group, and observed no failure.

generic/765 [not run] write atomic not supported by this block device

 generic/767         11s
 generic/768         13s
 generic/769         13s
 generic/770         32s
 generic/773        [not run] write atomic not supported by this block device
 generic/774        (skipped)
 generic/775         293s
 generic/776        [notrun] write atomic not supported by this block device
 generic/778         48s
 xfs/838            [not run] External volumes not in use, skipped this test
 xfs/839            [not run] XFS error injection requires CONFIG_XFS_DEBUG
 xfs/840            [not run] write atomic not supported by this block device
Quoted 2 lines

Action for fix will be appreciated. If I can do anything on my test node, please let me know.

.....

A bunch of threads waiting on the ILOCK here:

 down_write_nested+0x1c0/0x1f0
 xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
 xfs_dio_write_end_io+0x4b7/0x650 [xfs]
 iomap_dio_complete+0x140/0xb20
 iomap_dio_complete_work+0x58/0x90
 process_one_work+0x947/0x1760

And the holder:

 schedule+0xe5/0x2e0
 xlog_grant_head_wait+0x175/0xac0 [xfs]
 xlog_grant_head_check+0x312/0x3f0 [xfs]
 xfs_log_regrant+0x380/0x7d0 [xfs]
 xfs_trans_roll+0x2d9/0x420 [xfs]
 xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
 xfs_defer_finish_noroll+0x460/0xe70 [xfs]
 xfs_trans_commit+0xfc/0x180 [xfs]
 xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
 xfs_dio_write_end_io+0x4b7/0x650 [xfs]
 iomap_dio_complete+0x140/0xb20
 iomap_dio_complete_work+0x58/0x90
 process_one_work+0x947/0x1760

is waiting on log space whilst holding the ILOCK.

This looks to me like the test runs out of log space because of all the IO completions holding transaction reservations waiting on the ILOCK, whilst the ILOCK holder can't get enough log space to regrant on transaction roll to continue the transaction.

Oh:

STATIC void
xfs_calc_default_atomic_ioend_reservation(
struct xfs_mount *mp,
struct xfs_trans_resv *resp)

{
        /* Pick a default that will scale reasonably for the log size. */
        resp->tr_atomic_ioend = resp->tr_itruncate;
}

Which means each reservation is probably holding hundreds of KB to MB of journal reservation. If the journal is 64MB in size, then a few dozen o these might be all that is needed to consume all the reservation space.

Hmmm - there are at least 30 tasks stuck in xfs_reflink_end_atomic_cow(), another 30 stuck in xfs_vn_update_time() holding reservations, and significant number xfs_atomic_write_cow_iomap_begin()->xfs_trans_alloc_inode() holding tr_write reservations.

IOWs, smells of the journal being run out of reservation space, and no new space being able to be freed because all the dirty inodes in the journal that need to be written back to free up space are locked waiting for journal space to come free....

How big is the journal in the filesystem being tested? If you increase the size of the journal, does it go away? Can you get a dump of the transaction reservation sizes for the filesystem in question (the xfs_db logres command can do this, IIRC) and then run the math on the reservations held from the dump of all the blocked tasks holding locks to see if this matches the size of the journal in the fs?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#7 lore raw
In reply to John Garry
Quoted 31 lines · John Garry
Quoted 11 lines · John Garry
Quoted 7 lines · Darrick J. Wong
Quoted 6 lines

thanks

About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for CoW-based atomics could help avoid this issue as we seem to be bogged down
in lock contention.

Which lock is being contended, anyway?  It looks like the ILOCK?

Yeah, I think so Re. ilock. I had some perf data illustrating this from last year, which I can't seem to find, so I will re-generate it.

Here is perf call graph snippet when running fio with 72x threads issuing 4KB atomic writes on 1MB file:

--50.46%--xfs_file_dio_write_atomic

|
|--37.19%--iomap_dio_rw
|          |
|      --37.14%--__iomap_dio_rw
|           |
|       --36.50%--iomap_iter
|            |
|             --36.49%--xfs_atomic_write_cow_iomap_next
|                  |
|                  |--27.34%--xfs_ilock
|                  |          |
|                  |      --27.34%--down_write
|                  |           |
|                  |       --27.28%--rwsem_down_write_slowpath
Quoted 8 lines
|                  |            |
|                  |            |--24.65%--osq_lock
|                  |            |
|                  |             --2.35%--rwsem_spin_on_owner
|                  |
|                  |--7.62%--xfs_trans_alloc_inode
|                  |          |
|                  |      --7.59%--xfs_ilock

Where's the other half of the lock contention?

If this is all there is, it's because of concurrent submission of serialising IO, and xfs_atomic_write_cow_iomap_begin() is the first place in the IO submission path that we hit a serialising lock. That indicates a problem with high level IO serialisation, not a problem with the ILOCK.

Quoted 41 lines

Notice how much time we spend getting that ilock.

We could try something like this:

----8<------

[PATCH] xfs: serialize CoW-based atomic writes

We have had reports of system hangs when testing Cow-based atomic writes for large block sizes.

Analysis has shown large contention on the the per-inode ilock, superficially in creating the mapping in xfs_atomic_write_cow_iomap_begin().

Serialize atomic writes to reduce this contention. In some cases this may reduce performance, but high performance has not been a requirement so far for CoW-based atomic writes.

fs/xfs/xfs_file.c +4 -2
index d8202da15aca..ff48757cf0f0 100644
@@ -819,10 +819,12 @@ xfs_file_dio_write_atomic(
 	 * HW offload should be faster, so try that first if it is already
 	 * known that the write length is not too large.
 	 */
-	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max)
+	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max) {
 		dops = &xfs_atomic_write_cow_iomap_ops;
-	else
+		dio_flags |= IOMAP_DIO_FORCE_WAIT;
+	} else {
 		dops = &xfs_direct_write_iomap_ops;
+	}
 
 retry:
 	ret = xfs_ilock_iocb_for_write(iocb, &iolock);

fs/xfs/xfs_file.c xfs_file_dio_write_atomic( · line 846

Quoted 5 lines
fs/xfs/xfs_file.c +2 -0
@@ -840,6 +842,8 @@ xfs_file_dio_write_atomic(
 	}
 
 	trace_xfs_file_direct_write(iocb, from);
+	if (dio_flags & IOMAP_DIO_FORCE_WAIT)
+		inode_dio_wait(VFS_I(ip));

Based on my comment above, this seems like the wrong approach to me.

We serialise concurrent DIO write submission when necessary by taking the IOLOCK_EXCL instead of IOLOCK_SHARED (e.g. EOF extension). inode_dio_wait() waits for completion of all DIO (read and write), but we don't need to wait for completions if all we need to do is serialise IO submission.

Hence if we know we are going to do a software COW for the write, we should take IOLOCK_EXCL to serialise submission up high. Waiting until we are deep into the IO path and serialising on the internal metadata ILOCK is much, much harder to do efficiently and correctly.

i.e. we need to take the ILOCK vs transaction reservation vs unbound user-controlled concurrency deadlock problem out of
the equation. Using the IOLOCK achieves this because IO submission blocks on the IOLOCK rather than on the ILOCK whilst holding a transaction reservation.

Quoted 3 lines

I also notice that in xfs_atomic_write_cow_iomap_next() we drop the ilock for calling xfs_trans_alloc_inode() (which immediately grabs the same lock). It would seem that could be improved.

Doubt it - holding the ILOCK across transaction reservation guarantees journal space deadlocks will occur. That's why the suboptimal pattern of 'lock - check - unlock - trans_alloc - lock - check' exists throughout XFS inode modification paths.

FWIW, that's one of the anit-patterns that the patchset I posted recently begins to address:

https://lore.kernel.org/linux-xfs/[email protected]/

It creates consistently correct patterns in the IO path by changes all the individual alloc-lock-check-mod-commit-unlock loops to an "atomic" alloc-lock, check-mod-roll loop, commit-unlock pattern.

IOWs, multi-part extent modifications are all done under a single lock/unlock pair, and so appear atomic to all other extent map and modification operations. This makes many of the nasty IO corner cases that lead to sub-optimal behaviour (like data corruption or lock contention in COW workloads due to continual bouncing of the ILOCK between submission and completion contexts) largely go away...

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#10 lore raw
In reply to John Garry

On Sep 16, 2026 / 11:23, John Garry wrote:
[...]

Quoted 18 lines

We could try something like this:

----8<------

[PATCH] xfs: serialize CoW-based atomic writes

We have had reports of system hangs when testing Cow-based atomic writes for large block sizes.

Analysis has shown large contention on the the per-inode ilock, superficially in creating the mapping in xfs_atomic_write_cow_iomap_begin().

Serialize atomic writes to reduce this contention. In some cases this may reduce performance, but high performance has not been a requirement so far for CoW-based atomic writes.

[...]

Quoted 4 lines

---->8------

This is far from ideal, but I think that it should stops the hangs - having the kernel hang is intolerable.

Thanks. I applied the patch on top of xfs/for-next branch commit 0ca15a1a1151, then repeated the test case 100 times on top of the kernel. I did not observe the hang. This patch looks avoiding the hang.

#13 lore raw
In reply to Dave Chinner

On Sep 17, 2026 / 07:45, Dave Chinner wrote:
[...]

Quoted 6 lines

IOWs, smells of the journal being run out of reservation space, and no new space being able to be freed because all the dirty inodes in the journal that need to be written back to free up space are locked waiting for journal space to come free....

Quoted 7 lines

How big is the journal in the filesystem being tested? If you increase the size of the journal, does it go away? Can you get a dump of the transaction reservation sizes for the filesystem in question (the xfs_db logres command can do this, IIRC) and then run the math on the reservations held from the dump of all the blocked tasks holding locks to see if this matches the size of the journal in the fs?

Dave, thanks for the comments. I recreated the hang, and gathered relevant numbers [3]. Here, /dev/sdh is the SCRATCH_DEV. My interpretations are as follows (If I misunderstand anything correction will be appreciated):

- The log section size is 4096 * 16384 = 64MiB.
- The type 26 logres has 327552 size and logcount 5, then each blocking task consumes about 1.6MiB
- My test node has 24 nproc, and the test case creases 2 * nproc * LOAD_FACTOR threads. LOAD_FACTOR is 1, then it creates 48 fio threads.

Assuming all fio threads are blocked, and each thread consumes 16MiB in the log section, 1.6MiB * 48 = 76.8Mib will be consumed. This is beyond the log section size 64MiB. The sysfs attributes reserve_grant_head_bytes has the number close to 64MiB, and I guess it implies the log section is almost full. Based on these, Dave's guess look correct.

As suggested, now I'm trying to increase the journal/log section size. I added the mkfs options below:

MKFS_OPTIONS="-l size=128m"

With this, I repeated the test case 30 times, and did not observe the hang. It looks like the log section size increase avoids the hang. I will keep it on running to see if the hang is observed after 100 times run.

[3]

$ sudo xfs_info /dev/sdh
meta-data=/dev/sdh               isize=512    agcount=4, agsize=524288 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=1
         =                       exchange=1   metadir=0
data     =                       bsize=4096   blocks=2097152, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1, parent=1
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
         =                       rgcount=0    rgsize=0 extents
         =                       zoned=0      start=0 reserved=0
$ sudo xfs_db -c logres -r /dev/sdh
type 0 logres 193528 logcount 5 flags 0x4
type 1 logres 327552 logcount 5 flags 0x4
type 2 logres 428160 logcount 5 flags 0x4
type 3 logres 247204 logcount 5 flags 0x4
type 4 logres 230820 logcount 5 flags 0x4
type 5 logres 312612 logcount 6 flags 0x4
type 6 logres 311460 logcount 5 flags 0x4
type 7 logres 285696 logcount 2 flags 0x4
type 8 logres 311460 logcount 6 flags 0x4
type 9 logres 303992 logcount 2 flags 0x4
type 10 logres 2168 logcount 0 flags 0x0
type 11 logres 82176 logcount 2 flags 0x4
type 12 logres 116856 logcount 2 flags 0x4
type 13 logres 760 logcount 0 flags 0x0
type 14 logres 326784 logcount 1 flags 0x4
type 15 logres 23288 logcount 3 flags 0x4
type 16 logres 21760 logcount 0 flags 0x0
type 17 logres 164480 logcount 3 flags 0x4
type 18 logres 640 logcount 0 flags 0x0
type 19 logres 111864 logcount 2 flags 0x4
type 20 logres 4224 logcount 0 flags 0x0
type 21 logres 6512 logcount 0 flags 0x0
type 22 logres 232 logcount 1 flags 0x0
type 23 logres 197751 logcount 5 flags 0x4
type 24 logres 640 logcount 1 flags 0x0
type 25 logres 760 logcount 0 flags 0x0
type 26 logres 327552 logcount 5 flags 0x4
minlogsize logres 428160 logcount 5
$ grep . /sys/fs/xfs/sdh/log/*
/sys/fs/xfs/sdh/log/log_head_lsn:1:103190
/sys/fs/xfs/sdh/log/log_tail_lsn:1:101012
/sys/fs/xfs/sdh/log/reserve_grant_head_bytes:66053588
/sys/fs/xfs/sdh/log/write_grant_head_bytes:65718696
#8 lore raw
In reply to Dave Chinner
Quoted 41 lines · Dave Chinner
Quoted 31 lines · John Garry
Quoted 11 lines · John Garry
Quoted 7 lines · Darrick J. Wong
Quoted 6 lines

thanks

About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for CoW-based atomics could help avoid this issue as we seem to be bogged down
in lock contention.

Which lock is being contended, anyway?  It looks like the ILOCK?

Yeah, I think so Re. ilock. I had some perf data illustrating this from last year, which I can't seem to find, so I will re-generate it.

Here is perf call graph snippet when running fio with 72x threads issuing 4KB atomic writes on 1MB file:

--50.46%--xfs_file_dio_write_atomic

|
|--37.19%--iomap_dio_rw
|          |
|      --37.14%--__iomap_dio_rw
|           |
|       --36.50%--iomap_iter
|            |
|             --36.49%--xfs_atomic_write_cow_iomap_next
|                  |
|                  |--27.34%--xfs_ilock
|                  |          |
|                  |      --27.34%--down_write
|                  |           |
|                  |       --27.28%--rwsem_down_write_slowpath
Quoted 8 lines
|                  |            |
|                  |            |--24.65%--osq_lock
|                  |            |
|                  |             --2.35%--rwsem_spin_on_owner
|                  |
|                  |--7.62%--xfs_trans_alloc_inode
|                  |          |
|                  |      --7.59%--xfs_ilock
Quoted 1 line

Where's the other half of the lock contention?

I was expecting xfs_reflink_end_atomic_cow() -> xfs_ilock() to be a point of contention as well, but it does not show up. I suppose that it because all those threads at completion stage are sleeping.

However, there is this which I omitted:

--13.21%--xfs_file_write_checks
|

      --13.18%--kiocb_modified
         |
            --13.17%--file_update_time_flags
               |
                  --13.16%--xfs_vn_update_time
                     |
                        --12.99%--xfs_ilock
                           |
                              --12.98%--down_write
Quoted 92 lines

If this is all there is, it's because of concurrent submission of serialising IO, and xfs_atomic_write_cow_iomap_begin() is the first place in the IO submission path that we hit a serialising lock. That indicates a problem with high level IO serialisation, not a problem with the ILOCK.

Quoted 41 lines

Notice how much time we spend getting that ilock.

We could try something like this:

----8<------

[PATCH] xfs: serialize CoW-based atomic writes

We have had reports of system hangs when testing Cow-based atomic writes for large block sizes.

Analysis has shown large contention on the the per-inode ilock, superficially in creating the mapping in xfs_atomic_write_cow_iomap_begin().

Serialize atomic writes to reduce this contention. In some cases this may reduce performance, but high performance has not been a requirement so far for CoW-based atomic writes.

fs/xfs/xfs_file.c +4 -2
index d8202da15aca..ff48757cf0f0 100644
@@ -819,10 +819,12 @@ xfs_file_dio_write_atomic(
  	 * HW offload should be faster, so try that first if it is already
  	 * known that the write length is not too large.
  	 */
-	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max)
+	if (ocount > xfs_inode_buftarg(ip)->bt_awu_max) {
  		dops = &xfs_atomic_write_cow_iomap_ops;
-	else
+		dio_flags |= IOMAP_DIO_FORCE_WAIT;
+	} else {
  		dops = &xfs_direct_write_iomap_ops;
+	}
 
  retry:
  	ret = xfs_ilock_iocb_for_write(iocb, &iolock);

fs/xfs/xfs_file.c xfs_file_dio_write_atomic( · line 846

Quoted 5 lines
fs/xfs/xfs_file.c +2 -0
@@ -840,6 +842,8 @@ xfs_file_dio_write_atomic(
  	}
 
  	trace_xfs_file_direct_write(iocb, from);
+	if (dio_flags & IOMAP_DIO_FORCE_WAIT)
+		inode_dio_wait(VFS_I(ip));

Based on my comment above, this seems like the wrong approach to me.

We serialise concurrent DIO write submission when necessary by taking the IOLOCK_EXCL instead of IOLOCK_SHARED (e.g. EOF extension). inode_dio_wait() waits for completion of all DIO (read and write), but we don't need to wait for completions if all we need to do is serialise IO submission.

Hence if we know we are going to do a software COW for the write, we should take IOLOCK_EXCL to serialise submission up high. Waiting until we are deep into the IO path and serialising on the internal metadata ILOCK is much, much harder to do efficiently and correctly.

i.e. we need to take the ILOCK vs transaction reservation vs unbound user-controlled concurrency deadlock problem out of
the equation. Using the IOLOCK achieves this because IO submission blocks on the IOLOCK rather than on the ILOCK whilst holding a transaction reservation.

Quoted 3 lines

I also notice that in xfs_atomic_write_cow_iomap_next() we drop the ilock for calling xfs_trans_alloc_inode() (which immediately grabs the same lock). It would seem that could be improved.

Doubt it - holding the ILOCK across transaction reservation guarantees journal space deadlocks will occur. That's why the suboptimal pattern of 'lock - check - unlock - trans_alloc - lock - check' exists throughout XFS inode modification paths.

FWIW, that's one of the anit-patterns that the patchset I posted recently begins to address:

https://lore.kernel.org/linux-xfs/[email protected]/

It creates consistently correct patterns in the IO path by changes all the individual alloc-lock-check-mod-commit-unlock loops to an "atomic" alloc-lock, check-mod-roll loop, commit-unlock pattern.

Quoted 7 lines

IOWs, multi-part extent modifications are all done under a single lock/unlock pair, and so appear atomic to all other extent map and modification operations. This makes many of the nasty IO corner cases that lead to sub-optimal behaviour (like data corruption or lock contention in COW workloads due to continual bouncing of the ILOCK between submission and completion contexts) largely go away...

Ack, and unfortunately that series did not get much expert review.

JFYI, I have another log for a similar hang which we see one thread deadlock for log space, below.

fio-3.38
Starting 14 processes

[  247.169311] INFO: task kworker/34:1:568 blocked for more than 122 seconds.
[  247.169355]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.169370] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.169372] task:kworker/34:1    state:D stack:23048 pid:568 tgid:568   ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.169388] Workqueue: dio/sda iomap_dio_complete_work
[  247.169424] Call Trace:
[  247.169427]  <TASK>
[  247.169431]  __schedule+0x1179/0x44c0
[  247.169446]  ? kick_pool_pick+0x24c/0x640
[  247.169458]  ? __pfx___schedule+0x10/0x10
[  247.169464]  ? lock_release+0xc9/0x290
[  247.169471]  schedule+0xd2/0x260
[  247.169476]  schedule_preempt_disabled+0x11/0x20
[  247.169479]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.169486]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.169494]  ? xfs_dec_freecounter+0xec/0x580
[  247.169501]  down_write_nested+0x1b5/0x1e0
[  247.169508]  ? __pfx_down_write_nested+0x10/0x10
[  247.169512]  ? xfs_ilock+0x2a0/0x3d0
[  247.169518]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.169523]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.169527]  ? __lock_acquire+0x607/0x1f70
[  247.169532]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.169539]  iomap_dio_complete+0x136/0xaa0
[  247.169543]  ? process_scheduled_works+0xaee/0x1b30
[  247.169546]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.169552]  iomap_dio_complete_work+0x51/0x80
[  247.169555]  process_scheduled_works+0xc08/0x1b30
[  247.169570]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.169573]  ? lock_acquire+0x185/0x2e0
[  247.169577]  ? lock_is_held_type+0x9e/0x120
[  247.169582]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.169586]  worker_thread+0x4ad/0xb40
[  247.169591]  ? __kthread_parkme+0x99/0x180
[  247.169596]  ? __pfx_worker_thread+0x10/0x10
[  247.169600]  kthread+0x351/0x450
[  247.169603]  ? __pfx_kthread+0x10/0x10
[  247.169606]  ret_from_fork+0x563/0x800
[  247.169618]  ? __pfx_ret_from_fork+0x10/0x10
[  247.169621]  ? __pfx_kthread+0x10/0x10
[  247.169624]  ? __switch_to+0x58e/0xea0
[  247.169630]  ? __pfx_kthread+0x10/0x10
[  247.169632]  ret_from_fork_asm+0x1a/0x30
[  247.169651]  </TASK>
[  247.170111] INFO: task kworker/34:1:568 <writer> blocked on an rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.170169] INFO: task kworker/48:12:800 blocked for more than 122 seconds.
[  247.170172]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.170173] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.170174] task:kworker/48:12   state:D stack:26088 pid:800 tgid:800   ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.170188] Workqueue: dio/sda iomap_dio_complete_work
[  247.170196] Call Trace:
[  247.170198]  <TASK>
[  247.170200]  __schedule+0x1179/0x44c0
[  247.170208]  ? kick_pool_pick+0x24c/0x640
[  247.170216]  ? __pfx___schedule+0x10/0x10
[  247.170220]  ? lock_release+0xc9/0x290
[  247.170227]  schedule+0xd2/0x260
[  247.170231]  schedule_preempt_disabled+0x11/0x20
[  247.170234]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.170240]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.170249]  ? xfs_dec_freecounter+0xec/0x580
[  247.170254]  down_write_nested+0x1b5/0x1e0
[  247.170260]  ? __pfx_down_write_nested+0x10/0x10
[  247.170264]  ? xfs_ilock+0x2a0/0x3d0
[  247.170269]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.170274]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.170278]  ? __lock_acquire+0x607/0x1f70
[  247.170283]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.170289]  iomap_dio_complete+0x136/0xaa0
[  247.170293]  ? process_scheduled_works+0xaee/0x1b30
[  247.170296]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.170302]  iomap_dio_complete_work+0x51/0x80
[  247.170305]  process_scheduled_works+0xc08/0x1b30
[  247.170319]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.170323]  ? lock_acquire+0x185/0x2e0
[  247.170327]  ? lock_is_held_type+0x9e/0x120
[  247.170331]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.170335]  worker_thread+0x4ad/0xb40
[  247.170342]  ? __pfx_worker_thread+0x10/0x10
[  247.170345]  kthread+0x351/0x450
[  247.170348]  ? __pfx_kthread+0x10/0x10
[  247.170351]  ret_from_fork+0x563/0x800
[  247.170358]  ? __pfx_ret_from_fork+0x10/0x10
[  247.170371]  ? __pfx_kthread+0x10/0x10
[  247.170374]  ? __switch_to+0x58e/0xea0
[  247.170378]  ? __pfx_kthread+0x10/0x10
[  247.170381]  ret_from_fork_asm+0x1a/0x30
[  247.170391]  </TASK>
[  247.170688] INFO: task kworker/48:12:800 <writer> blocked on an rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.170694] INFO: task kworker/48:13:807 blocked for more than 122 seconds.
[  247.170696]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.170698] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.170699] task:kworker/48:13   state:D stack:26088 pid:807 tgid:807   ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.170708] Workqueue: dio/sda iomap_dio_complete_work
[  247.170713] Call Trace:
[  247.170715]  <TASK>
[  247.170718]  __schedule+0x1179/0x44c0
[  247.170722]  ? kick_pool_pick+0x24c/0x640
[  247.170729]  ? __pfx___schedule+0x10/0x10
[  247.170733]  ? lock_release+0xc9/0x290
[  247.170739]  schedule+0xd2/0x260
[  247.170742]  schedule_preempt_disabled+0x11/0x20
[  247.170746]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.170752]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.170759]  ? xfs_dec_freecounter+0xec/0x580
[  247.170764]  down_write_nested+0x1b5/0x1e0
[  247.170769]  ? __pfx_down_write_nested+0x10/0x10
[  247.170774]  ? xfs_ilock+0x2a0/0x3d0
[  247.170778]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.170781]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.170786]  ? __lock_acquire+0x607/0x1f70
[  247.170791]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.170796]  iomap_dio_complete+0x136/0xaa0
[  247.170799]  ? process_scheduled_works+0xaee/0x1b30
[  247.170802]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.170807]  iomap_dio_complete_work+0x51/0x80
[  247.170810]  process_scheduled_works+0xc08/0x1b30
[  247.170824]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.170827]  ? lock_acquire+0x185/0x2e0
[  247.170831]  ? lock_is_held_type+0x9e/0x120
[  247.170835]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.170839]  worker_thread+0x4ad/0xb40
[  247.170845]  ? __pfx_worker_thread+0x10/0x10
[  247.170848]  kthread+0x351/0x450
[  247.170850]  ? __pfx_kthread+0x10/0x10
[  247.170853]  ret_from_fork+0x563/0x800
[  247.170858]  ? __pfx_ret_from_fork+0x10/0x10
[  247.170861]  ? __pfx_kthread+0x10/0x10
[  247.170863]  ? __switch_to+0x58e/0xea0
[  247.170867]  ? __pfx_kthread+0x10/0x10
[  247.170870]  ret_from_fork_asm+0x1a/0x30
[  247.170878]  </TASK>
[  247.171134] INFO: task kworker/48:13:807 <writer> blocked on an rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.171153] INFO: task kworker/52:22:893 blocked for more than 122 seconds.
[  247.171155]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.171156] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.171157] task:kworker/52:22   state:D stack:26280 pid:893 tgid:893   ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.171166] Workqueue: dio/sda iomap_dio_complete_work
[  247.171171] Call Trace:
[  247.171172]  <TASK>
[  247.171174]  __schedule+0x1179/0x44c0
[  247.171179]  ? kick_pool_pick+0x24c/0x640
[  247.171186]  ? __pfx___schedule+0x10/0x10
[  247.171189]  ? lock_release+0xc9/0x290
[  247.171195]  schedule+0xd2/0x260
[  247.171198]  schedule_preempt_disabled+0x11/0x20
[  247.171202]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.171208]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.171216]  ? xfs_dec_freecounter+0xec/0x580
[  247.171221]  down_write_nested+0x1b5/0x1e0
[  247.171225]  ? __pfx_down_write_nested+0x10/0x10
[  247.171230]  ? xfs_ilock+0x2a0/0x3d0
[  247.171234]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.171238]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.171242]  ? __lock_acquire+0x607/0x1f70
[  247.171248]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.171253]  iomap_dio_complete+0x136/0xaa0
[  247.171256]  ? process_scheduled_works+0xaee/0x1b30
[  247.171259]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.171264]  iomap_dio_complete_work+0x51/0x80
[  247.171267]  process_scheduled_works+0xc08/0x1b30
[  247.171281]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.171285]  ? lock_acquire+0x185/0x2e0
[  247.171289]  ? lock_is_held_type+0x9e/0x120
[  247.171293]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.171297]  worker_thread+0x4ad/0xb40
[  247.171303]  ? __pfx_worker_thread+0x10/0x10
[  247.171306]  kthread+0x351/0x450
[  247.171308]  ? __pfx_kthread+0x10/0x10
[  247.171311]  ret_from_fork+0x563/0x800
[  247.171315]  ? __pfx_ret_from_fork+0x10/0x10
[  247.171318]  ? __pfx_kthread+0x10/0x10
[  247.171320]  ? __switch_to+0x58e/0xea0
[  247.171324]  ? __pfx_kthread+0x10/0x10
[  247.171327]  ret_from_fork_asm+0x1a/0x30
[  247.171335]  </TASK>
[  247.171539] INFO: task kworker/52:22:893 <writer> blocked on an rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.171555] INFO: task kworker/5:4:1006 blocked for more than 122 seconds.
[  247.171557]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.171558] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.171559] task:kworker/5:4     state:D stack:26456 pid:1006 tgid:1006  ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.171566] Workqueue: dio/sda iomap_dio_complete_work
[  247.171569] Call Trace:
[  247.171570]  <TASK>
[  247.171572]  __schedule+0x1179/0x44c0
[  247.171576]  ? kick_pool_pick+0x24c/0x640
[  247.171582]  ? __pfx___schedule+0x10/0x10
[  247.171586]  ? lock_release+0xc9/0x290
[  247.171591]  schedule+0xd2/0x260
[  247.171595]  schedule_preempt_disabled+0x11/0x20
[  247.171597]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.171603]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.171611]  ? xfs_dec_freecounter+0xec/0x580
[  247.171616]  down_write_nested+0x1b5/0x1e0
[  247.171619]  ? __pfx_down_write_nested+0x10/0x10
[  247.171623]  ? xfs_ilock+0x2a0/0x3d0
[  247.171627]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.171631]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.171635]  ? __lock_acquire+0x607/0x1f70
[  247.171640]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.171657]  iomap_dio_complete+0x136/0xaa0
[  247.171661]  ? process_scheduled_works+0xaee/0x1b30
[  247.171663]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.171667]  iomap_dio_complete_work+0x51/0x80
[  247.171670]  process_scheduled_works+0xc08/0x1b30
[  247.171684]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.171687]  ? lock_acquire+0x185/0x2e0
[  247.171691]  ? lock_is_held_type+0x9e/0x120
[  247.171695]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.171699]  worker_thread+0x4ad/0xb40
[  247.171705]  ? __pfx_worker_thread+0x10/0x10
[  247.171709]  kthread+0x351/0x450
[  247.171711]  ? __pfx_kthread+0x10/0x10
[  247.171714]  ret_from_fork+0x563/0x800
[  247.171718]  ? __pfx_ret_from_fork+0x10/0x10
[  247.171721]  ? __pfx_kthread+0x10/0x10
[  247.171723]  ? __switch_to+0x58e/0xea0
[  247.171726]  ? __pfx_kthread+0x10/0x10
[  247.171729]  ret_from_fork_asm+0x1a/0x30
[  247.171737]  </TASK>
[  247.171902] INFO: task kworker/5:4:1006 <writer> blocked on an rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.171905] INFO: task kworker/5:6:1009 blocked for more than 122 seconds.
[  247.171906]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.171908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.171908] task:kworker/5:6     state:D stack:25000 pid:1009 tgid:1009  ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.171915] Workqueue: dio/sda iomap_dio_complete_work
[  247.171919] Call Trace:
[  247.171920]  <TASK>
[  247.171922]  __schedule+0x1179/0x44c0
[  247.171929]  ? __pfx___schedule+0x10/0x10
[  247.171933]  ? lock_release+0xc9/0x290
[  247.171935]  ? _raw_spin_unlock_irq+0x23/0x40
[  247.171940]  schedule+0xd2/0x260
[  247.171944]  schedule_preempt_disabled+0x11/0x20
[  247.171947]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.171952]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.171960]  ? xfs_dec_freecounter+0xec/0x580
[  247.171965]  down_write_nested+0x1b5/0x1e0
[  247.171968]  ? __pfx_down_write_nested+0x10/0x10
[  247.171973]  ? xfs_ilock+0x2a0/0x3d0
[  247.171976]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.171980]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.171984]  ? __lock_acquire+0x607/0x1f70
[  247.171989]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.171994]  iomap_dio_complete+0x136/0xaa0
[  247.171997]  ? process_scheduled_works+0xaee/0x1b30
[  247.172000]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.172004]  iomap_dio_complete_work+0x51/0x80
[  247.172007]  process_scheduled_works+0xc08/0x1b30
[  247.172021]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.172024]  ? lock_acquire+0x185/0x2e0
[  247.172028]  ? lock_is_held_type+0x9e/0x120
[  247.172032]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.172036]  worker_thread+0x4ad/0xb40
[  247.172041]  ? __kthread_parkme+0x99/0x180
[  247.172045]  ? __pfx_worker_thread+0x10/0x10
[  247.172062]  kthread+0x351/0x450
[  247.172065]  ? __pfx_kthread+0x10/0x10
[  247.172068]  ret_from_fork+0x563/0x800
[  247.172072]  ? __pfx_ret_from_fork+0x10/0x10
[  247.172075]  ? __pfx_kthread+0x10/0x10
[  247.172077]  ? __switch_to+0x58e/0xea0
[  247.172081]  ? __pfx_kthread+0x10/0x10
[  247.172084]  ret_from_fork_asm+0x1a/0x30
[  247.172092]  </TASK>
[  247.172277] INFO: task kworker/5:6:1009 <writer> blocked on an rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.172282] INFO: task kworker/7:3:1019 blocked for more than 122 seconds.
[  247.172283]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.172285] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.172286] task:kworker/7:3     state:D stack:25912 pid:1019 tgid:1019  ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.172292] Workqueue: dio/sda iomap_dio_complete_work
[  247.172295] Call Trace:
[  247.172296]  <TASK>
[  247.172299]  __schedule+0x1179/0x44c0
[  247.172302]  ? kick_pool_pick+0x24c/0x640
[  247.172308]  ? __pfx___schedule+0x10/0x10
[  247.172312]  ? lock_release+0xc9/0x290
[  247.172317]  schedule+0xd2/0x260
[  247.172321]  xlog_grant_head_wait+0x112/0x9b0
[  247.172327]  xlog_grant_head_check+0x377/0x500
[  247.172331]  ? __pfx_xlog_grant_head_check+0x10/0x10
[  247.172337]  xfs_log_reserve+0x291/0x6f0
[  247.172343]  ? __pfx_xfs_log_reserve+0x10/0x10
[  247.172346]  ? __percpu_counter_compare+0xa8/0xf0
[  247.172352]  ? xfs_dec_freecounter+0xec/0x580
[  247.172357]  xfs_trans_alloc+0x205/0x900
[  247.172363]  xfs_reflink_end_atomic_cow+0x197/0x640
[  247.172367]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.172372]  ? __lock_acquire+0x607/0x1f70
[  247.172378]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.172383]  iomap_dio_complete+0x136/0xaa0
[  247.172386]  ? process_scheduled_works+0xaee/0x1b30
[  247.172388]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.172392]  iomap_dio_complete_work+0x51/0x80
[  247.172396]  process_scheduled_works+0xc08/0x1b30
[  247.172411]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.172415]  ? lock_acquire+0x185/0x2e0
[  247.172419]  ? lock_is_held_type+0x9e/0x120
[  247.172424]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.172428]  worker_thread+0x4ad/0xb40
[  247.172434]  ? __pfx_worker_thread+0x10/0x10
[  247.172437]  kthread+0x351/0x450
[  247.172439]  ? __pfx_kthread+0x10/0x10
[  247.172442]  ret_from_fork+0x563/0x800
[  247.172445]  ? __pfx_ret_from_fork+0x10/0x10
[  247.172448]  ? __pfx_kthread+0x10/0x10
[  247.172450]  ? __switch_to+0x58e/0xea0
[  247.172454]  ? __pfx_kthread+0x10/0x10
[  247.172457]  ret_from_fork_asm+0x1a/0x30
[  247.172465]  </TASK>
[  247.172471] INFO: task kworker/61:4:1051 blocked for more than 122 seconds.
[  247.172472]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.172473] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.172474] task:kworker/61:4    state:D stack:26016 pid:1051 tgid:1051  ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.172481] Workqueue: dio/sda iomap_dio_complete_work
[  247.172484] Call Trace:
[  247.172485]  <TASK>
[  247.172487]  __schedule+0x1179/0x44c0
[  247.172490]  ? kick_pool_pick+0x24c/0x640
[  247.172496]  ? __pfx___schedule+0x10/0x10
[  247.172500]  ? lock_release+0xc9/0x290
[  247.172504]  schedule+0xd2/0x260
[  247.172508]  schedule_preempt_disabled+0x11/0x20
[  247.172511]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.172517]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.172526]  ? xfs_dec_freecounter+0xec/0x580
[  247.172531]  down_write_nested+0x1b5/0x1e0
[  247.172534]  ? __pfx_down_write_nested+0x10/0x10
[  247.172539]  ? xfs_ilock+0x2a0/0x3d0
[  247.172543]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.172546]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.172550]  ? __lock_acquire+0x607/0x1f70
[  247.172555]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.172560]  iomap_dio_complete+0x136/0xaa0
[  247.172563]  ? process_scheduled_works+0xaee/0x1b30
[  247.172565]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.172569]  iomap_dio_complete_work+0x51/0x80
[  247.172572]  process_scheduled_works+0xc08/0x1b30
[  247.172586]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.172589]  ? lock_acquire+0x185/0x2e0
[  247.172594]  ? lock_is_held_type+0x9e/0x120
[  247.172597]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.172601]  worker_thread+0x4ad/0xb40
[  247.172607]  ? __pfx_worker_thread+0x10/0x10
[  247.172610]  kthread+0x351/0x450
[  247.172612]  ? __pfx_kthread+0x10/0x10
[  247.172615]  ret_from_fork+0x563/0x800
[  247.172618]  ? __pfx_ret_from_fork+0x10/0x10
[  247.172621]  ? __pfx_kthread+0x10/0x10
[  247.172623]  ? __switch_to+0x58e/0xea0
[  247.172626]  ? __pfx_kthread+0x10/0x10
[  247.172629]  ret_from_fork_asm+0x1a/0x30
[  247.172637]  </TASK>
[  247.172804] INFO: task kworker/61:4:1051 <writer> blocked on an rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.172810] INFO: task kworker/34:2:1071 blocked for more than 122 seconds.
[  247.172812]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.172813] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.172814] task:kworker/34:2    state:D stack:26360 pid:1071 tgid:1071  ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.172820] Workqueue: dio/sda iomap_dio_complete_work
[  247.172823] Call Trace:
[  247.172824]  <TASK>
[  247.172827]  __schedule+0x1179/0x44c0
[  247.172830]  ? kick_pool_pick+0x24c/0x640
[  247.172836]  ? __pfx___schedule+0x10/0x10
[  247.172840]  ? lock_release+0xc9/0x290
[  247.172844]  schedule+0xd2/0x260
[  247.172848]  schedule_preempt_disabled+0x11/0x20
[  247.172851]  rwsem_down_write_slowpath+0x3d6/0x1670
[  247.172857]  ? __pfx_rwsem_down_write_slowpath+0x10/0x10
[  247.172865]  ? xfs_dec_freecounter+0xec/0x580
[  247.172869]  down_write_nested+0x1b5/0x1e0
[  247.172872]  ? __pfx_down_write_nested+0x10/0x10
[  247.172876]  ? xfs_ilock+0x2a0/0x3d0
[  247.172880]  xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.172884]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.172888]  ? __lock_acquire+0x607/0x1f70
[  247.172893]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.172898]  iomap_dio_complete+0x136/0xaa0
[  247.172901]  ? process_scheduled_works+0xaee/0x1b30
[  247.172903]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.172907]  iomap_dio_complete_work+0x51/0x80
[  247.172910]  process_scheduled_works+0xc08/0x1b30
[  247.172924]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.172927]  ? lock_acquire+0x185/0x2e0
[  247.172931]  ? lock_is_held_type+0x9e/0x120
[  247.172935]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.172939]  worker_thread+0x4ad/0xb40
[  247.172944]  ? __pfx_worker_thread+0x10/0x10
[  247.172947]  kthread+0x351/0x450
[  247.172949]  ? __pfx_kthread+0x10/0x10
[  247.172952]  ret_from_fork+0x563/0x800
[  247.172956]  ? __pfx_ret_from_fork+0x10/0x10
[  247.172959]  ? __pfx_kthread+0x10/0x10
[  247.172961]  ? __switch_to+0x58e/0xea0
[  247.172964]  ? __pfx_kthread+0x10/0x10
[  247.172967]  ret_from_fork_asm+0x1a/0x30
[  247.172975]  </TASK>
[  247.173163] INFO: task kworker/34:2:1071 <writer> blocked on an rw-semaphore likely owned by task kworker/5:10:1933 <writer>
[  247.173167] INFO: task kworker/6:5:1080 blocked for more than 122 seconds.
[  247.173169]       Not tainted 7.3.0-rc2-00095-g75fd6e0692c3-dirty #265
[  247.173170] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.173171] task:kworker/6:5     state:D stack:26424 pid:1080 tgid:1080  ppid:2      task_flags:0x4248060 flags:0x00080000
[  247.173178] Workqueue: dio/sda iomap_dio_complete_work
[  247.173181] Call Trace:
[  247.173181]  <TASK>
[  247.173184]  __schedule+0x1179/0x44c0
[  247.173187]  ? kick_pool_pick+0x24c/0x640
[  247.173193]  ? __pfx___schedule+0x10/0x10
[  247.173197]  ? lock_release+0xc9/0x290
[  247.173201]  schedule+0xd2/0x260
[  247.173205]  xlog_grant_head_wait+0x112/0x9b0
[  247.173211]  xlog_grant_head_check+0x377/0x500
[  247.173214]  ? __pfx_xlog_grant_head_check+0x10/0x10
[  247.173221]  xfs_log_reserve+0x291/0x6f0
[  247.173225]  ? __pfx_xfs_log_reserve+0x10/0x10
[  247.173228]  ? __percpu_counter_compare+0xa8/0xf0
[  247.173233]  ? xfs_dec_freecounter+0xec/0x580
[  247.173237]  xfs_trans_alloc+0x205/0x900
[  247.173244]  xfs_reflink_end_atomic_cow+0x197/0x640
[  247.173248]  ? __pfx_xfs_reflink_end_atomic_cow+0x10/0x10
[  247.173252]  ? __lock_acquire+0x607/0x1f70
[  247.173257]  xfs_dio_write_end_io+0x58c/0x6e0
[  247.173262]  iomap_dio_complete+0x136/0xaa0
[  247.173265]  ? process_scheduled_works+0xaee/0x1b30
[  247.173268]  ? __pfx_aio_complete_rw+0x10/0x10
[  247.173272]  iomap_dio_complete_work+0x51/0x80
[  247.173275]  process_scheduled_works+0xc08/0x1b30
[  247.173288]  ? __pfx_process_scheduled_works+0x10/0x10
[  247.173291]  ? lock_acquire+0x185/0x2e0
[  247.173295]  ? lock_is_held_type+0x9e/0x120
[  247.173299]  ? __pfx_iomap_dio_complete_work+0x10/0x10
[  247.173303]  worker_thread+0x4ad/0xb40
[  247.173309]  ? __pfx_worker_thread+0x10/0x10
[  247.173312]  kthread+0x351/0x450
[  247.173314]  ? __pfx_kthread+0x10/0x10
[  247.173317]  ret_from_fork+0x563/0x800
[  247.173320]  ? __pfx_ret_from_fork+0x10/0x10
[  247.173323]  ? __pfx_kthread+0x10/0x10
[  247.173325]  ? __switch_to+0x58e/0xea0
[  247.173328]  ? __pfx_kthread+0x10/0x10
[  247.173331]  ret_from_fork_asm+0x1a/0x30
[  247.173339]  </TASK>
[  247.173340] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
[  247.173778]
[  247.173778] Showing all locks held in the system:
[  247.173782] locks held by pr/ttyS0/15: 2, last CPU#41:
[  247.173784]  #0: ffffffff892141f8 (console_srcu){....}-{0:0}, at: console_srcu_read_lock+0x46/0x60
[  247.173792]  #1: ffffffff8c4893d8 (&port_lock_key){-...}-{3:3}, at: univ8250_console_device_lock+0x56/0xa0
[  247.173839] locks held by khungtaskd/356: 1, last CPU#40:
[  247.173841]  #0: ffffffff89221460 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x36/0x1c0
[  247.173858] locks held by syslogd/464: 1, last CPU#59:
[  247.173860]  #0: ffff888105817cc8 (&u->iolock){+.+.}-{4:4}, at: __unix_dgram_recvmsg+0x1ad/0xc30
[  247.173872] locks held by kworker/34:1/568: 4, on CPU#34:
[  247.173874]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.173880]  #1: ffff88812a6a7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.173885]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.173891]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.173915] locks held by kworker/48:12/800: 4, on CPU#48:
[  247.173917]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.173922]  #1: ffff88814a0dfd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.173927]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.173932]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.173938] locks held by kworker/48:13/807: 4, on CPU#48:
[  247.173940]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.173945]  #1: ffff88814a12fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.173950]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.173955]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.173972] locks held by kworker/52:22/893: 4, on CPU#52:
[  247.173973]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.173979]  #1: ffff88814a4ffd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.173983]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.173988]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174009] locks held by kworker/5:4/1006: 4, on CPU#5:
[  247.174011]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174016]  #1: ffff88814b3efd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174021]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174026]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174032] locks held by kworker/5:6/1009: 4, on CPU#5:
[  247.174033]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174039]  #1: ffff88814b48fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174043]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174273]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174281] locks held by kworker/7:3/1019: 3, on CPU#7:
[  247.174282]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174288]  #1: ffff88814b687d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174293]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174302] locks held by kworker/61:4/1051: 4, on CPU#61:
[  247.174304]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174309]  #1: ffff88814bb0fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174314]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174319]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174327] locks held by kworker/34:2/1071: 4, on CPU#34:
[  247.174328]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174334]  #1: ffff88814bc57d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174338]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174343]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174349] locks held by kworker/6:5/1080: 3, on CPU#6:
[  247.174351]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174356]  #1: ffff88814bda7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174361]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174371] locks held by kworker/34:19/1123: 4, on CPU#34:
[  247.174373]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174378]  #1: ffff88814c187d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174383]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174387]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174395] locks held by kworker/7:10/1144: 3, on CPU#7:
[  247.174397]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174402]  #1: ffff88814c497d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174407]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174428] locks held by kworker/59:3/1308: 3, on CPU#59:
[  247.174430]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174435]  #1: ffff88813524fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174440]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174445] locks held by kworker/59:4/1309: 3, on CPU#59:
[  247.174446]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
Jobs: 14 (f=14):[  247.174452]  #1: ffff8881036b7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
 [W(14)][10.5%][[  247.174457]  #2: ffff88813af38628 
(sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174462] locks held by kworker/59:5/1310: 4, on CPU#59:
[  247.174463]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174469]  #1: ffff888135e6fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174473]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174478]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174498] locks held by kworker/28:2/1449: 4, on CPU#28:
[  247.174499]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174505]  #1: ffff888148d87d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174510]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174515]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174521] locks held by kworker/28:4/1454: 4, on CPU#28:
[  247.174522]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174527]  #1: ffff888148c7fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174532]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174537]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174542] locks held by kworker/28:6/1457: 3, on CPU#28:
[  247.174544]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174549]  #1: ffff888148b5fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174554]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174559] locks held by kworker/28:8/1461: 4, on CPU#28:
[  247.174561]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174566]  #1: ffff888148c5fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174571]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174576]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174588] locks held by kworker/28:9/1537: 4, on CPU#28:
[  247.174590]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174595]  #1: ffff888147537d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174600]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174605]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174612] locks held by kworker/28:13/1549: 4, on CPU#28:
[  247.174613]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174618]  #1: ffff88814770fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174623]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174628]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174635] locks held by kworker/60:10/1577: 4, on CPU#60:
[  247.174636]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174642]  #1: ffff8881446ffd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174656]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174661]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174666] locks held by kworker/60:11/1578: 4, on CPU#60:
[  247.174667]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174673]  #1: ffff888144707d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174677]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174682]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174699] locks held by kworker/59:10/1722: 3, on CPU#59:
[  247.174701]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174706]  #1: ffff888143dffd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174711]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174717] locks held by kworker/59:11/1726: 4, on CPU#59:
[  247.174719]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174724]  #1: ffff888143e2fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174729]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174733]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174739] locks held by kworker/59:12/1727: 4, on CPU#59:
[  247.174740]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174746]  #1: ffff888143e37d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174751]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174755]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174761] locks held by kworker/59:14/1729: 3, on CPU#59:
[  247.174762]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174767]  #1: ffff888143e4fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174772]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174780] locks held by kworker/59:18/1772: 4, on CPU#59:
[  247.174782]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174787]  #1: ffff888147097d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174792]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174797]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174804] locks held by kworker/59:20/1800: 4, on CPU#59:
[  247.174805]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174810]  #1: ffff8881471cfd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174815]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174820]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174828] locks held by kworker/25:7/1821: 4, on CPU#25:
[  247.174829]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174835]  #1: ffff888146b9fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174840]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174845]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174850] locks held by kworker/25:13/1834: 2, last CPU#25:
[  247.174852]  #0: ffff88810028cd40 ((wq_completion)events_power_efficient){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174857]  #1: ffff88814642fd30 ((work_completion)(&(&gc_work->dwork)->work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174863] locks held by kworker/48:36/1852: 3, on CPU#48:
[  247.174865]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174870]  #1: ffff88813f75fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174875]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174880] locks held by kworker/48:38/1854: 3, on CPU#48:
[  247.174881]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174887]  #1: ffff88812bfffd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174892]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174897] locks held by kworker/48:41/1859: 4, on CPU#48:
[  247.174899]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174904]  #1: ffff88814654fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174909]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174914]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174919] locks held by kworker/60:26/1863: 4, on CPU#60:
[  247.174920]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174926]  #1: ffff888146577d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174930]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174935]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174945] locks held by kworker/5:10/1933: 4, on CPU#5:
[  247.174947]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174952]  #1: ffff888146367d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174957]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174962]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.174967] locks held by kworker/5:13/1936: 3, on CPU#5:
[  247.174968]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174974]  #1: ffff88814637fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174979]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.174984] locks held by kworker/5:14/1937: 3, on CPU#5:
[  247.174985]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.174990]  #1: ffff88814638fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.174995]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175000] locks held by kworker/5:15/1938: 3, on CPU#5:
[  247.175002]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175007]  #1: ffff888146397d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175012]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175020] locks held by kworker/14:20/1967: 4, on CPU#14:
[  247.175022]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175027]  #1: ffff888142a0fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175032]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175037]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175071] locks held by kworker/62:21/2093: 4, on CPU#62:
[  247.175073]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175079]  #1: ffff8881454b7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175084]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175089]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175108] locks held by kworker/61:26/2190: 4, on CPU#61:
[  247.175110]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175115]  #1: ffff888145307d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175120]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175125]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175131] locks held by kworker/61:32/2196: 4, on CPU#61:
[  247.175133]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175138]  #1: ffff888145347d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175143]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175148]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175158] locks held by kworker/61:65/2234: 4, on CPU#61:
[  247.175160]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175165]  #1: ffff888144c87d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175170]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175175]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175181] locks held by kworker/61:71/2241: 4, on CPU#61:
[  247.175182]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175188]  #1: ffff888144ccfd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175193]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175198]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175204] locks held by kworker/61:73/2244: 4, on CPU#61:
[  247.175205]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175211]  #1: ffff888144cefd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175216]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175221]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175227] locks held by kworker/61:75/2246: 3, on CPU#61:
[  247.175228]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175234]  #1: ffff888144d07d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175239]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175244] locks held by kworker/61:77/2248: 3, on CPU#61:
[  247.175246]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175251]  #1: ffff888144d17d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175257]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175262] locks held by kworker/61:78/2249: 4, on CPU#61:
[  247.175264]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175270]  #1: ffff888144d27d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175275]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175280]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175286] locks held by kworker/61:80/2251: 3, on CPU#61:
[  247.175288]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175294]  #1: ffff888144d37d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175299]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175304] locks held by kworker/61:81/2252: 3, on CPU#61:
[  247.175306]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175311]  #1: ffff888144d3fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175317]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175329] locks held by kworker/35:38/2302: 4, on CPU#35:
[  247.175331]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175336]  #1: ffff8881448c7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175341]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175346]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175362] locks held by kworker/34:36/2368: 4, on CPU#34:
[  247.175364]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175370]  #1: ffff888145967d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175374]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175379]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175385] locks held by kworker/34:37/2369: 4, on CPU#34:
[  247.175386]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175392]  #1: ffff888145977d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175397]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175402]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175408] locks held by kworker/34:44/2376: 4, on CPU#34:
[  247.175410]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175415]  #1: ffff8881459c7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175420]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175425]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175448] locks held by kworker/7:27/2494: 3, on CPU#7:
[  247.175450]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175455]  #1: ffff8881432ffd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175461]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175466] locks held by kworker/6:28/2495: 4, on CPU#6:
[  247.175468]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175473]  #1: ffff888143307d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175479]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175484]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175490] locks held by kworker/7:31/2502: 3, on CPU#7:
[  247.175491]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175497]  #1: ffff88814335fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175502]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175507] locks held by kworker/6:37/2509: 3, on CPU#6:
[  247.175509]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175515]  #1: ffff8881433afd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175519]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175525] locks held by kworker/7:34/2513: 4, on CPU#7:
[  247.175526]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175532]  #1: ffff8881433d7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175537]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175542]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175547] locks held by kworker/6:44/2519: 3, on CPU#6:
[  247.175549]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175554]  #1: ffff888142c0fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175559]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175566] locks held by kworker/59:21/2537: 4, on CPU#59:
[  247.175568]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175574]  #1: ffff888142ce7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175579]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175584]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175590] locks held by kworker/59:22/2538: 3, on CPU#59:
[  247.175591]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175597]  #1: ffff888142cefd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175601]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175607] locks held by kworker/59:26/2542: 4, on CPU#59:
[  247.175608]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175614]  #1: ffff88813fe87d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175619]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175624]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175629] locks held by kworker/59:27/2543: 3, on CPU#59:
[  247.175631]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175636]  #1: ffff88813fe77d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175641]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175657] locks held by kworker/59:28/2544: 3, on CPU#59:
[  247.175659]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175664]  #1: ffff88813fcd7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175669]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175680] locks held by kworker/60:35/2583: 3, on CPU#60:
[  247.175682]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175687]  #1: ffff888142f8fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175692]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175700] locks held by kworker/26:7/2606: 4, on CPU#26:
[  247.175701]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175707]  #1: ffff888142477d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175711]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175716]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175727] locks held by kworker/14:27/2649: 3, on CPU#14:
[  247.175729]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175734]  #1: ffff888142617d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175739]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175747] locks held by kworker/14:39/2673: 4, on CPU#14:
[  247.175749]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175754]  #1: ffff888142727d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175759]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175764]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175770] locks held by kworker/14:49/2687: 4, on CPU#14:
[  247.175772]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175777]  #1: ffff8881427bfd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175782]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175787]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175792] locks held by kworker/14:50/2688: 4, on CPU#14:
[  247.175794]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175799]  #1: ffff8881427c7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175804]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175808]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175814] locks held by kworker/5:20/2691: 4, on CPU#5:
[  247.175815]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175820]  #1: ffff8881427e7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175825]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175830]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175836] locks held by kworker/5:22/2695: 4, on CPU#5:
[  247.175837]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175842]  #1: ffff888146ea7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175847]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175852]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175872] locks held by kworker/60:46/2815: 4, on CPU#60:
[  247.175874]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175879]  #1: ffff88813995fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175884]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175889]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175896] locks held by kworker/59:30/2827: 4, on CPU#59:
[  247.175898]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175903]  #1: ffff888138a77d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175908]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175913]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175918] locks held by kworker/59:31/2829: 3, on CPU#59:
[  247.175920]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175925]  #1: ffff888145a7fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175930]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175935] locks held by kworker/59:33/2831: 4, on CPU#59:
[  247.175936]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175942]  #1: ffff88813ceffd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175946]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175951]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.175957] locks held by kworker/59:35/2833: 3, on CPU#59:
[  247.175958]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175963]  #1: ffff88812c9afd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175968]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.175980] locks held by kworker/28:16/2887: 3, on CPU#28:
[  247.175981]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.175987]  #1: ffff88814d967d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.175992]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176000] locks held by kworker/48:50/2910: 4, on CPU#48:
[  247.176002]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176007]  #1: ffff88814db57d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176012]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176017]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176035] locks held by kworker/26:38/3006: 4, on CPU#26:
[  247.176036]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176042]  #1: ffff888148f3fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176058]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176065]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176081] locks held by kworker/59:43/3082: 4, on CPU#59:
[  247.176082]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176088]  #1: ffff8881350b7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176093]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176098]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176103] locks held by kworker/59:44/3084: 3, on CPU#59:
[  247.176105]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176110]  #1: ffff888145d17d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176115]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176120] locks held by kworker/59:45/3085: 3, on CPU#59:
[  247.176122]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176127]  #1: ffff88814d76fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176132]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176137] locks held by kworker/59:47/3087: 4, on CPU#59:
[  247.176139]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176144]  #1: ffff88814d49fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176149]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176154]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176165] locks held by kworker/28:19/3123: 3, on CPU#28:
[  247.176166]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176171]  #1: ffff888146a97d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176176]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176191] locks held by kworker/34:49/3179: 4, on CPU#34:
[  247.176192]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176198]  #1: ffff88814e587d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176203]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176208]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176214] locks held by kworker/34:50/3180: 4, on CPU#34:
[  247.176215]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176221]  #1: ffff88814e58fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176225]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176230]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176237] locks held by kworker/34:54/3184: 4, on CPU#34:
[  247.176238]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176244]  #1: ffff888142dc7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176249]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176254]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176259] locks held by kworker/34:55/3185: 4, on CPU#34:
[  247.176260]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176266]  #1: ffff888142dcfd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176271]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176276]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176300] locks held by fio/3278: 2, on CPU#6:
[  247.176302]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176309]  #1: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176315] locks held by fio/3279: 2, on CPU#62:
[  247.176317]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176322]  #1: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176327] locks held by fio/3280: 2, on CPU#7:
[  247.176329]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176334]  #1: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176340] locks held by fio/3281: 2, on CPU#28:
[  247.176341]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176346]  #1: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176351] locks held by fio/3282: 2, on CPU#60:
[  247.176353]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176358]  #1: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176363] locks held by fio/3283: 2, on CPU#34:
[  247.176364]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176369]  #1: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176375] locks held by fio/3284: 3, on CPU#26:
[  247.176376]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176381]  #1: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176386]  #2: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x28c/0x4d0
[  247.176391] locks held by fio/3285: 2, on CPU#14:
[  247.176393]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176398]  #1: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176403] locks held by fio/3286: 3, on CPU#26:
[  247.176405]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176410]  #1: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176415]  #2: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x28c/0x4d0
[  247.176421] locks held by fio/3287: 2, on CPU#48:
[  247.176422]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176428]  #1: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176433] locks held by fio/3288: 2, on CPU#5:
[  247.176434]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176440]  #1: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176445] locks held by fio/3289: 3, on CPU#26:
[  247.176446]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176452]  #1: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176457]  #2: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x28c/0x4d0
[  247.176462] locks held by fio/3290: 2, on CPU#61:
[  247.176463]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176469]  #1: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_atomic_write_cow_iomap_next+0x3ea/0xc60
[  247.176474] locks held by fio/3291: 3, on CPU#26:
[  247.176476]  #0: ffff8881247bae90 (&sb->s_type->i_mutex_key#14){++++}-{4:4}, at: xfs_ilock+0x253/0x3d0
[  247.176481]  #1: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176486]  #2: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_vn_update_time+0x28c/0x4d0
[  247.176505] locks held by kworker/62:53/3393: 3, on CPU#62:
[  247.176506]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176512]  #1: ffff88814759fd30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176517]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176523] locks held by kworker/62:54/3394: 4, on CPU#62:
[  247.176524]  #0: ffff888135aeb140 ((wq_completion)dio/sda#9){+.+.}-{0:0}, at: process_scheduled_works+0xff2/0x1b30
[  247.176530]  #1: ffff88814f5d7d30 ((work_completion)(&dio->aio.work)){+.+.}-{0:0}, at: process_scheduled_works+0xb9f/0x1b30
[  247.176535]  #2: ffff88813af38628 (sb_internal){.+.+}-{0:0}, at: xfs_trans_alloc+0xa7/0x900
[  247.176540]  #3: ffff8881247bac80 (&xfs_nondir_ilock_class){++++}-{4:4}, at: xfs_reflink_end_atomic_cow+0x2ea/0x640
[  247.176546]
[  247.176547] =============================================
[  247.176547]
#14 lore raw
In reply to Shin'ichiro Kawasaki

On Sep 17, 2026 / 15:52, Shin'ichiro Kawasaki wrote:
[...]

Quoted 8 lines

As suggested, now I'm trying to increase the journal/log section size. I added the mkfs options below:

MKFS_OPTIONS="-l size=128m"

With this, I repeated the test case 30 times, and did not observe the hang. It looks like the log section size increase avoids the hang. I will keep it on running to see if the hang is observed after 100 times run.

The test case did not hang all through the 100 times repeated runs. Then expanding the log section from 64MiB to 128MiB avoided the hang.

#9 lore raw
In reply to John Garry
Quoted 52 lines · John Garry
Quoted 41 lines · Dave Chinner
Quoted 31 lines · John Garry
Quoted 11 lines · John Garry
Quoted 7 lines · Darrick J. Wong
Quoted 6 lines

thanks

About the kernel code, I am wondering if using IOMAP_DIO_FORCE_WAIT for CoW-based atomics could help avoid this issue as we seem to be bogged down
in lock contention.

Which lock is being contended, anyway?  It looks like the ILOCK?

Yeah, I think so Re. ilock. I had some perf data illustrating this from last year, which I can't seem to find, so I will re-generate it.

Here is perf call graph snippet when running fio with 72x threads issuing 4KB atomic writes on 1MB file:

--50.46%--xfs_file_dio_write_atomic

|
|--37.19%--iomap_dio_rw
|          |
|      --37.14%--__iomap_dio_rw
|           |
|       --36.50%--iomap_iter
|            |
|             --36.49%--xfs_atomic_write_cow_iomap_next
|                  |
|                  |--27.34%--xfs_ilock
|                  |          |
|                  |      --27.34%--down_write
|                  |           |
|                  |       --27.28%--rwsem_down_write_slowpath
Quoted 8 lines
|                  |            |
|                  |            |--24.65%--osq_lock
|                  |            |
|                  |             --2.35%--rwsem_spin_on_owner
|                  |
|                  |--7.62%--xfs_trans_alloc_inode
|                  |          |
|                  |      --7.59%--xfs_ilock
Quoted 1 line

Where's the other half of the lock contention?

I was expecting xfs_reflink_end_atomic_cow() -> xfs_ilock() to be a point of contention as well, but it does not show up. I suppose that it because all those threads at completion stage are sleeping.

However, there is this which I omitted:

--13.21%--xfs_file_write_checks
|

Quoted 9 lines
      --13.18%--kiocb_modified
         |
            --13.17%--file_update_time_flags
               |
                  --13.16%--xfs_vn_update_time
                     |
                        --12.99%--xfs_ilock
                           |
                              --12.98%--down_write

Fair enough, I'd expect that from the all the threads blocked on the lock there in the hung task output.

....

Quoted 2 lines

JFYI, I have another log for a similar hang which we see one thread deadlock for log space, below.

Looks very similar. So, same questiosn about log size, etc.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#15 lore raw
In reply to Shin'ichiro Kawasaki
Quoted 5 lines · Shin'ichiro Kawasaki

On Sep 17, 2026 / 15:52, Shin'ichiro Kawasaki wrote:
[...]

Quoted 3 lines

As suggested, now I'm trying to increase the journal/log section size. I added the mkfs options below:

Quoted 8 lines
Quoted 5 lines

MKFS_OPTIONS="-l size=128m"

With this, I repeated the test case 30 times, and did not observe the hang. It looks like the log section size increase avoids the hang. I will keep it on running to see if the hang is observed after 100 times run.

The test case did not hang all through the 100 times repeated runs. Then expanding the log section from 64MiB to 128MiB avoided the hang.

Ok, so the fix for the moment is to make the software COW path for atomic writes use IOLOCK_EXCL, rather than IOLOCK_SHARED. That will result in all the IO submission blocking on the IOLOCK with no other resources held rather than blocking on the ILOCK whilst holding an unreleasable transaction reservation.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#16 lore raw
Quoted 12 lines · Dave Chinner
Quoted 11 lines · Shin'ichiro Kawasaki

On Sep 17, 2026 / 15:52, Shin'ichiro Kawasaki wrote:
[...]

Quoted 8 lines

As suggested, now I'm trying to increase the journal/log section size. I added the mkfs options below:

MKFS_OPTIONS="-l size=128m"

With this, I repeated the test case 30 times, and did not observe the hang. It looks like the log section size increase avoids the hang. I will keep it on running to see if the hang is observed after 100 times run.

Quoted 8 lines
Quoted 2 lines

The test case did not hang all through the 100 times repeated runs. Then expanding the log section from 64MiB to 128MiB avoided the hang.

Ok, so the fix for the moment is to make the software COW path for atomic writes use IOLOCK_EXCL, rather than IOLOCK_SHARED. That will result in all the IO submission blocking on the IOLOCK with no other resources held rather than blocking on the ILOCK whilst holding an unreleasable transaction reservation.

Thanks for the explanation and the fix suggestion. Based on the suggestion, I cooked a fix trial patch below [4]. I applied this patch on top of the xfs-linux/for-next branch kernel at the git hash 0ca15a1a115. I repeated the test case generic/774 100 times on the kernel, and observed no hang. Looks working good. I will do further testing to confirm no regression. Meanwhile, comments on the patch will be appreciated.

[4] fix trial patch

-----------------------------------------------------------------------------

From 2d2cbd8e3424953880f6ffc8e2ed5b4f844b0822 Mon Sep 17 00:00:00 2001
From: Shin'ichiro Kawasaki <[email protected]>
Date: Fri, 18 Sep 2026 00:00:00 +0900
Subject: [PATCH] xfs: serialise software COW atomic write submission on the
IOLOCK

When fstests generic/774 is repeated on a 8GiB device with a 64MiB journal size, the test case hangs. Dozens of IO completion workers are blocked on the ILOCK in xfs_reflink_end_atomic_cow():

  down_write_nested+0x1c0/0x1f0
  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
  iomap_dio_complete+0x140/0xb20
  iomap_dio_complete_work+0x58/0x90
  process_one_work+0x947/0x1760

while the ILOCK holder is waiting for journal space:

  xlog_grant_head_wait+0x175/0xac0 [xfs]
  xlog_grant_head_check+0x312/0x3f0 [xfs]
  xfs_log_regrant+0x380/0x7d0 [xfs]
  xfs_trans_roll+0x2d9/0x420 [xfs]
  xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
  xfs_defer_finish_noroll+0x460/0xe70 [xfs]
  xfs_trans_commit+0xfc/0x180 [xfs]
  xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
  xfs_dio_write_end_io+0x4b7/0x650 [xfs]

Each atomic write software COW submitter allocates around 1.6MiB of journal resource under ILOCK. When a few dozen of atomic writes are submitted, it is enough to run out of the 64MiB journal size. This caused the deadlock between ILOCK and the journal resource.

To avoid the deadlock, take the IOLOCK exclusively at the atomic write software COW submission. This serializes the submission in the IO path, so concurrent atomic writes block on the IOLOCK holding no other resources.

---
 fs/xfs/xfs_file.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
fs/xfs/xfs_file.c +14 -2
index 426a67b813a..8d10ae22884 100644
@@ -829,7 +829,7 @@ xfs_file_dio_write_atomic(
 	struct kiocb		*iocb,
 	struct iov_iter		*from)
 {
-	unsigned int		iolock = XFS_IOLOCK_SHARED;
+	unsigned int		iolock;
 	ssize_t			ret, ocount = iov_iter_count(from);
 	unsigned int		dio_flags = 0;
 	const struct iomap_ops	*dops;
@@ -844,6 +844,17 @@ xfs_file_dio_write_atomic(
 		dops = &xfs_direct_write_iomap_ops;
 
 retry:
+	/*
+	 * Concurrent atomic write COW submissions under ILOCK can run out
+	 * journal reservation resource and can deadlock with the IO
+	 * completions. To avoid the deadlock, serialize submissions of
+	 * atomic write COW by taking IOLOCK exclusively.
+	 */
+	if (dops == &xfs_atomic_write_cow_iomap_ops)
+		iolock = XFS_IOLOCK_EXCL;
+	else
+		iolock = XFS_IOLOCK_SHARED;
+
 	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
 	if (ret)
 		return ret;
@@ -853,7 +864,8 @@ xfs_file_dio_write_atomic(
 		goto out_unlock;
 
 	/* Demote similar to xfs_file_dio_write_aligned() */
-	if (iolock == XFS_IOLOCK_EXCL) {
+	if (iolock == XFS_IOLOCK_EXCL &&
+	    dops != &xfs_atomic_write_cow_iomap_ops) {
 		xfs_ilock_demote(ip, XFS_IOLOCK_EXCL);
 		iolock = XFS_IOLOCK_SHARED;
 	}
Signature · 1 line
2.55.0
#17 lore raw
Quoted 102 lines · Shin'ichiro Kawasaki
Quoted 12 lines · Dave Chinner
Quoted 11 lines · Shin'ichiro Kawasaki

On Sep 17, 2026 / 15:52, Shin'ichiro Kawasaki wrote:
[...]

Quoted 8 lines

As suggested, now I'm trying to increase the journal/log section size. I added the mkfs options below:

MKFS_OPTIONS="-l size=128m"

With this, I repeated the test case 30 times, and did not observe the hang. It looks like the log section size increase avoids the hang. I will keep it on running to see if the hang is observed after 100 times run.

Quoted 8 lines
Quoted 2 lines

The test case did not hang all through the 100 times repeated runs. Then expanding the log section from 64MiB to 128MiB avoided the hang.

Ok, so the fix for the moment is to make the software COW path for atomic writes use IOLOCK_EXCL, rather than IOLOCK_SHARED. That will result in all the IO submission blocking on the IOLOCK with no other resources held rather than blocking on the ILOCK whilst holding an unreleasable transaction reservation.

Thanks for the explanation and the fix suggestion. Based on the suggestion, I cooked a fix trial patch below [4]. I applied this patch on top of the xfs-linux/for-next branch kernel at the git hash 0ca15a1a115. I repeated the test case generic/774 100 times on the kernel, and observed no hang. Looks working good. I will do further testing to confirm no regression. Meanwhile, comments on the patch will be appreciated.

[4] fix trial patch

-----------------------------------------------------------------------------

From 2d2cbd8e3424953880f6ffc8e2ed5b4f844b0822 Mon Sep 17 00:00:00 2001
From: Shin'ichiro Kawasaki <[email protected]>
Date: Fri, 18 Sep 2026 00:00:00 +0900
Subject: [PATCH] xfs: serialise software COW atomic write submission on the
IOLOCK

When fstests generic/774 is repeated on a 8GiB device with a 64MiB journal size, the test case hangs. Dozens of IO completion workers are blocked on the ILOCK in xfs_reflink_end_atomic_cow():

  down_write_nested+0x1c0/0x1f0
  xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
  xfs_dio_write_end_io+0x4b7/0x650 [xfs]
  iomap_dio_complete+0x140/0xb20
  iomap_dio_complete_work+0x58/0x90
  process_one_work+0x947/0x1760

while the ILOCK holder is waiting for journal space:

  xlog_grant_head_wait+0x175/0xac0 [xfs]
  xlog_grant_head_check+0x312/0x3f0 [xfs]
  xfs_log_regrant+0x380/0x7d0 [xfs]
  xfs_trans_roll+0x2d9/0x420 [xfs]
  xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
  xfs_defer_finish_noroll+0x460/0xe70 [xfs]
  xfs_trans_commit+0xfc/0x180 [xfs]
  xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
  xfs_dio_write_end_io+0x4b7/0x650 [xfs]

Each atomic write software COW submitter allocates around 1.6MiB of journal resource under ILOCK. When a few dozen of atomic writes are submitted, it is enough to run out of the 64MiB journal size. This caused the deadlock between ILOCK and the journal resource.

To avoid the deadlock, take the IOLOCK exclusively at the atomic write software COW submission. This serializes the submission in the IO path, so concurrent atomic writes block on the IOLOCK holding no other resources.

---
 fs/xfs/xfs_file.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
fs/xfs/xfs_file.c +5 -1
index 426a67b813a..8d10ae22884 100644
@@ -829,7 +829,7 @@ xfs_file_dio_write_atomic(
 	struct kiocb		*iocb,
 	struct iov_iter		*from)
 {
-	unsigned int		iolock = XFS_IOLOCK_SHARED;
+	unsigned int		iolock;
 	ssize_t			ret, ocount = iov_iter_count(from);
 	unsigned int		dio_flags = 0;
 	const struct iomap_ops	*dops;
@@ -844,6 +844,17 @@ xfs_file_dio_write_atomic(
 		dops = &xfs_direct_write_iomap_ops;
 
 retry:
+	/*
+	 * Concurrent atomic write COW submissions under ILOCK can run out
+	 * journal reservation resource and can deadlock with the IO
+	 * completions. To avoid the deadlock, serialize submissions of

fs/xfs/xfs_file.c xfs_file_dio_write_atomic( · line 860

Quoted 10 lines
fs/xfs/xfs_file.c +7 -0
+	 * atomic write COW by taking IOLOCK exclusively.
+	 */
+	if (dops == &xfs_atomic_write_cow_iomap_ops)
+		iolock = XFS_IOLOCK_EXCL;
+	else
+		iolock = XFS_IOLOCK_SHARED;
+
 	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
 	if (ret)
 		return ret;

Urk, that's pretty nasty.

fs/xfs/xfs_file.c xfs_file_dio_write_atomic( · line 871

Quoted 10 lines
fs/xfs/xfs_file.c +2 -1
@@ -853,7 +864,8 @@ xfs_file_dio_write_atomic(
 		goto out_unlock;
 
 	/* Demote similar to xfs_file_dio_write_aligned() */
-	if (iolock == XFS_IOLOCK_EXCL) {
+	if (iolock == XFS_IOLOCK_EXCL &&
+	    dops != &xfs_atomic_write_cow_iomap_ops) {
 		xfs_ilock_demote(ip, XFS_IOLOCK_EXCL);
 		iolock = XFS_IOLOCK_SHARED;
 	}

And at this point, we now have several atomic write cow ops specific operations in this function (locking, the retry loop, etc).

This feels much more like there should be a separate function for the software cow path, and the fast path simply calls it on ENOPROTOOPT from the dio submission. That gets rid of all the conditionals and looping from the fast path. i.e

	if (ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
		return xfs_file_dio_write_atomic_cow();

	/* do normal DIO write */

	if (error == -ENOPROTOOPT)
		return xfs_file_dio_write_atomic_cow();
	return error;

That essentially makes xfs_file_dio_write_atomic() and xfs_file_dio_write_aligned() the same code, except for the above two checks, hence they could easily be collapsed back into a common implementation is:

	if ((iocb->ki_flags & IOCB_ATOMIC) &&
	    ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
		return xfs_file_dio_write_atomic_cow();

	/* do normal DIO write */

	if (error == -ENOPROTOOPT)
		return xfs_file_dio_write_atomic_cow();
	return error;

That seems like a much more natural breakdown that the current duplication of the DIO write submission code...

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#21 lore raw
In reply to Dave Chinner
Quoted 61 lines · Dave Chinner
Quoted 31 lines · Shin'ichiro Kawasaki

I observe the fstests test case generic/774 hangs, when I run it for xfs on 8GiB TCMU fileio devices. Actually I once reported this hang symptom in last October, and Darrick and John kindly took some actions [1]. Since then, the hang disappeared and I have not observed the hang almost one year. However, my test system started reporting the hang again last week.

[1] https://lore.kernel.org/linux-xfs/cmk52aqexackyz65phxgme55a3tdrermo3o4skr4lo4pwvvvcp@jmcblnfikbp2/#t

FYI, here I attache the kernel message [2]. I observed the hang last week with the kernel on the xfs/for-next branch at the commit 0ca15a1a1151 ("xfs: remove an extra cast in xfs_file_compat_ioctl"). Today, I tried again with the kernel at current xfs/for-next branch tip 4266ffdfd7cb ("xfs: remove an extra cast in xfs_file_compat_ioctl"), and observed the hang again. The hang can be recreated in stable manner by repeating the test case 20 times or so.

I also tried other test cases in atmoicwrites group, and observed no failure.

generic/765 [not run] write atomic not supported by this block device

 generic/767         11s
 generic/768         13s
 generic/769         13s
 generic/770         32s
 generic/773        [not run] write atomic not supported by this block device
 generic/774        (skipped)
 generic/775         293s
 generic/776        [notrun] write atomic not supported by this block device
 generic/778         48s
 xfs/838            [not run] External volumes not in use, skipped this test
 xfs/839            [not run] XFS error injection requires CONFIG_XFS_DEBUG
 xfs/840            [not run] write atomic not supported by this block device
Quoted 2 lines

Action for fix will be appreciated. If I can do anything on my test node, please let me know.

.....

A bunch of threads waiting on the ILOCK here:

 down_write_nested+0x1c0/0x1f0
 xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
 xfs_dio_write_end_io+0x4b7/0x650 [xfs]
 iomap_dio_complete+0x140/0xb20
 iomap_dio_complete_work+0x58/0x90
 process_one_work+0x947/0x1760

And the holder:

 schedule+0xe5/0x2e0
 xlog_grant_head_wait+0x175/0xac0 [xfs]
 xlog_grant_head_check+0x312/0x3f0 [xfs]
 xfs_log_regrant+0x380/0x7d0 [xfs]
 xfs_trans_roll+0x2d9/0x420 [xfs]
 xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
 xfs_defer_finish_noroll+0x460/0xe70 [xfs]
 xfs_trans_commit+0xfc/0x180 [xfs]
 xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
 xfs_dio_write_end_io+0x4b7/0x650 [xfs]
 iomap_dio_complete+0x140/0xb20
 iomap_dio_complete_work+0x58/0x90
 process_one_work+0x947/0x1760
Quoted 6 lines

is waiting on log space whilst holding the ILOCK.

This looks to me like the test runs out of log space because of all the IO completions holding transaction reservations waiting on the ILOCK, whilst the ILOCK holder can't get enough log space to regrant on transaction roll to continue the transaction.

Agreed.

Why are we calling xlog_grant_head_check from within xfs_reflink_end_atomic_cow? I think the reason for doing that is because we've exhausted t_cnt in the ticket (i.e. we've already rolled more than tr_logcount times).

Oh. tr_atomic_ioend.tr_logcount is 5 on a rmap+reflink filesystem, which it inherits from tr_itruncate. However, tr_itruncate is only intended to remove two extents from a file, so it only needs 5 rolls. Coincidentally we calculate 5 rolls for each atomic extent remapping operation.

For atomic write ioends, what if we increased tr_logcount to 5x the number of remappings that would have to occur to finish the write? That would preallocate all the permanent reservation we'd need before we take the ILOCK, which avoids the situation of needing to obtain more log space while holding ILOCK.

The downside is that you'd have to limit the software awu_max even further, perhaps to 40% of the log size divided by (tr_logres*tr_logcount). However, we'd still be able to handle concurrent atomic writes to different parts of the file, at least until fragmentation got bad.

<shrug>

--D

Quoted 43 lines

Oh:

STATIC void
xfs_calc_default_atomic_ioend_reservation(
struct xfs_mount *mp,
struct xfs_trans_resv *resp)

{
        /* Pick a default that will scale reasonably for the log size. */
        resp->tr_atomic_ioend = resp->tr_itruncate;
}

Which means each reservation is probably holding hundreds of KB to MB of journal reservation. If the journal is 64MB in size, then a few dozen o these might be all that is needed to consume all the reservation space.

Hmmm - there are at least 30 tasks stuck in xfs_reflink_end_atomic_cow(), another 30 stuck in xfs_vn_update_time() holding reservations, and significant number xfs_atomic_write_cow_iomap_begin()->xfs_trans_alloc_inode() holding tr_write reservations.

IOWs, smells of the journal being run out of reservation space, and no new space being able to be freed because all the dirty inodes in the journal that need to be written back to free up space are locked waiting for journal space to come free....

How big is the journal in the filesystem being tested? If you increase the size of the journal, does it go away? Can you get a dump of the transaction reservation sizes for the filesystem in question (the xfs_db logres command can do this, IIRC) and then run the math on the reservations held from the dump of all the blocked tasks holding locks to see if this matches the size of the journal in the fs?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#18 lore raw
In reply to Dave Chinner
Quoted 1 line · Dave Chinner

On Sat, Sep 19, 2026 at 08:53:39PM +0900, Shin'ichiro Kawasaki wrote:

[...]

Quoted 78 lines
Quoted 21 lines
fs/xfs/xfs_file.c +5 -1
index 426a67b813a..8d10ae22884 100644
@@ -829,7 +829,7 @@ xfs_file_dio_write_atomic(
 	struct kiocb		*iocb,
 	struct iov_iter		*from)
 {
-	unsigned int		iolock = XFS_IOLOCK_SHARED;
+	unsigned int		iolock;
 	ssize_t			ret, ocount = iov_iter_count(from);
 	unsigned int		dio_flags = 0;
 	const struct iomap_ops	*dops;
@@ -844,6 +844,17 @@ xfs_file_dio_write_atomic(
 		dops = &xfs_direct_write_iomap_ops;
 
 retry:
+	/*
+	 * Concurrent atomic write COW submissions under ILOCK can run out
+	 * journal reservation resource and can deadlock with the IO
+	 * completions. To avoid the deadlock, serialize submissions of

fs/xfs/xfs_file.c xfs_file_dio_write_atomic( · line 860

Quoted 10 lines
fs/xfs/xfs_file.c +7 -0
+	 * atomic write COW by taking IOLOCK exclusively.
+	 */
+	if (dops == &xfs_atomic_write_cow_iomap_ops)
+		iolock = XFS_IOLOCK_EXCL;
+	else
+		iolock = XFS_IOLOCK_SHARED;
+
 	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
 	if (ret)
 		return ret;

Urk, that's pretty nasty.

fs/xfs/xfs_file.c xfs_file_dio_write_atomic( · line 871

Quoted 10 lines
fs/xfs/xfs_file.c +2 -1
@@ -853,7 +864,8 @@ xfs_file_dio_write_atomic(
 		goto out_unlock;
 
 	/* Demote similar to xfs_file_dio_write_aligned() */
-	if (iolock == XFS_IOLOCK_EXCL) {
+	if (iolock == XFS_IOLOCK_EXCL &&
+	    dops != &xfs_atomic_write_cow_iomap_ops) {
 		xfs_ilock_demote(ip, XFS_IOLOCK_EXCL);
 		iolock = XFS_IOLOCK_SHARED;
 	}

And at this point, we now have several atomic write cow ops specific operations in this function (locking, the retry loop, etc).

This feels much more like there should be a separate function for the software cow path, and the fast path simply calls it on ENOPROTOOPT from the dio submission. That gets rid of all the conditionals and looping from the fast path. i.e

	if (ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
		return xfs_file_dio_write_atomic_cow();

	/* do normal DIO write */

	if (error == -ENOPROTOOPT)
		return xfs_file_dio_write_atomic_cow();
	return error;

That essentially makes xfs_file_dio_write_atomic() and xfs_file_dio_write_aligned() the same code, except for the above two checks, hence they could easily be collapsed back into a common implementation is:

	if ((iocb->ki_flags & IOCB_ATOMIC) &&
	    ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
		return xfs_file_dio_write_atomic_cow();

	/* do normal DIO write */

	if (error == -ENOPROTOOPT)
		return xfs_file_dio_write_atomic_cow();
	return error;
Quoted 2 lines

That seems like a much more natural breakdown that the current duplication of the DIO write submission code...

Thanks for the comment. I'll try to factor out the duplication based on your idea.

On the other hand, Darrick suggested antoher solution approach to allocate enough space before taking ILOCK by increasing tr_logcount. I wonder which way is the better: "allocate enough space" or "take exclusive IOLOCK". I'll do some experiment for the "allocate enough space" approach before working on the dupliaction clean up.

#22 lore raw
In reply to Darrick J. Wong
Quoted 1 line · Darrick J. Wong

On Thu, Sep 17, 2026 at 07:45:04AM +1000, Dave Chinner wrote:

[...]

Quoted 51 lines
Quoted 25 lines

A bunch of threads waiting on the ILOCK here:

 down_write_nested+0x1c0/0x1f0
 xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
 xfs_dio_write_end_io+0x4b7/0x650 [xfs]
 iomap_dio_complete+0x140/0xb20
 iomap_dio_complete_work+0x58/0x90
 process_one_work+0x947/0x1760

And the holder:

 schedule+0xe5/0x2e0
 xlog_grant_head_wait+0x175/0xac0 [xfs]
 xlog_grant_head_check+0x312/0x3f0 [xfs]
 xfs_log_regrant+0x380/0x7d0 [xfs]
 xfs_trans_roll+0x2d9/0x420 [xfs]
 xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
 xfs_defer_finish_noroll+0x460/0xe70 [xfs]
 xfs_trans_commit+0xfc/0x180 [xfs]
 xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
 xfs_dio_write_end_io+0x4b7/0x650 [xfs]
 iomap_dio_complete+0x140/0xb20
 iomap_dio_complete_work+0x58/0x90
 process_one_work+0x947/0x1760
Quoted 6 lines

is waiting on log space whilst holding the ILOCK.

This looks to me like the test runs out of log space because of all the IO completions holding transaction reservations waiting on the ILOCK, whilst the ILOCK holder can't get enough log space to regrant on transaction roll to continue the transaction.

Agreed.

Why are we calling xlog_grant_head_check from within xfs_reflink_end_atomic_cow? I think the reason for doing that is because we've exhausted t_cnt in the ticket (i.e. we've already rolled more than tr_logcount times).

Oh. tr_atomic_ioend.tr_logcount is 5 on a rmap+reflink filesystem, which it inherits from tr_itruncate. However, tr_itruncate is only intended to remove two extents from a file, so it only needs 5 rolls. Coincidentally we calculate 5 rolls for each atomic extent remapping operation.

For atomic write ioends, what if we increased tr_logcount to 5x the number of remappings that would have to occur to finish the write? That would preallocate all the permanent reservation we'd need before we take the ILOCK, which avoids the situation of needing to obtain more log space while holding ILOCK.

Quoted 5 lines

The downside is that you'd have to limit the software awu_max even further, perhaps to 40% of the log size divided by (tr_logres*tr_logcount). However, we'd still be able to handle concurrent atomic writes to different parts of the file, at least until fragmentation got bad.

Darrick, thanks for the idea. I wanted to try out the idea, so I cooled the experimental patch below. Is this what you meant? It is a very rough patch, and does not reflect the "limit the software awu_max even further" part, it is just for experiment.

Anyway, I applied this patch on top of the kernel xfs/for-next branch git hash 0ca15a1a1151, and repeated the test case g774. It did not hang after 100 times repeat, so it looks working.

From c90ec246b0b9e14097b0ad1d1ef5661261dd61db Mon Sep 17 00:00:00 2001
From: Shin'ichiro Kawasaki <[email protected]>
Date: Thu, 24 Sep 2026 19:08:32 +0900
Subject: [PATCH] xfs: try out Darrick's idea for g774 hang

[experimental patch]

Darrick's idea from
https://lore.kernel.org/linux-xfs/20260922002814.GK2705364@frogsfrogsfrogs/

The deadlock of g774 caused by grant space while holding ILOCK. To avoid the deadlock, allocate enough space before taking the ILOCK by increasing tr_logcount. 5 rolls are required for each extent, then multiply tr_logcount by 5.

---
 fs/xfs/xfs_reflink.c | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)
fs/xfs/xfs_reflink.c +32 -1
index 48013613663..9d3da713a01 100644
@@ -1020,6 +1020,28 @@ xfs_reflink_end_cow(
 	return error;
 }
 
+/*
+ * Number of transaction rolls needed to remap a single extent during an atomic
+ * write ioend.
+ */
+#define XFS_ATOMIC_IOEND_ROLLS	5
+
+/*
+ * Compute the number of log operations to reserve for an atomic write ioend
+ * that remaps @blockcount blocks with a @logres byte reservation.
+ */
+STATIC int
+xfs_calc_atomic_write_ioend_logcount(
+	struct xfs_mount	*mp,
+	xfs_extlen_t		blockcount,
+	unsigned int		logres)
+{
+	if (blockcount == 0 || logres == 0)
+		return M_RES(mp)->tr_atomic_ioend.tr_logcount;
+
+	return blockcount * XFS_ATOMIC_IOEND_ROLLS;
+}
+
 /*
  * Fully remap all of the file's data fork at once, which is the critical part
  * in achieving atomic behaviour.
@@ -1036,6 +1058,7 @@ xfs_reflink_end_atomic_cow(
 	xfs_fileoff_t			end_fsb;
 	int				error = 0;
 	struct xfs_mount		*mp = ip->i_mount;
+	struct xfs_trans_res		resv = M_RES(mp)->tr_atomic_ioend;
 	struct xfs_trans		*tp;
 	unsigned int			resblks;
 
@@ -1051,7 +1074,15 @@ xfs_reflink_end_atomic_cow(
 	resblks = (end_fsb - offset_fsb) *
 			XFS_NEXTENTADD_SPACE_RES(mp, 1, XFS_DATA_FORK);
 
-	error = xfs_trans_alloc(mp, &M_RES(mp)->tr_atomic_ioend, resblks, 0,
+	/*
+	 * Reserve enough log operations to remap every block of this write
+	 * before we take ILOCK, so that no roll below has to wait for grant
+	 * space while holding ILOCK.
+	 */
+	resv.tr_logcount = xfs_calc_atomic_write_ioend_logcount(mp,
+			end_fsb - offset_fsb, resv.tr_logres);
+
+	error = xfs_trans_alloc(mp, &resv, resblks, 0,
 			XFS_TRANS_RESERVE, &tp);
 	if (error)
 		return error;
Signature · 1 line
2.55.0
#23 lore raw
Quoted 136 lines · Shin'ichiro Kawasaki
Quoted 1 line · Darrick J. Wong

On Thu, Sep 17, 2026 at 07:45:04AM +1000, Dave Chinner wrote:

[...]

Quoted 51 lines
Quoted 25 lines

A bunch of threads waiting on the ILOCK here:

 down_write_nested+0x1c0/0x1f0
 xfs_reflink_end_atomic_cow+0x2f3/0x560 [xfs]
 xfs_dio_write_end_io+0x4b7/0x650 [xfs]
 iomap_dio_complete+0x140/0xb20
 iomap_dio_complete_work+0x58/0x90
 process_one_work+0x947/0x1760

And the holder:

 schedule+0xe5/0x2e0
 xlog_grant_head_wait+0x175/0xac0 [xfs]
 xlog_grant_head_check+0x312/0x3f0 [xfs]
 xfs_log_regrant+0x380/0x7d0 [xfs]
 xfs_trans_roll+0x2d9/0x420 [xfs]
 xfs_defer_trans_roll+0x11e/0x4b0 [xfs]
 xfs_defer_finish_noroll+0x460/0xe70 [xfs]
 xfs_trans_commit+0xfc/0x180 [xfs]
 xfs_reflink_end_atomic_cow+0x3b2/0x560 [xfs]
 xfs_dio_write_end_io+0x4b7/0x650 [xfs]
 iomap_dio_complete+0x140/0xb20
 iomap_dio_complete_work+0x58/0x90
 process_one_work+0x947/0x1760
Quoted 6 lines

is waiting on log space whilst holding the ILOCK.

This looks to me like the test runs out of log space because of all the IO completions holding transaction reservations waiting on the ILOCK, whilst the ILOCK holder can't get enough log space to regrant on transaction roll to continue the transaction.

Agreed.

Why are we calling xlog_grant_head_check from within xfs_reflink_end_atomic_cow? I think the reason for doing that is because we've exhausted t_cnt in the ticket (i.e. we've already rolled more than tr_logcount times).

Oh. tr_atomic_ioend.tr_logcount is 5 on a rmap+reflink filesystem, which it inherits from tr_itruncate. However, tr_itruncate is only intended to remove two extents from a file, so it only needs 5 rolls. Coincidentally we calculate 5 rolls for each atomic extent remapping operation.

For atomic write ioends, what if we increased tr_logcount to 5x the number of remappings that would have to occur to finish the write? That would preallocate all the permanent reservation we'd need before we take the ILOCK, which avoids the situation of needing to obtain more log space while holding ILOCK.

Quoted 5 lines

The downside is that you'd have to limit the software awu_max even further, perhaps to 40% of the log size divided by (tr_logres*tr_logcount). However, we'd still be able to handle concurrent atomic writes to different parts of the file, at least until fragmentation got bad.

Darrick, thanks for the idea. I wanted to try out the idea, so I cooled the experimental patch below. Is this what you meant? It is a very rough patch, and does not reflect the "limit the software awu_max even further" part, it is just for experiment.

Anyway, I applied this patch on top of the kernel xfs/for-next branch git hash 0ca15a1a1151, and repeated the test case g774. It did not hang after 100 times repeat, so it looks working.

From c90ec246b0b9e14097b0ad1d1ef5661261dd61db Mon Sep 17 00:00:00 2001
From: Shin'ichiro Kawasaki <[email protected]>
Date: Thu, 24 Sep 2026 19:08:32 +0900
Subject: [PATCH] xfs: try out Darrick's idea for g774 hang

[experimental patch]

Darrick's idea from
https://lore.kernel.org/linux-xfs/20260922002814.GK2705364@frogsfrogsfrogs/

The deadlock of g774 caused by grant space while holding ILOCK. To avoid the deadlock, allocate enough space before taking the ILOCK by increasing tr_logcount. 5 rolls are required for each extent, then multiply tr_logcount by 5.

---
 fs/xfs/xfs_reflink.c | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)
fs/xfs/xfs_reflink.c +25 -1
index 48013613663..9d3da713a01 100644
@@ -1020,6 +1020,28 @@ xfs_reflink_end_cow(
 	return error;
 }
 
+/*
+ * Number of transaction rolls needed to remap a single extent during an atomic
+ * write ioend.
+ */
+#define XFS_ATOMIC_IOEND_ROLLS	5
+
+/*
+ * Compute the number of log operations to reserve for an atomic write ioend
+ * that remaps @blockcount blocks with a @logres byte reservation.
+ */
+STATIC int
+xfs_calc_atomic_write_ioend_logcount(
+	struct xfs_mount	*mp,
+	xfs_extlen_t		blockcount,
+	unsigned int		logres)
+{
+	if (blockcount == 0 || logres == 0)
+		return M_RES(mp)->tr_atomic_ioend.tr_logcount;
+
+	return blockcount * XFS_ATOMIC_IOEND_ROLLS;
+}
+
 /*
  * Fully remap all of the file's data fork at once, which is the critical part
  * in achieving atomic behaviour.
@@ -1036,6 +1058,7 @@ xfs_reflink_end_atomic_cow(
 	xfs_fileoff_t			end_fsb;
 	int				error = 0;
 	struct xfs_mount		*mp = ip->i_mount;
+	struct xfs_trans_res		resv = M_RES(mp)->tr_atomic_ioend;
 	struct xfs_trans		*tp;
 	unsigned int			resblks;
 
@@ -1051,7 +1074,15 @@ xfs_reflink_end_atomic_cow(
 	resblks = (end_fsb - offset_fsb) *
 			XFS_NEXTENTADD_SPACE_RES(mp, 1, XFS_DATA_FORK);
 
-	error = xfs_trans_alloc(mp, &M_RES(mp)->tr_atomic_ioend, resblks, 0,
+	/*
+	 * Reserve enough log operations to remap every block of this write

fs/xfs/xfs_reflink.c xfs_reflink_end_atomic_cow( · line 1088

Quoted 10 lines
fs/xfs/xfs_reflink.c +7 -0
+	 * before we take ILOCK, so that no roll below has to wait for grant
+	 * space while holding ILOCK.
+	 */
+	resv.tr_logcount = xfs_calc_atomic_write_ioend_logcount(mp,
+			end_fsb - offset_fsb, resv.tr_logres);
+
+	error = xfs_trans_alloc(mp, &resv, resblks, 0,
 			XFS_TRANS_RESERVE, &tp);
 	if (error)
 		return error;

Yeah, I think you're on the right track with the dynamic tr_logcount changes but I wonder if you want to constrain the max software atomic write size so that we don't end up with gigantic chains that consume significant portions of the log for a single write?

Admittedly I see the multi-megabyte atomic write limits and marvel at how far we've drifted from the original requirements.

--D

fs/xfs/libxfs/xfs_trans_resv.c +7 -0
index 1c20a7c27aa1bf..2e84544cd22a0a 100644
@@ -1473,6 +1473,10 @@ xfs_calc_max_atomic_write_fsblocks(
        struct xfs_mount                *mp)
 {
        const struct xfs_trans_res      *resv = &M_RES(mp)->tr_atomic_ioend;
+       const unsigned int              max_log_bytes =
+               XFS_FSB_TO_B(mp, mp->m_sb.sb_logblocks * 10 / 4);
+       const unsigned int              log_bytes_per_aw_block =
+               (resv->tr_logres * resv->tr_logcount * 5) / 4;
        unsigned int                    per_intent = 0;
        unsigned int                    step_size = 0;
        unsigned int                    ret = 0;
@@ -1483,6 +1487,9 @@ xfs_calc_max_atomic_write_fsblocks(
 
                if (resv->tr_logres >= step_size)
                        ret = (resv->tr_logres - step_size) / per_intent;
+
+               if (ret > 0 && log_bytes_per_aw_block > max_log_bytes / ret)
+                       ret = max_log_bytes / log_bytes_per_aw_block;
        }
 
        trace_xfs_calc_max_atomic_write_fsblocks(mp, per_intent, step_size,
#19 lore raw
In reply to Shin'ichiro Kawasaki
Quoted 81 lines · Shin'ichiro Kawasaki
Quoted 1 line · Dave Chinner

On Sat, Sep 19, 2026 at 08:53:39PM +0900, Shin'ichiro Kawasaki wrote:

[...]

Quoted 78 lines
Quoted 21 lines
fs/xfs/xfs_file.c +5 -1
index 426a67b813a..8d10ae22884 100644
@@ -829,7 +829,7 @@ xfs_file_dio_write_atomic(
 	struct kiocb		*iocb,
 	struct iov_iter		*from)
 {
-	unsigned int		iolock = XFS_IOLOCK_SHARED;
+	unsigned int		iolock;
 	ssize_t			ret, ocount = iov_iter_count(from);
 	unsigned int		dio_flags = 0;
 	const struct iomap_ops	*dops;
@@ -844,6 +844,17 @@ xfs_file_dio_write_atomic(
 		dops = &xfs_direct_write_iomap_ops;
 
 retry:
+	/*
+	 * Concurrent atomic write COW submissions under ILOCK can run out
+	 * journal reservation resource and can deadlock with the IO
+	 * completions. To avoid the deadlock, serialize submissions of

fs/xfs/xfs_file.c xfs_file_dio_write_atomic( · line 860

Quoted 10 lines
fs/xfs/xfs_file.c +7 -0
+	 * atomic write COW by taking IOLOCK exclusively.
+	 */
+	if (dops == &xfs_atomic_write_cow_iomap_ops)
+		iolock = XFS_IOLOCK_EXCL;
+	else
+		iolock = XFS_IOLOCK_SHARED;
+
 	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
 	if (ret)
 		return ret;

Urk, that's pretty nasty.

fs/xfs/xfs_file.c xfs_file_dio_write_atomic( · line 871

Quoted 10 lines
fs/xfs/xfs_file.c +2 -1
@@ -853,7 +864,8 @@ xfs_file_dio_write_atomic(
 		goto out_unlock;
 
 	/* Demote similar to xfs_file_dio_write_aligned() */
-	if (iolock == XFS_IOLOCK_EXCL) {
+	if (iolock == XFS_IOLOCK_EXCL &&
+	    dops != &xfs_atomic_write_cow_iomap_ops) {
 		xfs_ilock_demote(ip, XFS_IOLOCK_EXCL);
 		iolock = XFS_IOLOCK_SHARED;
 	}

And at this point, we now have several atomic write cow ops specific operations in this function (locking, the retry loop, etc).

This feels much more like there should be a separate function for the software cow path, and the fast path simply calls it on ENOPROTOOPT from the dio submission. That gets rid of all the conditionals and looping from the fast path. i.e

	if (ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
		return xfs_file_dio_write_atomic_cow();

	/* do normal DIO write */

	if (error == -ENOPROTOOPT)
		return xfs_file_dio_write_atomic_cow();
	return error;

That essentially makes xfs_file_dio_write_atomic() and xfs_file_dio_write_aligned() the same code, except for the above two checks, hence they could easily be collapsed back into a common implementation is:

	if ((iocb->ki_flags & IOCB_ATOMIC) &&
	    ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
		return xfs_file_dio_write_atomic_cow();

	/* do normal DIO write */

	if (error == -ENOPROTOOPT)
		return xfs_file_dio_write_atomic_cow();
	return error;
Quoted 8 lines
Quoted 2 lines

That seems like a much more natural breakdown that the current duplication of the DIO write submission code...

Thanks for the comment. I'll try to factor out the duplication based on your idea.

On the other hand, Darrick suggested antoher solution approach to allocate enough space before taking ILOCK by increasing tr_logcount.

No, that doesn't fix anything. tr_logcount is an optimisation to minimise the number of blocking regrants a -typical- rolling transaction will take. It trades off an increase in initial write grant reservation space (i.e. they use more log space) to enable more transaction rolls without needing to refresh the write grant for the next operation in the transaction.

IOWs, it will reduce the number of concurrent transactions that can be running at any given time, but if the deferops intent processing can still run out of pre-reservation space and block waiting for a write grant.

Hence increasing tr_logcount just kicks the can down the road, making it slightly harder to trigger the deadlock at the cost of lowering modification concurrency in the filesystem.

Quoted 4 lines

I wonder which
way is the better: "allocate enough space" or "take exclusive IOLOCK". I'll do some experiment for the "allocate enough space" approach before working on the dupliaction clean up.

"allocate enough space" if not a fix - it's a bandaid. "take exclusive IOLOCK" reflects the fact that COW operations are inherently single threaded due to their reliance on holding the ILOCK_EXCL for long periods of time on both IO submission and IO completion.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#20 lore raw
Quoted 113 lines · Dave Chinner
Quoted 81 lines · Shin'ichiro Kawasaki
Quoted 1 line · Dave Chinner

On Sat, Sep 19, 2026 at 08:53:39PM +0900, Shin'ichiro Kawasaki wrote:

[...]

Quoted 78 lines
Quoted 21 lines
fs/xfs/xfs_file.c +5 -1
index 426a67b813a..8d10ae22884 100644
@@ -829,7 +829,7 @@ xfs_file_dio_write_atomic(
 	struct kiocb		*iocb,
 	struct iov_iter		*from)
 {
-	unsigned int		iolock = XFS_IOLOCK_SHARED;
+	unsigned int		iolock;
 	ssize_t			ret, ocount = iov_iter_count(from);
 	unsigned int		dio_flags = 0;
 	const struct iomap_ops	*dops;
@@ -844,6 +844,17 @@ xfs_file_dio_write_atomic(
 		dops = &xfs_direct_write_iomap_ops;
 
 retry:
+	/*
+	 * Concurrent atomic write COW submissions under ILOCK can run out
+	 * journal reservation resource and can deadlock with the IO
+	 * completions. To avoid the deadlock, serialize submissions of

fs/xfs/xfs_file.c xfs_file_dio_write_atomic( · line 860

Quoted 10 lines
fs/xfs/xfs_file.c +7 -0
+	 * atomic write COW by taking IOLOCK exclusively.
+	 */
+	if (dops == &xfs_atomic_write_cow_iomap_ops)
+		iolock = XFS_IOLOCK_EXCL;
+	else
+		iolock = XFS_IOLOCK_SHARED;
+
 	ret = xfs_ilock_iocb_for_write(iocb, &iolock);
 	if (ret)
 		return ret;

Urk, that's pretty nasty.

fs/xfs/xfs_file.c xfs_file_dio_write_atomic( · line 871

Quoted 10 lines
fs/xfs/xfs_file.c +2 -1
@@ -853,7 +864,8 @@ xfs_file_dio_write_atomic(
 		goto out_unlock;
 
 	/* Demote similar to xfs_file_dio_write_aligned() */
-	if (iolock == XFS_IOLOCK_EXCL) {
+	if (iolock == XFS_IOLOCK_EXCL &&
+	    dops != &xfs_atomic_write_cow_iomap_ops) {
 		xfs_ilock_demote(ip, XFS_IOLOCK_EXCL);
 		iolock = XFS_IOLOCK_SHARED;
 	}

And at this point, we now have several atomic write cow ops specific operations in this function (locking, the retry loop, etc).

This feels much more like there should be a separate function for the software cow path, and the fast path simply calls it on ENOPROTOOPT from the dio submission. That gets rid of all the conditionals and looping from the fast path. i.e

	if (ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
		return xfs_file_dio_write_atomic_cow();

	/* do normal DIO write */

	if (error == -ENOPROTOOPT)
		return xfs_file_dio_write_atomic_cow();
	return error;

That essentially makes xfs_file_dio_write_atomic() and xfs_file_dio_write_aligned() the same code, except for the above two checks, hence they could easily be collapsed back into a common implementation is:

	if ((iocb->ki_flags & IOCB_ATOMIC) &&
	    ocount > xfs_inode_buftarg(ip)->bt_awu_maxocount)
		return xfs_file_dio_write_atomic_cow();

	/* do normal DIO write */

	if (error == -ENOPROTOOPT)
		return xfs_file_dio_write_atomic_cow();
	return error;
Quoted 8 lines
Quoted 2 lines

That seems like a much more natural breakdown that the current duplication of the DIO write submission code...

Thanks for the comment. I'll try to factor out the duplication based on your idea.

On the other hand, Darrick suggested antoher solution approach to allocate enough space before taking ILOCK by increasing tr_logcount.

No, that doesn't fix anything. tr_logcount is an optimisation to minimise the number of blocking regrants a -typical- rolling transaction will take. It trades off an increase in initial write grant reservation space (i.e. they use more log space) to enable more transaction rolls without needing to refresh the write grant for the next operation in the transaction.

IOWs, it will reduce the number of concurrent transactions that can be running at any given time, but if the deferops intent processing can still run out of pre-reservation space and block waiting for a write grant.

Hence increasing tr_logcount just kicks the can down the road, making it slightly harder to trigger the deadlock at the cost of lowering modification concurrency in the filesystem.

Quoted 4 lines

I wonder which
way is the better: "allocate enough space" or "take exclusive IOLOCK". I'll do some experiment for the "allocate enough space" approach before working on the dupliaction clean up.

Quoted 5 lines

"allocate enough space" if not a fix - it's a bandaid. "take exclusive IOLOCK" reflects the fact that COW operations are inherently single threaded due to their reliance on holding the ILOCK_EXCL for long periods of time on both IO submission and IO completion.

Wellp, I'll let you and your LLM toy figure out the correct fix for this, then.

--D

#24 lore raw
In reply to Darrick J. Wong

On Sep 25, 2026 / 16:03, Darrick J. Wong wrote:
[...]

Quoted 8 lines

Yeah, I think you're on the right track with the dynamic tr_logcount changes but I wonder if you want to constrain the max software atomic write size so that we don't end up with gigantic chains that consume significant portions of the log for a single write?

Admittedly I see the multi-megabyte atomic write limits and marvel at how far we've drifted from the original requirements.

Darrick, thank you for the comments, and sorry for my slow response. Today, I found some time slot to try out max atomic write bytes capping. I took a look in your code snippet below, and was scratching my head.

Quoted 8 lines

--D

fs/xfs/libxfs/xfs_trans_resv.c +0 -0
index 1c20a7c27aa1bf..2e84544cd22a0a 100644

fs/xfs/libxfs/xfs_trans_resv.c xfs_calc_max_atomic_write_fsblocks( · line 1479

Quoted 7 lines
fs/xfs/libxfs/xfs_trans_resv.c +4 -0
@@ -1473,6 +1473,10 @@ xfs_calc_max_atomic_write_fsblocks(
        struct xfs_mount                *mp)
 {
        const struct xfs_trans_res      *resv = &M_RES(mp)->tr_atomic_ioend;
+       const unsigned int              max_log_bytes =
+               XFS_FSB_TO_B(mp, mp->m_sb.sb_logblocks * 10 / 4);
+       const unsigned int              log_bytes_per_aw_block =
+               (resv->tr_logres * resv->tr_logcount * 5) / 4;

I wonder what the numbers in the two lines above mean. I think 5 means the number of transaction rolls required for an atomic write. But not sure about 10 and 4. May I ask you to elaborate them?

Anyway, I applied this patch, and confirmed that awu_max reduced from 1024 to 64 in my test setup (4MiB to 256KiB). So, it looks working as expected.