Skip to contentSkip to conversation

Strange XFS problem

DISCUSSION Troels Hansen 3 participants 7 messages Sep 12, 2018 – Sep 13, 2018 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hi, we are facing an issue where we can't figure out if its XFS software related, or actually related to hardware, and can't quite figure out why we are facing the issues, though is doesn't seem hardware related.

The issue is with a 102Tb array on a Dell branded LSISAS 3508 (Perc H840).
Running Ubuntu with a 4.15.0-32 (Ubuntu branded), but we have also been running a number of 4.4.0-x with the same issues.

The XFS filsusyem is on a very busy NFS server, and when the issue occurs we see strange issues with NFS, while the system seems healthy on the local server, but at the same time some programs are having problems accessing the fs.

It occure roughly every 14 days, where we have to restart the fs to come back fully working. Sometimes refusing to unmount cleanly during shutdown, forcing us to fsck the fs on startup.

It looks like its hanging in xlog_grant_head_wait, but I don't know enough to determine what can make it hang there.

Hoping someone in here could have a look and point me in the right direction.

Below is a trace from the last crash we had:

Sep  9 23:23:51 ged kernel: [1436769.178935] INFO: task mysqld:2847 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.178999]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.179047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.179105] mysqld          D    0  2847      1 0x00000000
Sep  9 23:23:51 ged kernel: [1436769.179111] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.179123]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.179127]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.179216]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179277]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179330]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179381]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179428]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179478]  xfs_vn_update_time+0x5d/0x160 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179486]  file_update_time+0xbe/0x110
Sep  9 23:23:51 ged kernel: [1436769.179493]  ? tcp_recvmsg+0x317/0xab0
Sep  9 23:23:51 ged kernel: [1436769.179542]  xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179588]  xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179632]  xfs_file_write_iter+0x103/0x150 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179637]  new_sync_write+0xe5/0x140
Sep  9 23:23:51 ged kernel: [1436769.179641]  __vfs_write+0x29/0x40
Sep  9 23:23:51 ged kernel: [1436769.179645]  vfs_write+0xb8/0x1b0
Sep  9 23:23:51 ged kernel: [1436769.179649]  SyS_pwrite64+0x95/0xb0
Sep  9 23:23:51 ged kernel: [1436769.179655]  do_syscall_64+0x73/0x130
Sep  9 23:23:51 ged kernel: [1436769.179661]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Sep  9 23:23:51 ged kernel: [1436769.179664] RIP: 0033:0x7f16be838da3
Sep  9 23:23:51 ged kernel: [1436769.179667] RSP: 002b:00007f148c31b950 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
Sep  9 23:23:51 ged kernel: [1436769.179671] RAX: ffffffffffffffda RBX: 00007f148c31ba50 RCX: 00007f16be838da3
Sep  9 23:23:51 ged kernel: [1436769.179673] RDX: 0000000000000200 RSI: 00007f149ea80200 RDI: 0000000000000008
Sep  9 23:23:51 ged kernel: [1436769.179675] RBP: 00007f148c31bc00 R08: 000000000019d200 R09: 00007f148c31bc3c
Sep  9 23:23:51 ged kernel: [1436769.179677] R10: 000000000019d200 R11: 0000000000000293 R12: 0000000000000200
Sep  9 23:23:51 ged kernel: [1436769.179679] R13: 000000000019d200 R14: 0000000000000000 R15: 0000000000000000
Sep  9 23:23:51 ged kernel: [1436769.179686] INFO: task nfsd:1937 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.179738]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.179784] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.179842] nfsd            D    0  1937      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.179846] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.179852]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.179857]  ? sch_direct_xmit+0x73/0x1b0
Sep  9 23:23:51 ged kernel: [1436769.179860]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.179910]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179956]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180005]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180051]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180096]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180145]  xfs_vn_update_time+0x5d/0x160 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180150]  file_update_time+0xbe/0x110
Sep  9 23:23:51 ged kernel: [1436769.180199]  xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180244]  xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180248]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.180291]  xfs_file_write_iter+0x103/0x150 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180295]  do_iter_readv_writev+0x111/0x180
Sep  9 23:23:51 ged kernel: [1436769.180299]  do_iter_write+0x87/0x1a0
Sep  9 23:23:51 ged kernel: [1436769.180302]  vfs_iter_write+0x19/0x30
Sep  9 23:23:51 ged kernel: [1436769.180321]  nfsd_vfs_write+0xbe/0x380 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.180338]  nfsd4_write+0x16c/0x1a0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.180352]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.180365]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.180395]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.180416]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.180428]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.180434]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.180445]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.180451]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.180456]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.180461] INFO: task nfsd:1938 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.180511]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.180557] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.180615] nfsd            D    0  1938      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.180619] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.180623]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.180628]  ? sch_direct_xmit+0x73/0x1b0
Sep  9 23:23:51 ged kernel: [1436769.180631]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.180689]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180737]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180786]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180832]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180876]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180926]  xfs_vn_update_time+0x5d/0x160 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180931]  file_update_time+0xbe/0x110
Sep  9 23:23:51 ged kernel: [1436769.180980]  xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181025]  xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181029]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.181073]  xfs_file_write_iter+0x103/0x150 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181077]  do_iter_readv_writev+0x111/0x180
Sep  9 23:23:51 ged kernel: [1436769.181080]  do_iter_write+0x87/0x1a0
Sep  9 23:23:51 ged kernel: [1436769.181083]  vfs_iter_write+0x19/0x30
Sep  9 23:23:51 ged kernel: [1436769.181101]  nfsd_vfs_write+0xbe/0x380 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181117]  nfsd4_write+0x16c/0x1a0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181131]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181143]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181169]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.181189]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.181200]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181206]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.181217]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181222]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.181228]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.181233] INFO: task nfsd:1939 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.181282]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.181328] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.181385] nfsd            D    0  1939      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.181389] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.181394]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.181398]  ? sch_direct_xmit+0x73/0x1b0
Sep  9 23:23:51 ged kernel: [1436769.181401]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.181455]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181503]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181551]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181599]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181643]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181692]  xfs_vn_update_time+0x5d/0x160 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181698]  file_update_time+0xbe/0x110
Sep  9 23:23:51 ged kernel: [1436769.181747]  xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181792]  xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181796]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.181839]  xfs_file_write_iter+0x103/0x150 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181843]  do_iter_readv_writev+0x111/0x180
Sep  9 23:23:51 ged kernel: [1436769.181846]  do_iter_write+0x87/0x1a0
Sep  9 23:23:51 ged kernel: [1436769.181849]  vfs_iter_write+0x19/0x30
Sep  9 23:23:51 ged kernel: [1436769.181864]  nfsd_vfs_write+0xbe/0x380 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181880]  nfsd4_write+0x16c/0x1a0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181894]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181906]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181930]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.181951]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.181962]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181967]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.181978]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181984]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.181988]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.181993] INFO: task nfsd:1940 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.182041]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.182088] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.182146] nfsd            D    0  1940      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.182149] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.182153]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.182158]  ? __switch_to_asm+0x40/0x70
Sep  9 23:23:51 ged kernel: [1436769.182162]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.182213]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182259]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182308]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182354]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182398]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182448]  xfs_setattr_nonsize+0x242/0x5a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182494]  xfs_vn_setattr_nonsize+0x42/0x90 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182539]  xfs_vn_setattr+0x66/0xa0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182544]  notify_change+0x2e1/0x410
Sep  9 23:23:51 ged kernel: [1436769.182558]  nfsd_setattr+0x390/0x4f0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.182562]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.182577]  nfsd4_setattr+0x10e/0x140 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.182591]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.182603]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.182626]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.182646]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.182657]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.182662]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.182673]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.182710]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.182715]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.182720] INFO: task nfsd:1941 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.182771]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.182820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.182884] nfsd            D    0  1941      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.182897] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.182905]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.182914]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.182972]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183022]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183075]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183126]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183174]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183229]  xfs_create+0x1bf/0x6b0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183239]  ? get_acl+0x1d/0x100
Sep  9 23:23:51 ged kernel: [1436769.183248]  ? posix_acl_create+0x109/0x160
Sep  9 23:23:51 ged kernel: [1436769.183300]  xfs_generic_create+0x21a/0x2f0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183350]  xfs_vn_mknod+0x14/0x20 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183396]  xfs_vn_create+0x13/0x20 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183405]  vfs_create+0x12d/0x190
Sep  9 23:23:51 ged kernel: [1436769.183424]  do_nfsd_create+0x43f/0x580 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.183446]  nfsd4_open+0x20c/0x730 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.183466]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.183482]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.183509]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.183535]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.183553]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.183562]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.183578]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.183588]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.183596]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.183607] INFO: task nfsd:1942 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.183657]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.183703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.183760] nfsd            D    0  1942      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.183763] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.183768]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.183772]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.183823]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183870]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183917]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183964]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184008]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184056]  xfs_setattr_nonsize+0x242/0x5a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184077]  ? nfsd_setuser+0x11f/0x2b0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184127]  xfs_vn_setattr_nonsize+0x42/0x90 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184174]  xfs_vn_setattr+0x66/0xa0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184184]  ? current_time+0x32/0x70
Sep  9 23:23:51 ged kernel: [1436769.184194]  notify_change+0x2e1/0x410
Sep  9 23:23:51 ged kernel: [1436769.184212]  nfsd_setattr+0x390/0x4f0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184222]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.184241]  nfsd4_setattr+0x10e/0x140 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184260]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184278]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184306]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.184330]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.184345]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184356]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.184374]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184384]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.184395]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.184406] INFO: task nfsd:1943 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.184460]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.184508] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.184565] nfsd            D    0  1943      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.184568] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.184572]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.184577]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.184628]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184673]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184721]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184768]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184812]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184867]  xfs_setattr_nonsize+0x242/0x5a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184918]  xfs_vn_setattr_nonsize+0x42/0x90 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184966]  xfs_vn_setattr+0x66/0xa0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184977]  notify_change+0x2e1/0x410
Sep  9 23:23:51 ged kernel: [1436769.184997]  nfsd_setattr+0x390/0x4f0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185004]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.185020]  nfsd4_setattr+0x10e/0x140 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185034]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185046]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185074]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.185098]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.185113]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185119]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.185135]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185143]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.185151]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.185156] INFO: task nfsd:1944 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.185205]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.185251] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.185308] nfsd            D    0  1944      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.185311] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.185315]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.185321]  ? __slab_free+0xb2/0x2f0
Sep  9 23:23:51 ged kernel: [1436769.185324]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.185374]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185420]  xlog_grant_head_check+0xf1/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185467]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185513]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185556]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185606]  xfs_rename+0x133/0xab0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185610]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.185614]  ? sort+0x165/0x250
Sep  9 23:23:51 ged kernel: [1436769.185618]  ? iter_div_u64_rem+0x20/0x20
Sep  9 23:23:51 ged kernel: [1436769.185665]  xfs_vn_rename+0xd4/0x150 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185670]  vfs_rename+0x6a5/0x8c0
Sep  9 23:23:51 ged kernel: [1436769.185675]  ? lookup_one_len+0xf5/0x120
Sep  9 23:23:51 ged kernel: [1436769.185689]  nfsd_rename+0x359/0x3a0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185705]  nfsd4_rename+0x6e/0x190 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185719]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185731]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185753]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.185773]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.185785]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185790]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.185801]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185807]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.185812]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.185830] INFO: task bacula-fd:24743 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.185881]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.185927] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.185983] bacula-fd       D    0 24743      1 0x00000000
Sep  9 23:23:51 ged kernel: [1436769.185987] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.185991]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.185995]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.186045]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186090]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186136]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186182]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186225]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186274]  xfs_vn_update_time+0x5d/0x160 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186279]  touch_atime+0xb2/0xe0
Sep  9 23:23:51 ged kernel: [1436769.186286]  generic_file_read_iter+0x93a/0xba0
Sep  9 23:23:51 ged kernel: [1436769.186292]  ? page_cache_tree_insert+0xe0/0xe0
Sep  9 23:23:51 ged kernel: [1436769.186341]  xfs_file_buffered_aio_read+0x5d/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186385]  xfs_file_read_iter+0x68/0xc0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186389]  new_sync_read+0xe2/0x130
Sep  9 23:23:51 ged kernel: [1436769.186393]  __vfs_read+0x29/0x40
Sep  9 23:23:51 ged kernel: [1436769.186396]  vfs_read+0x93/0x130
Sep  9 23:23:51 ged kernel: [1436769.186399]  SyS_read+0x55/0xc0
Sep  9 23:23:51 ged kernel: [1436769.186405]  do_syscall_64+0x73/0x130
Sep  9 23:23:51 ged kernel: [1436769.186409]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Sep  9 23:23:51 ged kernel: [1436769.186412] RIP: 0033:0x7f919219b51d
Sep  9 23:23:51 ged kernel: [1436769.186414] RSP: 002b:00007f918d90c9a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
Sep  9 23:23:51 ged kernel: [1436769.186418] RAX: ffffffffffffffda RBX: 00007f917c00b1a0 RCX: 00007f919219b51d
Sep  9 23:23:51 ged kernel: [1436769.186420] RDX: 0000000000010000 RSI: 00007f917c00c8c0 RDI: 000000000000000e
Sep  9 23:23:51 ged kernel: [1436769.186422] RBP: 00007f917e12de68 R08: 000000000000000a R09: 0000000000000000
Sep  9 23:23:51 ged kernel: [1436769.186424] R10: 0000000000000000 R11: 0000000000000293 R12: 00007f917c096308
Sep  9 23:23:51 ged kernel: [1436769.186426] R13: 00007f917c00bfa8 R14: 00007f917c00b1a0 R15: 00007f917e12de68
#1 lore raw
Quoted 7 lines · Troels Hansen

Hi, we are facing an issue where we can't figure out if its XFS software related, or actually related to hardware, and can't quite figure out why we are facing the issues, though is doesn't seem hardware related.

The issue is with a 102Tb array on a Dell branded LSISAS 3508 (Perc H840).
Running Ubuntu with a 4.15.0-32 (Ubuntu branded), but we have also been running a number of 4.4.0-x with the same issues.

The XFS filsusyem is on a very busy NFS server, and when the issue occurs we see strange issues with NFS, while the system seems healthy on the local server, but at the same time some programs are having problems accessing the fs.

Quoted 8 lines

It occure roughly every 14 days, where we have to restart the fs to come back fully working. Sometimes refusing to unmount cleanly during shutdown, forcing us to fsck the fs on startup.

It looks like its hanging in xlog_grant_head_wait, but I don't know enough to determine what can make it hang there.

Hoping someone in here could have a look and point me in the right direction.

Below is a trace from the last crash we had:

It's a long shot, but can't do much without more information. At first, your filesystem is waiting for log space, so it could be your log is too small, your storage is too slow and/or misconfigured, or maybe you you've been hitting a bug somewhere!?

I don't remember any bug regarding log space issues or reservation problems happening recently, but I don't use to work on log reservation issues by now, so, maybe other can have more information if this might be any bug or not.

In the mean time, can you provide more information about the systems where you are hitting this issue?

Mainle the storage configuration and the xfs_info output of the affected filesystems.

FWIW:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

Cheers

Quoted 312 lines

Sep  9 23:23:51 ged kernel: [1436769.178935] INFO: task mysqld:2847 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.178999]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.179047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.179105] mysqld          D    0  2847      1 0x00000000
Sep  9 23:23:51 ged kernel: [1436769.179111] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.179123]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.179127]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.179216]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179277]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179330]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179381]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179428]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179478]  xfs_vn_update_time+0x5d/0x160 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179486]  file_update_time+0xbe/0x110
Sep  9 23:23:51 ged kernel: [1436769.179493]  ? tcp_recvmsg+0x317/0xab0
Sep  9 23:23:51 ged kernel: [1436769.179542]  xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179588]  xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179632]  xfs_file_write_iter+0x103/0x150 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179637]  new_sync_write+0xe5/0x140
Sep  9 23:23:51 ged kernel: [1436769.179641]  __vfs_write+0x29/0x40
Sep  9 23:23:51 ged kernel: [1436769.179645]  vfs_write+0xb8/0x1b0
Sep  9 23:23:51 ged kernel: [1436769.179649]  SyS_pwrite64+0x95/0xb0
Sep  9 23:23:51 ged kernel: [1436769.179655]  do_syscall_64+0x73/0x130
Sep  9 23:23:51 ged kernel: [1436769.179661]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Sep  9 23:23:51 ged kernel: [1436769.179664] RIP: 0033:0x7f16be838da3
Sep  9 23:23:51 ged kernel: [1436769.179667] RSP: 002b:00007f148c31b950 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
Sep  9 23:23:51 ged kernel: [1436769.179671] RAX: ffffffffffffffda RBX: 00007f148c31ba50 RCX: 00007f16be838da3
Sep  9 23:23:51 ged kernel: [1436769.179673] RDX: 0000000000000200 RSI: 00007f149ea80200 RDI: 0000000000000008
Sep  9 23:23:51 ged kernel: [1436769.179675] RBP: 00007f148c31bc00 R08: 000000000019d200 R09: 00007f148c31bc3c
Sep  9 23:23:51 ged kernel: [1436769.179677] R10: 000000000019d200 R11: 0000000000000293 R12: 0000000000000200
Sep  9 23:23:51 ged kernel: [1436769.179679] R13: 000000000019d200 R14: 0000000000000000 R15: 0000000000000000
Sep  9 23:23:51 ged kernel: [1436769.179686] INFO: task nfsd:1937 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.179738]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.179784] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.179842] nfsd            D    0  1937      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.179846] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.179852]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.179857]  ? sch_direct_xmit+0x73/0x1b0
Sep  9 23:23:51 ged kernel: [1436769.179860]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.179910]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179956]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180005]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180051]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180096]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180145]  xfs_vn_update_time+0x5d/0x160 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180150]  file_update_time+0xbe/0x110
Sep  9 23:23:51 ged kernel: [1436769.180199]  xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180244]  xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180248]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.180291]  xfs_file_write_iter+0x103/0x150 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180295]  do_iter_readv_writev+0x111/0x180
Sep  9 23:23:51 ged kernel: [1436769.180299]  do_iter_write+0x87/0x1a0
Sep  9 23:23:51 ged kernel: [1436769.180302]  vfs_iter_write+0x19/0x30
Sep  9 23:23:51 ged kernel: [1436769.180321]  nfsd_vfs_write+0xbe/0x380 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.180338]  nfsd4_write+0x16c/0x1a0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.180352]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.180365]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.180395]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.180416]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.180428]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.180434]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.180445]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.180451]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.180456]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.180461] INFO: task nfsd:1938 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.180511]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.180557] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.180615] nfsd            D    0  1938      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.180619] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.180623]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.180628]  ? sch_direct_xmit+0x73/0x1b0
Sep  9 23:23:51 ged kernel: [1436769.180631]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.180689]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180737]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180786]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180832]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180876]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180926]  xfs_vn_update_time+0x5d/0x160 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.180931]  file_update_time+0xbe/0x110
Sep  9 23:23:51 ged kernel: [1436769.180980]  xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181025]  xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181029]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.181073]  xfs_file_write_iter+0x103/0x150 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181077]  do_iter_readv_writev+0x111/0x180
Sep  9 23:23:51 ged kernel: [1436769.181080]  do_iter_write+0x87/0x1a0
Sep  9 23:23:51 ged kernel: [1436769.181083]  vfs_iter_write+0x19/0x30
Sep  9 23:23:51 ged kernel: [1436769.181101]  nfsd_vfs_write+0xbe/0x380 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181117]  nfsd4_write+0x16c/0x1a0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181131]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181143]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181169]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.181189]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.181200]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181206]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.181217]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181222]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.181228]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.181233] INFO: task nfsd:1939 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.181282]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.181328] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.181385] nfsd            D    0  1939      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.181389] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.181394]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.181398]  ? sch_direct_xmit+0x73/0x1b0
Sep  9 23:23:51 ged kernel: [1436769.181401]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.181455]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181503]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181551]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181599]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181643]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181692]  xfs_vn_update_time+0x5d/0x160 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181698]  file_update_time+0xbe/0x110
Sep  9 23:23:51 ged kernel: [1436769.181747]  xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181792]  xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181796]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.181839]  xfs_file_write_iter+0x103/0x150 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.181843]  do_iter_readv_writev+0x111/0x180
Sep  9 23:23:51 ged kernel: [1436769.181846]  do_iter_write+0x87/0x1a0
Sep  9 23:23:51 ged kernel: [1436769.181849]  vfs_iter_write+0x19/0x30
Sep  9 23:23:51 ged kernel: [1436769.181864]  nfsd_vfs_write+0xbe/0x380 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181880]  nfsd4_write+0x16c/0x1a0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181894]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181906]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181930]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.181951]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.181962]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181967]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.181978]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.181984]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.181988]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.181993] INFO: task nfsd:1940 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.182041]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.182088] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.182146] nfsd            D    0  1940      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.182149] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.182153]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.182158]  ? __switch_to_asm+0x40/0x70
Sep  9 23:23:51 ged kernel: [1436769.182162]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.182213]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182259]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182308]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182354]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182398]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182448]  xfs_setattr_nonsize+0x242/0x5a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182494]  xfs_vn_setattr_nonsize+0x42/0x90 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182539]  xfs_vn_setattr+0x66/0xa0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.182544]  notify_change+0x2e1/0x410
Sep  9 23:23:51 ged kernel: [1436769.182558]  nfsd_setattr+0x390/0x4f0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.182562]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.182577]  nfsd4_setattr+0x10e/0x140 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.182591]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.182603]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.182626]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.182646]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.182657]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.182662]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.182673]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.182710]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.182715]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.182720] INFO: task nfsd:1941 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.182771]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.182820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.182884] nfsd            D    0  1941      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.182897] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.182905]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.182914]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.182972]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183022]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183075]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183126]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183174]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183229]  xfs_create+0x1bf/0x6b0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183239]  ? get_acl+0x1d/0x100
Sep  9 23:23:51 ged kernel: [1436769.183248]  ? posix_acl_create+0x109/0x160
Sep  9 23:23:51 ged kernel: [1436769.183300]  xfs_generic_create+0x21a/0x2f0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183350]  xfs_vn_mknod+0x14/0x20 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183396]  xfs_vn_create+0x13/0x20 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183405]  vfs_create+0x12d/0x190
Sep  9 23:23:51 ged kernel: [1436769.183424]  do_nfsd_create+0x43f/0x580 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.183446]  nfsd4_open+0x20c/0x730 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.183466]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.183482]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.183509]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.183535]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.183553]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.183562]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.183578]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.183588]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.183596]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.183607] INFO: task nfsd:1942 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.183657]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.183703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.183760] nfsd            D    0  1942      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.183763] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.183768]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.183772]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.183823]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183870]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183917]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.183964]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184008]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184056]  xfs_setattr_nonsize+0x242/0x5a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184077]  ? nfsd_setuser+0x11f/0x2b0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184127]  xfs_vn_setattr_nonsize+0x42/0x90 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184174]  xfs_vn_setattr+0x66/0xa0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184184]  ? current_time+0x32/0x70
Sep  9 23:23:51 ged kernel: [1436769.184194]  notify_change+0x2e1/0x410
Sep  9 23:23:51 ged kernel: [1436769.184212]  nfsd_setattr+0x390/0x4f0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184222]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.184241]  nfsd4_setattr+0x10e/0x140 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184260]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184278]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184306]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.184330]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.184345]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184356]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.184374]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.184384]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.184395]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.184406] INFO: task nfsd:1943 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.184460]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.184508] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.184565] nfsd            D    0  1943      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.184568] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.184572]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.184577]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.184628]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184673]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184721]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184768]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184812]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184867]  xfs_setattr_nonsize+0x242/0x5a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184918]  xfs_vn_setattr_nonsize+0x42/0x90 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184966]  xfs_vn_setattr+0x66/0xa0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.184977]  notify_change+0x2e1/0x410
Sep  9 23:23:51 ged kernel: [1436769.184997]  nfsd_setattr+0x390/0x4f0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185004]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.185020]  nfsd4_setattr+0x10e/0x140 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185034]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185046]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185074]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.185098]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.185113]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185119]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.185135]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185143]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.185151]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.185156] INFO: task nfsd:1944 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.185205]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.185251] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.185308] nfsd            D    0  1944      2 0x80000000
Sep  9 23:23:51 ged kernel: [1436769.185311] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.185315]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.185321]  ? __slab_free+0xb2/0x2f0
Sep  9 23:23:51 ged kernel: [1436769.185324]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.185374]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185420]  xlog_grant_head_check+0xf1/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185467]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185513]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185556]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185606]  xfs_rename+0x133/0xab0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185610]  ? _cond_resched+0x1a/0x50
Sep  9 23:23:51 ged kernel: [1436769.185614]  ? sort+0x165/0x250
Sep  9 23:23:51 ged kernel: [1436769.185618]  ? iter_div_u64_rem+0x20/0x20
Sep  9 23:23:51 ged kernel: [1436769.185665]  xfs_vn_rename+0xd4/0x150 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.185670]  vfs_rename+0x6a5/0x8c0
Sep  9 23:23:51 ged kernel: [1436769.185675]  ? lookup_one_len+0xf5/0x120
Sep  9 23:23:51 ged kernel: [1436769.185689]  nfsd_rename+0x359/0x3a0 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185705]  nfsd4_rename+0x6e/0x190 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185719]  nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185731]  nfsd_dispatch+0xbb/0x250 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185753]  svc_process_common+0x380/0x690 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.185773]  svc_process+0xfa/0x1c0 [sunrpc]
Sep  9 23:23:51 ged kernel: [1436769.185785]  nfsd+0xe9/0x150 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185790]  kthread+0x105/0x140
Sep  9 23:23:51 ged kernel: [1436769.185801]  ? nfsd_destroy+0x60/0x60 [nfsd]
Sep  9 23:23:51 ged kernel: [1436769.185807]  ? kthread_destroy_worker+0x50/0x50
Sep  9 23:23:51 ged kernel: [1436769.185812]  ret_from_fork+0x35/0x40
Sep  9 23:23:51 ged kernel: [1436769.185830] INFO: task bacula-fd:24743 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.185881]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.185927] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.185983] bacula-fd       D    0 24743      1 0x00000000
Sep  9 23:23:51 ged kernel: [1436769.185987] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.185991]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.185995]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.186045]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186090]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186136]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186182]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186225]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186274]  xfs_vn_update_time+0x5d/0x160 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186279]  touch_atime+0xb2/0xe0
Sep  9 23:23:51 ged kernel: [1436769.186286]  generic_file_read_iter+0x93a/0xba0
Sep  9 23:23:51 ged kernel: [1436769.186292]  ? page_cache_tree_insert+0xe0/0xe0
Sep  9 23:23:51 ged kernel: [1436769.186341]  xfs_file_buffered_aio_read+0x5d/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186385]  xfs_file_read_iter+0x68/0xc0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.186389]  new_sync_read+0xe2/0x130
Sep  9 23:23:51 ged kernel: [1436769.186393]  __vfs_read+0x29/0x40
Sep  9 23:23:51 ged kernel: [1436769.186396]  vfs_read+0x93/0x130
Sep  9 23:23:51 ged kernel: [1436769.186399]  SyS_read+0x55/0xc0
Sep  9 23:23:51 ged kernel: [1436769.186405]  do_syscall_64+0x73/0x130
Sep  9 23:23:51 ged kernel: [1436769.186409]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Sep  9 23:23:51 ged kernel: [1436769.186412] RIP: 0033:0x7f919219b51d
Sep  9 23:23:51 ged kernel: [1436769.186414] RSP: 002b:00007f918d90c9a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
Sep  9 23:23:51 ged kernel: [1436769.186418] RAX: ffffffffffffffda RBX: 00007f917c00b1a0 RCX: 00007f919219b51d
Sep  9 23:23:51 ged kernel: [1436769.186420] RDX: 0000000000010000 RSI: 00007f917c00c8c0 RDI: 000000000000000e
Sep  9 23:23:51 ged kernel: [1436769.186422] RBP: 00007f917e12de68 R08: 000000000000000a R09: 0000000000000000
Sep  9 23:23:51 ged kernel: [1436769.186424] R10: 0000000000000000 R11: 0000000000000293 R12: 00007f917c096308
Sep  9 23:23:51 ged kernel: [1436769.186426] R13: 00007f917c00bfa8 R14: 00007f917c00b1a0 R15: 00007f917e12de68
Signature · 1 line
Carlos
#2 lore raw

Hi Carlos

Quoted 8 lines · ----- On Sep 12, 2018, at 12:59 PM, Carlos Maiolino [email protected] wrote:

In the mean time, can you provide more information about the systems where you are hitting this issue?

Mainle the storage configuration and the xfs_info output of the affected filesystems.

FWIW:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

$ uname -a
Linux ged 4.15.0-32-generic #35~16.04.1-Ubuntu SMP Fri Aug 10 21:54:34 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ sudo xfs_info /z/home
meta-data=/dev/sda               isize=256    agcount=102, agsize=268435328 blks
         =                       sectsz=512   attr=2, projid32bit=0
         =                       crc=0        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=27345289216, imaxpct=1
         =                       sunit=128    swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
$ xfs_repair -V
xfs_repair version 4.3.0

CPU:
2 x Xeon(R) Gold 6142, total 32cores, 64 threads.

$ cat /proc/meminfo 
MemTotal:       1583995652 kB
MemFree:        143862972 kB
MemAvailable:   1509722704 kB
Buffers:          388972 kB
Cached:         1237429160 kB
SwapCached:            0 kB
Active:         1026769972 kB
Inactive:       218745456 kB
Active(anon):    7542528 kB
Inactive(anon):   274984 kB
Active(file):   1019227444 kB
Inactive(file): 218470472 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       9765372 kB
SwapFree:        9765372 kB
Dirty:              4976 kB
Writeback:             0 kB
AnonPages:       7696592 kB
Mapped:            90216 kB
Shmem:            120212 kB
Slab:           144289884 kB
SReclaimable:   136424256 kB
SUnreclaim:      7865628 kB
KernelStack:       16176 kB
PageTables:        25844 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    801763196 kB
Committed_AS:   12122188 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     1826624 kB
DirectMap2M:    224344064 kB
DirectMap1G:    1385168896 kB
$ cat /proc/mounts
.....
/dev/sda /z/home xfs rw,relatime,attr2,inode64,sunit=1024,swidth=2048,noquota 0 0
$ cat /proc/partitions 
major minor  #blocks  name
   8        0 109381156864 sda
   8       16 39048970240 sdb
   8       32  585498624 sdc
   8       33    9765376 sdc1
   8       34          1 sdc2
   8       37  575732480 sdc5
  11        0    1048575 sr0

Hardware raid on Dell PERC H840

Raid50, 8 disk per span, 4 spans.
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 101.869 TB
Sector Size : 512
Parity Size : 14.552 TB
State : Optimal
Strip Size : 512 KB
Number Of Drives per span:8
Span Depth : 4

6Gb cache on controller, battery backed up.

Disks are 4Tb Toshiba 12G SAS disks.

Not running LVM.

A message that is not in the archive (sent off-list or not yet received): notmuch-sha1-960c954e5404b5b2f083d150633af0b7848ec14c

#4 lore raw

I think it's quite clear I hit 'send' by mistake :)

Quoted 55 lines · Carlos Maiolino

Hi,

Quoted 19 lines · Troels Hansen

Hi Carlos

Quoted 8 lines · ----- On Sep 12, 2018, at 12:59 PM, Carlos Maiolino [email protected] wrote:

In the mean time, can you provide more information about the systems where you are hitting this issue?

Mainle the storage configuration and the xfs_info output of the affected filesystems.

FWIW:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

$ uname -a
Linux ged 4.15.0-32-generic #35~16.04.1-Ubuntu SMP Fri Aug 10 21:54:34 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Quoted 11 lines
$ sudo xfs_info /z/home
meta-data=/dev/sda               isize=256    agcount=102, agsize=268435328 blks
         =                       sectsz=512   attr=2, projid32bit=0
         =                       crc=0        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=27345289216, imaxpct=1
         =                       sunit=128    swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Ok, the settings look reasonable for the size of the filesystem, agcount and log size looks sane, the stripe configuration doesn't though, based on your raid configuration. And you're using V4 filesystem, which change things a bit.

Quoted 5 lines

$ cat /proc/mounts
.....
/dev/sda /z/home xfs rw,relatime,attr2,inode64,sunit=1024,swidth=2048,noquota 0 0

Mounted with different alignment. 512KiB sunit and a 1MiB width.

Quoted 10 lines

Raid50, 8 disk per span, 4 spans.
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 101.869 TB
Sector Size : 512
Parity Size : 14.552 TB
State : Optimal
Strip Size : 512 KB
Number Of Drives per span:8
Span Depth : 4

Quoted 6 lines

So, if I understand correctly your configuration (I'm not really used to dell namings), you have:

4 RAID5 arrays using 8 disk each, nested in a RAID0 array, which has a 512KiB stripe size.

" If I'm right here, you should be using stripe unit 512K and stripe width
2MiB. (sunit 1024 swidth 4096)"

I didn't have too much time to think about it by now, and I am not sure if setting the correct stripe alignment is the culprit here (most likely not), and since I didn't think about it enough yet, I was postponing the e-mail and hit 'send' instead. My apologies about it.

I need to give it some extra thought to get some idea on how to help you here, or have any reason to ask you to use a newer kernel. Again, my apologies for hitting send too soon.

Quoted 10 lines
Quoted 7 lines

6Gb cache on controller, battery backed up.

Disks are 4Tb Toshiba 12G SAS disks.

Not running LVM.

Signature · 1 line
Carlos
Signature · 1 line
Carlos
#5 lore raw
In reply to Troels Hansen
Quoted 4 lines · Troels Hansen

Hi, we are facing an issue where we can't figure out if its XFS software related, or actually related to hardware, and can't quite figure out why we are facing the issues, though is doesn't seem hardware related.

The issue is with a 102Tb array on a Dell branded LSISAS 3508 (Perc H840).
Running Ubuntu with a 4.15.0-32 (Ubuntu branded), but we have also been running a number of 4.4.0-x with the same issues.

Smells of an IO overload problem from that.

Quoted 6 lines

The XFS filsusyem is on a very busy NFS server, and when the issue occurs we see strange issues with NFS, while the system seems healthy on the local server, but at the same time some programs are having problems accessing the fs.

It occure roughly every 14 days, where we have to restart the fs to come back fully working.

What happens on your network every 14 days or so? Is there a rogue client side backup or admin task running somewhere?

Quoted 1 line

Sometimes refusing to unmount cleanly during shutdown, forcing us to fsck the fs on startup.

Unclean shutdown doesn't require fsck to be run.

Quoted 5 lines

It looks like its hanging in xlog_grant_head_wait, but I don't know enough to determine what can make it hang there.

Hoping someone in here could have a look and point me in the right direction.

Below is a trace from the last crash we had:

Not a crash - it's a hung task warning.

Quoted 16 lines
Sep  9 23:23:51 ged kernel: [1436769.178935] INFO: task mysqld:2847 blocked for more than 120 seconds.
Sep  9 23:23:51 ged kernel: [1436769.178999]       Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep  9 23:23:51 ged kernel: [1436769.179047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  9 23:23:51 ged kernel: [1436769.179105] mysqld          D    0  2847      1 0x00000000
Sep  9 23:23:51 ged kernel: [1436769.179111] Call Trace:
Sep  9 23:23:51 ged kernel: [1436769.179123]  __schedule+0x3d6/0x8b0
Sep  9 23:23:51 ged kernel: [1436769.179127]  schedule+0x36/0x80
Sep  9 23:23:51 ged kernel: [1436769.179216]  xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179277]  xlog_grant_head_check+0x94/0x100 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179330]  xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179381]  xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179428]  xfs_trans_alloc+0xbe/0x130 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179478]  xfs_vn_update_time+0x5d/0x160 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179486]  file_update_time+0xbe/0x110
Sep  9 23:23:51 ged kernel: [1436769.179493]  ? tcp_recvmsg+0x317/0xab0
Sep  9 23:23:51 ged kernel: [1436769.179542]  xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Quoted 8 lines
Sep  9 23:23:51 ged kernel: [1436769.179588]  xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179632]  xfs_file_write_iter+0x103/0x150 [xfs]
Sep  9 23:23:51 ged kernel: [1436769.179637]  new_sync_write+0xe5/0x140
Sep  9 23:23:51 ged kernel: [1436769.179641]  __vfs_write+0x29/0x40
Sep  9 23:23:51 ged kernel: [1436769.179645]  vfs_write+0xb8/0x1b0
Sep  9 23:23:51 ged kernel: [1436769.179649]  SyS_pwrite64+0x95/0xb0
Sep  9 23:23:51 ged kernel: [1436769.179655]  do_syscall_64+0x73/0x130
Sep  9 23:23:51 ged kernel: [1436769.179661]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2

.....

Does this repeat every 120s?

These hung task warnings can happen if your workload has overloaded your raid array and everything doing IO hangs while it catches up. e.g. you have 6GB of random 4k writes in the controller NV cache and it takes minutes for it to flush (because random 4k writes are slow) and make room for new incoming IO....

If the warnings don't repeat, then it means it was a temporary overload. If the warnings repeat, but change processes and stack traces then it's a sustained overload condition. If exactly the same warnings repeat and/or has stalled and doesn't restart, then we've got some kind of hang occurring and we'll need to look into it further.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#6 lore raw
Quoted 5 lines

What happens on your network every 14 days or so? Is there a rogue client side backup or admin task running somewhere?

Well, we run nightly backups, but thats read ops.
When I look at the load, its not particular more loaded at that time, than normal work.

Quoted 2 lines

Does this repeat every 120s?

No, what I sent is the full trace. It happened around 23:23, but no more XFS errors in the log (which is on the ext4 OS disk).
It was working when I came in the following morning aroung 6:45, and worked for some time, but initially failed, and we had to reboot the server to get NFS exports to work. But, as I said, even though the fs was inaccessible from NFS I could ls the filesystem locally, but we really have no indication of it being an NFS problem, as we only see the XFS problem.

It could also boil down to a NFS problem, I just wasn't sure how to read the XFS trace.

Quoted 19 lines

These hung task warnings can happen if your workload has overloaded your raid array and everything doing IO hangs while it catches up. e.g. you have 6GB of random 4k writes in the controller NV cache and it takes minutes for it to flush (because random 4k writes are slow) and make room for new incoming IO....

If the warnings don't repeat, then it means it was a temporary overload. If the warnings repeat, but change processes and stack traces then it's a sustained overload condition. If exactly the same warnings repeat and/or has stalled and doesn't restart, then we've got some kind of hang occurring and we'll need to look into it further.

Cheers,

Dave.
--
Dave Chinner
[email protected]

#7 lore raw
Quoted 7 lines · Troels Hansen
Quoted 5 lines

What happens on your network every 14 days or so? Is there a rogue client side backup or admin task running somewhere?

Well, we run nightly backups, but thats read ops.

Yup, but that can get stuck modifying atime, like the bacula process in the hung process traces. :)

Hmmm - just a thought - it's hardware raid - it's not running a background admin op like a media scrub every 14 days, is it?

Quoted 1 line

When I look at the load, its not particular more loaded at that time, than normal work.

OK.

Quoted 4 lines
Quoted 1 line

Does this repeat every 120s?

No, what I sent is the full trace. It happened around 23:23, but no more XFS errors in the log (which is on the ext4 OS disk).

Ok, so those processes reported as hung have been woken and made progress again. It seems like a temporary overload situation.

Quoted 6 lines

It was working when I came in the following morning aroung 6:45, and worked for some time, but initially failed, and we had to reboot the server to get NFS exports to work. But, as I said, even though the fs was inaccessible from NFS I could ls the filesystem locally, but we really have no indication of it being an NFS problem, as we only see the XFS problem.

That could be the same problem, with all the kernel nfsds blocked waiting for the filesystem so no new NFS requests could be processed. How many kernel nfsd threads do you run? Local operations can still be done (don't go through nfsds), and they won't be slow if they hit the caches rather than have to retreive data from disk.

Quoted 2 lines

It could also boil down to a NFS problem, I just wasn't sure how to read the XFS trace.

Like you, I don't think this is an NFS problem - it smells more of how huge hardware writeback caches in front of slow disks using RAID5/6 behave.

i.e. Flushing 100MB of sequential write data from the cache takes a fraction or a second, flushing 100MB of random 4k write data to RAID5 luns can take minutes. While the hardware cache and flushing is supposed to be completely invisible to the OS, we can see it's impact via unexpectedly high device utilisations and long IO times for otherwise normal IO loads.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]