Skip to contentSkip to conversation

ocfs2 mount error

DISCUSSION Valentin Vidić 4 participants 14 messages Mar 10, 2024 – Mar 12, 2024 OCFS2

lore
Share & tools
lore
#0 lore raw

Hi,

I've started seeing the following mount error on recent kernels (6.7, 6.8) while the same script worked on older versions (6.1):

2024-03-10T21:04:30,623900+00:00 ocfs2: Registered cluster interface user
2024-03-10T21:04:30,627545+00:00 ocfs2: Unable to create the lockspace D6DCCE389FC84C8FB9E99ABE1D941825 (-17), because a ocfs2-tools program is running on this file system with the same name lockspace
2024-03-10T21:04:30,630607+00:00 (mount.ocfs2,1577,0):ocfs2_dlm_init:3357 ERROR: status = -17
2024-03-10T21:04:30,633153+00:00 (mount.ocfs2,1577,0):ocfs2_mount_volume:1791 ERROR: status = -17
2024-03-10T21:04:30,634947+00:00 (mount.ocfs2,1577,0):ocfs2_fill_super:1176 ERROR: status = -17

Script runs the commands in the following order:

mkfs.ocfs2
fsck.ocfs2
mount.ocfs2

fsck command tries to release the lockspace but it seems to be busy so it stays around and the mount command fails. Running dlm_tool leave manually removes it and after that the mount command works again.

A git bisect has led me to the following change:

61bed0baa4dba17dd06cdfe20481a580718d6c7c fs: dlm: use a non-static queue for callbacks

Could it be that some callback does not run correctly causing the lockspace to be busy? I have added some more debug info below including some logging of 5 kref_get and 4 kref_put that could be related.

# dlm_tool ls   
dlm lockspaces
name          D6DCCE389FC84C8FB9E99ABE1D941825
id            0x3a105115
flags         0x00000000 
change        member 1 joined 1 remove 0 failed 0 seq 1,1
members       1 
# dlm_tool status D6DCCE389FC84C8FB9E99ABE1D941825
cluster nodeid 1 quorate 1 ring seq 5 5
daemon now 1831 fence_pid 0 
node 1 M add 54 rem 0 fail 0 fence 0 at 0 0

/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 0000000020c29d80 Name (len=31) "S000000000000000000000200000000" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000003cfd484e Name (len=17) ".5f5b44890bdfac21" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 00000000c8e96ac4 Name (len=31) "M0000000000000000000018fe4c9cda" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000004c006d70 Name (len=31) "M0000000000000000000019fe4c9cda" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb ptr nodeid first_lkid flags !root_list_empty !recover_list_empty recover_locks_count len /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 0000000020c29d80 0 0 0 0 0 0 31 str S000000000000000000000200000000 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000003cfd484e 0 0 0 0 0 0 17 str .5f5b44890bdfac21 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 00000000c8e96ac4 0 0 0 0 0 0 31 str M0000000000000000000018fe4c9cda /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000004c006d70 0 0 0 0 0 0 31 str M0000000000000000000019fe4c9cda /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_locks:id nodeid remid pid xid exflags flags sts grmode rqmode time_ms r_nodeid r_len r_name /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_queued_asts:lkb_id lkb_flags mode flags sb_status sb_flags

2024-03-10T21:04:29,431465+00:00 dlm: kref_get 1
2024-03-10T21:04:29,432585+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:29,434951+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:29,436839+00:00 Call Trace:
2024-03-10T21:04:29,437077+00:00  <TASK>
2024-03-10T21:04:29,437308+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:29,437653+00:00  add_lkb+0x2f/0x300 [dlm]
2024-03-10T21:04:29,438000+00:00  ? rsb_insert+0x85/0xd0 [dlm]
2024-03-10T21:04:29,438378+00:00  _grant_lock+0x111/0x170 [dlm]
2024-03-10T21:04:29,438756+00:00  do_request+0x71/0x90 [dlm]
2024-03-10T21:04:29,439112+00:00  _request_lock+0x19b/0x270 [dlm]
2024-03-10T21:04:29,439509+00:00  request_lock+0xc7/0x110 [dlm]
2024-03-10T21:04:29,439887+00:00  dlm_user_request+0x10d/0x370 [dlm]
2024-03-10T21:04:29,440304+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:29,440763+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:29,441161+00:00  ? __pfx_fake_astfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,441578+00:00  ? __pfx_fake_bastfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,442016+00:00  device_write+0x5f1/0x790 [dlm]
2024-03-10T21:04:29,442413+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:29,442723+00:00  ? __x64_sys_fcntl+0x90/0xc0
2024-03-10T21:04:29,443082+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:29,443389+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:29,443729+00:00  ? exc_page_fault+0x73/0x170
2024-03-10T21:04:29,444086+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:29,453050+00:00 dlm: kref_get 1
2024-03-10T21:04:29,453556+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:29,454373+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:29,455462+00:00 Call Trace:
2024-03-10T21:04:29,455922+00:00  <TASK>
2024-03-10T21:04:29,456345+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:29,456903+00:00  dlm_user_add_ast+0x1fc/0x300 [dlm]
2024-03-10T21:04:29,457548+00:00  do_request+0x7b/0x90 [dlm]
2024-03-10T21:04:29,458128+00:00  _request_lock+0x19b/0x270 [dlm]
2024-03-10T21:04:29,458771+00:00  request_lock+0xc7/0x110 [dlm]
2024-03-10T21:04:29,459375+00:00  dlm_user_request+0x10d/0x370 [dlm]
2024-03-10T21:04:29,460021+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:29,460714+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:29,461363+00:00  ? __pfx_fake_astfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,462003+00:00  ? __pfx_fake_bastfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,462659+00:00  device_write+0x5f1/0x790 [dlm]
2024-03-10T21:04:29,463268+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:29,463791+00:00  ? __x64_sys_fcntl+0x90/0xc0
2024-03-10T21:04:29,464359+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:29,464891+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:29,465440+00:00  ? exc_page_fault+0x73/0x170
2024-03-10T21:04:29,466005+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:29,475107+00:00 dlm: kref_get 1
2024-03-10T21:04:29,475617+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:29,476460+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:29,477545+00:00 Call Trace:
2024-03-10T21:04:29,478031+00:00  <TASK>
2024-03-10T21:04:29,478494+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:29,479094+00:00  dlm_user_request+0x299/0x370 [dlm]
2024-03-10T21:04:29,479787+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:29,480544+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:29,481195+00:00  ? __pfx_fake_astfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,481848+00:00  ? __pfx_fake_bastfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,482518+00:00  device_write+0x5f1/0x790 [dlm]
2024-03-10T21:04:29,483140+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:29,483672+00:00  ? __x64_sys_fcntl+0x90/0xc0
2024-03-10T21:04:29,484254+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:29,484785+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:29,485380+00:00  ? exc_page_fault+0x73/0x170
2024-03-10T21:04:29,485954+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,416950+00:00 dlm: kref_get 1
2024-03-10T21:04:30,417698+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,418547+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,419629+00:00 Call Trace:
2024-03-10T21:04:30,420115+00:00  <TASK>
2024-03-10T21:04:30,420566+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,421157+00:00  find_lkb+0x4e/0xd0 [dlm]
2024-03-10T21:04:30,421758+00:00  dlm_user_unlock+0x6b/0x2f0 [dlm]
2024-03-10T21:04:30,422422+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,423140+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,423785+00:00  device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,424404+00:00  ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,425057+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:30,425606+00:00  ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,426268+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,426857+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:30,427415+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,427969+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,428534+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,429098+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,429657+00:00  ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,430211+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,440429+00:00 dlm: kref_put 1
2024-03-10T21:04:30,441102+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,441961+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,443065+00:00 Call Trace:
2024-03-10T21:04:30,443571+00:00  <TASK>
2024-03-10T21:04:30,444029+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,444618+00:00  _unlock_lock+0x73/0x110 [dlm]
2024-03-10T21:04:30,445257+00:00  unlock_lock.constprop.0+0x55/0x90 [dlm]
2024-03-10T21:04:30,445979+00:00  dlm_user_unlock+0x13b/0x2f0 [dlm]
2024-03-10T21:04:30,446641+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,447345+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,447979+00:00  device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,448593+00:00  ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,449460+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:30,450394+00:00  ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,451160+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,451773+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:30,452316+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,452894+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,453491+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,454053+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,454621+00:00  ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,455175+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,464305+00:00 dlm: kref_put 1
2024-03-10T21:04:30,464804+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,465649+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,466737+00:00 Call Trace:
2024-03-10T21:04:30,467220+00:00  <TASK>
2024-03-10T21:04:30,467684+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,468276+00:00  _unlock_lock+0x9e/0x110 [dlm]
2024-03-10T21:04:30,468912+00:00  unlock_lock.constprop.0+0x55/0x90 [dlm]
2024-03-10T21:04:30,469607+00:00  dlm_user_unlock+0x13b/0x2f0 [dlm]
2024-03-10T21:04:30,470261+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,470961+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,471611+00:00  device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,472221+00:00  ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,472867+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:30,473397+00:00  ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,474055+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,474633+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:30,475179+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,475759+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,476322+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,476883+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,477442+00:00  ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,478000+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,486964+00:00 dlm: kref_get 1
2024-03-10T21:04:30,487460+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,488304+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,489384+00:00 Call Trace:
2024-03-10T21:04:30,489869+00:00  <TASK>
2024-03-10T21:04:30,490320+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,490918+00:00  dlm_user_add_ast+0x1fc/0x300 [dlm]
2024-03-10T21:04:30,491609+00:00  _unlock_lock+0xbe/0x110 [dlm]
2024-03-10T21:04:30,492231+00:00  unlock_lock.constprop.0+0x55/0x90 [dlm]
2024-03-10T21:04:30,492914+00:00  dlm_user_unlock+0x13b/0x2f0 [dlm]
2024-03-10T21:04:30,493553+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,494238+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,494869+00:00  device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,495479+00:00  ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,496133+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:30,496660+00:00  ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,497318+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,497898+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:30,498428+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,498978+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,499532+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,500086+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,500639+00:00  ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,501192+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,510147+00:00 dlm: kref_put 1
2024-03-10T21:04:30,510651+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,511482+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,512564+00:00 Call Trace:
2024-03-10T21:04:30,513045+00:00  <TASK>
2024-03-10T21:04:30,513496+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,514081+00:00  __put_lkb+0x30/0x100 [dlm]
2024-03-10T21:04:30,514683+00:00  dlm_user_add_ast+0x1cc/0x300 [dlm]
2024-03-10T21:04:30,515337+00:00  _unlock_lock+0xbe/0x110 [dlm]
2024-03-10T21:04:30,515953+00:00  unlock_lock.constprop.0+0x55/0x90 [dlm]
2024-03-10T21:04:30,516633+00:00  dlm_user_unlock+0x13b/0x2f0 [dlm]
2024-03-10T21:04:30,517265+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,517957+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,518586+00:00  device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,519199+00:00  ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,519840+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:30,520365+00:00  ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,521021+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,521586+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:30,522100+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,522654+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,523212+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,523779+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,524333+00:00  ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,524888+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,533818+00:00 dlm: kref_put 1
2024-03-10T21:04:30,534317+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,535151+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,536247+00:00 Call Trace:
2024-03-10T21:04:30,536729+00:00  <TASK>
2024-03-10T21:04:30,537179+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,537755+00:00  __put_lkb+0x30/0x100 [dlm]
2024-03-10T21:04:30,538355+00:00  dlm_user_unlock+0x213/0x2f0 [dlm]
2024-03-10T21:04:30,539003+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,539698+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,540319+00:00  device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,540927+00:00  ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,541569+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:30,542093+00:00  ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,542759+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,543327+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:30,543851+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,544395+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,544949+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,545500+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,546059+00:00  ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,546621+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,547274+00:00 RIP: 0033:0x7fa4d45f0ba0
Signature · 1 line
Valentin
maintainer
#1 lore raw

Hi,

It seems that you are using fsdlm.
But from your running commands, I don't see how fsdlm get involved. Could you please give more detail on how to reproduce it?

Thanks,
Joseph

Quoted 292 lines · Valentin Vidić

Hi,

I've started seeing the following mount error on recent kernels (6.7, 6.8) while the same script worked on older versions (6.1):

2024-03-10T21:04:30,623900+00:00 ocfs2: Registered cluster interface user
2024-03-10T21:04:30,627545+00:00 ocfs2: Unable to create the lockspace D6DCCE389FC84C8FB9E99ABE1D941825 (-17), because a ocfs2-tools program is running on this file system with the same name lockspace
2024-03-10T21:04:30,630607+00:00 (mount.ocfs2,1577,0):ocfs2_dlm_init:3357 ERROR: status = -17
2024-03-10T21:04:30,633153+00:00 (mount.ocfs2,1577,0):ocfs2_mount_volume:1791 ERROR: status = -17
2024-03-10T21:04:30,634947+00:00 (mount.ocfs2,1577,0):ocfs2_fill_super:1176 ERROR: status = -17

Script runs the commands in the following order:

mkfs.ocfs2
fsck.ocfs2
mount.ocfs2

fsck command tries to release the lockspace but it seems to be busy so it stays around and the mount command fails. Running dlm_tool leave manually removes it and after that the mount command works again.

A git bisect has led me to the following change:

61bed0baa4dba17dd06cdfe20481a580718d6c7c fs: dlm: use a non-static queue for callbacks

Could it be that some callback does not run correctly causing the lockspace to be busy? I have added some more debug info below including some logging of 5 kref_get and 4 kref_put that could be related.

# dlm_tool ls   
dlm lockspaces
name          D6DCCE389FC84C8FB9E99ABE1D941825
id            0x3a105115
flags         0x00000000 
change        member 1 joined 1 remove 0 failed 0 seq 1,1
members       1 
# dlm_tool status D6DCCE389FC84C8FB9E99ABE1D941825
cluster nodeid 1 quorate 1 ring seq 5 5
daemon now 1831 fence_pid 0 
node 1 M add 54 rem 0 fail 0 fence 0 at 0 0

/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 0000000020c29d80 Name (len=31) "S000000000000000000000200000000" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000003cfd484e Name (len=17) ".5f5b44890bdfac21" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 00000000c8e96ac4 Name (len=31) "M0000000000000000000018fe4c9cda" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000004c006d70 Name (len=31) "M0000000000000000000019fe4c9cda" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb ptr nodeid first_lkid flags !root_list_empty !recover_list_empty recover_locks_count len /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 0000000020c29d80 0 0 0 0 0 0 31 str S000000000000000000000200000000 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000003cfd484e 0 0 0 0 0 0 17 str .5f5b44890bdfac21 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 00000000c8e96ac4 0 0 0 0 0 0 31 str M0000000000000000000018fe4c9cda /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000004c006d70 0 0 0 0 0 0 31 str M0000000000000000000019fe4c9cda /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_locks:id nodeid remid pid xid exflags flags sts grmode rqmode time_ms r_nodeid r_len r_name /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_queued_asts:lkb_id lkb_flags mode flags sb_status sb_flags

2024-03-10T21:04:29,431465+00:00 dlm: kref_get 1
2024-03-10T21:04:29,432585+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:29,434951+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:29,436839+00:00 Call Trace:
2024-03-10T21:04:29,437077+00:00  <TASK>
2024-03-10T21:04:29,437308+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:29,437653+00:00  add_lkb+0x2f/0x300 [dlm]
2024-03-10T21:04:29,438000+00:00  ? rsb_insert+0x85/0xd0 [dlm]
2024-03-10T21:04:29,438378+00:00  _grant_lock+0x111/0x170 [dlm]
2024-03-10T21:04:29,438756+00:00  do_request+0x71/0x90 [dlm]
2024-03-10T21:04:29,439112+00:00  _request_lock+0x19b/0x270 [dlm]
2024-03-10T21:04:29,439509+00:00  request_lock+0xc7/0x110 [dlm]
2024-03-10T21:04:29,439887+00:00  dlm_user_request+0x10d/0x370 [dlm]
2024-03-10T21:04:29,440304+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:29,440763+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:29,441161+00:00  ? __pfx_fake_astfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,441578+00:00  ? __pfx_fake_bastfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,442016+00:00  device_write+0x5f1/0x790 [dlm]
2024-03-10T21:04:29,442413+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:29,442723+00:00  ? __x64_sys_fcntl+0x90/0xc0
2024-03-10T21:04:29,443082+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:29,443389+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:29,443729+00:00  ? exc_page_fault+0x73/0x170
2024-03-10T21:04:29,444086+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:29,453050+00:00 dlm: kref_get 1
2024-03-10T21:04:29,453556+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:29,454373+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:29,455462+00:00 Call Trace:
2024-03-10T21:04:29,455922+00:00  <TASK>
2024-03-10T21:04:29,456345+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:29,456903+00:00  dlm_user_add_ast+0x1fc/0x300 [dlm]
2024-03-10T21:04:29,457548+00:00  do_request+0x7b/0x90 [dlm]
2024-03-10T21:04:29,458128+00:00  _request_lock+0x19b/0x270 [dlm]
2024-03-10T21:04:29,458771+00:00  request_lock+0xc7/0x110 [dlm]
2024-03-10T21:04:29,459375+00:00  dlm_user_request+0x10d/0x370 [dlm]
2024-03-10T21:04:29,460021+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:29,460714+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:29,461363+00:00  ? __pfx_fake_astfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,462003+00:00  ? __pfx_fake_bastfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,462659+00:00  device_write+0x5f1/0x790 [dlm]
2024-03-10T21:04:29,463268+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:29,463791+00:00  ? __x64_sys_fcntl+0x90/0xc0
2024-03-10T21:04:29,464359+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:29,464891+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:29,465440+00:00  ? exc_page_fault+0x73/0x170
2024-03-10T21:04:29,466005+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:29,475107+00:00 dlm: kref_get 1
2024-03-10T21:04:29,475617+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:29,476460+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:29,477545+00:00 Call Trace:
2024-03-10T21:04:29,478031+00:00  <TASK>
2024-03-10T21:04:29,478494+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:29,479094+00:00  dlm_user_request+0x299/0x370 [dlm]
2024-03-10T21:04:29,479787+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:29,480544+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:29,481195+00:00  ? __pfx_fake_astfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,481848+00:00  ? __pfx_fake_bastfn+0x10/0x10 [dlm]
2024-03-10T21:04:29,482518+00:00  device_write+0x5f1/0x790 [dlm]
2024-03-10T21:04:29,483140+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:29,483672+00:00  ? __x64_sys_fcntl+0x90/0xc0
2024-03-10T21:04:29,484254+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:29,484785+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:29,485380+00:00  ? exc_page_fault+0x73/0x170
2024-03-10T21:04:29,485954+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,416950+00:00 dlm: kref_get 1
2024-03-10T21:04:30,417698+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,418547+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,419629+00:00 Call Trace:
2024-03-10T21:04:30,420115+00:00  <TASK>
2024-03-10T21:04:30,420566+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,421157+00:00  find_lkb+0x4e/0xd0 [dlm]
2024-03-10T21:04:30,421758+00:00  dlm_user_unlock+0x6b/0x2f0 [dlm]
2024-03-10T21:04:30,422422+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,423140+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,423785+00:00  device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,424404+00:00  ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,425057+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:30,425606+00:00  ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,426268+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,426857+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:30,427415+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,427969+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,428534+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,429098+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,429657+00:00  ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,430211+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,440429+00:00 dlm: kref_put 1
2024-03-10T21:04:30,441102+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,441961+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,443065+00:00 Call Trace:
2024-03-10T21:04:30,443571+00:00  <TASK>
2024-03-10T21:04:30,444029+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,444618+00:00  _unlock_lock+0x73/0x110 [dlm]
2024-03-10T21:04:30,445257+00:00  unlock_lock.constprop.0+0x55/0x90 [dlm]
2024-03-10T21:04:30,445979+00:00  dlm_user_unlock+0x13b/0x2f0 [dlm]
2024-03-10T21:04:30,446641+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,447345+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,447979+00:00  device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,448593+00:00  ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,449460+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:30,450394+00:00  ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,451160+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,451773+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:30,452316+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,452894+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,453491+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,454053+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,454621+00:00  ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,455175+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,464305+00:00 dlm: kref_put 1
2024-03-10T21:04:30,464804+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,465649+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,466737+00:00 Call Trace:
2024-03-10T21:04:30,467220+00:00  <TASK>
2024-03-10T21:04:30,467684+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,468276+00:00  _unlock_lock+0x9e/0x110 [dlm]
2024-03-10T21:04:30,468912+00:00  unlock_lock.constprop.0+0x55/0x90 [dlm]
2024-03-10T21:04:30,469607+00:00  dlm_user_unlock+0x13b/0x2f0 [dlm]
2024-03-10T21:04:30,470261+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,470961+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,471611+00:00  device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,472221+00:00  ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,472867+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:30,473397+00:00  ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,474055+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,474633+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:30,475179+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,475759+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,476322+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,476883+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,477442+00:00  ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,478000+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,486964+00:00 dlm: kref_get 1
2024-03-10T21:04:30,487460+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,488304+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,489384+00:00 Call Trace:
2024-03-10T21:04:30,489869+00:00  <TASK>
2024-03-10T21:04:30,490320+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,490918+00:00  dlm_user_add_ast+0x1fc/0x300 [dlm]
2024-03-10T21:04:30,491609+00:00  _unlock_lock+0xbe/0x110 [dlm]
2024-03-10T21:04:30,492231+00:00  unlock_lock.constprop.0+0x55/0x90 [dlm]
2024-03-10T21:04:30,492914+00:00  dlm_user_unlock+0x13b/0x2f0 [dlm]
2024-03-10T21:04:30,493553+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,494238+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,494869+00:00  device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,495479+00:00  ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,496133+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:30,496660+00:00  ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,497318+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,497898+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:30,498428+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,498978+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,499532+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,500086+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,500639+00:00  ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,501192+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,510147+00:00 dlm: kref_put 1
2024-03-10T21:04:30,510651+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,511482+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,512564+00:00 Call Trace:
2024-03-10T21:04:30,513045+00:00  <TASK>
2024-03-10T21:04:30,513496+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,514081+00:00  __put_lkb+0x30/0x100 [dlm]
2024-03-10T21:04:30,514683+00:00  dlm_user_add_ast+0x1cc/0x300 [dlm]
2024-03-10T21:04:30,515337+00:00  _unlock_lock+0xbe/0x110 [dlm]
2024-03-10T21:04:30,515953+00:00  unlock_lock.constprop.0+0x55/0x90 [dlm]
2024-03-10T21:04:30,516633+00:00  dlm_user_unlock+0x13b/0x2f0 [dlm]
2024-03-10T21:04:30,517265+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,517957+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,518586+00:00  device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,519199+00:00  ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,519840+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:30,520365+00:00  ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,521021+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,521586+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:30,522100+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,522654+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,523212+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,523779+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,524333+00:00  ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,524888+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,533818+00:00 dlm: kref_put 1
2024-03-10T21:04:30,534317+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80
2024-03-10T21:04:30,535151+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
2024-03-10T21:04:30,536247+00:00 Call Trace:
2024-03-10T21:04:30,536729+00:00  <TASK>
2024-03-10T21:04:30,537179+00:00  dump_stack_lvl+0x43/0x60
2024-03-10T21:04:30,537755+00:00  __put_lkb+0x30/0x100 [dlm]
2024-03-10T21:04:30,538355+00:00  dlm_user_unlock+0x213/0x2f0 [dlm]
2024-03-10T21:04:30,539003+00:00  ? __kmalloc_node_track_caller+0x1a9/0x420
2024-03-10T21:04:30,539698+00:00  ? device_write+0x44/0x790 [dlm]
2024-03-10T21:04:30,540319+00:00  device_write+0x5bd/0x790 [dlm]
2024-03-10T21:04:30,540927+00:00  ? security_file_permission+0x32/0x50
2024-03-10T21:04:30,541569+00:00  vfs_write+0xec/0x470
2024-03-10T21:04:30,542093+00:00  ? syscall_exit_to_user_mode+0x81/0x220
2024-03-10T21:04:30,542759+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,543327+00:00  ksys_write+0x6b/0xf0
2024-03-10T21:04:30,543851+00:00  do_syscall_64+0x87/0x1a0
2024-03-10T21:04:30,544395+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,544949+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,545500+00:00  ? do_syscall_64+0x93/0x1a0
2024-03-10T21:04:30,546059+00:00  ? __irq_exit_rcu+0x3b/0xb0
2024-03-10T21:04:30,546621+00:00  entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,547274+00:00 RIP: 0033:0x7fa4d45f0ba0
#2 lore raw
Quoted 3 lines · Joseph Qi

It seems that you are using fsdlm.
But from your running commands, I don't see how fsdlm get involved. Could you please give more detail on how to reproduce it?

Sure, this is the original Pacemaker test script that failed:

https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads

Basically it starts Corosync and DLM services and than calls:

mkfs.ocfs2
fsck.ocfs2
mount.ocfs2

Signature · 1 line
Valentin
maintainer
#3 lore raw
Quoted 7 lines · Valentin Vidić
Quoted 3 lines · Joseph Qi

It seems that you are using fsdlm.
But from your running commands, I don't see how fsdlm get involved. Could you please give more detail on how to reproduce it?

Sure, this is the original Pacemaker test script that failed:

Quoted 8 lines

https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads

Basically it starts Corosync and DLM services and than calls:

mkfs.ocfs2
fsck.ocfs2
mount.ocfs2

Thanks for quick reply.
Now it confirms that it is using fsdlm.

Joseph

reviewer
#4 lore raw
Quoted 19 lines · Joseph Qi
Quoted 7 lines · Valentin Vidić
Quoted 3 lines · Joseph Qi

It seems that you are using fsdlm.
But from your running commands, I don't see how fsdlm get involved. Could you please give more detail on how to reproduce it?

Sure, this is the original Pacemaker test script that failed:

Quoted 8 lines

https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads

Basically it starts Corosync and DLM services and than calls:

mkfs.ocfs2
fsck.ocfs2
mount.ocfs2

Quoted 6 lines

Thanks for quick reply.
Now it confirms that it is using fsdlm.

Joseph

It looks like I met the same issue one year ago. check [1] and search for "error 17." in this mail.

[1]: https://lore.kernel.org/ocfs2-devel/20230626150916.je3egonzb3crvbl5@p15/

If my memory is still right, this root cause of this issue is that fsck forgets to clear the lockspace before exiting. I will investigate the debian's issue and provide a fix.

Thanks,
Heming

#5 lore raw
In reply to Valentin Vidić

Hi,

Quoted 6 lines · Valentin Vidić
Quoted 3 lines · Joseph Qi

It seems that you are using fsdlm.
But from your running commands, I don't see how fsdlm get involved. Could you please give more detail on how to reproduce it?

Quoted 8 lines

Sure, this is the original Pacemaker test script that failed:

https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads

Basically it starts Corosync and DLM services and than calls:

mkfs.ocfs2
fsck.ocfs2

when I try doing your steps (btw: thanks for sharing) to run fsck.ocfs2 it tells me:

fsck.ocfs2: Unable to access cluster service while initializing the DLM

Looking into strace right before that I see a:

connect(4, {sa_family=AF_UNIX, sun_path=@"ocfs2_controld_sock"}, 22) = -1 ECONNREFUSED (Connection refused)

it's true I don't have ocfs2_controld running, after looking into ocfs2-tools source code, I declare it as not compilable with recent corosync versions.

Maybe somebody else can help so I can try to reproduce it?

btw: a "trace-cmd stream -e dlm" would be nice to debug this issue. After I look at your debugging information it would be nice to see if the device() read for ast callbacks are there.

- Alex

#6 lore raw
Quoted 5 lines · Alexander Aring

when I try doing your steps (btw: thanks for sharing) to run fsck.ocfs2 it tells me:

fsck.ocfs2: Unable to access cluster service while initializing the DLM

Quoted 8 lines

Looking into strace right before that I see a:

connect(4, {sa_family=AF_UNIX, sun_path=@"ocfs2_controld_sock"}, 22) = -1 ECONNREFUSED (Connection refused)

it's true I don't have ocfs2_controld running, after looking into ocfs2-tools source code, I declare it as not compilable with recent corosync versions.

Not sure about that, because I also don't have ocfs2_controld running. This is ocfs2-tools v1.8.8 with corosync v3.1.7. Perhaps cluster stack should be set with:

echo pcmk > /sys/fs/ocfs2/cluster_stack

Quoted 3 lines

btw: a "trace-cmd stream -e dlm" would be nice to debug this issue. After I look at your debugging information it would be nice to see if the device() read for ast callbacks are there.

Here is a trace report for the fsck call:

cpus=1

      fsck.ocfs2-1696  [000]    54.621075: dlm_lock_start:       ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK res_name=2e39323539656662313866316162386331
      fsck.ocfs2-1696  [000]    54.692487: dlm_lock_end:         ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK error=0 res_name=2e39323539656662313866316162386331
      fsck.ocfs2-1696  [000]    54.695447: dlm_ast:              ls_id=2352585329 lkb_id=1 sb_flags= sb_status=0 res_name=2e39323539656662313866316162386331
      fsck.ocfs2-1696  [000]    54.695636: dlm_lock_start:       ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030
      fsck.ocfs2-1696  [000]    54.766959: dlm_lock_end:         ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030
      fsck.ocfs2-1696  [000]    54.770765: dlm_ast:              ls_id=2352585329 lkb_id=2 sb_flags= sb_status=0 res_name=53303030303030303030303030303030303030303030323030303030303030
      fsck.ocfs2-1696  [000]    54.770821: dlm_lock_start:       ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531
      fsck.ocfs2-1696  [000]    54.845613: dlm_lock_end:         ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
      fsck.ocfs2-1696  [000]    54.848395: dlm_ast:              ls_id=2352585329 lkb_id=3 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
      fsck.ocfs2-1696  [000]    54.868951: dlm_unlock_start:     ls_id=2352585329 lkb_id=3 flags=VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531
      fsck.ocfs2-1696  [000]    54.958319: dlm_unlock_end:       ls_id=2352585329 lkb_id=3 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
      fsck.ocfs2-1696  [000]    54.982397: dlm_ast:              ls_id=2352585329 lkb_id=3 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031383937333737316531
      fsck.ocfs2-1696  [000]    54.982450: dlm_lock_start:       ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531
      fsck.ocfs2-1696  [000]    55.056390: dlm_lock_end:         ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
      fsck.ocfs2-1696  [000]    55.058946: dlm_ast:              ls_id=2352585329 lkb_id=4 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
      fsck.ocfs2-1696  [000]    55.082546: dlm_unlock_start:     ls_id=2352585329 lkb_id=4 flags=VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531
      fsck.ocfs2-1696  [000]    55.182903: dlm_unlock_end:       ls_id=2352585329 lkb_id=4 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
      fsck.ocfs2-1696  [000]    55.210128: dlm_ast:              ls_id=2352585329 lkb_id=4 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031393937333737316531
      fsck.ocfs2-1696  [000]    55.530687: dlm_unlock_start:     ls_id=2352585329 lkb_id=2 flags=VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030
      fsck.ocfs2-1696  [000]    55.625015: dlm_unlock_end:       ls_id=2352585329 lkb_id=2 flags=VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030
      fsck.ocfs2-1696  [000]    55.650054: dlm_ast:              ls_id=2352585329 lkb_id=2 sb_flags= sb_status=-65538 res_name=53303030303030303030303030303030303030303030323030303030303030
      fsck.ocfs2-1696  [000]    55.672033: dlm_unlock_start:     ls_id=2352585329 lkb_id=1 flags=VALBLK res_name=2e39323539656662313866316162386331
      fsck.ocfs2-1696  [000]    55.766844: dlm_unlock_end:       ls_id=2352585329 lkb_id=1 flags=VALBLK error=0 res_name=2e39323539656662313866316162386331
      fsck.ocfs2-1696  [000]    55.791829: dlm_ast:              ls_id=2352585329 lkb_id=1 sb_flags= sb_status=-65538 res_name=2e39323539656662313866316162386331
Signature · 1 line
Valentin
#7 lore raw

Hi,

Quoted 16 lines · Valentin Vidić
Quoted 5 lines · Alexander Aring

when I try doing your steps (btw: thanks for sharing) to run fsck.ocfs2 it tells me:

fsck.ocfs2: Unable to access cluster service while initializing the DLM

Quoted 8 lines

Looking into strace right before that I see a:

connect(4, {sa_family=AF_UNIX, sun_path=@"ocfs2_controld_sock"}, 22) = -1 ECONNREFUSED (Connection refused)

it's true I don't have ocfs2_controld running, after looking into ocfs2-tools source code, I declare it as not compilable with recent corosync versions.

Quoted 6 lines

Not sure about that, because I also don't have ocfs2_controld running. This is ocfs2-tools v1.8.8 with corosync v3.1.7. Perhaps cluster stack should be set with:

echo pcmk > /sys/fs/ocfs2/cluster_stack

I step forward here, but still I am not able to set a cluster name. I hardcoded it in the kernel and "it seems" to work. How do you set the cluster name? It seems that setting the cluster name in mkfs is not the way to go.
I am able to run fsck and I see on "dlm_tool ls" a lockspace with a similar name you have.

The bad news is, I can't reproduce it. After fsck the lockspace releases successfully.

I see on your debug information that it uses dlm user space locks, in my case I am using dlm kernel locks and I am asking myself why. Can you tell me more about your cluster configuration?

Somehow in your case fsck is using the DLM user space locks functionality from libdlm (the user space DLM locking library) but in my case fsck goes over the ocfs2_fsdlm filesystem and then does DLM kernel locks.
I am asking myself how this is supposed to work.

Quoted 23 lines
Quoted 3 lines

btw: a "trace-cmd stream -e dlm" would be nice to debug this issue. After I look at your debugging information it would be nice to see if the device() read for ast callbacks are there.

Here is a trace report for the fsck call:

cpus=1

      fsck.ocfs2-1696  [000]    54.621075: dlm_lock_start:       ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK res_name=2e39323539656662313866316162386331
      fsck.ocfs2-1696  [000]    54.692487: dlm_lock_end:         ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK error=0 res_name=2e39323539656662313866316162386331
      fsck.ocfs2-1696  [000]    54.695447: dlm_ast:              ls_id=2352585329 lkb_id=1 sb_flags= sb_status=0 res_name=2e39323539656662313866316162386331
      fsck.ocfs2-1696  [000]    54.695636: dlm_lock_start:       ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030
      fsck.ocfs2-1696  [000]    54.766959: dlm_lock_end:         ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030
      fsck.ocfs2-1696  [000]    54.770765: dlm_ast:              ls_id=2352585329 lkb_id=2 sb_flags= sb_status=0 res_name=53303030303030303030303030303030303030303030323030303030303030
      fsck.ocfs2-1696  [000]    54.770821: dlm_lock_start:       ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531
      fsck.ocfs2-1696  [000]    54.845613: dlm_lock_end:         ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
      fsck.ocfs2-1696  [000]    54.848395: dlm_ast:              ls_id=2352585329 lkb_id=3 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
      fsck.ocfs2-1696  [000]    54.868951: dlm_unlock_start:     ls_id=2352585329 lkb_id=3 flags=VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531
      fsck.ocfs2-1696  [000]    54.958319: dlm_unlock_end:       ls_id=2352585329 lkb_id=3 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
      fsck.ocfs2-1696  [000]    54.982397: dlm_ast:              ls_id=2352585329 lkb_id=3 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031383937333737316531
      fsck.ocfs2-1696  [000]    54.982450: dlm_lock_start:       ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531
      fsck.ocfs2-1696  [000]    55.056390: dlm_lock_end:         ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
      fsck.ocfs2-1696  [000]    55.058946: dlm_ast:              ls_id=2352585329 lkb_id=4 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
      fsck.ocfs2-1696  [000]    55.082546: dlm_unlock_start:     ls_id=2352585329 lkb_id=4 flags=VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531
Quoted 9 lines
      fsck.ocfs2-1696  [000]    55.182903: dlm_unlock_end:       ls_id=2352585329 lkb_id=4 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
      fsck.ocfs2-1696  [000]    55.210128: dlm_ast:              ls_id=2352585329 lkb_id=4 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031393937333737316531
      fsck.ocfs2-1696  [000]    55.530687: dlm_unlock_start:     ls_id=2352585329 lkb_id=2 flags=VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030
      fsck.ocfs2-1696  [000]    55.625015: dlm_unlock_end:       ls_id=2352585329 lkb_id=2 flags=VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030
      fsck.ocfs2-1696  [000]    55.650054: dlm_ast:              ls_id=2352585329 lkb_id=2 sb_flags= sb_status=-65538 res_name=53303030303030303030303030303030303030303030323030303030303030
      fsck.ocfs2-1696  [000]    55.672033: dlm_unlock_start:     ls_id=2352585329 lkb_id=1 flags=VALBLK res_name=2e39323539656662313866316162386331
      fsck.ocfs2-1696  [000]    55.766844: dlm_unlock_end:       ls_id=2352585329 lkb_id=1 flags=VALBLK error=0 res_name=2e39323539656662313866316162386331
      fsck.ocfs2-1696  [000]    55.791829: dlm_ast:              ls_id=2352585329 lkb_id=1 sb_flags= sb_status=-65538 res_name=2e39323539656662313866316162386331

looks okay to me. For dlm user locks there is currently an issue with lvbarea, see [0]. Not sure if this is why you have such issues right now, it depends what fsck is doing with the lvb.

- Alex

[0] https://lore.kernel.org/gfs2/CAK-6q+h-RngJqJ6Rx2h4a0Qy1j+ZRvCn5E-3Qee7_P3pWXJonQ@mail.gmail.com/T/#ma861c86ab54991a0d9a2742c6983544289f3bdd0

#8 lore raw
Quoted 4 lines · Alexander Aring

I step forward here, but still I am not able to set a cluster name. I hardcoded it in the kernel and "it seems" to work. How do you set the cluster name? It seems that setting the cluster name in mkfs is not the way to go.

I think the cluster name comes from corosync, here is the config I have there:

totem {
version: 2
cluster_name: debian
crypto_cipher: none
crypto_hash: none
}

logging {
	fileline: off
	to_stderr: yes
	to_logfile: yes
	logfile: /var/log/corosync/corosync.log
	to_syslog: yes
	debug: off
	logger_subsys {
		subsys: QUORUM
		debug: off
	}
}
quorum {
	provider: corosync_votequorum
}
nodelist {
	node {
		name: node1
		nodeid: 1
		ring0_addr: 127.0.0.1
	}
}

The status of the services is as follows:

# corosync-quorumtool 
Quorum information
------------------
Date:             Tue Mar 12 07:49:18 2024
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          1
Ring ID:          1.5
Quorate:          Yes

Votequorum information
----------------------

Expected votes:   1
Highest expected: 1
Total votes:      1
Quorum:           1  
Flags:            Quorate 

Membership information
----------------------
Nodeid Votes Name
1 1 node1 (local)

# dlm_tool dump_config
daemon_debug=0
foreground=1 (cli option)
log_debug=0
protocol=detect
port=21064
debug_logfile=0
bind_all=0
mark=0
enable_fscontrol=0
enable_plock=1
plock_debug=0
plock_rate_limit=0
plock_ownership=0
drop_resources_time=10000
drop_resources_count=10
drop_resources_age=10000
post_join_delay=30
enable_fencing=1
enable_concurrent_fencing=0
enable_startup_fencing=1
repeat_failed_fencing=1
enable_quorum_fencing=1
enable_quorum_lockspace=1
enable_helper=1
help=-1
version=-1
Quoted 10 lines

I am able to run fsck and I see on "dlm_tool ls" a lockspace with a similar name you have.

The bad news is, I can't reproduce it. After fsck the lockspace releases successfully.

I see on your debug information that it uses dlm user space locks, in my case I am using dlm kernel locks and I am asking myself why. Can you tell me more about your cluster configuration?

Quoted 5 lines

Somehow in your case fsck is using the DLM user space locks functionality from libdlm (the user space DLM locking library) but in my case fsck goes over the ocfs2_fsdlm filesystem and then does DLM kernel locks.
I am asking myself how this is supposed to work.

I'm attaching the strace from fsck below, but it could be that ocfs2_dlmfs module is not loaded, so it falls back to user space locks?

Signature · 1 line
Valentin

execve("/usr/sbin/fsck.ocfs2", ["fsck.ocfs2", "-f", "-y", "/dev/loop0"], 0x7fff21dd7108 /* 27 vars */) = 0
brk(NULL) = 0x563054c7d000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=13065, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 13065, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f61b3401000
close(3) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b33ff000 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdlm_lt.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=18424, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 20536, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33f9000 mmap(0x7f61b33fa000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f61b33fa000 mmap(0x7f61b33fc000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33fc000 mmap(0x7f61b33fd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33fd000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcmap.so.4", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=26672, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 28816, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33f1000 mmap(0x7f61b33f3000, 12288, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33f3000 mmap(0x7f61b33f6000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b33f6000 mmap(0x7f61b33f7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b33f7000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcom_err.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=18344, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 20552, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33eb000 mmap(0x7f61b33ed000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33ed000 mmap(0x7f61b33ee000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33ee000 mmap(0x7f61b33ef000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33ef000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libaio.so.1t64", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=14184, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 16432, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33e6000 mmap(0x7f61b33e7000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f61b33e7000 mmap(0x7f61b33e8000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33e8000 mmap(0x7f61b33e9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33e9000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220x\2\0\0\0\0\0"..., 832) = 832 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1926256, ...}, AT_EMPTY_PATH) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 mmap(NULL, 1974096, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b3204000 mmap(0x7f61b322a000, 1396736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f61b322a000

mmap(0x7f61b337f000, 344064, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17b000) = 0x7f61b337f000
mmap(0x7f61b33d3000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1cf000) = 0x7f61b33d3000
mmap(0x7f61b33d9000, 53072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b33d9000

close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcorosync_common.so.4", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=13952, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 16400, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b31ff000 mmap(0x7f61b3200000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f61b3200000 mmap(0x7f61b3201000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b3201000 mmap(0x7f61b3202000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b3202000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libqb.so.100", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=165872, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b31fd000 mmap(NULL, 430816, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b3193000

mmap(0x7f61b319a000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f61b319a000
mmap(0x7f61b31b1000, 36864, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e000) = 0x7f61b31b1000
mmap(0x7f61b31ba000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x27000) = 0x7f61b31ba000

mmap(0x7f61b31bc000, 262880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b31bc000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libsystemd.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=918624, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 923208, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b30b1000

mmap(0x7f61b30c8000, 573440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f61b30c8000
mmap(0x7f61b3154000, 200704, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa3000) = 0x7f61b3154000
mmap(0x7f61b3185000, 53248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd3000) = 0x7f61b3185000

mmap(0x7f61b3192000, 1608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b3192000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200w\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=47288, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 45128, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b30a5000 mmap(0x7f61b30a8000, 20480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b30a8000 mmap(0x7f61b30ad000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f61b30ad000 mmap(0x7f61b30af000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x7f61b30af000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgcrypt.so.20", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=1336728, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 1340360, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2f5d000

mmap(0x7f61b2f6c000, 966656, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf000) = 0x7f61b2f6c000
mmap(0x7f61b3058000, 274432, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xfb000) = 0x7f61b3058000
mmap(0x7f61b309b000, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13d000) = 0x7f61b309b000

mmap(0x7f61b30a4000, 968, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b30a4000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/liblz4.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=149952, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 152032, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2f37000 mmap(0x7f61b2f3a000, 122880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b2f3a000 mmap(0x7f61b2f58000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7f61b2f58000 mmap(0x7f61b2f5b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f61b2f5b000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/liblzma.so.5", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=247800, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b2f35000 mmap(NULL, 245800, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2ef8000 mmap(0x7f61b2efc000, 167936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f61b2efc000 mmap(0x7f61b2f25000, 57344, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2d000) = 0x7f61b2f25000 mmap(0x7f61b2f33000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3b000) = 0x7f61b2f33000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libzstd.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=784376, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 786496, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2e37000 mmap(0x7f61b2e3c000, 679936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b2e3c000 mmap(0x7f61b2ee2000, 81920, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xab000) = 0x7f61b2ee2000 mmap(0x7f61b2ef6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xbe000) = 0x7f61b2ef6000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgpg-error.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=161864, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 164304, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2e0e000

mmap(0x7f61b2e13000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b2e13000
mmap(0x7f61b2e2a000, 45056, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x7f61b2e2a000
mmap(0x7f61b2e35000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f61b2e35000

close(3) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b2e0c000
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b2e09000
arch_prctl(ARCH_SET_FS, 0x7f61b2e09800) = 0
set_tid_address(0x7f61b2e09ad0) = 1654
set_robust_list(0x7f61b2e09ae0, 24) = 0
rseq(0x7f61b2e0a120, 0x20, 0, 0x53053053) = 0
mprotect(0x7f61b33d3000, 16384, PROT_READ) = 0
mprotect(0x7f61b2e35000, 4096, PROT_READ) = 0
mprotect(0x7f61b2ef6000, 4096, PROT_READ) = 0
mprotect(0x7f61b2f33000, 4096, PROT_READ) = 0
mprotect(0x7f61b2f5b000, 4096, PROT_READ) = 0
mprotect(0x7f61b309b000, 20480, PROT_READ) = 0
mprotect(0x7f61b30af000, 4096, PROT_READ) = 0
mprotect(0x7f61b3185000, 49152, PROT_READ) = 0
mprotect(0x7f61b31ba000, 4096, PROT_READ) = 0
mprotect(0x7f61b3202000, 4096, PROT_READ) = 0
mprotect(0x7f61b33e9000, 4096, PROT_READ) = 0
mprotect(0x7f61b33ef000, 4096, PROT_READ) = 0
mprotect(0x7f61b33f7000, 4096, PROT_READ) = 0
mprotect(0x7f61b33fd000, 4096, PROT_READ) = 0
mprotect(0x563052eed000, 4096, PROT_READ) = 0
mprotect(0x7f61b3435000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x7f61b3401000, 13065) = 0
getrandom("\xb3\xf1\xb9\xa1\x5c\xd8\xbe\xe2", 8, GRND_NONBLOCK) = 8
brk(NULL) = 0x563054c7d000
brk(0x563054c9e000) = 0x563054c9e000
prctl(PR_CAPBSET_READ, CAP_MAC_OVERRIDE) = 1
prctl(PR_CAPBSET_READ, 0x30 /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, CAP_CHECKPOINT_RESTORE) = 1
prctl(PR_CAPBSET_READ, 0x2c /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, 0x2a /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, 0x29 /* CAP_??? */) = -1 EINVAL (Invalid argument)
newfstatat(2, "", {st_mode=S_IFIFO|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0
write(2, "fsck.ocfs2 1.8.8\n", 17) = 17
newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0
openat(AT_FDCWD, "/proc/swaps", O_RDONLY) = 3
newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(3, "Filename\t\t\t\tType\t\tSize\t\tUsed\t\tPr"..., 1024) = 39
read(3, "", 1024) = 0
close(3) = 0
openat(AT_FDCWD, "/proc/mounts", O_RDONLY|O_CLOEXEC) = 3
newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0
newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(3, "sysfs /sys sysfs rw,nosuid,nodev"..., 1024) = 1024
newfstatat(AT_FDCWD, "sysfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "proc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "udev", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "devpts", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/dev/vda1", {st_mode=S_IFBLK|0660, st_rdev=makedev(0xfe, 0x1), ...}, 0) = 0
newfstatat(AT_FDCWD, "securityfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "cgroup2", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "pstore", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "bpf", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "systemd-1", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "hugetlbfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
read(3, "queue /dev/mqueue mqueue rw,nosu"..., 1024) = 616
newfstatat(AT_FDCWD, "mqueue", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tracefs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "debugfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "fusectl", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "configfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "binfmt_misc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "autopkgtest", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
read(3, "", 1024) = 0
newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
close(3) = 0
openat(AT_FDCWD, "/etc/mtab", O_RDONLY|O_CLOEXEC) = 3
newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0
newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(3, "sysfs /sys sysfs rw,nosuid,nodev"..., 1024) = 1024
newfstatat(AT_FDCWD, "sysfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "proc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "udev", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "devpts", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/dev/vda1", {st_mode=S_IFBLK|0660, st_rdev=makedev(0xfe, 0x1), ...}, 0) = 0
newfstatat(AT_FDCWD, "securityfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "cgroup2", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "pstore", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "bpf", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "systemd-1", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "hugetlbfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
read(3, "queue /dev/mqueue mqueue rw,nosu"..., 1024) = 616
newfstatat(AT_FDCWD, "mqueue", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tracefs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "debugfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "fusectl", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "configfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "binfmt_misc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "autopkgtest", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)
read(3, "", 1024) = 0
newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
close(3) = 0
newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0
openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_EXCL) = 3
close(3) = 0
rt_sigaction(SIGTERM, {sa_handler=0x563052e98e30, sa_mask=[TERM], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f61b3240510}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGINT, {sa_handler=0x563052e98e30, sa_mask=[INT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f61b3240510}, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=0}, 8) = 0 openat(AT_FDCWD, "/dev/loop0", O_RDWR|O_DIRECT) = 3
pread64(3, "\2\2\2\2\2\2\2\2this is an ocfs2 volume\0"..., 512, 0) = 512
uname({sysname="Linux", nodename="debian", ...}) = 0
ioctl(3, BLKROGET, [0]) = 0
pread64(3, "\2\2\2\2\2\2\2\2this is an ocfs2 volume\0"..., 512, 0) = 512
pread64(3, "\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2"..., 512, 1024) = 512
pread64(3, "OCFSV2\0\0L\23\t<\377\377\377\377\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 2048) = 1024
openat(AT_FDCWD, "/sys/fs/ocfs2/cluster_stack", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0644, st_size=4096, ...}, AT_EMPTY_PATH) = 0
read(4, "pcmk\n", 4096) = 5
close(4) = 0
openat(AT_FDCWD, "/sys/fs/o2cb/interface_revision", O_RDONLY) = 4

read(4, "5\n", 15)                      = 2
read(4, "", 13)                         = 0
close(4)                                = 0

newfstatat(AT_FDCWD, "/sys/kernel/config", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
statfs("/sys/kernel/config", {f_type=CONFIGFS_MAGIC, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0x20, 0]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0 rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0
openat(AT_FDCWD, "/dev/loop0", O_RDONLY) = 4
ioctl(4, BLKSSZGET, [512]) = 0
close(4) = 0
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\6\0\0\0\0\0@\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 14336) = 1024
openat(AT_FDCWD, "/sys/module/ocfs2_dlmfs/parameters/capabilities", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/proc/misc", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(4, "122 dlm_plock\n123 dlm-monitor\n12"..., 1024) = 220
close(4) = 0
newfstatat(AT_FDCWD, "/dev/misc/dlm-control", {st_mode=S_IFCHR|0666, st_rdev=makedev(0xa, 0x7c), ...}, 0) = 0
openat(AT_FDCWD, "/dev/misc/dlm-control", O_RDWR) = 4
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
read(4, "\6\0\0\0\0\0\0\0\2\0\0\0", 12) = 12
openat(AT_FDCWD, "/sys/fs/ocfs2/dlm_recover_callback_support", O_RDONLY) = 5
newfstatat(5, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0
read(5, "1\n", 4096) = 2
close(5) = 0
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 5

read(5, "\221cK\240s\342\265\212", 8)   = 8
close(5)                                = 0
brk(0x563054cc9000)                     = 0x563054cc9000
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0

write(4, "\6\0\0\0\0\0\0\0\2\0\0\0\4\1\0\0\0\0\0\0\0\0\0\00048E8C81F"..., 152) = 121
newfstatat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", 0x7ffdc4986d90, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/dev/misc/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5
newfstatat(5, "", {st_mode=S_IFDIR|0755, st_size=120, ...}, AT_EMPTY_PATH) = 0
getdents64(5, 0x563054ca9790 /* 6 entries */, 32768) = 200
newfstatat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", {st_mode=S_IFCHR|0660, st_rdev=makedev(0xa, 0x79), ...}, 0) = 0
close(5) = 0
openat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", O_RDWR) = 5
fchmod(5, 0600) = 0
fcntl(5, F_SETFD, FD_CLOEXEC) = 0
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\3\21\0\0\10\0\0\0\0\0\0\0\0\0\0\0"..., 137) = 1
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0h\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 104
openat(AT_FDCWD, "/sys/fs/ocfs2/dlm_recover_callback_support", O_RDONLY) = 6
newfstatat(6, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0
read(6, "1\n", 4096) = 2
close(6) = 0
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\5\37\0\0\t\0\0\0\0\0\0\0\0\0\0\0"..., 151) = 2
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
pread64(3, "INODE01\0L\23\t<\377\377\20\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 24576) = 1024
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\5\37\0\0\t\0\0\0\0\0\0\0\0\0\0\0"..., 151) = 3
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\0\0\0\0\0\0\10\0\0\0\3\0\0\0\0\0\0\0"..., 120) = 0
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
pread64(3, "INODE01\0L\23\t<\377\377\21\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 25600) = 1024
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\5\37\0\0\t\0\0\0\0\0\0\0\0\0\0\0"..., 151) = 4
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\0\0\0\0\0\0\10\0\0\0\4\0\0\0\0\0\0\0"..., 120) = 0
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72 rt_sigprocmask(SIG_UNBLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0
newfstatat(1, "", {st_mode=S_IFIFO|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0
write(1, "Checking OCFS2 filesystem in /de"..., 41) = 41

write(1, "  Label:              <NONE>\n", 29) = 29
write(1, "  UUID:               48E8C81F03"..., 55) = 55
write(1, "  Number of blocks:   204800\n", 29) = 29
write(1, "  Block size:         1024\n", 27) = 27
write(1, "  Number of clusters: 51200\n", 28) = 28
write(1, "  Cluster size:       4096\n", 27) = 27
write(1, "  Number of slots:    2\n\n", 25) = 25

sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=251826176, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0 mmap(NULL, 50368512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61afe00000
munmap(0x7f61afe00000, 50368512) = 0
mmap(NULL, 50368512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61afe00000
mmap(NULL, 3149824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61afa00000
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=251826176, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0
mlock(0x7f61afe00400, 50364416) = 0
mlock(0x7f61afa00010, 3147776) = 0
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\20\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 24576) = 1024
pread64(3, "\300;9\230\0\0\0\4\0\0\0\0\0\0\4\0\0\0\20\0\0\0\0\1\0\0\0\1\0\0\0\1"..., 1024, 552960) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\21\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 25600) = 1024
pread64(3, "\300;9\230\0\0\0\4\0\0\0\0\0\0\4\0\0\0\20\0\0\0\0\1\0\0\0\1\0\0\0\1"..., 1024, 4747264) = 1024
brk(0x563054cea000) = 0x563054cea000
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=201494528, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0

munlock(0x7f61afe00400, 50364416)       = 0
munmap(0x7f61afe00000, 50368512)        = 0
munlock(0x7f61afa00010, 3147776)        = 0
munmap(0x7f61afa00000, 3149824)         = 0

mmap(NULL, 40300544, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b0600000
munmap(0x7f61b0600000, 40300544) = 0
mmap(NULL, 40300544, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b0600000
brk(0x563054f61000) = 0x563054f61000
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=220368896, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0

mlock(0x7f61b0600400, 40296448)         = 0
mlock(0x563054cd9650, 2518528)          = 0
munlock(0x7f61b0600400, 40296448)       = 0
munmap(0x7f61b0600000, 40300544)        = 0
munlock(0x563054cd9650, 2518528)        = 0
mmap(NULL, 20152320, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b1a00000
munmap(0x7f61b1a00000, 20152320)        = 0
brk(0x563056031000)                     = 0x563056031000
brk(0x563056165000)                     = 0x563056165000

sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=216174592, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0
mlock(0x563054cd9800, 20148224) = 0
mlock(0x563056010a90, 1259264) = 0
pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 26624) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\23\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 27648) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\24\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 28672) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\24\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 28672) = 1024
pread64(3, "INODE01\0L\23\t<\377\377\25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 29696) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 29696) = 1024
write(1, "/dev/loop0 was run with -f, chec"..., 42) = 42
write(1, "Pass 0a: Checking cluster alloca"..., 44) = 44
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33247}, ru_stime={tv_sec=0, tv_usec=546077}, ...}) = 0
pread64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
io_setup(7, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 7, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615a800, aio_nbytes=1024, aio_offset=4096}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615ac00, aio_nbytes=1024, aio_offset=31457280}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615b000, aio_nbytes=1024, aio_offset=62914560}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615b400, aio_nbytes=1024, aio_offset=94371840}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615b800, aio_nbytes=1024, aio_offset=125829120}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615bc00, aio_nbytes=1024, aio_offset=157286400}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615c000, aio_nbytes=1024, aio_offset=188743680}]) = 7 io_pgetevents(0x7f61b3404000, 7, 7, [{data=0, obj=0x5630561553b0, res=1024, res2=0}, {data=0, obj=0x563056155370, res=1024, res2=0}, {data=0, obj=0x563056155330, res=1024, res2=0}, {data=0, obj=0x5630561552f0, res=1024, res2=0}, {data=0, obj=0x5630561552b0, res=1024, res2=0}, {data=0, obj=0x563056155270, res=1024, res2=0}, {data=0, obj=0x563056155230, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 7
io_destroy(0x7f61b3404000) = 0
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33261}, ru_stime={tv_sec=0, tv_usec=546303}, ...}) = 0
write(1, "Pass 0b: Checking inode allocati"..., 42) = 42
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33263}, ru_stime={tv_sec=0, tv_usec=546336}, ...}) = 0
pread64(3, "INODE01\0L\23\t<\377\377\4\0\0\0\0\0D\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 12288) = 1024
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056148800, aio_nbytes=1024, aio_offset=8192}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056158bb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
pread64(3, "INODE01\0L\23\t<\377\377\16\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 22528) = 1024
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056148800, aio_nbytes=1024, aio_offset=13172736}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056158bb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
brk(0x563056186000) = 0x563056186000
pread64(3, "INODE01\0L\23\t<\377\377\17\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 23552) = 1024
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33331}, ru_stime={tv_sec=0, tv_usec=547451}, ...}) = 0
write(1, "Pass 0c: Checking extent block a"..., 49) = 49
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33335}, ru_stime={tv_sec=0, tv_usec=547517}, ...}) = 0
pread64(3, "INODE01\0L\23\t<\377\377\f\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 20480) = 1024
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305616b800, aio_nbytes=1024, aio_offset=8941568}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x56305616abb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
pread64(3, "INODE01\0L\23\t<\377\377\r\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 21504) = 1024
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305616b800, aio_nbytes=1024, aio_offset=11038720}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056172fb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33390}, ru_stime={tv_sec=0, tv_usec=548424}, ...}) = 0
write(1, "Pass 1: Checking inodes and bloc"..., 35) = 35
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33394}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
brk(0x563056597000) = 0x563056597000
pread64(3, "INODE01\0L\23\t<\377\377\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 277504, 9216) = 277504
pread64(3, "INODE01\0L\23\t<\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576, 13173760) = 1048576
pread64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1047552, 14222336) = 1047552
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=38873}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
write(1, "Pass 2: Checking directory entri"..., 35) = 35
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=38975}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
io_setup(8, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 8, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056176400, aio_nbytes=1024, aio_offset=9216}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056176800, aio_nbytes=1024, aio_offset=18432}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056176c00, aio_nbytes=1024, aio_offset=19456}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177000, aio_nbytes=1024, aio_offset=286720}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177400, aio_nbytes=1024, aio_offset=287744}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177800, aio_nbytes=1024, aio_offset=288768}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177c00, aio_nbytes=1024, aio_offset=289792}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056178000, aio_nbytes=1024, aio_offset=13173760}]) = 8 io_pgetevents(0x7f61b3404000, 8, 8, [{data=0, obj=0x563056154bb0, res=1024, res2=0}, {data=0, obj=0x563056154b70, res=1024, res2=0}, {data=0, obj=0x563056154b30, res=1024, res2=0}, {data=0, obj=0x563056154af0, res=1024, res2=0}, {data=0, obj=0x563056154ab0, res=1024, res2=0}, {data=0, obj=0x563056154a70, res=1024, res2=0}, {data=0, obj=0x563056154a30, res=1024, res2=0}, {data=0, obj=0x5630561549f0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 8
io_destroy(0x7f61b3404000) = 0
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056163400, aio_nbytes=1024, aio_offset=8192}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056172fb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056157400, aio_nbytes=1024, aio_offset=13172736}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056172fb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
write(1, "[DX_TREE_MISSING] Directory 10 i"..., 57) = 57
write(1, " y\n", 3) = 3
pwrite64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
pread64(3, "p$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13139968) = 1024
pread64(3, "q$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13152256) = 1024
io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305657f800, aio_nbytes=1024, aio_offset=8941568}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056158bb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1
io_destroy(0x7f61b3404000) = 0
pwrite64(3, "GROUP01\0\0\1\0\10\376\7\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 8941568) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\f\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 20480) = 1024
pwrite64(3, "DXDIR01\0\211\270\365\260\265\3\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024
pwrite64(3, "DXDIR01\0\237\27d\200\241\1\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pwrite64(3, "DXDIR01\0\262@\254\245\334\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\177\233\217\370\327\7\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\377\204g\24\336\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0u\236n\322\363\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\n4z\265\317\7\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\0b`\367\342\7\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\303\271\272\342\363\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\0172\365\275\t\t\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\233\214\21a\213\4\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\27\363,B\232\4\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\231\337}\24\31\10\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\200\264I|\6\4\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0q\6\316\373U\17\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\261C\321\320k\2\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0!\324',\362\10\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\343\20\n\340\232\f\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0T\214\2611\3\5\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\336aa\3229\5\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0k\274\244\3l\r\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0s\322D\305\275\32\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\334\304\302\352s\35\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\200+]\37\254\r\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\241\320\204\4=\33\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\25\234Q\5\5\33\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0i\252\342\7\31\34\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024
pread64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13140992) = 1024
pread64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13142016) = 1024
pread64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13143040) = 1024
pread64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13164544) = 1024
pread64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13165568) = 1024
pread64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024
pread64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13153280) = 1024
pread64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13154304) = 1024
pread64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13155328) = 1024
pread64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13168640) = 1024
pread64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13169664) = 1024
pread64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=44996}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
write(1, "Pass 3: Checking directory conne"..., 40) = 40
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45065}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45123}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
write(1, "Pass 4a: Checking for orphaned i"..., 38) = 38
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45237}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45351}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
write(1, "Pass 4b: Checking inodes link co"..., 37) = 37
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45395}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45426}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
write(1, "Pass 5: Checking quota informati"..., 35) = 35
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45474}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
brk(0x5630569a1000) = 0x5630569a1000
io_setup(7, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 7, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056196800, aio_nbytes=1024, aio_offset=4096}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056196c00, aio_nbytes=1024, aio_offset=31457280}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197000, aio_nbytes=1024, aio_offset=62914560}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197400, aio_nbytes=1024, aio_offset=94371840}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197800, aio_nbytes=1024, aio_offset=125829120}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197c00, aio_nbytes=1024, aio_offset=157286400}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056198000, aio_nbytes=1024, aio_offset=188743680}]) = 7 io_pgetevents(0x7f61b3404000, 7, 7, [{data=0, obj=0x56305616ffb0, res=1024, res2=0}, {data=0, obj=0x56305616ff70, res=1024, res2=0}, {data=0, obj=0x56305616ff30, res=1024, res2=0}, {data=0, obj=0x56305616fef0, res=1024, res2=0}, {data=0, obj=0x56305616feb0, res=1024, res2=0}, {data=0, obj=0x56305616fe70, res=1024, res2=0}, {data=0, obj=0x56305616fe30, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 7
io_destroy(0x7f61b3404000) = 0
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36r\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024
pwrite64(3, "p$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13139968) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13140992) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024
pwrite64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13165568) = 1024
pwrite64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13164544) = 1024
pwrite64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13143040) = 1024
pwrite64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13142016) = 1024
pwrite64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13140992) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024
pwrite64(3, "\300$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13144064) = 2048
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024
pwrite64(3, "\300$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13148160) = 2048
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36r\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024
pwrite64(3, "q$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13152256) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13153280) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024
pwrite64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13169664) = 1024
pwrite64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13168640) = 1024
pwrite64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13155328) = 1024
pwrite64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13154304) = 1024
pwrite64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13153280) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024
pwrite64(3, "\301$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13156352) = 2048
pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024
pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024
pwrite64(3, "\301$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13160448) = 2048
getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=55343}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
write(1, "All passes succeeded.\n", 22) = 22
write(1, "\n", 1) = 1
pwrite64(3, "OCFSV2\0\0L\23\t<\377\377\377\377\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 2048) = 1024
pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 13135872) = 4096
rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\355\376\20\0\0\0\10\0\0\0\2\0\0\0\377\377\377\377"..., 120) = 0
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\0\0 \0\230\304\10\0\0\0\1\0\0\0\310\16\355\376"..., 120) = 0
read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
newfstatat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", {st_mode=S_IFLNK|0777, st_size=39, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(5, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0xa, 0x79), ...}, AT_EMPTY_PATH) = 0
close(5) = 0
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
write(4, "\6\0\0\0\0\0\0\0\2\0\0\0\5\1\0\0\0\0\0\0y\0\0\0\4\0\0\0\0\0\0\0"..., 120) = -1 EBUSY (Device or resource busy)
openat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", O_RDWR) = 5
close(5) = 0
openat(AT_FDCWD, "/dev/loop0", O_RDONLY) = 5
ioctl(5, BLKSSZGET, [512]) = 0
close(5) = 0
openat(AT_FDCWD, "/sys/fs/ocfs2/dlm_recover_callback_support", O_RDONLY) = 5
newfstatat(5, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0
read(5, "1\n", 4096) = 2
close(5) = 0
rt_sigprocmask(SIG_UNBLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0
pwrite64(3, "p$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13139968) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024
pwrite64(3, "q$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13152256) = 1024
pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024

munlock(0x563054cd9800, 20148224)       = 0
munlock(0x563056010a90, 1259264)        = 0
close(3)                                = 0
exit_group(0)                           = ?

+++ exited with 0 +++

#9 lore raw

Hi,

Quoted 102 lines · Valentin Vidić
Quoted 4 lines · Alexander Aring

I step forward here, but still I am not able to set a cluster name. I hardcoded it in the kernel and "it seems" to work. How do you set the cluster name? It seems that setting the cluster name in mkfs is not the way to go.

I think the cluster name comes from corosync, here is the config I have there:

totem {
version: 2
cluster_name: debian
crypto_cipher: none
crypto_hash: none
}

logging {
        fileline: off
        to_stderr: yes
        to_logfile: yes
        logfile: /var/log/corosync/corosync.log
        to_syslog: yes
        debug: off
        logger_subsys {
                subsys: QUORUM
                debug: off
        }
}
quorum {
        provider: corosync_votequorum
}
nodelist {
        node {
                name: node1
                nodeid: 1
                ring0_addr: 127.0.0.1
        }
}

The status of the services is as follows:

# corosync-quorumtool
Quorum information
------------------
Date:             Tue Mar 12 07:49:18 2024
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          1
Ring ID:          1.5
Quorate:          Yes

Votequorum information
----------------------

Expected votes:   1
Highest expected: 1
Total votes:      1
Quorum:           1
Flags:            Quorate

Membership information
----------------------
Nodeid Votes Name
1 1 node1 (local)

# dlm_tool dump_config
daemon_debug=0
foreground=1 (cli option)
log_debug=0
protocol=detect
port=21064
debug_logfile=0
bind_all=0
mark=0
enable_fscontrol=0
enable_plock=1
plock_debug=0
plock_rate_limit=0
plock_ownership=0
drop_resources_time=10000
drop_resources_count=10
drop_resources_age=10000
post_join_delay=30
enable_fencing=1
enable_concurrent_fencing=0
enable_startup_fencing=1
repeat_failed_fencing=1
enable_quorum_fencing=1
enable_quorum_lockspace=1
enable_helper=1
help=-1
version=-1
Quoted 10 lines

I am able to run fsck and I see on "dlm_tool ls" a lockspace with a similar name you have.

The bad news is, I can't reproduce it. After fsck the lockspace releases successfully.

I see on your debug information that it uses dlm user space locks, in my case I am using dlm kernel locks and I am asking myself why. Can you tell me more about your cluster configuration?

Quoted 8 lines
Quoted 5 lines

Somehow in your case fsck is using the DLM user space locks functionality from libdlm (the user space DLM locking library) but in my case fsck goes over the ocfs2_fsdlm filesystem and then does DLM kernel locks.
I am asking myself how this is supposed to work.

I'm attaching the strace from fsck below, but it could be that ocfs2_dlmfs module is not loaded, so it falls back to user space locks?

I was able to reproduce the issue and sent a fix out. If you want to extend your debian testing using different approach (not libdlm) this can be done by:

mount -t ocfs2_dlmfs none /dlm

this will somehow do some UAPI that triggers dlm kernel locking (so far I see it). How it is intended to work and which combination works or not? I have no idea, all I can say is that you should avoid using the same lockspace "name" for libdlm user and kernel dlm locking. I need to recheck if we have a check for that, that might break stuff but should not be used in such a way anyway... and maybe in future there is no difference between kernel/user locks.

I am curious about your testing at debian cluster, do you also test gfs2?

- Alex

#10 lore raw
Quoted 1 line · Alexander Aring

I was able to reproduce the issue and sent a fix out.

Yay, thanks a lot for testing, I can confirm the patches fix the problem for me!

Quoted 4 lines

If you want to extend your debian testing using different approach (not libdlm) this can be done by:

mount -t ocfs2_dlmfs none /dlm

I tried that but fsck than fails to make a lock, so I think ocfs2_dlmfs is only relevant when used with o2cb cluster type instead of corosync.

Quoted 1 line

I am curious about your testing at debian cluster, do you also test gfs2?

Yes, a similar test for gfs2 would be this:

https://salsa.debian.org/ha-team/gfs2-utils/-/blob/master/debian/tests/corosync?ref_type=heads

It is not really a cluster test since it only runs on one KVM machine. But as a package test it works well because even for 1-node cluster all the code (userspace and kernel) still needs to work correctly.

Signature · 1 line
Valentin
Tested-by Valentin Vidić
#11 lore raw

Hi,

Quoted 7 lines · Valentin Vidić
Quoted 1 line · Alexander Aring

I was able to reproduce the issue and sent a fix out.

Yay, thanks a lot for testing, I can confirm the patches fix the problem for me!

Then you can reply to the mail with:

that would help to bring the fix upstream.

Quoted 8 lines
Quoted 4 lines

If you want to extend your debian testing using different approach (not libdlm) this can be done by:

mount -t ocfs2_dlmfs none /dlm

I tried that but fsck than fails to make a lock, so I think ocfs2_dlmfs is only relevant when used with o2cb cluster type instead of corosync.

Yes, I think you are right. It is a replacement for corosync/dlm_controld handling. It seems I had both setup working at the same time, but it still uses kernel DLM locking.

Quoted 11 lines
Quoted 1 line

I am curious about your testing at debian cluster, do you also test gfs2?

Yes, a similar test for gfs2 would be this:

https://salsa.debian.org/ha-team/gfs2-utils/-/blob/master/debian/tests/corosync?ref_type=heads

It is not really a cluster test since it only runs on one KVM machine. But as a package test it works well because even for 1-node cluster all the code (userspace and kernel) still needs to work correctly.

I see, I currently have plans/ideas to add namespace support into kernel DLM that might help you by extending your testing. The general idea would be that you can do something like:

...
ip netns exec node1 mount /dev/fake_shared_block /mnt/node1 ip netns exec node2 mount /dev/fake_shared_block /mnt/node2 ip netns exec node3 mount /dev/fake_shared_block /mnt/node3
...

I hope that somehow explains the idea behind it. Setup your cluster networking with namespaces, e.g. selftests of Linux kernel is doing it
[0] and the DLM lockspaces are separated by net namespaces*. It "should" run like a cluster with several machines, it is only to do more testing and we don't need to synchronize tests over the network. However it will _not_ replace real cluster testing but it is a starting point to make testing "simpler". Somewhere is even a big fat "software" switch you can run wireshark on it to see all cluster traffic going on.

* In theory I am only abusing net namespaces for separate DLM lockspaces, but I don't want to introduce another kind of namespace now. However net namespaces fit here because it is necessary for a per node networking setup anyway.

- Alex

[0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/net/icmp.sh

#12 lore raw

Hi,

Quoted 24 lines · Alexander Aring

Hi,

Quoted 7 lines · Valentin Vidić
Quoted 1 line · Alexander Aring

I was able to reproduce the issue and sent a fix out.

Yay, thanks a lot for testing, I can confirm the patches fix the problem for me!

Then you can reply to the mail with:

that would help to bring the fix upstream.

Quoted 5 lines
Quoted 4 lines

If you want to extend your debian testing using different approach (not libdlm) this can be done by:

mount -t ocfs2_dlmfs none /dlm

Quoted 7 lines
Quoted 3 lines

I tried that but fsck than fails to make a lock, so I think ocfs2_dlmfs is only relevant when used with o2cb cluster type instead of corosync.

Yes, I think you are right. It is a replacement for corosync/dlm_controld handling. It seems I had both setup working at the same time, but it still uses kernel DLM locking.

It also requires a ocfs2-tools tool at a specific location:

/sbin/ocfs2_hb_ctl

maybe this is your issue why it wasn't working. My kernel was complaining about it because it was missing in /sbin/

- Alex

#13 lore raw
Quoted 5 lines · Alexander Aring

Then you can reply to the mail with:

that would help to bring the fix upstream.

Sure, replied to the other thread.

Quoted 9 lines

I see, I currently have plans/ideas to add namespace support into kernel DLM that might help you by extending your testing. The general idea would be that you can do something like:

...
ip netns exec node1 mount /dev/fake_shared_block /mnt/node1 ip netns exec node2 mount /dev/fake_shared_block /mnt/node2 ip netns exec node3 mount /dev/fake_shared_block /mnt/node3
...

Yeah, that would definitely be cool. And thanks for the pointers, I will definitely try to use namespaces for some simpler corosync tests first.

Quoted 6 lines

It also requires a ocfs2-tools tool at a specific location:

/sbin/ocfs2_hb_ctl

maybe this is your issue why it wasn't working. My kernel was complaining about it because it was missing in /sbin/

That binary is already in the Debian package. I think the error fsck reported was something like "Could not create domain".

Signature · 1 line
Valentin