ocfs2 mount error
Hi,
I've started seeing the following mount error on recent kernels (6.7, 6.8) while the same script worked on older versions (6.1):
2024-03-10T21:04:30,623900+00:00 ocfs2: Registered cluster interface user 2024-03-10T21:04:30,627545+00:00 ocfs2: Unable to create the lockspace D6DCCE389FC84C8FB9E99ABE1D941825 (-17), because a ocfs2-tools program is running on this file system with the same name lockspace 2024-03-10T21:04:30,630607+00:00 (mount.ocfs2,1577,0):ocfs2_dlm_init:3357 ERROR: status = -17 2024-03-10T21:04:30,633153+00:00 (mount.ocfs2,1577,0):ocfs2_mount_volume:1791 ERROR: status = -17 2024-03-10T21:04:30,634947+00:00 (mount.ocfs2,1577,0):ocfs2_fill_super:1176 ERROR: status = -17
Script runs the commands in the following order:
mkfs.ocfs2
fsck.ocfs2
mount.ocfs2
fsck command tries to release the lockspace but it seems to be busy so it stays around and the mount command fails. Running dlm_tool leave manually removes it and after that the mount command works again.
A git bisect has led me to the following change:
61bed0baa4dba17dd06cdfe20481a580718d6c7c fs: dlm: use a non-static queue for callbacks
Could it be that some callback does not run correctly causing the lockspace to be busy? I have added some more debug info below including some logging of 5 kref_get and 4 kref_put that could be related.
# dlm_tool ls dlm lockspaces name D6DCCE389FC84C8FB9E99ABE1D941825 id 0x3a105115 flags 0x00000000 change member 1 joined 1 remove 0 failed 0 seq 1,1 members 1
# dlm_tool status D6DCCE389FC84C8FB9E99ABE1D941825 cluster nodeid 1 quorate 1 ring seq 5 5 daemon now 1831 fence_pid 0 node 1 M add 54 rem 0 fail 0 fence 0 at 0 0
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 0000000020c29d80 Name (len=31) "S000000000000000000000200000000" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000003cfd484e Name (len=17) ".5f5b44890bdfac21" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 00000000c8e96ac4 Name (len=31) "M0000000000000000000018fe4c9cda" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource 000000004c006d70 Name (len=31) "M0000000000000000000019fe4c9cda" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb ptr nodeid first_lkid flags !root_list_empty !recover_list_empty recover_locks_count len /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 0000000020c29d80 0 0 0 0 0 0 31 str S000000000000000000000200000000 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000003cfd484e 0 0 0 0 0 0 17 str .5f5b44890bdfac21 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 00000000c8e96ac4 0 0 0 0 0 0 31 str M0000000000000000000018fe4c9cda /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb 000000004c006d70 0 0 0 0 0 0 31 str M0000000000000000000019fe4c9cda /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_locks:id nodeid remid pid xid exflags flags sts grmode rqmode time_ms r_nodeid r_len r_name /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_queued_asts:lkb_id lkb_flags mode flags sb_status sb_flags
2024-03-10T21:04:29,431465+00:00 dlm: kref_get 1 2024-03-10T21:04:29,432585+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:29,434951+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:29,436839+00:00 Call Trace: 2024-03-10T21:04:29,437077+00:00 <TASK> 2024-03-10T21:04:29,437308+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:29,437653+00:00 add_lkb+0x2f/0x300 [dlm] 2024-03-10T21:04:29,438000+00:00 ? rsb_insert+0x85/0xd0 [dlm] 2024-03-10T21:04:29,438378+00:00 _grant_lock+0x111/0x170 [dlm] 2024-03-10T21:04:29,438756+00:00 do_request+0x71/0x90 [dlm] 2024-03-10T21:04:29,439112+00:00 _request_lock+0x19b/0x270 [dlm] 2024-03-10T21:04:29,439509+00:00 request_lock+0xc7/0x110 [dlm] 2024-03-10T21:04:29,439887+00:00 dlm_user_request+0x10d/0x370 [dlm] 2024-03-10T21:04:29,440304+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:29,440763+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:29,441161+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,441578+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,442016+00:00 device_write+0x5f1/0x790 [dlm] 2024-03-10T21:04:29,442413+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:29,442723+00:00 ? __x64_sys_fcntl+0x90/0xc0 2024-03-10T21:04:29,443082+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:29,443389+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:29,443729+00:00 ? exc_page_fault+0x73/0x170 2024-03-10T21:04:29,444086+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:29,453050+00:00 dlm: kref_get 1 2024-03-10T21:04:29,453556+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:29,454373+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:29,455462+00:00 Call Trace: 2024-03-10T21:04:29,455922+00:00 <TASK> 2024-03-10T21:04:29,456345+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:29,456903+00:00 dlm_user_add_ast+0x1fc/0x300 [dlm] 2024-03-10T21:04:29,457548+00:00 do_request+0x7b/0x90 [dlm] 2024-03-10T21:04:29,458128+00:00 _request_lock+0x19b/0x270 [dlm] 2024-03-10T21:04:29,458771+00:00 request_lock+0xc7/0x110 [dlm] 2024-03-10T21:04:29,459375+00:00 dlm_user_request+0x10d/0x370 [dlm] 2024-03-10T21:04:29,460021+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:29,460714+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:29,461363+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,462003+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,462659+00:00 device_write+0x5f1/0x790 [dlm] 2024-03-10T21:04:29,463268+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:29,463791+00:00 ? __x64_sys_fcntl+0x90/0xc0 2024-03-10T21:04:29,464359+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:29,464891+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:29,465440+00:00 ? exc_page_fault+0x73/0x170 2024-03-10T21:04:29,466005+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:29,475107+00:00 dlm: kref_get 1 2024-03-10T21:04:29,475617+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:29,476460+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:29,477545+00:00 Call Trace: 2024-03-10T21:04:29,478031+00:00 <TASK> 2024-03-10T21:04:29,478494+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:29,479094+00:00 dlm_user_request+0x299/0x370 [dlm] 2024-03-10T21:04:29,479787+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:29,480544+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:29,481195+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,481848+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,482518+00:00 device_write+0x5f1/0x790 [dlm] 2024-03-10T21:04:29,483140+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:29,483672+00:00 ? __x64_sys_fcntl+0x90/0xc0 2024-03-10T21:04:29,484254+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:29,484785+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:29,485380+00:00 ? exc_page_fault+0x73/0x170 2024-03-10T21:04:29,485954+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,416950+00:00 dlm: kref_get 1 2024-03-10T21:04:30,417698+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,418547+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,419629+00:00 Call Trace: 2024-03-10T21:04:30,420115+00:00 <TASK> 2024-03-10T21:04:30,420566+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,421157+00:00 find_lkb+0x4e/0xd0 [dlm] 2024-03-10T21:04:30,421758+00:00 dlm_user_unlock+0x6b/0x2f0 [dlm] 2024-03-10T21:04:30,422422+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,423140+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,423785+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,424404+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,425057+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,425606+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,426268+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,426857+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,427415+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,427969+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,428534+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,429098+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,429657+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,430211+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,440429+00:00 dlm: kref_put 1 2024-03-10T21:04:30,441102+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,441961+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,443065+00:00 Call Trace: 2024-03-10T21:04:30,443571+00:00 <TASK> 2024-03-10T21:04:30,444029+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,444618+00:00 _unlock_lock+0x73/0x110 [dlm] 2024-03-10T21:04:30,445257+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] 2024-03-10T21:04:30,445979+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] 2024-03-10T21:04:30,446641+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,447345+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,447979+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,448593+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,449460+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,450394+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,451160+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,451773+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,452316+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,452894+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,453491+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,454053+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,454621+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,455175+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,464305+00:00 dlm: kref_put 1 2024-03-10T21:04:30,464804+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,465649+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,466737+00:00 Call Trace: 2024-03-10T21:04:30,467220+00:00 <TASK> 2024-03-10T21:04:30,467684+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,468276+00:00 _unlock_lock+0x9e/0x110 [dlm] 2024-03-10T21:04:30,468912+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] 2024-03-10T21:04:30,469607+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] 2024-03-10T21:04:30,470261+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,470961+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,471611+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,472221+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,472867+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,473397+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,474055+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,474633+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,475179+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,475759+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,476322+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,476883+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,477442+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,478000+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,486964+00:00 dlm: kref_get 1 2024-03-10T21:04:30,487460+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,488304+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,489384+00:00 Call Trace: 2024-03-10T21:04:30,489869+00:00 <TASK> 2024-03-10T21:04:30,490320+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,490918+00:00 dlm_user_add_ast+0x1fc/0x300 [dlm] 2024-03-10T21:04:30,491609+00:00 _unlock_lock+0xbe/0x110 [dlm] 2024-03-10T21:04:30,492231+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] 2024-03-10T21:04:30,492914+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] 2024-03-10T21:04:30,493553+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,494238+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,494869+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,495479+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,496133+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,496660+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,497318+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,497898+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,498428+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,498978+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,499532+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,500086+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,500639+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,501192+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,510147+00:00 dlm: kref_put 1 2024-03-10T21:04:30,510651+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,511482+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,512564+00:00 Call Trace: 2024-03-10T21:04:30,513045+00:00 <TASK> 2024-03-10T21:04:30,513496+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,514081+00:00 __put_lkb+0x30/0x100 [dlm] 2024-03-10T21:04:30,514683+00:00 dlm_user_add_ast+0x1cc/0x300 [dlm] 2024-03-10T21:04:30,515337+00:00 _unlock_lock+0xbe/0x110 [dlm] 2024-03-10T21:04:30,515953+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] 2024-03-10T21:04:30,516633+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] 2024-03-10T21:04:30,517265+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,517957+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,518586+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,519199+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,519840+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,520365+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,521021+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,521586+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,522100+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,522654+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,523212+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,523779+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,524333+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,524888+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76
2024-03-10T21:04:30,533818+00:00 dlm: kref_put 1 2024-03-10T21:04:30,534317+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,535151+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,536247+00:00 Call Trace: 2024-03-10T21:04:30,536729+00:00 <TASK> 2024-03-10T21:04:30,537179+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,537755+00:00 __put_lkb+0x30/0x100 [dlm] 2024-03-10T21:04:30,538355+00:00 dlm_user_unlock+0x213/0x2f0 [dlm] 2024-03-10T21:04:30,539003+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,539698+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,540319+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,540927+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,541569+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,542093+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,542759+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,543327+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,543851+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,544395+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,544949+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,545500+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,546059+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,546621+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 2024-03-10T21:04:30,547274+00:00 RIP: 0033:0x7fa4d45f0ba0
Signature · 1 line
Valentin
Joseph Qi
maintainerHi,
It seems that you are using fsdlm.
But from your running commands, I don't see how fsdlm get involved. Could you please give more detail on how to reproduce it?
Thanks,
Joseph
Quoted 292 lines · Valentin Vidić
Hi,
I've started seeing the following mount error on recent kernels (6.7, 6.8) while the same script worked on older versions (6.1):
2024-03-10T21:04:30,623900+00:00 ocfs2: Registered cluster interface user 2024-03-10T21:04:30,627545+00:00 ocfs2: Unable to create the lockspace D6DCCE389FC84C8FB9E99ABE1D941825 (-17), because a ocfs2-tools program is running on this file system with the same name lockspace 2024-03-10T21:04:30,630607+00:00 (mount.ocfs2,1577,0):ocfs2_dlm_init:3357 ERROR: status = -17 2024-03-10T21:04:30,633153+00:00 (mount.ocfs2,1577,0):ocfs2_mount_volume:1791 ERROR: status = -17 2024-03-10T21:04:30,634947+00:00 (mount.ocfs2,1577,0):ocfs2_fill_super:1176 ERROR: status = -17Script runs the commands in the following order:
mkfs.ocfs2
fsck.ocfs2
mount.ocfs2fsck command tries to release the lockspace but it seems to be busy so it stays around and the mount command fails. Running
dlm_toolleave manually removes it and after that the mount command works again.A git bisect has led me to the following change:
61bed0baa4dba17dd06cdfe20481a580718d6c7cfs: dlm: use a non-static queue for callbacksCould it be that some callback does not run correctly causing the lockspace to be busy? I have added some more debug info below including some logging of 5
kref_getand 4kref_putthat could be related.# dlm_tool ls dlm lockspaces name D6DCCE389FC84C8FB9E99ABE1D941825 id 0x3a105115 flags 0x00000000 change member 1 joined 1 remove 0 failed 0 seq 1,1 members 1# dlm_tool status D6DCCE389FC84C8FB9E99ABE1D941825 cluster nodeid 1 quorate 1 ring seq 5 5 daemon now 1831 fence_pid 0 node 1 M add 54 rem 0 fail 0 fence 0 at 0 0/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource
0000000020c29d80Name (len=31) "S000000000000000000000200000000" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource000000003cfd484eName (len=17) ".5f5b44890bdfac21" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource00000000c8e96ac4Name (len=31) "M0000000000000000000018fe4c9cda" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Resource000000004c006d70Name (len=31) "M0000000000000000000019fe4c9cda" /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Master Copy
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:LVB: 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825: b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Granted Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Conversion Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825:Waiting Queue
/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb ptr nodeidfirst_lkidflags !root_list_empty !recover_list_emptyrecover_locks_countlen /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb0000000020c29d800 0 0 0 0 0 31 str S000000000000000000000200000000 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 6f ec 98 7c e3 55 00 00 ef fb ff 7f fe ff ff ff df 11 59 d4 a4 7f 00 00 60 39 dc 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb000000003cfd484e0 0 0 0 0 0 17 str.5f5b44890bdfac21/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb00000000c8e96ac40 0 0 0 0 0 31 str M0000000000000000000018fe4c9cda /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 68 df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:rsb000000004c006d700 0 0 0 0 0 31 str M0000000000000000000019fe4c9cda /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_all:lvb 1 32 00 00 00 00 00 00 00 00 00 7c df 7c e3 55 00 00 b0 01 6f d4 a4 7f 00 00 e0 46 9d 7c e3 55 00 00 /sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_locks:id nodeid remid pid xid exflags flags sts grmode rqmodetime_msr_nodeidr_lenr_name/sys/kernel/debug/dlm/D6DCCE389FC84C8FB9E99ABE1D941825_queued_asts:lkb_idlkb_flagsmode flagssb_statussb_flags2024-03-10T21:04:29,431465+00:00 dlm: kref_get 1 2024-03-10T21:04:29,432585+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:29,434951+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:29,436839+00:00 Call Trace: 2024-03-10T21:04:29,437077+00:00 <TASK> 2024-03-10T21:04:29,437308+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:29,437653+00:00 add_lkb+0x2f/0x300 [dlm] 2024-03-10T21:04:29,438000+00:00 ? rsb_insert+0x85/0xd0 [dlm] 2024-03-10T21:04:29,438378+00:00 _grant_lock+0x111/0x170 [dlm] 2024-03-10T21:04:29,438756+00:00 do_request+0x71/0x90 [dlm] 2024-03-10T21:04:29,439112+00:00 _request_lock+0x19b/0x270 [dlm] 2024-03-10T21:04:29,439509+00:00 request_lock+0xc7/0x110 [dlm] 2024-03-10T21:04:29,439887+00:00 dlm_user_request+0x10d/0x370 [dlm] 2024-03-10T21:04:29,440304+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:29,440763+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:29,441161+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,441578+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,442016+00:00 device_write+0x5f1/0x790 [dlm] 2024-03-10T21:04:29,442413+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:29,442723+00:00 ? __x64_sys_fcntl+0x90/0xc0 2024-03-10T21:04:29,443082+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:29,443389+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:29,443729+00:00 ? exc_page_fault+0x73/0x170 2024-03-10T21:04:29,444086+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x762024-03-10T21:04:29,453050+00:00 dlm: kref_get 1 2024-03-10T21:04:29,453556+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:29,454373+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:29,455462+00:00 Call Trace: 2024-03-10T21:04:29,455922+00:00 <TASK> 2024-03-10T21:04:29,456345+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:29,456903+00:00 dlm_user_add_ast+0x1fc/0x300 [dlm] 2024-03-10T21:04:29,457548+00:00 do_request+0x7b/0x90 [dlm] 2024-03-10T21:04:29,458128+00:00 _request_lock+0x19b/0x270 [dlm] 2024-03-10T21:04:29,458771+00:00 request_lock+0xc7/0x110 [dlm] 2024-03-10T21:04:29,459375+00:00 dlm_user_request+0x10d/0x370 [dlm] 2024-03-10T21:04:29,460021+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:29,460714+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:29,461363+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,462003+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,462659+00:00 device_write+0x5f1/0x790 [dlm] 2024-03-10T21:04:29,463268+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:29,463791+00:00 ? __x64_sys_fcntl+0x90/0xc0 2024-03-10T21:04:29,464359+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:29,464891+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:29,465440+00:00 ? exc_page_fault+0x73/0x170 2024-03-10T21:04:29,466005+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x762024-03-10T21:04:29,475107+00:00 dlm: kref_get 1 2024-03-10T21:04:29,475617+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:29,476460+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:29,477545+00:00 Call Trace: 2024-03-10T21:04:29,478031+00:00 <TASK> 2024-03-10T21:04:29,478494+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:29,479094+00:00 dlm_user_request+0x299/0x370 [dlm] 2024-03-10T21:04:29,479787+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:29,480544+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:29,481195+00:00 ? __pfx_fake_astfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,481848+00:00 ? __pfx_fake_bastfn+0x10/0x10 [dlm] 2024-03-10T21:04:29,482518+00:00 device_write+0x5f1/0x790 [dlm] 2024-03-10T21:04:29,483140+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:29,483672+00:00 ? __x64_sys_fcntl+0x90/0xc0 2024-03-10T21:04:29,484254+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:29,484785+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:29,485380+00:00 ? exc_page_fault+0x73/0x170 2024-03-10T21:04:29,485954+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x762024-03-10T21:04:30,416950+00:00 dlm: kref_get 1 2024-03-10T21:04:30,417698+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,418547+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,419629+00:00 Call Trace: 2024-03-10T21:04:30,420115+00:00 <TASK> 2024-03-10T21:04:30,420566+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,421157+00:00 find_lkb+0x4e/0xd0 [dlm] 2024-03-10T21:04:30,421758+00:00 dlm_user_unlock+0x6b/0x2f0 [dlm] 2024-03-10T21:04:30,422422+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,423140+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,423785+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,424404+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,425057+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,425606+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,426268+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,426857+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,427415+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,427969+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,428534+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,429098+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,429657+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,430211+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x762024-03-10T21:04:30,440429+00:00 dlm: kref_put 1 2024-03-10T21:04:30,441102+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,441961+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,443065+00:00 Call Trace: 2024-03-10T21:04:30,443571+00:00 <TASK> 2024-03-10T21:04:30,444029+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,444618+00:00 _unlock_lock+0x73/0x110 [dlm] 2024-03-10T21:04:30,445257+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] 2024-03-10T21:04:30,445979+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] 2024-03-10T21:04:30,446641+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,447345+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,447979+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,448593+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,449460+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,450394+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,451160+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,451773+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,452316+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,452894+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,453491+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,454053+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,454621+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,455175+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x762024-03-10T21:04:30,464305+00:00 dlm: kref_put 1 2024-03-10T21:04:30,464804+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,465649+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,466737+00:00 Call Trace: 2024-03-10T21:04:30,467220+00:00 <TASK> 2024-03-10T21:04:30,467684+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,468276+00:00 _unlock_lock+0x9e/0x110 [dlm] 2024-03-10T21:04:30,468912+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] 2024-03-10T21:04:30,469607+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] 2024-03-10T21:04:30,470261+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,470961+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,471611+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,472221+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,472867+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,473397+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,474055+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,474633+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,475179+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,475759+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,476322+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,476883+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,477442+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,478000+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x762024-03-10T21:04:30,486964+00:00 dlm: kref_get 1 2024-03-10T21:04:30,487460+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,488304+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,489384+00:00 Call Trace: 2024-03-10T21:04:30,489869+00:00 <TASK> 2024-03-10T21:04:30,490320+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,490918+00:00 dlm_user_add_ast+0x1fc/0x300 [dlm] 2024-03-10T21:04:30,491609+00:00 _unlock_lock+0xbe/0x110 [dlm] 2024-03-10T21:04:30,492231+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] 2024-03-10T21:04:30,492914+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] 2024-03-10T21:04:30,493553+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,494238+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,494869+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,495479+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,496133+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,496660+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,497318+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,497898+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,498428+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,498978+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,499532+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,500086+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,500639+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,501192+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x762024-03-10T21:04:30,510147+00:00 dlm: kref_put 1 2024-03-10T21:04:30,510651+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,511482+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,512564+00:00 Call Trace: 2024-03-10T21:04:30,513045+00:00 <TASK> 2024-03-10T21:04:30,513496+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,514081+00:00 __put_lkb+0x30/0x100 [dlm] 2024-03-10T21:04:30,514683+00:00 dlm_user_add_ast+0x1cc/0x300 [dlm] 2024-03-10T21:04:30,515337+00:00 _unlock_lock+0xbe/0x110 [dlm] 2024-03-10T21:04:30,515953+00:00 unlock_lock.constprop.0+0x55/0x90 [dlm] 2024-03-10T21:04:30,516633+00:00 dlm_user_unlock+0x13b/0x2f0 [dlm] 2024-03-10T21:04:30,517265+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,517957+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,518586+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,519199+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,519840+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,520365+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,521021+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,521586+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,522100+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,522654+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,523212+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,523779+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,524333+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,524888+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x762024-03-10T21:04:30,533818+00:00 dlm: kref_put 1 2024-03-10T21:04:30,534317+00:00 CPU: 0 PID: 1562 Comm: fsck.ocfs2 Not tainted 6.8.0-rc7+ #80 2024-03-10T21:04:30,535151+00:00 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 2024-03-10T21:04:30,536247+00:00 Call Trace: 2024-03-10T21:04:30,536729+00:00 <TASK> 2024-03-10T21:04:30,537179+00:00 dump_stack_lvl+0x43/0x60 2024-03-10T21:04:30,537755+00:00 __put_lkb+0x30/0x100 [dlm] 2024-03-10T21:04:30,538355+00:00 dlm_user_unlock+0x213/0x2f0 [dlm] 2024-03-10T21:04:30,539003+00:00 ? __kmalloc_node_track_caller+0x1a9/0x420 2024-03-10T21:04:30,539698+00:00 ? device_write+0x44/0x790 [dlm] 2024-03-10T21:04:30,540319+00:00 device_write+0x5bd/0x790 [dlm] 2024-03-10T21:04:30,540927+00:00 ? security_file_permission+0x32/0x50 2024-03-10T21:04:30,541569+00:00 vfs_write+0xec/0x470 2024-03-10T21:04:30,542093+00:00 ? syscall_exit_to_user_mode+0x81/0x220 2024-03-10T21:04:30,542759+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,543327+00:00 ksys_write+0x6b/0xf0 2024-03-10T21:04:30,543851+00:00 do_syscall_64+0x87/0x1a0 2024-03-10T21:04:30,544395+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,544949+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,545500+00:00 ? do_syscall_64+0x93/0x1a0 2024-03-10T21:04:30,546059+00:00 ? __irq_exit_rcu+0x3b/0xb0 2024-03-10T21:04:30,546621+00:00 entry_SYSCALL_64_after_hwframe+0x6e/0x76 2024-03-10T21:04:30,547274+00:00 RIP: 0033:0x7fa4d45f0ba0
Quoted 3 lines · Joseph Qi
It seems that you are using fsdlm.
But from your running commands, I don't see how fsdlm get involved. Could you please give more detail on how to reproduce it?
Sure, this is the original Pacemaker test script that failed:
https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads
Basically it starts Corosync and DLM services and than calls:
mkfs.ocfs2
fsck.ocfs2
mount.ocfs2
Signature · 1 line
Valentin
Joseph Qi
maintainerQuoted 7 lines · Valentin Vidić
Quoted 3 lines · Joseph Qi
It seems that you are using fsdlm.
But from your running commands, I don't see how fsdlm get involved. Could you please give more detail on how to reproduce it?Sure, this is the original Pacemaker test script that failed:
Quoted 8 lines
https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads
Basically it starts Corosync and DLM services and than calls:
mkfs.ocfs2
fsck.ocfs2
mount.ocfs2
Thanks for quick reply.
Now it confirms that it is using fsdlm.
Joseph
Heming Zhao
reviewerQuoted 19 lines · Joseph Qi
Quoted 7 lines · Valentin Vidić
Quoted 3 lines · Joseph Qi
It seems that you are using fsdlm.
But from your running commands, I don't see how fsdlm get involved. Could you please give more detail on how to reproduce it?Sure, this is the original Pacemaker test script that failed:
Quoted 8 lines
https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads
Basically it starts Corosync and DLM services and than calls:
mkfs.ocfs2
fsck.ocfs2
mount.ocfs2
Quoted 6 lines
Thanks for quick reply.
Now it confirms that it is using fsdlm.Joseph
It looks like I met the same issue one year ago. check [1] and search for "error 17." in this mail.
[1]: https://lore.kernel.org/ocfs2-devel/20230626150916.je3egonzb3crvbl5@p15/
If my memory is still right, this root cause of this issue is that fsck forgets to clear the lockspace before exiting. I will investigate the debian's issue and provide a fix.
Thanks,
Heming
Hi,
Quoted 6 lines · Valentin Vidić
Quoted 3 lines · Joseph Qi
It seems that you are using fsdlm.
But from your running commands, I don't see how fsdlm get involved. Could you please give more detail on how to reproduce it?
Quoted 8 lines
Sure, this is the original Pacemaker test script that failed:
https://salsa.debian.org/ha-team/ocfs2-tools/-/blob/master/debian/tests/pcmk?ref_type=heads
Basically it starts Corosync and DLM services and than calls:
mkfs.ocfs2
fsck.ocfs2
when I try doing your steps (btw: thanks for sharing) to run fsck.ocfs2 it tells me:
fsck.ocfs2: Unable to access cluster service while initializing the DLM
Looking into strace right before that I see a:
connect(4, {sa_family=AF_UNIX, sun_path=@"ocfs2_controld_sock"}, 22) = -1 ECONNREFUSED (Connection refused)
it's true I don't have ocfs2_controld running, after looking into ocfs2-tools source code, I declare it as not compilable with recent corosync versions.
Maybe somebody else can help so I can try to reproduce it?
btw: a "trace-cmd stream -e dlm" would be nice to debug this issue. After I look at your debugging information it would be nice to see if the device() read for ast callbacks are there.
- Alex
Quoted 5 lines · Alexander Aring
when I try doing your steps (btw: thanks for sharing) to run fsck.ocfs2 it tells me:
fsck.ocfs2: Unable to access cluster service while initializing the DLM
Quoted 8 lines
Looking into strace right before that I see a:
connect(4, {sa_family=AF_UNIX, sun_path=@"ocfs2_controld_sock"}, 22)= -1 ECONNREFUSED (Connection refused)it's true I don't have
ocfs2_controldrunning, after looking into ocfs2-tools source code, I declare it as not compilable with recent corosync versions.
Not sure about that, because I also don't have ocfs2_controld running. This is ocfs2-tools v1.8.8 with corosync v3.1.7. Perhaps cluster stack should be set with:
echo pcmk > /sys/fs/ocfs2/cluster_stack
Quoted 3 lines
btw: a "trace-cmd stream -e dlm" would be nice to debug this issue. After I look at your debugging information it would be nice to see if the
device()read for ast callbacks are there.
Here is a trace report for the fsck call:
cpus=1
fsck.ocfs2-1696 [000] 54.621075: dlm_lock_start: ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK res_name=2e39323539656662313866316162386331
fsck.ocfs2-1696 [000] 54.692487: dlm_lock_end: ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK error=0 res_name=2e39323539656662313866316162386331
fsck.ocfs2-1696 [000] 54.695447: dlm_ast: ls_id=2352585329 lkb_id=1 sb_flags= sb_status=0 res_name=2e39323539656662313866316162386331
fsck.ocfs2-1696 [000] 54.695636: dlm_lock_start: ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 54.766959: dlm_lock_end: ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 54.770765: dlm_ast: ls_id=2352585329 lkb_id=2 sb_flags= sb_status=0 res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 54.770821: dlm_lock_start: ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531
fsck.ocfs2-1696 [000] 54.845613: dlm_lock_end: ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
fsck.ocfs2-1696 [000] 54.848395: dlm_ast: ls_id=2352585329 lkb_id=3 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
fsck.ocfs2-1696 [000] 54.868951: dlm_unlock_start: ls_id=2352585329 lkb_id=3 flags=VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531
fsck.ocfs2-1696 [000] 54.958319: dlm_unlock_end: ls_id=2352585329 lkb_id=3 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531
fsck.ocfs2-1696 [000] 54.982397: dlm_ast: ls_id=2352585329 lkb_id=3 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031383937333737316531
fsck.ocfs2-1696 [000] 54.982450: dlm_lock_start: ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.056390: dlm_lock_end: ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.058946: dlm_ast: ls_id=2352585329 lkb_id=4 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.082546: dlm_unlock_start: ls_id=2352585329 lkb_id=4 flags=VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.182903: dlm_unlock_end: ls_id=2352585329 lkb_id=4 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.210128: dlm_ast: ls_id=2352585329 lkb_id=4 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.530687: dlm_unlock_start: ls_id=2352585329 lkb_id=2 flags=VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 55.625015: dlm_unlock_end: ls_id=2352585329 lkb_id=2 flags=VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 55.650054: dlm_ast: ls_id=2352585329 lkb_id=2 sb_flags= sb_status=-65538 res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 55.672033: dlm_unlock_start: ls_id=2352585329 lkb_id=1 flags=VALBLK res_name=2e39323539656662313866316162386331
fsck.ocfs2-1696 [000] 55.766844: dlm_unlock_end: ls_id=2352585329 lkb_id=1 flags=VALBLK error=0 res_name=2e39323539656662313866316162386331
fsck.ocfs2-1696 [000] 55.791829: dlm_ast: ls_id=2352585329 lkb_id=1 sb_flags= sb_status=-65538 res_name=2e39323539656662313866316162386331Signature · 1 line
Valentin
Hi,
Quoted 16 lines · Valentin Vidić
Quoted 5 lines · Alexander Aring
when I try doing your steps (btw: thanks for sharing) to run fsck.ocfs2 it tells me:
fsck.ocfs2: Unable to access cluster service while initializing the DLM
Quoted 8 lines
Looking into strace right before that I see a:
connect(4, {sa_family=AF_UNIX, sun_path=@"ocfs2_controld_sock"}, 22)= -1 ECONNREFUSED (Connection refused)it's true I don't have
ocfs2_controldrunning, after looking into ocfs2-tools source code, I declare it as not compilable with recent corosync versions.
Quoted 6 lines
Not sure about that, because I also don't have
ocfs2_controldrunning. This is ocfs2-tools v1.8.8 with corosync v3.1.7. Perhaps cluster stack should be set with:echo pcmk >
/sys/fs/ocfs2/cluster_stack
I step forward here, but still I am not able to set a cluster name. I hardcoded it in the kernel and "it seems" to work. How do you set the cluster name? It seems that setting the cluster name in mkfs is not the way to go.
I am able to run fsck and I see on "dlm_tool ls" a lockspace with a similar name you have.
The bad news is, I can't reproduce it. After fsck the lockspace releases successfully.
I see on your debug information that it uses dlm user space locks, in my case I am using dlm kernel locks and I am asking myself why. Can you tell me more about your cluster configuration?
Somehow in your case fsck is using the DLM user space locks functionality from libdlm (the user space DLM locking library) but in my case fsck goes over the ocfs2_fsdlm filesystem and then does DLM kernel locks.
I am asking myself how this is supposed to work.
Quoted 23 lines
Quoted 3 lines
btw: a "trace-cmd stream -e dlm" would be nice to debug this issue. After I look at your debugging information it would be nice to see if the
device()read for ast callbacks are there.Here is a trace report for the fsck call:
cpus=1
fsck.ocfs2-1696 [000] 54.621075: dlm_lock_start: ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK res_name=2e39323539656662313866316162386331 fsck.ocfs2-1696 [000] 54.692487: dlm_lock_end: ls_id=2352585329 lkb_id=1 mode=PR flags=VALBLK error=0 res_name=2e39323539656662313866316162386331 fsck.ocfs2-1696 [000] 54.695447: dlm_ast: ls_id=2352585329 lkb_id=1 sb_flags= sb_status=0 res_name=2e39323539656662313866316162386331 fsck.ocfs2-1696 [000] 54.695636: dlm_lock_start: ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030 fsck.ocfs2-1696 [000] 54.766959: dlm_lock_end: ls_id=2352585329 lkb_id=2 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030 fsck.ocfs2-1696 [000] 54.770765: dlm_ast: ls_id=2352585329 lkb_id=2 sb_flags= sb_status=0 res_name=53303030303030303030303030303030303030303030323030303030303030 fsck.ocfs2-1696 [000] 54.770821: dlm_lock_start: ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531 fsck.ocfs2-1696 [000] 54.845613: dlm_lock_end: ls_id=2352585329 lkb_id=3 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531 fsck.ocfs2-1696 [000] 54.848395: dlm_ast: ls_id=2352585329 lkb_id=3 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031383937333737316531 fsck.ocfs2-1696 [000] 54.868951: dlm_unlock_start: ls_id=2352585329 lkb_id=3 flags=VALBLK res_name=4d303030303030303030303030303030303030303031383937333737316531 fsck.ocfs2-1696 [000] 54.958319: dlm_unlock_end: ls_id=2352585329 lkb_id=3 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031383937333737316531 fsck.ocfs2-1696 [000] 54.982397: dlm_ast: ls_id=2352585329 lkb_id=3 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031383937333737316531 fsck.ocfs2-1696 [000] 54.982450: dlm_lock_start: ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531 fsck.ocfs2-1696 [000] 55.056390: dlm_lock_end: ls_id=2352585329 lkb_id=4 mode=EX flags=NOQUEUE|VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531 fsck.ocfs2-1696 [000] 55.058946: dlm_ast: ls_id=2352585329 lkb_id=4 sb_flags= sb_status=0 res_name=4d303030303030303030303030303030303030303031393937333737316531 fsck.ocfs2-1696 [000] 55.082546: dlm_unlock_start: ls_id=2352585329 lkb_id=4 flags=VALBLK res_name=4d303030303030303030303030303030303030303031393937333737316531
Quoted 9 lines
fsck.ocfs2-1696 [000] 55.182903: dlm_unlock_end: ls_id=2352585329 lkb_id=4 flags=VALBLK error=0 res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.210128: dlm_ast: ls_id=2352585329 lkb_id=4 sb_flags= sb_status=-65538 res_name=4d303030303030303030303030303030303030303031393937333737316531
fsck.ocfs2-1696 [000] 55.530687: dlm_unlock_start: ls_id=2352585329 lkb_id=2 flags=VALBLK res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 55.625015: dlm_unlock_end: ls_id=2352585329 lkb_id=2 flags=VALBLK error=0 res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 55.650054: dlm_ast: ls_id=2352585329 lkb_id=2 sb_flags= sb_status=-65538 res_name=53303030303030303030303030303030303030303030323030303030303030
fsck.ocfs2-1696 [000] 55.672033: dlm_unlock_start: ls_id=2352585329 lkb_id=1 flags=VALBLK res_name=2e39323539656662313866316162386331
fsck.ocfs2-1696 [000] 55.766844: dlm_unlock_end: ls_id=2352585329 lkb_id=1 flags=VALBLK error=0 res_name=2e39323539656662313866316162386331
fsck.ocfs2-1696 [000] 55.791829: dlm_ast: ls_id=2352585329 lkb_id=1 sb_flags= sb_status=-65538 res_name=2e39323539656662313866316162386331looks okay to me. For dlm user locks there is currently an issue with lvbarea, see [0]. Not sure if this is why you have such issues right now, it depends what fsck is doing with the lvb.
- Alex
Quoted 4 lines · Alexander Aring
I step forward here, but still I am not able to set a cluster name. I hardcoded it in the kernel and "it seems" to work. How do you set the cluster name? It seems that setting the cluster name in mkfs is not the way to go.
I think the cluster name comes from corosync, here is the config I have there:
totem {
version: 2
cluster_name: debian
crypto_cipher: none
crypto_hash: none
}
logging {
fileline: off
to_stderr: yes
to_logfile: yes
logfile: /var/log/corosync/corosync.log
to_syslog: yes
debug: off
logger_subsys {
subsys: QUORUM
debug: off
}
}
quorum {
provider: corosync_votequorum
}
nodelist {
node {
name: node1
nodeid: 1
ring0_addr: 127.0.0.1
}
}The status of the services is as follows:
# corosync-quorumtool Quorum information ------------------ Date: Tue Mar 12 07:49:18 2024 Quorum provider: corosync_votequorum Nodes: 1 Node ID: 1 Ring ID: 1.5 Quorate: Yes
Votequorum information
----------------------
Expected votes: 1 Highest expected: 1 Total votes: 1 Quorum: 1 Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
1 1 node1 (local)
# dlm_tool dump_config daemon_debug=0 foreground=1 (cli option) log_debug=0 protocol=detect port=21064 debug_logfile=0 bind_all=0 mark=0 enable_fscontrol=0 enable_plock=1 plock_debug=0 plock_rate_limit=0 plock_ownership=0 drop_resources_time=10000 drop_resources_count=10 drop_resources_age=10000 post_join_delay=30 enable_fencing=1 enable_concurrent_fencing=0 enable_startup_fencing=1 repeat_failed_fencing=1 enable_quorum_fencing=1 enable_quorum_lockspace=1 enable_helper=1 help=-1 version=-1
Quoted 10 lines
I am able to run fsck and I see on "
dlm_toolls" a lockspace with a similar name you have.The bad news is, I can't reproduce it. After fsck the lockspace releases successfully.
I see on your debug information that it uses dlm user space locks, in my case I am using dlm kernel locks and I am asking myself why. Can you tell me more about your cluster configuration?
Quoted 5 lines
Somehow in your case fsck is using the DLM user space locks functionality from libdlm (the user space DLM locking library) but in my case fsck goes over the
ocfs2_fsdlmfilesystem and then does DLM kernel locks.
I am asking myself how this is supposed to work.
I'm attaching the strace from fsck below, but it could be that ocfs2_dlmfs module is not loaded, so it falls back to user space locks?
Signature · 1 line
Valentin
execve("/usr/sbin/fsck.ocfs2", ["fsck.ocfs2", "-f", "-y", "/dev/loop0"], 0x7fff21dd7108 /* 27 vars */) = 0
brk(NULL) = 0x563054c7d000access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=13065, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 13065, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f61b3401000
close(3) = 0mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b33ff000 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdlm_lt.so.3", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=18424, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 20536, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33f9000 mmap(0x7f61b33fa000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f61b33fa000 mmap(0x7f61b33fc000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33fc000 mmap(0x7f61b33fd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33fd000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcmap.so.4", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=26672, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 28816, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33f1000 mmap(0x7f61b33f3000, 12288, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33f3000 mmap(0x7f61b33f6000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b33f6000 mmap(0x7f61b33f7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b33f7000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcom_err.so.2", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=18344, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 20552, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33eb000 mmap(0x7f61b33ed000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33ed000 mmap(0x7f61b33ee000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33ee000 mmap(0x7f61b33ef000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b33ef000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libaio.so.1t64", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=14184, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 16432, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b33e6000 mmap(0x7f61b33e7000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f61b33e7000 mmap(0x7f61b33e8000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33e8000 mmap(0x7f61b33e9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b33e9000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220x\2\0\0\0\0\0"..., 832) = 832 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1926256, ...}, AT_EMPTY_PATH) = 0pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 mmap(NULL, 1974096, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b3204000 mmap(0x7f61b322a000, 1396736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f61b322a000
mmap(0x7f61b337f000, 344064, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17b000) = 0x7f61b337f000 mmap(0x7f61b33d3000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1cf000) = 0x7f61b33d3000 mmap(0x7f61b33d9000, 53072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b33d9000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcorosync_common.so.4", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=13952, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 16400, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b31ff000 mmap(0x7f61b3200000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f61b3200000 mmap(0x7f61b3201000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b3201000 mmap(0x7f61b3202000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f61b3202000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libqb.so.100", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=165872, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b31fd000 mmap(NULL, 430816, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b3193000
mmap(0x7f61b319a000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f61b319a000 mmap(0x7f61b31b1000, 36864, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e000) = 0x7f61b31b1000 mmap(0x7f61b31ba000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x27000) = 0x7f61b31ba000
mmap(0x7f61b31bc000, 262880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b31bc000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libsystemd.so.0", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=918624, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 923208, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b30b1000
mmap(0x7f61b30c8000, 573440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f61b30c8000 mmap(0x7f61b3154000, 200704, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa3000) = 0x7f61b3154000 mmap(0x7f61b3185000, 53248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd3000) = 0x7f61b3185000
mmap(0x7f61b3192000, 1608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b3192000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200w\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=47288, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 45128, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b30a5000 mmap(0x7f61b30a8000, 20480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b30a8000 mmap(0x7f61b30ad000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f61b30ad000 mmap(0x7f61b30af000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x7f61b30af000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgcrypt.so.20", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=1336728, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 1340360, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2f5d000
mmap(0x7f61b2f6c000, 966656, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf000) = 0x7f61b2f6c000 mmap(0x7f61b3058000, 274432, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xfb000) = 0x7f61b3058000 mmap(0x7f61b309b000, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13d000) = 0x7f61b309b000
mmap(0x7f61b30a4000, 968, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f61b30a4000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/liblz4.so.1", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=149952, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 152032, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2f37000 mmap(0x7f61b2f3a000, 122880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f61b2f3a000 mmap(0x7f61b2f58000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7f61b2f58000 mmap(0x7f61b2f5b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f61b2f5b000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/liblzma.so.5", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=247800, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b2f35000 mmap(NULL, 245800, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2ef8000 mmap(0x7f61b2efc000, 167936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f61b2efc000 mmap(0x7f61b2f25000, 57344, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2d000) = 0x7f61b2f25000 mmap(0x7f61b2f33000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3b000) = 0x7f61b2f33000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libzstd.so.1", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=784376, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 786496, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2e37000 mmap(0x7f61b2e3c000, 679936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b2e3c000 mmap(0x7f61b2ee2000, 81920, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xab000) = 0x7f61b2ee2000 mmap(0x7f61b2ef6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xbe000) = 0x7f61b2ef6000
close(3) = 0openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgpg-error.so.0", O_RDONLY|O_CLOEXEC) = 3read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=161864, ...}, AT_EMPTY_PATH) = 0mmap(NULL, 164304, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f61b2e0e000
mmap(0x7f61b2e13000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f61b2e13000 mmap(0x7f61b2e2a000, 45056, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x7f61b2e2a000 mmap(0x7f61b2e35000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f61b2e35000
close(3) = 0mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b2e0c000mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b2e09000arch_prctl(ARCH_SET_FS, 0x7f61b2e09800) = 0set_tid_address(0x7f61b2e09ad0) = 1654set_robust_list(0x7f61b2e09ae0, 24) = 0rseq(0x7f61b2e0a120, 0x20, 0, 0x53053053) = 0mprotect(0x7f61b33d3000, 16384, PROT_READ) = 0mprotect(0x7f61b2e35000, 4096, PROT_READ) = 0mprotect(0x7f61b2ef6000, 4096, PROT_READ) = 0mprotect(0x7f61b2f33000, 4096, PROT_READ) = 0mprotect(0x7f61b2f5b000, 4096, PROT_READ) = 0mprotect(0x7f61b309b000, 20480, PROT_READ) = 0mprotect(0x7f61b30af000, 4096, PROT_READ) = 0mprotect(0x7f61b3185000, 49152, PROT_READ) = 0mprotect(0x7f61b31ba000, 4096, PROT_READ) = 0mprotect(0x7f61b3202000, 4096, PROT_READ) = 0mprotect(0x7f61b33e9000, 4096, PROT_READ) = 0mprotect(0x7f61b33ef000, 4096, PROT_READ) = 0mprotect(0x7f61b33f7000, 4096, PROT_READ) = 0mprotect(0x7f61b33fd000, 4096, PROT_READ) = 0mprotect(0x563052eed000, 4096, PROT_READ) = 0mprotect(0x7f61b3435000, 8192, PROT_READ) = 0prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0munmap(0x7f61b3401000, 13065) = 0getrandom("\xb3\xf1\xb9\xa1\x5c\xd8\xbe\xe2", 8, GRND_NONBLOCK) = 8
brk(NULL) = 0x563054c7d000
brk(0x563054c9e000) = 0x563054c9e000prctl(PR_CAPBSET_READ, CAP_MAC_OVERRIDE) = 1prctl(PR_CAPBSET_READ, 0x30 /* CAP_??? */) = -1 EINVAL (Invalid argument)prctl(PR_CAPBSET_READ, CAP_CHECKPOINT_RESTORE) = 1prctl(PR_CAPBSET_READ, 0x2c /* CAP_??? */) = -1 EINVAL (Invalid argument)prctl(PR_CAPBSET_READ, 0x2a /* CAP_??? */) = -1 EINVAL (Invalid argument)prctl(PR_CAPBSET_READ, 0x29 /* CAP_??? */) = -1 EINVAL (Invalid argument)newfstatat(2, "", {st_mode=S_IFIFO|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0write(2, "fsck.ocfs2 1.8.8\n", 17) = 17newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0openat(AT_FDCWD, "/proc/swaps", O_RDONLY) = 3newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0read(3, "Filename\t\t\t\tType\t\tSize\t\tUsed\t\tPr"..., 1024) = 39read(3, "", 1024) = 0
close(3) = 0openat(AT_FDCWD, "/proc/mounts", O_RDONLY|O_CLOEXEC) = 3newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0read(3, "sysfs /sys sysfs rw,nosuid,nodev"..., 1024) = 1024newfstatat(AT_FDCWD, "sysfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "proc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "udev", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "devpts", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "/dev/vda1", {st_mode=S_IFBLK|0660, st_rdev=makedev(0xfe, 0x1), ...}, 0) = 0newfstatat(AT_FDCWD, "securityfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "cgroup2", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "pstore", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "bpf", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "systemd-1", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "hugetlbfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)read(3, "queue /dev/mqueue mqueue rw,nosu"..., 1024) = 616newfstatat(AT_FDCWD, "mqueue", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "tracefs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "debugfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "fusectl", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "configfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "binfmt_misc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "autopkgtest", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)read(3, "", 1024) = 0newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
close(3) = 0openat(AT_FDCWD, "/etc/mtab", O_RDONLY|O_CLOEXEC) = 3newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0read(3, "sysfs /sys sysfs rw,nosuid,nodev"..., 1024) = 1024newfstatat(AT_FDCWD, "sysfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "proc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "udev", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "devpts", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "/dev/vda1", {st_mode=S_IFBLK|0660, st_rdev=makedev(0xfe, 0x1), ...}, 0) = 0newfstatat(AT_FDCWD, "securityfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "cgroup2", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "pstore", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "bpf", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "systemd-1", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "hugetlbfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)read(3, "queue /dev/mqueue mqueue rw,nosu"..., 1024) = 616newfstatat(AT_FDCWD, "mqueue", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "tracefs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "debugfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "fusectl", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "configfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "binfmt_misc", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "autopkgtest", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)newfstatat(AT_FDCWD, "tmpfs", 0x7ffdc498aab0, 0) = -1 ENOENT (No such file or directory)read(3, "", 1024) = 0newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
close(3) = 0newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}, 0) = 0openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_EXCL) = 3
close(3) = 0
rt_sigaction(SIGTERM, {sa_handler=0x563052e98e30, sa_mask=[TERM], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f61b3240510}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0 rt_sigaction(SIGINT, {sa_handler=0x563052e98e30, sa_mask=[INT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f61b3240510}, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=0}, 8) = 0 openat(AT_FDCWD, "/dev/loop0", O_RDWR|O_DIRECT) = 3pread64(3, "\2\2\2\2\2\2\2\2this is an ocfs2 volume\0"..., 512, 0) = 512uname({sysname="Linux", nodename="debian", ...}) = 0ioctl(3, BLKROGET, [0]) = 0pread64(3, "\2\2\2\2\2\2\2\2this is an ocfs2 volume\0"..., 512, 0) = 512pread64(3, "\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2\2"..., 512, 1024) = 512pread64(3, "OCFSV2\0\0L\23\t<\377\377\377\377\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 2048) = 1024openat(AT_FDCWD, "/sys/fs/ocfs2/cluster_stack", O_RDONLY) = 4newfstatat(4, "", {st_mode=S_IFREG|0644, st_size=4096, ...}, AT_EMPTY_PATH) = 0read(4, "pcmk\n", 4096) = 5
close(4) = 0openat(AT_FDCWD, "/sys/fs/o2cb/interface_revision", O_RDONLY) = 4
read(4, "5\n", 15) = 2 read(4, "", 13) = 0 close(4) = 0
newfstatat(AT_FDCWD, "/sys/kernel/config", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
statfs("/sys/kernel/config", {f_type=CONFIGFS_MAGIC, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0x20, 0]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NOSUID|ST_NODEV|ST_NOEXEC|ST_RELATIME}) = 0 rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0openat(AT_FDCWD, "/dev/loop0", O_RDONLY) = 4ioctl(4, BLKSSZGET, [512]) = 0
close(4) = 0pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024pread64(3, "INODE01\0L\23\t<\377\377\6\0\0\0\0\0@\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 14336) = 1024openat(AT_FDCWD, "/sys/module/ocfs2_dlmfs/parameters/capabilities", O_RDONLY) = -1 ENOENT (No such file or directory)openat(AT_FDCWD, "/proc/misc", O_RDONLY) = 4newfstatat(4, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0read(4, "122 dlm_plock\n123 dlm-monitor\n12"..., 1024) = 220
close(4) = 0newfstatat(AT_FDCWD, "/dev/misc/dlm-control", {st_mode=S_IFCHR|0666, st_rdev=makedev(0xa, 0x7c), ...}, 0) = 0openat(AT_FDCWD, "/dev/misc/dlm-control", O_RDWR) = 4fcntl(4, F_SETFD, FD_CLOEXEC) = 0read(4, "\6\0\0\0\0\0\0\0\2\0\0\0", 12) = 12openat(AT_FDCWD, "/sys/fs/ocfs2/dlm_recover_callback_support", O_RDONLY) = 5newfstatat(5, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0read(5, "1\n", 4096) = 2
close(5) = 0openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 5
read(5, "\221cK\240s\342\265\212", 8) = 8 close(5) = 0 brk(0x563054cc9000) = 0x563054cc9000 fcntl(4, F_SETFD, FD_CLOEXEC) = 0
write(4, "\6\0\0\0\0\0\0\0\2\0\0\0\4\1\0\0\0\0\0\0\0\0\0\00048E8C81F"..., 152) = 121newfstatat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", 0x7ffdc4986d90, 0) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/dev/misc/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5newfstatat(5, "", {st_mode=S_IFDIR|0755, st_size=120, ...}, AT_EMPTY_PATH) = 0getdents64(5, 0x563054ca9790 /* 6 entries */, 32768) = 200
newfstatat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", {st_mode=S_IFCHR|0660, st_rdev=makedev(0xa, 0x79), ...}, 0) = 0
close(5) = 0openat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", O_RDWR) = 5fchmod(5, 0600) = 0fcntl(5, F_SETFD, FD_CLOEXEC) = 0write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\3\21\0\0\10\0\0\0\0\0\0\0\0\0\0\0"..., 137) = 1read(5, "\6\0\0\0\0\0\0\0\2\0\0\0h\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 104openat(AT_FDCWD, "/sys/fs/ocfs2/dlm_recover_callback_support", O_RDONLY) = 6newfstatat(6, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0read(6, "1\n", 4096) = 2
close(6) = 0pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\5\37\0\0\t\0\0\0\0\0\0\0\0\0\0\0"..., 151) = 2read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72pread64(3, "INODE01\0L\23\t<\377\377\20\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 24576) = 1024write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\5\37\0\0\t\0\0\0\0\0\0\0\0\0\0\0"..., 151) = 3read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\0\0\0\0\0\0\10\0\0\0\3\0\0\0\0\0\0\0"..., 120) = 0read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72pread64(3, "INODE01\0L\23\t<\377\377\21\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 25600) = 1024write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\1\1\0\0\5\37\0\0\t\0\0\0\0\0\0\0\0\0\0\0"..., 151) = 4read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\0\0\0\0\0\0\10\0\0\0\4\0\0\0\0\0\0\0"..., 120) = 0read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72 rt_sigprocmask(SIG_UNBLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0newfstatat(1, "", {st_mode=S_IFIFO|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0write(1, "Checking OCFS2 filesystem in /de"..., 41) = 41
write(1, " Label: <NONE>\n", 29) = 29 write(1, " UUID: 48E8C81F03"..., 55) = 55 write(1, " Number of blocks: 204800\n", 29) = 29 write(1, " Block size: 1024\n", 27) = 27 write(1, " Number of clusters: 51200\n", 28) = 28 write(1, " Cluster size: 4096\n", 27) = 27 write(1, " Number of slots: 2\n\n", 25) = 25
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=251826176, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0 mmap(NULL, 50368512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61afe00000munmap(0x7f61afe00000, 50368512) = 0mmap(NULL, 50368512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61afe00000mmap(NULL, 3149824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61afa00000
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=251826176, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0mlock(0x7f61afe00400, 50364416) = 0mlock(0x7f61afa00010, 3147776) = 0pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024pread64(3, "INODE01\0L\23\t<\377\377\20\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 24576) = 1024pread64(3, "\300;9\230\0\0\0\4\0\0\0\0\0\0\4\0\0\0\20\0\0\0\0\1\0\0\0\1\0\0\0\1"..., 1024, 552960) = 1024pread64(3, "INODE01\0L\23\t<\377\377\21\0\0\0\0\0\0\4\0\0\0\0\0\0\0\0\0\0"..., 1024, 25600) = 1024pread64(3, "\300;9\230\0\0\0\4\0\0\0\0\0\0\4\0\0\0\20\0\0\0\0\1\0\0\0\1\0\0\0\1"..., 1024, 4747264) = 1024
brk(0x563054cea000) = 0x563054cea000
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=201494528, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0
munlock(0x7f61afe00400, 50364416) = 0 munmap(0x7f61afe00000, 50368512) = 0 munlock(0x7f61afa00010, 3147776) = 0 munmap(0x7f61afa00000, 3149824) = 0
mmap(NULL, 40300544, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b0600000munmap(0x7f61b0600000, 40300544) = 0mmap(NULL, 40300544, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b0600000
brk(0x563054f61000) = 0x563054f61000
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=220368896, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0
mlock(0x7f61b0600400, 40296448) = 0 mlock(0x563054cd9650, 2518528) = 0 munlock(0x7f61b0600400, 40296448) = 0 munmap(0x7f61b0600000, 40300544) = 0 munlock(0x563054cd9650, 2518528) = 0 mmap(NULL, 20152320, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f61b1a00000 munmap(0x7f61b1a00000, 20152320) = 0 brk(0x563056031000) = 0x563056031000 brk(0x563056165000) = 0x563056165000
sysinfo({uptime=52, loads=[29440, 8064, 2688], totalram=1024217088, freeram=216174592, sharedram=21729280, bufferram=13910016, totalswap=0, freeswap=0, procs=113, totalhigh=0, freehigh=0, mem_unit=1}) = 0mlock(0x563054cd9800, 20148224) = 0mlock(0x563056010a90, 1259264) = 0pread64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024pread64(3, "INODE01\0L\23\t<\377\377\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 26624) = 1024pread64(3, "INODE01\0L\23\t<\377\377\23\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 27648) = 1024pread64(3, "INODE01\0L\23\t<\377\377\24\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 28672) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\24\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 28672) = 1024pread64(3, "INODE01\0L\23\t<\377\377\25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 29696) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 29696) = 1024write(1, "/dev/loop0 was run with -f, chec"..., 42) = 42write(1, "Pass 0a: Checking cluster alloca"..., 44) = 44getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33247}, ru_stime={tv_sec=0, tv_usec=546077}, ...}) = 0pread64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024io_setup(7, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 7, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615a800, aio_nbytes=1024, aio_offset=4096}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615ac00, aio_nbytes=1024, aio_offset=31457280}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615b000, aio_nbytes=1024, aio_offset=62914560}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615b400, aio_nbytes=1024, aio_offset=94371840}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615b800, aio_nbytes=1024, aio_offset=125829120}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615bc00, aio_nbytes=1024, aio_offset=157286400}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305615c000, aio_nbytes=1024, aio_offset=188743680}]) = 7 io_pgetevents(0x7f61b3404000, 7, 7, [{data=0, obj=0x5630561553b0, res=1024, res2=0}, {data=0, obj=0x563056155370, res=1024, res2=0}, {data=0, obj=0x563056155330, res=1024, res2=0}, {data=0, obj=0x5630561552f0, res=1024, res2=0}, {data=0, obj=0x5630561552b0, res=1024, res2=0}, {data=0, obj=0x563056155270, res=1024, res2=0}, {data=0, obj=0x563056155230, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 7io_destroy(0x7f61b3404000) = 0getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33261}, ru_stime={tv_sec=0, tv_usec=546303}, ...}) = 0write(1, "Pass 0b: Checking inode allocati"..., 42) = 42getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33263}, ru_stime={tv_sec=0, tv_usec=546336}, ...}) = 0pread64(3, "INODE01\0L\23\t<\377\377\4\0\0\0\0\0D\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 12288) = 1024io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056148800, aio_nbytes=1024, aio_offset=8192}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056158bb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1io_destroy(0x7f61b3404000) = 0pread64(3, "INODE01\0L\23\t<\377\377\16\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 22528) = 1024io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056148800, aio_nbytes=1024, aio_offset=13172736}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056158bb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1io_destroy(0x7f61b3404000) = 0
brk(0x563056186000) = 0x563056186000pread64(3, "INODE01\0L\23\t<\377\377\17\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 23552) = 1024getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33331}, ru_stime={tv_sec=0, tv_usec=547451}, ...}) = 0write(1, "Pass 0c: Checking extent block a"..., 49) = 49getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33335}, ru_stime={tv_sec=0, tv_usec=547517}, ...}) = 0pread64(3, "INODE01\0L\23\t<\377\377\f\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 20480) = 1024io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305616b800, aio_nbytes=1024, aio_offset=8941568}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x56305616abb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1io_destroy(0x7f61b3404000) = 0pread64(3, "INODE01\0L\23\t<\377\377\r\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 21504) = 1024io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305616b800, aio_nbytes=1024, aio_offset=11038720}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056172fb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1io_destroy(0x7f61b3404000) = 0getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33390}, ru_stime={tv_sec=0, tv_usec=548424}, ...}) = 0write(1, "Pass 1: Checking inodes and bloc"..., 35) = 35getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=33394}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
brk(0x563056597000) = 0x563056597000pread64(3, "INODE01\0L\23\t<\377\377\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 277504, 9216) = 277504pread64(3, "INODE01\0L\23\t<\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576, 13173760) = 1048576pread64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1047552, 14222336) = 1047552getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=38873}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0write(1, "Pass 2: Checking directory entri"..., 35) = 35getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=38975}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0io_setup(8, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 8, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056176400, aio_nbytes=1024, aio_offset=9216}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056176800, aio_nbytes=1024, aio_offset=18432}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056176c00, aio_nbytes=1024, aio_offset=19456}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177000, aio_nbytes=1024, aio_offset=286720}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177400, aio_nbytes=1024, aio_offset=287744}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177800, aio_nbytes=1024, aio_offset=288768}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056177c00, aio_nbytes=1024, aio_offset=289792}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056178000, aio_nbytes=1024, aio_offset=13173760}]) = 8 io_pgetevents(0x7f61b3404000, 8, 8, [{data=0, obj=0x563056154bb0, res=1024, res2=0}, {data=0, obj=0x563056154b70, res=1024, res2=0}, {data=0, obj=0x563056154b30, res=1024, res2=0}, {data=0, obj=0x563056154af0, res=1024, res2=0}, {data=0, obj=0x563056154ab0, res=1024, res2=0}, {data=0, obj=0x563056154a70, res=1024, res2=0}, {data=0, obj=0x563056154a30, res=1024, res2=0}, {data=0, obj=0x5630561549f0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 8io_destroy(0x7f61b3404000) = 0io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056163400, aio_nbytes=1024, aio_offset=8192}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056172fb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1io_destroy(0x7f61b3404000) = 0io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056157400, aio_nbytes=1024, aio_offset=13172736}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056172fb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1io_destroy(0x7f61b3404000) = 0write(1, "[DX_TREE_MISSING] Directory 10 i"..., 57) = 57write(1, " y\n", 3) = 3pwrite64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024pread64(3, "p$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13139968) = 1024pread64(3, "q$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13152256) = 1024io_setup(1, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 1, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x56305657f800, aio_nbytes=1024, aio_offset=8941568}]) = 1
io_pgetevents(0x7f61b3404000, 1, 1, [{data=0, obj=0x563056158bb0, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 1io_destroy(0x7f61b3404000) = 0pwrite64(3, "GROUP01\0\0\1\0\10\376\7\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 8941568) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\f\0\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0"..., 1024, 20480) = 1024pwrite64(3, "DXDIR01\0\211\270\365\260\265\3\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "\n\0\0\0\0\0\0\0\20\0\1\2.\0\0\0\n\0\0\0\0\0\0\0\20\0\2\2..\0\0"..., 1024, 286720) = 1024pwrite64(3, "DXDIR01\0\237\27d\200\241\1\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pwrite64(3, "DXDIR01\0\262@\254\245\334\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\177\233\217\370\327\7\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\377\204g\24\336\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0u\236n\322\363\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\n4z\265\317\7\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024
pwrite64(3, "DXDIR01\0\0b`\367\342\7\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\303\271\272\342\363\0\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\0172\365\275\t\t\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\233\214\21a\213\4\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\27\363,B\232\4\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\231\337}\24\31\10\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\200\264I|\6\4\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0q\6\316\373U\17\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\261C\321\320k\2\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0!\324',\362\10\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\343\20\n\340\232\f\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0T\214\2611\3\5\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\336aa\3229\5\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0k\274\244\3l\r\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0s\322D\305\275\32\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\334\304\302\352s\35\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\200+]\37\254\r\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\241\320\204\4=\33\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0\25\234Q\5\5\33\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "DXDIR01\0i\252\342\7\31\34\0\0\0\0\1\0L\23\t<\35\"\0\0\0\0\0\0"..., 1024, 8942592) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\2\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 10240) = 1024pread64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13140992) = 1024pread64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13142016) = 1024pread64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13143040) = 1024pread64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13164544) = 1024pread64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13165568) = 1024pread64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024pread64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13153280) = 1024pread64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13154304) = 1024pread64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13155328) = 1024pread64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13168640) = 1024pread64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13169664) = 1024pread64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=44996}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0write(1, "Pass 3: Checking directory conne"..., 40) = 40getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45065}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45123}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0write(1, "Pass 4a: Checking for orphaned i"..., 38) = 38getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45237}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45351}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0write(1, "Pass 4b: Checking inodes link co"..., 37) = 37getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45395}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45426}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0write(1, "Pass 5: Checking quota informati"..., 35) = 35getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=45474}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0
brk(0x5630569a1000) = 0x5630569a1000io_setup(7, [0x7f61b3404000]) = 0
io_submit(0x7f61b3404000, 7, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056196800, aio_nbytes=1024, aio_offset=4096}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056196c00, aio_nbytes=1024, aio_offset=31457280}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197000, aio_nbytes=1024, aio_offset=62914560}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197400, aio_nbytes=1024, aio_offset=94371840}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197800, aio_nbytes=1024, aio_offset=125829120}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056197c00, aio_nbytes=1024, aio_offset=157286400}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x563056198000, aio_nbytes=1024, aio_offset=188743680}]) = 7 io_pgetevents(0x7f61b3404000, 7, 7, [{data=0, obj=0x56305616ffb0, res=1024, res2=0}, {data=0, obj=0x56305616ff70, res=1024, res2=0}, {data=0, obj=0x56305616ff30, res=1024, res2=0}, {data=0, obj=0x56305616fef0, res=1024, res2=0}, {data=0, obj=0x56305616feb0, res=1024, res2=0}, {data=0, obj=0x56305616fe70, res=1024, res2=0}, {data=0, obj=0x56305616fe30, res=1024, res2=0}], NULL, {sigmask=NULL, sigsetsize=8}) = 7io_destroy(0x7f61b3404000) = 0pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "GROUP01\0\300\3\0\36r\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024pwrite64(3, "p$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13139968) = 1024pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13140992) = 1024pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024pwrite64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13165568) = 1024pwrite64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13164544) = 1024pwrite64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13143040) = 1024pwrite64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13142016) = 1024pwrite64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13140992) = 1024pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13166592) = 1024pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\26\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 30720) = 1024pwrite64(3, "\300$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13144064) = 2048pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\27\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 31744) = 1024pwrite64(3, "\300$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13148160) = 2048pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "GROUP01\0\300\3\0\36r\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024pwrite64(3, "q$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13152256) = 1024pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13153280) = 1024pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024pwrite64(3, "\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13169664) = 1024pwrite64(3, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13168640) = 1024pwrite64(3, "\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13155328) = 1024pwrite64(3, "\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13154304) = 1024pwrite64(3, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13153280) = 1024pwrite64(3, "\0\0\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 13170688) = 1024pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\30\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 32768) = 1024pwrite64(3, "\301$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13156352) = 2048pwrite64(3, "GROUP01\0\300\3\0\36q\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024pwrite64(3, "GROUP01\0\300\3\0\36p\17\0\0L\23\t<\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 4096) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\7\0\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 15360) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\31\0\0\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 33792) = 1024pwrite64(3, "\301$\365\f\0\0\0\0\1\0\0\0\1\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2048, 13160448) = 2048getrusage(RUSAGE_SELF, {ru_utime={tv_sec=0, tv_usec=55343}, ru_stime={tv_sec=0, tv_usec=548492}, ...}) = 0write(1, "All passes succeeded.\n", 22) = 22write(1, "\n", 1) = 1pwrite64(3, "OCFSV2\0\0L\23\t<\377\377\377\377\0\0\0\0\0\310\0\0\0\0\0\0\0\0\0\0"..., 1024, 2048) = 1024pwrite64(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 13135872) = 4096rt_sigprocmask(SIG_BLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\355\376\20\0\0\0\10\0\0\0\2\0\0\0\377\377\377\377"..., 120) = 0read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72write(5, "\6\0\0\0\0\0\0\0\2\0\0\0\2\1\0\0 \0\230\304\10\0\0\0\1\0\0\0\310\16\355\376"..., 120) = 0read(5, "\6\0\0\0\0\0\0\0\2\0\0\0H\0\0\0\20\243?\263a\177\0\0\0\0\0\0\0\0\0\0"..., 104) = 72
newfstatat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", {st_mode=S_IFLNK|0777, st_size=39, ...}, AT_SYMLINK_NOFOLLOW) = 0newfstatat(5, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0xa, 0x79), ...}, AT_EMPTY_PATH) = 0
close(5) = 0fcntl(4, F_SETFD, FD_CLOEXEC) = 0write(4, "\6\0\0\0\0\0\0\0\2\0\0\0\5\1\0\0\0\0\0\0y\0\0\0\4\0\0\0\0\0\0\0"..., 120) = -1 EBUSY (Device or resource busy)openat(AT_FDCWD, "/dev/misc/dlm_48E8C81F039C4F5E9CC099C954C14D46", O_RDWR) = 5
close(5) = 0openat(AT_FDCWD, "/dev/loop0", O_RDONLY) = 5ioctl(5, BLKSSZGET, [512]) = 0
close(5) = 0openat(AT_FDCWD, "/sys/fs/ocfs2/dlm_recover_callback_support", O_RDONLY) = 5newfstatat(5, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0read(5, "1\n", 4096) = 2
close(5) = 0rt_sigprocmask(SIG_UNBLOCK, ~[TRAP SEGV RTMIN RT_1], NULL, 8) = 0pwrite64(3, "p$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13139968) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\10\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 16384) = 1024pwrite64(3, "q$\365\f\0\0\0\0\200:\t\0\200:\t\0\20'\0\0\7\0\0\0\0\0\0\0\6\0\0\0"..., 1024, 13152256) = 1024pwrite64(3, "INODE01\0L\23\t<\377\377\t\0\0\0\0\0\2\0\0\0\0\0\0\0\0\0\0\0"..., 1024, 17408) = 1024
munlock(0x563054cd9800, 20148224) = 0 munlock(0x563056010a90, 1259264) = 0 close(3) = 0 exit_group(0) = ?
+++ exited with 0 +++
Hi,
Quoted 102 lines · Valentin Vidić
Quoted 4 lines · Alexander Aring
I step forward here, but still I am not able to set a cluster name. I hardcoded it in the kernel and "it seems" to work. How do you set the cluster name? It seems that setting the cluster name in mkfs is not the way to go.
I think the cluster name comes from corosync, here is the config I have there:
totem {
version: 2
cluster_name: debian
crypto_cipher: none
crypto_hash: none
}logging { fileline: off to_stderr: yes to_logfile: yes logfile: /var/log/corosync/corosync.log to_syslog: yes debug: off logger_subsys { subsys: QUORUM debug: off } } quorum { provider: corosync_votequorum } nodelist { node { name: node1 nodeid: 1 ring0_addr: 127.0.0.1 } }The status of the services is as follows:
# corosync-quorumtool Quorum information ------------------ Date: Tue Mar 12 07:49:18 2024 Quorum provider: corosync_votequorum Nodes: 1 Node ID: 1 Ring ID: 1.5 Quorate: YesVotequorum information
----------------------Expected votes: 1 Highest expected: 1 Total votes: 1 Quorum: 1 Flags: QuorateMembership information
----------------------
Nodeid Votes Name
1 1 node1 (local)# dlm_tool dump_config daemon_debug=0 foreground=1 (cli option) log_debug=0 protocol=detect port=21064 debug_logfile=0 bind_all=0 mark=0 enable_fscontrol=0 enable_plock=1 plock_debug=0 plock_rate_limit=0 plock_ownership=0 drop_resources_time=10000 drop_resources_count=10 drop_resources_age=10000 post_join_delay=30 enable_fencing=1 enable_concurrent_fencing=0 enable_startup_fencing=1 repeat_failed_fencing=1 enable_quorum_fencing=1 enable_quorum_lockspace=1 enable_helper=1 help=-1 version=-1Quoted 10 lines
I am able to run fsck and I see on "
dlm_toolls" a lockspace with a similar name you have.The bad news is, I can't reproduce it. After fsck the lockspace releases successfully.
I see on your debug information that it uses dlm user space locks, in my case I am using dlm kernel locks and I am asking myself why. Can you tell me more about your cluster configuration?
Quoted 8 lines
Quoted 5 lines
Somehow in your case fsck is using the DLM user space locks functionality from libdlm (the user space DLM locking library) but in my case fsck goes over the
ocfs2_fsdlmfilesystem and then does DLM kernel locks.
I am asking myself how this is supposed to work.I'm attaching the strace from fsck below, but it could be that
ocfs2_dlmfsmodule is not loaded, so it falls back to user space locks?
I was able to reproduce the issue and sent a fix out. If you want to extend your debian testing using different approach (not libdlm) this can be done by:
mount -t ocfs2_dlmfs none /dlm
this will somehow do some UAPI that triggers dlm kernel locking (so far I see it). How it is intended to work and which combination works or not? I have no idea, all I can say is that you should avoid using the same lockspace "name" for libdlm user and kernel dlm locking. I need to recheck if we have a check for that, that might break stuff but should not be used in such a way anyway... and maybe in future there is no difference between kernel/user locks.
I am curious about your testing at debian cluster, do you also test gfs2?
- Alex
Quoted 1 line · Alexander Aring
I was able to reproduce the issue and sent a fix out.
Yay, thanks a lot for testing, I can confirm the patches fix the problem for me!
Quoted 4 lines
If you want to extend your debian testing using different approach (not libdlm) this can be done by:
mount -t
ocfs2_dlmfsnone /dlm
I tried that but fsck than fails to make a lock, so I think ocfs2_dlmfs is only relevant when used with o2cb cluster type instead of corosync.
Quoted 1 line
I am curious about your testing at debian cluster, do you also test gfs2?
Yes, a similar test for gfs2 would be this:
https://salsa.debian.org/ha-team/gfs2-utils/-/blob/master/debian/tests/corosync?ref_type=heads
It is not really a cluster test since it only runs on one KVM machine. But as a package test it works well because even for 1-node cluster all the code (userspace and kernel) still needs to work correctly.
Signature · 1 line
Valentin
Alexander Aring
Tested-by Valentin VidićHi,
Quoted 7 lines · Valentin Vidić
Quoted 1 line · Alexander Aring
I was able to reproduce the issue and sent a fix out.
Yay, thanks a lot for testing, I can confirm the patches fix the problem for me!
Then you can reply to the mail with:
that would help to bring the fix upstream.
Quoted 8 lines
Quoted 4 lines
If you want to extend your debian testing using different approach (not libdlm) this can be done by:
mount -t
ocfs2_dlmfsnone /dlmI tried that but fsck than fails to make a lock, so I think
ocfs2_dlmfsis only relevant when used with o2cb cluster type instead of corosync.
Yes, I think you are right. It is a replacement for corosync/dlm_controld handling. It seems I had both setup working at the same time, but it still uses kernel DLM locking.
Quoted 11 lines
Quoted 1 line
I am curious about your testing at debian cluster, do you also test gfs2?
Yes, a similar test for gfs2 would be this:
https://salsa.debian.org/ha-team/gfs2-utils/-/blob/master/debian/tests/corosync?ref_type=heads
It is not really a cluster test since it only runs on one KVM machine. But as a package test it works well because even for 1-node cluster all the code (userspace and kernel) still needs to work correctly.
I see, I currently have plans/ideas to add namespace support into kernel DLM that might help you by extending your testing. The general idea would be that you can do something like:
...
ip netns exec node1 mount /dev/fake_shared_block /mnt/node1 ip netns exec node2 mount /dev/fake_shared_block /mnt/node2 ip netns exec node3 mount /dev/fake_shared_block /mnt/node3
...
I hope that somehow explains the idea behind it. Setup your cluster networking with namespaces, e.g. selftests of Linux kernel is doing it
[0] and the DLM lockspaces are separated by net namespaces*. It "should" run like a cluster with several machines, it is only to do more testing and we don't need to synchronize tests over the network. However it will _not_ replace real cluster testing but it is a starting point to make testing "simpler". Somewhere is even a big fat "software" switch you can run wireshark on it to see all cluster traffic going on.
* In theory I am only abusing net namespaces for separate DLM lockspaces, but I don't want to introduce another kind of namespace now. However net namespaces fit here because it is necessary for a per node networking setup anyway.
- Alex
Hi,
Quoted 24 lines · Alexander Aring
Hi,
Quoted 7 lines · Valentin Vidić
Quoted 1 line · Alexander Aring
I was able to reproduce the issue and sent a fix out.
Yay, thanks a lot for testing, I can confirm the patches fix the problem for me!
Then you can reply to the mail with:
Tested-by Valentin Vidić <[email protected]>that would help to bring the fix upstream.
Quoted 5 lines
Quoted 4 lines
If you want to extend your debian testing using different approach (not libdlm) this can be done by:
mount -t
ocfs2_dlmfsnone /dlm
Quoted 7 lines
Quoted 3 lines
I tried that but fsck than fails to make a lock, so I think
ocfs2_dlmfsis only relevant when used with o2cb cluster type instead of corosync.Yes, I think you are right. It is a replacement for corosync/dlm_controld handling. It seems I had both setup working at the same time, but it still uses kernel DLM locking.
It also requires a ocfs2-tools tool at a specific location:
/sbin/ocfs2_hb_ctl
maybe this is your issue why it wasn't working. My kernel was complaining about it because it was missing in /sbin/
- Alex
Quoted 5 lines · Alexander Aring
Then you can reply to the mail with:
Tested-by Valentin Vidić <[email protected]>that would help to bring the fix upstream.
Sure, replied to the other thread.
Quoted 9 lines
I see, I currently have plans/ideas to add namespace support into kernel DLM that might help you by extending your testing. The general idea would be that you can do something like:
...
ip netns exec node1 mount/dev/fake_shared_block/mnt/node1ip netns exec node2 mount/dev/fake_shared_block/mnt/node2ip netns exec node3 mount/dev/fake_shared_block/mnt/node3
...
Yeah, that would definitely be cool. And thanks for the pointers, I will definitely try to use namespaces for some simpler corosync tests first.
Quoted 6 lines
It also requires a ocfs2-tools tool at a specific location:
/sbin/ocfs2_hb_ctlmaybe this is your issue why it wasn't working. My kernel was complaining about it because it was missing in
/sbin/
That binary is already in the Debian package. I think the error fsck reported was something like "Could not create domain".
Signature · 1 line
Valentin