fix kernel crash after mounting when dlm doesn't ready
PATCH 1/1 fix kernel crash after mounting when dlm doesn't ready
How to trigger:
tb-ocfs1 # dd if=/dev/zero of=/dev/vdb bs=1M count=20 oflag=direct
tb-ocfs1 # mkfs.ocfs2 --cluster-stack=pcmk -N 4 /dev/vdb \
--cluster-name=ocfstst
== there is no dlm running ==
tb-ocfs1 # mount -t ocfs2 /dev/vdb /mnt
== kernel crash ==Crash log
``` kernel: DLM installed kernel: ocfs2: Registered cluster interface user kernel: dlm: no local IP address has been set kernel: dlm: cannot start dlm lowcomms -107 kernel: (mount.ocfs2,2225,0):ocfs2_dlm_init:3355 ERROR: status = -107 kernel: (mount.ocfs2,2225,0):ocfs2_mount_volume:1817 ERROR: status = -107 kernel: (mount.ocfs2,2225,0):ocfs2_fill_super:1186 ERROR: status = -107 kernel: BUG: kernel NULL pointer dereference, address: 0000000000000030 kernel: #PF: supervisor read access in kernel mode kernel: #PF: error_code(0x0000) - not-present page kernel: PGD 0 P4D 0 kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI kernel: CPU: 0 PID: 2225 Comm: mount.ocfs2 Not tainted 5.16.2-1-default #1 openSUSE Tumbleweed b40a195b7ff0f3399a616c3290f963c4ad189e84 kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014 kernel: RIP: 0010:ocfs2_clear_inode+0x2e9/0x720 [ocfs2] kernel: Code: 24 e8 9b 68 04 00 48 c7 c7 70 88 db c0 48 8b 80 98 03 00 00 48 8b 80 70 01 00 00 48 89 44 24 08 e8 3c 50 d8 d8 48 8b 44 24 08 <48> 8b 48 30 49 39 4f c8 0f 88 ff 00 00 00 48 c> kernel: RSP: 0018:ffffbbf000847bf0 EFLAGS: 00010246 kernel: RAX: 0000000000000000 RBX: ffff95f6834b8000 RCX: 0000000000000000 kernel: RDX: 0000000000000001 RSI: ffff95f6a1fbcbe0 RDI: ffffffffc0db8870 kernel: RBP: ffff95f6a1fbc6b8 R08: 00000ab5a9371b7a R09: 0000000000000230 kernel: R10: ffffbbf000847bc0 R11: ffffffffc0d53ea0 R12: ffff95f6a1fbc560 kernel: R13: ffff95f6a1fbc408 R14: ffff95f6834b8000 R15: ffff95f6a1fbc908 kernel: FS: 00007f366f151740(0000) GS:ffff95f6fdc00000(0000) knlGS:0000000000000000 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: CR2: 0000000000000030 CR3: 0000000003fcc004 CR4: 0000000000370ef0 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 kernel: Call Trace: kernel: <TASK> kernel: ? ocfs2_evict_inode+0x1fe/0x630 [ocfs2 411bc..281] kernel: evict+0xc0/0x1c0 kernel: ocfs2_release_system_inodes+0x21/0xc0 [ocfs2 411bc..281] kernel: ocfs2_dismount_volume+0x10b/0x2d0 [ocfs2 411bc..281] kernel: ocfs2_fill_super+0xaf/0x19e0 [ocfs2 411bc..281] kernel: mount_bdev+0x182/0x1b0 kernel: ? ocfs2_initialize_super.isra.0+0xf50/0xf50 [ocfs2 411bc..281] kernel: legacy_get_tree+0x24/0x40 kernel: vfs_get_tree+0x22/0xb0 kernel: path_mount+0x465/0xac0 kernel: __x64_sys_mount+0x103/0x140 kernel: do_syscall_64+0x59/0x80 kernel: ? syscall_exit_to_user_mode+0x18/0x40 kernel: ? do_syscall_64+0x69/0x80 kernel: ? syscall_exit_to_user_mode+0x18/0x40 kernel: ? do_syscall_64+0x69/0x80 kernel: ? exc_page_fault+0x68/0x150 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae ```
Analysis:
ocfs2_fill_super
ocfs2_mount_volume
ocfs2_dlm_init //failed, journal still doesn't be initized.
goto read_super_error
ocfs2_dismount_volume
ocfs2_release_system_inodes
...
evict
...
ocfs2_clear_inode
ocfs2_checkpoint_inode
ocfs2_ci_fully_checkpointed
time_after(journal->j_trans_id, ci->ci_last_trans)
+ journal is empty, crash!--- fs/ocfs2/inode.c | 3 ++- fs/ocfs2/journal.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
fs/ocfs2/inode.c +2 -1
@@ -1205,7 +1205,8 @@ static void ocfs2_clear_inode(struct inode *inode)* the journal is flushed before journal shutdown. Thus it is safe to * have inodes get cleaned up after journal shutdown. */ -jbd2_journal_release_jbd_inode (osb->journal->j_journal, +if (osb->journal) +jbd2_journal_release_jbd_inode (osb->journal->j_journal, &oi->ip_jinode); }
fs/ocfs2/journal.h +1 -1
@@ -189,7 +189,7 @@ static inline void ocfs2_checkpoint_inode(struct inode *inode) {struct ocfs2_super *osb =OCFS2_SB (inode->i_sb); -if (ocfs2_mount_local (osb)) +if (!osb->journal ||ocfs2_mount_local (osb))return ;if (!ocfs2_ci_fully_checkpointed (INODE_CACHE (inode))) {
Signature · 7 lines
2.34.1 _______________________________________________ Ocfs2-devel mailing list [email protected] https://oss.oracle.com/mailman/listinfo/ocfs2-devel
It looks this issue is introduced by the following commit:da5e7c87827e ocfs2: cleanup journal init and shutdown
Before that, journal was initialized in ocfs2_initialize_super().
Quoted 63 lines · Heming Zhao
How to trigger:
tb-ocfs1 # dd if=/dev/zero of=/dev/vdb bs=1M count=20 oflag=direct
tb-ocfs1 # mkfs.ocfs2 --cluster-stack=pcmk -N 4 /dev/vdb \ --cluster-name=ocfstst == there is no dlm running == tb-ocfs1 # mount -t ocfs2 /dev/vdb /mnt == kernel crash ==Crash log
``` kernel: DLM installed kernel: ocfs2: Registered cluster interface user kernel: dlm: no local IP address has been set kernel: dlm: cannot start dlm lowcomms -107 kernel: (mount.ocfs2,2225,0):ocfs2_dlm_init:3355 ERROR: status = -107 kernel: (mount.ocfs2,2225,0):ocfs2_mount_volume:1817 ERROR: status = -107 kernel: (mount.ocfs2,2225,0):ocfs2_fill_super:1186 ERROR: status = -107 kernel: BUG: kernel NULL pointer dereference, address: 0000000000000030 kernel: #PF: supervisor read access in kernel mode kernel: #PF: error_code(0x0000) - not-present page kernel: PGD 0 P4D 0 kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI kernel: CPU: 0 PID: 2225 Comm: mount.ocfs2 Not tainted 5.16.2-1-default #1 openSUSE Tumbleweed b40a195b7ff0f3399a616c3290f963c4ad189e84 kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014 kernel: RIP: 0010:ocfs2_clear_inode+0x2e9/0x720 [ocfs2] kernel: Code: 24 e8 9b 68 04 00 48 c7 c7 70 88 db c0 48 8b 80 98 03 00 00 48 8b 80 70 01 00 00 48 89 44 24 08 e8 3c 50 d8 d8 48 8b 44 24 08 <48> 8b 48 30 49 39 4f c8 0f 88 ff 00 00 00 48 c> kernel: RSP: 0018:ffffbbf000847bf0 EFLAGS: 00010246 kernel: RAX: 0000000000000000 RBX: ffff95f6834b8000 RCX: 0000000000000000 kernel: RDX: 0000000000000001 RSI: ffff95f6a1fbcbe0 RDI: ffffffffc0db8870 kernel: RBP: ffff95f6a1fbc6b8 R08: 00000ab5a9371b7a R09: 0000000000000230 kernel: R10: ffffbbf000847bc0 R11: ffffffffc0d53ea0 R12: ffff95f6a1fbc560 kernel: R13: ffff95f6a1fbc408 R14: ffff95f6834b8000 R15: ffff95f6a1fbc908 kernel: FS: 00007f366f151740(0000) GS:ffff95f6fdc00000(0000) knlGS:0000000000000000 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: CR2: 0000000000000030 CR3: 0000000003fcc004 CR4: 0000000000370ef0 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 kernel: Call Trace: kernel: <TASK> kernel: ? ocfs2_evict_inode+0x1fe/0x630 [ocfs2 411bc..281] kernel: evict+0xc0/0x1c0 kernel: ocfs2_release_system_inodes+0x21/0xc0 [ocfs2 411bc..281] kernel: ocfs2_dismount_volume+0x10b/0x2d0 [ocfs2 411bc..281] kernel: ocfs2_fill_super+0xaf/0x19e0 [ocfs2 411bc..281] kernel: mount_bdev+0x182/0x1b0 kernel: ? ocfs2_initialize_super.isra.0+0xf50/0xf50 [ocfs2 411bc..281] kernel: legacy_get_tree+0x24/0x40 kernel: vfs_get_tree+0x22/0xb0 kernel: path_mount+0x465/0xac0 kernel: __x64_sys_mount+0x103/0x140 kernel: do_syscall_64+0x59/0x80 kernel: ? syscall_exit_to_user_mode+0x18/0x40 kernel: ? do_syscall_64+0x69/0x80 kernel: ? syscall_exit_to_user_mode+0x18/0x40 kernel: ? do_syscall_64+0x69/0x80 kernel: ? exc_page_fault+0x68/0x150 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae ```
Quoted 6 lines
Analysis:
ocfs2_fill_super
ocfs2_mount_volume
ocfs2_dlm_init//failed, journal still doesn't be initized.
gotoread_super_error
This is not a part of call trace, I don't think it is needed here.
Quoted 11 lines
ocfs2_dismount_volume
ocfs2_release_system_inodes
...
evict
...
ocfs2_clear_inode
ocfs2_checkpoint_inode
ocfs2_ci_fully_checkpointed
time_after(journal->j_trans_id, ci->ci_last_trans)
+ journal is empty, crash!Suggest we use the following way to description this commit:
<call trace>
<reproducer>
<your analysis and how to fix>
BTW, as I mentioned at the first, a fixes tag should be mentioned here. Could you also please check all possible use of journal during ocfs2_dismount_volume()?
Thanks,
Joseph
Quoted 25 lines
Signed-off-by Heming Zhao <[email protected]>--- fs/ocfs2/inode.c | 3 ++- fs/ocfs2/journal.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)fs/ocfs2/inode.c +2
-1@@ -1205,7 +1205,8 @@ static void ocfs2_clear_inode(struct inode *inode)* the journal is flushed before journal shutdown. Thus it is safe to * have inodes get cleaned up after journal shutdown. */ -jbd2_journal_release_jbd_inode (osb->journal->j_journal, +if (osb->journal) +jbd2_journal_release_jbd_inode (osb->journal->j_journal, &oi->ip_jinode); }fs/ocfs2/journal.h +0
-0
fs/ocfs2/journal.h static inline void ocfs2_checkpoint_inode(struct inode *inode) · line 195
Quoted 8 lines
fs/ocfs2/journal.h +1
-1@@ -189,7 +189,7 @@ static inline void ocfs2_checkpoint_inode(struct inode *inode) {struct ocfs2_super *osb =OCFS2_SB (inode->i_sb); -if (ocfs2_mount_local (osb)) +if (!osb->journal ||ocfs2_mount_local (osb))return ;if (!ocfs2_ci_fully_checkpointed (INODE_CACHE (inode))) {
_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
Hello Joseph,
Thank you for your review.
Quoted 4 lines · Joseph Qi
It looks this issue is introduced by the following commit:
da5e7c87827eocfs2: cleanup journal init and shutdownBefore that, journal was initialized in
ocfs2_initialize_super().
I agree, and will add Fixes in V2.
Quoted 65 lines · Heming Zhao
Quoted 63 lines
How to trigger:
tb-ocfs1 # dd if=/dev/zero of=/dev/vdb bs=1M count=20 oflag=direct
tb-ocfs1 # mkfs.ocfs2 --cluster-stack=pcmk -N 4 /dev/vdb \ --cluster-name=ocfstst == there is no dlm running == tb-ocfs1 # mount -t ocfs2 /dev/vdb /mnt == kernel crash ==Crash log
``` kernel: DLM installed kernel: ocfs2: Registered cluster interface user kernel: dlm: no local IP address has been set kernel: dlm: cannot start dlm lowcomms -107 kernel: (mount.ocfs2,2225,0):ocfs2_dlm_init:3355 ERROR: status = -107 kernel: (mount.ocfs2,2225,0):ocfs2_mount_volume:1817 ERROR: status = -107 kernel: (mount.ocfs2,2225,0):ocfs2_fill_super:1186 ERROR: status = -107 kernel: BUG: kernel NULL pointer dereference, address: 0000000000000030 kernel: #PF: supervisor read access in kernel mode kernel: #PF: error_code(0x0000) - not-present page kernel: PGD 0 P4D 0 kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI kernel: CPU: 0 PID: 2225 Comm: mount.ocfs2 Not tainted 5.16.2-1-default #1 openSUSE Tumbleweed b40a195b7ff0f3399a616c3290f963c4ad189e84 kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a-rebuilt.opensuse.org 04/01/2014 kernel: RIP: 0010:ocfs2_clear_inode+0x2e9/0x720 [ocfs2] kernel: Code: 24 e8 9b 68 04 00 48 c7 c7 70 88 db c0 48 8b 80 98 03 00 00 48 8b 80 70 01 00 00 48 89 44 24 08 e8 3c 50 d8 d8 48 8b 44 24 08 <48> 8b 48 30 49 39 4f c8 0f 88 ff 00 00 00 48 c> kernel: RSP: 0018:ffffbbf000847bf0 EFLAGS: 00010246 kernel: RAX: 0000000000000000 RBX: ffff95f6834b8000 RCX: 0000000000000000 kernel: RDX: 0000000000000001 RSI: ffff95f6a1fbcbe0 RDI: ffffffffc0db8870 kernel: RBP: ffff95f6a1fbc6b8 R08: 00000ab5a9371b7a R09: 0000000000000230 kernel: R10: ffffbbf000847bc0 R11: ffffffffc0d53ea0 R12: ffff95f6a1fbc560 kernel: R13: ffff95f6a1fbc408 R14: ffff95f6834b8000 R15: ffff95f6a1fbc908 kernel: FS: 00007f366f151740(0000) GS:ffff95f6fdc00000(0000) knlGS:0000000000000000 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: CR2: 0000000000000030 CR3: 0000000003fcc004 CR4: 0000000000370ef0 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 kernel: Call Trace: kernel: <TASK> kernel: ? ocfs2_evict_inode+0x1fe/0x630 [ocfs2 411bc..281] kernel: evict+0xc0/0x1c0 kernel: ocfs2_release_system_inodes+0x21/0xc0 [ocfs2 411bc..281] kernel: ocfs2_dismount_volume+0x10b/0x2d0 [ocfs2 411bc..281] kernel: ocfs2_fill_super+0xaf/0x19e0 [ocfs2 411bc..281] kernel: mount_bdev+0x182/0x1b0 kernel: ? ocfs2_initialize_super.isra.0+0xf50/0xf50 [ocfs2 411bc..281] kernel: legacy_get_tree+0x24/0x40 kernel: vfs_get_tree+0x22/0xb0 kernel: path_mount+0x465/0xac0 kernel: __x64_sys_mount+0x103/0x140 kernel: do_syscall_64+0x59/0x80 kernel: ? syscall_exit_to_user_mode+0x18/0x40 kernel: ? do_syscall_64+0x69/0x80 kernel: ? syscall_exit_to_user_mode+0x18/0x40 kernel: ? do_syscall_64+0x69/0x80 kernel: ? exc_page_fault+0x68/0x150 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae ```
Quoted 7 lines
Quoted 6 lines
Analysis:
ocfs2_fill_super
ocfs2_mount_volume
ocfs2_dlm_init//failed, journal still doesn't be initized.
gotoread_super_errorThis is not a part of call trace, I don't think it is needed here.
OK, will drop in v2.
Quoted 13 lines
Quoted 12 lines
ocfs2_dismount_volume ocfs2_release_system_inodes ... evict ... ocfs2_clear_inode ocfs2_checkpoint_inode ocfs2_ci_fully_checkpointed time_after(journal->j_trans_id, ci->ci_last_trans) + journal is empty, crash!
Quoted 4 lines
Suggest we use the following way to description this commit:
<call trace>
<reproducer>
<your analysis and how to fix>
OK, will follow this style.
Quoted 4 lines
BTW, as I mentioned at the first, a fixes tag should be mentioned here. Could you also please check all possible use of journal during
ocfs2_dismount_volume()?
I verified my patch before file V1, there was no crash anymore with no dlm case.
Under your comment, there may have other places need to fix in ocfs2_dismount_volume(). The reverse operations in ocfs2_dismount_volume() for range [ocfs2_initialize_super, \ocfs2_journal_init) should be checked.
Quoted 4 lines
Thanks,
Joseph
Thanks,
Heming
_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel