Skip to contentSkip to conversation

ocfs2: fix kernel crash after mounting when journal doesn't ready

PATCH Superseded Heming Zhao via Ocfs2-devel 1 participant 2 messages Mar 18, 2022 OCFS2

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 ocfs2: fix kernel crash after mounting when journal doesn't ready

Call trace:

ocfs2: Registered cluster interface user
dlm: no local IP address has been set
dlm: cannot start dlm lowcomms -107 (mount.ocfs2,2225,0):ocfs2_dlm_init:3355 ERROR: status = -107 (mount.ocfs2,2225,0):ocfs2_mount_volume:1817 ERROR: status = -107 (mount.ocfs2,2225,0):ocfs2_fill_super:1186 ERROR: status = -107

  BUG: kernel NULL pointer dereference, address: 0000000000000030
  #PF: supervisor read access in kernel mode
  #PF: error_code(0x0000) - not-present page
  PGD 0 P4D 0
  Oops: 0000 [#1] PREEMPT SMP NOPTI
  CPU: 0 PID: 2225 Comm: mount.ocfs2 Not tainted 5.16.2-1-default #1
openSUSE Tumbleweed b40...e84
  Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ...
  RIP: 0010:ocfs2_clear_inode+0x2e9/0x720 [ocfs2]
  ...
  Call Trace:
   <TASK>
   ? ocfs2_evict_inode+0x1fe/0x630 [ocfs2 411bc..281]
   evict+0xc0/0x1c0
   ocfs2_release_system_inodes+0x21/0xc0 [ocfs2 411bc..281]
   ocfs2_dismount_volume+0x10b/0x2d0 [ocfs2 411bc..281]
   ocfs2_fill_super+0xaf/0x19e0 [ocfs2 411bc..281]
   mount_bdev+0x182/0x1b0
   ? ocfs2_initialize_super.isra.0+0xf50/0xf50 [ocfs2 411bc..281]
   legacy_get_tree+0x24/0x40
   vfs_get_tree+0x22/0xb0
   path_mount+0x465/0xac0
   __x64_sys_mount+0x103/0x140
   do_syscall_64+0x59/0x80
   ? syscall_exit_to_user_mode+0x18/0x40
   ? do_syscall_64+0x69/0x80
   ? syscall_exit_to_user_mode+0x18/0x40
   ? do_syscall_64+0x69/0x80
   ? exc_page_fault+0x68/0x150
   entry_SYSCALL_64_after_hwframe+0x44/0xae

How to trigger:

  tb-ocfs1 # dd if=/dev/zero of=/dev/vdb bs=1M count=20 oflag=direct
  tb-ocfs1 # mkfs.ocfs2 --cluster-stack=pcmk -N 4 /dev/vdb \
             --cluster-name=ocfstst
             == there is no dlm running ==
  tb-ocfs1 # mount -t ocfs2 /dev/vdb /mnt
             == kernel crash ==

Analysis:

After commit da5e7c87827e8 ("ocfs2: cleanup journal init and shutdown"), Journal init later than before, it makes NULL pointer access in free routine.

ocfs2_fill_super
+ ocfs2_mount_volume
| ocfs2_dlm_init //failed, osb->journal still NULL.
+ ...
+ ocfs2_dismount_volume

    ocfs2_release_system_inodes
      ...
       evict
        ...
         ocfs2_clear_inode
          ocfs2_checkpoint_inode
           ocfs2_ci_fully_checkpointed
            time_after(journal->j_trans_id, ci->ci_last_trans)
             + journal is empty, crash!

How to fix:

create a new function _inode_direct_free() to handle journal non-exist case.

Fixes da5e7c87827e8 ("ocfs2: cleanup journal init and shutdown")
---
 fs/ocfs2/localalloc.c | 12 ++++++++
 fs/ocfs2/super.c      | 67 ++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 75 insertions(+), 4 deletions(-)
fs/ocfs2/localalloc.c +12 -0
index 5f6bacbeef6b..c7694c2a1000 100644
@@ -412,6 +412,18 @@ void ocfs2_shutdown_local_alloc(struct ocfs2_super *osb)
 		goto out_mutex;
 	}
 
+	if (!osb->journal) {
+		/* Does here make sense to handle bh? */
+		if (osb->local_alloc_bh)
+			brelse(osb->local_alloc_bh);
+		osb->local_alloc_bh = NULL;
+		osb->local_alloc_state = OCFS2_LA_UNUSED;
+
+		/*  After out_unlock calling iput() is safe, in theory system inode
+		 * still be hold in dismount flow */
+		goto out_unlock;
+	}
+
 	/* WINDOW_MOVE_CREDITS is a bit heavy... */
 	handle = ocfs2_start_trans(osb, OCFS2_WINDOW_MOVE_CREDITS);
 	if (IS_ERR(handle)) {
fs/ocfs2/super.c +63 -4

110 lines not on this page · raw message

Signature · 7 lines
2.34.1


_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
#1 lore raw

Sorry, I forgot to add changelog in patch v3, will resend it. Please review with resent v3 patch.

Thanks

Quoted 84 lines · Heming Zhao
Call trace:
   ocfs2: Registered cluster interface user
   dlm: no local IP address has been set
   dlm: cannot start dlm lowcomms -107
   (mount.ocfs2,2225,0):ocfs2_dlm_init:3355 ERROR: status = -107
   (mount.ocfs2,2225,0):ocfs2_mount_volume:1817 ERROR: status = -107
   (mount.ocfs2,2225,0):ocfs2_fill_super:1186 ERROR: status = -107
   BUG: kernel NULL pointer dereference, address: 0000000000000030
   #PF: supervisor read access in kernel mode
   #PF: error_code(0x0000) - not-present page
   PGD 0 P4D 0
   Oops: 0000 [#1] PREEMPT SMP NOPTI
   CPU: 0 PID: 2225 Comm: mount.ocfs2 Not tainted 5.16.2-1-default #1
openSUSE Tumbleweed b40...e84
   Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ...
   RIP: 0010:ocfs2_clear_inode+0x2e9/0x720 [ocfs2]
   ...
   Call Trace:
    <TASK>
    ? ocfs2_evict_inode+0x1fe/0x630 [ocfs2 411bc..281]
    evict+0xc0/0x1c0
    ocfs2_release_system_inodes+0x21/0xc0 [ocfs2 411bc..281]
    ocfs2_dismount_volume+0x10b/0x2d0 [ocfs2 411bc..281]
    ocfs2_fill_super+0xaf/0x19e0 [ocfs2 411bc..281]
    mount_bdev+0x182/0x1b0
    ? ocfs2_initialize_super.isra.0+0xf50/0xf50 [ocfs2 411bc..281]
    legacy_get_tree+0x24/0x40
    vfs_get_tree+0x22/0xb0
    path_mount+0x465/0xac0
    __x64_sys_mount+0x103/0x140
    do_syscall_64+0x59/0x80
    ? syscall_exit_to_user_mode+0x18/0x40
    ? do_syscall_64+0x69/0x80
    ? syscall_exit_to_user_mode+0x18/0x40
    ? do_syscall_64+0x69/0x80
    ? exc_page_fault+0x68/0x150
    entry_SYSCALL_64_after_hwframe+0x44/0xae

How to trigger:

   tb-ocfs1 # dd if=/dev/zero of=/dev/vdb bs=1M count=20 oflag=direct
   tb-ocfs1 # mkfs.ocfs2 --cluster-stack=pcmk -N 4 /dev/vdb \
              --cluster-name=ocfstst
              == there is no dlm running ==
   tb-ocfs1 # mount -t ocfs2 /dev/vdb /mnt
              == kernel crash ==

Analysis:

After commit da5e7c87827e8 ("ocfs2: cleanup journal init and shutdown"), Journal init later than before, it makes NULL pointer access in free routine.

ocfs2_fill_super
+ ocfs2_mount_volume
| ocfs2_dlm_init //failed, osb->journal still NULL.
+ ...
+ ocfs2_dismount_volume

     ocfs2_release_system_inodes
       ...
        evict
         ...
          ocfs2_clear_inode
           ocfs2_checkpoint_inode
            ocfs2_ci_fully_checkpointed
             time_after(journal->j_trans_id, ci->ci_last_trans)
              + journal is empty, crash!

How to fix:

create a new function _inode_direct_free() to handle journal non-exist case.

Fixes da5e7c87827e8 ("ocfs2: cleanup journal init and shutdown")
---
  fs/ocfs2/localalloc.c | 12 ++++++++
  fs/ocfs2/super.c      | 67 ++++++++++++++++++++++++++++++++++++++++---
  2 files changed, 75 insertions(+), 4 deletions(-)
fs/ocfs2/localalloc.c +0 -0
index 5f6bacbeef6b..c7694c2a1000 100644

fs/ocfs2/localalloc.c

Quoted 6 lines
fs/ocfs2/localalloc.c +1 -0
...
 
+put_slot:
  	/* No cluster connection means we've failed during mount, so skip
  	 * all the steps which depended on that to complete. */
  	if (osb->cconn) {

_______________________________________________
Ocfs2-devel mailing list
[email protected]
https://oss.oracle.com/mailman/listinfo/ocfs2-devel