ext4: Avoid panic during forced reboot
Jan Kara
reviewerPATCH 1/1 ext4: Avoid panic during forced reboot
When admin calls "reboot -f" - i.e., does a hard system reboot by directly calling reboot(2) - ext4 filesystem mounted with errors=panic can panic the system. This happens because the underlying device gets disabled without unmounting the filesystem and thus some syscall running in parallel to reboot(2) can result in the filesystem getting IO errors.
This is somewhat surprising to the users so try improve the behavior by switching to errors=remount-ro behavior when the system is running reboot(2).
--- fs/ext4/super.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
What do people think about this? Arguably this change has some potential for breakage if e.g. an ext4 error would somehow block reboot(2). I don't see how that would be possible but I wanted to raise this just in case...
fs/ext4/super.c +12 -1
@@ -430,6 +430,12 @@ static void ext4_journal_commit_callback(journal_t *journal, transaction_t *txn)spin_unlock (&sbi->s_md_lock); } +static bool system_going_down (void ) +{ +return system_state ==SYSTEM_HALT || system_state ==SYSTEM_POWER_OFF + || system_state ==SYSTEM_RESTART ; +} +/* Deal with the reporting of failure conditions on a filesystem such as * inconsistencies detected or read IO failures. *
@@ -460,7 +466,12 @@ static void ext4_handle_error(struct super_block *sb)if (journal)jbd2_journal_abort (journal, -EIO ); } -if (test_opt (sb,ERRORS_RO )) { +/* +* We force ERRORS_RO behavior when system is rebooting. Otherwise we +* could panic during 'reboot -f' as the underlying device got already +* disabled. +*/ +if (test_opt (sb,ERRORS_RO ) ||system_going_down ()) {ext4_msg (sb,KERN_CRIT ,"Remounting filesystem read-only" );/* * Make sure updated value of ->s_mount_flags will be visible
Signature · 1 line
2.16.4
Quoted 37 lines · Jan Kara
When admin calls "reboot -f" - i.e., does a hard system reboot by directly calling reboot(2) - ext4 filesystem mounted with errors=panic can panic the system. This happens because the underlying device gets disabled without unmounting the filesystem and thus some syscall running in parallel to reboot(2) can result in the filesystem getting IO errors.
This is somewhat surprising to the users so try improve the behavior by switching to errors=remount-ro behavior when the system is running reboot(2).
Signed-off-by Jan Kara <[email protected]>--- fs/ext4/super.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)What do people think about this? Arguably this change has some potential for breakage if e.g. an ext4 error would somehow block reboot(2). I don't see how that would be possible but I wanted to raise this just in case...
fs/ext4/super.c +6
-0@@ -430,6 +430,12 @@ static void ext4_journal_commit_callback(journal_t *journal, transaction_t *txn)spin_unlock (&sbi->s_md_lock); } +static bool system_going_down (void ) +{ +return system_state ==SYSTEM_HALT || system_state ==SYSTEM_POWER_OFF + || system_state ==SYSTEM_RESTART ; +} +/* Deal with the reporting of failure conditions on a filesystem such as * inconsistencies detected or read IO failures. *
fs/ext4/super.c static void ext4_handle_error(struct super_block *sb) · line 474
Quoted 10 lines
fs/ext4/super.c +6
-1@@ -460,7 +466,12 @@ static void ext4_handle_error(struct super_block *sb)if (journal)jbd2_journal_abort (journal, -EIO ); } -if (test_opt (sb,ERRORS_RO )) { +/* +* We force ERRORS_RO behavior when system is rebooting. Otherwise we +* could panic during 'reboot -f' as the underlying device got already +* disabled. +*/ +if (test_opt (sb,ERRORS_RO ) ||system_going_down ()) {
If I read the code correctly then this will not avoid the panic().
There is no return in this branch. Therefore it still continues with the ERRORS_PANIC check...
Or do I miss anything, please?
Best Regards,
Petr
Quoted 5 lines
fs/ext4/super.c +0
-0ext4_msg (sb,KERN_CRIT ,"Remounting filesystem read-only" );/* * Make sure updated value of ->s_mount_flags will be visible Signature · 1 line
2.16.4
Jan Kara
reviewerQuoted 51 lines · Petr Mladek
Quoted 37 lines · Jan Kara
When admin calls "reboot -f" - i.e., does a hard system reboot by directly calling reboot(2) - ext4 filesystem mounted with errors=panic can panic the system. This happens because the underlying device gets disabled without unmounting the filesystem and thus some syscall running in parallel to reboot(2) can result in the filesystem getting IO errors.
This is somewhat surprising to the users so try improve the behavior by switching to errors=remount-ro behavior when the system is running reboot(2).
Signed-off-by Jan Kara <[email protected]>--- fs/ext4/super.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)What do people think about this? Arguably this change has some potential for breakage if e.g. an ext4 error would somehow block reboot(2). I don't see how that would be possible but I wanted to raise this just in case...
fs/ext4/super.c +6
-0@@ -430,6 +430,12 @@ static void ext4_journal_commit_callback(journal_t *journal, transaction_t *txn)spin_unlock (&sbi->s_md_lock); } +static bool system_going_down (void ) +{ +return system_state ==SYSTEM_HALT || system_state ==SYSTEM_POWER_OFF + || system_state ==SYSTEM_RESTART ; +} +/* Deal with the reporting of failure conditions on a filesystem such as * inconsistencies detected or read IO failures. * fs/ext4/super.c static void ext4_handle_error(struct super_block *sb) · line 474
Quoted 10 lines
fs/ext4/super.c +6
-1@@ -460,7 +466,12 @@ static void ext4_handle_error(struct super_block *sb)if (journal)jbd2_journal_abort (journal, -EIO ); } -if (test_opt (sb,ERRORS_RO )) { +/* +* We force ERRORS_RO behavior when system is rebooting. Otherwise we +* could panic during 'reboot -f' as the underlying device got already +* disabled. +*/ +if (test_opt (sb,ERRORS_RO ) ||system_going_down ()) {If I read the code correctly then this will not avoid the
panic().
Quoted 4 lines
There is no return in this branch. Therefore it still continues with the
ERRORS_PANICcheck...Or do I miss anything, please?
No, you are right. Attached is a fixed up patch.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
From 15b830d4e877ed908c733ab3219801d1026af256 Mon Sep 17 00:00:00 2001
From: Jan Kara <[email protected]>
Date: Tue, 12 Mar 2019 10:38:19 +0100
Subject: [PATCH] ext4: Avoid panic during forced reboot
When admin calls "reboot -f" - i.e., does a hard system reboot by directly calling reboot(2) - ext4 filesystem mounted with errors=panic can panic the system. This happens because the underlying device gets disabled without unmounting the filesystem and thus some syscall running in parallel to reboot(2) can result in the filesystem getting IO errors.
This is somewhat surprising to the users so try improve the behavior by switching to errors=remount-ro behavior when the system is running reboot(2).
--- fs/ext4/super.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
fs/ext4/super.c +13 -3
@@ -430,6 +430,12 @@ static void ext4_journal_commit_callback(journal_t *journal, transaction_t *txn)spin_unlock (&sbi->s_md_lock); } +static bool system_going_down (void ) +{ +return system_state ==SYSTEM_HALT || system_state ==SYSTEM_POWER_OFF + || system_state ==SYSTEM_RESTART ; +} +/* Deal with the reporting of failure conditions on a filesystem such as * inconsistencies detected or read IO failures. *
@@ -460,7 +466,12 @@ static void ext4_handle_error(struct super_block *sb)if (journal)jbd2_journal_abort (journal, -EIO ); } -if (test_opt (sb,ERRORS_RO )) { +/* +* We force ERRORS_RO behavior when system is rebooting. Otherwise we +* could panic during 'reboot -f' as the underlying device got already +* disabled. +*/ +if (test_opt (sb,ERRORS_RO ) ||system_going_down ()) {ext4_msg (sb,KERN_CRIT ,"Remounting filesystem read-only" );/* * Make sure updated value of ->s_mount_flags will be visible
@@ -468,8 +479,7 @@ static void ext4_handle_error(struct super_block *sb)*/ smp_wmb (); sb->s_flags |=SB_RDONLY ; - } -if (test_opt (sb,ERRORS_PANIC )) { + }else if (test_opt (sb,ERRORS_PANIC )) {if (EXT4_SB (sb)->s_journal && !(EXT4_SB (sb)->s_journal->j_flags &JBD2_REC_ERR ))return ;
Signature · 1 line
2.16.4
Quoted 80 lines · Jan Kara
Quoted 11 lines · Petr Mladek
Quoted 10 lines · Jan Kara
When admin calls "reboot -f" - i.e., does a hard system reboot by directly calling reboot(2) - ext4 filesystem mounted with errors=panic can panic the system. This happens because the underlying device gets disabled without unmounting the filesystem and thus some syscall running in parallel to reboot(2) can result in the filesystem getting IO errors.
This is somewhat surprising to the users so try improve the behavior by switching to errors=remount-ro behavior when the system is running reboot(2).
Quoted 13 lines
fs/ext4/super.c static void ext4_handle_error(struct super_block *sb) · line 474
Quoted 11 lines
fs/ext4/super.c +6
-1@@ -460,7 +466,12 @@ static void ext4_handle_error(struct super_block *sb)if (journal)jbd2_journal_abort (journal, -EIO ); } -if (test_opt (sb,ERRORS_RO )) { +/* +* We force ERRORS_RO behavior when system is rebooting. Otherwise we +* could panic during 'reboot -f' as the underlying device got already +* disabled. +*/ +if (test_opt (sb,ERRORS_RO ) ||system_going_down ()) {If I read the code correctly then this will not avoid the
panic().No, you are right. Attached is a fixed up patch.
Quoted 1 line
From
15b830d4e877ed908c733ab3219801d1026af256Mon Sep 17 00:00:00 2001From: Jan Kara <[email protected]>
Date: Tue, 12 Mar 2019 10:38:19 +0100
Subject: [PATCH] ext4: Avoid panic during forced rebootWhen admin calls "reboot -f" - i.e., does a hard system reboot by directly calling reboot(2) - ext4 filesystem mounted with errors=panic can panic the system. This happens because the underlying device gets disabled without unmounting the filesystem and thus some syscall running in parallel to reboot(2) can result in the filesystem getting IO errors.
This is somewhat surprising to the users so try improve the behavior by switching to errors=remount-ro behavior when the system is running reboot(2).
Signed-off-by Jan Kara <[email protected]>--- fs/ext4/super.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)fs/ext4/super.c +12
-1@@ -430,6 +430,12 @@ static void ext4_journal_commit_callback(journal_t *journal, transaction_t *txn)spin_unlock (&sbi->s_md_lock); } +static bool system_going_down (void ) +{ +return system_state ==SYSTEM_HALT || system_state ==SYSTEM_POWER_OFF + || system_state ==SYSTEM_RESTART ; +} +/* Deal with the reporting of failure conditions on a filesystem such as * inconsistencies detected or read IO failures. * @@ -460,7 +466,12 @@ static void ext4_handle_error(struct super_block *sb)if (journal)jbd2_journal_abort (journal, -EIO ); } -if (test_opt (sb,ERRORS_RO )) { +/* +* We force ERRORS_RO behavior when system is rebooting. Otherwise we +* could panic during 'reboot -f' as the underlying device got already +* disabled. +*/ +if (test_opt (sb,ERRORS_RO ) ||system_going_down ()) {ext4_msg (sb,KERN_CRIT ,"Remounting filesystem read-only" );/* * Make sure updated value of ->s_mount_flags will be visible
fs/ext4/super.c static void ext4_handle_error(struct super_block *sb) · line 482
Quoted 6 lines
fs/ext4/super.c +1
-2@@ -468,8 +479,7 @@ static void ext4_handle_error(struct super_block *sb)*/ smp_wmb (); sb->s_flags |=SB_RDONLY ; - } -if (test_opt (sb,ERRORS_PANIC )) { + }else if (test_opt (sb,ERRORS_PANIC )) {
I think that this would work. Just note that it changes a bit the semantic also when the system is in the running state.
ERRORS_RO option always takes precedence over ERRORS_PANIC now. It was the other way before.
Best Regards,
Petr
Quoted 7 lines
fs/ext4/super.c +0
-0if (EXT4_SB (sb)->s_journal && !(EXT4_SB (sb)->s_journal->j_flags &JBD2_REC_ERR ))return ;Signature · 1 line
2.16.4
Jan Kara
reviewerQuoted 91 lines · Petr Mladek
Quoted 80 lines · Jan Kara
Quoted 11 lines · Petr Mladek
Quoted 10 lines · Jan Kara
When admin calls "reboot -f" - i.e., does a hard system reboot by directly calling reboot(2) - ext4 filesystem mounted with errors=panic can panic the system. This happens because the underlying device gets disabled without unmounting the filesystem and thus some syscall running in parallel to reboot(2) can result in the filesystem getting IO errors.
This is somewhat surprising to the users so try improve the behavior by switching to errors=remount-ro behavior when the system is running reboot(2).
Quoted 13 lines
fs/ext4/super.c static void ext4_handle_error(struct super_block *sb) · line 474
Quoted 11 lines
fs/ext4/super.c +6
-1@@ -460,7 +466,12 @@ static void ext4_handle_error(struct super_block *sb)if (journal)jbd2_journal_abort (journal, -EIO ); } -if (test_opt (sb,ERRORS_RO )) { +/* +* We force ERRORS_RO behavior when system is rebooting. Otherwise we +* could panic during 'reboot -f' as the underlying device got already +* disabled. +*/ +if (test_opt (sb,ERRORS_RO ) ||system_going_down ()) {If I read the code correctly then this will not avoid the
panic().No, you are right. Attached is a fixed up patch.
Quoted 1 line
From
15b830d4e877ed908c733ab3219801d1026af256Mon Sep 17 00:00:00 2001From: Jan Kara <[email protected]>
Date: Tue, 12 Mar 2019 10:38:19 +0100
Subject: [PATCH] ext4: Avoid panic during forced rebootWhen admin calls "reboot -f" - i.e., does a hard system reboot by directly calling reboot(2) - ext4 filesystem mounted with errors=panic can panic the system. This happens because the underlying device gets disabled without unmounting the filesystem and thus some syscall running in parallel to reboot(2) can result in the filesystem getting IO errors.
This is somewhat surprising to the users so try improve the behavior by switching to errors=remount-ro behavior when the system is running reboot(2).
Signed-off-by Jan Kara <[email protected]>--- fs/ext4/super.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)fs/ext4/super.c +12
-1@@ -430,6 +430,12 @@ static void ext4_journal_commit_callback(journal_t *journal, transaction_t *txn)spin_unlock (&sbi->s_md_lock); } +static bool system_going_down (void ) +{ +return system_state ==SYSTEM_HALT || system_state ==SYSTEM_POWER_OFF + || system_state ==SYSTEM_RESTART ; +} +/* Deal with the reporting of failure conditions on a filesystem such as * inconsistencies detected or read IO failures. * @@ -460,7 +466,12 @@ static void ext4_handle_error(struct super_block *sb)if (journal)jbd2_journal_abort (journal, -EIO ); } -if (test_opt (sb,ERRORS_RO )) { +/* +* We force ERRORS_RO behavior when system is rebooting. Otherwise we +* could panic during 'reboot -f' as the underlying device got already +* disabled. +*/ +if (test_opt (sb,ERRORS_RO ) ||system_going_down ()) {ext4_msg (sb,KERN_CRIT ,"Remounting filesystem read-only" );/* * Make sure updated value of ->s_mount_flags will be visible fs/ext4/super.c static void ext4_handle_error(struct super_block *sb) · line 482
Quoted 6 lines
fs/ext4/super.c +1
-2@@ -468,8 +479,7 @@ static void ext4_handle_error(struct super_block *sb)*/ smp_wmb (); sb->s_flags |=SB_RDONLY ; - } -if (test_opt (sb,ERRORS_PANIC )) { + }else if (test_opt (sb,ERRORS_PANIC )) {I think that this would work. Just note that it changes a bit the semantic also when the system is in the running state.
Quoted 2 lines
ERRORS_ROoption always takes precedence overERRORS_PANICnow. It was the other way before.
I know. But they couldn't be set both at the same time so this should not matter.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Theodore Ts'o
maintainerQuoted 11 lines
From
15b830d4e877ed908c733ab3219801d1026af256Mon Sep 17 00:00:00 2001
From: Jan Kara <[email protected]>
Date: Tue, 12 Mar 2019 10:38:19 +0100
Subject: [PATCH] ext4: Avoid panic during forced rebootWhen admin calls "reboot -f" - i.e., does a hard system reboot by directly calling reboot(2) - ext4 filesystem mounted with errors=panic can panic the system. This happens because the underlying device gets disabled without unmounting the filesystem and thus some syscall running in parallel to reboot(2) can result in the filesystem getting IO errors.
Quoted 5 lines
This is somewhat surprising to the users so try improve the behavior by switching to errors=remount-ro behavior when the system is running reboot(2).
Signed-off-by Jan Kara <[email protected]>
Looks good to me; thanks, applied.
- Ted