Skip to contentSkip to conversation

dm verity: skip verity work on I/O errors when system is shutting down

PATCH In review Hyeongseok Kim 2 participants 6 messages Dec 3, 2020 – Dec 10, 2020 Device mapper

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 dm verity: skip verity work on I/O errors when system is shutting down

If emergency system shutdown is called, like by thermal shutdown, dm device could be alive when the block device couldn't process I/O requests anymore. In this status, the handling of I/O errors by new dm I/O requests or by those already in-flight can lead to a verity corruption state, which is misjudgment.
So, skip verity work for I/O error when system is shutting down.

---
 drivers/md/dm-verity-target.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/md/dm-verity-target.c +11 -1
index f74982dcbea0..ba62c537798b 100644
@@ -64,6 +64,15 @@ struct buffer_aux {
 	int hash_verified;
 };
 
+/*
+ * While system shutdown, skip verity work for I/O error.
+ */
+static inline bool verity_is_system_shutting_down(void)
+{
+	return system_state == SYSTEM_HALT || system_state == SYSTEM_POWER_OFF
+		|| system_state == SYSTEM_RESTART;
+}
+
 /*
  * Initialize struct buffer_aux for a freshly created buffer.
  */
@@ -564,7 +573,8 @@ static void verity_end_io(struct bio *bio)
 {
 	struct dm_verity_io *io = bio->bi_private;
 
-	if (bio->bi_status && !verity_fec_is_enabled(io->v)) {
+	if (bio->bi_status &&
+		(!verity_fec_is_enabled(io->v) || verity_is_system_shutting_down())) {
 		verity_finish_io(io, bio->bi_status);
 		return;
 	}
Signature · 6 lines
2.27.0.83.g0313f36

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
#1 lore raw

Hi,

Quoted 7 lines · Hyeongseok Kim

If emergency system shutdown is called, like by thermal shutdown, dm device could be alive when the block device couldn't process I/O requests anymore. In this status, the handling of I/O errors by new dm I/O requests or by those already in-flight can lead to a verity corruption state, which is misjudgment.
So, skip verity work for I/O error when system is shutting down.

Thank you for the patch. I agree that attempting to correct I/O errors when the system is shutting down, and thus generating more I/O that's likely going to fail, is not a good idea.

Quoted 12 lines
---
 drivers/md/dm-verity-target.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/md/dm-verity-target.c +0 -0
index f74982dcbea0..ba62c537798b 100644
@@ -64,6 +64,15 @@ struct buffer_aux {
        int hash_verified;

drivers/md/dm-verity-target.c struct buffer_aux { · line 74

Quoted 10 lines
drivers/md/dm-verity-target.c +8 -0
 };
 
+/*
+ * While system shutdown, skip verity work for I/O error.
+ */
+static inline bool verity_is_system_shutting_down(void)
+{
+       return system_state == SYSTEM_HALT || system_state == SYSTEM_POWER_OFF
+               || system_state == SYSTEM_RESTART;
+}

Which of these states does the system get to during an emergency shutdown? Can we simplify this by changing the test to system_state > SYSTEM_RUNNING?

Quoted 5 lines
drivers/md/dm-verity-target.c +1 -0
+
 /*
  * Initialize struct buffer_aux for a freshly created buffer.
  */

drivers/md/dm-verity-target.c static void verity_end_io(struct bio *bio) · line 580

Quoted 9 lines
drivers/md/dm-verity-target.c +2 -1
@@ -564,7 +573,8 @@ static void verity_end_io(struct bio *bio)
 {
        struct dm_verity_io *io = bio->bi_private;
 
-       if (bio->bi_status && !verity_fec_is_enabled(io->v)) {
+       if (bio->bi_status &&
+               (!verity_fec_is_enabled(io->v) || verity_is_system_shutting_down())) {
                verity_finish_io(io, bio->bi_status);
                return;
        }

Otherwise, this looks good to me. However, I'm now wondering if an I/O error should ever result in verity_handle_err() being called. Without FEC, dm-verity won't call verity_handle_err() when I/O fails, but with FEC enabled, it currently does, assuming error correction fails. Any thoughts?

Sami

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

author
#2 lore raw
Quoted 26 lines · Sami Tolvanen

Hi,

Quoted 6 lines · Hyeongseok Kim

If emergency system shutdown is called, like by thermal shutdown, dm device could be alive when the block device couldn't process I/O requests anymore. In this status, the handling of I/O errors by new dm I/O requests or by those already in-flight can lead to a verity corruption state, which is misjudgment.
So, skip verity work for I/O error when system is shutting down.

Thank you for the patch. I agree that attempting to correct I/O errors when the system is shutting down, and thus generating more I/O that's likely going to fail, is not a good idea.

Quoted 13 lines
---
  drivers/md/dm-verity-target.c | 12 +++++++++++-
  1 file changed, 11 insertions(+), 1 deletion(-)
drivers/md/dm-verity-target.c +0 -0
index f74982dcbea0..ba62c537798b 100644
@@ -64,6 +64,15 @@ struct buffer_aux {
         int hash_verified;
  };
 
Quoted 11 lines

drivers/md/dm-verity-target.c struct buffer_aux { · line 74

Quoted 8 lines
drivers/md/dm-verity-target.c +8 -0
+/*
+ * While system shutdown, skip verity work for I/O error.
+ */
+static inline bool verity_is_system_shutting_down(void)
+{
+       return system_state == SYSTEM_HALT || system_state == SYSTEM_POWER_OFF
+               || system_state == SYSTEM_RESTART;
+}

Which of these states does the system get to during an emergency shutdown? Can we simplify this by changing the test to system_state > SYSTEM_RUNNING?

I only saw that it was SYSTEM_POWER_OFF or SYSTEM_RESTART, I wonder if I/O error can occur in SYSTEM_SUSPEND state. As far as I know, this could be happen in emergency shutdown case,
can you explain if you have a case when I/O error can occur by SYSTEM_SUSPEND?

Quoted 8 lines
Quoted 8 lines
drivers/md/dm-verity-target.c +1 -0
+
  /*
   * Initialize struct buffer_aux for a freshly created buffer.
   */
@@ -564,7 +573,8 @@ static void verity_end_io(struct bio *bio)
  {
         struct dm_verity_io *io = bio->bi_private;
 
Quoted 11 lines

drivers/md/dm-verity-target.c static void verity_end_io(struct bio *bio) · line 580

Quoted 6 lines
drivers/md/dm-verity-target.c +2 -1
-       if (bio->bi_status && !verity_fec_is_enabled(io->v)) {
+       if (bio->bi_status &&
+               (!verity_fec_is_enabled(io->v) || verity_is_system_shutting_down())) {
                 verity_finish_io(io, bio->bi_status);
                 return;
         }

Otherwise, this looks good to me. However, I'm now wondering if an I/O error should ever result in verity_handle_err() being called. Without FEC, dm-verity won't call verity_handle_err() when I/O fails, but with FEC enabled, it currently does, assuming error correction fails. Any thoughts?

Yes, I have thought about this, and to be honest, I think verity or FEC should not call verity_handle_error() in case of I/O errors. But, because I couldn't know the ability of FEC, I only focused on not breaking curent logics other than system shutdown && I/O errors case.

Quoted 3 lines

Sami

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

Reviewed-by
#3 lore raw
Quoted 40 lines · hyeongseok
Quoted 26 lines · Sami Tolvanen

Hi,

Quoted 6 lines · Hyeongseok Kim

If emergency system shutdown is called, like by thermal shutdown, dm device could be alive when the block device couldn't process I/O requests anymore. In this status, the handling of I/O errors by new dm I/O requests or by those already in-flight can lead to a verity corruption state, which is misjudgment.
So, skip verity work for I/O error when system is shutting down.

Thank you for the patch. I agree that attempting to correct I/O errors when the system is shutting down, and thus generating more I/O that's likely going to fail, is not a good idea.

Quoted 13 lines
---
  drivers/md/dm-verity-target.c | 12 +++++++++++-
  1 file changed, 11 insertions(+), 1 deletion(-)
drivers/md/dm-verity-target.c +0 -0
index f74982dcbea0..ba62c537798b 100644
@@ -64,6 +64,15 @@ struct buffer_aux {
         int hash_verified;
  };
 
Quoted 11 lines

drivers/md/dm-verity-target.c struct buffer_aux { · line 74

Quoted 8 lines
drivers/md/dm-verity-target.c +8 -0
+/*
+ * While system shutdown, skip verity work for I/O error.
+ */
+static inline bool verity_is_system_shutting_down(void)
+{
+       return system_state == SYSTEM_HALT || system_state == SYSTEM_POWER_OFF
+               || system_state == SYSTEM_RESTART;
+}

Which of these states does the system get to during an emergency shutdown? Can we simplify this by changing the test to system_state > SYSTEM_RUNNING?

Quoted 2 lines

I only saw that it was SYSTEM_POWER_OFF or SYSTEM_RESTART, I wonder if I/O error can occur in SYSTEM_SUSPEND state.

OK, so think the current version is fine then.

Quoted 3 lines

As far as I know, this could be happen in emergency shutdown case, can you explain if you have a case when I/O error can occur by SYSTEM_SUSPEND?

No, I don't have a case where that would happen.

Quoted 8 lines
Quoted 5 lines

Otherwise, this looks good to me. However, I'm now wondering if an I/O error should ever result in verity_handle_err() being called. Without FEC, dm-verity won't call verity_handle_err() when I/O fails, but with FEC enabled, it currently does, assuming error correction fails. Any thoughts?

Yes, I have thought about this, and to be honest, I think verity or FEC should not call verity_handle_error() in case of I/O errors.

I tend to agree. We could simply check the original bio->bi_status in verity_verify_io() and if we failed to correct an I/O error, return an error instead of going into verity_handle_err(). Any thoughts?

Quoted 2 lines

But, because I couldn't know the ability of FEC, I only focused on not breaking curent logics other than system shutdown && I/O errors case.

Sure, makes sense. We can address that separately.

Sami

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

author
#4 lore raw
Quoted 55 lines · Sami Tolvanen
Quoted 38 lines · hyeongseok
Quoted 26 lines · Sami Tolvanen

Hi,

Quoted 6 lines · Hyeongseok Kim

If emergency system shutdown is called, like by thermal shutdown, dm device could be alive when the block device couldn't process I/O requests anymore. In this status, the handling of I/O errors by new dm I/O requests or by those already in-flight can lead to a verity corruption state, which is misjudgment.
So, skip verity work for I/O error when system is shutting down.

Thank you for the patch. I agree that attempting to correct I/O errors when the system is shutting down, and thus generating more I/O that's likely going to fail, is not a good idea.

Quoted 13 lines
---
   drivers/md/dm-verity-target.c | 12 +++++++++++-
   1 file changed, 11 insertions(+), 1 deletion(-)
drivers/md/dm-verity-target.c +0 -0
index f74982dcbea0..ba62c537798b 100644
@@ -64,6 +64,15 @@ struct buffer_aux {
          int hash_verified;
   };
 
Quoted 11 lines

drivers/md/dm-verity-target.c struct buffer_aux { · line 74

Quoted 8 lines
drivers/md/dm-verity-target.c +8 -0
+/*
+ * While system shutdown, skip verity work for I/O error.
+ */
+static inline bool verity_is_system_shutting_down(void)
+{
+       return system_state == SYSTEM_HALT || system_state == SYSTEM_POWER_OFF
+               || system_state == SYSTEM_RESTART;
+}

Which of these states does the system get to during an emergency shutdown? Can we simplify this by changing the test to system_state > SYSTEM_RUNNING?

Quoted 2 lines

I only saw that it was SYSTEM_POWER_OFF or SYSTEM_RESTART, I wonder if I/O error can occur in SYSTEM_SUSPEND state.

OK, so think the current version is fine then.

Quoted 3 lines

As far as I know, this could be happen in emergency shutdown case, can you explain if you have a case when I/O error can occur by SYSTEM_SUSPEND?

No, I don't have a case where that would happen.

Quoted 7 lines
Quoted 5 lines

Otherwise, this looks good to me. However, I'm now wondering if an I/O error should ever result in verity_handle_err() being called. Without FEC, dm-verity won't call verity_handle_err() when I/O fails, but with FEC enabled, it currently does, assuming error correction fails. Any thoughts?

Yes, I have thought about this, and to be honest, I think verity or FEC should not call verity_handle_error() in case of I/O errors.

Quoted 7 lines

I tend to agree. We could simply check the original bio->bi_status in verity_verify_io() and if we failed to correct an I/O error, return an error instead of going into verity_handle_err(). Any thoughts?

Quoted 2 lines

But, because I couldn't know the ability of FEC, I only focused on not breaking curent logics other than system shutdown && I/O errors case.

Sure, makes sense. We can addrwess that separately.

Sounds reasonable. I hope the next improvements.

Quoted 5 lines

Sami

Thank you for the review.

Hyeongseok

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

#5 lore raw

Hi Mike,

How do you think about this patch?

Quoted 66 lines · hyeongseok
Quoted 56 lines · Sami Tolvanen
Quoted 39 lines · hyeongseok
Quoted 26 lines · Sami Tolvanen

Hi,

Quoted 6 lines · Hyeongseok Kim

If emergency system shutdown is called, like by thermal shutdown, dm device could be alive when the block device couldn't process I/O requests anymore. In this status, the handling of I/O errors by new dm I/O requests or by those already in-flight can lead to a verity corruption state, which is misjudgment.
So, skip verity work for I/O error when system is shutting down.

Thank you for the patch. I agree that attempting to correct I/O errors when the system is shutting down, and thus generating more I/O that's likely going to fail, is not a good idea.

Quoted 13 lines

---
   drivers/md/dm-verity-target.c | 12 +++++++++++-

   1 file changed, 11 insertions(+), 1 deletion(-)
drivers/md/dm-verity-target.c +0 -0
index f74982dcbea0..ba62c537798b 100644
@@ -64,6 +64,15 @@ struct buffer_aux {
           int hash_verified;
    };
 
Quoted 12 lines
Quoted 9 lines
drivers/md/dm-verity-target.c +6 -0
+/*
+ * While system shutdown, skip verity work for I/O error.
+ */
+static inline bool verity_is_system_shutting_down(void)
+{
+       return system_state == SYSTEM_HALT || system_state == 

SYSTEM_POWER_OFF
+               || system_state == SYSTEM_RESTART; +}

Which of these states does the system get to during an emergency shutdown? Can we simplify this by changing the test to system_state > SYSTEM_RUNNING?

Quoted 2 lines

I only saw that it was SYSTEM_POWER_OFF or SYSTEM_RESTART, I wonder if I/O error can occur in SYSTEM_SUSPEND state.

OK, so think the current version is fine then.

Quoted 3 lines

As far as I know, this could be happen in emergency shutdown case, can you explain if you have a case when I/O error can occur by SYSTEM_SUSPEND?

No, I don't have a case where that would happen.

Quoted 7 lines
Quoted 5 lines

Otherwise, this looks good to me. However, I'm now wondering if an I/O error should ever result in verity_handle_err() being called. Without FEC, dm-verity won't call verity_handle_err() when I/O fails, but with FEC enabled, it currently does, assuming error correction fails. Any thoughts?

Yes, I have thought about this, and to be honest, I think verity or FEC should not call verity_handle_error() in case of I/O errors.

Quoted 7 lines

I tend to agree. We could simply check the original bio->bi_status in verity_verify_io() and if we failed to correct an I/O error, return an error instead of going into verity_handle_err(). Any thoughts?

Quoted 2 lines

But, because I couldn't know the ability of FEC, I only focused on not breaking curent logics other than system shutdown && I/O errors case.

Sure, makes sense. We can addrwess that separately.

Sounds reasonable. I hope the next improvements.

Quoted 1 line
Quoted 9 lines
Quoted 4 lines

Sami

Thank you for the review.

Hyeongseok

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel