kexec_file: skip checksum verification when safe
Subsystems (3)
PATCH v2 1/1 kexec_file: skip checksum verification when safe
Checksum verification is needed
1. for crash kernels. In a crash, we can't be sure the kernel is intact.
2. if we're worried about relocating the kernel into a region used by some DMA that wasn't properly cancelled.
If KHO is enabled then relocations will happen to KHO scratch, which is free from DMA regions.
If we used CMA to allocate segments then relocations are not going to happen at all.
Therefore, we can safely disable checksum verification in both of those cases.
Instead of adding a new variable to purgatory, just skip adding regions and save the default value of SHA256 hash.
Saves ~250ms on my 4.0 GHz CPU. This is an important saving for the live-update project.
---
v2:
- also skip checksum verification if KHO is enabled
- small fixes from reviews
My original idea was to do 2 changes:
1. Skip checksum if all segments are CMA.
2. If KHO is enabled, allocate the kernel inside kho_scratch using CMA.
This way we could skip both relocations and checksum verification when KHO is enabled.
But I realized that step 2 might not be possible on warm boots. I have no idea how to fix that (except weird ideas like 2 kho_scratches that we swap on every warm boot), so I decided to just skip checksum verification when KHO is enabled. This unfortunately means relocations will still happen.
--- kernel/kexec_file.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
kernel/kexec_file.c +27 -0
@@ -27,6 +27,7 @@#include <linux/syscalls.h> #include <linux/vmalloc.h> #include <linux/dma-map-ops.h> +#include <linux/kexec_handover.h> #include "kexec_internal.h" #ifdef CONFIG_KEXEC_SIG
@@ -798,6 +799,16 @@ int kexec_add_buffer(struct kexec_buf *kbuf)return 0 ; } +static bool kexec_only_cma_segments (struct kimage *image) +{ +for (int i =0 ; i < image->nr_segments; i++) { +if (!image->segment_cma[i]) +return false; + } + +return true; +} +/* Calculate and store the digest of segments */ static int kexec_calculate_store_digests (struct kimage *image) {
@@ -822,6 +833,21 @@ static int kexec_calculate_store_digests(struct kimage *image)sha256_init (&sctx); +/* +* If KHO is enabled, the destinations are located in KHO scratch. +* KHO scratch can only contain early boot allocations and movable +* allocations. That means there is no risk of memory corruption by +* uncancelled DMA. +* +* If all segments were loaded into contiguous memory, there will be no +* relocations at all, so also no risk no corruption. +*/ +if (image->type !=KEXEC_TYPE_CRASH && + (kho_is_enabled () ||kexec_only_cma_segments (image))) { +pr_debug ("disabling checksum verification in purgatory\n" ); + goto skip_checksum; + } +for (j = i =0 ; i < image->nr_segments; i++) {struct kexec_segment *ksegment;
@@ -867,6 +893,7 @@ static int kexec_calculate_store_digests(struct kimage *image) j++; } +skip_checksum:sha256_final (&sctx, digest); ret = kexec_purgatory_get_set_symbol(image, "purgatory_sha_regions",
Signature · 1 line
2.54.0.929.g9b7fa37559-goog
Quoted 25 lines · Michal Clapinski
Checksum verification is needed
1. for crash kernels. In a crash, we can't be sure the kernel is intact.
2. if we're worried about relocating the kernel into a region used by some DMA that wasn't properly cancelled.If KHO is enabled then relocations will happen to KHO scratch, which is free from DMA regions.
If we used CMA to allocate segments then relocations are not going to happen at all.
Therefore, we can safely disable checksum verification in both of those cases.Instead of adding a new variable to purgatory, just skip adding regions and save the default value of SHA256 hash.
Saves ~250ms on my 4.0 GHz CPU. This is an important saving for the live-update project.
Signed-off-by Michal Clapinski <[email protected]>---
v2:
- also skip checksum verification if KHO is enabled
- small fixes from reviews
Quoted 7 lines
My original idea was to do 2 changes:
1. Skip checksum if all segments are CMA.
2. If KHO is enabled, allocate the kernel insidekho_scratchusing CMA.This way we could skip both relocations and checksum verification when KHO is enabled.
But I realized that step 2 might not be possible on warm boots.
AFAIU we only relocate into scratch since relocating anywhere else might over-write preserved memory. If there is no relocation, there is no need for the kernel image to be in scratch, since the image won't be preserved memory anyway.
So perhaps we can just use CMA directly, and only fall back to kho_locate_mem_hole() if that fails? This should be a simple enough change.
Do you know how much time we can save by skipping relocations? I would guess it is in the hundreds of milliseconds.
Can you try this (COMPLETELY UNTESTED) patch out and see if it works and if it further improves kexec time?
--- 8< ---
kernel/kexec_file.c +8 -8
@@ -720,14 +720,6 @@ int kexec_locate_mem_hole(struct kexec_buf *kbuf)if (kbuf->mem !=KEXEC_BUF_MEM_UNKNOWN )return 0 ; -/* -* If KHO is active, only use KHO scratch memory. All other memory -* could potentially be handed over. -*/ - ret =kho_locate_mem_hole (kbuf, locate_mem_hole_callback); -if (ret <=0 ) -return ret; -/* * Try to find a free physically contiguous block of memory first. With that, we * can avoid any copying at kexec time.
@@ -735,6 +727,14 @@ int kexec_locate_mem_hole(struct kexec_buf *kbuf)if (!kexec_alloc_contig (kbuf))return 0 ; +/* +* If KHO is active and relocations are to be done,, only use KHO +* scratch memory. All other memory could potentially be handed over. +*/ + ret =kho_locate_mem_hole (kbuf, locate_mem_hole_callback); +if (ret <=0 ) +return ret; +if (!IS_ENABLED (CONFIG_ARCH_KEEP_MEMBLOCK )) ret =kexec_walk_resources (kbuf, locate_mem_hole_callback);else
--- >8 ---
Of course this is not directly related to this patch so it shouldn't block it, but I reckon we might be able to squeeze a bit more performance out this way as a follow up.
Quoted 39 lines
I have no idea how to fix that (except weird ideas like 2
kho_scratchesthat we swap on every warm boot), so I decided to just skip checksum verification when KHO is enabled. This unfortunately means relocations will still happen.--- kernel/kexec_file.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)kernel/kexec_file.c +11
-0@@ -27,6 +27,7 @@#include <linux/syscalls.h> #include <linux/vmalloc.h> #include <linux/dma-map-ops.h> +#include <linux/kexec_handover.h> #include "kexec_internal.h" #ifdef CONFIG_KEXEC_SIG @@ -798,6 +799,16 @@ int kexec_add_buffer(struct kexec_buf *kbuf)return 0 ; } +static bool kexec_only_cma_segments (struct kimage *image) +{ +for (int i =0 ; i < image->nr_segments; i++) { +if (!image->segment_cma[i]) +return false; + } + +return true; +} +/* Calculate and store the digest of segments */ static int kexec_calculate_store_digests (struct kimage *image) {@@ -822,6 +833,21 @@ static int kexec_calculate_store_digests(struct kimage *image)
kernel/kexec_file.c static int kexec_calculate_store_digests(struct kimage *image) · line 843
Quoted 10 lines
kernel/kexec_file.c +8
-0sha256_init (&sctx); +/* +* If KHO is enabled, the destinations are located in KHO scratch. +* KHO scratch can only contain early boot allocations and movable +* allocations. That means there is no risk of memory corruption by +* uncancelled DMA. +* +* If all segments were loaded into contiguous memory, there will be no +* relocations at all, so also no risk no corruption.
Typo: "so also no risk *of* corruption".
We can fix that up when applying I think, so no need for a v3 just for this.
Other than this,
Quoted 18 lines
kernel/kexec_file.c +8
-0+*/ +if (image->type !=KEXEC_TYPE_CRASH && + (kho_is_enabled () ||kexec_only_cma_segments (image))) { +pr_debug ("disabling checksum verification in purgatory\n" ); + goto skip_checksum; + } +for (j = i =0 ; i < image->nr_segments; i++) {struct kexec_segment *ksegment;@@ -867,6 +893,7 @@ static int kexec_calculate_store_digests(struct kimage *image) j++; } +skip_checksum:sha256_final (&sctx, digest); ret = kexec_purgatory_get_set_symbol(image, "purgatory_sha_regions",
Signature · 2 lines
Regards, Pratyush Yadav
Quoted 36 lines · Pratyush Yadav
Quoted 25 lines · Michal Clapinski
Checksum verification is needed
1. for crash kernels. In a crash, we can't be sure the kernel is intact.
2. if we're worried about relocating the kernel into a region used by some DMA that wasn't properly cancelled.If KHO is enabled then relocations will happen to KHO scratch, which is free from DMA regions.
If we used CMA to allocate segments then relocations are not going to happen at all.
Therefore, we can safely disable checksum verification in both of those cases.Instead of adding a new variable to purgatory, just skip adding regions and save the default value of SHA256 hash.
Saves ~250ms on my 4.0 GHz CPU. This is an important saving for the live-update project.
Signed-off-by Michal Clapinski <[email protected]>---
v2:
- also skip checksum verification if KHO is enabled
- small fixes from reviewsQuoted 7 lines
My original idea was to do 2 changes:
1. Skip checksum if all segments are CMA.
2. If KHO is enabled, allocate the kernel insidekho_scratchusing CMA.This way we could skip both relocations and checksum verification when KHO is enabled.
But I realized that step 2 might not be possible on warm boots.
Quoted 8 lines
AFAIU we only relocate into scratch since relocating anywhere else might over-write preserved memory. If there is no relocation, there is no need for the kernel image to be in scratch, since the image won't be preserved memory anyway.
So perhaps we can just use CMA directly, and only fall back to
kho_locate_mem_hole()if that fails? This should be a simple enough change.
I agree that it will work. However, the user would need to have CMA memory and it would need to have enough contiguous memory available. Do you think running out of CMA memory is a real problem?
Quoted 2 lines
Do you know how much time we can save by skipping relocations? I would guess it is in the hundreds of milliseconds.
It's smaller than the variance between runs. Maybe 10ms. Everything between exiting the old kernel and TSC initialization in the new kernel takes ~70ms.
Theoretically if we didn't have to do relocations, we could try unpacking the kernel before kexec, which would save a little bit more time. But again, definitely less than 0.1s.
Quoted 125 lines
Can you try this (COMPLETELY UNTESTED) patch out and see if it works and if it further improves kexec time?
--- 8< ---
kernel/kexec_file.c +8
-9@@ -720,14 +720,6 @@ int kexec_locate_mem_hole(struct kexec_buf *kbuf)if (kbuf->mem !=KEXEC_BUF_MEM_UNKNOWN )return 0 ; -/* -* If KHO is active, only use KHO scratch memory. All other memory -* could potentially be handed over. -*/ - ret =kho_locate_mem_hole (kbuf, locate_mem_hole_callback); -if (ret <=0 ) -return ret; -/* * Try to find a free physically contiguous block of memory first. With that, we * can avoid any copying at kexec time. @@ -735,6 +727,14 @@ int kexec_locate_mem_hole(struct kexec_buf *kbuf)if (!kexec_alloc_contig (kbuf))return 0 ; +/* +* If KHO is active and relocations are to be done,, only use KHO +* scratch memory. All other memory could potentially be handed over. +*/ + ret =kho_locate_mem_hole (kbuf, locate_mem_hole_callback); +if (ret <=0 ) +return ret; +if (!IS_ENABLED (CONFIG_ARCH_KEEP_MEMBLOCK )) ret =kexec_walk_resources (kbuf, locate_mem_hole_callback);else --- >8 ---Of course this is not directly related to this patch so it shouldn't block it, but I reckon we might be able to squeeze a bit more performance out this way as a follow up.
Quoted 39 lines
I have no idea how to fix that (except weird ideas like 2
kho_scratchesthat we swap on every warm boot), so I decided to just skip checksum verification when KHO is enabled. This unfortunately means relocations will still happen.--- kernel/kexec_file.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)kernel/kexec_file.c +11
-0@@ -27,6 +27,7 @@#include <linux/syscalls.h> #include <linux/vmalloc.h> #include <linux/dma-map-ops.h> +#include <linux/kexec_handover.h> #include "kexec_internal.h" #ifdef CONFIG_KEXEC_SIG @@ -798,6 +799,16 @@ int kexec_add_buffer(struct kexec_buf *kbuf)return 0 ; } +static bool kexec_only_cma_segments (struct kimage *image) +{ +for (int i =0 ; i < image->nr_segments; i++) { +if (!image->segment_cma[i]) +return false; + } + +return true; +} +/* Calculate and store the digest of segments */ static int kexec_calculate_store_digests (struct kimage *image) {@@ -822,6 +833,21 @@ static int kexec_calculate_store_digests(struct kimage *image)kernel/kexec_file.c static int kexec_calculate_store_digests(struct kimage *image) · line 843
Quoted 10 lines
kernel/kexec_file.c +8
-0sha256_init (&sctx); +/* +* If KHO is enabled, the destinations are located in KHO scratch. +* KHO scratch can only contain early boot allocations and movable +* allocations. That means there is no risk of memory corruption by +* uncancelled DMA. +* +* If all segments were loaded into contiguous memory, there will be no +* relocations at all, so also no risk no corruption. Typo: "so also no risk *of* corruption".
We can fix that up when applying I think, so no need for a v3 just for this.
Other than this,
Reviewed-by Pratyush Yadav (Google) <[email protected]>Quoted 11 lines
kernel/kexec_file.c +7
-0+*/ +if (image->type !=KEXEC_TYPE_CRASH && + (kho_is_enabled () ||kexec_only_cma_segments (image))) { +pr_debug ("disabling checksum verification in purgatory\n" ); + goto skip_checksum; + } +for (j = i =0 ; i < image->nr_segments; i++) {struct kexec_segment *ksegment;kernel/kexec_file.c static int kexec_calculate_store_digests(struct kimage *image) · line 899
Quoted 7 lines
kernel/kexec_file.c +1
-0@@ -867,6 +893,7 @@ static int kexec_calculate_store_digests(struct kimage *image) j++; } +skip_checksum:sha256_final (&sctx, digest); ret = kexec_purgatory_get_set_symbol(image, "purgatory_sha_regions",--
Regards,
Pratyush Yadav
Pratyush Yadav
maintainerQuoted 42 lines · Michał Cłapiński
Quoted 41 lines · Pratyush Yadav
Quoted 25 lines · Michal Clapinski
Checksum verification is needed
1. for crash kernels. In a crash, we can't be sure the kernel is intact.
2. if we're worried about relocating the kernel into a region used by some DMA that wasn't properly cancelled.If KHO is enabled then relocations will happen to KHO scratch, which is free from DMA regions.
If we used CMA to allocate segments then relocations are not going to happen at all.
Therefore, we can safely disable checksum verification in both of those cases.Instead of adding a new variable to purgatory, just skip adding regions and save the default value of SHA256 hash.
Saves ~250ms on my 4.0 GHz CPU. This is an important saving for the live-update project.
Signed-off-by Michal Clapinski <[email protected]>---
v2:
- also skip checksum verification if KHO is enabled
- small fixes from reviewsQuoted 7 lines
My original idea was to do 2 changes:
1. Skip checksum if all segments are CMA.
2. If KHO is enabled, allocate the kernel insidekho_scratchusing CMA.This way we could skip both relocations and checksum verification when KHO is enabled.
But I realized that step 2 might not be possible on warm boots.AFAIU we only relocate into scratch since relocating anywhere else might over-write preserved memory. If there is no relocation, there is no need for the kernel image to be in scratch, since the image won't be preserved memory anyway.
Quoted 7 lines
Quoted 3 lines
So perhaps we can just use CMA directly, and only fall back to
kho_locate_mem_hole()if that fails? This should be a simple enough change.I agree that it will work. However, the user would need to have CMA memory and it would need to have enough contiguous memory available. Do you think running out of CMA memory is a real problem?
No idea. I think that depends heavily on how much memory drivers are using, and I have no numbers for that.
Anyway, if the user doesn't have memory available in CMA, we will still fall back to the normal path so kexec load will still at least keep working.
Quoted 11 lines
Quoted 3 lines
Do you know how much time we can save by skipping relocations? I would guess it is in the hundreds of milliseconds.
It's smaller than the variance between runs. Maybe 10ms. Everything between exiting the old kernel and TSC initialization in the new kernel takes ~70ms.
Theoretically if we didn't have to do relocations, we could try unpacking the kernel before kexec, which would save a little bit more time. But again, definitely less than 0.1s.
Hmm, I thought it would take longer. I don't think we are at a point yet where we should try to save 10s of milliseconds.
Thanks for trying it out though.
Quoted 45 lines
Quoted 45 lines
Can you try this (COMPLETELY UNTESTED) patch out and see if it works and if it further improves kexec time?
--- 8< ---
kernel/kexec_file.c +8
-9@@ -720,14 +720,6 @@ int kexec_locate_mem_hole(struct kexec_buf *kbuf)if (kbuf->mem !=KEXEC_BUF_MEM_UNKNOWN )return 0 ; -/* -* If KHO is active, only use KHO scratch memory. All other memory -* could potentially be handed over. -*/ - ret =kho_locate_mem_hole (kbuf, locate_mem_hole_callback); -if (ret <=0 ) -return ret; -/* * Try to find a free physically contiguous block of memory first. With that, we * can avoid any copying at kexec time. @@ -735,6 +727,14 @@ int kexec_locate_mem_hole(struct kexec_buf *kbuf)if (!kexec_alloc_contig (kbuf))return 0 ; +/* +* If KHO is active and relocations are to be done,, only use KHO +* scratch memory. All other memory could potentially be handed over. +*/ + ret =kho_locate_mem_hole (kbuf, locate_mem_hole_callback); +if (ret <=0 ) +return ret; +if (!IS_ENABLED (CONFIG_ARCH_KEEP_MEMBLOCK )) ret =kexec_walk_resources (kbuf, locate_mem_hole_callback);else --- >8 ---Of course this is not directly related to this patch so it shouldn't block it, but I reckon we might be able to squeeze a bit more performance out this way as a follow up.
[...]
Signature · 2 lines
Regards, Pratyush Yadav
Pasha Tatashin
maintainerQuoted 132 lines · Pratyush Yadav
Quoted 25 lines · Michal Clapinski
Checksum verification is needed
1. for crash kernels. In a crash, we can't be sure the kernel is intact.
2. if we're worried about relocating the kernel into a region used by some DMA that wasn't properly cancelled.If KHO is enabled then relocations will happen to KHO scratch, which is free from DMA regions.
If we used CMA to allocate segments then relocations are not going to happen at all.
Therefore, we can safely disable checksum verification in both of those cases.Instead of adding a new variable to purgatory, just skip adding regions and save the default value of SHA256 hash.
Saves ~250ms on my 4.0 GHz CPU. This is an important saving for the live-update project.
Signed-off-by Michal Clapinski <[email protected]>---
v2:
- also skip checksum verification if KHO is enabled
- small fixes from reviewsQuoted 7 lines
My original idea was to do 2 changes:
1. Skip checksum if all segments are CMA.
2. If KHO is enabled, allocate the kernel insidekho_scratchusing CMA.This way we could skip both relocations and checksum verification when KHO is enabled.
But I realized that step 2 might not be possible on warm boots.AFAIU we only relocate into scratch since relocating anywhere else might over-write preserved memory. If there is no relocation, there is no need for the kernel image to be in scratch, since the image won't be preserved memory anyway.
So perhaps we can just use CMA directly, and only fall back to
kho_locate_mem_hole()if that fails? This should be a simple enough change.Do you know how much time we can save by skipping relocations? I would guess it is in the hundreds of milliseconds.
Can you try this (COMPLETELY UNTESTED) patch out and see if it works and if it further improves kexec time?
--- 8< ---
kernel/kexec_file.c +8
-9@@ -720,14 +720,6 @@ int kexec_locate_mem_hole(struct kexec_buf *kbuf)if (kbuf->mem !=KEXEC_BUF_MEM_UNKNOWN )return 0 ; -/* -* If KHO is active, only use KHO scratch memory. All other memory -* could potentially be handed over. -*/ - ret =kho_locate_mem_hole (kbuf, locate_mem_hole_callback); -if (ret <=0 ) -return ret; -/* * Try to find a free physically contiguous block of memory first. With that, we * can avoid any copying at kexec time. @@ -735,6 +727,14 @@ int kexec_locate_mem_hole(struct kexec_buf *kbuf)if (!kexec_alloc_contig (kbuf))return 0 ; +/* +* If KHO is active and relocations are to be done,, only use KHO +* scratch memory. All other memory could potentially be handed over. +*/ + ret =kho_locate_mem_hole (kbuf, locate_mem_hole_callback); +if (ret <=0 ) +return ret; +if (!IS_ENABLED (CONFIG_ARCH_KEEP_MEMBLOCK )) ret =kexec_walk_resources (kbuf, locate_mem_hole_callback);else --- >8 ---Of course this is not directly related to this patch so it shouldn't block it, but I reckon we might be able to squeeze a bit more performance out this way as a follow up.
Quoted 41 lines
I have no idea how to fix that (except weird ideas like 2
kho_scratchesthat we swap on every warm boot), so I decided to just skip checksum verification when KHO is enabled. This unfortunately means relocations will still happen.--- kernel/kexec_file.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)kernel/kexec_file.c +11
-0@@ -27,6 +27,7 @@#include <linux/syscalls.h> #include <linux/vmalloc.h> #include <linux/dma-map-ops.h> +#include <linux/kexec_handover.h> #include "kexec_internal.h" #ifdef CONFIG_KEXEC_SIG @@ -798,6 +799,16 @@ int kexec_add_buffer(struct kexec_buf *kbuf)return 0 ; } +static bool kexec_only_cma_segments (struct kimage *image) +{ +for (int i =0 ; i < image->nr_segments; i++) { +if (!image->segment_cma[i]) +return false; + } + +return true; +} +/* Calculate and store the digest of segments */ static int kexec_calculate_store_digests (struct kimage *image) {@@ -822,6 +833,21 @@ static int kexec_calculate_store_digests(struct kimage *image)sha256_init (&sctx);
Quoted 10 lines
kernel/kexec_file.c static int kexec_calculate_store_digests(struct kimage *image) · line 843
Quoted 8 lines
kernel/kexec_file.c +8
-0+/* +* If KHO is enabled, the destinations are located in KHO scratch. +* KHO scratch can only contain early boot allocations and movable +* allocations. That means there is no risk of memory corruption by +* uncancelled DMA. +* +* If all segments were loaded into contiguous memory, there will be no +* relocations at all, so also no risk no corruption. Typo: "so also no risk *of* corruption".
Missed this fix when applied forced updated, to address this.
Quoted 30 lines
We can fix that up when applying I think, so no need for a v3 just for this.
Other than this,
Reviewed-by Pratyush Yadav (Google) <[email protected]>Quoted 18 lines
kernel/kexec_file.c +8
-0+*/ +if (image->type !=KEXEC_TYPE_CRASH && + (kho_is_enabled () ||kexec_only_cma_segments (image))) { +pr_debug ("disabling checksum verification in purgatory\n" ); + goto skip_checksum; + } +for (j = i =0 ; i < image->nr_segments; i++) {struct kexec_segment *ksegment;@@ -867,6 +893,7 @@ static int kexec_calculate_store_digests(struct kimage *image) j++; } +skip_checksum:sha256_final (&sctx, digest); ret = kexec_purgatory_get_set_symbol(image, "purgatory_sha_regions",Signature · 2 lines
Regards, Pratyush Yadav
Quoted 6 lines · Michal Clapinski
Checksum verification is needed
1. for crash kernels. In a crash, we can't be sure the kernel is intact.
2. if we're worried about relocating the kernel into a region used by some DMA that wasn't properly cancelled.
Quoted 8 lines
If KHO is enabled then relocations will happen to KHO scratch, which is free from DMA regions.
If we used CMA to allocate segments then relocations are not going to happen at all.
Therefore, we can safely disable checksum verification in both of those cases.[...]
Applied, thanks!
[1/1] kexec_file: skip checksum verification when safe
commit: 459a08d029bf6f026b25063708a63bdaa8ccc0b1
Best regards,
Signature · 1 line
Pasha Tatashin <[email protected]>