RISC-V Hibernation Support
Subsystems (2)
v7 cover RISC-V Hibernation Support
This series adds RISC-V Hibernation/suspend to disk support. Low level Arch functions were created to support hibernation. swsusp_arch_suspend() relies code from __cpu_suspend_enter() to write cpu state onto the stack, then calling swsusp_save() to save the memory image.
Arch specific hibernation header is implemented and is utilized by the arch_hibernation_header_restore() and arch_hibernation_header_save() functions. The arch specific hibernation header consists of satp, hartid, and the cpu_resume address. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume() creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into __hibernate_cpu_resume() to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.
To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
- CONFIG_ARCH_HIBERNATION_HEADER
- CONFIG_ARCH_HIBERNATION_POSSIBLE
At high-level, this series includes the following changes:
1) Change suspend_save_csrs() and suspend_restore_csrs()
to public function as these functions are common to suspend/hibernation. (patch 1)
2) Refactor the common code in the __cpu_resume_enter() function and __hibernate_cpu_resume() function. The common code are used by hibernation and suspend. (patch 2)
3) Enhance kernel_page_present() function to support huge page. (patch 3)
4) Add arch/riscv low level functions to support hibernation/suspend to disk. (patch 4)
The above patches are based on kernel v6.3-rc3 and are tested on StarFive VF2 SBC board and Qemu.
ACPI platform mode is not supported in this series.
Changes since v6:
- Rebased to kernel v6.3-rc3
- Resolved nit
Changes since v5:
- Rebased to kernel v6.3-rc2
- Removed extra line at the commit msg
- Added comment to describe the reason to map the kernel address
Changes since v4:
- Rebased to kernel v6.3-rc1
- Resolved typo(s)
- Removed unnecessary helper function
- Removed unnecessary "addr" local variable
- Removed typecast of 'int'
- Used def_bool HIBERNATION
- Used "mv a0, zero" instead of "add a0, zero, zero"
- Make linear region as executable and writable when restoring the image
Changes since v3:
- Rebased to kernel v6.2
- Temporary page table code refactoring by reference to ARM64
- Resolved typo(s) and grammars
- Resolved documentation errors
- Resolved clang build issue
- Removed unnecessary comments
- Used kzalloc instead of kcalloc
Changes since v2:
- Rebased to kernel v6.2-rc5
- Refactor the common code used by hibernation and suspend
- Create copy_page macro
- Solved other comments from Andrew and Conor
Changes since v1:
- Rebased to kernel v6.2-rc3
- Fixed bot's compilation error
Sia Jee Heng (4):
RISC-V: Change suspend_save_csrs and suspend_restore_csrs to public
function
RISC-V: Factor out common code of __cpu_resume_enter()
RISC-V: mm: Enable huge page support to kernel_page_present() function
RISC-V: Add arch functions to support hibernation/suspend-to-disk
arch/riscv/Kconfig | 6 + arch/riscv/include/asm/assembler.h | 82 ++++++ arch/riscv/include/asm/suspend.h | 22 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 427 +++++++++++++++++++++++++++++ arch/riscv/kernel/suspend.c | 4 +- arch/riscv/kernel/suspend_entry.S | 34 +-- arch/riscv/mm/pageattr.c | 8 + 10 files changed, 633 insertions(+), 33 deletions(-) create mode 100644 arch/riscv/include/asm/assembler.h create mode 100644 arch/riscv/kernel/hibernate-asm.S create mode 100644 arch/riscv/kernel/hibernate.c
base-commit: fff5a5e7f528b2ed2c335991399a766c2cf01103
Signature · 1 line
2.34.1
PATCH v7 1/4 RISC-V: Change suspend_save_csrs and suspend_restore_csrs to public function
Currently suspend_save_csrs() and suspend_restore_csrs() functions are statically defined in the suspend.c. Change the function's attribute to public so that the functions can be used by hibernation as well.
--- arch/riscv/include/asm/suspend.h | 3 +++ arch/riscv/kernel/suspend.c | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-)
arch/riscv/include/asm/suspend.h +3 -0
@@ -33,4 +33,7 @@ int cpu_suspend(unsigned long arg,/* Low-level CPU resume entry function */ int __cpu_resume_enter (unsigned long hartid,unsigned long context); +/* Used to save and restore the CSRs */ +void suspend_save_csrs (struct suspend_context *context); +void suspend_restore_csrs (struct suspend_context *context);#endif
arch/riscv/kernel/suspend.c +2 -2
@@ -8,7 +8,7 @@#include <asm/csr.h> #include <asm/suspend.h> -static void suspend_save_csrs (struct suspend_context *context) +void suspend_save_csrs (struct suspend_context *context) { context->scratch =csr_read (CSR_SCRATCH ); context->tvec =csr_read (CSR_TVEC );
@@ -29,7 +29,7 @@ static void suspend_save_csrs(struct suspend_context *context)#endif } -static void suspend_restore_csrs (struct suspend_context *context) +void suspend_restore_csrs (struct suspend_context *context) {csr_write (CSR_SCRATCH , context->scratch);csr_write (CSR_TVEC , context->tvec);
Signature · 1 line
2.34.1
PATCH v7 2/4 RISC-V: Factor out common code of __cpu_resume_enter()
In reply to Sia Jee HengThe cpu_resume() function is very similar for the suspend to disk and suspend to ram cases. Factor out the common code into suspend_restore_csrs macro and suspend_restore_regs macro.
--- arch/riscv/include/asm/assembler.h | 62 ++++++++++++++++++++++++++++++ arch/riscv/kernel/suspend_entry.S | 34 ++-------------- 2 files changed, 65 insertions(+), 31 deletions(-) create mode 100644 arch/riscv/include/asm/assembler.h
arch/riscv/include/asm/assembler.h new file +62 -0
@@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Copyright (C) 2023 StarFive Technology Co., Ltd. +* +* Author: Jee Heng Sia <[email protected]> +*/ + +#ifndef __ASSEMBLY__ +#error "Only include this from assembly code" +#endif + +#ifndef __ASM_ASSEMBLER_H +#define __ASM_ASSEMBLER_H + +#include <asm/asm.h> +#include <asm/asm-offsets.h> +#include <asm/csr.h> + +/* +* suspend_restore_csrs - restore CSRs +*/ + .macro suspend_restore_csrs + REG_L t0, (SUSPEND_CONTEXT_REGS + PT_EPC)(a0) + csrw CSR_EPC, t0 + REG_L t0, (SUSPEND_CONTEXT_REGS + PT_STATUS)(a0) + csrw CSR_STATUS, t0 + REG_L t0, (SUSPEND_CONTEXT_REGS + PT_BADADDR)(a0) + csrw CSR_TVAL, t0 + REG_L t0, (SUSPEND_CONTEXT_REGS + PT_CAUSE)(a0) + csrw CSR_CAUSE, t0 + .endm + +/* + * suspend_restore_regs - Restore registers (except A0 and T0-T6) + */ + .macro suspend_restore_regs + REG_L ra, (SUSPEND_CONTEXT_REGS + PT_RA)(a0) + REG_L sp, (SUSPEND_CONTEXT_REGS + PT_SP)(a0) + REG_L gp, (SUSPEND_CONTEXT_REGS + PT_GP)(a0) + REG_L tp, (SUSPEND_CONTEXT_REGS + PT_TP)(a0) + REG_L s0, (SUSPEND_CONTEXT_REGS + PT_S0)(a0) + REG_L s1, (SUSPEND_CONTEXT_REGS + PT_S1)(a0) + REG_L a1, (SUSPEND_CONTEXT_REGS + PT_A1)(a0) + REG_L a2, (SUSPEND_CONTEXT_REGS + PT_A2)(a0) + REG_L a3, (SUSPEND_CONTEXT_REGS + PT_A3)(a0) + REG_L a4, (SUSPEND_CONTEXT_REGS + PT_A4)(a0) + REG_L a5, (SUSPEND_CONTEXT_REGS + PT_A5)(a0) + REG_L a6, (SUSPEND_CONTEXT_REGS + PT_A6)(a0) + REG_L a7, (SUSPEND_CONTEXT_REGS + PT_A7)(a0) + REG_L s2, (SUSPEND_CONTEXT_REGS + PT_S2)(a0) + REG_L s3, (SUSPEND_CONTEXT_REGS + PT_S3)(a0) + REG_L s4, (SUSPEND_CONTEXT_REGS + PT_S4)(a0) + REG_L s5, (SUSPEND_CONTEXT_REGS + PT_S5)(a0) + REG_L s6, (SUSPEND_CONTEXT_REGS + PT_S6)(a0) + REG_L s7, (SUSPEND_CONTEXT_REGS + PT_S7)(a0) + REG_L s8, (SUSPEND_CONTEXT_REGS + PT_S8)(a0) + REG_L s9, (SUSPEND_CONTEXT_REGS + PT_S9)(a0) + REG_L s10, (SUSPEND_CONTEXT_REGS + PT_S10)(a0) + REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0) + .endm + +#endif /* __ASM_ASSEMBLER_H */
arch/riscv/kernel/suspend_entry.S +3 -31
@@ -7,6 +7,7 @@ #include <linux/linkage.h> #include <asm/asm.h> #include <asm/asm-offsets.h> +#include <asm/assembler.h> #include <asm/csr.h> #include <asm/xip_fixup.h>
@@ -83,39 +84,10 @@ ENTRY(__cpu_resume_enter) add a0, a1, zero /* Restore CSRs */ - REG_L t0, (SUSPEND_CONTEXT_REGS + PT_EPC)(a0) - csrw CSR_EPC, t0 - REG_L t0, (SUSPEND_CONTEXT_REGS + PT_STATUS)(a0) - csrw CSR_STATUS, t0 - REG_L t0, (SUSPEND_CONTEXT_REGS + PT_BADADDR)(a0) - csrw CSR_TVAL, t0 - REG_L t0, (SUSPEND_CONTEXT_REGS + PT_CAUSE)(a0) - csrw CSR_CAUSE, t0 + suspend_restore_csrs /* Restore registers (except A0 and T0-T6) */ - REG_L ra, (SUSPEND_CONTEXT_REGS + PT_RA)(a0) - REG_L sp, (SUSPEND_CONTEXT_REGS + PT_SP)(a0) - REG_L gp, (SUSPEND_CONTEXT_REGS + PT_GP)(a0) - REG_L tp, (SUSPEND_CONTEXT_REGS + PT_TP)(a0) - REG_L s0, (SUSPEND_CONTEXT_REGS + PT_S0)(a0) - REG_L s1, (SUSPEND_CONTEXT_REGS + PT_S1)(a0) - REG_L a1, (SUSPEND_CONTEXT_REGS + PT_A1)(a0) - REG_L a2, (SUSPEND_CONTEXT_REGS + PT_A2)(a0) - REG_L a3, (SUSPEND_CONTEXT_REGS + PT_A3)(a0) - REG_L a4, (SUSPEND_CONTEXT_REGS + PT_A4)(a0) - REG_L a5, (SUSPEND_CONTEXT_REGS + PT_A5)(a0) - REG_L a6, (SUSPEND_CONTEXT_REGS + PT_A6)(a0) - REG_L a7, (SUSPEND_CONTEXT_REGS + PT_A7)(a0) - REG_L s2, (SUSPEND_CONTEXT_REGS + PT_S2)(a0) - REG_L s3, (SUSPEND_CONTEXT_REGS + PT_S3)(a0) - REG_L s4, (SUSPEND_CONTEXT_REGS + PT_S4)(a0) - REG_L s5, (SUSPEND_CONTEXT_REGS + PT_S5)(a0) - REG_L s6, (SUSPEND_CONTEXT_REGS + PT_S6)(a0) - REG_L s7, (SUSPEND_CONTEXT_REGS + PT_S7)(a0) - REG_L s8, (SUSPEND_CONTEXT_REGS + PT_S8)(a0) - REG_L s9, (SUSPEND_CONTEXT_REGS + PT_S9)(a0) - REG_L s10, (SUSPEND_CONTEXT_REGS + PT_S10)(a0) - REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0) + suspend_restore_regs /* Return zero value */ add a0, zero, zero
Signature · 1 line
2.34.1
PATCH v7 3/4 RISC-V: mm: Enable huge page support to kernel_page_present() function
In reply to Sia Jee HengCurrently kernel_page_present() function doesn't support huge page detection causes the function to mistakenly return false to the hibernation core.
Add huge page detection to the function to solve the problem.
--- arch/riscv/mm/pageattr.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/riscv/mm/pageattr.c +8 -0
@@ -217,18 +217,26 @@ bool kernel_page_present(struct page *page) pgd =pgd_offset_k (addr);if (!pgd_present (*pgd))return false; +if (pgd_leaf (*pgd)) +return true; p4d =p4d_offset (pgd, addr);if (!p4d_present (*p4d))return false; +if (p4d_leaf (*p4d)) +return true; pud =pud_offset (p4d, addr);if (!pud_present (*pud))return false; +if (pud_leaf (*pud)) +return true; pmd =pmd_offset (pud, addr);if (!pmd_present (*pmd))return false; +if (pmd_leaf (*pmd)) +return true; pte =pte_offset_kernel (pmd, addr);return pte_present (*pte);
Signature · 1 line
2.34.1
PATCH v7 4/4 RISC-V: Add arch functions to support hibernation/suspend-to-disk
In reply to Sia Jee HengLow level Arch functions were created to support hibernation. swsusp_arch_suspend() relies code from __cpu_suspend_enter() to write cpu state onto the stack, then calling swsusp_save() to save the memory image.
Arch specific hibernation header is implemented and is utilized by the arch_hibernation_header_restore() and arch_hibernation_header_save() functions. The arch specific hibernation header consists of satp, hartid, and the cpu_resume address. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume() creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into __hibernate_cpu_resume() to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.
To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
- CONFIG_ARCH_HIBERNATION_HEADER
- CONFIG_ARCH_HIBERNATION_POSSIBLE
--- arch/riscv/Kconfig | 6 + arch/riscv/include/asm/assembler.h | 20 ++ arch/riscv/include/asm/suspend.h | 19 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 427 +++++++++++++++++++++++++++++ 7 files changed, 555 insertions(+) create mode 100644 arch/riscv/kernel/hibernate-asm.S create mode 100644 arch/riscv/kernel/hibernate.c
arch/riscv/Kconfig +6 -0
@@ -707,6 +707,12 @@ menu "Power management options" source "kernel/power/Kconfig" +config ARCH_HIBERNATION_POSSIBLE + def_bool y + +config ARCH_HIBERNATION_HEADER + def_bool HIBERNATION + endmenu # "Power management options" menu "CPU Power Management"
arch/riscv/include/asm/assembler.h +20 -0
@@ -59,4 +59,24 @@ REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0) .endm +/* + * copy_page - copy 1 page (4KB) of data from source to destination + * @a0 - destination + * @a1 - source + */ + .macro copy_page a0, a1 + lui a2, 0x1 + add a2, a2, a0 +1 : + REG_L t0, 0(a1) + REG_L t1, SZREG(a1) + + REG_S t0, 0(a0) + REG_S t1, SZREG(a0) + + addi a0, a0, 2 * SZREG + addi a1, a1, 2 * SZREG + bne a2, a0, 1b + .endm + #endif /* __ASM_ASSEMBLER_H */
arch/riscv/include/asm/suspend.h +19 -0
@@ -21,6 +21,11 @@ struct suspend_context {#endif }; +/* +* Used by hibernation core and cleared during resume sequence +*/ +extern int in_suspend; +/* Low-level CPU suspend entry function */ int __cpu_suspend_enter (struct suspend_context *context);
@@ -36,4 +41,18 @@ int __cpu_resume_enter(unsigned long hartid, unsigned long context);/* Used to save and restore the CSRs */ void suspend_save_csrs (struct suspend_context *context);void suspend_restore_csrs (struct suspend_context *context); + +/* Low-level API to support hibernation */ +int swsusp_arch_suspend (void ); +int swsusp_arch_resume (void ); +int arch_hibernation_header_save (void *addr,unsigned int max_size); +int arch_hibernation_header_restore (void *addr); +int __hibernate_cpu_resume (void ); + +/* Used to resume on the CPU we hibernated on */ +int hibernate_resume_nonboot_cpu_disable (void ); + +asmlinkage void hibernate_restore_image(unsigned long resume_satp, unsigned long satp_temp, +unsigned long cpu_resume); +asmlinkage int hibernate_core_restore_code(void);#endif
arch/riscv/kernel/Makefile +1 -0
@@ -64,6 +64,7 @@ obj-$(CONFIG_MODULES) += module.o obj-$(CONFIG_MODULE_SECTIONS) += module-sections.o obj-$(CONFIG_CPU_PM) += suspend_entry.o suspend.o +obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_FUNCTION_TRACER) += mcount.o ftrace.o obj-$(CONFIG_DYNAMIC_FTRACE) += mcount-dyn.o
arch/riscv/kernel/asm-offsets.c +5 -0
@@ -9,6 +9,7 @@#include <linux/kbuild.h> #include <linux/mm.h> #include <linux/sched.h> +#include <linux/suspend.h> #include <asm/kvm_host.h> #include <asm/thread_info.h> #include <asm/ptrace.h>
@@ -116,6 +117,10 @@ void asm_offsets(void)OFFSET (SUSPEND_CONTEXT_REGS , suspend_context, regs); +OFFSET (HIBERN_PBE_ADDR , pbe, address); +OFFSET (HIBERN_PBE_ORIG , pbe, orig_address); +OFFSET (HIBERN_PBE_NEXT , pbe, next); +OFFSET (KVM_ARCH_GUEST_ZERO , kvm_vcpu_arch, guest_context.zero);OFFSET (KVM_ARCH_GUEST_RA , kvm_vcpu_arch, guest_context.ra);OFFSET (KVM_ARCH_GUEST_SP , kvm_vcpu_arch, guest_context.sp);
arch/riscv/kernel/hibernate-asm.S new file +77 -0
77 lines not on this page · raw message
arch/riscv/kernel/hibernate.c new file +427 -0
427 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v7 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-disk
Quoted 28 lines · Sia Jee Heng
Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid, and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLESigned-off-by Sia Jee Heng <[email protected]>Reviewed-by Ley Foon Tan <[email protected]>Reviewed-by Mason Huo <[email protected]>Reviewed-by Conor Dooley <[email protected]>
Quoted 12 lines
--- arch/riscv/Kconfig | 6 + arch/riscv/include/asm/assembler.h | 20 ++ arch/riscv/include/asm/suspend.h | 19 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 427 +++++++++++++++++++++++++++++ 7 files changed, 555 insertions(+) create mode 100644 arch/riscv/kernel/hibernate-asm.S create mode 100644 arch/riscv/kernel/hibernate.c
Re: [PATCH v7 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-disk
Quoted 42 lines · Andrew Jones
Quoted 28 lines · Sia Jee Heng
Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid, and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLESigned-off-by Sia Jee Heng <[email protected]>Reviewed-by Ley Foon Tan <[email protected]>Reviewed-by Mason Huo <[email protected]>Reviewed-by Conor Dooley <[email protected]>Quoted 12 lines
--- arch/riscv/Kconfig | 6 + arch/riscv/include/asm/assembler.h | 20 ++ arch/riscv/include/asm/suspend.h | 19 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 427 +++++++++++++++++++++++++++++ 7 files changed, 555 insertions(+) create mode 100644 arch/riscv/kernel/hibernate-asm.S create mode 100644 arch/riscv/kernel/hibernate.c
Quoted 1 line
Reviewed-by Andrew Jones <[email protected]>
Actually, I forgot to point out something. As hibernation depends on suspend.c for save/restore_csrs, we need something like
arch/riscv/Kconfig +1 -1
@@ -54,7 +54,7 @@ config RISCV select CLINT_TIMER if !MMU select CLONE_BACKWARDS select COMMON_CLK - select CPU_PM if CPU_IDLE + select CPU_PM if CPU_IDLE || HIBERNATION select EDAC_SUPPORT select GENERIC_ARCH_TOPOLOGY select GENERIC_ATOMIC64 if !64BIT
Thanks,
drew
RE: [PATCH v7 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-disk
Quoted 64 lines
-----Original Message-----
From: Andrew Jones <[email protected]>
Sent: Monday, March 27, 2023 9:23 PM
To: JeeHeng Sia <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>; Conor Dooley <[email protected]>
Subject: Re: [PATCH v7 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-diskQuoted 42 lines · Andrew Jones
Quoted 28 lines · Sia Jee Heng
Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid, and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLESigned-off-by Sia Jee Heng <[email protected]>Reviewed-by Ley Foon Tan <[email protected]>Reviewed-by Mason Huo <[email protected]>Reviewed-by Conor Dooley <[email protected]>Quoted 12 lines
--- arch/riscv/Kconfig | 6 + arch/riscv/include/asm/assembler.h | 20 ++ arch/riscv/include/asm/suspend.h | 19 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 427 +++++++++++++++++++++++++++++ 7 files changed, 555 insertions(+) create mode 100644 arch/riscv/kernel/hibernate-asm.S create mode 100644 arch/riscv/kernel/hibernate.cQuoted 1 line
Reviewed-by Andrew Jones <[email protected]>Actually, I forgot to point out something. As hibernation depends on
suspend.cfor save/restore_csrs, we need something likearch/riscv/Kconfig +0
-0
arch/riscv/Kconfig config RISCV · line 57
Quoted 5 lines
arch/riscv/Kconfig +1
-1@@ -54,7 +54,7 @@ config RISCV select CLINT_TIMER if !MMU select CLONE_BACKWARDS select COMMON_CLK - select CPU_PM if CPU_IDLE + select CPU_PM if CPU_IDLE || HIBERNATION
Sure. Will add it
Quoted 7 lines
arch/riscv/Kconfig +0
-0select EDAC_SUPPORT select GENERIC_ARCH_TOPOLOGY select GENERIC_ATOMIC64 if !64BITThanks,
drew
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 36 lines · Sia Jee Heng
This series adds RISC-V Hibernation/suspend to disk support. Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid, and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLEAt high-level, this series includes the following changes:
1) Changesuspend_save_csrs()andsuspend_restore_csrs()
to public function as these functions are common to suspend/hibernation. (patch 1)
2) Refactor the common code in the__cpu_resume_enter()function and__hibernate_cpu_resume()function. The common code are used by hibernation and suspend. (patch 2)
3) Enhancekernel_page_present()function to support huge page. (patch 3)
4) Addarch/riscvlow level functions to support hibernation/suspend to disk. (patch 4)
Quoted 4 lines
The above patches are based on kernel v6.3-rc3 and are tested on StarFive VF2 SBC board and Qemu.
ACPI platform mode is not supported in this series.
I tested this on QEMU, but, FYI, I had to use a raw backing file for the swap disk, rather than a qcow2 backing file, otherwise it didn't resume. It's probably worth looking into why that is.
Thanks,
drew
Quoted 47 lines
-----Original Message-----
From: Andrew Jones <[email protected]>
Sent: Monday, March 27, 2023 9:14 PM
To: JeeHeng Sia <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>
Subject: Re: [PATCH v7 0/4] RISC-V Hibernation SupportQuoted 36 lines · Sia Jee Heng
This series adds RISC-V Hibernation/suspend to disk support. Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid, and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLEAt high-level, this series includes the following changes:
1) Changesuspend_save_csrs()andsuspend_restore_csrs()
to public function as these functions are common to suspend/hibernation. (patch 1)
2) Refactor the common code in the__cpu_resume_enter()function and__hibernate_cpu_resume()function. The common code are used by hibernation and suspend. (patch 2)
3) Enhancekernel_page_present()function to support huge page. (patch 3)
4) Addarch/riscvlow level functions to support hibernation/suspend to disk. (patch 4)
Quoted 8 lines
Quoted 4 lines
The above patches are based on kernel v6.3-rc3 and are tested on StarFive VF2 SBC board and Qemu.
ACPI platform mode is not supported in this series.I tested this on QEMU, but, FYI, I had to use a raw backing file for the swap disk, rather than a qcow2 backing file, otherwise it didn't resume. It's probably worth looking into why that is.
Thanks for your time. The raw file format is closer to the actual physical disk. Although I can look into the qcow2 format for QEMU in the near future, it shouldn't be a blocking factor for this patch series to be upstreamed.
Quoted 3 lines
Thanks,
drew
Quoted 62 lines
-----Original Message-----
From: JeeHeng Sia
Sent: Tuesday, March 28, 2023 2:37 PM
To: 'Andrew Jones' <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>
Subject: RE: [PATCH v7 0/4] RISC-V Hibernation SupportQuoted 50 lines
-----Original Message-----
From: Andrew Jones <[email protected]>
Sent: Monday, March 27, 2023 9:14 PM
To: JeeHeng Sia <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>
Subject: Re: [PATCH v7 0/4] RISC-V Hibernation SupportQuoted 40 lines · Sia Jee Heng
This series adds RISC-V Hibernation/suspend to disk support. Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid, and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLEAt high-level, this series includes the following changes:
1) Changesuspend_save_csrs()andsuspend_restore_csrs()
to public function as these functions are common to suspend/hibernation. (patch 1)
2) Refactor the common code in the__cpu_resume_enter()function and__hibernate_cpu_resume()function. The common code are used by hibernation and suspend. (patch 2)
3) Enhancekernel_page_present()function to support huge page. (patch 3)
4) Addarch/riscvlow level functions to support hibernation/suspend to disk. (patch 4)The above patches are based on kernel v6.3-rc3 and are tested on StarFive VF2 SBC board and Qemu.
ACPI platform mode is not supported in this series.
Quoted 5 lines
Quoted 3 lines
I tested this on QEMU, but, FYI, I had to use a raw backing file for the swap disk, rather than a qcow2 backing file, otherwise it didn't resume. It's probably worth looking into why that is.
Thanks for your time. The raw file format is closer to the actual physical disk. Although I can look into the qcow2 format for QEMU in the near future, it shouldn't be a blocking factor for this patch series to be upstreamed.
FYI, I managed to reproduce the hibernation issue that Andrew reported. The hibernation resume failed while retrieving pages from the disk, specifically in the kernel/power/swap.c - swap_read_page() function and the snapshot_write_next() function in the kernel/power/snapshot.c. I found that adding a delay to the functions (by adding a printk) allowed the page retrieval process to progress further. Through this exercise, I have begun to suspect that there may be an issue with coherency handling in between the hibernation core and the QEMU qcow2 driver. I will add it to my AR list and shall help to investigate the issue in the near future.
Quoted 3 lines
Quoted 3 lines
Thanks,
drew