RISC-V Hibernation Support
Subsystems (2)
v5 cover RISC-V Hibernation Support
This series adds RISC-V Hibernation/suspend to disk support. Low level Arch functions were created to support hibernation. swsusp_arch_suspend() relies code from __cpu_suspend_enter() to write cpu state onto the stack, then calling swsusp_save() to save the memory image.
Arch specific hibernation header is implemented and is utilized by the arch_hibernation_header_restore() and arch_hibernation_header_save() functions. The arch specific hibernation header consists of satp, hartid, and the cpu_resume address. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume() creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into __hibernate_cpu_resume() to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.
To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
- CONFIG_ARCH_HIBERNATION_HEADER
- CONFIG_ARCH_HIBERNATION_POSSIBLE
At high-level, this series includes the following changes:
1) Change suspend_save_csrs() and suspend_restore_csrs()
to public function as these functions are common to suspend/hibernation. (patch 1)
2) Refactor the common code in the __cpu_resume_enter() function and __hibernate_cpu_resume() function. The common code are used by hibernation and suspend. (patch 2)
3) Enhance kernel_page_present() function to support huge page. (patch 3)
4) Add arch/riscv low level functions to support hibernation/suspend to disk. (patch 4)
The above patches are based on kernel v6.3-rc1 and are tested on StarFive VF2 SBC board and Qemu.
ACPI platform mode is not supported in this series.
Changes since v4:
- Rebased to kernel v6.3-rc1
- Resolved typo(s)
- Removed unnecessary helper function
- Removed unnecessary "addr" local variable
- Removed typecast of 'int'
- Used def_bool HIBERNATION
- Used "mv a0, zero" instead of "add a0, zero, zero"
- Make linear region as executable and writable when restoring the image
Changes since v3:
- Rebased to kernel v6.2
- Temporary page table code refactoring by reference to ARM64
- Resolved typo(s) and grammars
- Resolved documentation errors
- Resolved clang build issue
- Removed unnecessary comments
- Used kzalloc instead of kcalloc
Changes since v2:
- Rebased to kernel v6.2-rc5
- Refactor the common code used by hibernation and suspend
- Create copy_page macro
- Solved other comments from Andrew and Conor
Changes since v1:
- Rebased to kernel v6.2-rc3
- Fixed bot's compilation error
Sia Jee Heng (4):
RISC-V: Change suspend_save_csrs and suspend_restore_csrs to public
function
RISC-V: Factor out common code of __cpu_resume_enter()
RISC-V: mm: Enable huge page support to kernel_page_present() function
RISC-V: Add arch functions to support hibernation/suspend-to-disk
arch/riscv/Kconfig | 6 + arch/riscv/include/asm/assembler.h | 82 ++++++ arch/riscv/include/asm/suspend.h | 22 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 424 +++++++++++++++++++++++++++++ arch/riscv/kernel/suspend.c | 4 +- arch/riscv/kernel/suspend_entry.S | 34 +-- arch/riscv/mm/pageattr.c | 8 + 10 files changed, 630 insertions(+), 33 deletions(-) create mode 100644 arch/riscv/include/asm/assembler.h create mode 100644 arch/riscv/kernel/hibernate-asm.S create mode 100644 arch/riscv/kernel/hibernate.c
base-commit: 63355b9884b3d1677de6bd1517cd2b8a9bf53978
Signature · 1 line
2.34.1
PATCH v5 1/4 RISC-V: Change suspend_save_csrs and suspend_restore_csrs to public function
Currently suspend_save_csrs() and suspend_restore_csrs() functions are statically defined in the suspend.c. Change the function's attribute to public so that the functions can be used by hibernation as well.
--- arch/riscv/include/asm/suspend.h | 3 +++ arch/riscv/kernel/suspend.c | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-)
arch/riscv/include/asm/suspend.h +3 -0
@@ -33,4 +33,7 @@ int cpu_suspend(unsigned long arg,/* Low-level CPU resume entry function */ int __cpu_resume_enter (unsigned long hartid,unsigned long context); +/* Used to save and restore the CSRs */ +void suspend_save_csrs (struct suspend_context *context); +void suspend_restore_csrs (struct suspend_context *context);#endif
arch/riscv/kernel/suspend.c +2 -2
@@ -8,7 +8,7 @@#include <asm/csr.h> #include <asm/suspend.h> -static void suspend_save_csrs (struct suspend_context *context) +void suspend_save_csrs (struct suspend_context *context) { context->scratch =csr_read (CSR_SCRATCH ); context->tvec =csr_read (CSR_TVEC );
@@ -29,7 +29,7 @@ static void suspend_save_csrs(struct suspend_context *context)#endif } -static void suspend_restore_csrs (struct suspend_context *context) +void suspend_restore_csrs (struct suspend_context *context) {csr_write (CSR_SCRATCH , context->scratch);csr_write (CSR_TVEC , context->tvec);
Signature · 1 line
2.34.1
PATCH v5 2/4 RISC-V: Factor out common code of __cpu_resume_enter()
In reply to Sia Jee HengThe cpu_resume() function is very similar for the suspend to disk and suspend to ram cases. Factor out the common code into suspend_restore_csrs macro and suspend_restore_regs macro.
--- arch/riscv/include/asm/assembler.h | 62 ++++++++++++++++++++++++++++++ arch/riscv/kernel/suspend_entry.S | 34 ++-------------- 2 files changed, 65 insertions(+), 31 deletions(-) create mode 100644 arch/riscv/include/asm/assembler.h
arch/riscv/include/asm/assembler.h new file +62 -0
@@ -0,0 +1,62 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Copyright (C) 2023 StarFive Technology Co., Ltd. +* +* Author: Jee Heng Sia <[email protected]> +*/ + +#ifndef __ASSEMBLY__ +#error "Only include this from assembly code" +#endif + +#ifndef __ASM_ASSEMBLER_H +#define __ASM_ASSEMBLER_H + +#include <asm/asm.h> +#include <asm/asm-offsets.h> +#include <asm/csr.h> + +/* +* suspend_restore_csrs - restore CSRs +*/ + .macro suspend_restore_csrs + REG_L t0, (SUSPEND_CONTEXT_REGS + PT_EPC)(a0) + csrw CSR_EPC, t0 + REG_L t0, (SUSPEND_CONTEXT_REGS + PT_STATUS)(a0) + csrw CSR_STATUS, t0 + REG_L t0, (SUSPEND_CONTEXT_REGS + PT_BADADDR)(a0) + csrw CSR_TVAL, t0 + REG_L t0, (SUSPEND_CONTEXT_REGS + PT_CAUSE)(a0) + csrw CSR_CAUSE, t0 + .endm + +/* + * suspend_restore_regs - Restore registers (except A0 and T0-T6) + */ + .macro suspend_restore_regs + REG_L ra, (SUSPEND_CONTEXT_REGS + PT_RA)(a0) + REG_L sp, (SUSPEND_CONTEXT_REGS + PT_SP)(a0) + REG_L gp, (SUSPEND_CONTEXT_REGS + PT_GP)(a0) + REG_L tp, (SUSPEND_CONTEXT_REGS + PT_TP)(a0) + REG_L s0, (SUSPEND_CONTEXT_REGS + PT_S0)(a0) + REG_L s1, (SUSPEND_CONTEXT_REGS + PT_S1)(a0) + REG_L a1, (SUSPEND_CONTEXT_REGS + PT_A1)(a0) + REG_L a2, (SUSPEND_CONTEXT_REGS + PT_A2)(a0) + REG_L a3, (SUSPEND_CONTEXT_REGS + PT_A3)(a0) + REG_L a4, (SUSPEND_CONTEXT_REGS + PT_A4)(a0) + REG_L a5, (SUSPEND_CONTEXT_REGS + PT_A5)(a0) + REG_L a6, (SUSPEND_CONTEXT_REGS + PT_A6)(a0) + REG_L a7, (SUSPEND_CONTEXT_REGS + PT_A7)(a0) + REG_L s2, (SUSPEND_CONTEXT_REGS + PT_S2)(a0) + REG_L s3, (SUSPEND_CONTEXT_REGS + PT_S3)(a0) + REG_L s4, (SUSPEND_CONTEXT_REGS + PT_S4)(a0) + REG_L s5, (SUSPEND_CONTEXT_REGS + PT_S5)(a0) + REG_L s6, (SUSPEND_CONTEXT_REGS + PT_S6)(a0) + REG_L s7, (SUSPEND_CONTEXT_REGS + PT_S7)(a0) + REG_L s8, (SUSPEND_CONTEXT_REGS + PT_S8)(a0) + REG_L s9, (SUSPEND_CONTEXT_REGS + PT_S9)(a0) + REG_L s10, (SUSPEND_CONTEXT_REGS + PT_S10)(a0) + REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0) + .endm + +#endif /* __ASM_ASSEMBLER_H */
arch/riscv/kernel/suspend_entry.S +3 -31
@@ -7,6 +7,7 @@ #include <linux/linkage.h> #include <asm/asm.h> #include <asm/asm-offsets.h> +#include <asm/assembler.h> #include <asm/csr.h> #include <asm/xip_fixup.h>
@@ -83,39 +84,10 @@ ENTRY(__cpu_resume_enter) add a0, a1, zero /* Restore CSRs */ - REG_L t0, (SUSPEND_CONTEXT_REGS + PT_EPC)(a0) - csrw CSR_EPC, t0 - REG_L t0, (SUSPEND_CONTEXT_REGS + PT_STATUS)(a0) - csrw CSR_STATUS, t0 - REG_L t0, (SUSPEND_CONTEXT_REGS + PT_BADADDR)(a0) - csrw CSR_TVAL, t0 - REG_L t0, (SUSPEND_CONTEXT_REGS + PT_CAUSE)(a0) - csrw CSR_CAUSE, t0 + suspend_restore_csrs /* Restore registers (except A0 and T0-T6) */ - REG_L ra, (SUSPEND_CONTEXT_REGS + PT_RA)(a0) - REG_L sp, (SUSPEND_CONTEXT_REGS + PT_SP)(a0) - REG_L gp, (SUSPEND_CONTEXT_REGS + PT_GP)(a0) - REG_L tp, (SUSPEND_CONTEXT_REGS + PT_TP)(a0) - REG_L s0, (SUSPEND_CONTEXT_REGS + PT_S0)(a0) - REG_L s1, (SUSPEND_CONTEXT_REGS + PT_S1)(a0) - REG_L a1, (SUSPEND_CONTEXT_REGS + PT_A1)(a0) - REG_L a2, (SUSPEND_CONTEXT_REGS + PT_A2)(a0) - REG_L a3, (SUSPEND_CONTEXT_REGS + PT_A3)(a0) - REG_L a4, (SUSPEND_CONTEXT_REGS + PT_A4)(a0) - REG_L a5, (SUSPEND_CONTEXT_REGS + PT_A5)(a0) - REG_L a6, (SUSPEND_CONTEXT_REGS + PT_A6)(a0) - REG_L a7, (SUSPEND_CONTEXT_REGS + PT_A7)(a0) - REG_L s2, (SUSPEND_CONTEXT_REGS + PT_S2)(a0) - REG_L s3, (SUSPEND_CONTEXT_REGS + PT_S3)(a0) - REG_L s4, (SUSPEND_CONTEXT_REGS + PT_S4)(a0) - REG_L s5, (SUSPEND_CONTEXT_REGS + PT_S5)(a0) - REG_L s6, (SUSPEND_CONTEXT_REGS + PT_S6)(a0) - REG_L s7, (SUSPEND_CONTEXT_REGS + PT_S7)(a0) - REG_L s8, (SUSPEND_CONTEXT_REGS + PT_S8)(a0) - REG_L s9, (SUSPEND_CONTEXT_REGS + PT_S9)(a0) - REG_L s10, (SUSPEND_CONTEXT_REGS + PT_S10)(a0) - REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0) + suspend_restore_regs /* Return zero value */ add a0, zero, zero
Signature · 1 line
2.34.1
PATCH v5 3/4 RISC-V: mm: Enable huge page support to kernel_page_present() function
In reply to Sia Jee HengCurrently kernel_page_present() function doesn't support huge page detection causes the function to mistakenly return false to the hibernation core.
Add huge page detection to the function to solve the problem.
--- arch/riscv/mm/pageattr.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/riscv/mm/pageattr.c +8 -0
@@ -217,18 +217,26 @@ bool kernel_page_present(struct page *page) pgd =pgd_offset_k (addr);if (!pgd_present (*pgd))return false; +if (pgd_leaf (*pgd)) +return true; p4d =p4d_offset (pgd, addr);if (!p4d_present (*p4d))return false; +if (p4d_leaf (*p4d)) +return true; pud =pud_offset (p4d, addr);if (!pud_present (*pud))return false; +if (pud_leaf (*pud)) +return true; pmd =pmd_offset (pud, addr);if (!pmd_present (*pmd))return false; +if (pmd_leaf (*pmd)) +return true; pte =pte_offset_kernel (pmd, addr);return pte_present (*pte);
Signature · 1 line
2.34.1
PATCH v5 4/4 RISC-V: Add arch functions to support hibernation/suspend-to-disk
In reply to Sia Jee HengLow level Arch functions were created to support hibernation. swsusp_arch_suspend() relies code from __cpu_suspend_enter() to write cpu state onto the stack, then calling swsusp_save() to save the memory image.
Arch specific hibernation header is implemented and is utilized by the arch_hibernation_header_restore() and arch_hibernation_header_save() functions. The arch specific hibernation header consists of satp, hartid, and the cpu_resume address. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume() creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into __hibernate_cpu_resume() to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.
To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
- CONFIG_ARCH_HIBERNATION_HEADER
- CONFIG_ARCH_HIBERNATION_POSSIBLE
--- arch/riscv/Kconfig | 6 + arch/riscv/include/asm/assembler.h | 20 ++ arch/riscv/include/asm/suspend.h | 19 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 424 +++++++++++++++++++++++++++++ 7 files changed, 552 insertions(+) create mode 100644 arch/riscv/kernel/hibernate-asm.S create mode 100644 arch/riscv/kernel/hibernate.c
arch/riscv/Kconfig +6 -0
@@ -707,6 +707,12 @@ menu "Power management options" source "kernel/power/Kconfig" +config ARCH_HIBERNATION_POSSIBLE + def_bool y + +config ARCH_HIBERNATION_HEADER + def_bool HIBERNATION + endmenu # "Power management options" menu "CPU Power Management"
arch/riscv/include/asm/assembler.h +20 -0
@@ -59,4 +59,24 @@ REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0) .endm +/* + * copy_page - copy 1 page (4KB) of data from source to destination + * @a0 - destination + * @a1 - source + */ + .macro copy_page a0, a1 + lui a2, 0x1 + add a2, a2, a0 +1 : + REG_L t0, 0(a1) + REG_L t1, SZREG(a1) + + REG_S t0, 0(a0) + REG_S t1, SZREG(a0) + + addi a0, a0, 2 * SZREG + addi a1, a1, 2 * SZREG + bne a2, a0, 1b + .endm + #endif /* __ASM_ASSEMBLER_H */
arch/riscv/include/asm/suspend.h +19 -0
@@ -21,6 +21,11 @@ struct suspend_context {#endif }; +/* +* Used by hibernation core and cleared during resume sequence +*/ +extern int in_suspend; +/* Low-level CPU suspend entry function */ int __cpu_suspend_enter (struct suspend_context *context);
@@ -36,4 +41,18 @@ int __cpu_resume_enter(unsigned long hartid, unsigned long context);/* Used to save and restore the CSRs */ void suspend_save_csrs (struct suspend_context *context);void suspend_restore_csrs (struct suspend_context *context); + +/* Low-level API to support hibernation */ +int swsusp_arch_suspend (void ); +int swsusp_arch_resume (void ); +int arch_hibernation_header_save (void *addr,unsigned int max_size); +int arch_hibernation_header_restore (void *addr); +int __hibernate_cpu_resume (void ); + +/* Used to resume on the CPU we hibernated on */ +int hibernate_resume_nonboot_cpu_disable (void ); + +asmlinkage void hibernate_restore_image(unsigned long resume_satp, unsigned long satp_temp, +unsigned long cpu_resume); +asmlinkage int hibernate_core_restore_code(void);#endif
arch/riscv/kernel/Makefile +1 -0
@@ -64,6 +64,7 @@ obj-$(CONFIG_MODULES) += module.o obj-$(CONFIG_MODULE_SECTIONS) += module-sections.o obj-$(CONFIG_CPU_PM) += suspend_entry.o suspend.o +obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_FUNCTION_TRACER) += mcount.o ftrace.o obj-$(CONFIG_DYNAMIC_FTRACE) += mcount-dyn.o
arch/riscv/kernel/asm-offsets.c +5 -0
@@ -9,6 +9,7 @@#include <linux/kbuild.h> #include <linux/mm.h> #include <linux/sched.h> +#include <linux/suspend.h> #include <asm/kvm_host.h> #include <asm/thread_info.h> #include <asm/ptrace.h>
@@ -116,6 +117,10 @@ void asm_offsets(void)OFFSET (SUSPEND_CONTEXT_REGS , suspend_context, regs); +OFFSET (HIBERN_PBE_ADDR , pbe, address); +OFFSET (HIBERN_PBE_ORIG , pbe, orig_address); +OFFSET (HIBERN_PBE_NEXT , pbe, next); +OFFSET (KVM_ARCH_GUEST_ZERO , kvm_vcpu_arch, guest_context.zero);OFFSET (KVM_ARCH_GUEST_RA , kvm_vcpu_arch, guest_context.ra);OFFSET (KVM_ARCH_GUEST_SP , kvm_vcpu_arch, guest_context.sp);
arch/riscv/kernel/hibernate-asm.S new file +77 -0
77 lines not on this page · raw message
arch/riscv/kernel/hibernate.c new file +424 -0
424 lines not on this page · raw message
Signature · 1 line
2.34.1
Alexandre Ghiti
reviewerRe: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-disk
Quoted 607 lines · Sia Jee Heng
Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid, and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLESigned-off-by Sia Jee Heng <[email protected]>Reviewed-by Ley Foon Tan <[email protected]>Reviewed-by Mason Huo <[email protected]>--- arch/riscv/Kconfig | 6 + arch/riscv/include/asm/assembler.h | 20 ++ arch/riscv/include/asm/suspend.h | 19 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 424 +++++++++++++++++++++++++++++ 7 files changed, 552 insertions(+)create mode 100644
arch/riscv/kernel/hibernate-asm.Screate mode 100644arch/riscv/kernel/hibernate.carch/riscv/Kconfig +6
-0@@ -707,6 +707,12 @@ menu "Power management options" source "kernel/power/Kconfig" +config ARCH_HIBERNATION_POSSIBLE + def_bool y + +config ARCH_HIBERNATION_HEADER + def_bool HIBERNATION + endmenu # "Power management options" menu "CPU Power Management"arch/riscv/include/asm/assembler.h +20
-0@@ -59,4 +59,24 @@ REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0) .endm +/* + * copy_page - copy 1 page (4KB) of data from source to destination + * @a0 - destination + * @a1 - source + */ + .macro copy_page a0, a1 + lui a2, 0x1 + add a2, a2, a0 +1 : + REG_L t0, 0(a1) + REG_L t1, SZREG(a1) + + REG_S t0, 0(a0) + REG_S t1, SZREG(a0) + + addi a0, a0, 2 * SZREG + addi a1, a1, 2 * SZREG + bne a2, a0, 1b + .endm + #endif /* __ASM_ASSEMBLER_H */arch/riscv/include/asm/suspend.h +19
-0@@ -21,6 +21,11 @@ struct suspend_context {#endif }; +/* +* Used by hibernation core and cleared during resume sequence +*/ +extern int in_suspend; +/* Low-level CPU suspend entry function */ int __cpu_suspend_enter (struct suspend_context *context);@@ -36,4 +41,18 @@ int __cpu_resume_enter(unsigned long hartid, unsigned long context);/* Used to save and restore the CSRs */ void suspend_save_csrs (struct suspend_context *context);void suspend_restore_csrs (struct suspend_context *context); + +/* Low-level API to support hibernation */ +int swsusp_arch_suspend (void ); +int swsusp_arch_resume (void ); +int arch_hibernation_header_save (void *addr,unsigned int max_size); +int arch_hibernation_header_restore (void *addr); +int __hibernate_cpu_resume (void ); + +/* Used to resume on the CPU we hibernated on */ +int hibernate_resume_nonboot_cpu_disable (void ); + +asmlinkage void hibernate_restore_image(unsigned long resume_satp, unsigned long satp_temp, +unsigned long cpu_resume); +asmlinkage int hibernate_core_restore_code(void);#endif arch/riscv/kernel/Makefile +1
-0@@ -64,6 +64,7 @@ obj-$(CONFIG_MODULES) += module.o obj-$(CONFIG_MODULE_SECTIONS) += module-sections.o obj-$(CONFIG_CPU_PM) += suspend_entry.o suspend.o +obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_FUNCTION_TRACER) += mcount.o ftrace.o obj-$(CONFIG_DYNAMIC_FTRACE) += mcount-dyn.oarch/riscv/kernel/asm-offsets.c +5
-0@@ -9,6 +9,7 @@#include <linux/kbuild.h> #include <linux/mm.h> #include <linux/sched.h> +#include <linux/suspend.h> #include <asm/kvm_host.h> #include <asm/thread_info.h> #include <asm/ptrace.h> @@ -116,6 +117,10 @@ void asm_offsets(void)OFFSET (SUSPEND_CONTEXT_REGS , suspend_context, regs); +OFFSET (HIBERN_PBE_ADDR , pbe, address); +OFFSET (HIBERN_PBE_ORIG , pbe, orig_address); +OFFSET (HIBERN_PBE_NEXT , pbe, next); +OFFSET (KVM_ARCH_GUEST_ZERO , kvm_vcpu_arch, guest_context.zero);OFFSET (KVM_ARCH_GUEST_RA , kvm_vcpu_arch, guest_context.ra);OFFSET (KVM_ARCH_GUEST_SP , kvm_vcpu_arch, guest_context.sp);arch/riscv/kernel/hibernate-asm.S new file +77
-0@@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Hibernation low level support for RISCV. + * + * Copyright (C) 2023 StarFive Technology Co., Ltd. + * + * Author: Jee Heng Sia <[email protected]> + */ + +#include <asm/asm.h> +#include <asm/asm-offsets.h> +#include <asm/assembler.h> +#include <asm/csr.h> + +#include <linux/linkage.h> + +/* + * int __hibernate_cpu_resume(void) + * Switch back to the hibernated image's page table prior to restoring the CPU + * context. + * + * Always returns 0 + */ +ENTRY(__hibernate_cpu_resume) + /* switch to hibernated image's page table. */ + csrw CSR_SATP, s0 + sfence.vma + + REG_L a0, hibernate_cpu_context + + suspend_restore_csrs + suspend_restore_regs + + /* Return zero value. */ + mv a0, zero + + ret +END(__hibernate_cpu_resume) + +/* + * Prepare to restore the image. + * a0: satp of saved page tables. + * a1: satp of temporary page tables. + * a2: cpu_resume. + */ +ENTRY(hibernate_restore_image) + mv s0, a0 + mv s1, a1 + mv s2, a2 + REG_L s4, restore_pblist + REG_L a1, relocated_restore_code + + jalr a1 +END(hibernate_restore_image) + +/* + * The below code will be executed from a 'safe' page. + * It first switches to the temporary page table, then starts to copy the pages + * back to the original memory location. Finally, it jumps to __hibernate_cpu_resume() + * to restore the CPU context. + */ +ENTRY(hibernate_core_restore_code) + /* switch to temp page table. */ + csrw satp, s1 + sfence.vma +.Lcopy: + /* The below code will restore the hibernated image. */ + REG_L a1, HIBERN_PBE_ADDR(s4) + REG_L a0, HIBERN_PBE_ORIG(s4) + + copy_page a0, a1 + + REG_L s4, HIBERN_PBE_NEXT(s4) + bnez s4, .Lcopy + + jalr s2 +END(hibernate_core_restore_code)arch/riscv/kernel/hibernate.c new file +363
-0@@ -0,0 +1,424 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Hibernation support for RISCV +* +* Copyright (C) 2023 StarFive Technology Co., Ltd. +* +* Author: Jee Heng Sia <[email protected]> +*/ + +#include <asm/barrier.h> +#include <asm/cacheflush.h> +#include <asm/mmu_context.h> +#include <asm/page.h> +#include <asm/pgalloc.h> +#include <asm/pgtable.h> +#include <asm/sections.h> +#include <asm/set_memory.h> +#include <asm/smp.h> +#include <asm/suspend.h> + +#include <linux/cpu.h> +#include <linux/memblock.h> +#include <linux/pm.h> +#include <linux/sched.h> +#include <linux/suspend.h> +#include <linux/utsname.h> + +/* The logical cpu number we should resume on, initialised to a non-cpu number. */ +static int sleep_cpu = -EINVAL ; + +/* Pointer to the temporary resume page table. */ +static pgd_t *resume_pg_dir; + +/* CPU context to be saved. */ +struct suspend_context *hibernate_cpu_context; +EXPORT_SYMBOL_GPL (hibernate_cpu_context); + +unsigned long relocated_restore_code; +EXPORT_SYMBOL_GPL (relocated_restore_code); + +/** +* struct arch_hibernate_hdr_invariants - container to store kernel build version. +* @uts_version: to save the build number and date so that the we do not resume with +* a different kernel. +*/ +struct arch_hibernate_hdr_invariants { +char uts_version[__NEW_UTS_LEN +1 ]; +}; + +/** +* struct arch_hibernate_hdr - helper parameters that help us to restore the image. +* @invariants: container to store kernel build version. +* @hartid: to make sure same boot_cpu executes the hibernate/restore code. +* @saved_satp: original page table used by the hibernated image. +* @restore_cpu_addr: the kernel's image address to restore the CPU context. +*/ +static struct arch_hibernate_hdr { +struct arch_hibernate_hdr_invariants invariants; +unsigned long hartid; +unsigned long saved_satp; +unsigned long restore_cpu_addr; +} resume_hdr; + +static void arch_hdr_invariants (struct arch_hibernate_hdr_invariants *i) +{ +memset (i,0 ,sizeof (*i)); +memcpy (i->uts_version,init_utsname ()->version,sizeof (i->uts_version)); +} + +/* +* Check if the given pfn is in the 'nosave' section. +*/ +int pfn_is_nosave (unsigned long pfn) +{ +unsigned long nosave_begin_pfn =sym_to_pfn (&__nosave_begin); +unsigned long nosave_end_pfn =sym_to_pfn (&__nosave_end -1 ); + +return ((pfn >= nosave_begin_pfn) && (pfn <= nosave_end_pfn)); +} + +void notrace save_processor_state(void) +{ +WARN_ON (num_online_cpus () !=1 ); +} + +void notrace restore_processor_state(void) +{ +} + +/* +* Helper parameters need to be saved to the hibernation image header. +*/ +int arch_hibernation_header_save (void *addr,unsigned int max_size) +{ +struct arch_hibernate_hdr *hdr = addr; + +if (max_size <sizeof (*hdr)) +return -EOVERFLOW ; + +arch_hdr_invariants (&hdr->invariants); + + hdr->hartid =cpuid_to_hartid_map (sleep_cpu); + hdr->saved_satp =csr_read (CSR_SATP ); + hdr->restore_cpu_addr = (unsigned long )__hibernate_cpu_resume; + +return 0 ; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_save); + +/* +* Retrieve the helper parameters from the hibernation image header. +*/ +int arch_hibernation_header_restore (void *addr) +{ +struct arch_hibernate_hdr_invariants invariants; +struct arch_hibernate_hdr *hdr = addr; +int ret =0 ; + +arch_hdr_invariants (&invariants); + +if (memcmp (&hdr->invariants, &invariants,sizeof (invariants))) { +pr_crit ("Hibernate image not generated by this kernel!\n" ); +return -EINVAL ; + } + + sleep_cpu =riscv_hartid_to_cpuid (hdr->hartid); +if (sleep_cpu <0 ) { +pr_crit ("Hibernated on a CPU not known to this kernel!\n" ); + sleep_cpu = -EINVAL ; +return -EINVAL ; + } + +#ifdef CONFIG_SMP + ret =bringup_hibernate_cpu (sleep_cpu); +if (ret) { + sleep_cpu = -EINVAL ; +return ret; + } +#endif + resume_hdr = *hdr; + +return ret; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_restore); + +int swsusp_arch_suspend (void ) +{ +int ret =0 ; + +if (__cpu_suspend_enter (hibernate_cpu_context)) { + sleep_cpu =smp_processor_id (); +suspend_save_csrs (hibernate_cpu_context); + ret =swsusp_save (); + }else { +suspend_restore_csrs (hibernate_cpu_context); +flush_tlb_all (); +flush_icache_all (); + +/* +* Tell the hibernation core that we've just restored the memory. +*/ + in_suspend =0 ; + sleep_cpu = -EINVAL ; + } + +return ret; +} + +static int temp_pgtable_map_pte (pmd_t *dst_pmdp,pmd_t *src_pmdp,unsigned long start, +unsigned long end,pgprot_t prot) +{ +pte_t *src_ptep; +pte_t *dst_ptep; + +if (pmd_none (READ_ONCE (*dst_pmdp))) { + dst_ptep = (pte_t *)get_safe_page (GFP_ATOMIC ); +if (!dst_ptep) +return -ENOMEM ; + +pmd_populate_kernel (NULL , dst_pmdp, dst_ptep); + } + + dst_ptep =pte_offset_kernel (dst_pmdp, start); + src_ptep =pte_offset_kernel (src_pmdp, start); + +do { +pte_t pte =READ_ONCE (*src_ptep); + +if (pte_present (pte)) +set_pte (dst_ptep,__pte (pte_val (pte) |pgprot_val (prot))); + }while (dst_ptep++, src_ptep++, start +=PAGE_SIZE , start < end); + +return 0 ; +} + +static int temp_pgtable_map_pmd (pud_t *dst_pudp,pud_t *src_pudp,unsigned long start, +unsigned long end,pgprot_t prot) +{ +unsigned long next; +unsigned long ret; +pmd_t *src_pmdp; +pmd_t *dst_pmdp; + +if (pud_none (READ_ONCE (*dst_pudp))) { + dst_pmdp = (pmd_t *)get_safe_page (GFP_ATOMIC ); +if (!dst_pmdp) +return -ENOMEM ; + +pud_populate (NULL , dst_pudp, dst_pmdp); + } + + dst_pmdp =pmd_offset (dst_pudp, start); + src_pmdp =pmd_offset (src_pudp, start); + +do { +pmd_t pmd =READ_ONCE (*src_pmdp); + + next =pmd_addr_end (start, end); + +if (pmd_none (pmd)) +continue ; + +if (pmd_leaf (pmd)) { +set_pmd (dst_pmdp,__pmd (pmd_val (pmd) |pgprot_val (prot))); + }else { + ret =temp_pgtable_map_pte (dst_pmdp, src_pmdp, start, next, prot); +if (ret) +return -ENOMEM ; + } + }while (dst_pmdp++, src_pmdp++, start = next, start != end); + +return 0 ; +} + +static int temp_pgtable_map_pud (p4d_t *dst_p4dp,p4d_t *src_p4dp,unsigned long start, +unsigned long end,pgprot_t prot) +{ +unsigned long next; +unsigned long ret; +pud_t *dst_pudp; +pud_t *src_pudp; + +if (p4d_none (READ_ONCE (*dst_p4dp))) { + dst_pudp = (pud_t *)get_safe_page (GFP_ATOMIC ); +if (!dst_pudp) +return -ENOMEM ; + +p4d_populate (NULL , dst_p4dp, dst_pudp); + } + + dst_pudp =pud_offset (dst_p4dp, start); + src_pudp =pud_offset (src_p4dp, start); + +do { +pud_t pud =READ_ONCE (*src_pudp); + + next =pud_addr_end (start, end); + +if (pud_none (pud)) +continue ; + +if (pud_leaf (pud)) { +set_pud (dst_pudp,__pud (pud_val (pud) |pgprot_val (prot))); + }else { + ret =temp_pgtable_map_pmd (dst_pudp, src_pudp, start, next, prot); +if (ret) +return -ENOMEM ; + } + }while (dst_pudp++, src_pudp++, start = next, start != end); + +return 0 ; +} + +static int temp_pgtable_map_p4d (pgd_t *dst_pgdp,pgd_t *src_pgdp,unsigned long start, +unsigned long end,pgprot_t prot) +{ +unsigned long next; +unsigned long ret; +p4d_t *dst_p4dp; +p4d_t *src_p4dp; + +if (pgd_none (READ_ONCE (*dst_pgdp))) { + dst_p4dp = (p4d_t *)get_safe_page (GFP_ATOMIC ); +if (!dst_p4dp) +return -ENOMEM ; + +pgd_populate (NULL , dst_pgdp, dst_p4dp); + } + + dst_p4dp =p4d_offset (dst_pgdp, start); + src_p4dp =p4d_offset (src_pgdp, start); + +do { +p4d_t p4d =READ_ONCE (*src_p4dp); + + next =p4d_addr_end (start, end); + +if (p4d_none (p4d)) +continue ; + +if (p4d_leaf (p4d)) { +set_p4d (dst_p4dp,__p4d (p4d_val (p4d) |pgprot_val (prot))); + }else { + ret =temp_pgtable_map_pud (dst_p4dp, src_p4dp, start, next, prot); +if (ret) +return -ENOMEM ; + } + }while (dst_p4dp++, src_p4dp++, start = next, start != end); + +return 0 ; +} + +static int temp_pgtable_mapping (pgd_t *pgdp,unsigned long start,unsigned long end,pgprot_t prot) +{ +pgd_t *dst_pgdp =pgd_offset_pgd (pgdp, start); +pgd_t *src_pgdp =pgd_offset_k (start); +unsigned long next; +unsigned long ret; + +do { +pgd_t pgd =READ_ONCE (*src_pgdp); + + next =pgd_addr_end (start, end); + +if (pgd_none (pgd)) +continue ; + +if (pgd_leaf (pgd)) { +set_pgd (dst_pgdp,__pgd (pgd_val (pgd) |pgprot_val (prot))); + }else { + ret =temp_pgtable_map_p4d (dst_pgdp, src_pgdp, start, next, prot); +if (ret) +return -ENOMEM ; + } + }while (dst_pgdp++, src_pgdp++, start = next, start != end); + +return 0 ; +} + +static unsigned long relocate_restore_code (void ) +{ +void *page = (void *)get_safe_page (GFP_ATOMIC ); + +if (!page) +return -ENOMEM ; + +copy_page (page, hibernate_core_restore_code); + +/* Make the page containing the relocated code executable. */ +set_memory_x ((unsigned long )page,1 ); + +return (unsigned long )page; +} + +int swsusp_arch_resume (void ) +{ +unsigned long end = (unsigned long )pfn_to_virt (max_low_pfn); +unsigned long start =PAGE_OFFSET ; +int ret; + +/* +* Memory allocated by get_safe_page() will be dealt with by the hibernation core, +* we don't need to free it here.
arch/riscv/kernel/hibernate.c line 373
Quoted 10 lines
arch/riscv/kernel/hibernate.c new file +10
-0+*/ + resume_pg_dir = (pgd_t *)get_safe_page (GFP_ATOMIC ); +if (!resume_pg_dir) +return -ENOMEM ; + +/* +* Create a temporary page table and map the whole linear region as executable and +* writable. +*/ + ret =temp_pgtable_mapping (resume_pg_dir, start, end,__pgprot (_PAGE_WRITE | _PAGE_EXEC));
Hmm weird protections, PAGE_PRESENT and PAGE_READ are not even set. Shouldn't you use PAGE_KERNEL_EXEC instead?
Quoted 11 lines
arch/riscv/kernel/hibernate.c new file +11
-0+if (ret) +return ret; + +/* Move the restore code to a new page so that it doesn't get overwritten by itself. */ + relocated_restore_code =relocate_restore_code (); +if (relocated_restore_code == -ENOMEM ) +return -ENOMEM ; + +/* +* Map the __hibernate_cpu_resume() address to the temporary page table so that the +* restore code can jumps to it after finished restore the image. The next execution
arch/riscv/kernel/hibernate.c line 394
Quoted 10 lines
arch/riscv/kernel/hibernate.c new file +10
-0+* code doesn't find itself in a different address space after switching over to the +* original page table used by the hibernated image. +* +*/ + start = (unsigned long )resume_hdr.restore_cpu_addr; + end = start +PAGE_SIZE ; + + ret =temp_pgtable_mapping (resume_pg_dir, start, end,__pgprot (_PAGE_WRITE | _PAGE_EXEC)); +if (ret) +return ret;
You set the linear mapping as executable so no need for this.
Quoted 20 lines
arch/riscv/kernel/hibernate.c new file +20
-0+ +hibernate_restore_image (resume_hdr.saved_satp, (PFN_DOWN (__pa (resume_pg_dir)) | satp_mode), + resume_hdr.restore_cpu_addr); + +return 0 ; +} + +#ifdef CONFIG_PM_SLEEP_SMP +int hibernate_resume_nonboot_cpu_disable (void ) +{ +if (sleep_cpu <0 ) { +pr_err ("Failing to resume from hibernate on an unknown CPU\n" ); +return -ENODEV ; + } + +return freeze_secondary_cpus (sleep_cpu); +} +#endif + +static int __init riscv_hibernate_init (void )
arch/riscv/kernel/hibernate.c line 424
Quoted 10 lines
arch/riscv/kernel/hibernate.c new file +10
-0+{ + hibernate_cpu_context =kzalloc (sizeof (*hibernate_cpu_context),GFP_KERNEL ); + +if (WARN_ON (!hibernate_cpu_context)) +return -ENOMEM ; + +return 0 ; +} + +early_initcall (riscv_hibernate_init);
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Alexandre Ghiti
reviewerRe: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-disk
Quoted 629 lines · Alexandre Ghiti
Quoted 627 lines · Sia Jee Heng
Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid, and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLESigned-off-by Sia Jee Heng <[email protected]>Reviewed-by Ley Foon Tan <[email protected]>Reviewed-by Mason Huo <[email protected]>---
arch/riscv/Kconfig | 6 +arch/riscv/include/asm/assembler.h| 20 ++ arch/riscv/include/asm/suspend.h | 19 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 424 +++++++++++++++++++++++++++++7 files changed, 552 insertions(+)create mode 100644
arch/riscv/kernel/hibernate-asm.Screate mode 100644arch/riscv/kernel/hibernate.carch/riscv/Kconfig +0
-0@@ -707,6 +707,12 @@ menu "Power management options"source "
kernel/power/Kconfig"
+configARCH_HIBERNATION_POSSIBLE
+def_boolyarch/riscv/Kconfig +4
-0+ +config ARCH_HIBERNATION_HEADER + def_bool HIBERNATION +endmenu # "Power management options"
menu "CPU Power Management"arch/riscv/include/asm/assembler.h +0
-0@@ -59,4 +59,24 @@ REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0).endm
+/*
+ *copy_page- copy 1 page (4KB) of data from source to destination
+ * @a0 - destination
+ * @a1 - source
+ */
+ .macrocopy_pagea0, a1
+ lui a2,0x1
+ add a2, a2, a0
+1 :
+ REG_L t0, 0(a1)
+ REG_L t1, SZREG(a1)arch/riscv/include/asm/assembler.h +9
-0+ + REG_S t0, 0(a0) + REG_S t1, SZREG(a0) + + addi a0, a0, 2 * SZREG + addi a1, a1, 2 * SZREG + bne a2, a0, 1b + .endm +#endif /*
__ASM_ASSEMBLER_H*/arch/riscv/include/asm/suspend.h +0
-0@@ -21,6 +21,11 @@ struct suspend_context { #endif };+/*
+ * Used by hibernation core and cleared during resume sequence
+ */arch/riscv/include/asm/suspend.h +2
-0+extern int in_suspend; +/* Low-level CPU suspend entry function */
int__cpu_suspend_enter(struct suspend_context *context);
@@ -36,4 +41,18 @@ int__cpu_resume_enter(unsigned long hartid, unsigned long context); /* Used to save and restore the CSRs */
voidsuspend_save_csrs(struct suspend_context *context);
voidsuspend_restore_csrs(struct suspend_context *context);arch/riscv/include/asm/suspend.h +12
-0+ +/* Low-level API to support hibernation */ +int swsusp_arch_suspend (void ); +int swsusp_arch_resume (void ); +int arch_hibernation_header_save (void *addr,unsigned int max_size); +int arch_hibernation_header_restore (void *addr); +int __hibernate_cpu_resume (void ); + +/* Used to resume on the CPU we hibernated on */ +int hibernate_resume_nonboot_cpu_disable (void ); + +asmlinkage void hibernate_restore_image(unsigned long resume_satp,unsigned long
satp_temp,
+ unsigned longcpu_resume); +asmlinkage inthibernate_core_restore_code(void); #endifarch/riscv/kernel/Makefile +0
-0@@ -64,6 +64,7 @@ obj-$(CONFIG_MODULES) += module.oobj-$(CONFIG_MODULE_SECTIONS) += module-sections.o obj-$(CONFIG_CPU_PM) += suspend_entry.o suspend.o +obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_FUNCTION_TRACER) += mcount.o ftrace.o obj-$(CONFIG_DYNAMIC_FTRACE) += mcount-dyn.o
arch/riscv/kernel/asm-offsets.c +4
-0@@ -9,6 +9,7 @@ #include <linux/kbuild.h> #include <linux/mm.h> #include <linux/sched.h> +#include <linux/suspend.h> #include <asm/kvm_host.h> #include <asm/thread_info.h> #include <asm/ptrace.h>@@ -116,6 +117,10 @@ void asm_offsets(void) OFFSET(SUSPEND_CONTEXT_REGS, suspend_context, regs); + OFFSET(HIBERN_PBE_ADDR, pbe, address); + OFFSET(HIBERN_PBE_ORIG, pbe, orig_address); + OFFSET(HIBERN_PBE_NEXT, pbe, next); + OFFSET(KVM_ARCH_GUEST_ZERO, kvm_vcpu_arch, guest_context.zero); OFFSET(KVM_ARCH_GUEST_RA, kvm_vcpu_arch, guest_context.ra); OFFSET(KVM_ARCH_GUEST_SP, kvm_vcpu_arch, guest_context.sp);arch/riscv/kernel/hibernate-asm.S new file +19
-0@@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Hibernation low level support for RISCV. + * + * Copyright (C) 2023 StarFive Technology Co., Ltd. + * + * Author: Jee Heng Sia <[email protected]> + */ + +#include <asm/asm.h> +#include <asm/asm-offsets.h> +#include <asm/assembler.h> +#include <asm/csr.h> + +#include <linux/linkage.h> + +/* + * int __hibernate_cpu_resume(void) + * Switch back to the hibernated image's page table prior torestoring the CPU
+ * context.
+ *
+ * Always returns 0
+ */arch/riscv/kernel/hibernate-asm.S new file +35
-0+ENTRY(__hibernate_cpu_resume) + /* switch to hibernated image's page table. */ + csrw CSR_SATP, s0 + sfence.vma + + REG_L a0, hibernate_cpu_context + + suspend_restore_csrs + suspend_restore_regs + + /* Return zero value. */ + mv a0, zero + + ret +END(__hibernate_cpu_resume) + +/* + * Prepare to restore the image. + * a0: satp of saved page tables. + * a1: satp of temporary page tables. + * a2: cpu_resume. + */ +ENTRY(hibernate_restore_image) + mv s0, a0 + mv s1, a1 + mv s2, a2 + REG_L s4, restore_pblist + REG_L a1, relocated_restore_code + + jalr a1 +END(hibernate_restore_image) + +/* + * The below code will be executed from a 'safe' page. + * It first switches to the temporary page table, then starts tocopy the pages
+ * back to the original memory location. Finally, it jumps to__hibernate_cpu_resume()
+ * to restore the CPU context.
+ */arch/riscv/kernel/hibernate-asm.S new file +16
-0+ENTRY(hibernate_core_restore_code) + /* switch to temp page table. */ + csrw satp, s1 + sfence.vma +.Lcopy: + /* The below code will restore the hibernated image. */ + REG_L a1, HIBERN_PBE_ADDR(s4) + REG_L a0, HIBERN_PBE_ORIG(s4) + + copy_page a0, a1 + + REG_L s4, HIBERN_PBE_NEXT(s4) + bnez s4, .Lcopy + + jalr s2 +END(hibernate_core_restore_code)arch/riscv/kernel/hibernate.c new file +28
-0@@ -0,0 +1,424 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Hibernation support for RISCV +* +* Copyright (C) 2023 StarFive Technology Co., Ltd. +* +* Author: Jee Heng Sia <[email protected]> +*/ + +#include <asm/barrier.h> +#include <asm/cacheflush.h> +#include <asm/mmu_context.h> +#include <asm/page.h> +#include <asm/pgalloc.h> +#include <asm/pgtable.h> +#include <asm/sections.h> +#include <asm/set_memory.h> +#include <asm/smp.h> +#include <asm/suspend.h> + +#include <linux/cpu.h> +#include <linux/memblock.h> +#include <linux/pm.h> +#include <linux/sched.h> +#include <linux/suspend.h> +#include <linux/utsname.h> + +/* The logical cpu number we should resume on, initialised to a non-cpu number. */
arch/riscv/kernel/hibernate.c new file +14
-0+static int sleep_cpu = -EINVAL ; + +/* Pointer to the temporary resume page table. */ +static pgd_t *resume_pg_dir; + +/* CPU context to be saved. */ +struct suspend_context *hibernate_cpu_context; +EXPORT_SYMBOL_GPL (hibernate_cpu_context); + +unsigned long relocated_restore_code; +EXPORT_SYMBOL_GPL (relocated_restore_code); + +/** +* struct arch_hibernate_hdr_invariants - container to store kernel build version.
+ * @uts_version: to save the build number and date so that the we do not resume with
+ * a different kernel.
+ */arch/riscv/kernel/hibernate.c new file +6
-0+struct arch_hibernate_hdr_invariants { + char uts_version[__NEW_UTS_LEN + 1]; +}; + +/** +* struct arch_hibernate_hdr - helper parameters that help us to restore the image.
+ * @invariants: container to store kernel build version.
+ * @hartid: to make sure sameboot_cpuexecutes the hibernate/restore code.
+ * @saved_satp: original page table used by the hibernated image.
+ * @restore_cpu_addr: the kernel's image address to restore the CPU context.
+ */arch/riscv/kernel/hibernate.c new file +8
-0+static struct arch_hibernate_hdr { + struct arch_hibernate_hdr_invariants invariants; + unsigned long hartid; + unsigned long saved_satp; + unsigned long restore_cpu_addr; +} resume_hdr; + +static void arch_hdr_invariants(struct arch_hibernate_hdr_invariants*i)
+{
+memset(i, 0, sizeof(*i));
+memcpy(i->uts_version, init_utsname()->version, sizeof(i->uts_version));arch/riscv/kernel/hibernate.c new file +93
-0+} + +/* +* Check if the given pfn is in the 'nosave' section. +*/ +int pfn_is_nosave (unsigned long pfn) +{ + unsigned long nosave_begin_pfn = sym_to_pfn(&__nosave_begin); + unsigned long nosave_end_pfn = sym_to_pfn(&__nosave_end - 1); + + return ((pfn >= nosave_begin_pfn) && (pfn <= nosave_end_pfn)); +} + +void notrace save_processor_state(void) +{ + WARN_ON(num_online_cpus() != 1); +} + +void notrace restore_processor_state(void) +{ +} + +/* +* Helper parameters need to be saved to the hibernation image header. +*/ +int arch_hibernation_header_save (void *addr,unsigned int max_size) +{ + struct arch_hibernate_hdr *hdr = addr; + + if (max_size < sizeof(*hdr)) + return -EOVERFLOW; + + arch_hdr_invariants(&hdr->invariants); + + hdr->hartid = cpuid_to_hartid_map(sleep_cpu); + hdr->saved_satp = csr_read(CSR_SATP); + hdr->restore_cpu_addr = (unsigned long)__hibernate_cpu_resume; + + return 0; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_save); + +/* +* Retrieve the helper parameters from the hibernation image header. +*/ +int arch_hibernation_header_restore (void *addr) +{ + struct arch_hibernate_hdr_invariants invariants; + struct arch_hibernate_hdr *hdr = addr; + int ret = 0; + + arch_hdr_invariants(&invariants); + + if (memcmp(&hdr->invariants, &invariants, sizeof(invariants))) { + pr_crit("Hibernate image not generated by this kernel!\n"); + return -EINVAL; + } + + sleep_cpu = riscv_hartid_to_cpuid(hdr->hartid); + if (sleep_cpu < 0) { + pr_crit("Hibernated on a CPU not known to this kernel!\n"); + sleep_cpu = -EINVAL; + return -EINVAL; + } + +#ifdef CONFIG_SMP + ret = bringup_hibernate_cpu(sleep_cpu); + if (ret) { + sleep_cpu = -EINVAL; + return ret; + } +#endif + resume_hdr = *hdr; + + return ret; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_restore); + +int swsusp_arch_suspend (void ) +{ + int ret = 0; + + if (__cpu_suspend_enter(hibernate_cpu_context)) { + sleep_cpu = smp_processor_id(); + suspend_save_csrs(hibernate_cpu_context); + ret = swsusp_save(); + } else { + suspend_restore_csrs(hibernate_cpu_context); + flush_tlb_all(); + flush_icache_all(); + + /* +* Tell the hibernation core that we've just restored the memory.
+ */
+in_suspend= 0;
+sleep_cpu=-EINVAL;
+ }arch/riscv/kernel/hibernate.c new file +5
-0+ + return ret; +} + +static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +26
-0+{ + pte_t *src_ptep; + pte_t *dst_ptep; + + if (pmd_none(READ_ONCE(*dst_pmdp))) { + dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC); + if (!dst_ptep) + return -ENOMEM; + + pmd_populate_kernel(NULL, dst_pmdp, dst_ptep); + } + + dst_ptep = pte_offset_kernel(dst_pmdp, start); + src_ptep = pte_offset_kernel(src_pmdp, start); + + do { + pte_t pte = READ_ONCE(*src_ptep); + + if (pte_present(pte)) + set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); + } while (dst_ptep++, src_ptep++, start += PAGE_SIZE, start < end); + + return 0; +} + +static int temp_pgtable_map_pmd(pud_t *dst_pudp, pud_t *src_pudp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + pmd_t *src_pmdp; + pmd_t *dst_pmdp; + + if (pud_none(READ_ONCE(*dst_pudp))) { + dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC); + if (!dst_pmdp) + return -ENOMEM; + + pud_populate(NULL, dst_pudp, dst_pmdp); + } + + dst_pmdp = pmd_offset(dst_pudp, start); + src_pmdp = pmd_offset(src_pudp, start); + + do { + pmd_t pmd = READ_ONCE(*src_pmdp); + + next = pmd_addr_end(start, end); + + if (pmd_none(pmd)) + continue; + + if (pmd_leaf(pmd)) { + set_pmd(dst_pmdp, __pmd(pmd_val(pmd) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pte(dst_pmdp, src_pmdp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pmdp++, src_pmdp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_map_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + pud_t *dst_pudp; + pud_t *src_pudp; + + if (p4d_none(READ_ONCE(*dst_p4dp))) { + dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC); + if (!dst_pudp) + return -ENOMEM; + + p4d_populate(NULL, dst_p4dp, dst_pudp); + } + + dst_pudp = pud_offset(dst_p4dp, start); + src_pudp = pud_offset(src_p4dp, start); + + do { + pud_t pud = READ_ONCE(*src_pudp); + + next = pud_addr_end(start, end); + + if (pud_none(pud)) + continue; + + if (pud_leaf(pud)) { + set_pud(dst_pudp, __pud(pud_val(pud) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pmd(dst_pudp, src_pudp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pudp++, src_pudp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_map_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + p4d_t *dst_p4dp; + p4d_t *src_p4dp; + + if (pgd_none(READ_ONCE(*dst_pgdp))) { + dst_p4dp = (p4d_t *)get_safe_page(GFP_ATOMIC); + if (!dst_p4dp) + return -ENOMEM; + + pgd_populate(NULL, dst_pgdp, dst_p4dp); + } + + dst_p4dp = p4d_offset(dst_pgdp, start); + src_p4dp = p4d_offset(src_pgdp, start); + + do { + p4d_t p4d = READ_ONCE(*src_p4dp); + + next = p4d_addr_end(start, end); + + if (p4d_none(p4d)) + continue; + + if (p4d_leaf(p4d)) { + set_p4d(dst_p4dp, __p4d(p4d_val(p4d) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pud(dst_p4dp, src_p4dp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_p4dp++, src_p4dp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_mapping(pgd_t *pgdp, unsigned long start,unsigned long end,
pgprot_tprot)arch/riscv/kernel/hibernate.c new file +18
-0+{ + pgd_t *dst_pgdp = pgd_offset_pgd(pgdp, start); + pgd_t *src_pgdp = pgd_offset_k(start); + unsigned long next; + unsigned long ret; + + do { + pgd_t pgd = READ_ONCE(*src_pgdp); + + next = pgd_addr_end(start, end); + + if (pgd_none(pgd)) + continue; + + if (pgd_leaf(pgd)) { + set_pgd(dst_pgdp, __pgd(pgd_val(pgd) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_p4d(dst_pgdp, src_pgdp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pgdp++, src_pgdp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +27
-0+ + return 0; +} + +static unsigned long relocate_restore_code (void ) +{ + void *page = (void *)get_safe_page(GFP_ATOMIC); + + if (!page) + return -ENOMEM; + + copy_page(page, hibernate_core_restore_code); + + /* Make the page containing the relocated code executable. */ + set_memory_x((unsigned long)page, 1); + + return (unsigned long)page; +} + +int swsusp_arch_resume (void ) +{ + unsigned long end = (unsigned long)pfn_to_virt(max_low_pfn); + unsigned long start = PAGE_OFFSET; + int ret; + + /* +* Memory allocated by get_safe_page() will be dealt with by the hibernation core,
+ * we don't need to free it here.
+ */
+resume_pg_dir= (pgd_t*)get_safe_page(GFP_ATOMIC);
+ if (!resume_pg_dir)
+ return-ENOMEM;
+
+ /*
+ * Create a temporary page table and map the whole linear region
Quoted 8 lines
Quoted 5 lines
as executable and
+ * writable.
+ */
+ ret =temp_pgtable_mapping(resume_pg_dir, start, end, __pgprot(_PAGE_WRITE | _PAGE_EXEC));Hmm weird protections,
PAGE_PRESENTandPAGE_READare not even set. Shouldn't you usePAGE_KERNEL_EXECinstead?
Forget this one, that's the protections you add, so that's good :)
Quoted 70 lines
Quoted 20 lines
+ if (ret)
+ return ret;
+
+ /* Move the restore code to a new page so that it doesn't get overwritten by itself. */
+relocated_restore_code=relocate_restore_code();
+ if (relocated_restore_code==-ENOMEM)
+ return-ENOMEM;
+
+ /*
+ * Map the__hibernate_cpu_resume()address to the temporary page table so that the
+ * restore code can jumps to it after finished restore the image. The next execution
+ * code doesn't find itself in a different address space after switching over to the
+ * original page table used by the hibernated image. + *Quoted 8 lines
+ */
+ start = (unsigned long)resume_hdr.restore_cpu_addr; + end = start +PAGE_SIZE;
+
+ ret =temp_pgtable_mapping(resume_pg_dir, start, end, __pgprot(_PAGE_WRITE | _PAGE_EXEC));
+ if (ret)
+ return ret;You set the linear mapping as executable so no need for this.
Quoted 24 lines
+
+ hibernate_restore_image(resume_hdr.saved_satp, (PFN_DOWN(__pa(resume_pg_dir)) |satp_mode), + resume_hdr.restore_cpu_addr);arch/riscv/kernel/hibernate.c new file +19
-0+ + return 0; +} + +#ifdef CONFIG_PM_SLEEP_SMP +int hibernate_resume_nonboot_cpu_disable (void ) +{ + if (sleep_cpu < 0) { + pr_err("Failing to resume from hibernate on an unknown CPU\n"); + return -ENODEV; + } + + return freeze_secondary_cpus(sleep_cpu); +} +#endif + +static int __init riscv_hibernate_init (void ) +{ + hibernate_cpu_context = kzalloc(sizeof(*hibernate_cpu_context),
GFP_KERNEL);arch/riscv/kernel/hibernate.c line 424
Quoted 8 lines
arch/riscv/kernel/hibernate.c new file +8
-0+ + if (WARN_ON(!hibernate_cpu_context)) + return -ENOMEM; + + return 0; +} + +early_initcall (riscv_hibernate_init);_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
RE: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-disk
Quoted 659 lines
-----Original Message-----
From: Alexandre Ghiti <[email protected]>
Sent: Wednesday, March 8, 2023 5:28 PM
To: JeeHeng Sia <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>
Subject: Re: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-diskQuoted 637 lines · Alexandre Ghiti
Quoted 635 lines · Sia Jee Heng
Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid,
and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLESigned-off-by Sia Jee Heng <[email protected]>Reviewed-by Ley Foon Tan <[email protected]>Reviewed-by Mason Huo <[email protected]>---
arch/riscv/Kconfig | 6 +arch/riscv/include/asm/assembler.h| 20 ++ arch/riscv/include/asm/suspend.h | 19 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 424 +++++++++++++++++++++++++++++7 files changed, 552 insertions(+)create mode 100644
arch/riscv/kernel/hibernate-asm.Screate mode 100644arch/riscv/kernel/hibernate.carch/riscv/Kconfig +0
-0@@ -707,6 +707,12 @@ menu "Power management options"source "
kernel/power/Kconfig"
+configARCH_HIBERNATION_POSSIBLE
+def_boolyarch/riscv/Kconfig +4
-0+ +config ARCH_HIBERNATION_HEADER + def_bool HIBERNATION +endmenu # "Power management options"
menu "CPU Power Management"arch/riscv/include/asm/assembler.h +0
-0@@ -59,4 +59,24 @@ REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0).endm
+/*
+ *copy_page- copy 1 page (4KB) of data from source to destination
+ * @a0 - destination
+ * @a1 - source
+ */
+ .macrocopy_pagea0, a1
+ lui a2,0x1
+ add a2, a2, a0
+1 :
+ REG_L t0, 0(a1)
+ REG_L t1, SZREG(a1)arch/riscv/include/asm/assembler.h +9
-0+ + REG_S t0, 0(a0) + REG_S t1, SZREG(a0) + + addi a0, a0, 2 * SZREG + addi a1, a1, 2 * SZREG + bne a2, a0, 1b + .endm +#endif /*
__ASM_ASSEMBLER_H*/arch/riscv/include/asm/suspend.h +0
-0@@ -21,6 +21,11 @@ struct suspend_context { #endif };+/*
+ * Used by hibernation core and cleared during resume sequence
+ */arch/riscv/include/asm/suspend.h +2
-0+extern int in_suspend; +/* Low-level CPU suspend entry function */
int__cpu_suspend_enter(struct suspend_context *context); @@ -36,4 +41,18 @@ int__cpu_resume_enter(unsigned long hartid, unsigned long context);
/* Used to save and restore the CSRs */
voidsuspend_save_csrs(struct suspend_context *context); voidsuspend_restore_csrs(struct suspend_context *context);arch/riscv/include/asm/suspend.h +12
-0+ +/* Low-level API to support hibernation */ +int swsusp_arch_suspend (void ); +int swsusp_arch_resume (void ); +int arch_hibernation_header_save (void *addr,unsigned int max_size); +int arch_hibernation_header_restore (void *addr); +int __hibernate_cpu_resume (void ); + +/* Used to resume on the CPU we hibernated on */ +int hibernate_resume_nonboot_cpu_disable (void ); + +asmlinkage void hibernate_restore_image(unsigned long resume_satp,unsigned long
satp_temp,
+ unsigned longcpu_resume); +asmlinkage inthibernate_core_restore_code(void); #endifarch/riscv/kernel/Makefile +0
-0@@ -64,6 +64,7 @@ obj-$(CONFIG_MODULES) += module.oobj-$(CONFIG_MODULE_SECTIONS) += module-sections.o obj-$(CONFIG_CPU_PM) += suspend_entry.o suspend.o +obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_FUNCTION_TRACER) += mcount.o ftrace.o obj-$(CONFIG_DYNAMIC_FTRACE) += mcount-dyn.o
arch/riscv/kernel/asm-offsets.c +4
-0@@ -9,6 +9,7 @@ #include <linux/kbuild.h> #include <linux/mm.h> #include <linux/sched.h> +#include <linux/suspend.h> #include <asm/kvm_host.h> #include <asm/thread_info.h> #include <asm/ptrace.h>@@ -116,6 +117,10 @@ void asm_offsets(void) OFFSET(SUSPEND_CONTEXT_REGS, suspend_context, regs); + OFFSET(HIBERN_PBE_ADDR, pbe, address); + OFFSET(HIBERN_PBE_ORIG, pbe, orig_address); + OFFSET(HIBERN_PBE_NEXT, pbe, next); + OFFSET(KVM_ARCH_GUEST_ZERO, kvm_vcpu_arch, guest_context.zero); OFFSET(KVM_ARCH_GUEST_RA, kvm_vcpu_arch, guest_context.ra); OFFSET(KVM_ARCH_GUEST_SP, kvm_vcpu_arch, guest_context.sp);arch/riscv/kernel/hibernate-asm.S new file +19
-0@@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Hibernation low level support for RISCV. + * + * Copyright (C) 2023 StarFive Technology Co., Ltd. + * + * Author: Jee Heng Sia <[email protected]> + */ + +#include <asm/asm.h> +#include <asm/asm-offsets.h> +#include <asm/assembler.h> +#include <asm/csr.h> + +#include <linux/linkage.h> + +/* + * int __hibernate_cpu_resume(void) + * Switch back to the hibernated image's page table prior torestoring the CPU
+ * context.
+ *
+ * Always returns 0
+ */arch/riscv/kernel/hibernate-asm.S new file +35
-0+ENTRY(__hibernate_cpu_resume) + /* switch to hibernated image's page table. */ + csrw CSR_SATP, s0 + sfence.vma + + REG_L a0, hibernate_cpu_context + + suspend_restore_csrs + suspend_restore_regs + + /* Return zero value. */ + mv a0, zero + + ret +END(__hibernate_cpu_resume) + +/* + * Prepare to restore the image. + * a0: satp of saved page tables. + * a1: satp of temporary page tables. + * a2: cpu_resume. + */ +ENTRY(hibernate_restore_image) + mv s0, a0 + mv s1, a1 + mv s2, a2 + REG_L s4, restore_pblist + REG_L a1, relocated_restore_code + + jalr a1 +END(hibernate_restore_image) + +/* + * The below code will be executed from a 'safe' page. + * It first switches to the temporary page table, then starts tocopy the pages
+ * back to the original memory location. Finally, it jumps to__hibernate_cpu_resume()
+ * to restore the CPU context.
+ */arch/riscv/kernel/hibernate-asm.S new file +16
-0+ENTRY(hibernate_core_restore_code) + /* switch to temp page table. */ + csrw satp, s1 + sfence.vma +.Lcopy: + /* The below code will restore the hibernated image. */ + REG_L a1, HIBERN_PBE_ADDR(s4) + REG_L a0, HIBERN_PBE_ORIG(s4) + + copy_page a0, a1 + + REG_L s4, HIBERN_PBE_NEXT(s4) + bnez s4, .Lcopy + + jalr s2 +END(hibernate_core_restore_code)arch/riscv/kernel/hibernate.c new file +28
-0@@ -0,0 +1,424 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Hibernation support for RISCV +* +* Copyright (C) 2023 StarFive Technology Co., Ltd. +* +* Author: Jee Heng Sia <[email protected]> +*/ + +#include <asm/barrier.h> +#include <asm/cacheflush.h> +#include <asm/mmu_context.h> +#include <asm/page.h> +#include <asm/pgalloc.h> +#include <asm/pgtable.h> +#include <asm/sections.h> +#include <asm/set_memory.h> +#include <asm/smp.h> +#include <asm/suspend.h> + +#include <linux/cpu.h> +#include <linux/memblock.h> +#include <linux/pm.h> +#include <linux/sched.h> +#include <linux/suspend.h> +#include <linux/utsname.h> + +/* The logical cpu number we should resume on, initialised to a non-cpu number. */
arch/riscv/kernel/hibernate.c new file +14
-0+static int sleep_cpu = -EINVAL ; + +/* Pointer to the temporary resume page table. */ +static pgd_t *resume_pg_dir; + +/* CPU context to be saved. */ +struct suspend_context *hibernate_cpu_context; +EXPORT_SYMBOL_GPL (hibernate_cpu_context); + +unsigned long relocated_restore_code; +EXPORT_SYMBOL_GPL (relocated_restore_code); + +/** +* struct arch_hibernate_hdr_invariants - container to store kernel build version.
+ * @uts_version: to save the build number and date so that the we do not resume with
+ * a different kernel.
+ */arch/riscv/kernel/hibernate.c new file +6
-0+struct arch_hibernate_hdr_invariants { + char uts_version[__NEW_UTS_LEN + 1]; +}; + +/** +* struct arch_hibernate_hdr - helper parameters that help us to restore the image.
+ * @invariants: container to store kernel build version.
+ * @hartid: to make sure sameboot_cpuexecutes the hibernate/restore code.
+ * @saved_satp: original page table used by the hibernated image.
+ * @restore_cpu_addr: the kernel's image address to restore the CPU context.
+ */arch/riscv/kernel/hibernate.c new file +8
-0+static struct arch_hibernate_hdr { + struct arch_hibernate_hdr_invariants invariants; + unsigned long hartid; + unsigned long saved_satp; + unsigned long restore_cpu_addr; +} resume_hdr; + +static void arch_hdr_invariants(struct arch_hibernate_hdr_invariants*i)
+{
+memset(i, 0, sizeof(*i));
+memcpy(i->uts_version, init_utsname()->version, sizeof(i->uts_version));arch/riscv/kernel/hibernate.c new file +93
-0+} + +/* +* Check if the given pfn is in the 'nosave' section. +*/ +int pfn_is_nosave (unsigned long pfn) +{ + unsigned long nosave_begin_pfn = sym_to_pfn(&__nosave_begin); + unsigned long nosave_end_pfn = sym_to_pfn(&__nosave_end - 1); + + return ((pfn >= nosave_begin_pfn) && (pfn <= nosave_end_pfn)); +} + +void notrace save_processor_state(void) +{ + WARN_ON(num_online_cpus() != 1); +} + +void notrace restore_processor_state(void) +{ +} + +/* +* Helper parameters need to be saved to the hibernation image header. +*/ +int arch_hibernation_header_save (void *addr,unsigned int max_size) +{ + struct arch_hibernate_hdr *hdr = addr; + + if (max_size < sizeof(*hdr)) + return -EOVERFLOW; + + arch_hdr_invariants(&hdr->invariants); + + hdr->hartid = cpuid_to_hartid_map(sleep_cpu); + hdr->saved_satp = csr_read(CSR_SATP); + hdr->restore_cpu_addr = (unsigned long)__hibernate_cpu_resume; + + return 0; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_save); + +/* +* Retrieve the helper parameters from the hibernation image header. +*/ +int arch_hibernation_header_restore (void *addr) +{ + struct arch_hibernate_hdr_invariants invariants; + struct arch_hibernate_hdr *hdr = addr; + int ret = 0; + + arch_hdr_invariants(&invariants); + + if (memcmp(&hdr->invariants, &invariants, sizeof(invariants))) { + pr_crit("Hibernate image not generated by this kernel!\n"); + return -EINVAL; + } + + sleep_cpu = riscv_hartid_to_cpuid(hdr->hartid); + if (sleep_cpu < 0) { + pr_crit("Hibernated on a CPU not known to this kernel!\n"); + sleep_cpu = -EINVAL; + return -EINVAL; + } + +#ifdef CONFIG_SMP + ret = bringup_hibernate_cpu(sleep_cpu); + if (ret) { + sleep_cpu = -EINVAL; + return ret; + } +#endif + resume_hdr = *hdr; + + return ret; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_restore); + +int swsusp_arch_suspend (void ) +{ + int ret = 0; + + if (__cpu_suspend_enter(hibernate_cpu_context)) { + sleep_cpu = smp_processor_id(); + suspend_save_csrs(hibernate_cpu_context); + ret = swsusp_save(); + } else { + suspend_restore_csrs(hibernate_cpu_context); + flush_tlb_all(); + flush_icache_all(); + + /* +* Tell the hibernation core that we've just restored the memory.
+ */
+in_suspend= 0;
+sleep_cpu=-EINVAL;
+ }arch/riscv/kernel/hibernate.c new file +5
-0+ + return ret; +} + +static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +26
-0+{ + pte_t *src_ptep; + pte_t *dst_ptep; + + if (pmd_none(READ_ONCE(*dst_pmdp))) { + dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC); + if (!dst_ptep) + return -ENOMEM; + + pmd_populate_kernel(NULL, dst_pmdp, dst_ptep); + } + + dst_ptep = pte_offset_kernel(dst_pmdp, start); + src_ptep = pte_offset_kernel(src_pmdp, start); + + do { + pte_t pte = READ_ONCE(*src_ptep); + + if (pte_present(pte)) + set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); + } while (dst_ptep++, src_ptep++, start += PAGE_SIZE, start < end); + + return 0; +} + +static int temp_pgtable_map_pmd(pud_t *dst_pudp, pud_t *src_pudp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + pmd_t *src_pmdp; + pmd_t *dst_pmdp; + + if (pud_none(READ_ONCE(*dst_pudp))) { + dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC); + if (!dst_pmdp) + return -ENOMEM; + + pud_populate(NULL, dst_pudp, dst_pmdp); + } + + dst_pmdp = pmd_offset(dst_pudp, start); + src_pmdp = pmd_offset(src_pudp, start); + + do { + pmd_t pmd = READ_ONCE(*src_pmdp); + + next = pmd_addr_end(start, end); + + if (pmd_none(pmd)) + continue; + + if (pmd_leaf(pmd)) { + set_pmd(dst_pmdp, __pmd(pmd_val(pmd) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pte(dst_pmdp, src_pmdp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pmdp++, src_pmdp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_map_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + pud_t *dst_pudp; + pud_t *src_pudp; + + if (p4d_none(READ_ONCE(*dst_p4dp))) { + dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC); + if (!dst_pudp) + return -ENOMEM; + + p4d_populate(NULL, dst_p4dp, dst_pudp); + } + + dst_pudp = pud_offset(dst_p4dp, start); + src_pudp = pud_offset(src_p4dp, start); + + do { + pud_t pud = READ_ONCE(*src_pudp); + + next = pud_addr_end(start, end); + + if (pud_none(pud)) + continue; + + if (pud_leaf(pud)) { + set_pud(dst_pudp, __pud(pud_val(pud) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pmd(dst_pudp, src_pudp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pudp++, src_pudp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_map_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + p4d_t *dst_p4dp; + p4d_t *src_p4dp; + + if (pgd_none(READ_ONCE(*dst_pgdp))) { + dst_p4dp = (p4d_t *)get_safe_page(GFP_ATOMIC); + if (!dst_p4dp) + return -ENOMEM; + + pgd_populate(NULL, dst_pgdp, dst_p4dp); + } + + dst_p4dp = p4d_offset(dst_pgdp, start); + src_p4dp = p4d_offset(src_pgdp, start); + + do { + p4d_t p4d = READ_ONCE(*src_p4dp); + + next = p4d_addr_end(start, end); + + if (p4d_none(p4d)) + continue; + + if (p4d_leaf(p4d)) { + set_p4d(dst_p4dp, __p4d(p4d_val(p4d) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pud(dst_p4dp, src_p4dp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_p4dp++, src_p4dp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_mapping(pgd_t *pgdp, unsigned long start,unsigned long end,
pgprot_tprot)arch/riscv/kernel/hibernate.c new file +18
-0+{ + pgd_t *dst_pgdp = pgd_offset_pgd(pgdp, start); + pgd_t *src_pgdp = pgd_offset_k(start); + unsigned long next; + unsigned long ret; + + do { + pgd_t pgd = READ_ONCE(*src_pgdp); + + next = pgd_addr_end(start, end); + + if (pgd_none(pgd)) + continue; + + if (pgd_leaf(pgd)) { + set_pgd(dst_pgdp, __pgd(pgd_val(pgd) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_p4d(dst_pgdp, src_pgdp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pgdp++, src_pgdp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +27
-0+ + return 0; +} + +static unsigned long relocate_restore_code (void ) +{ + void *page = (void *)get_safe_page(GFP_ATOMIC); + + if (!page) + return -ENOMEM; + + copy_page(page, hibernate_core_restore_code); + + /* Make the page containing the relocated code executable. */ + set_memory_x((unsigned long)page, 1); + + return (unsigned long)page; +} + +int swsusp_arch_resume (void ) +{ + unsigned long end = (unsigned long)pfn_to_virt(max_low_pfn); + unsigned long start = PAGE_OFFSET; + int ret; + + /* +* Memory allocated by get_safe_page() will be dealt with by the hibernation core,
+ * we don't need to free it here.
+ */
+resume_pg_dir= (pgd_t*)get_safe_page(GFP_ATOMIC);
+ if (!resume_pg_dir)
+ return-ENOMEM;
+
+ /*
+ * Create a temporary page table and map the whole linear region as executable andQuoted 8 lines
Quoted 4 lines
+ * writable.
+ */
+ ret =temp_pgtable_mapping(resume_pg_dir, start, end, __pgprot(_PAGE_WRITE | _PAGE_EXEC));Hmm weird protections,
PAGE_PRESENTandPAGE_READare not even set. Shouldn't you usePAGE_KERNEL_EXECinstead?
Quoted 1 line
Forget this one, that's the protections you add, so that's good :)
OK.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
+ if (ret)
+ return ret;
+
+ /* Move the restore code to a new page so that it doesn't get overwritten by itself. */
+relocated_restore_code=relocate_restore_code();
+ if (relocated_restore_code==-ENOMEM)
+ return-ENOMEM;
+
+ /*
+ * Map the__hibernate_cpu_resume()address to the temporary page table so that the
+ * restore code can jumps to it after finished restore the image. The next execution
+ * code doesn't find itself in a different address space after switching over to the
+ * original page table used by the hibernated image. + *
+ */
+ start = (unsigned long)resume_hdr.restore_cpu_addr; + end = start +PAGE_SIZE;
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
+
+ ret =temp_pgtable_mapping(resume_pg_dir, start, end, __pgprot(_PAGE_WRITE | _PAGE_EXEC));
+ if (ret)
+ return ret;You set the linear mapping as executable so no need for this.
We need this mapping because the kernel addresses and the linear addresses in riscv-64bit are in different boundary. For example:
Linear address start with 0xff60000000000000
Kernel address start with 0xffffffff80000000
Quoted 39 lines
Quoted 39 lines
Quoted 26 lines
+
+ hibernate_restore_image(resume_hdr.saved_satp, (PFN_DOWN(__pa(resume_pg_dir)) |satp_mode), + resume_hdr.restore_cpu_addr);arch/riscv/kernel/hibernate.c new file +19
-0+ + return 0; +} + +#ifdef CONFIG_PM_SLEEP_SMP +int hibernate_resume_nonboot_cpu_disable (void ) +{ + if (sleep_cpu < 0) { + pr_err("Failing to resume from hibernate on an unknown CPU\n"); + return -ENODEV; + } + + return freeze_secondary_cpus(sleep_cpu); +} +#endif + +static int __init riscv_hibernate_init (void ) +{ + hibernate_cpu_context = kzalloc(sizeof(*hibernate_cpu_context),
GFP_KERNEL);arch/riscv/kernel/hibernate.c line 424
Quoted 8 lines
arch/riscv/kernel/hibernate.c new file +8
-0+ + if (WARN_ON(!hibernate_cpu_context)) + return -ENOMEM; + + return 0; +} + +early_initcall (riscv_hibernate_init);_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Alexandre Ghiti
reviewerRe: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-disk
Hi JeeHeng,
Quoted 687 lines · JeeHeng Sia
Quoted 655 lines
-----Original Message-----
From: Alexandre Ghiti <[email protected]>
Sent: Wednesday, March 8, 2023 5:28 PM
To: JeeHeng Sia <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>
Subject: Re: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-diskQuoted 635 lines · Alexandre Ghiti
Quoted 634 lines · Sia Jee Heng
Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid,
and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLESigned-off-by Sia Jee Heng <[email protected]>Reviewed-by Ley Foon Tan <[email protected]>Reviewed-by Mason Huo <[email protected]>--- arch/riscv/Kconfig | 6 + arch/riscv/include/asm/assembler.h | 20 ++ arch/riscv/include/asm/suspend.h | 19 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 424 +++++++++++++++++++++++++++++ 7 files changed, 552 insertions(+)create mode 100644
arch/riscv/kernel/hibernate-asm.Screate mode 100644arch/riscv/kernel/hibernate.carch/riscv/Kconfig +5
-0@@ -707,6 +707,12 @@ menu "Power management options" source "kernel/power/Kconfig" +config ARCH_HIBERNATION_POSSIBLE + def_bool y + +config ARCH_HIBERNATION_HEADER + def_bool HIBERNATION + endmenu # "Power management options" menu "CPU Power Management"arch/riscv/include/asm/assembler.h +19
-0@@ -59,4 +59,24 @@ REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0) .endm +/* +* copy_page - copy 1 page (4KB) of data from source to destination +* @a0 - destination +* @a1 - source +*/ + .macro copy_page a0, a1 + lui a2, 0x1 + add a2, a2, a0 +1 : + REG_L t0, 0(a1) + REG_L t1, SZREG(a1) + + REG_S t0, 0(a0) + REG_S t1, SZREG(a0) + + addi a0, a0, 2 * SZREG + addi a1, a1, 2 * SZREG + bne a2, a0, 1b + .endm + #endif /* __ASM_ASSEMBLER_H */arch/riscv/include/asm/suspend.h +4
-0@@ -21,6 +21,11 @@ struct suspend_context { #endif }; +/* +* Used by hibernation core and cleared during resume sequence +*/ +extern int in_suspend; + /* Low-level CPU suspend entry function */ int __cpu_suspend_enter(struct suspend_context *context); @@ -36,4 +41,18 @@ int __cpu_resume_enter(unsigned long hartid,unsigned long context);
/* Used to save and restore the CSRs */
voidsuspend_save_csrs(struct suspend_context *context); voidsuspend_restore_csrs(struct suspend_context *context);arch/riscv/include/asm/suspend.h +12
-0+ +/* Low-level API to support hibernation */ +int swsusp_arch_suspend (void ); +int swsusp_arch_resume (void ); +int arch_hibernation_header_save (void *addr,unsigned int max_size); +int arch_hibernation_header_restore (void *addr); +int __hibernate_cpu_resume (void ); + +/* Used to resume on the CPU we hibernated on */ +int hibernate_resume_nonboot_cpu_disable (void ); + +asmlinkage void hibernate_restore_image(unsigned long resume_satp,unsigned long
satp_temp,
+ unsigned longcpu_resume);arch/riscv/include/asm/suspend.h +1
-0+asmlinkage int hibernate_core_restore_code(void); #endifarch/riscv/kernel/Makefile +1
-0@@ -64,6 +64,7 @@ obj-$(CONFIG_MODULES) += module.o obj-$(CONFIG_MODULE_SECTIONS) += module-sections.o obj-$(CONFIG_CPU_PM) += suspend_entry.o suspend.o +obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_FUNCTION_TRACER) += mcount.o ftrace.o obj-$(CONFIG_DYNAMIC_FTRACE) += mcount-dyn.oarch/riscv/kernel/asm-offsets.c +4
-0@@ -9,6 +9,7 @@ #include <linux/kbuild.h> #include <linux/mm.h> #include <linux/sched.h> +#include <linux/suspend.h> #include <asm/kvm_host.h> #include <asm/thread_info.h> #include <asm/ptrace.h>@@ -116,6 +117,10 @@ void asm_offsets(void) OFFSET(SUSPEND_CONTEXT_REGS, suspend_context, regs); + OFFSET(HIBERN_PBE_ADDR, pbe, address); + OFFSET(HIBERN_PBE_ORIG, pbe, orig_address); + OFFSET(HIBERN_PBE_NEXT, pbe, next); + OFFSET(KVM_ARCH_GUEST_ZERO, kvm_vcpu_arch, guest_context.zero); OFFSET(KVM_ARCH_GUEST_RA, kvm_vcpu_arch, guest_context.ra); OFFSET(KVM_ARCH_GUEST_SP, kvm_vcpu_arch, guest_context.sp);arch/riscv/kernel/hibernate-asm.S new file +19
-0@@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Hibernation low level support for RISCV. + * + * Copyright (C) 2023 StarFive Technology Co., Ltd. + * + * Author: Jee Heng Sia <[email protected]> + */ + +#include <asm/asm.h> +#include <asm/asm-offsets.h> +#include <asm/assembler.h> +#include <asm/csr.h> + +#include <linux/linkage.h> + +/* + * int __hibernate_cpu_resume(void) + * Switch back to the hibernated image's page table prior torestoring the CPU
+ * context.
+ *
+ * Always returns 0
+ */arch/riscv/kernel/hibernate-asm.S new file +35
-0+ENTRY(__hibernate_cpu_resume) + /* switch to hibernated image's page table. */ + csrw CSR_SATP, s0 + sfence.vma + + REG_L a0, hibernate_cpu_context + + suspend_restore_csrs + suspend_restore_regs + + /* Return zero value. */ + mv a0, zero + + ret +END(__hibernate_cpu_resume) + +/* + * Prepare to restore the image. + * a0: satp of saved page tables. + * a1: satp of temporary page tables. + * a2: cpu_resume. + */ +ENTRY(hibernate_restore_image) + mv s0, a0 + mv s1, a1 + mv s2, a2 + REG_L s4, restore_pblist + REG_L a1, relocated_restore_code + + jalr a1 +END(hibernate_restore_image) + +/* + * The below code will be executed from a 'safe' page. + * It first switches to the temporary page table, then starts tocopy the pages
+ * back to the original memory location. Finally, it jumps to__hibernate_cpu_resume()
+ * to restore the CPU context.
+ */arch/riscv/kernel/hibernate-asm.S new file +16
-0+ENTRY(hibernate_core_restore_code) + /* switch to temp page table. */ + csrw satp, s1 + sfence.vma +.Lcopy: + /* The below code will restore the hibernated image. */ + REG_L a1, HIBERN_PBE_ADDR(s4) + REG_L a0, HIBERN_PBE_ORIG(s4) + + copy_page a0, a1 + + REG_L s4, HIBERN_PBE_NEXT(s4) + bnez s4, .Lcopy + + jalr s2 +END(hibernate_core_restore_code)arch/riscv/kernel/hibernate.c new file +28
-0@@ -0,0 +1,424 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Hibernation support for RISCV +* +* Copyright (C) 2023 StarFive Technology Co., Ltd. +* +* Author: Jee Heng Sia <[email protected]> +*/ + +#include <asm/barrier.h> +#include <asm/cacheflush.h> +#include <asm/mmu_context.h> +#include <asm/page.h> +#include <asm/pgalloc.h> +#include <asm/pgtable.h> +#include <asm/sections.h> +#include <asm/set_memory.h> +#include <asm/smp.h> +#include <asm/suspend.h> + +#include <linux/cpu.h> +#include <linux/memblock.h> +#include <linux/pm.h> +#include <linux/sched.h> +#include <linux/suspend.h> +#include <linux/utsname.h> + +/* The logical cpu number we should resume on, initialised to a non-cpu number. */
arch/riscv/kernel/hibernate.c new file +14
-0+static int sleep_cpu = -EINVAL ; + +/* Pointer to the temporary resume page table. */ +static pgd_t *resume_pg_dir; + +/* CPU context to be saved. */ +struct suspend_context *hibernate_cpu_context; +EXPORT_SYMBOL_GPL (hibernate_cpu_context); + +unsigned long relocated_restore_code; +EXPORT_SYMBOL_GPL (relocated_restore_code); + +/** +* struct arch_hibernate_hdr_invariants - container to store kernel build version.
+ * @uts_version: to save the build number and date so that the we do not resume with
+ * a different kernel.
+ */arch/riscv/kernel/hibernate.c new file +6
-0+struct arch_hibernate_hdr_invariants { + char uts_version[__NEW_UTS_LEN + 1]; +}; + +/** +* struct arch_hibernate_hdr - helper parameters that help us to restore the image.
+ * @invariants: container to store kernel build version.
+ * @hartid: to make sure sameboot_cpuexecutes the hibernate/restore code.
+ * @saved_satp: original page table used by the hibernated image.
+ * @restore_cpu_addr: the kernel's image address to restore the CPU context.
+ */arch/riscv/kernel/hibernate.c new file +8
-0+static struct arch_hibernate_hdr { + struct arch_hibernate_hdr_invariants invariants; + unsigned long hartid; + unsigned long saved_satp; + unsigned long restore_cpu_addr; +} resume_hdr; + +static void arch_hdr_invariants(struct arch_hibernate_hdr_invariants*i)
+{
+memset(i, 0, sizeof(*i));
+memcpy(i->uts_version, init_utsname()->version, sizeof(i->uts_version));arch/riscv/kernel/hibernate.c new file +93
-0+} + +/* +* Check if the given pfn is in the 'nosave' section. +*/ +int pfn_is_nosave (unsigned long pfn) +{ + unsigned long nosave_begin_pfn = sym_to_pfn(&__nosave_begin); + unsigned long nosave_end_pfn = sym_to_pfn(&__nosave_end - 1); + + return ((pfn >= nosave_begin_pfn) && (pfn <= nosave_end_pfn)); +} + +void notrace save_processor_state(void) +{ + WARN_ON(num_online_cpus() != 1); +} + +void notrace restore_processor_state(void) +{ +} + +/* +* Helper parameters need to be saved to the hibernation image header. +*/ +int arch_hibernation_header_save (void *addr,unsigned int max_size) +{ + struct arch_hibernate_hdr *hdr = addr; + + if (max_size < sizeof(*hdr)) + return -EOVERFLOW; + + arch_hdr_invariants(&hdr->invariants); + + hdr->hartid = cpuid_to_hartid_map(sleep_cpu); + hdr->saved_satp = csr_read(CSR_SATP); + hdr->restore_cpu_addr = (unsigned long)__hibernate_cpu_resume; + + return 0; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_save); + +/* +* Retrieve the helper parameters from the hibernation image header. +*/ +int arch_hibernation_header_restore (void *addr) +{ + struct arch_hibernate_hdr_invariants invariants; + struct arch_hibernate_hdr *hdr = addr; + int ret = 0; + + arch_hdr_invariants(&invariants); + + if (memcmp(&hdr->invariants, &invariants, sizeof(invariants))) { + pr_crit("Hibernate image not generated by this kernel!\n"); + return -EINVAL; + } + + sleep_cpu = riscv_hartid_to_cpuid(hdr->hartid); + if (sleep_cpu < 0) { + pr_crit("Hibernated on a CPU not known to this kernel!\n"); + sleep_cpu = -EINVAL; + return -EINVAL; + } + +#ifdef CONFIG_SMP + ret = bringup_hibernate_cpu(sleep_cpu); + if (ret) { + sleep_cpu = -EINVAL; + return ret; + } +#endif + resume_hdr = *hdr; + + return ret; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_restore); + +int swsusp_arch_suspend (void ) +{ + int ret = 0; + + if (__cpu_suspend_enter(hibernate_cpu_context)) { + sleep_cpu = smp_processor_id(); + suspend_save_csrs(hibernate_cpu_context); + ret = swsusp_save(); + } else { + suspend_restore_csrs(hibernate_cpu_context); + flush_tlb_all(); + flush_icache_all(); + + /* +* Tell the hibernation core that we've just restored the memory.
+ */
+in_suspend= 0;
+sleep_cpu=-EINVAL;
+ }arch/riscv/kernel/hibernate.c new file +5
-0+ + return ret; +} + +static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +26
-0+{ + pte_t *src_ptep; + pte_t *dst_ptep; + + if (pmd_none(READ_ONCE(*dst_pmdp))) { + dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC); + if (!dst_ptep) + return -ENOMEM; + + pmd_populate_kernel(NULL, dst_pmdp, dst_ptep); + } + + dst_ptep = pte_offset_kernel(dst_pmdp, start); + src_ptep = pte_offset_kernel(src_pmdp, start); + + do { + pte_t pte = READ_ONCE(*src_ptep); + + if (pte_present(pte)) + set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); + } while (dst_ptep++, src_ptep++, start += PAGE_SIZE, start < end); + + return 0; +} + +static int temp_pgtable_map_pmd(pud_t *dst_pudp, pud_t *src_pudp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + pmd_t *src_pmdp; + pmd_t *dst_pmdp; + + if (pud_none(READ_ONCE(*dst_pudp))) { + dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC); + if (!dst_pmdp) + return -ENOMEM; + + pud_populate(NULL, dst_pudp, dst_pmdp); + } + + dst_pmdp = pmd_offset(dst_pudp, start); + src_pmdp = pmd_offset(src_pudp, start); + + do { + pmd_t pmd = READ_ONCE(*src_pmdp); + + next = pmd_addr_end(start, end); + + if (pmd_none(pmd)) + continue; + + if (pmd_leaf(pmd)) { + set_pmd(dst_pmdp, __pmd(pmd_val(pmd) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pte(dst_pmdp, src_pmdp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pmdp++, src_pmdp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_map_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + pud_t *dst_pudp; + pud_t *src_pudp; + + if (p4d_none(READ_ONCE(*dst_p4dp))) { + dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC); + if (!dst_pudp) + return -ENOMEM; + + p4d_populate(NULL, dst_p4dp, dst_pudp); + } + + dst_pudp = pud_offset(dst_p4dp, start); + src_pudp = pud_offset(src_p4dp, start); + + do { + pud_t pud = READ_ONCE(*src_pudp); + + next = pud_addr_end(start, end); + + if (pud_none(pud)) + continue; + + if (pud_leaf(pud)) { + set_pud(dst_pudp, __pud(pud_val(pud) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pmd(dst_pudp, src_pudp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pudp++, src_pudp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_map_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + p4d_t *dst_p4dp; + p4d_t *src_p4dp; + + if (pgd_none(READ_ONCE(*dst_pgdp))) { + dst_p4dp = (p4d_t *)get_safe_page(GFP_ATOMIC); + if (!dst_p4dp) + return -ENOMEM; + + pgd_populate(NULL, dst_pgdp, dst_p4dp); + } + + dst_p4dp = p4d_offset(dst_pgdp, start); + src_p4dp = p4d_offset(src_pgdp, start); + + do { + p4d_t p4d = READ_ONCE(*src_p4dp); + + next = p4d_addr_end(start, end); + + if (p4d_none(p4d)) + continue; + + if (p4d_leaf(p4d)) { + set_p4d(dst_p4dp, __p4d(p4d_val(p4d) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pud(dst_p4dp, src_p4dp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_p4dp++, src_p4dp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_mapping(pgd_t *pgdp, unsigned long start,unsigned long end,
pgprot_tprot)arch/riscv/kernel/hibernate.c new file +18
-0+{ + pgd_t *dst_pgdp = pgd_offset_pgd(pgdp, start); + pgd_t *src_pgdp = pgd_offset_k(start); + unsigned long next; + unsigned long ret; + + do { + pgd_t pgd = READ_ONCE(*src_pgdp); + + next = pgd_addr_end(start, end); + + if (pgd_none(pgd)) + continue; + + if (pgd_leaf(pgd)) { + set_pgd(dst_pgdp, __pgd(pgd_val(pgd) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_p4d(dst_pgdp, src_pgdp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pgdp++, src_pgdp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +27
-0+ + return 0; +} + +static unsigned long relocate_restore_code (void ) +{ + void *page = (void *)get_safe_page(GFP_ATOMIC); + + if (!page) + return -ENOMEM; + + copy_page(page, hibernate_core_restore_code); + + /* Make the page containing the relocated code executable. */ + set_memory_x((unsigned long)page, 1); + + return (unsigned long)page; +} + +int swsusp_arch_resume (void ) +{ + unsigned long end = (unsigned long)pfn_to_virt(max_low_pfn); + unsigned long start = PAGE_OFFSET; + int ret; + + /* +* Memory allocated by get_safe_page() will be dealt with by the hibernation core,
+ * we don't need to free it here.
+ */
+resume_pg_dir= (pgd_t*)get_safe_page(GFP_ATOMIC);
+ if (!resume_pg_dir)
+ return-ENOMEM;
+
+ /*
+ * Create a temporary page table and map the whole linear regionQuoted 8 lines
Quoted 5 lines
as executable and
+ * writable.
+ */
+ ret =temp_pgtable_mapping(resume_pg_dir, start, end, __pgprot(_PAGE_WRITE | _PAGE_EXEC));Hmm weird protections,
PAGE_PRESENTandPAGE_READare not even set. Shouldn't you usePAGE_KERNEL_EXECinstead?Quoted 1 line
Forget this one, that's the protections you add, so that's good :)
OK.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
+ if (ret)
+ return ret;
+
+ /* Move the restore code to a new page so that it doesn't get overwritten by itself. */
+relocated_restore_code=relocate_restore_code();
+ if (relocated_restore_code==-ENOMEM)
+ return-ENOMEM;
+
+ /*
+ * Map the__hibernate_cpu_resume()address to the temporary page table so that the
+ * restore code can jumps to it after finished restore the image. The next execution
+ * code doesn't find itself in a different address space after switching over to the
+ * original page table used by the hibernated image. + *
+ */
+ start = (unsigned long)resume_hdr.restore_cpu_addr;Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
+ end = start +
PAGE_SIZE;
+
+ ret =temp_pgtable_mapping(resume_pg_dir, start, end, __pgprot(_PAGE_WRITE | _PAGE_EXEC));
+ if (ret)
+ return ret;You set the linear mapping as executable so no need for this.
Quoted 3 lines
We need this mapping because the kernel addresses and the linear addresses in riscv-64bit are in different boundary. For example:
Linear address start with0xff60000000000000
Kernel address start with0xffffffff80000000
Ok I understand now that you need to map this into the temporary page table: one nit though, since you copy the kernel mapping, no need to use PAGE_WRITE | PAGE_EXEC, the protections are already correct.
Thanks,
Alex
Quoted 41 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
+
+ hibernate_restore_image(resume_hdr.saved_satp, (PFN_DOWN(__pa(resume_pg_dir)) |satp_mode), + resume_hdr.restore_cpu_addr);arch/riscv/kernel/hibernate.c new file +19
-0+ + return 0; +} + +#ifdef CONFIG_PM_SLEEP_SMP +int hibernate_resume_nonboot_cpu_disable (void ) +{ + if (sleep_cpu < 0) { + pr_err("Failing to resume from hibernate on an unknown CPU\n"); + return -ENODEV; + } + + return freeze_secondary_cpus(sleep_cpu); +} +#endif + +static int __init riscv_hibernate_init (void ) +{ + hibernate_cpu_context = kzalloc(sizeof(*hibernate_cpu_context),
GFP_KERNEL);arch/riscv/kernel/hibernate.c new file +4
-0+ + if (WARN_ON(!hibernate_cpu_context)) + return -ENOMEM; +Quoted 8 lines
Quoted 8 lines
arch/riscv/kernel/hibernate.c line 424
Quoted 4 lines
+ return 0;
arch/riscv/kernel/hibernate.c new file +3
-0+} + +early_initcall (riscv_hibernate_init);_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
RE: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-disk
Hi Alex,
Quoted 698 lines
-----Original Message-----
From: Alexandre Ghiti <[email protected]>
Sent: Friday, March 10, 2023 5:59 PM
To: JeeHeng Sia <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>
Subject: Re: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-diskHi JeeHeng,
Quoted 687 lines · JeeHeng Sia
Quoted 655 lines
-----Original Message-----
From: Alexandre Ghiti <[email protected]>
Sent: Wednesday, March 8, 2023 5:28 PM
To: JeeHeng Sia <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>
Subject: Re: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-diskQuoted 635 lines · Alexandre Ghiti
Quoted 634 lines · Sia Jee Heng
Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid,
and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLESigned-off-by Sia Jee Heng <[email protected]>Reviewed-by Ley Foon Tan <[email protected]>Reviewed-by Mason Huo <[email protected]>--- arch/riscv/Kconfig | 6 + arch/riscv/include/asm/assembler.h | 20 ++ arch/riscv/include/asm/suspend.h | 19 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 424 +++++++++++++++++++++++++++++ 7 files changed, 552 insertions(+)create mode 100644
arch/riscv/kernel/hibernate-asm.Screate mode 100644arch/riscv/kernel/hibernate.carch/riscv/Kconfig +5
-0@@ -707,6 +707,12 @@ menu "Power management options" source "kernel/power/Kconfig" +config ARCH_HIBERNATION_POSSIBLE + def_bool y + +config ARCH_HIBERNATION_HEADER + def_bool HIBERNATION + endmenu # "Power management options" menu "CPU Power Management"arch/riscv/include/asm/assembler.h +19
-0@@ -59,4 +59,24 @@ REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0) .endm +/* +* copy_page - copy 1 page (4KB) of data from source to destination +* @a0 - destination +* @a1 - source +*/ + .macro copy_page a0, a1 + lui a2, 0x1 + add a2, a2, a0 +1 : + REG_L t0, 0(a1) + REG_L t1, SZREG(a1) + + REG_S t0, 0(a0) + REG_S t1, SZREG(a0) + + addi a0, a0, 2 * SZREG + addi a1, a1, 2 * SZREG + bne a2, a0, 1b + .endm + #endif /* __ASM_ASSEMBLER_H */arch/riscv/include/asm/suspend.h +4
-0@@ -21,6 +21,11 @@ struct suspend_context { #endif }; +/* +* Used by hibernation core and cleared during resume sequence +*/ +extern int in_suspend; + /* Low-level CPU suspend entry function */ int __cpu_suspend_enter(struct suspend_context *context); @@ -36,4 +41,18 @@ int __cpu_resume_enter(unsigned long hartid,unsigned long context);
/* Used to save and restore the CSRs */
voidsuspend_save_csrs(struct suspend_context *context); voidsuspend_restore_csrs(struct suspend_context *context);arch/riscv/include/asm/suspend.h +12
-0+ +/* Low-level API to support hibernation */ +int swsusp_arch_suspend (void ); +int swsusp_arch_resume (void ); +int arch_hibernation_header_save (void *addr,unsigned int max_size); +int arch_hibernation_header_restore (void *addr); +int __hibernate_cpu_resume (void ); + +/* Used to resume on the CPU we hibernated on */ +int hibernate_resume_nonboot_cpu_disable (void ); + +asmlinkage void hibernate_restore_image(unsigned long resume_satp,unsigned long
satp_temp,
+ unsigned longcpu_resume);arch/riscv/include/asm/suspend.h +1
-0+asmlinkage int hibernate_core_restore_code(void); #endifarch/riscv/kernel/Makefile +1
-0@@ -64,6 +64,7 @@ obj-$(CONFIG_MODULES) += module.o obj-$(CONFIG_MODULE_SECTIONS) += module-sections.o obj-$(CONFIG_CPU_PM) += suspend_entry.o suspend.o +obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_FUNCTION_TRACER) += mcount.o ftrace.o obj-$(CONFIG_DYNAMIC_FTRACE) += mcount-dyn.oarch/riscv/kernel/asm-offsets.c +4
-0@@ -9,6 +9,7 @@ #include <linux/kbuild.h> #include <linux/mm.h> #include <linux/sched.h> +#include <linux/suspend.h> #include <asm/kvm_host.h> #include <asm/thread_info.h> #include <asm/ptrace.h>@@ -116,6 +117,10 @@ void asm_offsets(void) OFFSET(SUSPEND_CONTEXT_REGS, suspend_context, regs); + OFFSET(HIBERN_PBE_ADDR, pbe, address); + OFFSET(HIBERN_PBE_ORIG, pbe, orig_address); + OFFSET(HIBERN_PBE_NEXT, pbe, next); + OFFSET(KVM_ARCH_GUEST_ZERO, kvm_vcpu_arch, guest_context.zero); OFFSET(KVM_ARCH_GUEST_RA, kvm_vcpu_arch, guest_context.ra); OFFSET(KVM_ARCH_GUEST_SP, kvm_vcpu_arch, guest_context.sp);arch/riscv/kernel/hibernate-asm.S new file +19
-0@@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Hibernation low level support for RISCV. + * + * Copyright (C) 2023 StarFive Technology Co., Ltd. + * + * Author: Jee Heng Sia <[email protected]> + */ + +#include <asm/asm.h> +#include <asm/asm-offsets.h> +#include <asm/assembler.h> +#include <asm/csr.h> + +#include <linux/linkage.h> + +/* + * int __hibernate_cpu_resume(void) + * Switch back to the hibernated image's page table prior torestoring the CPU
+ * context.
+ *
+ * Always returns 0
+ */arch/riscv/kernel/hibernate-asm.S new file +35
-0+ENTRY(__hibernate_cpu_resume) + /* switch to hibernated image's page table. */ + csrw CSR_SATP, s0 + sfence.vma + + REG_L a0, hibernate_cpu_context + + suspend_restore_csrs + suspend_restore_regs + + /* Return zero value. */ + mv a0, zero + + ret +END(__hibernate_cpu_resume) + +/* + * Prepare to restore the image. + * a0: satp of saved page tables. + * a1: satp of temporary page tables. + * a2: cpu_resume. + */ +ENTRY(hibernate_restore_image) + mv s0, a0 + mv s1, a1 + mv s2, a2 + REG_L s4, restore_pblist + REG_L a1, relocated_restore_code + + jalr a1 +END(hibernate_restore_image) + +/* + * The below code will be executed from a 'safe' page. + * It first switches to the temporary page table, then starts tocopy the pages
+ * back to the original memory location. Finally, it jumps to__hibernate_cpu_resume()
+ * to restore the CPU context.
+ */arch/riscv/kernel/hibernate-asm.S new file +16
-0+ENTRY(hibernate_core_restore_code) + /* switch to temp page table. */ + csrw satp, s1 + sfence.vma +.Lcopy: + /* The below code will restore the hibernated image. */ + REG_L a1, HIBERN_PBE_ADDR(s4) + REG_L a0, HIBERN_PBE_ORIG(s4) + + copy_page a0, a1 + + REG_L s4, HIBERN_PBE_NEXT(s4) + bnez s4, .Lcopy + + jalr s2 +END(hibernate_core_restore_code)arch/riscv/kernel/hibernate.c new file +28
-0@@ -0,0 +1,424 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Hibernation support for RISCV +* +* Copyright (C) 2023 StarFive Technology Co., Ltd. +* +* Author: Jee Heng Sia <[email protected]> +*/ + +#include <asm/barrier.h> +#include <asm/cacheflush.h> +#include <asm/mmu_context.h> +#include <asm/page.h> +#include <asm/pgalloc.h> +#include <asm/pgtable.h> +#include <asm/sections.h> +#include <asm/set_memory.h> +#include <asm/smp.h> +#include <asm/suspend.h> + +#include <linux/cpu.h> +#include <linux/memblock.h> +#include <linux/pm.h> +#include <linux/sched.h> +#include <linux/suspend.h> +#include <linux/utsname.h> + +/* The logical cpu number we should resume on, initialised to a non-cpu number. */
arch/riscv/kernel/hibernate.c new file +14
-0+static int sleep_cpu = -EINVAL ; + +/* Pointer to the temporary resume page table. */ +static pgd_t *resume_pg_dir; + +/* CPU context to be saved. */ +struct suspend_context *hibernate_cpu_context; +EXPORT_SYMBOL_GPL (hibernate_cpu_context); + +unsigned long relocated_restore_code; +EXPORT_SYMBOL_GPL (relocated_restore_code); + +/** +* struct arch_hibernate_hdr_invariants - container to store kernel build version.
+ * @uts_version: to save the build number and date so that the we do not resume with
+ * a different kernel.
+ */arch/riscv/kernel/hibernate.c new file +6
-0+struct arch_hibernate_hdr_invariants { + char uts_version[__NEW_UTS_LEN + 1]; +}; + +/** +* struct arch_hibernate_hdr - helper parameters that help us to restore the image.
+ * @invariants: container to store kernel build version.
+ * @hartid: to make sure sameboot_cpuexecutes the hibernate/restore code.
+ * @saved_satp: original page table used by the hibernated image.
+ * @restore_cpu_addr: the kernel's image address to restore the CPU context.
+ */arch/riscv/kernel/hibernate.c new file +8
-0+static struct arch_hibernate_hdr { + struct arch_hibernate_hdr_invariants invariants; + unsigned long hartid; + unsigned long saved_satp; + unsigned long restore_cpu_addr; +} resume_hdr; + +static void arch_hdr_invariants(struct arch_hibernate_hdr_invariants*i)
+{
+memset(i, 0, sizeof(*i));
+memcpy(i->uts_version, init_utsname()->version, sizeof(i->uts_version));arch/riscv/kernel/hibernate.c new file +93
-0+} + +/* +* Check if the given pfn is in the 'nosave' section. +*/ +int pfn_is_nosave (unsigned long pfn) +{ + unsigned long nosave_begin_pfn = sym_to_pfn(&__nosave_begin); + unsigned long nosave_end_pfn = sym_to_pfn(&__nosave_end - 1); + + return ((pfn >= nosave_begin_pfn) && (pfn <= nosave_end_pfn)); +} + +void notrace save_processor_state(void) +{ + WARN_ON(num_online_cpus() != 1); +} + +void notrace restore_processor_state(void) +{ +} + +/* +* Helper parameters need to be saved to the hibernation image header. +*/ +int arch_hibernation_header_save (void *addr,unsigned int max_size) +{ + struct arch_hibernate_hdr *hdr = addr; + + if (max_size < sizeof(*hdr)) + return -EOVERFLOW; + + arch_hdr_invariants(&hdr->invariants); + + hdr->hartid = cpuid_to_hartid_map(sleep_cpu); + hdr->saved_satp = csr_read(CSR_SATP); + hdr->restore_cpu_addr = (unsigned long)__hibernate_cpu_resume; + + return 0; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_save); + +/* +* Retrieve the helper parameters from the hibernation image header. +*/ +int arch_hibernation_header_restore (void *addr) +{ + struct arch_hibernate_hdr_invariants invariants; + struct arch_hibernate_hdr *hdr = addr; + int ret = 0; + + arch_hdr_invariants(&invariants); + + if (memcmp(&hdr->invariants, &invariants, sizeof(invariants))) { + pr_crit("Hibernate image not generated by this kernel!\n"); + return -EINVAL; + } + + sleep_cpu = riscv_hartid_to_cpuid(hdr->hartid); + if (sleep_cpu < 0) { + pr_crit("Hibernated on a CPU not known to this kernel!\n"); + sleep_cpu = -EINVAL; + return -EINVAL; + } + +#ifdef CONFIG_SMP + ret = bringup_hibernate_cpu(sleep_cpu); + if (ret) { + sleep_cpu = -EINVAL; + return ret; + } +#endif + resume_hdr = *hdr; + + return ret; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_restore); + +int swsusp_arch_suspend (void ) +{ + int ret = 0; + + if (__cpu_suspend_enter(hibernate_cpu_context)) { + sleep_cpu = smp_processor_id(); + suspend_save_csrs(hibernate_cpu_context); + ret = swsusp_save(); + } else { + suspend_restore_csrs(hibernate_cpu_context); + flush_tlb_all(); + flush_icache_all(); + + /* +* Tell the hibernation core that we've just restored the memory.
+ */
+in_suspend= 0;
+sleep_cpu=-EINVAL;
+ }arch/riscv/kernel/hibernate.c new file +5
-0+ + return ret; +} + +static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +26
-0+{ + pte_t *src_ptep; + pte_t *dst_ptep; + + if (pmd_none(READ_ONCE(*dst_pmdp))) { + dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC); + if (!dst_ptep) + return -ENOMEM; + + pmd_populate_kernel(NULL, dst_pmdp, dst_ptep); + } + + dst_ptep = pte_offset_kernel(dst_pmdp, start); + src_ptep = pte_offset_kernel(src_pmdp, start); + + do { + pte_t pte = READ_ONCE(*src_ptep); + + if (pte_present(pte)) + set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); + } while (dst_ptep++, src_ptep++, start += PAGE_SIZE, start < end); + + return 0; +} + +static int temp_pgtable_map_pmd(pud_t *dst_pudp, pud_t *src_pudp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + pmd_t *src_pmdp; + pmd_t *dst_pmdp; + + if (pud_none(READ_ONCE(*dst_pudp))) { + dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC); + if (!dst_pmdp) + return -ENOMEM; + + pud_populate(NULL, dst_pudp, dst_pmdp); + } + + dst_pmdp = pmd_offset(dst_pudp, start); + src_pmdp = pmd_offset(src_pudp, start); + + do { + pmd_t pmd = READ_ONCE(*src_pmdp); + + next = pmd_addr_end(start, end); + + if (pmd_none(pmd)) + continue; + + if (pmd_leaf(pmd)) { + set_pmd(dst_pmdp, __pmd(pmd_val(pmd) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pte(dst_pmdp, src_pmdp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pmdp++, src_pmdp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_map_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + pud_t *dst_pudp; + pud_t *src_pudp; + + if (p4d_none(READ_ONCE(*dst_p4dp))) { + dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC); + if (!dst_pudp) + return -ENOMEM; + + p4d_populate(NULL, dst_p4dp, dst_pudp); + } + + dst_pudp = pud_offset(dst_p4dp, start); + src_pudp = pud_offset(src_p4dp, start); + + do { + pud_t pud = READ_ONCE(*src_pudp); + + next = pud_addr_end(start, end); + + if (pud_none(pud)) + continue; + + if (pud_leaf(pud)) { + set_pud(dst_pudp, __pud(pud_val(pud) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pmd(dst_pudp, src_pudp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pudp++, src_pudp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_map_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + p4d_t *dst_p4dp; + p4d_t *src_p4dp; + + if (pgd_none(READ_ONCE(*dst_pgdp))) { + dst_p4dp = (p4d_t *)get_safe_page(GFP_ATOMIC); + if (!dst_p4dp) + return -ENOMEM; + + pgd_populate(NULL, dst_pgdp, dst_p4dp); + } + + dst_p4dp = p4d_offset(dst_pgdp, start); + src_p4dp = p4d_offset(src_pgdp, start); + + do { + p4d_t p4d = READ_ONCE(*src_p4dp); + + next = p4d_addr_end(start, end); + + if (p4d_none(p4d)) + continue; + + if (p4d_leaf(p4d)) { + set_p4d(dst_p4dp, __p4d(p4d_val(p4d) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pud(dst_p4dp, src_p4dp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_p4dp++, src_p4dp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_mapping(pgd_t *pgdp, unsigned long start,unsigned long end,
pgprot_tprot)arch/riscv/kernel/hibernate.c new file +18
-0+{ + pgd_t *dst_pgdp = pgd_offset_pgd(pgdp, start); + pgd_t *src_pgdp = pgd_offset_k(start); + unsigned long next; + unsigned long ret; + + do { + pgd_t pgd = READ_ONCE(*src_pgdp); + + next = pgd_addr_end(start, end); + + if (pgd_none(pgd)) + continue; + + if (pgd_leaf(pgd)) { + set_pgd(dst_pgdp, __pgd(pgd_val(pgd) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_p4d(dst_pgdp, src_pgdp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pgdp++, src_pgdp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +27
-0+ + return 0; +} + +static unsigned long relocate_restore_code (void ) +{ + void *page = (void *)get_safe_page(GFP_ATOMIC); + + if (!page) + return -ENOMEM; + + copy_page(page, hibernate_core_restore_code); + + /* Make the page containing the relocated code executable. */ + set_memory_x((unsigned long)page, 1); + + return (unsigned long)page; +} + +int swsusp_arch_resume (void ) +{ + unsigned long end = (unsigned long)pfn_to_virt(max_low_pfn); + unsigned long start = PAGE_OFFSET; + int ret; + + /* +* Memory allocated by get_safe_page() will be dealt with by the hibernation core,
+ * we don't need to free it here.
+ */
+resume_pg_dir= (pgd_t*)get_safe_page(GFP_ATOMIC);
+ if (!resume_pg_dir)
+ return-ENOMEM;
+
+ /*
+ * Create a temporary page table and map the whole linear regionQuoted 8 lines
Quoted 5 lines
as executable and
+ * writable.
+ */
+ ret =temp_pgtable_mapping(resume_pg_dir, start, end, __pgprot(_PAGE_WRITE | _PAGE_EXEC));Hmm weird protections,
PAGE_PRESENTandPAGE_READare not even set. Shouldn't you usePAGE_KERNEL_EXECinstead?Quoted 1 line
Forget this one, that's the protections you add, so that's good :)
OK.
Quoted 30 lines
Quoted 30 lines
Quoted 20 lines
+ if (ret)
+ return ret;
+
+ /* Move the restore code to a new page so that it doesn't get overwritten by itself. */
+relocated_restore_code=relocate_restore_code();
+ if (relocated_restore_code==-ENOMEM)
+ return-ENOMEM;
+
+ /*
+ * Map the__hibernate_cpu_resume()address to the temporary page table so that the
+ * restore code can jumps to it after finished restore the image. The next execution
+ * code doesn't find itself in a different address space after switching over to the
+ * original page table used by the hibernated image. + *Quoted 8 lines
+ */
+ start = (unsigned long)resume_hdr.restore_cpu_addr; + end = start +PAGE_SIZE;
+
+ ret =temp_pgtable_mapping(resume_pg_dir, start, end, __pgprot(_PAGE_WRITE | _PAGE_EXEC));
+ if (ret)
+ return ret;You set the linear mapping as executable so no need for this.
Quoted 8 lines
Quoted 3 lines
We need this mapping because the kernel addresses and the linear addresses in riscv-64bit are in different boundary. For example:
Linear address start with0xff60000000000000
Kernel address start with0xffffffff80000000Ok I understand now that you need to map this into the temporary page
table: one nit though, since you copy the kernel mapping, no need to usePAGE_WRITE|PAGE_EXEC, the protections are already correct.
Here is the thing. For riscv_64bit, I am agreed with you that we don’t need to use PAGE_WRITE | PAGE_EXEC.
However, we need to use PAGE_WRITE | PAGE_EXEC for riscv_32bit because the linear addresses and kernel addresses are similar. The attribute becomes not writable if following the "original" page table. I am thinking to use #ifdef config_32bit to split the path for 64bit and 32bit but I just afraid the linear addresses and kernel addresses might change again in the future. To be on the safe side, I let both riscv_32bit & riscv_64bit to use the PAGE_WRITE | PAGE_EXEC. For example:
Linear address and kernel address for 32bit started with 0xc0000000
Linear address for 64bit start with 0xff60000000000000
Kernel address for 64bit start with 0xffffffff80000000
Please let me know your thoughts.
Quoted 47 lines
Thanks,
Alex
Quoted 41 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
+
+ hibernate_restore_image(resume_hdr.saved_satp, (PFN_DOWN(__pa(resume_pg_dir)) |satp_mode), + resume_hdr.restore_cpu_addr);arch/riscv/kernel/hibernate.c new file +19
-0+ + return 0; +} + +#ifdef CONFIG_PM_SLEEP_SMP +int hibernate_resume_nonboot_cpu_disable (void ) +{ + if (sleep_cpu < 0) { + pr_err("Failing to resume from hibernate on an unknown CPU\n"); + return -ENODEV; + } + + return freeze_secondary_cpus(sleep_cpu); +} +#endif + +static int __init riscv_hibernate_init (void ) +{ + hibernate_cpu_context = kzalloc(sizeof(*hibernate_cpu_context),
GFP_KERNEL);arch/riscv/kernel/hibernate.c new file +4
-0+ + if (WARN_ON(!hibernate_cpu_context)) + return -ENOMEM; +Quoted 8 lines
Quoted 8 lines
arch/riscv/kernel/hibernate.c line 424
Quoted 4 lines
+ return 0;
arch/riscv/kernel/hibernate.c new file +3
-0+} + +early_initcall (riscv_hibernate_init);_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Alexandre Ghiti
reviewerRe: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-disk
Quoted 698 lines · JeeHeng Sia
Hi Alex,
Quoted 696 lines
-----Original Message-----
From: Alexandre Ghiti <[email protected]>
Sent: Friday, March 10, 2023 5:59 PM
To: JeeHeng Sia <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>
Subject: Re: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-diskHi JeeHeng,
Quoted 684 lines · JeeHeng Sia
Quoted 652 lines
-----Original Message-----
From: Alexandre Ghiti <[email protected]>
Sent: Wednesday, March 8, 2023 5:28 PM
To: JeeHeng Sia <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>
Subject: Re: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-diskQuoted 634 lines · Alexandre Ghiti
Quoted 633 lines · Sia Jee Heng
Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid,
and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLESigned-off-by Sia Jee Heng <[email protected]>Reviewed-by Ley Foon Tan <[email protected]>Reviewed-by Mason Huo <[email protected]>--- arch/riscv/Kconfig | 6 + arch/riscv/include/asm/assembler.h | 20 ++ arch/riscv/include/asm/suspend.h | 19 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 424 +++++++++++++++++++++++++++++ 7 files changed, 552 insertions(+)create mode 100644
arch/riscv/kernel/hibernate-asm.Screate mode 100644arch/riscv/kernel/hibernate.carch/riscv/Kconfig +5
-0@@ -707,6 +707,12 @@ menu "Power management options" source "kernel/power/Kconfig" +config ARCH_HIBERNATION_POSSIBLE + def_bool y + +config ARCH_HIBERNATION_HEADER + def_bool HIBERNATION + endmenu # "Power management options" menu "CPU Power Management"arch/riscv/include/asm/assembler.h +19
-0@@ -59,4 +59,24 @@ REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0) .endm +/* +* copy_page - copy 1 page (4KB) of data from source to destination +* @a0 - destination +* @a1 - source +*/ + .macro copy_page a0, a1 + lui a2, 0x1 + add a2, a2, a0 +1 : + REG_L t0, 0(a1) + REG_L t1, SZREG(a1) + + REG_S t0, 0(a0) + REG_S t1, SZREG(a0) + + addi a0, a0, 2 * SZREG + addi a1, a1, 2 * SZREG + bne a2, a0, 1b + .endm + #endif /* __ASM_ASSEMBLER_H */arch/riscv/include/asm/suspend.h +4
-0@@ -21,6 +21,11 @@ struct suspend_context { #endif }; +/* +* Used by hibernation core and cleared during resume sequence +*/ +extern int in_suspend; + /* Low-level CPU suspend entry function */ int __cpu_suspend_enter(struct suspend_context *context); @@ -36,4 +41,18 @@ int __cpu_resume_enter(unsigned long hartid,unsigned long context);
/* Used to save and restore the CSRs */
voidsuspend_save_csrs(struct suspend_context *context); voidsuspend_restore_csrs(struct suspend_context *context);arch/riscv/include/asm/suspend.h +12
-0+ +/* Low-level API to support hibernation */ +int swsusp_arch_suspend (void ); +int swsusp_arch_resume (void ); +int arch_hibernation_header_save (void *addr,unsigned int max_size); +int arch_hibernation_header_restore (void *addr); +int __hibernate_cpu_resume (void ); + +/* Used to resume on the CPU we hibernated on */ +int hibernate_resume_nonboot_cpu_disable (void ); + +asmlinkage void hibernate_restore_image(unsigned long resume_satp,unsigned long
satp_temp,
+ unsigned longcpu_resume);arch/riscv/include/asm/suspend.h +1
-0+asmlinkage int hibernate_core_restore_code(void); #endifarch/riscv/kernel/Makefile +1
-0@@ -64,6 +64,7 @@ obj-$(CONFIG_MODULES) += module.o obj-$(CONFIG_MODULE_SECTIONS) += module-sections.o obj-$(CONFIG_CPU_PM) += suspend_entry.o suspend.o +obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_FUNCTION_TRACER) += mcount.o ftrace.o obj-$(CONFIG_DYNAMIC_FTRACE) += mcount-dyn.oarch/riscv/kernel/asm-offsets.c +4
-0@@ -9,6 +9,7 @@ #include <linux/kbuild.h> #include <linux/mm.h> #include <linux/sched.h> +#include <linux/suspend.h> #include <asm/kvm_host.h> #include <asm/thread_info.h> #include <asm/ptrace.h>@@ -116,6 +117,10 @@ void asm_offsets(void) OFFSET(SUSPEND_CONTEXT_REGS, suspend_context, regs); + OFFSET(HIBERN_PBE_ADDR, pbe, address); + OFFSET(HIBERN_PBE_ORIG, pbe, orig_address); + OFFSET(HIBERN_PBE_NEXT, pbe, next); + OFFSET(KVM_ARCH_GUEST_ZERO, kvm_vcpu_arch, guest_context.zero); OFFSET(KVM_ARCH_GUEST_RA, kvm_vcpu_arch, guest_context.ra); OFFSET(KVM_ARCH_GUEST_SP, kvm_vcpu_arch, guest_context.sp);arch/riscv/kernel/hibernate-asm.S new file +19
-0@@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Hibernation low level support for RISCV. + * + * Copyright (C) 2023 StarFive Technology Co., Ltd. + * + * Author: Jee Heng Sia <[email protected]> + */ + +#include <asm/asm.h> +#include <asm/asm-offsets.h> +#include <asm/assembler.h> +#include <asm/csr.h> + +#include <linux/linkage.h> + +/* + * int __hibernate_cpu_resume(void) + * Switch back to the hibernated image's page table prior torestoring the CPU
+ * context.
+ *
+ * Always returns 0
+ */arch/riscv/kernel/hibernate-asm.S new file +35
-0+ENTRY(__hibernate_cpu_resume) + /* switch to hibernated image's page table. */ + csrw CSR_SATP, s0 + sfence.vma + + REG_L a0, hibernate_cpu_context + + suspend_restore_csrs + suspend_restore_regs + + /* Return zero value. */ + mv a0, zero + + ret +END(__hibernate_cpu_resume) + +/* + * Prepare to restore the image. + * a0: satp of saved page tables. + * a1: satp of temporary page tables. + * a2: cpu_resume. + */ +ENTRY(hibernate_restore_image) + mv s0, a0 + mv s1, a1 + mv s2, a2 + REG_L s4, restore_pblist + REG_L a1, relocated_restore_code + + jalr a1 +END(hibernate_restore_image) + +/* + * The below code will be executed from a 'safe' page. + * It first switches to the temporary page table, then starts tocopy the pages
+ * back to the original memory location. Finally, it jumps to__hibernate_cpu_resume()
+ * to restore the CPU context.
+ */arch/riscv/kernel/hibernate-asm.S new file +16
-0+ENTRY(hibernate_core_restore_code) + /* switch to temp page table. */ + csrw satp, s1 + sfence.vma +.Lcopy: + /* The below code will restore the hibernated image. */ + REG_L a1, HIBERN_PBE_ADDR(s4) + REG_L a0, HIBERN_PBE_ORIG(s4) + + copy_page a0, a1 + + REG_L s4, HIBERN_PBE_NEXT(s4) + bnez s4, .Lcopy + + jalr s2 +END(hibernate_core_restore_code)arch/riscv/kernel/hibernate.c new file +28
-0@@ -0,0 +1,424 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Hibernation support for RISCV +* +* Copyright (C) 2023 StarFive Technology Co., Ltd. +* +* Author: Jee Heng Sia <[email protected]> +*/ + +#include <asm/barrier.h> +#include <asm/cacheflush.h> +#include <asm/mmu_context.h> +#include <asm/page.h> +#include <asm/pgalloc.h> +#include <asm/pgtable.h> +#include <asm/sections.h> +#include <asm/set_memory.h> +#include <asm/smp.h> +#include <asm/suspend.h> + +#include <linux/cpu.h> +#include <linux/memblock.h> +#include <linux/pm.h> +#include <linux/sched.h> +#include <linux/suspend.h> +#include <linux/utsname.h> + +/* The logical cpu number we should resume on, initialised to a non-cpu number. */
arch/riscv/kernel/hibernate.c new file +14
-0+static int sleep_cpu = -EINVAL ; + +/* Pointer to the temporary resume page table. */ +static pgd_t *resume_pg_dir; + +/* CPU context to be saved. */ +struct suspend_context *hibernate_cpu_context; +EXPORT_SYMBOL_GPL (hibernate_cpu_context); + +unsigned long relocated_restore_code; +EXPORT_SYMBOL_GPL (relocated_restore_code); + +/** +* struct arch_hibernate_hdr_invariants - container to store kernel build version.
+ * @uts_version: to save the build number and date so that the we do not resume with
+ * a different kernel.
+ */arch/riscv/kernel/hibernate.c new file +6
-0+struct arch_hibernate_hdr_invariants { + char uts_version[__NEW_UTS_LEN + 1]; +}; + +/** +* struct arch_hibernate_hdr - helper parameters that help us to restore the image.
+ * @invariants: container to store kernel build version.
+ * @hartid: to make sure sameboot_cpuexecutes the hibernate/restore code.
+ * @saved_satp: original page table used by the hibernated image.
+ * @restore_cpu_addr: the kernel's image address to restore the CPU context.
+ */arch/riscv/kernel/hibernate.c new file +8
-0+static struct arch_hibernate_hdr { + struct arch_hibernate_hdr_invariants invariants; + unsigned long hartid; + unsigned long saved_satp; + unsigned long restore_cpu_addr; +} resume_hdr; + +static void arch_hdr_invariants(struct arch_hibernate_hdr_invariants*i)
+{
+memset(i, 0, sizeof(*i));
+memcpy(i->uts_version, init_utsname()->version, sizeof(i->uts_version));arch/riscv/kernel/hibernate.c new file +93
-0+} + +/* +* Check if the given pfn is in the 'nosave' section. +*/ +int pfn_is_nosave (unsigned long pfn) +{ + unsigned long nosave_begin_pfn = sym_to_pfn(&__nosave_begin); + unsigned long nosave_end_pfn = sym_to_pfn(&__nosave_end - 1); + + return ((pfn >= nosave_begin_pfn) && (pfn <= nosave_end_pfn)); +} + +void notrace save_processor_state(void) +{ + WARN_ON(num_online_cpus() != 1); +} + +void notrace restore_processor_state(void) +{ +} + +/* +* Helper parameters need to be saved to the hibernation image header. +*/ +int arch_hibernation_header_save (void *addr,unsigned int max_size) +{ + struct arch_hibernate_hdr *hdr = addr; + + if (max_size < sizeof(*hdr)) + return -EOVERFLOW; + + arch_hdr_invariants(&hdr->invariants); + + hdr->hartid = cpuid_to_hartid_map(sleep_cpu); + hdr->saved_satp = csr_read(CSR_SATP); + hdr->restore_cpu_addr = (unsigned long)__hibernate_cpu_resume; + + return 0; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_save); + +/* +* Retrieve the helper parameters from the hibernation image header. +*/ +int arch_hibernation_header_restore (void *addr) +{ + struct arch_hibernate_hdr_invariants invariants; + struct arch_hibernate_hdr *hdr = addr; + int ret = 0; + + arch_hdr_invariants(&invariants); + + if (memcmp(&hdr->invariants, &invariants, sizeof(invariants))) { + pr_crit("Hibernate image not generated by this kernel!\n"); + return -EINVAL; + } + + sleep_cpu = riscv_hartid_to_cpuid(hdr->hartid); + if (sleep_cpu < 0) { + pr_crit("Hibernated on a CPU not known to this kernel!\n"); + sleep_cpu = -EINVAL; + return -EINVAL; + } + +#ifdef CONFIG_SMP + ret = bringup_hibernate_cpu(sleep_cpu); + if (ret) { + sleep_cpu = -EINVAL; + return ret; + } +#endif + resume_hdr = *hdr; + + return ret; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_restore); + +int swsusp_arch_suspend (void ) +{ + int ret = 0; + + if (__cpu_suspend_enter(hibernate_cpu_context)) { + sleep_cpu = smp_processor_id(); + suspend_save_csrs(hibernate_cpu_context); + ret = swsusp_save(); + } else { + suspend_restore_csrs(hibernate_cpu_context); + flush_tlb_all(); + flush_icache_all(); + + /* +* Tell the hibernation core that we've just restored the memory.
+ */
+in_suspend= 0;
+sleep_cpu=-EINVAL;
+ }arch/riscv/kernel/hibernate.c new file +5
-0+ + return ret; +} + +static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +26
-0+{ + pte_t *src_ptep; + pte_t *dst_ptep; + + if (pmd_none(READ_ONCE(*dst_pmdp))) { + dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC); + if (!dst_ptep) + return -ENOMEM; + + pmd_populate_kernel(NULL, dst_pmdp, dst_ptep); + } + + dst_ptep = pte_offset_kernel(dst_pmdp, start); + src_ptep = pte_offset_kernel(src_pmdp, start); + + do { + pte_t pte = READ_ONCE(*src_ptep); + + if (pte_present(pte)) + set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); + } while (dst_ptep++, src_ptep++, start += PAGE_SIZE, start < end); + + return 0; +} + +static int temp_pgtable_map_pmd(pud_t *dst_pudp, pud_t *src_pudp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + pmd_t *src_pmdp; + pmd_t *dst_pmdp; + + if (pud_none(READ_ONCE(*dst_pudp))) { + dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC); + if (!dst_pmdp) + return -ENOMEM; + + pud_populate(NULL, dst_pudp, dst_pmdp); + } + + dst_pmdp = pmd_offset(dst_pudp, start); + src_pmdp = pmd_offset(src_pudp, start); + + do { + pmd_t pmd = READ_ONCE(*src_pmdp); + + next = pmd_addr_end(start, end); + + if (pmd_none(pmd)) + continue; + + if (pmd_leaf(pmd)) { + set_pmd(dst_pmdp, __pmd(pmd_val(pmd) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pte(dst_pmdp, src_pmdp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pmdp++, src_pmdp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_map_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + pud_t *dst_pudp; + pud_t *src_pudp; + + if (p4d_none(READ_ONCE(*dst_p4dp))) { + dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC); + if (!dst_pudp) + return -ENOMEM; + + p4d_populate(NULL, dst_p4dp, dst_pudp); + } + + dst_pudp = pud_offset(dst_p4dp, start); + src_pudp = pud_offset(src_p4dp, start); + + do { + pud_t pud = READ_ONCE(*src_pudp); + + next = pud_addr_end(start, end); + + if (pud_none(pud)) + continue; + + if (pud_leaf(pud)) { + set_pud(dst_pudp, __pud(pud_val(pud) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pmd(dst_pudp, src_pudp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pudp++, src_pudp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_map_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + p4d_t *dst_p4dp; + p4d_t *src_p4dp; + + if (pgd_none(READ_ONCE(*dst_pgdp))) { + dst_p4dp = (p4d_t *)get_safe_page(GFP_ATOMIC); + if (!dst_p4dp) + return -ENOMEM; + + pgd_populate(NULL, dst_pgdp, dst_p4dp); + } + + dst_p4dp = p4d_offset(dst_pgdp, start); + src_p4dp = p4d_offset(src_pgdp, start); + + do { + p4d_t p4d = READ_ONCE(*src_p4dp); + + next = p4d_addr_end(start, end); + + if (p4d_none(p4d)) + continue; + + if (p4d_leaf(p4d)) { + set_p4d(dst_p4dp, __p4d(p4d_val(p4d) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pud(dst_p4dp, src_p4dp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_p4dp++, src_p4dp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_mapping(pgd_t *pgdp, unsigned long start,unsigned long end,
pgprot_tprot)arch/riscv/kernel/hibernate.c new file +18
-0+{ + pgd_t *dst_pgdp = pgd_offset_pgd(pgdp, start); + pgd_t *src_pgdp = pgd_offset_k(start); + unsigned long next; + unsigned long ret; + + do { + pgd_t pgd = READ_ONCE(*src_pgdp); + + next = pgd_addr_end(start, end); + + if (pgd_none(pgd)) + continue; + + if (pgd_leaf(pgd)) { + set_pgd(dst_pgdp, __pgd(pgd_val(pgd) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_p4d(dst_pgdp, src_pgdp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pgdp++, src_pgdp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +27
-0+ + return 0; +} + +static unsigned long relocate_restore_code (void ) +{ + void *page = (void *)get_safe_page(GFP_ATOMIC); + + if (!page) + return -ENOMEM; + + copy_page(page, hibernate_core_restore_code); + + /* Make the page containing the relocated code executable. */ + set_memory_x((unsigned long)page, 1); + + return (unsigned long)page; +} + +int swsusp_arch_resume (void ) +{ + unsigned long end = (unsigned long)pfn_to_virt(max_low_pfn); + unsigned long start = PAGE_OFFSET; + int ret; + + /* +* Memory allocated by get_safe_page() will be dealt with by the hibernation core,
+ * we don't need to free it here.
+ */
+resume_pg_dir= (pgd_t*)get_safe_page(GFP_ATOMIC); + if (!resume_pg_dir)
+ return-ENOMEM;
+
+ /*Quoted 8 lines
Quoted 6 lines
+ * Create a temporary page table and map the whole linear region as executable and
+ * writable.
+ */
+ ret =temp_pgtable_mapping(resume_pg_dir, start, end, __pgprot(_PAGE_WRITE | _PAGE_EXEC));Hmm weird protections,
PAGE_PRESENTandPAGE_READare not even set. Shouldn't you usePAGE_KERNEL_EXECinstead?Quoted 1 line
Forget this one, that's the protections you add, so that's good :)
OK.
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
+ if (ret)
+ return ret;
+
+ /* Move the restore code to a new page so that it doesn't get overwritten by itself. */
+relocated_restore_code=relocate_restore_code();
+ if (relocated_restore_code==-ENOMEM)
+ return-ENOMEM;
+
+ /*
+ * Map the__hibernate_cpu_resume()address to the temporary page table so that the
+ * restore code can jumps to it after finished restore the image. The next execution
+ * code doesn't find itself in a different address space after switching over to the
+ * original page table used by the hibernated image. + *
+ */Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
+ start = (unsigned long)resume_hdr.restore_cpu_addr; + end = start +
PAGE_SIZE;
+
+ ret =temp_pgtable_mapping(resume_pg_dir, start, end, __pgprot(_PAGE_WRITE | _PAGE_EXEC));
+ if (ret)
+ return ret;You set the linear mapping as executable so no need for this.
We need this mapping because the kernel addresses and the linear addresses in riscv-64bit are in different boundary. For example:
Linear address start with0xff60000000000000
Kernel address start with0xffffffff80000000
Quoted 5 lines
Quoted 3 lines
Ok I understand now that you need to map this into the temporary page
table: one nit though, since you copy the kernel mapping, no need to usePAGE_WRITE|PAGE_EXEC, the protections are already correct.Here is the thing. For
riscv_64bit, I am agreed with you that we don’t need to usePAGE_WRITE|PAGE_EXEC.
However, we need to usePAGE_WRITE|PAGE_EXECforriscv_32bitbecause the linear addresses and kernel addresses are similar. The attribute becomes not writable if following the "original" page table. I am thinking to use #ifdefconfig_32bitto split the path for 64bit and 32bit but I just afraid the linear addresses and kernel addresses might change again in the future. To be on the safe side, I let bothriscv_32bit&riscv_64bitto use thePAGE_WRITE|PAGE_EXEC.
Indeed, you really thought about everything, well done! I would add a comment that explains just that above, no need for other ifdefs IMO.
Quoted 52 lines
For example:
Linear address and kernel address for 32bit started with0xc0000000Linear address for 64bit start with0xff60000000000000Kernel address for 64bit start with0xffffffff80000000Please let me know your thoughts.
Quoted 45 lines
Thanks,
Alex
Quoted 40 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
+
+ hibernate_restore_image(resume_hdr.saved_satp, (PFN_DOWN(__pa(resume_pg_dir)) |satp_mode), + resume_hdr.restore_cpu_addr);arch/riscv/kernel/hibernate.c new file +19
-0+ + return 0; +} + +#ifdef CONFIG_PM_SLEEP_SMP +int hibernate_resume_nonboot_cpu_disable (void ) +{ + if (sleep_cpu < 0) { + pr_err("Failing to resume from hibernate on an unknown CPU\n"); + return -ENODEV; + } + + return freeze_secondary_cpus(sleep_cpu); +} +#endif + +static int __init riscv_hibernate_init (void ) +{ + hibernate_cpu_context = kzalloc(sizeof(*hibernate_cpu_context),
GFP_KERNEL);arch/riscv/kernel/hibernate.c new file +4
-0+ + if (WARN_ON(!hibernate_cpu_context)) + return -ENOMEM; +Quoted 8 lines
Quoted 8 lines
arch/riscv/kernel/hibernate.c line 424
Quoted 4 lines
+ return 0;
arch/riscv/kernel/hibernate.c new file +3
-0+} + +early_initcall (riscv_hibernate_init);_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
RE: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-disk
Quoted 716 lines
-----Original Message-----
From: Alexandre Ghiti <[email protected]>
Sent: Friday, March 10, 2023 6:50 PM
To: JeeHeng Sia <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>
Subject: Re: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-diskQuoted 695 lines · JeeHeng Sia
Hi Alex,
Quoted 693 lines
-----Original Message-----
From: Alexandre Ghiti <[email protected]>
Sent: Friday, March 10, 2023 5:59 PM
To: JeeHeng Sia <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>
Subject: Re: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-diskHi JeeHeng,
Quoted 682 lines · JeeHeng Sia
Quoted 652 lines
-----Original Message-----
From: Alexandre Ghiti <[email protected]>
Sent: Wednesday, March 8, 2023 5:28 PM
To: JeeHeng Sia <[email protected]>; [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; Leyfoon Tan <[email protected]>; Mason Huo <[email protected]>
Subject: Re: [PATCH v5 4/4] RISC-V: Add arch functions to support hibernation/suspend-to-diskQuoted 634 lines · Alexandre Ghiti
Quoted 633 lines · Sia Jee Heng
Low level Arch functions were created to support hibernation.
swsusp_arch_suspend()relies code from__cpu_suspend_enter()to write cpu state onto the stack, then callingswsusp_save()to save the memory image.Arch specific hibernation header is implemented and is utilized by the
arch_hibernation_header_restore()andarch_hibernation_header_save()functions. The arch specific hibernation header consists of satp, hartid,
and thecpu_resumeaddress. The kernel built version is also need to be saved into the hibernation image header to making sure only the same kernel is restore when resume.
swsusp_arch_resume()creates a temporary page table that covering only the linear map. It copies the restore code to a 'safe' page, then start to restore the memory image. Once completed, it restores the original kernel's page table. It then calls into__hibernate_cpu_resume()to restore the CPU context. Finally, it follows the normal hibernation path back to the hibernation core.To enable hibernation/suspend to disk into RISCV, the below config need to be enabled:
-CONFIG_ARCH_HIBERNATION_HEADER
-CONFIG_ARCH_HIBERNATION_POSSIBLESigned-off-by Sia Jee Heng <[email protected]>Reviewed-by Ley Foon Tan <[email protected]>Reviewed-by Mason Huo <[email protected]>--- arch/riscv/Kconfig | 6 + arch/riscv/include/asm/assembler.h | 20 ++ arch/riscv/include/asm/suspend.h | 19 ++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/hibernate-asm.S | 77 ++++++ arch/riscv/kernel/hibernate.c | 424 +++++++++++++++++++++++++++++ 7 files changed, 552 insertions(+)create mode 100644
arch/riscv/kernel/hibernate-asm.Screate mode 100644arch/riscv/kernel/hibernate.carch/riscv/Kconfig +5
-0@@ -707,6 +707,12 @@ menu "Power management options" source "kernel/power/Kconfig" +config ARCH_HIBERNATION_POSSIBLE + def_bool y + +config ARCH_HIBERNATION_HEADER + def_bool HIBERNATION + endmenu # "Power management options" menu "CPU Power Management"arch/riscv/include/asm/assembler.h +19
-0@@ -59,4 +59,24 @@ REG_L s11, (SUSPEND_CONTEXT_REGS + PT_S11)(a0) .endm +/* +* copy_page - copy 1 page (4KB) of data from source to destination +* @a0 - destination +* @a1 - source +*/ + .macro copy_page a0, a1 + lui a2, 0x1 + add a2, a2, a0 +1 : + REG_L t0, 0(a1) + REG_L t1, SZREG(a1) + + REG_S t0, 0(a0) + REG_S t1, SZREG(a0) + + addi a0, a0, 2 * SZREG + addi a1, a1, 2 * SZREG + bne a2, a0, 1b + .endm + #endif /* __ASM_ASSEMBLER_H */arch/riscv/include/asm/suspend.h +4
-0@@ -21,6 +21,11 @@ struct suspend_context { #endif }; +/* +* Used by hibernation core and cleared during resume sequence +*/ +extern int in_suspend; + /* Low-level CPU suspend entry function */ int __cpu_suspend_enter(struct suspend_context *context); @@ -36,4 +41,18 @@ int __cpu_resume_enter(unsigned long hartid,unsigned long context);
/* Used to save and restore the CSRs */
voidsuspend_save_csrs(struct suspend_context *context); voidsuspend_restore_csrs(struct suspend_context *context);arch/riscv/include/asm/suspend.h +12
-0+ +/* Low-level API to support hibernation */ +int swsusp_arch_suspend (void ); +int swsusp_arch_resume (void ); +int arch_hibernation_header_save (void *addr,unsigned int max_size); +int arch_hibernation_header_restore (void *addr); +int __hibernate_cpu_resume (void ); + +/* Used to resume on the CPU we hibernated on */ +int hibernate_resume_nonboot_cpu_disable (void ); + +asmlinkage void hibernate_restore_image(unsigned long resume_satp,unsigned long
satp_temp,
+ unsigned longcpu_resume);arch/riscv/include/asm/suspend.h +1
-0+asmlinkage int hibernate_core_restore_code(void); #endifarch/riscv/kernel/Makefile +1
-0@@ -64,6 +64,7 @@ obj-$(CONFIG_MODULES) += module.o obj-$(CONFIG_MODULE_SECTIONS) += module-sections.o obj-$(CONFIG_CPU_PM) += suspend_entry.o suspend.o +obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_FUNCTION_TRACER) += mcount.o ftrace.o obj-$(CONFIG_DYNAMIC_FTRACE) += mcount-dyn.oarch/riscv/kernel/asm-offsets.c +4
-0@@ -9,6 +9,7 @@ #include <linux/kbuild.h> #include <linux/mm.h> #include <linux/sched.h> +#include <linux/suspend.h> #include <asm/kvm_host.h> #include <asm/thread_info.h> #include <asm/ptrace.h>@@ -116,6 +117,10 @@ void asm_offsets(void) OFFSET(SUSPEND_CONTEXT_REGS, suspend_context, regs); + OFFSET(HIBERN_PBE_ADDR, pbe, address); + OFFSET(HIBERN_PBE_ORIG, pbe, orig_address); + OFFSET(HIBERN_PBE_NEXT, pbe, next); + OFFSET(KVM_ARCH_GUEST_ZERO, kvm_vcpu_arch, guest_context.zero); OFFSET(KVM_ARCH_GUEST_RA, kvm_vcpu_arch, guest_context.ra); OFFSET(KVM_ARCH_GUEST_SP, kvm_vcpu_arch, guest_context.sp);arch/riscv/kernel/hibernate-asm.S new file +19
-0@@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Hibernation low level support for RISCV. + * + * Copyright (C) 2023 StarFive Technology Co., Ltd. + * + * Author: Jee Heng Sia <[email protected]> + */ + +#include <asm/asm.h> +#include <asm/asm-offsets.h> +#include <asm/assembler.h> +#include <asm/csr.h> + +#include <linux/linkage.h> + +/* + * int __hibernate_cpu_resume(void) + * Switch back to the hibernated image's page table prior torestoring the CPU
+ * context.
+ *
+ * Always returns 0
+ */arch/riscv/kernel/hibernate-asm.S new file +35
-0+ENTRY(__hibernate_cpu_resume) + /* switch to hibernated image's page table. */ + csrw CSR_SATP, s0 + sfence.vma + + REG_L a0, hibernate_cpu_context + + suspend_restore_csrs + suspend_restore_regs + + /* Return zero value. */ + mv a0, zero + + ret +END(__hibernate_cpu_resume) + +/* + * Prepare to restore the image. + * a0: satp of saved page tables. + * a1: satp of temporary page tables. + * a2: cpu_resume. + */ +ENTRY(hibernate_restore_image) + mv s0, a0 + mv s1, a1 + mv s2, a2 + REG_L s4, restore_pblist + REG_L a1, relocated_restore_code + + jalr a1 +END(hibernate_restore_image) + +/* + * The below code will be executed from a 'safe' page. + * It first switches to the temporary page table, then starts tocopy the pages
+ * back to the original memory location. Finally, it jumps to__hibernate_cpu_resume()
+ * to restore the CPU context.
+ */arch/riscv/kernel/hibernate-asm.S new file +16
-0+ENTRY(hibernate_core_restore_code) + /* switch to temp page table. */ + csrw satp, s1 + sfence.vma +.Lcopy: + /* The below code will restore the hibernated image. */ + REG_L a1, HIBERN_PBE_ADDR(s4) + REG_L a0, HIBERN_PBE_ORIG(s4) + + copy_page a0, a1 + + REG_L s4, HIBERN_PBE_NEXT(s4) + bnez s4, .Lcopy + + jalr s2 +END(hibernate_core_restore_code)arch/riscv/kernel/hibernate.c new file +28
-0@@ -0,0 +1,424 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Hibernation support for RISCV +* +* Copyright (C) 2023 StarFive Technology Co., Ltd. +* +* Author: Jee Heng Sia <[email protected]> +*/ + +#include <asm/barrier.h> +#include <asm/cacheflush.h> +#include <asm/mmu_context.h> +#include <asm/page.h> +#include <asm/pgalloc.h> +#include <asm/pgtable.h> +#include <asm/sections.h> +#include <asm/set_memory.h> +#include <asm/smp.h> +#include <asm/suspend.h> + +#include <linux/cpu.h> +#include <linux/memblock.h> +#include <linux/pm.h> +#include <linux/sched.h> +#include <linux/suspend.h> +#include <linux/utsname.h> + +/* The logical cpu number we should resume on, initialised to a non-cpu number. */
arch/riscv/kernel/hibernate.c new file +14
-0+static int sleep_cpu = -EINVAL ; + +/* Pointer to the temporary resume page table. */ +static pgd_t *resume_pg_dir; + +/* CPU context to be saved. */ +struct suspend_context *hibernate_cpu_context; +EXPORT_SYMBOL_GPL (hibernate_cpu_context); + +unsigned long relocated_restore_code; +EXPORT_SYMBOL_GPL (relocated_restore_code); + +/** +* struct arch_hibernate_hdr_invariants - container to store kernel build version.
+ * @uts_version: to save the build number and date so that the we do not resume with
+ * a different kernel.
+ */arch/riscv/kernel/hibernate.c new file +6
-0+struct arch_hibernate_hdr_invariants { + char uts_version[__NEW_UTS_LEN + 1]; +}; + +/** +* struct arch_hibernate_hdr - helper parameters that help us to restore the image.
+ * @invariants: container to store kernel build version.
+ * @hartid: to make sure sameboot_cpuexecutes the hibernate/restore code.
+ * @saved_satp: original page table used by the hibernated image.
+ * @restore_cpu_addr: the kernel's image address to restore the CPU context.
+ */arch/riscv/kernel/hibernate.c new file +8
-0+static struct arch_hibernate_hdr { + struct arch_hibernate_hdr_invariants invariants; + unsigned long hartid; + unsigned long saved_satp; + unsigned long restore_cpu_addr; +} resume_hdr; + +static void arch_hdr_invariants(struct arch_hibernate_hdr_invariants*i)
+{
+memset(i, 0, sizeof(*i));
+memcpy(i->uts_version, init_utsname()->version, sizeof(i->uts_version));arch/riscv/kernel/hibernate.c new file +93
-0+} + +/* +* Check if the given pfn is in the 'nosave' section. +*/ +int pfn_is_nosave (unsigned long pfn) +{ + unsigned long nosave_begin_pfn = sym_to_pfn(&__nosave_begin); + unsigned long nosave_end_pfn = sym_to_pfn(&__nosave_end - 1); + + return ((pfn >= nosave_begin_pfn) && (pfn <= nosave_end_pfn)); +} + +void notrace save_processor_state(void) +{ + WARN_ON(num_online_cpus() != 1); +} + +void notrace restore_processor_state(void) +{ +} + +/* +* Helper parameters need to be saved to the hibernation image header. +*/ +int arch_hibernation_header_save (void *addr,unsigned int max_size) +{ + struct arch_hibernate_hdr *hdr = addr; + + if (max_size < sizeof(*hdr)) + return -EOVERFLOW; + + arch_hdr_invariants(&hdr->invariants); + + hdr->hartid = cpuid_to_hartid_map(sleep_cpu); + hdr->saved_satp = csr_read(CSR_SATP); + hdr->restore_cpu_addr = (unsigned long)__hibernate_cpu_resume; + + return 0; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_save); + +/* +* Retrieve the helper parameters from the hibernation image header. +*/ +int arch_hibernation_header_restore (void *addr) +{ + struct arch_hibernate_hdr_invariants invariants; + struct arch_hibernate_hdr *hdr = addr; + int ret = 0; + + arch_hdr_invariants(&invariants); + + if (memcmp(&hdr->invariants, &invariants, sizeof(invariants))) { + pr_crit("Hibernate image not generated by this kernel!\n"); + return -EINVAL; + } + + sleep_cpu = riscv_hartid_to_cpuid(hdr->hartid); + if (sleep_cpu < 0) { + pr_crit("Hibernated on a CPU not known to this kernel!\n"); + sleep_cpu = -EINVAL; + return -EINVAL; + } + +#ifdef CONFIG_SMP + ret = bringup_hibernate_cpu(sleep_cpu); + if (ret) { + sleep_cpu = -EINVAL; + return ret; + } +#endif + resume_hdr = *hdr; + + return ret; +} +EXPORT_SYMBOL_GPL (arch_hibernation_header_restore); + +int swsusp_arch_suspend (void ) +{ + int ret = 0; + + if (__cpu_suspend_enter(hibernate_cpu_context)) { + sleep_cpu = smp_processor_id(); + suspend_save_csrs(hibernate_cpu_context); + ret = swsusp_save(); + } else { + suspend_restore_csrs(hibernate_cpu_context); + flush_tlb_all(); + flush_icache_all(); + + /* +* Tell the hibernation core that we've just restored the memory.
+ */
+in_suspend= 0;
+sleep_cpu=-EINVAL;
+ }arch/riscv/kernel/hibernate.c new file +5
-0+ + return ret; +} + +static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +26
-0+{ + pte_t *src_ptep; + pte_t *dst_ptep; + + if (pmd_none(READ_ONCE(*dst_pmdp))) { + dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC); + if (!dst_ptep) + return -ENOMEM; + + pmd_populate_kernel(NULL, dst_pmdp, dst_ptep); + } + + dst_ptep = pte_offset_kernel(dst_pmdp, start); + src_ptep = pte_offset_kernel(src_pmdp, start); + + do { + pte_t pte = READ_ONCE(*src_ptep); + + if (pte_present(pte)) + set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); + } while (dst_ptep++, src_ptep++, start += PAGE_SIZE, start < end); + + return 0; +} + +static int temp_pgtable_map_pmd(pud_t *dst_pudp, pud_t *src_pudp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + pmd_t *src_pmdp; + pmd_t *dst_pmdp; + + if (pud_none(READ_ONCE(*dst_pudp))) { + dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC); + if (!dst_pmdp) + return -ENOMEM; + + pud_populate(NULL, dst_pudp, dst_pmdp); + } + + dst_pmdp = pmd_offset(dst_pudp, start); + src_pmdp = pmd_offset(src_pudp, start); + + do { + pmd_t pmd = READ_ONCE(*src_pmdp); + + next = pmd_addr_end(start, end); + + if (pmd_none(pmd)) + continue; + + if (pmd_leaf(pmd)) { + set_pmd(dst_pmdp, __pmd(pmd_val(pmd) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pte(dst_pmdp, src_pmdp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pmdp++, src_pmdp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_map_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + pud_t *dst_pudp; + pud_t *src_pudp; + + if (p4d_none(READ_ONCE(*dst_p4dp))) { + dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC); + if (!dst_pudp) + return -ENOMEM; + + p4d_populate(NULL, dst_p4dp, dst_pudp); + } + + dst_pudp = pud_offset(dst_p4dp, start); + src_pudp = pud_offset(src_p4dp, start); + + do { + pud_t pud = READ_ONCE(*src_pudp); + + next = pud_addr_end(start, end); + + if (pud_none(pud)) + continue; + + if (pud_leaf(pud)) { + set_pud(dst_pudp, __pud(pud_val(pud) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pmd(dst_pudp, src_pudp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pudp++, src_pudp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_map_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp,unsigned long start,
+ unsigned long end,pgprot_tprot)arch/riscv/kernel/hibernate.c new file +29
-0+{ + unsigned long next; + unsigned long ret; + p4d_t *dst_p4dp; + p4d_t *src_p4dp; + + if (pgd_none(READ_ONCE(*dst_pgdp))) { + dst_p4dp = (p4d_t *)get_safe_page(GFP_ATOMIC); + if (!dst_p4dp) + return -ENOMEM; + + pgd_populate(NULL, dst_pgdp, dst_p4dp); + } + + dst_p4dp = p4d_offset(dst_pgdp, start); + src_p4dp = p4d_offset(src_pgdp, start); + + do { + p4d_t p4d = READ_ONCE(*src_p4dp); + + next = p4d_addr_end(start, end); + + if (p4d_none(p4d)) + continue; + + if (p4d_leaf(p4d)) { + set_p4d(dst_p4dp, __p4d(p4d_val(p4d) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_pud(dst_p4dp, src_p4dp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_p4dp++, src_p4dp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +5
-0+ + return 0; +} + +static int temp_pgtable_mapping(pgd_t *pgdp, unsigned long start,unsigned long end,
pgprot_tprot)arch/riscv/kernel/hibernate.c new file +18
-0+{ + pgd_t *dst_pgdp = pgd_offset_pgd(pgdp, start); + pgd_t *src_pgdp = pgd_offset_k(start); + unsigned long next; + unsigned long ret; + + do { + pgd_t pgd = READ_ONCE(*src_pgdp); + + next = pgd_addr_end(start, end); + + if (pgd_none(pgd)) + continue; + + if (pgd_leaf(pgd)) { + set_pgd(dst_pgdp, __pgd(pgd_val(pgd) | pgprot_val(prot))); + } else { + ret = temp_pgtable_map_p4d(dst_pgdp, src_pgdp, start,next, prot);
+ if (ret)
+ return-ENOMEM;
+ }
+ } while (dst_pgdp++, src_pgdp++, start = next, start != end);arch/riscv/kernel/hibernate.c new file +27
-0+ + return 0; +} + +static unsigned long relocate_restore_code (void ) +{ + void *page = (void *)get_safe_page(GFP_ATOMIC); + + if (!page) + return -ENOMEM; + + copy_page(page, hibernate_core_restore_code); + + /* Make the page containing the relocated code executable. */ + set_memory_x((unsigned long)page, 1); + + return (unsigned long)page; +} + +int swsusp_arch_resume (void ) +{ + unsigned long end = (unsigned long)pfn_to_virt(max_low_pfn); + unsigned long start = PAGE_OFFSET; + int ret; + + /* +* Memory allocated by get_safe_page() will be dealt with by the hibernation core,
+ * we don't need to free it here.
+ */
+resume_pg_dir= (pgd_t*)get_safe_page(GFP_ATOMIC); + if (!resume_pg_dir)
+ return-ENOMEM;
+
+ /*Quoted 8 lines
Quoted 6 lines
+ * Create a temporary page table and map the whole linear region as executable and
+ * writable.
+ */
+ ret =temp_pgtable_mapping(resume_pg_dir, start, end, __pgprot(_PAGE_WRITE | _PAGE_EXEC));Hmm weird protections,
PAGE_PRESENTandPAGE_READare not even set. Shouldn't you usePAGE_KERNEL_EXECinstead?Quoted 1 line
Forget this one, that's the protections you add, so that's good :)
OK.
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
+ if (ret)
+ return ret;
+
+ /* Move the restore code to a new page so that it doesn't get overwritten by itself. */
+relocated_restore_code=relocate_restore_code();
+ if (relocated_restore_code==-ENOMEM)
+ return-ENOMEM;
+
+ /*
+ * Map the__hibernate_cpu_resume()address to the temporary page table so that the
+ * restore code can jumps to it after finished restore the image. The next execution
+ * code doesn't find itself in a different address space after switching over to the
+ * original page table used by the hibernated image. + *
+ */Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
+ start = (unsigned long)resume_hdr.restore_cpu_addr; + end = start +
PAGE_SIZE;
+
+ ret =temp_pgtable_mapping(resume_pg_dir, start, end, __pgprot(_PAGE_WRITE | _PAGE_EXEC));
+ if (ret)
+ return ret;You set the linear mapping as executable so no need for this.
We need this mapping because the kernel addresses and the linear addresses in riscv-64bit are in different boundary. For
example:
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Linear address start with
0xff60000000000000
Kernel address start with0xffffffff80000000Ok I understand now that you need to map this into the temporary page
table: one nit though, since you copy the kernel mapping, no need to usePAGE_WRITE|PAGE_EXEC, the protections are already correct.Here is the thing. For
riscv_64bit, I am agreed with you that we don’t need to usePAGE_WRITE|PAGE_EXEC. However, we need to usePAGE_WRITE|PAGE_EXECforriscv_32bitbecause the linear addresses and kernel addresses are similar.
Quoted 7 lines
The attribute becomes not writable if following the "original" page table. I am thinking to use #ifdef
config_32bitto split the path for 64bit and 32bit but I just afraid the linear addresses and kernel addresses might change again in the future. To be on the safe side, I let bothriscv_32bit&riscv_64bitto use thePAGE_WRITE|PAGE_EXEC.Indeed, you really thought about everything, well done! I would add a comment that explains just that above, no need for other ifdefs IMO.
Sure, can add comment to explain the above situation.
Quoted 50 lines
Quoted 50 lines
For example:
Linear address and kernel address for 32bit started with0xc0000000Linear address for 64bit start with0xff60000000000000Kernel address for 64bit start with0xffffffff80000000Please let me know your thoughts.
Quoted 41 lines
Thanks,
Alex
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 24 lines
+
+ hibernate_restore_image(resume_hdr.saved_satp, (PFN_DOWN(__pa(resume_pg_dir)) |satp_mode), + resume_hdr.restore_cpu_addr);arch/riscv/kernel/hibernate.c new file +19
-0+ + return 0; +} + +#ifdef CONFIG_PM_SLEEP_SMP +int hibernate_resume_nonboot_cpu_disable (void ) +{ + if (sleep_cpu < 0) { + pr_err("Failing to resume from hibernate on an unknown CPU\n"); + return -ENODEV; + } + + return freeze_secondary_cpus(sleep_cpu); +} +#endif + +static int __init riscv_hibernate_init (void ) +{ + hibernate_cpu_context = kzalloc(sizeof(*hibernate_cpu_context),
GFP_KERNEL);arch/riscv/kernel/hibernate.c
Quoted 8 lines
arch/riscv/kernel/hibernate.c new file +8
-0+ + if (WARN_ON(!hibernate_cpu_context)) + return -ENOMEM; + + return 0; +} + +early_initcall (riscv_hibernate_init);_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv