Skip to contentSkip to conversation

arm64: hibernate: dust off the suspend/cleanup rework

PATCH In review Bradley Morgan 4 participants 23 messages Aug 25 – Sep 9 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover arm64: hibernate: dust off the suspend/cleanup rework

This comes out of the hibernate resume discussion where Will asked if the restored kernel's hyp vectors are reinstalled before we come back from swsusp_arch_suspend() and free the reserved pages. Rather than answer with a one off fix, this series reworks resume to look like a resume from idle, which makes the answer fall out on its own.

The series leans on Mark Rutland's old work for inspiration, he had the same end goal in a branch from back then:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/suspend/cleanup

The short answer to the question: yes, but only because the rework makes resume look like a resume from idle. We enter cpu_resume() at the highest available EL with the MMU off, and the resumed kernel installs its own hyp stub vectors into VBAR_EL2 through init_kernel_el() before anything else runs, so by the time swsusp_free() releases the trans_pgd copies nothing points at them anymore.

The series is structured so the first patch stands alone as a stable worthy fix for the missing hvc arguments. The rest is the cleanup: the stale init_kernel_el() comment gets corrected, MTE tag pages get freed after save instead of leaking on failed hibernates, the fragile _cpu_resume() calling convention gets split, and the last patch reworks hibernate to use the regular cpu_resume() logic end to end.

Everything object compiles and passes checkpatch and sparse. I could not boot or run a hibernate cycle on real hardware here, so that still wants doing before anyone gets excited.

Bradley Morgan (7):
arm64: hibernate: pass HVC_SET_VECTORS args to the resume hvc
arm64: head: correct comment for init_kernel_el()
arm64: hibernate: free MTE tag pages after saving
arm64: hibernate: fix _cpu_resume() calling convention
arm64: hibernate: use dcache_by_myline_op
arm64: hibernate: only clean to PoC
arm64: hibernate: use regular cpu_resume() logic

 arch/arm64/include/asm/suspend.h  |  1 -
 arch/arm64/kernel/head.S          |  6 +--
 arch/arm64/kernel/hibernate-asm.S | 87 +++++++++++++++++++------------
 arch/arm64/kernel/hibernate.c     | 66 +++++++++++------------
 arch/arm64/kernel/sleep.S         |  6 +--
 5 files changed, 88 insertions(+), 78 deletions(-)
Signature · 1 line
2.47.3
#1 lore raw

PATCH 1/7 arm64: hibernate: pass HVC_SET_VECTORS args to the resume hvc

swsusp_arch_suspend_exit() reinstalls the restored kernel's hyp stub vectors with an hvc, but never passes the arguments. x0 is not set to HVC_SET_VECTORS and x1 is not set to the vector address, so the stub dispatch falls through and returns without writing vbar_el2. EL2 is left pointing at the trans_pgd copy of the vectors, a page that swsusp_free() releases right after resume.

Set the arguments up the same way __hyp_set_vectors() does.

Fixes 788bfdd97434 ("arm64: trans_pgd: hibernate: Add trans_pgd_copy_el2_vectors")
---
 arch/arm64/kernel/hibernate-asm.S | 2 ++
 1 file changed, 2 insertions(+)
arch/arm64/kernel/hibernate-asm.S +2 -0
index 0e1d9c3c6a93..2baefe7a82d3 100644
@@ -89,6 +89,8 @@ alternative_insn "dc cvau, x4",  "dc civac, x4",  ARM64_WORKAROUND_CLEAN_CACHE
 	isb
 
 	cbz	x24, 3f		/* Do we need to re-initialise EL2? */
+	mov	x1, x24
+	mov	x0, #HVC_SET_VECTORS
 	hvc	#0
 3:	ret
 SYM_CODE_END(swsusp_arch_suspend_exit)
Signature · 1 line
2.47.3
#2 lore raw

PATCH 2/7 arm64: head: correct comment for init_kernel_el()

In reply to Bradley Morgan

The comment above init_kernel_el() still says the function configures the CPU to execute at the highest reachable EL, but that stopped being true a while back. Ever since commit ae4b7e38e9a94798 ("arm64: Allow sticky E2H when entering EL1"), init_kernel_el() always drops to EL1, and it is finalise_el2() that brings us back up to EL2 when we want it.

Update the comment to match what the code actually does.

Cc Ard Biesheuvel <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc James Morse <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/kernel/head.S | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm64/kernel/head.S +3 -3
index 87a822e5c4ca..c6301557eee1 100644
@@ -254,9 +254,9 @@ SYM_FUNC_END(__primary_switched)
 	.section ".idmap.text","a"
 
 /*
- * Starting from EL2 or EL1, configure the CPU to execute at the highest
- * reachable EL supported by the kernel in a chosen default state. If dropping
- * from EL2 to EL1, configure EL2 before configuring EL1.
+ * Starting from EL2 or EL1, configure the CPU to execute at EL1.
+ * If dropping from EL2 to EL1, configure EL2 before configuring EL1.
+ * To use VHE we'll upgrade back to EL2 later in finalise_el2().
  *
  * Since we cannot always rely on ERET synchronizing writes to sysregs (e.g. if
  * SCTLR_ELx.EOS is clear), we place an ISB prior to ERET.
Signature · 1 line
2.47.3
reviewer
#3 lore raw

Re: [PATCH 2/7] arm64: head: correct comment for init_kernel_el()

Quoted 14 lines · Bradley Morgan

The comment above init_kernel_el() still says the function configures the CPU to execute at the highest reachable EL, but that stopped being true a while back. Ever since commit ae4b7e38e9a94798 ("arm64: Allow sticky E2H when entering EL1"), init_kernel_el() always drops to EL1, and it is finalise_el2() that brings us back up to EL2 when we want it.

Update the comment to match what the code actually does.

Cc Ard Biesheuvel <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc James Morse <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
Quoted 3 lines
---
 arch/arm64/kernel/head.S | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

This looks to be my patch [1], but with a (partially) rewritten commit message, and my S-o-B dropped.

There was admittedly one typo in that commit message that needed to be fixed, but I don't think the rest of the changes were necessary, and I don't think it's approriate to drop my S-o-B.

I think similar is true for the rest of the series. The patch fixing the _cpu_resume() only seems to have a paraphrased commit message.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/commit/?id=ee4323ff6a2ba4c3989b38f13b2c1a516d6c27e4

Mark.

Quoted 20 lines
arch/arm64/kernel/head.S +3 -3
index 87a822e5c4ca..c6301557eee1 100644
@@ -254,9 +254,9 @@ SYM_FUNC_END(__primary_switched)
 	.section ".idmap.text","a"
 
 /*
- * Starting from EL2 or EL1, configure the CPU to execute at the highest
- * reachable EL supported by the kernel in a chosen default state. If dropping
- * from EL2 to EL1, configure EL2 before configuring EL1.
+ * Starting from EL2 or EL1, configure the CPU to execute at EL1.
+ * If dropping from EL2 to EL1, configure EL2 before configuring EL1.
+ * To use VHE we'll upgrade back to EL2 later in finalise_el2().
  *
  * Since we cannot always rely on ERET synchronizing writes to sysregs (e.g. if
  * SCTLR_ELx.EOS is clear), we place an ISB prior to ERET.
Signature · 1 line
2.47.3
#4 lore raw

Re: [PATCH 2/7] arm64: head: correct comment for init_kernel_el()

Quoted 46 lines · Mark Rutland
Quoted 14 lines · Bradley Morgan

The comment above init_kernel_el() still says the function configures the CPU to execute at the highest reachable EL, but that stopped being true a while back. Ever since commit ae4b7e38e9a94798 ("arm64: Allow sticky E2H when entering EL1"), init_kernel_el() always drops to EL1, and it is finalise_el2() that brings us back up to EL2 when we want it.

Update the comment to match what the code actually does.

Cc Ard Biesheuvel <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc James Morse <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
Quoted 3 lines
---
 arch/arm64/kernel/head.S | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

This looks to be my patch [1], but with a (partially) rewritten commit message, and my S-o-B dropped.

There was admittedly one typo in that commit message that needed to be fixed, but I don't think the rest of the changes were necessary, and I don't think it's approriate to drop my S-o-B.

I think similar is true for the rest of the series. The patch fixing the _cpu_resume() only seems to have a paraphrased commit message.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/commit/?id=ee4323ff6a2ba4c3989b38f13b2c1a516d6c27e4

Mark.

arch/arm64/kernel/head.S SYM_FUNC_END(__primary_switched) · line 257

Quoted 10 lines
arch/arm64/kernel/head.S +0 -1
index 87a822e5c4ca..c6301557eee1 100644
@@ -254,9 +254,9 @@ SYM_FUNC_END(__primary_switched)
 	.section ".idmap.text","a"
 
 /*
- * Starting from EL2 or EL1, configure the CPU to execute at the

highest

arch/arm64/kernel/head.S SYM_FUNC_END(__primary_switched) · line 258

Quoted 1 line
arch/arm64/kernel/head.S +0 -1
- * reachable EL supported by the kernel in a chosen default state. If
Quoted 11 lines

dropping

arch/arm64/kernel/head.S SYM_FUNC_END(__primary_switched)

Quoted 6 lines
arch/arm64/kernel/head.S +3 -1
- * from EL2 to EL1, configure EL2 before configuring EL1.
+ * Starting from EL2 or EL1, configure the CPU to execute at EL1.
+ * If dropping from EL2 to EL1, configure EL2 before configuring EL1.
+ * To use VHE we'll upgrade back to EL2 later in finalise_el2().
  *
  * Since we cannot always rely on ERET synchronizing writes to sysregs

(e.g. if

Quoted 3 lines
arch/arm64/kernel/head.S +0 -0
  * SCTLR_ELx.EOS is clear), we place an ISB prior to ERET.
Signature · 1 line
2.47.3

Hi mark, do you **WANT** your sob? I don't mind.

I dropped it because it seemed you didn't care about it (or were too busy),hence, yeah.

--- Thanks!
https://lore.kernel.org/all/[email protected]/

reviewer
#5 lore raw

Re: [PATCH 2/7] arm64: head: correct comment for init_kernel_el()

Quoted 55 lines · Bradley Morgan
Quoted 53 lines · Mark Rutland
Quoted 14 lines · Bradley Morgan

The comment above init_kernel_el() still says the function configures the CPU to execute at the highest reachable EL, but that stopped being true a while back. Ever since commit ae4b7e38e9a94798 ("arm64: Allow sticky E2H when entering EL1"), init_kernel_el() always drops to EL1, and it is finalise_el2() that brings us back up to EL2 when we want it.

Update the comment to match what the code actually does.

Cc Ard Biesheuvel <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc James Morse <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
Quoted 3 lines
---
 arch/arm64/kernel/head.S | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

This looks to be my patch [1], but with a (partially) rewritten commit message, and my S-o-B dropped.

There was admittedly one typo in that commit message that needed to be fixed, but I don't think the rest of the changes were necessary, and I don't think it's approriate to drop my S-o-B.

I think similar is true for the rest of the series. The patch fixing the _cpu_resume() only seems to have a paraphrased commit message.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/commit/?id=ee4323ff6a2ba4c3989b38f13b2c1a516d6c27e4

Mark.

arch/arm64/kernel/head.S SYM_FUNC_END(__primary_switched) · line 257

Quoted 10 lines
arch/arm64/kernel/head.S +0 -1
index 87a822e5c4ca..c6301557eee1 100644
@@ -254,9 +254,9 @@ SYM_FUNC_END(__primary_switched)
 	.section ".idmap.text","a"
 
 /*
- * Starting from EL2 or EL1, configure the CPU to execute at the

highest

arch/arm64/kernel/head.S SYM_FUNC_END(__primary_switched) · line 258

Quoted 1 line
arch/arm64/kernel/head.S +0 -1
- * reachable EL supported by the kernel in a chosen default state. If

dropping

Quoted 6 lines
arch/arm64/kernel/head.S +3 -1
- * from EL2 to EL1, configure EL2 before configuring EL1.
+ * Starting from EL2 or EL1, configure the CPU to execute at EL1.
+ * If dropping from EL2 to EL1, configure EL2 before configuring EL1.
+ * To use VHE we'll upgrade back to EL2 later in finalise_el2().
  *
  * Since we cannot always rely on ERET synchronizing writes to sysregs
Quoted 5 lines
Quoted 4 lines

(e.g. if

Quoted 3 lines
arch/arm64/kernel/head.S +0 -0
  * SCTLR_ELx.EOS is clear), we place an ISB prior to ERET.
Signature · 1 line
2.47.3

Hi mark, do you **WANT** your sob? I don't mind.

Please read Documentation/process/submitting-patches.rst

The SoB lines form a chain of authorship and delivery. You *must not* drop an SoB line.

Quoted 2 lines

I dropped it because it seemed you didn't care about it (or were too busy),hence, yeah.

I was busy, then on holiday. Neither has any relevance to the SoB lines.

I do care about these patches, and I'm going to have to review them regardless. The reason I hadn't posted them was that this is a subtle area that needs careful handling.

Mark.

#6 lore raw

Re: [PATCH 2/7] arm64: head: correct comment for init_kernel_el()

Quoted 77 lines · Mark Rutland
Quoted 5 lines · Bradley Morgan
Quoted 3 lines · Mark Rutland
Quoted 2 lines · Bradley Morgan

The comment above init_kernel_el() still says the function configures the CPU to execute at the highest reachable EL, but that stopped

being

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

true a while back. Ever since commit ae4b7e38e9a94798 ("arm64: Allow sticky E2H when entering EL1"), init_kernel_el() always drops to EL1, and it is finalise_el2() that brings us back up to EL2 when we want

it.

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

Update the comment to match what the code actually does.

Cc Ard Biesheuvel <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc James Morse <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/kernel/head.S | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
Quoted 8 lines
Quoted 8 lines

This looks to be my patch [1], but with a (partially) rewritten commit message, and my S-o-B dropped.

There was admittedly one typo in that commit message that needed to be fixed, but I don't think the rest of the changes were necessary, and I don't think it's approriate to drop my S-o-B.

I think similar is true for the rest of the series. The patch fixing

the

Quoted 20 lines
Quoted 3 lines

_cpu_resume() only seems to have a paraphrased commit message.

[1]

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/commit/?id=ee4323ff6a2ba4c3989b38f13b2c1a516d6c27e4

Quoted 15 lines

Mark.

arch/arm64/kernel/head.S SYM_FUNC_END(__primary_switched) · line 257

Quoted 10 lines
arch/arm64/kernel/head.S +0 -1
index 87a822e5c4ca..c6301557eee1 100644
@@ -254,9 +254,9 @@ SYM_FUNC_END(__primary_switched)
 	.section ".idmap.text","a"
 
 /*
- * Starting from EL2 or EL1, configure the CPU to execute at the

highest

Quoted 1 line
arch/arm64/kernel/head.S +0 -1
- * reachable EL supported by the kernel in a chosen default state.

If

Quoted 4 lines
Quoted 4 lines

dropping

Quoted 3 lines
arch/arm64/kernel/head.S +2 -1
- * from EL2 to EL1, configure EL2 before configuring EL1.
+ * Starting from EL2 or EL1, configure the CPU to execute at EL1.
+ * If dropping from EL2 to EL1, configure EL2 before configuring

EL1.

Quoted 3 lines
Quoted 3 lines

arch/arm64/kernel/head.S SYM_FUNC_END(__primary_switched)

Quoted 3 lines
arch/arm64/kernel/head.S +1 -0
+ * To use VHE we'll upgrade back to EL2 later in finalise_el2().
  *
  * Since we cannot always rely on ERET synchronizing writes to

sysregs

Quoted 5 lines
Quoted 4 lines

(e.g. if

Quoted 3 lines
arch/arm64/kernel/head.S +0 -0
  * SCTLR_ELx.EOS is clear), we place an ISB prior to ERET.
Signature · 1 line
2.47.3

Hi mark, do you **WANT** your sob? I don't mind.

Please read Documentation/process/submitting-patches.rst

The SoB lines form a chain of authorship and delivery. You *must not* drop an SoB line.

Quoted 2 lines

I dropped it because it seemed you didn't care about it (or were too busy),hence, yeah.

Quoted 7 lines

I was busy, then on holiday. Neither has any relevance to the SoB lines.

I do care about these patches, and I'm going to have to review them regardless. The reason I hadn't posted them was that this is a subtle area that needs careful handling.

Mark.

Sorry mate,

V2 will have your SOB on all patches

I removed them due to like, what if you said

"Remove my sob these patches are terrible and unrelated"

Well, this is a corner case, but I had to think this through,

Yep, V2 will have your sob on all 7 patches.
--- Thanks!
https://lore.kernel.org/all/[email protected]/

#7 lore raw

PATCH 3/7 arm64: hibernate: free MTE tag pages after saving

In reply to Bradley Morgan

swsusp_arch_suspend() calls swsusp_mte_save_tags() to stash the userspace MTE tags, but we only free them on a successful resume in swsusp_mte_restore_tags(). If hibernation fails partway, the tags stay allocated, and the next time we try to hibernate, swsusp_mte_save_tags() warns about duplicate entries in the mte_pages xarray.

Reproducing it is easy, just make a hibernate fail twice in a row.

| # echo test_resume > /sys/power/disk
| # echo disk > /sys/power/state
...
| PM: Cannot find swap device, try swapon -a
| PM: Cannot get swap writer
...
| # echo disk > /sys/power/state
...

| ------------[ cut here ]------------
| swsusp: save_tags: Duplicate entry
| WARNING: CPU: 0 PID: 123 at arch/arm64/kernel/hibernate.c:234 \
| swsusp_arch_suspend+0x3f0/0x5b0

...

Free the tag storage right after swsusp_save() has copied it into the hibernation image. Where hibernation succeeds, the tags come back from the image and get consumed by swsusp_mte_restore_tags() as before.

Fixes ee11f332af96 ("arm64: mte: Save tags when hibernating")
---
 arch/arm64/kernel/hibernate.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/arm64/kernel/hibernate.c +10 -0
index 7bf117427777..8ac29058a839 100644
@@ -241,14 +241,18 @@ static int save_tags(struct page *page, unsigned long pfn)
 static void swsusp_mte_free_storage(void)
 {
 	XA_STATE(xa_state, &mte_pages, 0);
+	int n = 0;
 	void *tags;
 
 	xa_lock(&mte_pages);
 	xas_for_each(&xa_state, tags, ULONG_MAX) {
 		mte_free_tag_storage(tags);
+		n++;
 	}
 	xa_unlock(&mte_pages);
 
+	pr_info("Freed %d MTE pages\n", n);
+
 	xa_destroy(&mte_pages);
 }
 
@@ -355,6 +359,12 @@ int swsusp_arch_suspend(void)
 
 		sleep_cpu = smp_processor_id();
 		ret = swsusp_save();
+
+		/*
+		 * Hibernation can still fail from here, so free the tags now,
+		 * a second attempt would warn on duplicate entries otherwise.
+		 */
+		swsusp_mte_free_storage();
 	} else {
 		/* Clean kernel core startup/idle code to PoC*/
 		dcache_clean_inval_poc((unsigned long)__mmuoff_data_start,
Signature · 1 line
2.47.3
#8 lore raw

PATCH 4/7 arm64: hibernate: fix _cpu_resume() calling convention

In reply to Bradley Morgan

The hibernate path borrows bits of the normal idle suspend/resume code and open codes the rest. It skips cpu_resume() and has swsusp_arch_suspend_exit() branch straight into _cpu_resume() at the kernel's native EL with the MMU on.

Commit 82e4958800c01daa ("arm64: head: Move all finalise_el2 calls to after __enable_mmu") changed the calling convention for _cpu_resume() to expect EL1 with the boot mode in x19, but we never updated swsusp_arch_suspend_exit() to match.

So when swsusp_arch_suspend_exit() calls _cpu_resume(), x19 holds the final struct pbe next pointer, which must be NULL since it marks the end of the list. _cpu_resume() then passes that to finalise_el2() in x0, and finalise_el2() only issues an HVC when the value is BOOT_CPU_MODE_EL2 and we are at EL1, so it happens not to fire. That is the right outcome, but it is pure luck rather than design.

Split _cpu_resume() so this is less fragile.

__cpu_resume_switched() is the shared part, used by both the idle and hibernate code. It takes no arguments and issues no HVC. The name is meant to match __primary_switched and __secondary_switched.

__cpu_resume_switched_finalise_el2() is for the idle path only. It pulls the boot mode from x19 and calls finalise_el2() before __cpu_resume_switched(). It is kept local to sleep.S so the odd calling convention does not leak any further.

Fixes 82e4958800c01daa ("arm64: head: Move all finalise_el2 calls to after __enable_mmu")
Cc Ard Biesheuvel <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc James Morse <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/include/asm/suspend.h | 2 +-
 arch/arm64/kernel/hibernate.c    | 2 +-
 arch/arm64/kernel/sleep.S        | 9 ++++++---
 3 files changed, 8 insertions(+), 5 deletions(-)
arch/arm64/include/asm/suspend.h +1 -1
index e9ce68d50ba4..1b7570902264 100644
@@ -41,7 +41,7 @@ extern int cpu_suspend(unsigned long arg, int (*fn)(unsigned long));
 extern void cpu_resume(void);
 int __cpu_suspend_enter(struct sleep_stack_data *state);
 void __cpu_suspend_exit(void);
-void _cpu_resume(void);
+void __cpu_resume_switched(void);
 
 int swsusp_arch_suspend(void);
 int swsusp_arch_resume(void);
arch/arm64/kernel/hibernate.c +1 -1
index 8ac29058a839..30b02e39a397 100644
@@ -114,7 +114,7 @@ int arch_hibernation_header_save(void *addr, unsigned int max_size)
 
 	arch_hdr_invariants(&hdr->invariants);
 	hdr->ttbr1_el1		= __pa_symbol(swapper_pg_dir);
-	hdr->reenter_kernel	= _cpu_resume;
+	hdr->reenter_kernel	= __cpu_resume_switched;
 
 	/* We can't use __hyp_get_vectors() because kvm may still be loaded */
 	if (el2_reset_needed())
arch/arm64/kernel/sleep.S +6 -3
index f093cdf71be1..da45ab63bd9c 100644
@@ -107,16 +107,19 @@ SYM_CODE_START(cpu_resume)
 	adrp	x1, swapper_pg_dir
 	adrp	x2, idmap_pg_dir
 	bl	__enable_mmu
-	ldr	x8, =_cpu_resume
+	ldr	x8, =__cpu_resume_switched_finalise_el2
 	br	x8
 SYM_CODE_END(cpu_resume)
 	.ltorg
 	.popsection
 
-SYM_FUNC_START(_cpu_resume)
+SYM_FUNC_START_LOCAL(__cpu_resume_switched_finalise_el2)
 	mov	x0, x19
 	bl	finalise_el2
+	b	__cpu_resume_switched
+SYM_FUNC_END(__cpu_resume_switched_finalise_el2)
 
+SYM_FUNC_START(__cpu_resume_switched)
 	mrs	x1, mpidr_el1
 	adr_l	x8, mpidr_hash		// x8 = struct mpidr_hash virt address
 
@@ -152,4 +155,4 @@ SYM_FUNC_START(_cpu_resume)
 	ldp	x29, lr, [x29]
 	mov	x0, #0
 	ret
-SYM_FUNC_END(_cpu_resume)
+SYM_FUNC_END(__cpu_resume_switched)
Signature · 1 line
2.47.3
#9 lore raw

PATCH 5/7 arm64: hibernate: use dcache_by_myline_op

In reply to Bradley Morgan

swsusp_arch_suspend_exit() used to open code its cache maintenance, but the dcache_by_myline_op macro does everything that open coded version did, including the ARM64_WORKAROUND_CLEAN_CACHE handling. So use it.

Since this was first written the macro got split, with the trailing DSB moved out into a wrapper, so use dcache_by_myline_op_nosync here and keep relying on the DSB that already sits after the restore loop. That keeps this a true no-op.

Cc Ard Biesheuvel <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc James Morse <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/kernel/hibernate-asm.S | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)
arch/arm64/kernel/hibernate-asm.S +3 -8
index 2baefe7a82d3..f7613404f99e 100644
@@ -66,16 +66,11 @@ SYM_CODE_START(swsusp_arch_suspend_exit)
 
 	copy_page	x0, x1, x2, x3, x4, x5, x6, x7, x8, x9
 
+	/* Clean the restored page to the PoU */
+	mov	x0, x10
 	add	x1, x10, #PAGE_SIZE
-	/* Clean the copied page to PoU - based on caches_clean_inval_pou() */
 	raw_dcache_line_size x2, x3
-	sub	x3, x2, #1
-	bic	x4, x10, x3
-2:	/* clean D line / unified line */
-alternative_insn "dc cvau, x4",  "dc civac, x4",  ARM64_WORKAROUND_CLEAN_CACHE
-	add	x4, x4, x2
-	cmp	x4, x1
-	b.lo	2b
+	dcache_by_myline_op_nosync cvau, x0, x1, x2, x3
 
 	ldr	x19, [x19, #HIBERN_PBE_NEXT]
 	cbnz	x19, 1b
Signature · 1 line
2.47.3
#10 lore raw

PATCH 6/7 arm64: hibernate: only clean to PoC

In reply to Bradley Morgan

The hibernate code has to clean parts of the kernel to the PoC so they can be used with the MMU off. We only need to clean to the PoC, but we have been using dcache_clean_inval_poc(), which does a clean plus invalidate, just because that is what was available.

When hibernate was added in commit 82869ac57b5d ("arm64: kernel: Add
support for hibernate/suspend-to-disk"), the only cache maintenance
exposed outside the DMA code was __flush_dcache_area(), so that is what
got used. Commit d34fdb7081394cbf ("arm64: mm: convert __dma_* routines to use start, size") later exposed dcache_clean_area_poc(), and commit fade9c2c6ee2baea
("arm64: Rename arm64-internal cache maintenance functions") renamed
things again, but the hibernate code never switched over either time.

The invalidate is pointless anyway, since the mappings we clean through are cacheable and the CPU can just pull the memory straight back into cache right after. Use dcache_clean_poc() instead.

Cc Ard Biesheuvel <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc James Morse <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/kernel/hibernate.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
arch/arm64/kernel/hibernate.c +8 -9
index 30b02e39a397..60e264299a4d 100644
@@ -367,18 +367,17 @@ int swsusp_arch_suspend(void)
 		swsusp_mte_free_storage();
 	} else {
 		/* Clean kernel core startup/idle code to PoC*/
-		dcache_clean_inval_poc((unsigned long)__mmuoff_data_start,
-				    (unsigned long)__mmuoff_data_end);
-		dcache_clean_inval_poc((unsigned long)__idmap_text_start,
-				    (unsigned long)__idmap_text_end);
+		dcache_clean_poc((unsigned long)__mmuoff_data_start,
+				 (unsigned long)__mmuoff_data_end);
+		dcache_clean_poc((unsigned long)__idmap_text_start,
+				 (unsigned long)__idmap_text_end);
 
 		/* Clean kvm setup code to PoC? */
 		if (el2_reset_needed()) {
-			dcache_clean_inval_poc(
-				(unsigned long)__hyp_idmap_text_start,
-				(unsigned long)__hyp_idmap_text_end);
-			dcache_clean_inval_poc((unsigned long)__hyp_text_start,
-					    (unsigned long)__hyp_text_end);
+			dcache_clean_poc((unsigned long)__hyp_idmap_text_start,
+					 (unsigned long)__hyp_idmap_text_end);
+			dcache_clean_poc((unsigned long)__hyp_text_start,
+					 (unsigned long)__hyp_text_end);
 		}
 
 		swsusp_mte_restore_tags();
Signature · 1 line
2.47.3
#11 lore raw

PATCH 7/7 arm64: hibernate: use regular cpu_resume() logic

In reply to Bradley Morgan

Right now the hibernate path borrows parts of the idle suspend/resume code and writes the rest itself. It skips cpu_resume() and branches straight into __cpu_resume_switched() at the kernel's native EL with the MMU on. That split has been the source of a few subtle bugs, and it makes the code a pain to maintain.

Rework it so a resume from hibernate looks like a resume from idle. We enter cpu_resume() at the highest available EL with the MMU off, and leave the resumed kernel responsible for restoring all its CPU state the same way idle does. That makes the handoff cleaner and should be more robust.

With this, swsusp_arch_suspend_exit() now cleans the whole kernel Image to the PoC instead of leaving the resumed kernel to clean bits of itself. That is nicer on its own, since we cannot forget to clean some portion of the Image that gets added later, and the maintenance is cheap next to the page copying that dominates resume time anyway.

It also means __cpu_resume_switched() no longer needs to be visible outside sleep.S, so it can merge back with __cpu_resume_switched_finalise_el2().

Since this was first written, two things landed upstream and are folded in here. Commit 684bde100117931f ("arm64: hibernate: mask DAIF before restoring hibernated kernel") made swsusp_arch_resume() mask all DAIF exceptions before calling hibernate_exit() and mark it unreachable(), and that masking is kept around the new hibernate_exit() call. Commit 2c92eff008a2 ("arm64: Provide dcache_by_myline_op_nosync helper") split dcache_by_myline_op into a _nosync form, so use that and rely on the existing DSB after the maintenance. swsusp_arch_resume() also keeps its __nocfi annotation from commit e2f8216ca2d8 ("arm64: Set __nocfi on swsusp_arch_resume()").

Cc Ard Biesheuvel <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc James Morse <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm64/include/asm/suspend.h  |  1 -
 arch/arm64/kernel/hibernate-asm.S | 78 ++++++++++++++++++++-----------
 arch/arm64/kernel/hibernate.c     | 55 ++++++++--------------
 arch/arm64/kernel/sleep.S         |  7 +--
 4 files changed, 71 insertions(+), 70 deletions(-)
arch/arm64/include/asm/suspend.h +0 -1
index 1b7570902264..3aba69df06b9 100644
@@ -41,7 +41,6 @@ extern int cpu_suspend(unsigned long arg, int (*fn)(unsigned long));
 extern void cpu_resume(void);
 int __cpu_suspend_enter(struct sleep_stack_data *state);
 void __cpu_suspend_exit(void);
-void __cpu_resume_switched(void);
 
 int swsusp_arch_suspend(void);
 int swsusp_arch_resume(void);
arch/arm64/kernel/hibernate-asm.S +50 -28

111 lines not on this page · raw message

arch/arm64/kernel/hibernate.c +19 -36

87 lines not on this page · raw message

arch/arm64/kernel/sleep.S +2 -5
index da45ab63bd9c..8c63ffdf6d75 100644
@@ -107,19 +107,16 @@ SYM_CODE_START(cpu_resume)
 	adrp	x1, swapper_pg_dir
 	adrp	x2, idmap_pg_dir
 	bl	__enable_mmu
-	ldr	x8, =__cpu_resume_switched_finalise_el2
+	ldr	x8, =__cpu_resume_switched
 	br	x8
 SYM_CODE_END(cpu_resume)
 	.ltorg
 	.popsection
 
-SYM_FUNC_START_LOCAL(__cpu_resume_switched_finalise_el2)
+SYM_FUNC_START_LOCAL(__cpu_resume_switched)
 	mov	x0, x19
 	bl	finalise_el2
-	b	__cpu_resume_switched
-SYM_FUNC_END(__cpu_resume_switched_finalise_el2)
 
-SYM_FUNC_START(__cpu_resume_switched)
 	mrs	x1, mpidr_el1
 	adr_l	x8, mpidr_hash		// x8 = struct mpidr_hash virt address
 
Signature · 1 line
2.47.3
Tested-by on the v1 series
#12 lore raw
In reply to Bradley Morgan

Hi Bradley,

Quoted 4 lines · Bradley Morgan

Everything object compiles and passes checkpatch and sparse. I could not boot or run a hibernate cycle on real hardware here, so that still wants doing before anyone gets excited.

You can easily test hibernate under QEMU/mach-virt: just create a file on the host, mkswap it and pass it to the VM as a virtio drive (e.g., /dev/vda)

You can then swapon it in the VM and do 'echo disk >/sys/power/state', which will perform the hibernate and exit QEMU. If you then restart it with 'resume=/dev/vda' on the command line, it will perform the resume from hibernate at boot.

I've had a go with your series on a NV capable system (Mac Mini M2). Without the series applied, I can hibernate and resume 4 vCPUs without issue, so it is not entirely clear to me what your series intends to fix.

With the series applied resume from hibernate still works - I did see one intermittent failure where the secondaries failed to come up, but I only saw it once and so I cannot say for certain whether your series triggered it or not.

#13 lore raw
Quoted 2 lines · Ard Biesheuvel

Hi Bradley,

Ard!

Quoted 6 lines · Bradley Morgan
Quoted 4 lines

Everything object compiles and passes checkpatch and sparse. I could not boot or run a hibernate cycle on real hardware here, so that still wants doing before anyone gets excited.

Quoted 9 lines

You can easily test hibernate under QEMU/mach-virt: just create a file on the host, mkswap it and pass it to the VM as a virtio drive (e.g., /dev/vda)
You can then swapon it in the VM and do 'echo disk >/sys/power/state', which
will perform the hibernate and exit QEMU. If you then restart it with 'resume=/dev/vda' on the command line, it will perform the resume from hibernate at boot.

Ack. Tried on PKVM last night.

Quoted 6 lines

I've had a go with your series on a NV capable system (Mac Mini M2). Without
the series applied, I can hibernate and resume 4 vCPUs without issue, so it
is not entirely clear to me what your series intends to fix.

Quoted 7 lines

With the series applied resume from hibernate still works - I did see one intermittent failure where the secondaries failed to come up, but I only saw it once and so I cannot say for certain whether your series triggered it or not.

For the whole series?

Quoted 7 lines

--- Thanks!
https://lore.kernel.org/all/[email protected]/

reviewer
#14 lore raw
In reply to Bradley Morgan
Quoted 10 lines · Bradley Morgan

This comes out of the hibernate resume discussion where Will asked if the restored kernel's hyp vectors are reinstalled before we come back from swsusp_arch_suspend() and free the reserved pages. Rather than answer with a one off fix, this series reworks resume to look like a resume from idle, which makes the answer fall out on its own.

The series leans on Mark Rutland's old work for inspiration, he had the same end goal in a branch from back then:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/suspend/cleanup

Most of the series seems to be a rebased of those patches, just with paraphrased commit messages and my S-o-B lines dropped. As commented on patch 4, I do not think that is appropriate.

I'll take a more in-depth look shortly.

Catalin, Will, for the moment please do not pick these up.

Mark.

Quoted 36 lines

The short answer to the question: yes, but only because the rework makes resume look like a resume from idle. We enter cpu_resume() at the highest available EL with the MMU off, and the resumed kernel installs its own hyp stub vectors into VBAR_EL2 through init_kernel_el() before anything else runs, so by the time swsusp_free() releases the trans_pgd copies nothing points at them anymore.

The series is structured so the first patch stands alone as a stable worthy fix for the missing hvc arguments. The rest is the cleanup: the stale init_kernel_el() comment gets corrected, MTE tag pages get freed after save instead of leaking on failed hibernates, the fragile _cpu_resume() calling convention gets split, and the last patch reworks hibernate to use the regular cpu_resume() logic end to end.

Everything object compiles and passes checkpatch and sparse. I could not boot or run a hibernate cycle on real hardware here, so that still wants doing before anyone gets excited.

Bradley Morgan (7):
arm64: hibernate: pass HVC_SET_VECTORS args to the resume hvc
arm64: head: correct comment for init_kernel_el()
arm64: hibernate: free MTE tag pages after saving
arm64: hibernate: fix _cpu_resume() calling convention
arm64: hibernate: use dcache_by_myline_op
arm64: hibernate: only clean to PoC
arm64: hibernate: use regular cpu_resume() logic

 arch/arm64/include/asm/suspend.h  |  1 -
 arch/arm64/kernel/head.S          |  6 +--
 arch/arm64/kernel/hibernate-asm.S | 87 +++++++++++++++++++------------
 arch/arm64/kernel/hibernate.c     | 66 +++++++++++------------
 arch/arm64/kernel/sleep.S         |  6 +--
 5 files changed, 88 insertions(+), 78 deletions(-)
Signature · 1 line
2.47.3
#15 lore raw
Quoted 11 lines · Mark Rutland
Quoted 10 lines · Bradley Morgan

This comes out of the hibernate resume discussion where Will asked if the restored kernel's hyp vectors are reinstalled before we come back from swsusp_arch_suspend() and free the reserved pages. Rather than answer with a one off fix, this series reworks resume to look like a resume from idle, which makes the answer fall out on its own.

The series leans on Mark Rutland's old work for inspiration, he had the same end goal in a branch from back then:

Quoted 5 lines

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/suspend/cleanup

Most of the series seems to be a rebased of those patches, just with paraphrased commit messages and my S-o-B lines dropped. As commented on patch 4, I do not think that is appropriate.

Same question as patch 2

If you want your S-O-B added, ok fine, but yeah.

Do you think this series itself is good?

Quoted 2 lines

I'll take a more in-depth look shortly.

thanks!

Quoted 42 lines

Catalin, Will, for the moment please do not pick these up.

Mark.

Quoted 2 lines

The short answer to the question: yes, but only because the rework makes resume look like a resume from idle. We enter cpu_resume() at the

highest

Quoted 34 lines

available EL with the MMU off, and the resumed kernel installs its own hyp stub vectors into VBAR_EL2 through init_kernel_el() before anything else runs, so by the time swsusp_free() releases the trans_pgd copies nothing points at them anymore.

The series is structured so the first patch stands alone as a stable worthy fix for the missing hvc arguments. The rest is the cleanup: the stale init_kernel_el() comment gets corrected, MTE tag pages get freed after save instead of leaking on failed hibernates, the fragile _cpu_resume() calling convention gets split, and the last patch reworks hibernate to use the regular cpu_resume() logic end to end.

Everything object compiles and passes checkpatch and sparse. I could not boot or run a hibernate cycle on real hardware here, so that still wants doing before anyone gets excited.

Bradley Morgan (7):
arm64: hibernate: pass HVC_SET_VECTORS args to the resume hvc
arm64: head: correct comment for init_kernel_el()
arm64: hibernate: free MTE tag pages after saving
arm64: hibernate: fix _cpu_resume() calling convention
arm64: hibernate: use dcache_by_myline_op
arm64: hibernate: only clean to PoC
arm64: hibernate: use regular cpu_resume() logic

 arch/arm64/include/asm/suspend.h  |  1 -
 arch/arm64/kernel/head.S          |  6 +--
 arch/arm64/kernel/hibernate-asm.S | 87 +++++++++++++++++++------------
 arch/arm64/kernel/hibernate.c     | 66 +++++++++++------------
 arch/arm64/kernel/sleep.S         |  6 +--
 5 files changed, 88 insertions(+), 78 deletions(-)
Signature · 1 line
2.47.3

--- Thanks!
https://lore.kernel.org/all/[email protected]/

reviewer
#16 lore raw
Quoted 15 lines · Bradley Morgan
Quoted 13 lines · Mark Rutland
Quoted 10 lines · Bradley Morgan

This comes out of the hibernate resume discussion where Will asked if the restored kernel's hyp vectors are reinstalled before we come back from swsusp_arch_suspend() and free the reserved pages. Rather than answer with a one off fix, this series reworks resume to look like a resume from idle, which makes the answer fall out on its own.

The series leans on Mark Rutland's old work for inspiration, he had the same end goal in a branch from back then:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/suspend/cleanup

Quoted 7 lines
Quoted 3 lines

Most of the series seems to be a rebased of those patches, just with paraphrased commit messages and my S-o-B lines dropped. As commented on patch 4, I do not think that is appropriate.

Same question as patch 2

If you want your S-O-B added, ok fine, but yeah.

As on patch 2, please read the documentation before continuing futher.

Quoted 1 line

Do you think this series itself is good?

No. At present I think it's in a worse state than it was in my branch, because portions have changed, with no rationale and no clear benefit.

Hence my ask to Catalin and Will to NOT pick this up for now.

Mark.

Quoted 51 lines
Quoted 3 lines

I'll take a more in-depth look shortly.

thanks!

Quoted 42 lines

Catalin, Will, for the moment please do not pick these up.

Mark.

Quoted 2 lines

The short answer to the question: yes, but only because the rework makes resume look like a resume from idle. We enter cpu_resume() at the

highest

Quoted 34 lines

available EL with the MMU off, and the resumed kernel installs its own hyp stub vectors into VBAR_EL2 through init_kernel_el() before anything else runs, so by the time swsusp_free() releases the trans_pgd copies nothing points at them anymore.

The series is structured so the first patch stands alone as a stable worthy fix for the missing hvc arguments. The rest is the cleanup: the stale init_kernel_el() comment gets corrected, MTE tag pages get freed after save instead of leaking on failed hibernates, the fragile _cpu_resume() calling convention gets split, and the last patch reworks hibernate to use the regular cpu_resume() logic end to end.

Everything object compiles and passes checkpatch and sparse. I could not boot or run a hibernate cycle on real hardware here, so that still wants doing before anyone gets excited.

Bradley Morgan (7):
arm64: hibernate: pass HVC_SET_VECTORS args to the resume hvc
arm64: head: correct comment for init_kernel_el()
arm64: hibernate: free MTE tag pages after saving
arm64: hibernate: fix _cpu_resume() calling convention
arm64: hibernate: use dcache_by_myline_op
arm64: hibernate: only clean to PoC
arm64: hibernate: use regular cpu_resume() logic

 arch/arm64/include/asm/suspend.h  |  1 -
 arch/arm64/kernel/head.S          |  6 +--
 arch/arm64/kernel/hibernate-asm.S | 87 +++++++++++++++++++------------
 arch/arm64/kernel/hibernate.c     | 66 +++++++++++------------
 arch/arm64/kernel/sleep.S         |  6 +--
 5 files changed, 88 insertions(+), 78 deletions(-)
Signature · 1 line
2.47.3

--- Thanks!
https://lore.kernel.org/all/[email protected]/

#17 lore raw
Quoted 22 lines · Mark Rutland
Quoted 10 lines · Bradley Morgan
Quoted 8 lines · Mark Rutland
Quoted 7 lines · Bradley Morgan

This comes out of the hibernate resume discussion where Will asked if the restored kernel's hyp vectors are reinstalled before we come back from swsusp_arch_suspend() and free the reserved pages. Rather than answer with a one off fix, this series reworks resume to look like a resume from idle, which makes the answer fall out on its own.

The series leans on Mark Rutland's old work for inspiration, he had

the

Quoted 10 lines
Quoted 3 lines
Quoted 3 lines

same end goal in a branch from back then:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/suspend/cleanup

Quoted 4 lines

Most of the series seems to be a rebased of those patches, just with paraphrased commit messages and my S-o-B lines dropped. As commented on patch 4, I do not think that is appropriate.

Quoted 6 lines
Quoted 3 lines

Same question as patch 2

If you want your S-O-B added, ok fine, but yeah.

As on patch 2, please read the documentation before continuing futher.

Ack.

Quoted 4 lines
Quoted 1 line

Do you think this series itself is good?

No. At present I think it's in a worse state than it was in my branch, because portions have changed, with no rationale and no clear benefit.

Sorry. I'm sure the points you will bring up on review will go into V2, andthen it should be better then.

Quoted 60 lines

Hence my ask to Catalin and Will to NOT pick this up for now.

Mark.

Quoted 12 lines
Quoted 3 lines

I'll take a more in-depth look shortly.

thanks!

Quoted 6 lines

Catalin, Will, for the moment please do not pick these up.

Mark.

Quoted 1 line

The short answer to the question: yes, but only because the rework

makes

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

resume look like a resume from idle. We enter cpu_resume() at the

highest

Quoted 1 line

available EL with the MMU off, and the resumed kernel installs its

own

Quoted 1 line
Quoted 1 line
Quoted 1 line

hyp stub vectors into VBAR_EL2 through init_kernel_el() before

anything

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

else runs, so by the time swsusp_free() releases the trans_pgd copies nothing points at them anymore.

The series is structured so the first patch stands alone as a stable worthy fix for the missing hvc arguments. The rest is the cleanup:

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

stale init_kernel_el() comment gets corrected, MTE tag pages get

freed

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

after save instead of leaking on failed hibernates, the fragile _cpu_resume() calling convention gets split, and the last patch

reworks

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

hibernate to use the regular cpu_resume() logic end to end.

Everything object compiles and passes checkpatch and sparse. I could

not

Quoted 1 line
Quoted 1 line
Quoted 1 line

boot or run a hibernate cycle on real hardware here, so that still

wants

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

doing before anyone gets excited.

Bradley Morgan (7):
arm64: hibernate: pass HVC_SET_VECTORS args to the resume hvc
arm64: head: correct comment for init_kernel_el()
arm64: hibernate: free MTE tag pages after saving
arm64: hibernate: fix _cpu_resume() calling convention
arm64: hibernate: use dcache_by_myline_op
arm64: hibernate: only clean to PoC
arm64: hibernate: use regular cpu_resume() logic

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
 arch/arm64/include/asm/suspend.h  |  1 -
 arch/arm64/kernel/head.S          |  6 +--
 arch/arm64/kernel/hibernate-asm.S | 87

+++++++++++++++++++------------

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
 arch/arm64/kernel/hibernate.c     | 66 +++++++++++------------
 arch/arm64/kernel/sleep.S         |  6 +--
 5 files changed, 88 insertions(+), 78 deletions(-)
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Signature · 1 line
2.47.3

--- Thanks!

https://lore.kernel.org/all/[email protected]/

--- Thanks!
https://lore.kernel.org/all/[email protected]/

maintainer
#18 lore raw
In reply to Mark Rutland
Quoted 24 lines · Mark Rutland
Quoted 15 lines · Bradley Morgan
Quoted 13 lines · Mark Rutland
Quoted 10 lines · Bradley Morgan

This comes out of the hibernate resume discussion where Will asked if the restored kernel's hyp vectors are reinstalled before we come back from swsusp_arch_suspend() and free the reserved pages. Rather than answer with a one off fix, this series reworks resume to look like a resume from idle, which makes the answer fall out on its own.

The series leans on Mark Rutland's old work for inspiration, he had the same end goal in a branch from back then:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/suspend/cleanup

Quoted 7 lines
Quoted 3 lines

Most of the series seems to be a rebased of those patches, just with paraphrased commit messages and my S-o-B lines dropped. As commented on patch 4, I do not think that is appropriate.

Same question as patch 2

If you want your S-O-B added, ok fine, but yeah.

Quoted 8 lines

As on patch 2, please read the documentation before continuing futher.

Quoted 1 line

Do you think this series itself is good?

No. At present I think it's in a worse state than it was in my branch, because portions have changed, with no rationale and no clear benefit.

Hence my ask to Catalin and Will to NOT pick this up for now.

I wasn't going to pick them up with the incorrect authorship, but it's worth noting that Bradley's patch _does_ fix one of the hibernation problems and has been tested by Vladimir:

https://lore.kernel.org/all/[email protected]/

I would prefer to take your complete set of patches, but if you don't want that then should I just take the above for now? Prior to this series, there has been no activity for nearly two years on your branch, so I'd prefer to get _something_ in, even if it only fixes some of the problems.

Will

#19 lore raw
Quoted 44 lines · Will Deacon
Quoted 5 lines · Mark Rutland
Quoted 4 lines · Bradley Morgan
Quoted 2 lines · Mark Rutland
Quoted 1 line · Bradley Morgan

This comes out of the hibernate resume discussion where Will asked

if

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

the restored kernel's hyp vectors are reinstalled before we come

back

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

from swsusp_arch_suspend() and free the reserved pages. Rather than answer with a one off fix, this series reworks resume to look like

a

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

resume from idle, which makes the answer fall out on its own.

The series leans on Mark Rutland's old work for inspiration, he had

the

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

same end goal in a branch from back then:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/suspend/cleanup

Quoted 3 lines
Quoted 3 lines

Most of the series seems to be a rebased of those patches, just with paraphrased commit messages and my S-o-B lines dropped. As commented

on

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

patch 4, I do not think that is appropriate.

Same question as patch 2

If you want your S-O-B added, ok fine, but yeah.

Quoted 8 lines

As on patch 2, please read the documentation before continuing futher.

Quoted 1 line

Do you think this series itself is good?

No. At present I think it's in a worse state than it was in my branch, because portions have changed, with no rationale and no clear benefit.

Hence my ask to Catalin and Will to NOT pick this up for now.

I wasn't going to pick them up with the incorrect authorship, but it's worth noting that Bradley's patch _does_ fix one of the hibernation problems and has been tested by Vladimir:

Quoted 8 lines

https://lore.kernel.org/all/[email protected]/

I would prefer to take your complete set of patches, but if you don't want that then should I just take the above for now? Prior to this series, there has been no activity for nearly two years on your branch, so I'd prefer to get _something_ in, even if it only fixes some of the problems.

So your taking the bugfix? I'll do V2 later, you'd need to review my sleep patch tho.

Quoted 1 line

Will

--- Thanks!
https://lore.kernel.org/all/[email protected]/

maintainer
#20 lore raw
Quoted 54 lines · Bradley Morgan
Quoted 44 lines · Will Deacon
Quoted 5 lines · Mark Rutland
Quoted 4 lines · Bradley Morgan
Quoted 2 lines · Mark Rutland
Quoted 1 line · Bradley Morgan

This comes out of the hibernate resume discussion where Will asked

if

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

the restored kernel's hyp vectors are reinstalled before we come

back

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

from swsusp_arch_suspend() and free the reserved pages. Rather than answer with a one off fix, this series reworks resume to look like

a

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

resume from idle, which makes the answer fall out on its own.

The series leans on Mark Rutland's old work for inspiration, he had

the

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

same end goal in a branch from back then:

https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/suspend/cleanup

Quoted 3 lines
Quoted 3 lines

Most of the series seems to be a rebased of those patches, just with paraphrased commit messages and my S-o-B lines dropped. As commented

on

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

patch 4, I do not think that is appropriate.

Same question as patch 2

If you want your S-O-B added, ok fine, but yeah.

Quoted 8 lines

As on patch 2, please read the documentation before continuing futher.

Quoted 1 line

Do you think this series itself is good?

No. At present I think it's in a worse state than it was in my branch, because portions have changed, with no rationale and no clear benefit.

Hence my ask to Catalin and Will to NOT pick this up for now.

I wasn't going to pick them up with the incorrect authorship, but it's worth noting that Bradley's patch _does_ fix one of the hibernation problems and has been tested by Vladimir:

Quoted 8 lines

https://lore.kernel.org/all/[email protected]/

I would prefer to take your complete set of patches, but if you don't want that then should I just take the above for now? Prior to this series, there has been no activity for nearly two years on your branch, so I'd prefer to get _something_ in, even if it only fixes some of the problems.

Quoted 2 lines

So your taking the bugfix? I'll do V2 later, you'd need to review my sleep patch tho.

I'm asking Mark how he would like to proceed, so no need to send a v2 just yet.

Will

A message that is not in the archive (sent off-list or not yet received): aplRx8f5NprO9dSU@willie-the-truck_quarantine

reviewer Acked-by
#22 lore raw

Hi Will,

Quoted 10 lines · Will Deacon

it's worth noting that Bradley's patch _does_ fix one of the hibernation problems and has been tested by Vladimir:

https://lore.kernel.org/all/[email protected]/

I would prefer to take your complete set of patches, but if you don't want that then should I just take the above for now? Prior to this series, there has been no activity for nearly two years on your branch, so I'd prefer to get _something_ in, even if it only fixes some of the problems.

Sorry for the delay.

I'd prefer if you picked up Bradley's single patch for now. I'll follow up with a respin of the rest of the cleanup and fixes shortly.

Could you add a note to the commit message as to how Vladimir was able to trigger a hang with CONFIG_PAGE_POISONING=y and page_poison=on? It's helpful for anyone testing a backport, etc.

With that:

For context, when I originally wrote my patches, I hadn't managed to trigger failures at top level without additional hacks, and so this hadn't been a high priority. Given that Vladimir can trigger this specific issue with existing options, it makes sense to pick that up.

I had a vague recollection that this specific fix lead to a secondary issue, but digging through my notes I can't see what that would have been, and I suspect I had just gotten confused w.r.t. what KVM was doing. If I find there was a real secondary issue I'll follow up with a fix .

Thanks,
Mark.

#23 lore raw
Quoted 14 lines · Mark Rutland

Hi Will,

Quoted 10 lines · Will Deacon

it's worth noting that Bradley's patch _does_ fix one of the hibernation problems and has been tested by Vladimir:

https://lore.kernel.org/all/[email protected]/

I would prefer to take your complete set of patches, but if you don't want that then should I just take the above for now? Prior to this series, there has been no activity for nearly two years on your branch, so I'd prefer to get _something_ in, even if it only fixes some of the problems.

Quoted 4 lines

Sorry for the delay.

I'd prefer if you picked up Bradley's single patch for now. I'll follow up with a respin of the rest of the cleanup and fixes shortly.

Oh, so I didn't need to do all that? Urgh..

Quoted 14 lines

Could you add a note to the commit message as to how Vladimir was able to trigger a hang with CONFIG_PAGE_POISONING=y and page_poison=on? It's helpful for anyone testing a backport, etc.

With that:

For context, when I originally wrote my patches, I hadn't managed to trigger failures at top level without additional hacks, and so this hadn't been a high priority. Given that Vladimir can trigger this specific issue with existing options, it makes sense to pick that up.

Quoted 8 lines

I had a vague recollection that this specific fix lead to a secondary issue, but digging through my notes I can't see what that would have been, and I suspect I had just gotten confused w.r.t. what KVM was doing. If I find there was a real secondary issue I'll follow up with a fix .

Thanks,
Mark.

--- Thanks!
https://lore.kernel.org/all/[email protected]/