arm64: kernel: Add support for hibernate/suspend-to-disk

From: Geoff Levand <[email protected]>

To allow the assembler macros defined in arch/arm64/mm/proc-macros.S to be used outside the mm code move the contents of proc-macros.S to asm/assembler.h. Also, delete proc-macros.S, and fix up all references to proc-macros.S.

Signed-off-by Geoff Levand <[email protected]>

Signed-off-by James Morse <[email protected]>

---
 arch/arm64/include/asm/assembler.h | 48 +++++++++++++++++++++++++++-
 arch/arm64/kernel/head.S           |  1 -
 arch/arm64/kvm/hyp-init.S          |  1 -
 arch/arm64/mm/cache.S              |  2 --
 arch/arm64/mm/proc-macros.S        | 64 --------------------------------------
 arch/arm64/mm/proc.S               |  3 --
 6 files changed, 47 insertions(+), 72 deletions(-)
 delete mode 100644 arch/arm64/mm/proc-macros.S

arch/arm64/include/asm/assembler.h +0 -0

index b51f2cc22ca9..91cb311d33de 100644

Quoted 1 line · James Morse

From: Geoff Levand <[email protected]>

Why the two stars?

Some architectures require code written to memory as if it were data to be 'cleaned' from any data caches so that the processor can fetch them as new instructions.

During resume from hibernate, the snapshot code copies some pages directly, meaning these architectures do not get a chance to perform their cache maintenance. Add a call to flush_icache_range(), which is provided by architectures that require it, to perform the maintenance.

This mirrors the kernel's behaviour when loading kernel modules and when mapping executable pages to user space.

Signed-off-by James Morse <[email protected]>

---
 kernel/power/snapshot.c | 4 ++++
 1 file changed, 4 insertions(+)

kernel/power/snapshot.c +4 -0

index 5235dd4e1e2f..139fc449ad75 100644
@@ -31,6 +31,7 @@
 #include <linux/ktime.h>
 
 #include <asm/uaccess.h>
+#include <asm/cacheflush.h>
 #include <asm/mmu_context.h>
 #include <asm/pgtable.h>
 #include <asm/tlbflush.h>

@@ -1196,9 +1197,12 @@ static unsigned int count_data_pages(void)
 static inline void do_copy_page(long *dst, long *src)
 {
 	int n;
+	unsigned long __maybe_unused start = (unsigned long)dst;
 
 	for (n = PAGE_SIZE / sizeof(long); n; n--)
 		*dst++ = *src++;
+
+	flush_icache_range(start, start+PAGE_SIZE);
 }
 
 

Signature · 1 line

2.1.4

Hi Pavel, Rafael,

Do you have any feedback on this patch ?

It is fundamental to this series and affects Hibernate core code so if you have any feedback that would be much appreciated.

Quoted 32 lines

Some architectures require code written to memory as if it were data to be 'cleaned' from any data caches so that the processor can fetch them as new instructions.

During resume from hibernate, the snapshot code copies some pages directly, meaning these architectures do not get a chance to perform their cache maintenance. Add a call to flush_icache_range(), which is provided by architectures that require it, to perform the maintenance.

This mirrors the kernel's behaviour when loading kernel modules and when mapping executable pages to user space.

Signed-off-by James Morse <[email protected]>

---
 kernel/power/snapshot.c | 4 ++++
 1 file changed, 4 insertions(+)

kernel/power/snapshot.c +1 -0

index 5235dd4e1e2f..139fc449ad75 100644
@@ -31,6 +31,7 @@
 #include <linux/ktime.h>
 
 #include <asm/uaccess.h>
+#include <asm/cacheflush.h>
 #include <asm/mmu_context.h>
 #include <asm/pgtable.h>
 #include <asm/tlbflush.h>

@@ -1196,9 +1197,12 @@ static unsigned int count_data_pages(void)
 static inline void do_copy_page(long *dst, long *src)
 {

kernel/power/snapshot.c static unsigned int count_data_pages(void) · line 1208

Quoted 10 lines

kernel/power/snapshot.c +3 -0

 	int n;
+	unsigned long __maybe_unused start = (unsigned long)dst;
 
 	for (n = PAGE_SIZE / sizeof(long); n; n--)
 		*dst++ = *src++;
+
+	flush_icache_range(start, start+PAGE_SIZE);
 }
 
 

--
To unsubscribe from this list: send the line "unsubscribe linux-pm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 6 lines · Lorenzo Pieralisi

Hi Pavel, Rafael,

Do you have any feedback on this patch ?

It is fundamental to this series and affects Hibernate core code so if you have any feedback that would be much appreciated.

I'm really not familiar with the flush_icache_range() interface.

What exactly does it do?

Rafael

Quoted 47 lines · James Morse

Quoted 32 lines

Some architectures require code written to memory as if it were data to be 'cleaned' from any data caches so that the processor can fetch them as new instructions.

During resume from hibernate, the snapshot code copies some pages directly, meaning these architectures do not get a chance to perform their cache maintenance. Add a call to flush_icache_range(), which is provided by architectures that require it, to perform the maintenance.

This mirrors the kernel's behaviour when loading kernel modules and when mapping executable pages to user space.

Signed-off-by James Morse <[email protected]>

---
 kernel/power/snapshot.c | 4 ++++
 1 file changed, 4 insertions(+)

kernel/power/snapshot.c +1 -0

index 5235dd4e1e2f..139fc449ad75 100644
@@ -31,6 +31,7 @@
 #include <linux/ktime.h>
 
 #include <asm/uaccess.h>
+#include <asm/cacheflush.h>
 #include <asm/mmu_context.h>
 #include <asm/pgtable.h>
 #include <asm/tlbflush.h>

@@ -1196,9 +1197,12 @@ static unsigned int count_data_pages(void)
 static inline void do_copy_page(long *dst, long *src)
 {

kernel/power/snapshot.c static unsigned int count_data_pages(void) · line 1208

Quoted 10 lines

kernel/power/snapshot.c +3 -0

 	int n;
+	unsigned long __maybe_unused start = (unsigned long)dst;
 
 	for (n = PAGE_SIZE / sizeof(long); n; n--)
 		*dst++ = *src++;
+
+	flush_icache_range(start, start+PAGE_SIZE);
 }
 
 

--
To unsubscribe from this list: send the line "unsubscribe linux-pm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines

I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

Quoted 11 lines · Rafael J. Wysocki

Quoted 6 lines · Lorenzo Pieralisi

Hi Pavel, Rafael,

Do you have any feedback on this patch ?

It is fundamental to this series and affects Hibernate core code so if you have any feedback that would be much appreciated.

I'm really not familiar with the flush_icache_range() interface.

What exactly does it do?

It is used to sync a memory range that is written into (eg loading modules, copying from snapshot is basically the same thing, reads from storage and restore pages that might well be executable code), in particular to sync the I-cache and the D-cache, eg on arm64 the page that the snapshot code is copying might be executable code that has to be cleaned from the D-cache so that it is made visible to the I-cache.

On x86 it is a NOP AFAIK.

Quoted 20 lines · Lorenzo Pieralisi

Quoted 11 lines · Rafael J. Wysocki

Quoted 6 lines · Lorenzo Pieralisi

Hi Pavel, Rafael,

Do you have any feedback on this patch ?

It is fundamental to this series and affects Hibernate core code so if you have any feedback that would be much appreciated.

I'm really not familiar with the flush_icache_range() interface.

What exactly does it do?

It is used to sync a memory range that is written into (eg loading modules, copying from snapshot is basically the same thing, reads from storage and restore pages that might well be executable code), in particular to sync the I-cache and the D-cache, eg on arm64 the page that the snapshot code is copying might be executable code that has to be cleaned from the D-cache so that it is made visible to the I-cache.

Quoted 1 line

On x86 it is a NOP AFAIK.

If that's the case, I have no problems with this change as long as the code works on architectures with non-trivial flush_icache_range().

[Cc'ed maintainers of affected arches]

Quoted 21 lines · Rafael J. Wysocki

Quoted 20 lines · Lorenzo Pieralisi

Quoted 11 lines · Rafael J. Wysocki

Quoted 6 lines · Lorenzo Pieralisi

Hi Pavel, Rafael,

Do you have any feedback on this patch ?

It is fundamental to this series and affects Hibernate core code so if you have any feedback that would be much appreciated.

I'm really not familiar with the flush_icache_range() interface.

What exactly does it do?

It is used to sync a memory range that is written into (eg loading modules, copying from snapshot is basically the same thing, reads from storage and restore pages that might well be executable code), in particular to sync the I-cache and the D-cache, eg on arm64 the page that the snapshot code is copying might be executable code that has to be cleaned from the D-cache so that it is made visible to the I-cache.

Quoted 4 lines

Quoted 1 line

On x86 it is a NOP AFAIK.

If that's the case, I have no problems with this change as long as the code works on architectures with non-trivial flush_icache_range().

I Cc'ed the respective arches maintainers, it should work (it may make resuming a bit slower, owing to the cache syncing), problem is that we have no way of testing it on platforms other than arm/arm64.

How do you want us to go on about this ? Should we add a config option to prevent calling flush_icache_range() on all platforms (where it is not a nop) ?

Quoted 26 lines · Lorenzo Pieralisi

[Cc'ed maintainers of affected arches]

Quoted 23 lines · Rafael J. Wysocki

Quoted 21 lines · Lorenzo Pieralisi

Quoted 11 lines · Rafael J. Wysocki

Quoted 6 lines · Lorenzo Pieralisi

Hi Pavel, Rafael,

Do you have any feedback on this patch ?

It is fundamental to this series and affects Hibernate core code so if you have any feedback that would be much appreciated.

I'm really not familiar with the flush_icache_range() interface.

What exactly does it do?

It is used to sync a memory range that is written into (eg loading modules, copying from snapshot is basically the same thing, reads from storage and restore pages that might well be executable code), in particular to sync the I-cache and the D-cache, eg on arm64 the page that the snapshot code is copying might be executable code that has to be cleaned from the D-cache so that it is made visible to the I-cache.

On x86 it is a NOP AFAIK.

Quoted 6 lines

Quoted 2 lines

If that's the case, I have no problems with this change as long as the code works on architectures with non-trivial flush_icache_range().

I Cc'ed the respective arches maintainers, it should work (it may make resuming a bit slower, owing to the cache syncing), problem is that we have no way of testing it on platforms other than arm/arm64.

Sure you can find x86 and x86-64 machine near you?

And as hibernation is only supported on x86 and arm, that should be ok.

Or just merge it to the next and let the world do testing for you...

Quoted 3 lines

How do you want us to go on about this ? Should we add a config option to prevent calling flush_icache_range() on all platforms (where it is not a nop) ?

Config option is definitely not an option ;-).

-----Original Message-----
From: linux-pm-owner at vger.kernel.org [mailto:linux-pm-
owner at vger.kernel.org] On Behalf Of Lorenzo Pieralisi
Sent: Wednesday, November 11, 2015 7:41 PM
To: James Morse; Rafael J. Wysocki; Pavel Machek
Cc: linux-arm-kernel at lists.infradead.org; linux-pm at vger.kernel.org; Will Deacon; Sudeep Holla; Kevin Kang; Geoff Levand; Catalin Marinas; Mark Rutland; AKASHI Takahiro; wangfei; Marc Zyngier
Subject: Re: [PATCH v2 10/11] PM / Hibernate: clean cached pages on architectures that require it

Hi Pavel, Rafael,

Do you have any feedback on this patch ?

It is fundamental to this series and affects Hibernate core code so if you have any feedback that would be much appreciated.

Thanks a lot,
Lorenzo

Quoted 34 lines · James Morse

Some architectures require code written to memory as if it were data to be 'cleaned' from any data caches so that the processor can fetch them as new instructions.

During resume from hibernate, the snapshot code copies some pages directly, meaning these architectures do not get a chance to perform their cache maintenance. Add a call to flush_icache_range(), which is provided by architectures that require it, to perform the maintenance.

This mirrors the kernel's behaviour when loading kernel modules and when mapping executable pages to user space.

Signed-off-by James Morse <[email protected]>

---
 kernel/power/snapshot.c | 4 ++++
 1 file changed, 4 insertions(+)

kernel/power/snapshot.c index renamed from kernel/power/snapshot.c +0 -0

5235dd4e1e2f..139fc449ad75 100644

kernel/power/snapshot.c +2 -0

@@ -31,6 +31,7 @@
 #include <linux/ktime.h>
 
 #include <asm/uaccess.h>
+#include <asm/cacheflush.h>
 #include <asm/mmu_context.h>
 #include <asm/pgtable.h>
 #include <asm/tlbflush.h>

@@ -1196,9 +1197,12 @@ static unsigned int count_data_pages(void)
 static inline void do_copy_page(long *dst, long *src)  {
 	int n;
+	unsigned long __maybe_unused start = (unsigned long)dst;
 

kernel/power/snapshot.c static unsigned int count_data_pages(void) · line 1206

Quoted 5 lines

kernel/power/snapshot.c +2 -0

 	for (n = PAGE_SIZE / sizeof(long); n; n--)
 		*dst++ = *src++;
+
+	flush_icache_range(start, start+PAGE_SIZE);
 }

Quoted 4 lines

Quoted 4 lines

static inline void do_copy_page(long *dst, long *src) {

kernel/power/snapshot.c +1 -0

 	int n;
+	unsigned long __maybe_unused start = (unsigned long)dst;

Quoted 5 lines

kernel/power/snapshot.c static unsigned int count_data_pages(void) · line 1206

Quoted 5 lines

kernel/power/snapshot.c +2 -0

 	for (n = PAGE_SIZE / sizeof(long); n; n--)
 		*dst++ = *src++;
+
+	flush_icache_range(start, start+PAGE_SIZE);
 }

How about invalid all icache lines before doing do_copy_page, since do_copy_page might deal with both data pages and execute pages, that might be redundant to do it for every page?

Some architectures require code written to memory as if it were data to be 'cleaned' from any data caches so that the processor can fetch them as new instructions.

During resume from hibernate, the snapshot code copies some pages directly, meaning these architectures do not get a chance to perform their cache maintenance. Add a call to flush_icache_range(), which is provided by architectures that require it, to perform the maintenance.

This mirrors the kernel's behaviour when loading kernel modules and when mapping executable pages to user space.

Signed-off-by James Morse <[email protected]>

Quoted 9 lines · James Morse

Some architectures require code written to memory as if it were data to be 'cleaned' from any data caches so that the processor can fetch them as new instructions.

During resume from hibernate, the snapshot code copies some pages directly, meaning these architectures do not get a chance to perform their cache maintenance. Add a call to flush_icache_range(), which is provided by architectures that require it, to perform the maintenance.

Quoted 4 lines

This mirrors the kernel's behaviour when loading kernel modules and when mapping executable pages to user space.

Signed-off-by James Morse <[email protected]>

Looks reasonable.

Acked-by Pavel Machek <[email protected]>

Quoted 5 lines

---
 kernel/power/snapshot.c | 4 ++++
 1 file changed, 4 insertions(+)

kernel/power/snapshot.c static unsigned int count_data_pages(void) · line 1200

Quoted 9 lines

kernel/power/snapshot.c +1 -0

index 5235dd4e1e2f..139fc449ad75 100644
@@ -1196,9 +1197,12 @@ static unsigned int count_data_pages(void)
 static inline void do_copy_page(long *dst, long *src)
 {
 	int n;
+	unsigned long __maybe_unused start = (unsigned long)dst;

Why the "maybe unused"?

../kernel/power/snapshot.c: In function ?do_copy_page?:
../kernel/power/snapshot.c:1200:16: warning: unused variable ?start? [-Wunused-variable]
unsigned long start = (unsigned long)dst;

kernel/power/snapshot.c static unsigned int count_data_pages(void) · line 1206

Quoted 5 lines

kernel/power/snapshot.c +2 -0

 	for (n = PAGE_SIZE / sizeof(long); n; n--)
 		*dst++ = *src++;
+
+	flush_icache_range(start, start+PAGE_SIZE);
 }

Quoted 10 lines · Pavel Machek

Quoted 9 lines · James Morse

Some architectures require code written to memory as if it were data to be 'cleaned' from any data caches so that the processor can fetch them as new instructions.

During resume from hibernate, the snapshot code copies some pages directly, meaning these architectures do not get a chance to perform their cache maintenance. Add a call to flush_icache_range(), which is provided by architectures that require it, to perform the maintenance.

Quoted 8 lines

Quoted 4 lines

This mirrors the kernel's behaviour when loading kernel modules and when mapping executable pages to user space.

Signed-off-by James Morse <[email protected]>

Looks reasonable.

Acked-by Pavel Machek <[email protected]>

Thanks!

Quoted 5 lines

Quoted 5 lines

---
 kernel/power/snapshot.c | 4 ++++
 1 file changed, 4 insertions(+)

Quoted 11 lines

kernel/power/snapshot.c static unsigned int count_data_pages(void) · line 1200

Quoted 9 lines

kernel/power/snapshot.c +1 -0

index 5235dd4e1e2f..139fc449ad75 100644
@@ -1196,9 +1197,12 @@ static unsigned int count_data_pages(void)
 static inline void do_copy_page(long *dst, long *src)
 {
 	int n;
+	unsigned long __maybe_unused start = (unsigned long)dst;

Why the "maybe unused"?

To avoid a build warning on x86_64, and any other architectures that don't use the arguments in their flush_icache_range() implementation.

Some architectures require code written to memory as if it were data to be 'cleaned' from any data caches so that the processor can fetch them as new instructions.

During resume from hibernate, the snapshot code copies some pages directly, meaning these architectures do not get a chance to perform their cache maintenance. Add a call to flush_icache_range(), which is provided by architectures that require it, to perform the maintenance.

This mirrors the kernel's behaviour when loading kernel modules and when mapping executable pages to user space.

The implementation assumes that exactly the same kernel is booted on the same hardware, and that the kernel is loaded at the same physical address.

Signed-off-by James Morse <[email protected]>

+/*
+ * Corrupt memory.
+ *

+ * Because this code has to be copied to a safe_page, it can't call out to
+ * other functions by pc-relative address. Also remember that it

+ * mid-way through over-writing other functions. For this reason it contains
+ * a copy of copy_page() and code from flush_icache_range().
+ *
+ * All of memory gets written to, including code. We need to clean the kernel
+ * text to the PoC before secondary cores can be booted. Because kernel modules,

+ * and executable pages mapped to user space are also written as data, we
+ * clean all pages we touch to the PoU.

Quoted 2 lines

The implementation assumes that exactly the same kernel is booted on the same hardware, and that the kernel is loaded at the same physical address.

BTW... on newer implementations (and I have patch for x86, too), we try to make it so that resume kernel does not have to be same as suspend one. It would be nice to move there with arm64, too.

Quoted 1 line

Signed-off-by James Morse <[email protected]>

Acked-by Pavel Machek <[email protected]>

Quoted 3 lines

+/*
+ * Corrupt memory.
+ *

Umm. Really?

Quoted 2 lines

+ * Because this code has to be copied to a safe_page, it can't call out to
+ * other functions by pc-relative address. Also remember that it

PC-relative?

Quoted 5 lines

+ * mid-way through over-writing other functions. For this reason it contains
+ * a copy of copy_page() and code from flush_icache_range().
+ *
+ * All of memory gets written to, including code. We need to clean the kernel
+ * text to the PoC before secondary cores can be booted. Because kernel modules,

"the kernel modules" and no ","?

Quoted 2 lines

+ * and executable pages mapped to user space are also written as data, we
+ * clean all pages we touch to the PoU.

What is PoC and PoU?

Quoted 6 lines

Quoted 2 lines

The implementation assumes that exactly the same kernel is booted on the same hardware, and that the kernel is loaded at the same physical address.

BTW... on newer implementations (and I have patch for x86, too), we try to make it so that resume kernel does not have to be same as suspend one. It would be nice to move there with arm64, too.

Yes, that is a neat trick, can I leave it as future work?

Quoted 4 lines

Quoted 2 lines

+ * Because this code has to be copied to a safe_page, it can't call out to
+ * other functions by pc-relative address. Also remember that it

PC-relative?

The linker may (often!) use program-counter relative addresses for loads and stores. This code gets copied, so the linker doesn't know where the code will be executed from, so any instructions using pc-relative addresses will get the wrong result, (if they reference something outside the function).

Quoted 4 lines

Quoted 2 lines

+ * and executable pages mapped to user space are also written as data, we
+ * clean all pages we touch to the PoU.

What is PoC and PoU?

They are points in the CPU's cache hierarchy:

ARM processors are of a 'modified Harvard' architecture, their paths to read instructions and data are different. The 'Point of Unification' is the first point in the cache hierarchy that is the same for both. On ARM, flush_icache_range() makes sure code written as data is pushed through any data caches to this point, and then evicts any stale copies in the instruction caches.

PoC is the 'Point of Coherency', it is the first point that is the same for all devices, (e.g. a cpu with caches turned on, and one with them off), it is normally main memory. The kernel text has to be pushed to this point, so that secondary cores, while running early-boot code with their MMU and caches turned off, don't get incorrect code/data from before resume.

I have resisted the urge to draw some ascii-art!

Quoted 9 lines · Pavel Machek

Quoted 6 lines

Quoted 2 lines

The implementation assumes that exactly the same kernel is booted on the same hardware, and that the kernel is loaded at the same physical address.

BTW... on newer implementations (and I have patch for x86, too), we try to make it so that resume kernel does not have to be same as suspend one. It would be nice to move there with arm64, too.

Yes, that is a neat trick, can I leave it as future work?

Yes. But it is really not hard.

Quoted 5 lines

Quoted 4 lines

Quoted 2 lines

+ * Because this code has to be copied to a safe_page, it can't call out to
+ * other functions by pc-relative address. Also remember that it

PC-relative?

Quoted 5 lines

The linker may (often!) use program-counter relative addresses for loads and stores. This code gets copied, so the linker doesn't know where the code will be executed from, so any instructions using pc-relative addresses will get the wrong result, (if they reference something outside the function).

I was wondering if it should be spelled "PC-relative", not "pc-relative" :-).

Quoted 20 lines

Quoted 4 lines

Quoted 2 lines

+ * and executable pages mapped to user space are also written as data, we
+ * clean all pages we touch to the PoU.

What is PoC and PoU?

They are points in the CPU's cache hierarchy:

ARM processors are of a 'modified Harvard' architecture, their paths to read instructions and data are different. The 'Point of Unification' is the first point in the cache hierarchy that is the same for both. On ARM, flush_icache_range() makes sure code written as data is pushed through any data caches to this point, and then evicts any stale copies in the instruction caches.

PoC is the 'Point of Coherency', it is the first point that is the same for all devices, (e.g. a cpu with caches turned on, and one with them off), it is normally main memory. The kernel text has to be pushed to this point, so that secondary cores, while running early-boot code with their MMU and caches turned off, don't get incorrect code/data from before resume.

Quoted 1 line

I have resisted the urge to draw some ascii-art!

That's ok, you just might want to replace PoI -> 'Point of Unification' and PoC -> 'Point of Coherency' in the comments. That should make googling easier for people not familiar with arm terminology.

Quoted 19 lines

Quoted 18 lines

Quoted 1 line

What is PoC and PoU?

They are points in the CPU's cache hierarchy:

ARM processors are of a 'modified Harvard' architecture, their paths to read instructions and data are different. The 'Point of Unification' is the first point in the cache hierarchy that is the same for both. On ARM, flush_icache_range() makes sure code written as data is pushed through any data caches to this point, and then evicts any stale copies in the instruction caches.

PoC is the 'Point of Coherency', it is the first point that is the same for all devices, (e.g. a cpu with caches turned on, and one with them off), it is normally main memory. The kernel text has to be pushed to this point, so that secondary cores, while running early-boot code with their MMU and caches turned off, don't get incorrect code/data from before resume.

I have resisted the urge to draw some ascii-art!

Quoted 4 lines

That's ok, you just might want to replace PoI -> 'Point of Unification' and PoC -> 'Point of Coherency' in the comments. That should make googling easier for people not familiar with arm terminology.

There aren't any other points under arch/arm64 that use the full expansion, but it can't hurt to include both.

Hi,

Quoted 10 lines · Pavel Machek

Quoted 9 lines · Pavel Machek

Quoted 6 lines

Quoted 2 lines

The implementation assumes that exactly the same kernel is booted on the same hardware, and that the kernel is loaded at the same physical address.

BTW... on newer implementations (and I have patch for x86, too), we try to make it so that resume kernel does not have to be same as suspend one. It would be nice to move there with arm64, too.

Yes, that is a neat trick, can I leave it as future work?

Quoted 1 line

Yes. But it is really not hard.

I think its harder than it looks:
It means the MMU has to be turned off, as two different kernels may not have used the same configuration for the MMU - and I don't think its safe to change while the MMU is running. There are also going to be complications with resetting the hypervisor/el2 configuration, which I need to spend more time thinking about (and probably ask for advice!).