Skip to contentSkip to conversation

s390x: pv: Fixes and improvements

PATCH Superseded Janosch Frank 3 participants 11 messages Sep 3, 2020 – Sep 7, 2020

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Janosch Frank

author maintainer
#0 lore raw

cover s390x: pv: Fixes and improvements

Using the destroy call instead of the export on a VM shutdown, we can clear out a protected guest much faster.

The 3f exception can in fact be triggered by userspace and therefore should not panic the whole system, but send a SIGSEGV to the culprit process.

Janosch Frank (2):
s390x: uv: Add destroy page call
s390x: Add 3f program exception handler

 arch/s390/include/asm/uv.h   |  7 +++++++
 arch/s390/kernel/pgm_check.S |  2 +-
 arch/s390/kernel/uv.c        | 21 +++++++++++++++++++++
 arch/s390/mm/fault.c         | 23 +++++++++++++++++++++++
 arch/s390/mm/gmap.c          |  2 +-
 5 files changed, 53 insertions(+), 2 deletions(-)
Signature · 1 line
2.25.1
author maintainer
#1 lore raw

PATCH 1/2 s390x: uv: Add destroy page call

We don't need to export pages if we destroy the VM configuration afterwards anyway. Instead we can destroy the page which will zero it and then make it accessible to the host.

Destroying is about twice as fast as the export.

---
 arch/s390/include/asm/uv.h |  7 +++++++
 arch/s390/kernel/uv.c      | 21 +++++++++++++++++++++
 arch/s390/mm/gmap.c        |  2 +-
 3 files changed, 29 insertions(+), 1 deletion(-)
arch/s390/include/asm/uv.h +7 -0
index cff4b4c99b75..0325fc0469b7 100644
@@ -33,6 +33,7 @@
 #define UVC_CMD_DESTROY_SEC_CPU		0x0121
 #define UVC_CMD_CONV_TO_SEC_STOR	0x0200
 #define UVC_CMD_CONV_FROM_SEC_STOR	0x0201
+#define UVC_CMD_DESTR_SEC_STOR		0x0202
 #define UVC_CMD_SET_SEC_CONF_PARAMS	0x0300
 #define UVC_CMD_UNPACK_IMG		0x0301
 #define UVC_CMD_VERIFY_IMG		0x0302
@@ -344,6 +345,7 @@ static inline int is_prot_virt_host(void)
 }
 
 int gmap_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb);
+int uv_destroy_page(unsigned long paddr);
 int uv_convert_from_secure(unsigned long paddr);
 int gmap_convert_to_secure(struct gmap *gmap, unsigned long gaddr);
 
@@ -354,6 +356,11 @@ void adjust_to_uv_max(unsigned long *vmax);
 static inline void setup_uv(void) {}
 static inline void adjust_to_uv_max(unsigned long *vmax) {}
 
+static inline int uv_destroy_page(unsigned long paddr)
+{
+	return 0;
+}
+
 static inline int uv_convert_from_secure(unsigned long paddr)
 {
 	return 0;
arch/s390/kernel/uv.c +21 -0
index c296e5c8dbf9..e07a5859f1ea 100644
@@ -118,6 +118,27 @@ static int uv_pin_shared(unsigned long paddr)
 	return 0;
 }
 
+/*
+ * Requests the Ultravisor to destroy a guest page and make it
+ * accessible to the host. The destroy clears the page instead of
+ * exporting.
+ *
+ * @paddr: Absolute host address of page to be destroyed
+ */
+int uv_destroy_page(unsigned long paddr)
+{
+	struct uv_cb_cfs uvcb = {
+		.header.cmd = UVC_CMD_DESTR_SEC_STOR,
+		.header.len = sizeof(uvcb),
+		.paddr = paddr
+	};
+
+	if (uv_call(0, (u64)&uvcb))
+		return -EINVAL;
+	return 0;
+}
+
+
 /*
  * Requests the Ultravisor to encrypt a guest page and make it
  * accessible to the host for paging (export).
arch/s390/mm/gmap.c +1 -1
index 373542ca1113..cfb0017f33a7 100644
@@ -2679,7 +2679,7 @@ static int __s390_reset_acc(pte_t *ptep, unsigned long addr,
 	pte_t pte = READ_ONCE(*ptep);
 
 	if (pte_present(pte))
-		WARN_ON_ONCE(uv_convert_from_secure(pte_val(pte) & PAGE_MASK));
+		WARN_ON_ONCE(uv_destroy_page(pte_val(pte) & PAGE_MASK));
 	return 0;
 }
 
Signature · 1 line
2.25.1
maintainer
#2 lore raw

Re: [PATCH 1/2] s390x: uv: Add destroy page call

Quoted 22 lines · Janosch Frank
arch/s390/kernel/uv.c +14 -0
+int uv_destroy_page(unsigned long paddr)
+{
+	struct uv_cb_cfs uvcb = {
+		.header.cmd = UVC_CMD_DESTR_SEC_STOR,
+		.header.len = sizeof(uvcb),
+		.paddr = paddr
+	};
+
+	if (uv_call(0, (u64)&uvcb))
+		return -EINVAL;
+	return 0;
+}
+
+
 /*
  * Requests the Ultravisor to encrypt a guest page and make it
  * accessible to the host for paging (export).
arch/s390/mm/gmap.c +0 -0
index 373542ca1113..cfb0017f33a7 100644

arch/s390/mm/gmap.c static int __s390_reset_acc(pte_t *ptep, unsigned long addr, · line 2682

Quoted 5 lines
arch/s390/mm/gmap.c +1 -1
@@ -2679,7 +2679,7 @@ static int __s390_reset_acc(pte_t *ptep, unsigned long addr,
 	pte_t pte = READ_ONCE(*ptep);
 
 	if (pte_present(pte))
-		WARN_ON_ONCE(uv_convert_from_secure(pte_val(pte) & PAGE_MASK));
+		WARN_ON_ONCE(uv_destroy_page(pte_val(pte) & PAGE_MASK));

Why not put the WARN_ON_ONCE() into uv_destroy_page() and make that function return void?

author maintainer
#3 lore raw

Re: [PATCH 1/2] s390x: uv: Add destroy page call

Quoted 15 lines · Heiko Carstens
Quoted 4 lines · Janosch Frank
arch/s390/kernel/uv.c +4 -0
+int uv_destroy_page(unsigned long paddr)
+{
+	struct uv_cb_cfs uvcb = {
+		.header.cmd = UVC_CMD_DESTR_SEC_STOR,

arch/s390/kernel/uv.c static int uv_pin_shared(unsigned long paddr) · line 141

Quoted 10 lines
arch/s390/kernel/uv.c +10 -0
+		.header.len = sizeof(uvcb),
+		.paddr = paddr
+	};
+
+	if (uv_call(0, (u64)&uvcb))
+		return -EINVAL;
+	return 0;
+}
+
+

I need to remove one of those \n

Quoted 10 lines
Quoted 10 lines

/*
* Requests the Ultravisor to encrypt a guest page and make it
* accessible to the host for paging (export).

arch/s390/mm/gmap.c +0 -0
index 373542ca1113..cfb0017f33a7 100644
@@ -2679,7 +2679,7 @@ static int __s390_reset_acc(pte_t *ptep, unsigned long addr,
 	pte_t pte = READ_ONCE(*ptep);
 
Quoted 7 lines

arch/s390/mm/gmap.c static int __s390_reset_acc(pte_t *ptep, unsigned long addr, · line 2682

Quoted 3 lines
arch/s390/mm/gmap.c +1 -1
 	if (pte_present(pte))
-		WARN_ON_ONCE(uv_convert_from_secure(pte_val(pte) & PAGE_MASK));
+		WARN_ON_ONCE(uv_destroy_page(pte_val(pte) & PAGE_MASK));

Why not put the WARN_ON_ONCE() into uv_destroy_page() and make that function return void?

If you prefer that, I'll change the patch.

I think we'd need a proper print of the return codes of the UVC anyway, the warn isn't very helpful if you want to debug after the fact.

maintainer
#4 lore raw

Re: [PATCH 1/2] s390x: uv: Add destroy page call

Quoted 9 lines · Janosch Frank
Quoted 9 lines
Quoted 9 lines

* Requests the Ultravisor to encrypt a guest page and make it
* accessible to the host for paging (export).

arch/s390/mm/gmap.c +0 -0
index 373542ca1113..cfb0017f33a7 100644
@@ -2679,7 +2679,7 @@ static int __s390_reset_acc(pte_t *ptep, unsigned long addr,
 	pte_t pte = READ_ONCE(*ptep);
 
Quoted 8 lines
Quoted 7 lines

arch/s390/mm/gmap.c static int __s390_reset_acc(pte_t *ptep, unsigned long addr, · line 2682

Quoted 3 lines
arch/s390/mm/gmap.c +1 -1
 	if (pte_present(pte))
-		WARN_ON_ONCE(uv_convert_from_secure(pte_val(pte) & PAGE_MASK));
+		WARN_ON_ONCE(uv_destroy_page(pte_val(pte) & PAGE_MASK));

Why not put the WARN_ON_ONCE() into uv_destroy_page() and make that function return void?

If you prefer that, I'll change the patch.

Seems to be better to me. Otherwise you start to sprinkle WARN_ONs all over the code, _if_ there would be more callers.

Quoted 2 lines

I think we'd need a proper print of the return codes of the UVC anyway, the warn isn't very helpful if you want to debug after the fact.

Maybe a new debug feature? Well, but that's something that hasn't do anything with this code.

author maintainer
#5 lore raw

Re: [PATCH 1/2] s390x: uv: Add destroy page call

Quoted 17 lines · Heiko Carstens
Quoted 16 lines · Janosch Frank
Quoted 16 lines

arch/s390/mm/gmap.c static int __s390_reset_acc(pte_t *ptep, unsigned long addr, · line 2682

Quoted 12 lines

* Requests the Ultravisor to encrypt a guest page and make it
* accessible to the host for paging (export).

arch/s390/mm/gmap.c +1 -1
index 373542ca1113..cfb0017f33a7 100644
@@ -2679,7 +2679,7 @@ static int __s390_reset_acc(pte_t *ptep, unsigned long addr,
 	pte_t pte = READ_ONCE(*ptep);
 
 	if (pte_present(pte))
-		WARN_ON_ONCE(uv_convert_from_secure(pte_val(pte) & PAGE_MASK));
+		WARN_ON_ONCE(uv_destroy_page(pte_val(pte) & PAGE_MASK));

Why not put the WARN_ON_ONCE() into uv_destroy_page() and make that function return void?

Quoted 4 lines
Quoted 1 line

If you prefer that, I'll change the patch.

Seems to be better to me. Otherwise you start to sprinkle WARN_ONs all over the code, _if_ there would be more callers.

The other call sites currently don't care about the return codes which is not optimal.

I'd prefer to leave it as is and put a debug item on the todo list which takes care of providing more debug data on error.

Quoted 6 lines
Quoted 3 lines

I think we'd need a proper print of the return codes of the UVC anyway, the warn isn't very helpful if you want to debug after the fact.

Maybe a new debug feature? Well, but that's something that hasn't do anything with this code.

author maintainer
#6 lore raw

PATCH 2/2 s390x: Add 3f program exception handler

In reply to Janosch Frank

Program exception 3f (secure storage violation) can only be detected when the CPU is running in SIE with a format 4 state description, e.g. running a protected guest. Because of this and because user space partly controls the guest memory mapping and can trigger this exception, we want to send a SIGSEGV to the process running the guest and not panic the kernel.

CC <[email protected]> # 5.7+
Fixes 084ea4d611a3 ("s390/mm: add (non)secure page access exceptions handlers")
---
 arch/s390/kernel/pgm_check.S |  2 +-
 arch/s390/mm/fault.c         | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+), 1 deletion(-)
arch/s390/kernel/pgm_check.S +1 -1
index 2c27907a5ffc..9a92638360ee 100644
@@ -80,7 +80,7 @@ PGM_CHECK(do_dat_exception)		/* 3b */
 PGM_CHECK_DEFAULT			/* 3c */
 PGM_CHECK(do_secure_storage_access)	/* 3d */
 PGM_CHECK(do_non_secure_storage_access)	/* 3e */
-PGM_CHECK_DEFAULT			/* 3f */
+PGM_CHECK(do_secure_storage_violation)	/* 3f */
 PGM_CHECK(monitor_event_exception)	/* 40 */
 PGM_CHECK_DEFAULT			/* 41 */
 PGM_CHECK_DEFAULT			/* 42 */
arch/s390/mm/fault.c +23 -0
index 4c8c063bce5b..20abb7c5c540 100644
@@ -859,6 +859,24 @@ void do_non_secure_storage_access(struct pt_regs *regs)
 }
 NOKPROBE_SYMBOL(do_non_secure_storage_access);
 
+void do_secure_storage_violation(struct pt_regs *regs)
+{
+	char buf[TASK_COMM_LEN];
+
+	/*
+	 * Either KVM messed up the secure guest mapping or the same
+	 * page is mapped into multiple secure guests.
+	 *
+	 * This exception is only triggered when a guest 2 is running
+	 * and can therefore never occur in kernel context.
+	 */
+	printk_ratelimited(KERN_WARNING
+			   "Secure storage violation in task: %s, pid %d\n",
+			   get_task_comm(buf, current), task_pid_nr(current));
+	send_sig(SIGSEGV, current, 0);
+}
+NOKPROBE_SYMBOL(do_secure_storage_violation);
+
 #else
 void do_secure_storage_access(struct pt_regs *regs)
 {
@@ -869,4 +887,9 @@ void do_non_secure_storage_access(struct pt_regs *regs)
 {
 	default_trap_handler(regs);
 }
+
+void do_secure_storage_violation(struct pt_regs *regs)
+{
+	default_trap_handler(regs);
+}
 #endif
Signature · 1 line
2.25.1
Acked-by on v1 2/2
#7 lore raw

Re: [PATCH 2/2] s390x: Add 3f program exception handler

Quoted 11 lines · Janosch Frank

Program exception 3f (secure storage violation) can only be detected when the CPU is running in SIE with a format 4 state description, e.g. running a protected guest. Because of this and because user space partly controls the guest memory mapping and can trigger this exception, we want to send a SIGSEGV to the process running the guest and not panic the kernel.

CC <[email protected]> # 5.7+
Fixes 084ea4d611a3 ("s390/mm: add (non)secure page access exceptions handlers")

I guess we will pick this up via the s390 tree?

Quoted 58 lines
---
 arch/s390/kernel/pgm_check.S |  2 +-
 arch/s390/mm/fault.c         | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+), 1 deletion(-)
arch/s390/kernel/pgm_check.S +1 -1
index 2c27907a5ffc..9a92638360ee 100644
@@ -80,7 +80,7 @@ PGM_CHECK(do_dat_exception)		/* 3b */
 PGM_CHECK_DEFAULT			/* 3c */
 PGM_CHECK(do_secure_storage_access)	/* 3d */
 PGM_CHECK(do_non_secure_storage_access)	/* 3e */
-PGM_CHECK_DEFAULT			/* 3f */
+PGM_CHECK(do_secure_storage_violation)	/* 3f */
 PGM_CHECK(monitor_event_exception)	/* 40 */
 PGM_CHECK_DEFAULT			/* 41 */
 PGM_CHECK_DEFAULT			/* 42 */
arch/s390/mm/fault.c +23 -0
index 4c8c063bce5b..20abb7c5c540 100644
@@ -859,6 +859,24 @@ void do_non_secure_storage_access(struct pt_regs *regs)
 }
 NOKPROBE_SYMBOL(do_non_secure_storage_access);
 
+void do_secure_storage_violation(struct pt_regs *regs)
+{
+	char buf[TASK_COMM_LEN];
+
+	/*
+	 * Either KVM messed up the secure guest mapping or the same
+	 * page is mapped into multiple secure guests.
+	 *
+	 * This exception is only triggered when a guest 2 is running
+	 * and can therefore never occur in kernel context.
+	 */
+	printk_ratelimited(KERN_WARNING
+			   "Secure storage violation in task: %s, pid %d\n",
+			   get_task_comm(buf, current), task_pid_nr(current));
+	send_sig(SIGSEGV, current, 0);
+}
+NOKPROBE_SYMBOL(do_secure_storage_violation);
+
 #else
 void do_secure_storage_access(struct pt_regs *regs)
 {
@@ -869,4 +887,9 @@ void do_non_secure_storage_access(struct pt_regs *regs)
 {
 	default_trap_handler(regs);
 }
+
+void do_secure_storage_violation(struct pt_regs *regs)
+{
+	default_trap_handler(regs);
+}
 #endif
maintainer
#8 lore raw

Re: [PATCH 2/2] s390x: Add 3f program exception handler

In reply to Janosch Frank
Quoted 41 lines · Janosch Frank

Program exception 3f (secure storage violation) can only be detected when the CPU is running in SIE with a format 4 state description, e.g. running a protected guest. Because of this and because user space partly controls the guest memory mapping and can trigger this exception, we want to send a SIGSEGV to the process running the guest and not panic the kernel.

CC <[email protected]> # 5.7+
Fixes 084ea4d611a3 ("s390/mm: add (non)secure page access exceptions handlers")
---
 arch/s390/kernel/pgm_check.S |  2 +-
 arch/s390/mm/fault.c         | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+), 1 deletion(-)
arch/s390/kernel/pgm_check.S +1 -1
index 2c27907a5ffc..9a92638360ee 100644
@@ -80,7 +80,7 @@ PGM_CHECK(do_dat_exception)		/* 3b */
 PGM_CHECK_DEFAULT			/* 3c */
 PGM_CHECK(do_secure_storage_access)	/* 3d */
 PGM_CHECK(do_non_secure_storage_access)	/* 3e */
-PGM_CHECK_DEFAULT			/* 3f */
+PGM_CHECK(do_secure_storage_violation)	/* 3f */
 PGM_CHECK(monitor_event_exception)	/* 40 */
 PGM_CHECK_DEFAULT			/* 41 */
 PGM_CHECK_DEFAULT			/* 42 */
arch/s390/mm/fault.c +4 -0
index 4c8c063bce5b..20abb7c5c540 100644
@@ -859,6 +859,24 @@ void do_non_secure_storage_access(struct pt_regs *regs)
 }
 NOKPROBE_SYMBOL(do_non_secure_storage_access);
 
+void do_secure_storage_violation(struct pt_regs *regs)
+{
+	char buf[TASK_COMM_LEN];
+

arch/s390/mm/fault.c void do_non_secure_storage_access(struct pt_regs *regs) · line 875

Quoted 10 lines
arch/s390/mm/fault.c +10 -0
+	/*
+	 * Either KVM messed up the secure guest mapping or the same
+	 * page is mapped into multiple secure guests.
+	 *
+	 * This exception is only triggered when a guest 2 is running
+	 * and can therefore never occur in kernel context.
+	 */
+	printk_ratelimited(KERN_WARNING
+			   "Secure storage violation in task: %s, pid %d\n",
+			   get_task_comm(buf, current), task_pid_nr(current));

Why get_task_comm() and task_pid_nr() instead of simply current->comm and current->pid?
Also: is the dmesg message of any value?

arch/s390/mm/fault.c void do_non_secure_storage_access(struct pt_regs *regs) · line 878

Quoted 3 lines
arch/s390/mm/fault.c +3 -0
+	send_sig(SIGSEGV, current, 0);
+}
+NOKPROBE_SYMBOL(do_secure_storage_violation);

Why is this NOKPROBE? Can this deadlock?

author maintainer
#9 lore raw

Re: [PATCH 2/2] s390x: Add 3f program exception handler

Quoted 47 lines · Heiko Carstens
Quoted 46 lines · Janosch Frank

Program exception 3f (secure storage violation) can only be detected when the CPU is running in SIE with a format 4 state description, e.g. running a protected guest. Because of this and because user space partly controls the guest memory mapping and can trigger this exception, we want to send a SIGSEGV to the process running the guest and not panic the kernel.

CC <[email protected]> # 5.7+
Fixes 084ea4d611a3 ("s390/mm: add (non)secure page access exceptions handlers")
---
 arch/s390/kernel/pgm_check.S |  2 +-
 arch/s390/mm/fault.c         | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+), 1 deletion(-)
arch/s390/kernel/pgm_check.S +1 -1
index 2c27907a5ffc..9a92638360ee 100644
@@ -80,7 +80,7 @@ PGM_CHECK(do_dat_exception)		/* 3b */
 PGM_CHECK_DEFAULT			/* 3c */
 PGM_CHECK(do_secure_storage_access)	/* 3d */
 PGM_CHECK(do_non_secure_storage_access)	/* 3e */
-PGM_CHECK_DEFAULT			/* 3f */
+PGM_CHECK(do_secure_storage_violation)	/* 3f */
 PGM_CHECK(monitor_event_exception)	/* 40 */
 PGM_CHECK_DEFAULT			/* 41 */
 PGM_CHECK_DEFAULT			/* 42 */
arch/s390/mm/fault.c +9 -0
index 4c8c063bce5b..20abb7c5c540 100644
@@ -859,6 +859,24 @@ void do_non_secure_storage_access(struct pt_regs *regs)
 }
 NOKPROBE_SYMBOL(do_non_secure_storage_access);
 
+void do_secure_storage_violation(struct pt_regs *regs)
+{
+	char buf[TASK_COMM_LEN];
+
+	/*
+	 * Either KVM messed up the secure guest mapping or the same
+	 * page is mapped into multiple secure guests.
+	 *
+	 * This exception is only triggered when a guest 2 is running
Quoted 8 lines

arch/s390/mm/fault.c void do_non_secure_storage_access(struct pt_regs *regs) · line 875

Quoted 5 lines
arch/s390/mm/fault.c +5 -0
+	 * and can therefore never occur in kernel context.
+	 */
+	printk_ratelimited(KERN_WARNING
+			   "Secure storage violation in task: %s, pid %d\n",
+			   get_task_comm(buf, current), task_pid_nr(current));

Why get_task_comm() and task_pid_nr() instead of simply current->comm and current->pid?

Normally if there are functions to get data I assume those should be used.

Quoted 1 line

Also: is the dmesg message of any value?

Yes, it's import for administrators to know that an exception caused this segfault and not some memory shenanigans.

As the exception only occurs if a guest runs in unsupported modes like sharing the memory between two secure guests it's a good first indication what went wrong.

Quoted 5 lines

arch/s390/mm/fault.c void do_non_secure_storage_access(struct pt_regs *regs)

Quoted 4 lines
arch/s390/mm/fault.c +3 -0
+	send_sig(SIGSEGV, current, 0);
+}
+NOKPROBE_SYMBOL(do_secure_storage_violation);>

Why is this NOKPROBE? Can this deadlock?

Right, we don't need to export this at all, this was copied over from the export function above.

maintainer
#10 lore raw

Re: [PATCH 2/2] s390x: Add 3f program exception handler

Quoted 8 lines · Janosch Frank
Quoted 6 lines

arch/s390/mm/fault.c void do_non_secure_storage_access(struct pt_regs *regs) · line 875

Quoted 3 lines
arch/s390/mm/fault.c +3 -0
+	printk_ratelimited(KERN_WARNING
+			   "Secure storage violation in task: %s, pid %d\n",
+			   get_task_comm(buf, current), task_pid_nr(current));

Why get_task_comm() and task_pid_nr() instead of simply current->comm and current->pid?

Normally if there are functions to get data I assume those should be used.

Could be used, however I don't see why you need that extra complexity here for both of them.

Quoted 7 lines
Quoted 1 line

Also: is the dmesg message of any value?

Yes, it's import for administrators to know that an exception caused this segfault and not some memory shenanigans.

As the exception only occurs if a guest runs in unsupported modes like sharing the memory between two secure guests it's a good first indication what went wrong.

Yes, fine with me. Just not sure of how help this is when pid namespaces come into play...