Skip to contentSkip to conversation

KVM: s390: vsie: fixes and cleanups

PATCH In review David Hildenbrand 3 participants 22 messages Apr 3, 2020 – Apr 7, 2020

Subsystems (5)
Stable branch KVM s390 s390 MM +1 more subsystems
lore
Share & tools
lore
#0 lore raw

v2 cover KVM: s390: vsie: fixes and cleanups

Some vsie/gmap fixes and two cleanups/improvements.

Patch #1 fixes an issue reported by Janosch. It was never observed so far, because KVM usually doesn't use a region 1 table for it's guest (unless memory would be exceeding something like 16 EB, which isn't even supported by the HW). Older QEMU+KVM or other hypervisors can trigger this.

Patch #2 fixes a code path that probably was never taken and will most probably not be taken very often in the future - unless somebody really messes up the page tables for a guest (or writes a test for it). At some point, a test case for this would be nice.

Patch #3 fixes a rare possible race. Don't think this is stable material.

Gave it some testing with my limited access to somewhat-fast s390x machines. Booted a Linux kernel, supplying all possible number of page table hiearchies.

v1 -> v2:
- "KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks" -- Fix WARN_ON_ONCE
- "gmap_table_walk() simplifications"
-- Also init "table" directly

David Hildenbrand (5):
KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks
KVM: s390: vsie: Fix delivery of addressing exceptions
KVM: s390: vsie: Fix possible race when shadowing region 3 tables
KVM: s390: vsie: Move conditional reschedule
KVM: s390: vsie: gmap_table_walk() simplifications

 arch/s390/kvm/vsie.c |  4 ++--
 arch/s390/mm/gmap.c  | 17 +++++++++++------
 2 files changed, 13 insertions(+), 8 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH v2 1/5 KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks

In case we have a region 1 ASCE, our shadow/g3 address can have any value. Unfortunately, (-1UL << 64) is undefined and triggers sometimes, rejecting valid shadow addresses when trying to walk our shadow table hierarchy.

The result is that the prefix cannot get mapped and will loop basically forever trying to map it (-EAGAIN loop).

After all, the broken check is only a sanity check, our table shadowing code in kvm_s390_shadow_tables() already checks these conditions, injecting proper translation exceptions. Turn it into a WARN_ON_ONCE().

Fixes 4be130a08420 ("s390/mm: add shadow gmap support")
Cc <[email protected]> # v4.8+
---
 arch/s390/mm/gmap.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/s390/mm/gmap.c +5 -1
index 2fbece47ef6f..b93dd54b234a 100644
@@ -787,14 +787,18 @@ static void gmap_call_notifier(struct gmap *gmap, unsigned long start,
 static inline unsigned long *gmap_table_walk(struct gmap *gmap,
 					     unsigned long gaddr, int level)
 {
+	const int asce_type = gmap->asce & _ASCE_TYPE_MASK;
 	unsigned long *table;
 
 	if ((gmap->asce & _ASCE_TYPE_MASK) + 4 < (level * 4))
 		return NULL;
 	if (gmap_is_shadow(gmap) && gmap->removed)
 		return NULL;
-	if (gaddr & (-1UL << (31 + ((gmap->asce & _ASCE_TYPE_MASK) >> 2)*11)))
+
+	if (WARN_ON_ONCE(asce_type != _ASCE_TYPE_REGION1 &&
+			 gaddr & (-1UL << (31 + (asce_type >> 2) * 11))))
 		return NULL;
+
 	table = gmap->table;
 	switch (gmap->asce & _ASCE_TYPE_MASK) {
 	case _ASCE_TYPE_REGION1:
Signature · 1 line
2.25.1
Reviewed-by on v2 1/5
#2 lore raw

Re: [PATCH v2 1/5] KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks

Quoted 4 lines · David Hildenbrand

In case we have a region 1 ASCE, our shadow/g3 address can have any value. Unfortunately, (-1UL << 64) is undefined and triggers sometimes, rejecting valid shadow addresses when trying to walk our shadow table hierarchy.

I thin the range of the addresses do not matter. Took me a while to understand maybe rephrase that:

In case we have a region 1 the following calculation
(31 + ((gmap->asce & _ASCE_TYPE_MASK) >> 2)*11)
results in 64. As shifts beyond the size are undefined the compiler is free to use instructions like sllg. sllg will only use 6 bits of the shift value (here 64) resulting in no shift at all. That means that ALL addresses will be rejected.

With that this makes sense.

Quoted 42 lines

The result is that the prefix cannot get mapped and will loop basically forever trying to map it (-EAGAIN loop).

After all, the broken check is only a sanity check, our table shadowing code in kvm_s390_shadow_tables() already checks these conditions, injecting proper translation exceptions. Turn it into a WARN_ON_ONCE().

Fixes 4be130a08420 ("s390/mm: add shadow gmap support")
Cc <[email protected]> # v4.8+
---
 arch/s390/mm/gmap.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/s390/mm/gmap.c +5 -1
index 2fbece47ef6f..b93dd54b234a 100644
@@ -787,14 +787,18 @@ static void gmap_call_notifier(struct gmap *gmap, unsigned long start,
 static inline unsigned long *gmap_table_walk(struct gmap *gmap,
 					     unsigned long gaddr, int level)
 {
+	const int asce_type = gmap->asce & _ASCE_TYPE_MASK;
 	unsigned long *table;
 
 	if ((gmap->asce & _ASCE_TYPE_MASK) + 4 < (level * 4))
 		return NULL;
 	if (gmap_is_shadow(gmap) && gmap->removed)
 		return NULL;
-	if (gaddr & (-1UL << (31 + ((gmap->asce & _ASCE_TYPE_MASK) >> 2)*11)))
+
+	if (WARN_ON_ONCE(asce_type != _ASCE_TYPE_REGION1 &&
+			 gaddr & (-1UL << (31 + (asce_type >> 2) * 11))))
 		return NULL;
+
 	table = gmap->table;
 	switch (gmap->asce & _ASCE_TYPE_MASK) {
 	case _ASCE_TYPE_REGION1:
#3 lore raw

Re: [PATCH v2 1/5] KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks

Quoted 12 lines

Am 03.04.2020 um 19:56 schrieb Christian Borntraeger <[email protected]>:



Quoted 5 lines · David Hildenbrand

In case we have a region 1 ASCE, our shadow/g3 address can have any value. Unfortunately, (-1UL << 64) is undefined and triggers sometimes, rejecting valid shadow addresses when trying to walk our shadow table hierarchy.

Quoted 8 lines

I thin the range of the addresses do not matter. Took me a while to understand maybe rephrase that:

In case we have a region 1 the following calculation
(31 + ((gmap->asce & _ASCE_TYPE_MASK) >> 2)*11)
results in 64. As shifts beyond the size are undefined the compiler is free to use instructions like sllg. sllg will only use 6 bits of the shift value (here 64) resulting in no shift at all. That means that ALL addresses will be rejected.

Interestingly, it would not fail when shadowing the r2t, but only when trying to shadow the r3t.

Quoted 5 lines

With that this makes sense.

In case there are no other comments, can you fixup when applying, or do you want me to resend?

Cheers

#4 lore raw

Re: [PATCH v2 1/5] KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks

Quoted 24 lines · David Hildenbrand
Quoted 12 lines

Am 03.04.2020 um 19:56 schrieb Christian Borntraeger <[email protected]>:



Quoted 5 lines · David Hildenbrand

In case we have a region 1 ASCE, our shadow/g3 address can have any value. Unfortunately, (-1UL << 64) is undefined and triggers sometimes, rejecting valid shadow addresses when trying to walk our shadow table hierarchy.

Quoted 8 lines

I thin the range of the addresses do not matter. Took me a while to understand maybe rephrase that:

In case we have a region 1 the following calculation
(31 + ((gmap->asce & _ASCE_TYPE_MASK) >> 2)*11)
results in 64. As shifts beyond the size are undefined the compiler is free to use instructions like sllg. sllg will only use 6 bits of the shift value (here 64) resulting in no shift at all. That means that ALL addresses will be rejected.

Interestingly, it would not fail when shadowing the r2t, but only when trying to shadow the r3t.

Quoted 1 line
Quoted 6 lines
Quoted 4 lines

With that this makes sense.

In case there are no other comments, can you fixup when applying, or do you want me to resend?

I can fixup.

#5 lore raw

Re: [PATCH v2 1/5] KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks

In reply to David Hildenbrand
Quoted 11 lines · David Hildenbrand

In case we have a region 1 ASCE, our shadow/g3 address can have any value. Unfortunately, (-1UL << 64) is undefined and triggers sometimes, rejecting valid shadow addresses when trying to walk our shadow table hierarchy.

The result is that the prefix cannot get mapped and will loop basically forever trying to map it (-EAGAIN loop).

After all, the broken check is only a sanity check, our table shadowing code in kvm_s390_shadow_tables() already checks these conditions, injecting proper translation exceptions. Turn it into a WARN_ON_ONCE().

After some testing I now triggered this warning:

[  541.633114] ------------[ cut here ]------------
[  541.633128] WARNING: CPU: 38 PID: 2812 at arch/s390/mm/gmap.c:799 gmap_shadow_pgt_lookup+0x98/0x1a0
[  541.633129] Modules linked in: vhost_net vhost macvtap macvlan tap kvm xt_CHECKSUM xt_MASQUERADE nf_nat_tftp nf_conntrack_tftp xt_CT tun bridge stp llc xt_tcpudp ip6t_REJECT nf_reject_ipv6 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ip6table_filter ip6_tables iptable_filter rpcrdma sunrpc rdma_ucm rdma_cm iw_cm ib_cm configfs mlx5_ib s390_trng ghash_s390 prng aes_s390 ib_uverbs des_s390 ib_core libdes sha3_512_s390 genwqe_card sha3_256_s390 vfio_ccw crc_itu_t vfio_mdev sha512_s390 mdev vfio_iommu_type1 sha1_s390 vfio eadm_sch zcrypt_cex4 sch_fq_codel ip_tables x_tables mlx5_core sha256_s390 sha_common pkey zcrypt rng_core autofs4
[  541.633164] CPU: 38 PID: 2812 Comm: CPU 0/KVM Not tainted 5.6.0+ #354
[  541.633166] Hardware name: IBM 3906 M04 704 (LPAR)
[  541.633167] Krnl PSW : 0704d00180000000 00000014e05dc454 (gmap_shadow_pgt_lookup+0x9c/0x1a0)
[  541.633169]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
[  541.633171] Krnl GPRS: 0000000000000000 0000001f00000000 0000001f487b8000 ffffffff80000000
[  541.633172]            ffffffffffffffff 000003e003defa18 000003e003defa1c 000003e003defa18
[  541.633173]            fffffffffffff000 000003e003defa18 000003e003defa28 0000001f70e06300
[  541.633174]            0000001f43484000 00000000043ed200 000003e003def978 000003e003def920
[  541.633203] Krnl Code: 00000014e05dc448: b9800038		ngr	%r3,%r8
                          00000014e05dc44c: a7840014		brc	8,00000014e05dc474
                         #00000014e05dc450: af000000		mc	0,0
                         >00000014e05dc454: a728fff5		lhi	%r2,-11
                          00000014e05dc458: a7180000		lhi	%r1,0
                          00000014e05dc45c: b2fa0070		niai	7,0
                          00000014e05dc460: 4010b04a		sth	%r1,74(%r11)
                          00000014e05dc464: b9140022		lgfr	%r2,%r2
[  541.633215] Call Trace:
[  541.633218]  [<00000014e05dc454>] gmap_shadow_pgt_lookup+0x9c/0x1a0 
[  541.633257]  [<000003ff804c57d6>] kvm_s390_shadow_fault+0x66/0x1e8 [kvm] 
[  541.633265]  [<000003ff804c72dc>] vsie_run+0x43c/0x710 [kvm] 
[  541.633273]  [<000003ff804c85ca>] kvm_s390_handle_vsie+0x632/0x750 [kvm] 
[  541.633281]  [<000003ff804c123c>] kvm_s390_handle_b2+0x84/0x4e0 [kvm] 
[  541.633289]  [<000003ff804b46b2>] kvm_handle_sie_intercept+0x172/0xcb8 [kvm] 
[  541.633297]  [<000003ff804b18a8>] __vcpu_run+0x658/0xc90 [kvm] 
[  541.633305]  [<000003ff804b2920>] kvm_arch_vcpu_ioctl_run+0x248/0x858 [kvm] 
[  541.633313]  [<000003ff8049d454>] kvm_vcpu_ioctl+0x284/0x7b0 [kvm] 
[  541.633316]  [<00000014e087d5ae>] ksys_ioctl+0xae/0xe8 
[  541.633318]  [<00000014e087d652>] __s390x_sys_ioctl+0x2a/0x38 
[  541.633323]  [<00000014e0ff02a2>] system_call+0x2a6/0x2c8 
[  541.633323] Last Breaking-Event-Address:
[  541.633334]  [<000003ff804983e0>] kvm_running_vcpu+0x3ea9ee997d8/0x3ea9ee99950 [kvm]
[  541.633335] ---[ end trace f69b6021855ea189 ]---

Unfortunately no dump at that point in time.
I have other tests which are clearly fixed by this patch, so we should propbably go forward anyway. Question is, is this just another bug we need to fix or is the assumption that somebody else checked all conditions so we can warn false?

#6 lore raw

Re: [PATCH v2 1/5] KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks

Quoted 53 lines · Christian Borntraeger
Quoted 11 lines · David Hildenbrand

In case we have a region 1 ASCE, our shadow/g3 address can have any value. Unfortunately, (-1UL << 64) is undefined and triggers sometimes, rejecting valid shadow addresses when trying to walk our shadow table hierarchy.

The result is that the prefix cannot get mapped and will loop basically forever trying to map it (-EAGAIN loop).

After all, the broken check is only a sanity check, our table shadowing code in kvm_s390_shadow_tables() already checks these conditions, injecting proper translation exceptions. Turn it into a WARN_ON_ONCE().

After some testing I now triggered this warning:

[  541.633114] ------------[ cut here ]------------
[  541.633128] WARNING: CPU: 38 PID: 2812 at arch/s390/mm/gmap.c:799 gmap_shadow_pgt_lookup+0x98/0x1a0
[  541.633129] Modules linked in: vhost_net vhost macvtap macvlan tap kvm xt_CHECKSUM xt_MASQUERADE nf_nat_tftp nf_conntrack_tftp xt_CT tun bridge stp llc xt_tcpudp ip6t_REJECT nf_reject_ipv6 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ip6table_filter ip6_tables iptable_filter rpcrdma sunrpc rdma_ucm rdma_cm iw_cm ib_cm configfs mlx5_ib s390_trng ghash_s390 prng aes_s390 ib_uverbs des_s390 ib_core libdes sha3_512_s390 genwqe_card sha3_256_s390 vfio_ccw crc_itu_t vfio_mdev sha512_s390 mdev vfio_iommu_type1 sha1_s390 vfio eadm_sch zcrypt_cex4 sch_fq_codel ip_tables x_tables mlx5_core sha256_s390 sha_common pkey zcrypt rng_core autofs4
[  541.633164] CPU: 38 PID: 2812 Comm: CPU 0/KVM Not tainted 5.6.0+ #354
[  541.633166] Hardware name: IBM 3906 M04 704 (LPAR)
[  541.633167] Krnl PSW : 0704d00180000000 00000014e05dc454 (gmap_shadow_pgt_lookup+0x9c/0x1a0)
[  541.633169]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
[  541.633171] Krnl GPRS: 0000000000000000 0000001f00000000 0000001f487b8000 ffffffff80000000
[  541.633172]            ffffffffffffffff 000003e003defa18 000003e003defa1c 000003e003defa18
[  541.633173]            fffffffffffff000 000003e003defa18 000003e003defa28 0000001f70e06300
[  541.633174]            0000001f43484000 00000000043ed200 000003e003def978 000003e003def920
[  541.633203] Krnl Code: 00000014e05dc448: b9800038		ngr	%r3,%r8
                          00000014e05dc44c: a7840014		brc	8,00000014e05dc474
                         #00000014e05dc450: af000000		mc	0,0
                         >00000014e05dc454: a728fff5		lhi	%r2,-11
                          00000014e05dc458: a7180000		lhi	%r1,0
                          00000014e05dc45c: b2fa0070		niai	7,0
                          00000014e05dc460: 4010b04a		sth	%r1,74(%r11)
                          00000014e05dc464: b9140022		lgfr	%r2,%r2
[  541.633215] Call Trace:
[  541.633218]  [<00000014e05dc454>] gmap_shadow_pgt_lookup+0x9c/0x1a0 
[  541.633257]  [<000003ff804c57d6>] kvm_s390_shadow_fault+0x66/0x1e8 [kvm] 
[  541.633265]  [<000003ff804c72dc>] vsie_run+0x43c/0x710 [kvm] 
[  541.633273]  [<000003ff804c85ca>] kvm_s390_handle_vsie+0x632/0x750 [kvm] 
[  541.633281]  [<000003ff804c123c>] kvm_s390_handle_b2+0x84/0x4e0 [kvm] 
[  541.633289]  [<000003ff804b46b2>] kvm_handle_sie_intercept+0x172/0xcb8 [kvm] 
[  541.633297]  [<000003ff804b18a8>] __vcpu_run+0x658/0xc90 [kvm] 
[  541.633305]  [<000003ff804b2920>] kvm_arch_vcpu_ioctl_run+0x248/0x858 [kvm] 
[  541.633313]  [<000003ff8049d454>] kvm_vcpu_ioctl+0x284/0x7b0 [kvm] 
[  541.633316]  [<00000014e087d5ae>] ksys_ioctl+0xae/0xe8 
[  541.633318]  [<00000014e087d652>] __s390x_sys_ioctl+0x2a/0x38 
[  541.633323]  [<00000014e0ff02a2>] system_call+0x2a6/0x2c8 
[  541.633323] Last Breaking-Event-Address:
[  541.633334]  [<000003ff804983e0>] kvm_running_vcpu+0x3ea9ee997d8/0x3ea9ee99950 [kvm]
[  541.633335] ---[ end trace f69b6021855ea189 ]---
Quoted 4 lines

Unfortunately no dump at that point in time.
I have other tests which are clearly fixed by this patch, so we should propbably go forward anyway. Question is, is this just another bug we need to fix or is the assumption that somebody else checked all conditions so we can warn false?

Yeah, I think it is via

kvm_s390_shadow_fault()->gmap_shadow_pgt_lookup()->gmap_table_walk()

where we just peek if there is already something shadowed. If not, we go via the full kvm_s390_shadow_tables() path.

So we could either do sanity checks in gmap_shadow_pgt_lookup(), or rather drop the WARN_ON_ONCE. I think the latter makes sense, now that we understood the problem.

Signature · 3 lines
Thanks,

David / dhildenb
#7 lore raw

Re: [PATCH v2 1/5] KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks

Quoted 61 lines · David Hildenbrand
Quoted 53 lines · Christian Borntraeger
Quoted 11 lines · David Hildenbrand

In case we have a region 1 ASCE, our shadow/g3 address can have any value. Unfortunately, (-1UL << 64) is undefined and triggers sometimes, rejecting valid shadow addresses when trying to walk our shadow table hierarchy.

The result is that the prefix cannot get mapped and will loop basically forever trying to map it (-EAGAIN loop).

After all, the broken check is only a sanity check, our table shadowing code in kvm_s390_shadow_tables() already checks these conditions, injecting proper translation exceptions. Turn it into a WARN_ON_ONCE().

After some testing I now triggered this warning:

[  541.633114] ------------[ cut here ]------------
[  541.633128] WARNING: CPU: 38 PID: 2812 at arch/s390/mm/gmap.c:799 gmap_shadow_pgt_lookup+0x98/0x1a0
[  541.633129] Modules linked in: vhost_net vhost macvtap macvlan tap kvm xt_CHECKSUM xt_MASQUERADE nf_nat_tftp nf_conntrack_tftp xt_CT tun bridge stp llc xt_tcpudp ip6t_REJECT nf_reject_ipv6 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ip6table_filter ip6_tables iptable_filter rpcrdma sunrpc rdma_ucm rdma_cm iw_cm ib_cm configfs mlx5_ib s390_trng ghash_s390 prng aes_s390 ib_uverbs des_s390 ib_core libdes sha3_512_s390 genwqe_card sha3_256_s390 vfio_ccw crc_itu_t vfio_mdev sha512_s390 mdev vfio_iommu_type1 sha1_s390 vfio eadm_sch zcrypt_cex4 sch_fq_codel ip_tables x_tables mlx5_core sha256_s390 sha_common pkey zcrypt rng_core autofs4
[  541.633164] CPU: 38 PID: 2812 Comm: CPU 0/KVM Not tainted 5.6.0+ #354
[  541.633166] Hardware name: IBM 3906 M04 704 (LPAR)
[  541.633167] Krnl PSW : 0704d00180000000 00000014e05dc454 (gmap_shadow_pgt_lookup+0x9c/0x1a0)
[  541.633169]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
[  541.633171] Krnl GPRS: 0000000000000000 0000001f00000000 0000001f487b8000 ffffffff80000000
[  541.633172]            ffffffffffffffff 000003e003defa18 000003e003defa1c 000003e003defa18
[  541.633173]            fffffffffffff000 000003e003defa18 000003e003defa28 0000001f70e06300
[  541.633174]            0000001f43484000 00000000043ed200 000003e003def978 000003e003def920
[  541.633203] Krnl Code: 00000014e05dc448: b9800038		ngr	%r3,%r8
                          00000014e05dc44c: a7840014		brc	8,00000014e05dc474
                         #00000014e05dc450: af000000		mc	0,0
                         >00000014e05dc454: a728fff5		lhi	%r2,-11
                          00000014e05dc458: a7180000		lhi	%r1,0
                          00000014e05dc45c: b2fa0070		niai	7,0
                          00000014e05dc460: 4010b04a		sth	%r1,74(%r11)
                          00000014e05dc464: b9140022		lgfr	%r2,%r2
[  541.633215] Call Trace:
[  541.633218]  [<00000014e05dc454>] gmap_shadow_pgt_lookup+0x9c/0x1a0 
[  541.633257]  [<000003ff804c57d6>] kvm_s390_shadow_fault+0x66/0x1e8 [kvm] 
[  541.633265]  [<000003ff804c72dc>] vsie_run+0x43c/0x710 [kvm] 
[  541.633273]  [<000003ff804c85ca>] kvm_s390_handle_vsie+0x632/0x750 [kvm] 
[  541.633281]  [<000003ff804c123c>] kvm_s390_handle_b2+0x84/0x4e0 [kvm] 
[  541.633289]  [<000003ff804b46b2>] kvm_handle_sie_intercept+0x172/0xcb8 [kvm] 
[  541.633297]  [<000003ff804b18a8>] __vcpu_run+0x658/0xc90 [kvm] 
[  541.633305]  [<000003ff804b2920>] kvm_arch_vcpu_ioctl_run+0x248/0x858 [kvm] 
[  541.633313]  [<000003ff8049d454>] kvm_vcpu_ioctl+0x284/0x7b0 [kvm] 
[  541.633316]  [<00000014e087d5ae>] ksys_ioctl+0xae/0xe8 
[  541.633318]  [<00000014e087d652>] __s390x_sys_ioctl+0x2a/0x38 
[  541.633323]  [<00000014e0ff02a2>] system_call+0x2a6/0x2c8 
[  541.633323] Last Breaking-Event-Address:
[  541.633334]  [<000003ff804983e0>] kvm_running_vcpu+0x3ea9ee997d8/0x3ea9ee99950 [kvm]
[  541.633335] ---[ end trace f69b6021855ea189 ]---
Quoted 4 lines

Unfortunately no dump at that point in time.
I have other tests which are clearly fixed by this patch, so we should propbably go forward anyway. Question is, is this just another bug we need to fix or is the assumption that somebody else checked all conditions so we can warn false?

Yeah, I think it is via

Quoted 8 lines

kvm_s390_shadow_fault()->gmap_shadow_pgt_lookup()->gmap_table_walk()

where we just peek if there is already something shadowed. If not, we go via the full kvm_s390_shadow_tables() path.

So we could either do sanity checks in gmap_shadow_pgt_lookup(), or rather drop the WARN_ON_ONCE. I think the latter makes sense, now that we understood the problem.

Ok, so I will drop the WARN_ON_ONCE and fixup the commit message.

#8 lore raw

Re: [PATCH v2 1/5] KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks

Quoted 66 lines · Christian Borntraeger
Quoted 63 lines · David Hildenbrand
Quoted 53 lines · Christian Borntraeger
Quoted 11 lines · David Hildenbrand

In case we have a region 1 ASCE, our shadow/g3 address can have any value. Unfortunately, (-1UL << 64) is undefined and triggers sometimes, rejecting valid shadow addresses when trying to walk our shadow table hierarchy.

The result is that the prefix cannot get mapped and will loop basically forever trying to map it (-EAGAIN loop).

After all, the broken check is only a sanity check, our table shadowing code in kvm_s390_shadow_tables() already checks these conditions, injecting proper translation exceptions. Turn it into a WARN_ON_ONCE().

After some testing I now triggered this warning:

[  541.633114] ------------[ cut here ]------------
[  541.633128] WARNING: CPU: 38 PID: 2812 at arch/s390/mm/gmap.c:799 gmap_shadow_pgt_lookup+0x98/0x1a0
[  541.633129] Modules linked in: vhost_net vhost macvtap macvlan tap kvm xt_CHECKSUM xt_MASQUERADE nf_nat_tftp nf_conntrack_tftp xt_CT tun bridge stp llc xt_tcpudp ip6t_REJECT nf_reject_ipv6 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ip6table_filter ip6_tables iptable_filter rpcrdma sunrpc rdma_ucm rdma_cm iw_cm ib_cm configfs mlx5_ib s390_trng ghash_s390 prng aes_s390 ib_uverbs des_s390 ib_core libdes sha3_512_s390 genwqe_card sha3_256_s390 vfio_ccw crc_itu_t vfio_mdev sha512_s390 mdev vfio_iommu_type1 sha1_s390 vfio eadm_sch zcrypt_cex4 sch_fq_codel ip_tables x_tables mlx5_core sha256_s390 sha_common pkey zcrypt rng_core autofs4
[  541.633164] CPU: 38 PID: 2812 Comm: CPU 0/KVM Not tainted 5.6.0+ #354
[  541.633166] Hardware name: IBM 3906 M04 704 (LPAR)
[  541.633167] Krnl PSW : 0704d00180000000 00000014e05dc454 (gmap_shadow_pgt_lookup+0x9c/0x1a0)
[  541.633169]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:1 PM:0 RI:0 EA:3
[  541.633171] Krnl GPRS: 0000000000000000 0000001f00000000 0000001f487b8000 ffffffff80000000
[  541.633172]            ffffffffffffffff 000003e003defa18 000003e003defa1c 000003e003defa18
[  541.633173]            fffffffffffff000 000003e003defa18 000003e003defa28 0000001f70e06300
[  541.633174]            0000001f43484000 00000000043ed200 000003e003def978 000003e003def920
[  541.633203] Krnl Code: 00000014e05dc448: b9800038		ngr	%r3,%r8
                          00000014e05dc44c: a7840014		brc	8,00000014e05dc474
                         #00000014e05dc450: af000000		mc	0,0
                         >00000014e05dc454: a728fff5		lhi	%r2,-11
                          00000014e05dc458: a7180000		lhi	%r1,0
                          00000014e05dc45c: b2fa0070		niai	7,0
                          00000014e05dc460: 4010b04a		sth	%r1,74(%r11)
                          00000014e05dc464: b9140022		lgfr	%r2,%r2
[  541.633215] Call Trace:
[  541.633218]  [<00000014e05dc454>] gmap_shadow_pgt_lookup+0x9c/0x1a0 
[  541.633257]  [<000003ff804c57d6>] kvm_s390_shadow_fault+0x66/0x1e8 [kvm] 
[  541.633265]  [<000003ff804c72dc>] vsie_run+0x43c/0x710 [kvm] 
[  541.633273]  [<000003ff804c85ca>] kvm_s390_handle_vsie+0x632/0x750 [kvm] 
[  541.633281]  [<000003ff804c123c>] kvm_s390_handle_b2+0x84/0x4e0 [kvm] 
[  541.633289]  [<000003ff804b46b2>] kvm_handle_sie_intercept+0x172/0xcb8 [kvm] 
[  541.633297]  [<000003ff804b18a8>] __vcpu_run+0x658/0xc90 [kvm] 
[  541.633305]  [<000003ff804b2920>] kvm_arch_vcpu_ioctl_run+0x248/0x858 [kvm] 
[  541.633313]  [<000003ff8049d454>] kvm_vcpu_ioctl+0x284/0x7b0 [kvm] 
[  541.633316]  [<00000014e087d5ae>] ksys_ioctl+0xae/0xe8 
[  541.633318]  [<00000014e087d652>] __s390x_sys_ioctl+0x2a/0x38 
[  541.633323]  [<00000014e0ff02a2>] system_call+0x2a6/0x2c8 
[  541.633323] Last Breaking-Event-Address:
[  541.633334]  [<000003ff804983e0>] kvm_running_vcpu+0x3ea9ee997d8/0x3ea9ee99950 [kvm]
[  541.633335] ---[ end trace f69b6021855ea189 ]---
Quoted 4 lines

Unfortunately no dump at that point in time.
I have other tests which are clearly fixed by this patch, so we should propbably go forward anyway. Question is, is this just another bug we need to fix or is the assumption that somebody else checked all conditions so we can warn false?

Yeah, I think it is via

kvm_s390_shadow_fault()->gmap_shadow_pgt_lookup()->gmap_table_walk()

Quoted 9 lines
Quoted 6 lines

where we just peek if there is already something shadowed. If not, we go via the full kvm_s390_shadow_tables() path.

So we could either do sanity checks in gmap_shadow_pgt_lookup(), or rather drop the WARN_ON_ONCE. I think the latter makes sense, now that we understood the problem.

Ok, so I will drop the WARN_ON_ONCE and fixup the commit message.

Perfect, thanks!

Signature · 3 lines
Thanks,

David / dhildenb
maintainer Reviewed-by
#9 lore raw

Re: [PATCH v2 1/5] KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks

In reply to Christian Borntraeger
Quoted 87 lines · On Tue, 7 Apr 2020 09:52:53 +0200 Christian Borntraeger…
Quoted 86 lines · David Hildenbrand
Quoted 75 lines · Christian Borntraeger
Quoted 5 lines · David Hildenbrand

In case we have a region 1 ASCE, our shadow/g3 address can have any value. Unfortunately, (-1UL << 64) is undefined and triggers sometimes, rejecting valid shadow addresses when trying to walk our shadow table hierarchy.

Quoted 7 lines

The result is that the prefix cannot get mapped and will loop basically forever trying to map it (-EAGAIN loop).

After all, the broken check is only a sanity check, our table shadowing code in kvm_s390_shadow_tables() already checks these conditions, injecting proper translation exceptions. Turn it into a WARN_ON_ONCE().

After some testing I now triggered this warning:

[  541.633114] ------------[ cut here ]------------
[  541.633128] WARNING: CPU: 38 PID: 2812 at

arch/s390/mm/gmap.c:799 gmap_shadow_pgt_lookup+0x98/0x1a0 [ 541.633129] Modules linked in: vhost_net vhost macvtap macvlan tap kvm xt_CHECKSUM xt_MASQUERADE nf_nat_tftp nf_conntrack_tftp xt_CT tun bridge stp llc xt_tcpudp ip6t_REJECT nf_reject_ipv6 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ip6table_filter ip6_tables iptable_filter rpcrdma sunrpc rdma_ucm rdma_cm iw_cm ib_cm configfs mlx5_ib s390_trng ghash_s390 prng aes_s390 ib_uverbs des_s390 ib_core libdes sha3_512_s390 genwqe_card sha3_256_s390 vfio_ccw crc_itu_t vfio_mdev sha512_s390 mdev vfio_iommu_type1 sha1_s390 vfio eadm_sch zcrypt_cex4 sch_fq_codel ip_tables x_tables mlx5_core sha256_s390 sha_common pkey zcrypt rng_core autofs4 [ 541.633164] CPU: 38 PID: 2812 Comm: CPU 0/KVM Not tainted 5.6.0+ #354 [ 541.633166] Hardware name: IBM 3906 M04 704 (LPAR) [ 541.633167] Krnl PSW : 0704d00180000000 00000014e05dc454 (gmap_shadow_pgt_lookup+0x9c/0x1a0) [
541.633169] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3
CC:1 PM:0 RI:0 EA:3 [ 541.633171] Krnl GPRS: 0000000000000000 0000001f00000000 0000001f487b8000 ffffffff80000000 [ 541.633172]

          ffffffffffffffff 000003e003defa18 000003e003defa1c
000003e003defa18 [  541.633173]            fffffffffffff000
000003e003defa18 000003e003defa28 0000001f70e06300 [  541.633174]
          0000001f43484000 00000000043ed200 000003e003def978

000003e003def920 [ 541.633203] Krnl Code: 00000014e05dc448:
b9800038 ngr %r3,%r8 00000014e05dc44c:
a7840014 brc 8,00000014e05dc474
#00000014e05dc450: af000000 mc 0,0

                         >00000014e05dc454: a728fff5
                         >	lhi	%r2,-11  
                          00000014e05dc458: a7180000
	lhi	%r1,0 00000014e05dc45c: b2fa0070
niai	7,0 00000014e05dc460: 4010b04a
sth	%r1,74(%r11) 00000014e05dc464: b9140022
lgfr	%r2,%r2 [  541.633215] Call Trace:
[  541.633218]  [<00000014e05dc454>]
gmap_shadow_pgt_lookup+0x9c/0x1a0 [  541.633257]
[<000003ff804c57d6>] kvm_s390_shadow_fault+0x66/0x1e8 [kvm] [
541.633265]  [<000003ff804c72dc>] vsie_run+0x43c/0x710 [kvm] [
541.633273]  [<000003ff804c85ca>] kvm_s390_handle_vsie+0x632/0x750
[kvm] [  541.633281]  [<000003ff804c123c>]
kvm_s390_handle_b2+0x84/0x4e0 [kvm] [  541.633289]
[<000003ff804b46b2>] kvm_handle_sie_intercept+0x172/0xcb8 [kvm] [
541.633297]  [<000003ff804b18a8>] __vcpu_run+0x658/0xc90 [kvm] [
541.633305]  [<000003ff804b2920>]
kvm_arch_vcpu_ioctl_run+0x248/0x858 [kvm] [  541.633313]
[<000003ff8049d454>] kvm_vcpu_ioctl+0x284/0x7b0 [kvm] [
541.633316]  [<00000014e087d5ae>] ksys_ioctl+0xae/0xe8 [
541.633318]  [<00000014e087d652>] __s390x_sys_ioctl+0x2a/0x38 [
541.633323]  [<00000014e0ff02a2>] system_call+0x2a6/0x2c8 [

541.633323] Last Breaking-Event-Address: [ 541.633334] [<000003ff804983e0>] kvm_running_vcpu+0x3ea9ee997d8/0x3ea9ee99950 [kvm] [ 541.633335] ---[ end trace f69b6021855ea189 ]---

Quoted 5 lines

Unfortunately no dump at that point in time.
I have other tests which are clearly fixed by this patch, so we should propbably go forward anyway. Question is, is this just another bug we need to fix or is the assumption that somebody else checked all conditions so we can warn false?

Yeah, I think it is via

kvm_s390_shadow_fault()->gmap_shadow_pgt_lookup()->gmap_table_walk()

Quoted 9 lines
Quoted 6 lines

where we just peek if there is already something shadowed. If not, we go via the full kvm_s390_shadow_tables() path.

So we could either do sanity checks in gmap_shadow_pgt_lookup(), or rather drop the WARN_ON_ONCE. I think the latter makes sense, now that we understood the problem.

Ok, so I will drop the WARN_ON_ONCE and fixup the commit message.

with those fixes, you can also add:

#10 lore raw

PATCH v2 2/5 KVM: s390: vsie: Fix delivery of addressing exceptions

In reply to David Hildenbrand

Whenever we get an -EFAULT, we failed to read in guest 2 physical address space. Such addressing exceptions are reported via a program intercept to the nested hypervisor.

We faked the intercept, we have to return to guest 2. Instead, right now we would be returning -EFAULT from the intercept handler, eventually crashing the VM.

Addressing exceptions can only happen if the g2->g3 page tables reference invalid g2 addresses (say, either a table or the final page is not accessible - so something that basically never happens in sane environments.

Identified by manual code inspection.

Fixes a3508fbe9dc6 ("KVM: s390: vsie: initial support for nested virtualization")
Cc <[email protected]> # v4.8+
---
 arch/s390/kvm/vsie.c | 1 +
 1 file changed, 1 insertion(+)
arch/s390/kvm/vsie.c +1 -0
index 076090f9e666..4f6c22d72072 100644
@@ -1202,6 +1202,7 @@ static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 		scb_s->iprcc = PGM_ADDRESSING;
 		scb_s->pgmilc = 4;
 		scb_s->gpsw.addr = __rewind_psw(scb_s->gpsw, 4);
+		rc = 1;
 	}
 	return rc;
 }
Signature · 1 line
2.25.1
maintainer Reviewed-by on v2 2/5
#11 lore raw

Re: [PATCH v2 2/5] KVM: s390: vsie: Fix delivery of addressing exceptions

Quoted 27 lines · On Fri, 3 Apr 2020 17:30:47 +0200 David Hildenbrand <[email protected]> wrote:

Whenever we get an -EFAULT, we failed to read in guest 2 physical address space. Such addressing exceptions are reported via a program intercept to the nested hypervisor.

We faked the intercept, we have to return to guest 2. Instead, right now we would be returning -EFAULT from the intercept handler, eventually crashing the VM.

Addressing exceptions can only happen if the g2->g3 page tables reference invalid g2 addresses (say, either a table or the final page is not accessible - so something that basically never happens in sane environments.

Identified by manual code inspection.

Fixes a3508fbe9dc6 ("KVM: s390: vsie: initial support for nested

virtualization") Cc: <[email protected]> # v4.8+

---
 arch/s390/kvm/vsie.c | 1 +
 1 file changed, 1 insertion(+)
arch/s390/kvm/vsie.c +0 -0
index 076090f9e666..4f6c22d72072 100644

arch/s390/kvm/vsie.c static int vsie_run(struct kvm_vcpu *vcpu, · line 1208

Quoted 7 lines
arch/s390/kvm/vsie.c +1 -0
@@ -1202,6 +1202,7 @@ static int vsie_run(struct kvm_vcpu *vcpu,
 struct vsie_page *vsie_page) scb_s->iprcc = PGM_ADDRESSING;
 		scb_s->pgmilc = 4;
 		scb_s->gpsw.addr = __rewind_psw(scb_s->gpsw, 4);
+		rc = 1;
 	}
 	return rc;
 }

so, the reason why we never noticed this issue before is simply that nobody tried running a misbehaving nested guest?

#12 lore raw

Re: [PATCH v2 2/5] KVM: s390: vsie: Fix delivery of addressing exceptions

Quoted 25 lines · Claudio Imbrenda
Quoted 22 lines · On Fri, 3 Apr 2020 17:30:47 +0200 David Hildenbrand <[email protected]> wrote:

Whenever we get an -EFAULT, we failed to read in guest 2 physical address space. Such addressing exceptions are reported via a program intercept to the nested hypervisor.

We faked the intercept, we have to return to guest 2. Instead, right now we would be returning -EFAULT from the intercept handler, eventually crashing the VM.

Addressing exceptions can only happen if the g2->g3 page tables reference invalid g2 addresses (say, either a table or the final page is not accessible - so something that basically never happens in sane environments.

Identified by manual code inspection.

Fixes a3508fbe9dc6 ("KVM: s390: vsie: initial support for nested

virtualization") Cc: <[email protected]> # v4.8+

---
 arch/s390/kvm/vsie.c | 1 +
 1 file changed, 1 insertion(+)
Quoted 15 lines

arch/s390/kvm/vsie.c static int vsie_run(struct kvm_vcpu *vcpu, · line 1208

Quoted 12 lines
arch/s390/kvm/vsie.c +1 -0
index 076090f9e666..4f6c22d72072 100644
@@ -1202,6 +1202,7 @@ static int vsie_run(struct kvm_vcpu *vcpu,
 struct vsie_page *vsie_page) scb_s->iprcc = PGM_ADDRESSING;
 		scb_s->pgmilc = 4;
 		scb_s->gpsw.addr = __rewind_psw(scb_s->gpsw, 4);
+		rc = 1;
 	}
 	return rc;
 }

so, the reason why we never noticed this issue before is simply that nobody tried running a misbehaving nested guest?

Yes, actually, a misbehaving nested hypervisor.

Quoted 3 lines

Thanks!

Signature · 3 lines
Thanks,

David / dhildenb
#13 lore raw

PATCH v2 3/5 KVM: s390: vsie: Fix possible race when shadowing region 3 tables

In reply to David Hildenbrand

We have to properly retry again by returning -EINVAL immediately in case somebody else instantiated the table concurrently. We missed to add the goto in this function only. The code now matches the other, similar shadowing functions.

We are overwriting an existing region 2 table entry. All allocated pages are added to the crst_list to be freed later, so they are not lost forever. However, when unshadowing the region 2 table, we wouldn't trigger unshadowing of the original shadowed region 3 table that we replaced. It would get unshadowed when the original region 3 table is modified. As it's not connected to the page table hierarchy anymore, it's not going to get used anymore. However, for a limited time, this page table will stick around, so it's in some sense a temporary memory leak.

Identified by manual code inspection. I don't think this classifies as stable material.

Fixes 998f637cc4b9 ("s390/mm: avoid races on region/segment/page table shadowing")
---
 arch/s390/mm/gmap.c | 1 +
 1 file changed, 1 insertion(+)
arch/s390/mm/gmap.c +1 -0
index b93dd54b234a..24ef30fb0833 100644
@@ -1844,6 +1844,7 @@ int gmap_shadow_r3t(struct gmap *sg, unsigned long saddr, unsigned long r3t,
 		goto out_free;
 	} else if (*table & _REGION_ENTRY_ORIGIN) {
 		rc = -EAGAIN;		/* Race with shadow */
+		goto out_free;
 	}
 	crst_table_init(s_r3t, _REGION3_ENTRY_EMPTY);
 	/* mark as invalid as long as the parent table is not protected */
Signature · 1 line
2.25.1
maintainer Reviewed-by on v2 3/5
#14 lore raw

Re: [PATCH v2 3/5] KVM: s390: vsie: Fix possible race when shadowing region 3 tables

Quoted 36 lines · On Fri, 3 Apr 2020 17:30:48 +0200 David Hildenbrand <[email protected]> wrote:

We have to properly retry again by returning -EINVAL immediately in case somebody else instantiated the table concurrently. We missed to add the goto in this function only. The code now matches the other, similar shadowing functions.

We are overwriting an existing region 2 table entry. All allocated pages are added to the crst_list to be freed later, so they are not lost forever. However, when unshadowing the region 2 table, we wouldn't trigger unshadowing of the original shadowed region 3 table that we replaced. It would get unshadowed when the original region 3 table is modified. As it's not connected to the page table hierarchy anymore, it's not going to get used anymore. However, for a limited time, this page table will stick around, so it's in some sense a temporary memory leak.

Identified by manual code inspection. I don't think this classifies as stable material.

Fixes 998f637cc4b9 ("s390/mm: avoid races on region/segment/page

table shadowing") Signed-off-by: David Hildenbrand <[email protected]>

---
 arch/s390/mm/gmap.c | 1 +
 1 file changed, 1 insertion(+)
arch/s390/mm/gmap.c +1 -0
index b93dd54b234a..24ef30fb0833 100644
@@ -1844,6 +1844,7 @@ int gmap_shadow_r3t(struct gmap *sg, unsigned
 long saddr, unsigned long r3t, goto out_free;
 	} else if (*table & _REGION_ENTRY_ORIGIN) {
 		rc = -EAGAIN;		/* Race with shadow */
+		goto out_free;
 	}
 	crst_table_init(s_r3t, _REGION3_ENTRY_EMPTY);
 	/* mark as invalid as long as the parent table is not
Quoted 1 line

protected */

#15 lore raw

PATCH v2 4/5 KVM: s390: vsie: Move conditional reschedule

In reply to David Hildenbrand

Let's move it to the outer loop, in case we ever run again into long loops, trying to map the prefix. While at it, convert it to cond_resched().

---
 arch/s390/kvm/vsie.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/s390/kvm/vsie.c +1 -2
index 4f6c22d72072..ef05b4e167fb 100644
@@ -1000,8 +1000,6 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 
 	handle_last_fault(vcpu, vsie_page);
 
-	if (need_resched())
-		schedule();
 	if (test_cpu_flag(CIF_MCCK_PENDING))
 		s390_handle_mcck();
 
@@ -1185,6 +1183,7 @@ static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 		    kvm_s390_vcpu_has_irq(vcpu, 0) ||
 		    kvm_s390_vcpu_sie_inhibited(vcpu))
 			break;
+		cond_resched();
 	}
 
 	if (rc == -EFAULT) {
Signature · 1 line
2.25.1
Reviewed-by on v2 4/5
#16 lore raw

Re: [PATCH v2 4/5] KVM: s390: vsie: Move conditional reschedule

Quoted 4 lines · David Hildenbrand

Let's move it to the outer loop, in case we ever run again into long loops, trying to map the prefix. While at it, convert it to cond_resched().

Quoted 26 lines
---
 arch/s390/kvm/vsie.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/s390/kvm/vsie.c +1 -2
index 4f6c22d72072..ef05b4e167fb 100644
@@ -1000,8 +1000,6 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 
 	handle_last_fault(vcpu, vsie_page);
 
-	if (need_resched())
-		schedule();
 	if (test_cpu_flag(CIF_MCCK_PENDING))
 		s390_handle_mcck();
 
@@ -1185,6 +1183,7 @@ static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 		    kvm_s390_vcpu_has_irq(vcpu, 0) ||
 		    kvm_s390_vcpu_sie_inhibited(vcpu))
 			break;
+		cond_resched();
 	}
 
 	if (rc == -EFAULT) {
maintainer Reviewed-by on v2 4/5
#17 lore raw

Re: [PATCH v2 4/5] KVM: s390: vsie: Move conditional reschedule

In reply to David Hildenbrand
Quoted 24 lines · On Fri, 3 Apr 2020 17:30:49 +0200 David Hildenbrand <[email protected]> wrote:

Let's move it to the outer loop, in case we ever run again into long loops, trying to map the prefix. While at it, convert it to cond_resched().

---
 arch/s390/kvm/vsie.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/s390/kvm/vsie.c +0 -2
index 4f6c22d72072..ef05b4e167fb 100644
@@ -1000,8 +1000,6 @@ static int do_vsie_run(struct kvm_vcpu *vcpu,
 struct vsie_page *vsie_page) 
 	handle_last_fault(vcpu, vsie_page);
 
-	if (need_resched())
-		schedule();
 	if (test_cpu_flag(CIF_MCCK_PENDING))
 		s390_handle_mcck();
 
@@ -1185,6 +1183,7 @@ static int vsie_run(struct kvm_vcpu *vcpu,

struct vsie_page *vsie_page) kvm_s390_vcpu_has_irq(vcpu, 0) ||

arch/s390/kvm/vsie.c static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · line 1189

Quoted 6 lines
arch/s390/kvm/vsie.c +1 -0
 		    kvm_s390_vcpu_sie_inhibited(vcpu))
 			break;
+		cond_resched();
 	}
 
 	if (rc == -EFAULT) {
#18 lore raw

PATCH v2 5/5 KVM: s390: vsie: gmap_table_walk() simplifications

In reply to David Hildenbrand

Let's use asce_type where applicable. Also, simplify our sanity check for valid table levels and convert it into a WARN_ON_ONCE(). Check if we even have a valid gmap shadow as the very first step.

---
 arch/s390/mm/gmap.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
arch/s390/mm/gmap.c +5 -5
index 24ef30fb0833..a2bd8d7792e9 100644
@@ -788,19 +788,19 @@ static inline unsigned long *gmap_table_walk(struct gmap *gmap,
 					     unsigned long gaddr, int level)
 {
 	const int asce_type = gmap->asce & _ASCE_TYPE_MASK;
-	unsigned long *table;
+	unsigned long *table = gmap->table;
 
-	if ((gmap->asce & _ASCE_TYPE_MASK) + 4 < (level * 4))
-		return NULL;
 	if (gmap_is_shadow(gmap) && gmap->removed)
 		return NULL;
 
+	if (WARN_ON_ONCE(level > (asce_type >> 2) + 1))
+		return NULL;
+
 	if (WARN_ON_ONCE(asce_type != _ASCE_TYPE_REGION1 &&
 			 gaddr & (-1UL << (31 + (asce_type >> 2) * 11))))
 		return NULL;
 
-	table = gmap->table;
-	switch (gmap->asce & _ASCE_TYPE_MASK) {
+	switch (asce_type) {
 	case _ASCE_TYPE_REGION1:
 		table += (gaddr & _REGION1_INDEX) >> _REGION1_SHIFT;
 		if (level == 4)
Signature · 1 line
2.25.1
Reviewed-by on v2 5/5
#19 lore raw

Re: [PATCH v2 5/5] KVM: s390: vsie: gmap_table_walk() simplifications

Quoted 5 lines · David Hildenbrand

Let's use asce_type where applicable. Also, simplify our sanity check for valid table levels and convert it into a WARN_ON_ONCE(). Check if we even have a valid gmap shadow as the very first step.

Quoted 34 lines
---
 arch/s390/mm/gmap.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
arch/s390/mm/gmap.c +5 -5
index 24ef30fb0833..a2bd8d7792e9 100644
@@ -788,19 +788,19 @@ static inline unsigned long *gmap_table_walk(struct gmap *gmap,
 					     unsigned long gaddr, int level)
 {
 	const int asce_type = gmap->asce & _ASCE_TYPE_MASK;
-	unsigned long *table;
+	unsigned long *table = gmap->table;
 
-	if ((gmap->asce & _ASCE_TYPE_MASK) + 4 < (level * 4))
-		return NULL;
 	if (gmap_is_shadow(gmap) && gmap->removed)
 		return NULL;
 
+	if (WARN_ON_ONCE(level > (asce_type >> 2) + 1))
+		return NULL;
+
 	if (WARN_ON_ONCE(asce_type != _ASCE_TYPE_REGION1 &&
 			 gaddr & (-1UL << (31 + (asce_type >> 2) * 11))))
 		return NULL;
 
-	table = gmap->table;
-	switch (gmap->asce & _ASCE_TYPE_MASK) {
+	switch (asce_type) {
 	case _ASCE_TYPE_REGION1:
 		table += (gaddr & _REGION1_INDEX) >> _REGION1_SHIFT;
 		if (level == 4)
maintainer Reviewed-by on v2 5/5
#20 lore raw

Re: [PATCH v2 5/5] KVM: s390: vsie: gmap_table_walk() simplifications

In reply to David Hildenbrand
Quoted 28 lines · On Fri, 3 Apr 2020 17:30:50 +0200 David Hildenbrand <[email protected]> wrote:

Let's use asce_type where applicable. Also, simplify our sanity check for valid table levels and convert it into a WARN_ON_ONCE(). Check if we even have a valid gmap shadow as the very first step.

---
 arch/s390/mm/gmap.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
arch/s390/mm/gmap.c +4 -3
index 24ef30fb0833..a2bd8d7792e9 100644
@@ -788,19 +788,19 @@ static inline unsigned long
 *gmap_table_walk(struct gmap *gmap, unsigned long gaddr, int level)
 {
 	const int asce_type = gmap->asce & _ASCE_TYPE_MASK;
-	unsigned long *table;
+	unsigned long *table = gmap->table;
 
-	if ((gmap->asce & _ASCE_TYPE_MASK) + 4 < (level * 4))
-		return NULL;
 	if (gmap_is_shadow(gmap) && gmap->removed)
 		return NULL;
 
+	if (WARN_ON_ONCE(level > (asce_type >> 2) + 1))
+		return NULL;
+

arch/s390/mm/gmap.c static inline unsigned long · line 806

Quoted 10 lines
arch/s390/mm/gmap.c +1 -2
 	if (WARN_ON_ONCE(asce_type != _ASCE_TYPE_REGION1 &&
 			 gaddr & (-1UL << (31 + (asce_type >> 2) *
 11)))) return NULL;
 
-	table = gmap->table;
-	switch (gmap->asce & _ASCE_TYPE_MASK) {
+	switch (asce_type) {
 	case _ASCE_TYPE_REGION1:
 		table += (gaddr & _REGION1_INDEX) >> _REGION1_SHIFT;
 		if (level == 4)
#21 lore raw
In reply to David Hildenbrand

Series applied. thanks.
I will schedule the first 3 for master, 4 and 5 for next.

Quoted 35 lines · David Hildenbrand

Some vsie/gmap fixes and two cleanups/improvements.

Patch #1 fixes an issue reported by Janosch. It was never observed so far, because KVM usually doesn't use a region 1 table for it's guest (unless memory would be exceeding something like 16 EB, which isn't even supported by the HW). Older QEMU+KVM or other hypervisors can trigger this.

Patch #2 fixes a code path that probably was never taken and will most probably not be taken very often in the future - unless somebody really messes up the page tables for a guest (or writes a test for it). At some point, a test case for this would be nice.

Patch #3 fixes a rare possible race. Don't think this is stable material.

Gave it some testing with my limited access to somewhat-fast s390x machines. Booted a Linux kernel, supplying all possible number of page table hiearchies.

v1 -> v2:
- "KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks" -- Fix WARN_ON_ONCE
- "gmap_table_walk() simplifications"
-- Also init "table" directly

David Hildenbrand (5):
KVM: s390: vsie: Fix region 1 ASCE sanity shadow address checks
KVM: s390: vsie: Fix delivery of addressing exceptions
KVM: s390: vsie: Fix possible race when shadowing region 3 tables
KVM: s390: vsie: Move conditional reschedule
KVM: s390: vsie: gmap_table_walk() simplifications

 arch/s390/kvm/vsie.c |  4 ++--
 arch/s390/mm/gmap.c  | 17 +++++++++++------
 2 files changed, 13 insertions(+), 8 deletions(-)