Skip to contentSkip to conversation

KVM: s390: Permit ESA mode guests in VSIE

PATCH Superseded Eric Farman 4 participants 16 messages Apr 1 – Apr 6

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover KVM: s390: Permit ESA mode guests in VSIE

Hi Hendrik, Janosch, et al,

Attached is a small series to allow ESA-mode guests in s390 VSIE. It continues to expect that z/Architecture is the default, unless opted in by userspace.

I look forward to your thoughts; thanks!

Eric Farman (3):
KVM: s390: vsie: Allow non-zarch guests
KVM: s390: vsie: Disable some bits when in ESA mode
KVM: s390: vsie: Accommodate ESA prefix pages

Hendrik Brueckner (1):
KVM: s390: Add KVM capability for ESA mode guests

 Documentation/virt/kvm/api.rst         |  8 +++++++
 arch/s390/include/asm/kvm_host.h       |  1 +
 arch/s390/include/asm/kvm_host_types.h |  3 +--
 arch/s390/kvm/kvm-s390.c               |  6 ++++++
 arch/s390/kvm/kvm-s390.h               |  5 ++++-
 arch/s390/kvm/vsie.c                   | 29 ++++++++++++++++++++++----
 include/uapi/linux/kvm.h               |  1 +
 7 files changed, 46 insertions(+), 7 deletions(-)
Signature · 1 line
2.51.0
#1 lore raw

PATCH 1/4 KVM: s390: vsie: Allow non-zarch guests

Linux/KVM runs in z/Architecture-only mode. Although z/Architecture is built upon a long history of hardware refinements, any other CPU mode is not permitted.

Allow a userspace to explicitly enable the use of ESA mode for nested guests, otherwise usage will be rejected.

---
 arch/s390/include/asm/kvm_host.h | 1 +
 arch/s390/kvm/vsie.c             | 8 +++++---
 2 files changed, 6 insertions(+), 3 deletions(-)
arch/s390/include/asm/kvm_host.h +1 -0
index 3039c88daa63..8a4f4a39f7a2 100644
@@ -656,6 +656,7 @@ struct kvm_arch {
 	int user_stsi;
 	int user_instr0;
 	int user_operexec;
+	int allow_vsie_esamode;
 	struct s390_io_adapter *adapters[MAX_S390_IO_ADAPTERS];
 	wait_queue_head_t ipte_wq;
 	int ipte_lock_count;
arch/s390/kvm/vsie.c +5 -3
index 72895dddc39a..888370a02ef7 100644
@@ -125,8 +125,9 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 	struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
 	int newflags, cpuflags = atomic_read(&scb_o->cpuflags);
 
-	/* we don't allow ESA/390 guests */
-	if (!(cpuflags & CPUSTAT_ZARCH))
+	newflags = 0;
+	/* we don't allow ESA/390 guests unless explicitly enabled */
+	if (!(cpuflags & CPUSTAT_ZARCH) && !vcpu->kvm->arch.allow_vsie_esamode)
 		return set_validity_icpt(scb_s, 0x0001U);
 
 	if (cpuflags & (CPUSTAT_RRF | CPUSTAT_MCDS))
@@ -135,7 +136,8 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 		return set_validity_icpt(scb_s, 0x0007U);
 
 	/* intervention requests will be set later */
-	newflags = CPUSTAT_ZARCH;
+	if (cpuflags & CPUSTAT_ZARCH)
+		newflags = CPUSTAT_ZARCH;
 	if (cpuflags & CPUSTAT_GED && test_kvm_facility(vcpu->kvm, 8))
 		newflags |= CPUSTAT_GED;
 	if (cpuflags & CPUSTAT_GED2 && test_kvm_facility(vcpu->kvm, 78)) {
Signature · 1 line
2.51.0
maintainer Reviewed-by on v1 1/4
#2 lore raw

Re: [PATCH v1 1/4] KVM: s390: vsie: Allow non-zarch guests

Quoted 30 lines · Eric Farman

Linux/KVM runs in z/Architecture-only mode. Although z/Architecture is built upon a long history of hardware refinements, any other CPU mode is not permitted.

Allow a userspace to explicitly enable the use of ESA mode for nested guests, otherwise usage will be rejected.

---
  arch/s390/include/asm/kvm_host.h | 1 +
  arch/s390/kvm/vsie.c             | 8 +++++---
  2 files changed, 6 insertions(+), 3 deletions(-)
arch/s390/include/asm/kvm_host.h +1 -0
index 3039c88daa63..8a4f4a39f7a2 100644
@@ -656,6 +656,7 @@ struct kvm_arch {
  	int user_stsi;
  	int user_instr0;
  	int user_operexec;
+	int allow_vsie_esamode;
  	struct s390_io_adapter *adapters[MAX_S390_IO_ADAPTERS];
  	wait_queue_head_t ipte_wq;
  	int ipte_lock_count;
arch/s390/kvm/vsie.c +0 -0
index 72895dddc39a..888370a02ef7 100644

arch/s390/kvm/vsie.c static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · line 128

Quoted 6 lines
arch/s390/kvm/vsie.c +1 -2
@@ -125,8 +125,9 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
  	struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
  	int newflags, cpuflags = atomic_read(&scb_o->cpuflags);
 
-	/* we don't allow ESA/390 guests */
-	if (!(cpuflags & CPUSTAT_ZARCH))
+	newflags = 0;

Not sure why you have that here but removed setting newflags below.

arch/s390/kvm/vsie.c static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · line 138

Quoted 10 lines
arch/s390/kvm/vsie.c +2 -1
+	/* we don't allow ESA/390 guests unless explicitly enabled */
+	if (!(cpuflags & CPUSTAT_ZARCH) && !vcpu->kvm->arch.allow_vsie_esamode)
  		return set_validity_icpt(scb_s, 0x0001U);
 
  	if (cpuflags & (CPUSTAT_RRF | CPUSTAT_MCDS))
@@ -135,7 +136,8 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
  		return set_validity_icpt(scb_s, 0x0007U);
 
  	/* intervention requests will be set later */
-	newflags = CPUSTAT_ZARCH;

I'd leave this but set it to 0.

arch/s390/kvm/vsie.c static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · line 143

Quoted 5 lines
arch/s390/kvm/vsie.c +2 -0
+	if (cpuflags & CPUSTAT_ZARCH)
+		newflags = CPUSTAT_ZARCH;
  	if (cpuflags & CPUSTAT_GED && test_kvm_facility(vcpu->kvm, 8))
  		newflags |= CPUSTAT_GED;
  	if (cpuflags & CPUSTAT_GED2 && test_kvm_facility(vcpu->kvm, 78)) {

I can fix that up if you want.

#3 lore raw

Re: [PATCH v1 1/4] KVM: s390: vsie: Allow non-zarch guests

Quoted 34 lines · Janosch Frank
Quoted 33 lines · Eric Farman

Linux/KVM runs in z/Architecture-only mode. Although z/Architecture is built upon a long history of hardware refinements, any other CPU mode is not permitted.

Allow a userspace to explicitly enable the use of ESA mode for nested guests, otherwise usage will be rejected.

---
  arch/s390/include/asm/kvm_host.h | 1 +
  arch/s390/kvm/vsie.c             | 8 +++++---
  2 files changed, 6 insertions(+), 3 deletions(-)
arch/s390/include/asm/kvm_host.h +1 -0
index 3039c88daa63..8a4f4a39f7a2 100644
@@ -656,6 +656,7 @@ struct kvm_arch {
  	int user_stsi;
  	int user_instr0;
  	int user_operexec;
+	int allow_vsie_esamode;
  	struct s390_io_adapter *adapters[MAX_S390_IO_ADAPTERS];
  	wait_queue_head_t ipte_wq;
  	int ipte_lock_count;
arch/s390/kvm/vsie.c +0 -0
index 72895dddc39a..888370a02ef7 100644
@@ -125,8 +125,9 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
  	struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
  	int newflags, cpuflags = atomic_read(&scb_o->cpuflags);
 
Quoted 5 lines

arch/s390/kvm/vsie.c static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · line 128

Quoted 3 lines
arch/s390/kvm/vsie.c +1 -2
-	/* we don't allow ESA/390 guests */
-	if (!(cpuflags & CPUSTAT_ZARCH))
+	newflags = 0;

Not sure why you have that here but removed setting newflags below.

Yeah, I don't have any good reason for it here.

Quoted 14 lines

arch/s390/kvm/vsie.c static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · line 138

Quoted 11 lines
arch/s390/kvm/vsie.c +2 -1
+	/* we don't allow ESA/390 guests unless explicitly enabled */
+	if (!(cpuflags & CPUSTAT_ZARCH) && !vcpu->kvm->arch.allow_vsie_esamode)
  		return set_validity_icpt(scb_s, 0x0001U);
 
  	if (cpuflags & (CPUSTAT_RRF | CPUSTAT_MCDS))
@@ -135,7 +136,8 @@ static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
  		return set_validity_icpt(scb_s, 0x0007U);
 
  	/* intervention requests will be set later */
-	newflags = CPUSTAT_ZARCH;

I'd leave this but set it to 0.

Quoted 7 lines

arch/s390/kvm/vsie.c static int prepare_cpuflags(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · line 143

Quoted 5 lines
arch/s390/kvm/vsie.c +2 -0
+	if (cpuflags & CPUSTAT_ZARCH)
+		newflags = CPUSTAT_ZARCH;
  	if (cpuflags & CPUSTAT_GED && test_kvm_facility(vcpu->kvm, 8))
  		newflags |= CPUSTAT_GED;
  	if (cpuflags & CPUSTAT_GED2 && test_kvm_facility(vcpu->kvm, 78)) {

I can fix that up if you want.

This makes more sense. And if you don't mind, that would be wonderful!

Quoted 2 lines

Thank you!

#4 lore raw

PATCH 2/4 KVM: s390: vsie: Disable some bits when in ESA mode

In reply to Eric Farman

In the event that a nested guest is put in ESA mode, ensure that some bits are scrubbed from the shadow SCB.

---
 arch/s390/kvm/vsie.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/s390/kvm/vsie.c +14 -0
index 888370a02ef7..584fc7803632 100644
@@ -387,6 +387,17 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 	return 0;
 }
 
+static void shadow_esa(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
+{
+	struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
+
+	/* Ensure these bits are indeed turned off */
+	scb_s->eca &= ~ECA_VX;
+	scb_s->ecb &= ~ECB_GS;
+	scb_s->ecb3 &= ~ECB3_RI;
+	scb_s->ecd &= ~ECD_HOSTREGMGMT;
+}
+
 /* shadow (round up/down) the ibc to avoid validity icpt */
 static void prepare_ibc(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 {
@@ -590,6 +601,9 @@ static int shadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 	scb_s->hpid = HPID_VSIE;
 	scb_s->cpnc = scb_o->cpnc;
 
+	if (!(atomic_read(&scb_s->cpuflags) & CPUSTAT_ZARCH))
+		shadow_esa(vcpu, vsie_page);
+
 	prepare_ibc(vcpu, vsie_page);
 	rc = shadow_crycb(vcpu, vsie_page);
 out:
Signature · 1 line
2.51.0
maintainer Reviewed-by on v1 2/4
#5 lore raw

Re: [PATCH v1 2/4] KVM: s390: vsie: Disable some bits when in ESA mode

Am 01.04.26 um 04:09 schrieb Eric Farman:

Quoted 4 lines

In the event that a nested guest is put in ESA mode, ensure that some bits are scrubbed from the shadow SCB.

apart from one thing below

Quoted 11 lines
---
  arch/s390/kvm/vsie.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
arch/s390/kvm/vsie.c +0 -0
index 888370a02ef7..584fc7803632 100644
@@ -387,6 +387,17 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
  	return 0;
  }

arch/s390/kvm/vsie.c static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · line 398

Quoted 10 lines
arch/s390/kvm/vsie.c +9 -0
 
+static void shadow_esa(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
+{
+	struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
+
+	/* Ensure these bits are indeed turned off */
+	scb_s->eca &= ~ECA_VX;
+	scb_s->ecb &= ~ECB_GS;
+	scb_s->ecb3 &= ~ECB3_RI;
+	scb_s->ecd &= ~ECD_HOSTREGMGMT;

shouldnt we also remove the TE bit (transactional execution)?

Quoted 15 lines
arch/s390/kvm/vsie.c +5 -0
+}
+
  /* shadow (round up/down) the ibc to avoid validity icpt */
  static void prepare_ibc(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
  {
@@ -590,6 +601,9 @@ static int shadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
  	scb_s->hpid = HPID_VSIE;
  	scb_s->cpnc = scb_o->cpnc;
 
+	if (!(atomic_read(&scb_s->cpuflags) & CPUSTAT_ZARCH))
+		shadow_esa(vcpu, vsie_page);
+
  	prepare_ibc(vcpu, vsie_page);
  	rc = shadow_crycb(vcpu, vsie_page);
  out:
#6 lore raw

Re: [PATCH v1 2/4] KVM: s390: vsie: Disable some bits when in ESA mode

Quoted 22 lines · Christian Borntraeger

Am 01.04.26 um 04:09 schrieb Eric Farman:

Quoted 4 lines

In the event that a nested guest is put in ESA mode, ensure that some bits are scrubbed from the shadow SCB.

apart from one thing below

Quoted 12 lines
---
  arch/s390/kvm/vsie.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
arch/s390/kvm/vsie.c +0 -0
index 888370a02ef7..584fc7803632 100644
@@ -387,6 +387,17 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
  	return 0;
  }
 
Quoted 11 lines

arch/s390/kvm/vsie.c static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · line 398

Quoted 9 lines
arch/s390/kvm/vsie.c +9 -0
+static void shadow_esa(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
+{
+	struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
+
+	/* Ensure these bits are indeed turned off */
+	scb_s->eca &= ~ECA_VX;
+	scb_s->ecb &= ~ECB_GS;
+	scb_s->ecb3 &= ~ECB3_RI;
+	scb_s->ecd &= ~ECD_HOSTREGMGMT;

shouldnt we also remove the TE bit (transactional execution)?

Ahh, good point. Yeah, that should be removed.

@Janosch, I'll spin a quick v2 with this and the other nits, so you don't have to do the fixups. Just want to doublecheck this one.

Quoted 16 lines
Quoted 16 lines
arch/s390/kvm/vsie.c +5 -0
+}
+
  /* shadow (round up/down) the ibc to avoid validity icpt */
  static void prepare_ibc(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
  {
@@ -590,6 +601,9 @@ static int shadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
  	scb_s->hpid = HPID_VSIE;
  	scb_s->cpnc = scb_o->cpnc;
 
+	if (!(atomic_read(&scb_s->cpuflags) & CPUSTAT_ZARCH))
+		shadow_esa(vcpu, vsie_page);
+
  	prepare_ibc(vcpu, vsie_page);
  	rc = shadow_crycb(vcpu, vsie_page);
  out:
maintainer
#7 lore raw

Re: [PATCH v1 2/4] KVM: s390: vsie: Disable some bits when in ESA mode

Quoted 35 lines · Eric Farman
Quoted 22 lines · Christian Borntraeger

Am 01.04.26 um 04:09 schrieb Eric Farman:

Quoted 4 lines

In the event that a nested guest is put in ESA mode, ensure that some bits are scrubbed from the shadow SCB.

apart from one thing below

Quoted 12 lines
---
   arch/s390/kvm/vsie.c | 14 ++++++++++++++
   1 file changed, 14 insertions(+)
arch/s390/kvm/vsie.c +0 -0
index 888370a02ef7..584fc7803632 100644
@@ -387,6 +387,17 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
   	return 0;
   }
 
Quoted 11 lines

arch/s390/kvm/vsie.c static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page) · line 398

Quoted 9 lines
arch/s390/kvm/vsie.c +9 -0
+static void shadow_esa(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
+{
+	struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
+
+	/* Ensure these bits are indeed turned off */
+	scb_s->eca &= ~ECA_VX;
+	scb_s->ecb &= ~ECB_GS;
+	scb_s->ecb3 &= ~ECB3_RI;
+	scb_s->ecd &= ~ECD_HOSTREGMGMT;

shouldnt we also remove the TE bit (transactional execution)?

Quoted 4 lines

Ahh, good point. Yeah, that should be removed.

@Janosch, I'll spin a quick v2 with this and the other nits, so you don't have to do the fixups. Just want to doublecheck this one.

Great, thanks

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
arch/s390/kvm/vsie.c +5 -0
+}
+
   /* shadow (round up/down) the ibc to avoid validity icpt */
   static void prepare_ibc(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
   {
@@ -590,6 +601,9 @@ static int shadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
   	scb_s->hpid = HPID_VSIE;
   	scb_s->cpnc = scb_o->cpnc;
 
+	if (!(atomic_read(&scb_s->cpuflags) & CPUSTAT_ZARCH))
+		shadow_esa(vcpu, vsie_page);
+
   	prepare_ibc(vcpu, vsie_page);
   	rc = shadow_crycb(vcpu, vsie_page);
   out:
#8 lore raw

PATCH 3/4 KVM: s390: vsie: Accommodate ESA prefix pages

In reply to Eric Farman

The prefix page address occupies a different number of bits for z/Architecture versus ESA mode. Adjust the definition to cover both, and permit an ESA mode address within the nested codepath.

---
 arch/s390/include/asm/kvm_host_types.h | 3 +--
 arch/s390/kvm/kvm-s390.h               | 5 ++++-
 arch/s390/kvm/vsie.c                   | 7 ++++++-
 3 files changed, 11 insertions(+), 4 deletions(-)
arch/s390/include/asm/kvm_host_types.h +1 -2
index 1394d3fb648f..3f50942bdfe6 100644
@@ -137,8 +137,7 @@ struct mcck_volatile_info {
 struct kvm_s390_sie_block {
 	atomic_t cpuflags;		/* 0x0000 */
 	__u32 : 1;			/* 0x0004 */
-	__u32 prefix : 18;
-	__u32 : 1;
+	__u32 prefix : 19;
 	__u32 ibc : 12;
 	__u8	reserved08[4];		/* 0x0008 */
 #define PROG_IN_SIE (1<<0)
arch/s390/kvm/kvm-s390.h +4 -1
index bf1d7798c1af..c0ad8effe3d0 100644
@@ -122,7 +122,9 @@ static inline int kvm_is_ucontrol(struct kvm *kvm)
 #endif
 }
 
-#define GUEST_PREFIX_SHIFT 13
+#define GUEST_PREFIX_SHIFT 12
+#define GUEST_PREFIX_ZARCH 0x7fffe
+#define GUEST_PREFIX_ESA   0x7ffff
 static inline u32 kvm_s390_get_prefix(struct kvm_vcpu *vcpu)
 {
 	return vcpu->arch.sie_block->prefix << GUEST_PREFIX_SHIFT;
@@ -133,6 +135,7 @@ static inline void kvm_s390_set_prefix(struct kvm_vcpu *vcpu, u32 prefix)
 	VCPU_EVENT(vcpu, 3, "set prefix of cpu %03u to 0x%x", vcpu->vcpu_id,
 		   prefix);
 	vcpu->arch.sie_block->prefix = prefix >> GUEST_PREFIX_SHIFT;
+	vcpu->arch.sie_block->prefix &= GUEST_PREFIX_ZARCH;
 	kvm_make_request(KVM_REQ_TLB_FLUSH, vcpu);
 	kvm_make_request(KVM_REQ_REFRESH_GUEST_PREFIX, vcpu);
 }
arch/s390/kvm/vsie.c +6 -1
index 584fc7803632..e45378a01731 100644
@@ -479,7 +479,7 @@ static int shadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 	struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
 	/* READ_ONCE does not work on bitfields - use a temporary variable */
 	const uint32_t __new_prefix = scb_o->prefix;
-	const uint32_t new_prefix = READ_ONCE(__new_prefix);
+	uint32_t new_prefix = READ_ONCE(__new_prefix);
 	const bool wants_tx = READ_ONCE(scb_o->ecb) & ECB_TE;
 	bool had_tx = scb_s->ecb & ECB_TE;
 	unsigned long new_mso = 0;
@@ -527,6 +527,11 @@ static int shadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 
 	scb_s->icpua = scb_o->icpua;
 
+	if (!(atomic_read(&scb_s->cpuflags) & CPUSTAT_ZARCH))
+		new_prefix &= GUEST_PREFIX_ESA;
+	else
+		new_prefix &= GUEST_PREFIX_ZARCH;
+
 	if (!(atomic_read(&scb_s->cpuflags) & CPUSTAT_SM))
 		new_mso = READ_ONCE(scb_o->mso) & 0xfffffffffff00000UL;
 	/* if the hva of the prefix changes, we have to remap the prefix */
Signature · 1 line
2.51.0
maintainer
#9 lore raw

Re: [PATCH v1 3/4] KVM: s390: vsie: Accommodate ESA prefix pages

Quoted 31 lines · Eric Farman

The prefix page address occupies a different number of bits for z/Architecture versus ESA mode. Adjust the definition to cover both, and permit an ESA mode address within the nested codepath.

---
  arch/s390/include/asm/kvm_host_types.h | 3 +--
  arch/s390/kvm/kvm-s390.h               | 5 ++++-
  arch/s390/kvm/vsie.c                   | 7 ++++++-
  3 files changed, 11 insertions(+), 4 deletions(-)
arch/s390/include/asm/kvm_host_types.h +1 -2
index 1394d3fb648f..3f50942bdfe6 100644
@@ -137,8 +137,7 @@ struct mcck_volatile_info {
  struct kvm_s390_sie_block {
  	atomic_t cpuflags;		/* 0x0000 */
  	__u32 : 1;			/* 0x0004 */
-	__u32 prefix : 18;
-	__u32 : 1;
+	__u32 prefix : 19;
  	__u32 ibc : 12;
  	__u8	reserved08[4];		/* 0x0008 */
  #define PROG_IN_SIE (1<<0)
arch/s390/kvm/kvm-s390.h +0 -0
index bf1d7798c1af..c0ad8effe3d0 100644

arch/s390/kvm/kvm-s390.h static inline int kvm_is_ucontrol(struct kvm *kvm) · line 127

Quoted 7 lines
arch/s390/kvm/kvm-s390.h +3 -1
@@ -122,7 +122,9 @@ static inline int kvm_is_ucontrol(struct kvm *kvm)
  #endif
  }
 
-#define GUEST_PREFIX_SHIFT 13
+#define GUEST_PREFIX_SHIFT 12
+#define GUEST_PREFIX_ZARCH 0x7fffe
+#define GUEST_PREFIX_ESA   0x7ffff

Those should be GUEST_PREFIX_MASK_XXX.

GUEST_PREFIX_MASK_ZARCH can be used for set prefix and sigp set prefix emulation at a later time. Those functions currently use numeric constants.

The rest looks fine.

#10 lore raw

Re: [PATCH v1 3/4] KVM: s390: vsie: Accommodate ESA prefix pages

Quoted 35 lines · Janosch Frank
Quoted 34 lines · Eric Farman

The prefix page address occupies a different number of bits for z/Architecture versus ESA mode. Adjust the definition to cover both, and permit an ESA mode address within the nested codepath.

---
  arch/s390/include/asm/kvm_host_types.h | 3 +--
  arch/s390/kvm/kvm-s390.h               | 5 ++++-
  arch/s390/kvm/vsie.c                   | 7 ++++++-
  3 files changed, 11 insertions(+), 4 deletions(-)
arch/s390/include/asm/kvm_host_types.h +1 -2
index 1394d3fb648f..3f50942bdfe6 100644
@@ -137,8 +137,7 @@ struct mcck_volatile_info {
  struct kvm_s390_sie_block {
  	atomic_t cpuflags;		/* 0x0000 */
  	__u32 : 1;			/* 0x0004 */
-	__u32 prefix : 18;
-	__u32 : 1;
+	__u32 prefix : 19;
  	__u32 ibc : 12;
  	__u8	reserved08[4];		/* 0x0008 */
  #define PROG_IN_SIE (1<<0)
arch/s390/kvm/kvm-s390.h +0 -0
index bf1d7798c1af..c0ad8effe3d0 100644
@@ -122,7 +122,9 @@ static inline int kvm_is_ucontrol(struct kvm *kvm)
  #endif
  }
 
Quoted 6 lines

arch/s390/kvm/kvm-s390.h static inline int kvm_is_ucontrol(struct kvm *kvm) · line 127

Quoted 4 lines
arch/s390/kvm/kvm-s390.h +3 -1
-#define GUEST_PREFIX_SHIFT 13
+#define GUEST_PREFIX_SHIFT 12
+#define GUEST_PREFIX_ZARCH 0x7fffe
+#define GUEST_PREFIX_ESA   0x7ffff

Those should be GUEST_PREFIX_MASK_XXX.

Oops, I even thought it would be better with a MASK in there; just forgot to add it.

Quoted 3 lines

GUEST_PREFIX_MASK_ZARCH can be used for set prefix and sigp set prefix emulation at a later time. Those functions currently use numeric constants.

Sure. I'll add that to the followup list.

Quoted 2 lines

The rest looks fine.

#11 lore raw

PATCH 4/4 KVM: s390: Add KVM capability for ESA mode guests

In reply to Eric Farman

From: Hendrik Brueckner <[email protected]>

Now that all the bits are properly addressed, provide a mechanism for testing ESA mode guests in nested configurations.

[EF: Update commit message]
---
 Documentation/virt/kvm/api.rst | 8 ++++++++
 arch/s390/kvm/kvm-s390.c       | 6 ++++++
 include/uapi/linux/kvm.h       | 1 +
 3 files changed, 15 insertions(+)
Documentation/virt/kvm/api.rst +8 -0
index 032516783e96..feabfee0f714 100644
@@ -9436,6 +9436,14 @@ KVM exits with the register state of either the L1 or L2 guest
 depending on which executed at the time of an exit. Userspace must
 take care to differentiate between these cases.
 
+8.47 KVM_CAP_S390_VSIE_ESAMODE
+------------------------------
+
+:Architectures: s390
+
+The presence of this capability indicates that the nested KVM guest can
+start in ESA mode.
+
 9. Known KVM API problems
 =========================
 
arch/s390/kvm/kvm-s390.c +6 -0
index d7838334a338..3856af54b6fe 100644
@@ -629,6 +629,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_IRQFD_RESAMPLE:
 	case KVM_CAP_S390_USER_OPEREXEC:
 	case KVM_CAP_S390_KEYOP:
+	case KVM_CAP_S390_VSIE_ESAMODE:
 		r = 1;
 		break;
 	case KVM_CAP_SET_GUEST_DEBUG2:
@@ -926,6 +927,11 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap)
 		icpt_operexc_on_all_vcpus(kvm);
 		r = 0;
 		break;
+	case KVM_CAP_S390_VSIE_ESAMODE:
+		VM_EVENT(kvm, 3, "%s", "ENABLE: CAP_S390_VSIE_ESAMODE");
+		kvm->arch.allow_vsie_esamode = 1;
+		r = 0;
+		break;
 	default:
 		r = -EINVAL;
 		break;
include/uapi/linux/kvm.h +1 -0
index 80364d4dbebb..9710184d883d 100644
@@ -989,6 +989,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_ARM_SEA_TO_USER 245
 #define KVM_CAP_S390_USER_OPEREXEC 246
 #define KVM_CAP_S390_KEYOP 247
+#define KVM_CAP_S390_VSIE_ESAMODE 248
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
Signature · 1 line
2.51.0
maintainer Reviewed-by on v1 4/4
#12 lore raw

Re: [PATCH v1 4/4] KVM: s390: Add KVM capability for ESA mode guests

Quoted 7 lines · Eric Farman

From: Hendrik Brueckner <[email protected]>

Now that all the bits are properly addressed, provide a mechanism for testing ESA mode guests in nested configurations.

[EF: Update commit message]

Should be:
[[email protected]: Updated commit message]

Quoted 1 line
Quoted 61 lines
---
  Documentation/virt/kvm/api.rst | 8 ++++++++
  arch/s390/kvm/kvm-s390.c       | 6 ++++++
  include/uapi/linux/kvm.h       | 1 +
  3 files changed, 15 insertions(+)
Documentation/virt/kvm/api.rst +8 -0
index 032516783e96..feabfee0f714 100644
@@ -9436,6 +9436,14 @@ KVM exits with the register state of either the L1 or L2 guest
  depending on which executed at the time of an exit. Userspace must
  take care to differentiate between these cases.
 
+8.47 KVM_CAP_S390_VSIE_ESAMODE
+------------------------------
+
+:Architectures: s390
+
+The presence of this capability indicates that the nested KVM guest can
+start in ESA mode.
+
  9. Known KVM API problems
  =========================
 
arch/s390/kvm/kvm-s390.c +6 -0
index d7838334a338..3856af54b6fe 100644
@@ -629,6 +629,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
  	case KVM_CAP_IRQFD_RESAMPLE:
  	case KVM_CAP_S390_USER_OPEREXEC:
  	case KVM_CAP_S390_KEYOP:
+	case KVM_CAP_S390_VSIE_ESAMODE:
  		r = 1;
  		break;
  	case KVM_CAP_SET_GUEST_DEBUG2:
@@ -926,6 +927,11 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap)
  		icpt_operexc_on_all_vcpus(kvm);
  		r = 0;
  		break;
+	case KVM_CAP_S390_VSIE_ESAMODE:
+		VM_EVENT(kvm, 3, "%s", "ENABLE: CAP_S390_VSIE_ESAMODE");
+		kvm->arch.allow_vsie_esamode = 1;
+		r = 0;
+		break;
  	default:
  		r = -EINVAL;
  		break;
include/uapi/linux/kvm.h +1 -0
index 80364d4dbebb..9710184d883d 100644
@@ -989,6 +989,7 @@ struct kvm_enable_cap {
  #define KVM_CAP_ARM_SEA_TO_USER 245
  #define KVM_CAP_S390_USER_OPEREXEC 246
  #define KVM_CAP_S390_KEYOP 247
+#define KVM_CAP_S390_VSIE_ESAMODE 248
 
  struct kvm_irq_routing_irqchip {
  	__u32 irqchip;
#13 lore raw
In reply to Eric Farman
Quoted 7 lines · Eric Farman

Hi Hendrik, Janosch, et al,

Attached is a small series to allow ESA-mode guests in s390 VSIE. It continues to expect that z/Architecture is the default, unless opted in by userspace.

I look forward to your thoughts; thanks!

What is the use case for this? Migrating existin VMs from 31-bit hosts to 64-bit hosts?

maintainer
#14 lore raw
Quoted 12 lines · Christoph Hellwig
Quoted 7 lines · Eric Farman

Hi Hendrik, Janosch, et al,

Attached is a small series to allow ESA-mode guests in s390 VSIE. It continues to expect that z/Architecture is the default, unless opted in by userspace.

I look forward to your thoughts; thanks!

What is the use case for this? Migrating existin VMs from 31-bit hosts to 64-bit hosts?

KVM is the latest hypervisor in a long list on s390. As such we implemented the newest architecture as our starting point and only architecture (zArch/64bit).

There are others which don't start their guests from zArch mode but start them in ESA and slowly move them through modes until they reach zArch. Those other hypervisors often implement emulation for features and devices that KVM doesn't support so there's some demand for using them.

Why do we nest those hypervisors under KVM? We have products that work with the KVM API so it's easier to nest than to re-write the product to use other hypervisors in the first place.

#15 lore raw
Quoted 4 lines · Janosch Frank

KVM is the latest hypervisor in a long list on s390. As such we implemented the newest architecture as our starting point and only architecture (zArch/64bit).

Quoted 8 lines

There are others which don't start their guests from zArch mode but start them in ESA and slowly move them through modes until they reach zArch. Those other hypervisors often implement emulation for features and devices that KVM doesn't support so there's some demand for using them.

Why do we nest those hypervisors under KVM? We have products that work with the KVM API so it's easier to nest than to re-write the product to use other hypervisors in the first place.

Makes sense. Maybe mention this in the cover letter, and also that you're not looking into full device model support for older systems?