Skip to contentSkip to conversation

Fix doorbell emulation for v2 API on PPC

PATCH Applied Gautam Menghani 4 participants 10 messages Jun 5, 2024 – Jun 20, 2024

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v2 cover Fix doorbell emulation for v2 API on PPC

Doorbell emulation for KVM on PAPR guests is broken as support for DPDES was not added in initial patch series [1].
Add DPDES support and doorbell handling support for V2 API.

[1] lore.kernel.org/linuxppc-dev/[email protected]

Changes in v2:
1. Split DPDES support into its own patch

Gautam Menghani (2):
arch/powerpc/kvm: Add DPDES support in helper library for Guest state
buffer
arch/powerpc/kvm: Fix doorbell emulation for v2 API

 Documentation/arch/powerpc/kvm-nested.rst     | 4 +++-
 arch/powerpc/include/asm/guest-state-buffer.h | 3 ++-
 arch/powerpc/include/asm/kvm_book3s.h         | 1 +
 arch/powerpc/kvm/book3s_hv.c                  | 5 +++++
 arch/powerpc/kvm/book3s_hv_nestedv2.c         | 7 +++++++
 arch/powerpc/kvm/test-guest-state-buffer.c    | 2 +-
 6 files changed, 19 insertions(+), 3 deletions(-)
Signature · 1 line
2.45.1
#1 lore raw

PATCH v2 1/2 arch/powerpc/kvm: Add DPDES support in helper library for Guest state buffer

Add support for using DPDES in the library for using guest state buffers. DPDES support is needed for enabling usage of doorbells in a L2 KVM on PAPR guest.

Fixes 6ccbbc33f06a ("KVM: PPC: Add helper library for Guest State Buffers")
---
 Documentation/arch/powerpc/kvm-nested.rst     | 4 +++-
 arch/powerpc/include/asm/guest-state-buffer.h | 3 ++-
 arch/powerpc/include/asm/kvm_book3s.h         | 1 +
 arch/powerpc/kvm/book3s_hv_nestedv2.c         | 7 +++++++
 arch/powerpc/kvm/test-guest-state-buffer.c    | 2 +-
 5 files changed, 14 insertions(+), 3 deletions(-)
Documentation/arch/powerpc/kvm-nested.rst +3 -1
index 630602a8aa00..5defd13cc6c1 100644
@@ -546,7 +546,9 @@ table information.
 +--------+-------+----+--------+----------------------------------+
 | 0x1052 | 0x08  | RW |   T    | CTRL                             |
 +--------+-------+----+--------+----------------------------------+
-| 0x1053-|       |    |        | Reserved                         |
+| 0x1053 | 0x08  | RW |   T    | DPDES                            |
++--------+-------+----+--------+----------------------------------+
+| 0x1054-|       |    |        | Reserved                         |
 | 0x1FFF |       |    |        |                                  |
 +--------+-------+----+--------+----------------------------------+
 | 0x2000 | 0x04  | RW |   T    | CR                               |
arch/powerpc/include/asm/guest-state-buffer.h +2 -1
index 808149f31576..d107abe1468f 100644
@@ -81,6 +81,7 @@
 #define KVMPPC_GSID_HASHKEYR			0x1050
 #define KVMPPC_GSID_HASHPKEYR			0x1051
 #define KVMPPC_GSID_CTRL			0x1052
+#define KVMPPC_GSID_DPDES			0x1053
 
 #define KVMPPC_GSID_CR				0x2000
 #define KVMPPC_GSID_PIDR			0x2001
@@ -110,7 +111,7 @@
 #define KVMPPC_GSE_META_COUNT (KVMPPC_GSE_META_END - KVMPPC_GSE_META_START + 1)
 
 #define KVMPPC_GSE_DW_REGS_START KVMPPC_GSID_GPR(0)
-#define KVMPPC_GSE_DW_REGS_END KVMPPC_GSID_CTRL
+#define KVMPPC_GSE_DW_REGS_END KVMPPC_GSID_DPDES
 #define KVMPPC_GSE_DW_REGS_COUNT \
 	(KVMPPC_GSE_DW_REGS_END - KVMPPC_GSE_DW_REGS_START + 1)
 
arch/powerpc/include/asm/kvm_book3s.h +1 -0
index 3e1e2a698c9e..10618622d7ef 100644
@@ -594,6 +594,7 @@ static inline u##size kvmppc_get_##reg(struct kvm_vcpu *vcpu)		\
 
 
 KVMPPC_BOOK3S_VCORE_ACCESSOR(vtb, 64, KVMPPC_GSID_VTB)
+KVMPPC_BOOK3S_VCORE_ACCESSOR(dpdes, 64, KVMPPC_GSID_DPDES)
 KVMPPC_BOOK3S_VCORE_ACCESSOR_GET(arch_compat, 32, KVMPPC_GSID_LOGICAL_PVR)
 KVMPPC_BOOK3S_VCORE_ACCESSOR_GET(lpcr, 64, KVMPPC_GSID_LPCR)
 KVMPPC_BOOK3S_VCORE_ACCESSOR_SET(tb_offset, 64, KVMPPC_GSID_TB_OFFSET)
arch/powerpc/kvm/book3s_hv_nestedv2.c +7 -0
index 8e6f5355f08b..36863fff2a99 100644
@@ -311,6 +311,10 @@ static int gs_msg_ops_vcpu_fill_info(struct kvmppc_gs_buff *gsb,
 			rc = kvmppc_gse_put_u64(gsb, iden,
 						vcpu->arch.vcore->vtb);
 			break;
+		case KVMPPC_GSID_DPDES:
+			rc = kvmppc_gse_put_u64(gsb, iden,
+						vcpu->arch.vcore->dpdes);
+			break;
 		case KVMPPC_GSID_LPCR:
 			rc = kvmppc_gse_put_u64(gsb, iden,
 						vcpu->arch.vcore->lpcr);
@@ -543,6 +547,9 @@ static int gs_msg_ops_vcpu_refresh_info(struct kvmppc_gs_msg *gsm,
 		case KVMPPC_GSID_VTB:
 			vcpu->arch.vcore->vtb = kvmppc_gse_get_u64(gse);
 			break;
+		case KVMPPC_GSID_DPDES:
+			vcpu->arch.vcore->dpdes = kvmppc_gse_get_u64(gse);
+			break;
 		case KVMPPC_GSID_LPCR:
 			vcpu->arch.vcore->lpcr = kvmppc_gse_get_u64(gse);
 			break;
arch/powerpc/kvm/test-guest-state-buffer.c +1 -1
index 4720b8dc8837..2571ccc618c9 100644
@@ -151,7 +151,7 @@ static void test_gs_bitmap(struct kunit *test)
 		i++;
 	}
 
-	for (u16 iden = KVMPPC_GSID_GPR(0); iden <= KVMPPC_GSID_CTRL; iden++) {
+	for (u16 iden = KVMPPC_GSID_GPR(0); iden <= KVMPPC_GSE_DW_REGS_END; iden++) {
 		kvmppc_gsbm_set(&gsbm, iden);
 		kvmppc_gsbm_set(&gsbm1, iden);
 		KUNIT_EXPECT_TRUE(test, kvmppc_gsbm_test(&gsbm, iden));
Signature · 1 line
2.45.1
reviewer Reviewed-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] arch/powerpc/kvm: Add DPDES support in helper library for Guest state buffer

Quoted 4 lines · Gautam Menghani

Add support for using DPDES in the library for using guest state buffers. DPDES support is needed for enabling usage of doorbells in a L2 KVM on PAPR guest.

Quoted 97 lines
Fixes 6ccbbc33f06a ("KVM: PPC: Add helper library for Guest State Buffers")
---
 Documentation/arch/powerpc/kvm-nested.rst     | 4 +++-
 arch/powerpc/include/asm/guest-state-buffer.h | 3 ++-
 arch/powerpc/include/asm/kvm_book3s.h         | 1 +
 arch/powerpc/kvm/book3s_hv_nestedv2.c         | 7 +++++++
 arch/powerpc/kvm/test-guest-state-buffer.c    | 2 +-
 5 files changed, 14 insertions(+), 3 deletions(-)
Documentation/arch/powerpc/kvm-nested.rst +3 -1
index 630602a8aa00..5defd13cc6c1 100644
@@ -546,7 +546,9 @@ table information.
 +--------+-------+----+--------+----------------------------------+
 | 0x1052 | 0x08  | RW |   T    | CTRL                             |
 +--------+-------+----+--------+----------------------------------+
-| 0x1053-|       |    |        | Reserved                         |
+| 0x1053 | 0x08  | RW |   T    | DPDES                            |
++--------+-------+----+--------+----------------------------------+
+| 0x1054-|       |    |        | Reserved                         |
 | 0x1FFF |       |    |        |                                  |
 +--------+-------+----+--------+----------------------------------+
 | 0x2000 | 0x04  | RW |   T    | CR                               |
arch/powerpc/include/asm/guest-state-buffer.h +2 -1
index 808149f31576..d107abe1468f 100644
@@ -81,6 +81,7 @@
 #define KVMPPC_GSID_HASHKEYR			0x1050
 #define KVMPPC_GSID_HASHPKEYR			0x1051
 #define KVMPPC_GSID_CTRL			0x1052
+#define KVMPPC_GSID_DPDES			0x1053
 
 #define KVMPPC_GSID_CR				0x2000
 #define KVMPPC_GSID_PIDR			0x2001
@@ -110,7 +111,7 @@
 #define KVMPPC_GSE_META_COUNT (KVMPPC_GSE_META_END - KVMPPC_GSE_META_START + 1)
 
 #define KVMPPC_GSE_DW_REGS_START KVMPPC_GSID_GPR(0)
-#define KVMPPC_GSE_DW_REGS_END KVMPPC_GSID_CTRL
+#define KVMPPC_GSE_DW_REGS_END KVMPPC_GSID_DPDES
 #define KVMPPC_GSE_DW_REGS_COUNT \
 	(KVMPPC_GSE_DW_REGS_END - KVMPPC_GSE_DW_REGS_START + 1)
 
arch/powerpc/include/asm/kvm_book3s.h +1 -0
index 3e1e2a698c9e..10618622d7ef 100644
@@ -594,6 +594,7 @@ static inline u##size kvmppc_get_##reg(struct kvm_vcpu *vcpu)		\
 
 
 KVMPPC_BOOK3S_VCORE_ACCESSOR(vtb, 64, KVMPPC_GSID_VTB)
+KVMPPC_BOOK3S_VCORE_ACCESSOR(dpdes, 64, KVMPPC_GSID_DPDES)
 KVMPPC_BOOK3S_VCORE_ACCESSOR_GET(arch_compat, 32, KVMPPC_GSID_LOGICAL_PVR)
 KVMPPC_BOOK3S_VCORE_ACCESSOR_GET(lpcr, 64, KVMPPC_GSID_LPCR)
 KVMPPC_BOOK3S_VCORE_ACCESSOR_SET(tb_offset, 64, KVMPPC_GSID_TB_OFFSET)
arch/powerpc/kvm/book3s_hv_nestedv2.c +7 -0
index 8e6f5355f08b..36863fff2a99 100644
@@ -311,6 +311,10 @@ static int gs_msg_ops_vcpu_fill_info(struct kvmppc_gs_buff *gsb,
 			rc = kvmppc_gse_put_u64(gsb, iden,
 						vcpu->arch.vcore->vtb);
 			break;
+		case KVMPPC_GSID_DPDES:
+			rc = kvmppc_gse_put_u64(gsb, iden,
+						vcpu->arch.vcore->dpdes);
+			break;
 		case KVMPPC_GSID_LPCR:
 			rc = kvmppc_gse_put_u64(gsb, iden,
 						vcpu->arch.vcore->lpcr);
@@ -543,6 +547,9 @@ static int gs_msg_ops_vcpu_refresh_info(struct kvmppc_gs_msg *gsm,
 		case KVMPPC_GSID_VTB:
 			vcpu->arch.vcore->vtb = kvmppc_gse_get_u64(gse);
 			break;
+		case KVMPPC_GSID_DPDES:
+			vcpu->arch.vcore->dpdes = kvmppc_gse_get_u64(gse);
+			break;
 		case KVMPPC_GSID_LPCR:
 			vcpu->arch.vcore->lpcr = kvmppc_gse_get_u64(gse);
 			break;
arch/powerpc/kvm/test-guest-state-buffer.c +1 -1
index 4720b8dc8837..2571ccc618c9 100644
@@ -151,7 +151,7 @@ static void test_gs_bitmap(struct kunit *test)
 		i++;
 	}
 
-	for (u16 iden = KVMPPC_GSID_GPR(0); iden <= KVMPPC_GSID_CTRL; iden++) {
+	for (u16 iden = KVMPPC_GSID_GPR(0); iden <= KVMPPC_GSE_DW_REGS_END; iden++) {
 		kvmppc_gsbm_set(&gsbm, iden);
 		kvmppc_gsbm_set(&gsbm1, iden);
 		KUNIT_EXPECT_TRUE(test, kvmppc_gsbm_test(&gsbm, iden));
#3 lore raw

PATCH v2 2/2 arch/powerpc/kvm: Fix doorbell emulation for v2 API

In reply to Gautam Menghani

Doorbell emulation is broken for KVM on PAPR guests as support for DPDES was not added in the initial patch series. Due to this, a KVM on PAPR guest with SMT > 1 cannot be booted with the XICS interrupt controller as doorbells are setup in the initial probe path when using XICS (pSeries_smp_probe()).

Command to replicate the above bug:

qemu-system-ppc64 \
	-drive file=rhel.qcow2,format=qcow2 \
	-m 20G \
	-smp 8,cores=1,threads=8 \
	-cpu  host \
	-nographic \
	-machine pseries,ic-mode=xics -accel kvm

Add doorbell state handling support in the host
KVM code to fix doorbell emulation.

Fixes 19d31c5f1157 ("KVM: PPC: Add support for nestedv2 guests")
---
 arch/powerpc/kvm/book3s_hv.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/powerpc/kvm/book3s_hv.c +5 -0
index 35cb014a0c51..21c69647d27c 100644
@@ -4116,6 +4116,11 @@ static int kvmhv_vcpu_entry_nestedv2(struct kvm_vcpu *vcpu, u64 time_limit,
 	int trap;
 	long rc;
 
+	if (vcpu->arch.doorbell_request) {
+		vcpu->arch.doorbell_request = 0;
+		kvmppc_set_dpdes(vcpu, 1);
+	}
+
 	io = &vcpu->arch.nestedv2_io;
 
 	msr = mfmsr();
Signature · 1 line
2.45.1
reviewer Reviewed-by on v2 2/2
#4 lore raw

Re: [PATCH v2 2/2] arch/powerpc/kvm: Fix doorbell emulation for v2 API

Quoted 16 lines · Gautam Menghani

Doorbell emulation is broken for KVM on PAPR guests as support for DPDES was not added in the initial patch series. Due to this, a KVM on PAPR guest with SMT > 1 cannot be booted with the XICS interrupt controller as doorbells are setup in the initial probe path when using XICS (pSeries_smp_probe()).

Command to replicate the above bug:

qemu-system-ppc64 \
	-drive file=rhel.qcow2,format=qcow2 \
	-m 20G \
	-smp 8,cores=1,threads=8 \
	-cpu  host \
	-nographic \
	-machine pseries,ic-mode=xics -accel kvm
Quoted 2 lines

Add doorbell state handling support in the host
KVM code to fix doorbell emulation.

Quoted 24 lines
Fixes 19d31c5f1157 ("KVM: PPC: Add support for nestedv2 guests")
---
 arch/powerpc/kvm/book3s_hv.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/powerpc/kvm/book3s_hv.c +5 -0
index 35cb014a0c51..21c69647d27c 100644
@@ -4116,6 +4116,11 @@ static int kvmhv_vcpu_entry_nestedv2(struct kvm_vcpu *vcpu, u64 time_limit,
 	int trap;
 	long rc;
 
+	if (vcpu->arch.doorbell_request) {
+		vcpu->arch.doorbell_request = 0;
+		kvmppc_set_dpdes(vcpu, 1);
+	}
+
 	io = &vcpu->arch.nestedv2_io;
 
 	msr = mfmsr();
#5 lore raw
In reply to Gautam Menghani
Quoted 3 lines · Gautam Menghani

Doorbell emulation for KVM on PAPR guests is broken as support for DPDES was not added in initial patch series [1].
Add DPDES support and doorbell handling support for V2 API.

Looks good, thanks. So fix for v1 doorbells is coming?

Thanks,
Nick

Quoted 18 lines

[1] lore.kernel.org/linuxppc-dev/[email protected]

Changes in v2:
1. Split DPDES support into its own patch

Gautam Menghani (2):
arch/powerpc/kvm: Add DPDES support in helper library for Guest state
buffer
arch/powerpc/kvm: Fix doorbell emulation for v2 API

 Documentation/arch/powerpc/kvm-nested.rst     | 4 +++-
 arch/powerpc/include/asm/guest-state-buffer.h | 3 ++-
 arch/powerpc/include/asm/kvm_book3s.h         | 1 +
 arch/powerpc/kvm/book3s_hv.c                  | 5 +++++
 arch/powerpc/kvm/book3s_hv_nestedv2.c         | 7 +++++++
 arch/powerpc/kvm/test-guest-state-buffer.c    | 2 +-
 6 files changed, 19 insertions(+), 3 deletions(-)
#6 lore raw
Quoted 6 lines · Nicholas Piggin
Quoted 3 lines · Gautam Menghani

Doorbell emulation for KVM on PAPR guests is broken as support for DPDES was not added in initial patch series [1].
Add DPDES support and doorbell handling support for V2 API.

Looks good, thanks. So fix for v1 doorbells is coming?

Yes I've root caused the doorbell breakage in V1 API to commit 7c3ded5735141ff4d049747c9f76672a8b737c49. I'll send out a fix soon.

Thanks,
Gautam

#7 lore raw
In reply to Gautam Menghani
Quoted 14 lines · Gautam Menghani

Doorbell emulation for KVM on PAPR guests is broken as support for DPDES was not added in initial patch series [1].
Add DPDES support and doorbell handling support for V2 API.

[1] lore.kernel.org/linuxppc-dev/[email protected]

Changes in v2:
1. Split DPDES support into its own patch

Gautam Menghani (2):
arch/powerpc/kvm: Add DPDES support in helper library for Guest state
buffer
arch/powerpc/kvm: Fix doorbell emulation for v2 API

Quoted 11 lines
 Documentation/arch/powerpc/kvm-nested.rst     | 4 +++-
 arch/powerpc/include/asm/guest-state-buffer.h | 3 ++-
 arch/powerpc/include/asm/kvm_book3s.h         | 1 +
 arch/powerpc/kvm/book3s_hv.c                  | 5 +++++
 arch/powerpc/kvm/book3s_hv_nestedv2.c         | 7 +++++++
 arch/powerpc/kvm/test-guest-state-buffer.c    | 2 +-
 6 files changed, 19 insertions(+), 3 deletions(-)
Signature · 1 line
2.45.1

Hi Michael,

This patch series is to be backported for all kernels >= 6.7. So the tag should be
Cc: [email protected] # v6.7+

and not
Cc: [email protected] # v6.7

Should I send a new version of this series or can you please make this change when pulling in your tree?

Thanks,
Gautam

#8 lore raw
Quoted 34 lines · Gautam Menghani
Quoted 14 lines · Gautam Menghani

Doorbell emulation for KVM on PAPR guests is broken as support for DPDES was not added in initial patch series [1].
Add DPDES support and doorbell handling support for V2 API.

[1] lore.kernel.org/linuxppc-dev/[email protected]

Changes in v2:
1. Split DPDES support into its own patch

Gautam Menghani (2):
arch/powerpc/kvm: Add DPDES support in helper library for Guest state
buffer
arch/powerpc/kvm: Fix doorbell emulation for v2 API

Quoted 11 lines
 Documentation/arch/powerpc/kvm-nested.rst     | 4 +++-
 arch/powerpc/include/asm/guest-state-buffer.h | 3 ++-
 arch/powerpc/include/asm/kvm_book3s.h         | 1 +
 arch/powerpc/kvm/book3s_hv.c                  | 5 +++++
 arch/powerpc/kvm/book3s_hv_nestedv2.c         | 7 +++++++
 arch/powerpc/kvm/test-guest-state-buffer.c    | 2 +-
 6 files changed, 19 insertions(+), 3 deletions(-)
Signature · 1 line
2.45.1

Hi Michael,

This patch series is to be backported for all kernels >= 6.7. So the tag should be
Cc: [email protected] # v6.7+

Quoted 5 lines

and not
Cc: [email protected] # v6.7

Should I send a new version of this series or can you please make this change when pulling in your tree?

I can make the change when applying.

cheers

#9 lore raw
In reply to Gautam Menghani
Quoted 10 lines · Gautam Menghani

Doorbell emulation for KVM on PAPR guests is broken as support for DPDES was not added in initial patch series [1].
Add DPDES support and doorbell handling support for V2 API.

[1] lore.kernel.org/linuxppc-dev/[email protected]

Changes in v2:
1. Split DPDES support into its own patch

[...]

Applied to powerpc/topic/ppc-kvm.

[1/2] arch/powerpc/kvm: Add DPDES support in helper library for Guest state buffer

      https://git.kernel.org/powerpc/c/55dfb8bed6fe8bda390cc71cca878d11a9407099
[2/2] arch/powerpc/kvm: Fix doorbell emulation for v2 API
      https://git.kernel.org/powerpc/c/54ec2bd9e0173b75daf84675d07c56584f96564b

cheers