Skip to contentSkip to conversation

KVM: PPC: Book3S: Add support for H_IPOLL and H_XIRR_X in XICS emulation

PATCH In review Paul Mackerras 3 participants 6 messages May 24, 2013 – May 30, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: PPC: Book3S: Add support for H_IPOLL and H_XIRR_X in XICS emulation

This adds the remaining two hypercalls defined by PAPR for manipulating the XICS interrupt controller, H_IPOLL and H_XIRR_X. H_IPOLL returns information about the priority and pending interrupts for a virtual cpu, without changing any state. H_XIRR_X is like H_XIRR in that it reads and acknowledges the highest-priority pending interrupt, but it also returns the timestamp (timebase register value) from when the interrupt was first received by the hypervisor. Currently we just return the current time, since we don't do any software queueing of virtual interrupts inside the XICS emulation code.

These hcalls are not currently used by Linux guests, but may be in future.

---
Unfortunately I missed these two hcalls in the previous submissions. It would be good to get this patch into 3.10 so we don't have a kernel version with these calls missing from the API, in case future guest kernels want to use them.

Alex, given you're on vacation at the moment, are you OK with Ben taking this through his tree?

 arch/powerpc/include/asm/hvcall.h |    1 +
 arch/powerpc/kvm/book3s_hv.c      |    2 ++
 arch/powerpc/kvm/book3s_pr_papr.c |    2 ++
 arch/powerpc/kvm/book3s_xics.c    |   29 +++++++++++++++++++++++++++++
 4 files changed, 34 insertions(+)
arch/powerpc/include/asm/hvcall.h +1 -0
index cf4df8e..0c7f2bf 100644
@@ -264,6 +264,7 @@
 #define H_GET_MPP		0x2D4
 #define H_HOME_NODE_ASSOCIATIVITY 0x2EC
 #define H_BEST_ENERGY		0x2F4
+#define H_XIRR_X		0x2FC
 #define H_RANDOM		0x300
 #define H_COP			0x304
 #define H_GET_MPP_X		0x314
arch/powerpc/kvm/book3s_hv.c +2 -0
index 065c9df..7e2059e 100644
@@ -562,6 +562,8 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)
 	case H_CPPR:
 	case H_EOI:
 	case H_IPI:
+	case H_IPOLL:
+	case H_XIRR_X:
 		if (kvmppc_xics_enabled(vcpu)) {
 			ret = kvmppc_xics_hcall(vcpu, req);
 			break;
arch/powerpc/kvm/book3s_pr_papr.c +2 -0
index 3750e3c..91d4b45 100644
@@ -292,6 +292,8 @@ int kvmppc_h_pr(struct kvm_vcpu *vcpu, unsigned long cmd)
 	case H_CPPR:
 	case H_EOI:
 	case H_IPI:
+	case H_IPOLL:
+	case H_XIRR_X:
 		if (kvmppc_xics_enabled(vcpu))
 			return kvmppc_h_pr_xics_hcall(vcpu, cmd);
 		break;
arch/powerpc/kvm/book3s_xics.c +29 -0
index f7a1037..94c1dd4 100644
@@ -650,6 +650,23 @@ static noinline int kvmppc_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
 	return H_SUCCESS;
 }
 
+static int kvmppc_h_ipoll(struct kvm_vcpu *vcpu, unsigned long server)
+{
+	union kvmppc_icp_state state;
+	struct kvmppc_icp *icp;
+
+	icp = vcpu->arch.icp;
+	if (icp->server_num != server) {
+		icp = kvmppc_xics_find_server(vcpu->kvm, server);
+		if (!icp)
+			return H_PARAMETER;
+	}
+	state = ACCESS_ONCE(icp->state);
+	kvmppc_set_gpr(vcpu, 4, ((u32)state.cppr << 24) | state.xisr);
+	kvmppc_set_gpr(vcpu, 5, state.mfrr);
+	return H_SUCCESS;
+}
+
 static noinline void kvmppc_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr)
 {
 	union kvmppc_icp_state old_state, new_state;
@@ -787,6 +804,18 @@ int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 req)
 	if (!xics || !vcpu->arch.icp)
 		return H_HARDWARE;
 
+	/* These requests don't have real-mode implementations at present */
+	switch (req) {
+	case H_XIRR_X:
+		res = kvmppc_h_xirr(vcpu);
+		kvmppc_set_gpr(vcpu, 4, res);
+		kvmppc_set_gpr(vcpu, 5, get_tb());
+		return rc;
+	case H_IPOLL:
+		rc = kvmppc_h_ipoll(vcpu, kvmppc_get_gpr(vcpu, 4));
+		return rc;
+	}
+
 	/* Check for real mode returning too hard */
 	if (xics->real_mode)
 		return kvmppc_xics_rm_complete(vcpu, req);
Signature · 1 line
1.7.10.4
#1 lore raw
Quoted 14 lines · Paul Mackerras

This adds the remaining two hypercalls defined by PAPR for manipulating the XICS interrupt controller, H_IPOLL and H_XIRR_X. H_IPOLL returns information about the priority and pending interrupts for a virtual cpu, without changing any state. H_XIRR_X is like H_XIRR in that it reads and acknowledges the highest-priority pending interrupt, but it also returns the timestamp (timebase register value) from when the interrupt was first received by the hypervisor. Currently we just return the current time, since we don't do any software queueing of virtual interrupts inside the XICS emulation code.

These hcalls are not currently used by Linux guests, but may be in future.

Quoted 8 lines

---
Unfortunately I missed these two hcalls in the previous submissions. It would be good to get this patch into 3.10 so we don't have a kernel version with these calls missing from the API, in case future guest kernels want to use them.

Alex, given you're on vacation at the moment, are you OK with Ben taking this through his tree?

I believe Alex is staying far away from e-mail on his vacation. He's asked me to fill in for him while he's gone.

The patch itself seems reasonable (though I don't know much about XICS, and do have one question...), but I'll leave it up to Gleb/Marcelo/Ben if it should go in for 3.10 and via which tree. I understand the desire to not have an incomplete ABI in a released version, but Linus is already grumbling about how much went into rc3, and you say the hcalls aren't currently used... Are they likely to be used in any timeframe in which we'd reasonably care about 3.10? If so, would the effect of not having them implemented be such that it would be worse than not having in-kernel XICS at all?

Quoted 11 lines
arch/powerpc/kvm/book3s_xics.c +1 -0
@@ -787,6 +804,18 @@ int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32  
 req)
 	if (!xics || !vcpu->arch.icp)
 		return H_HARDWARE;
 
+	/* These requests don't have real-mode implementations at  

present */

arch/powerpc/kvm/book3s_xics.c +4 -0
+	switch (req) {
+	case H_XIRR_X:
+		res = kvmppc_h_xirr(vcpu);
+		kvmppc_set_gpr(vcpu, 4, res);

arch/powerpc/kvm/book3s_xics.c int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 req) · line 821

Quoted 10 lines
arch/powerpc/kvm/book3s_xics.c +7 -0
+		kvmppc_set_gpr(vcpu, 5, get_tb());
+		return rc;
+	case H_IPOLL:
+		rc = kvmppc_h_ipoll(vcpu, kvmppc_get_gpr(vcpu, 4));
+		return rc;
+	}
+
 	/* Check for real mode returning too hard */
 	if (xics->real_mode)
 		return kvmppc_xics_rm_complete(vcpu, req);

Could you explain what's going on here relative to kvmppc_xics_rm_complete()? What does "returning too hard" mean, and why must rm_action not be checked for these hcalls?

-Scott

#2 lore raw
Quoted 10 lines · Scott Wood

I believe Alex is staying far away from e-mail on his vacation. He's asked me to fill in for him while he's gone.

The patch itself seems reasonable (though I don't know much about XICS, and do have one question...), but I'll leave it up to Gleb/Marcelo/Ben if it should go in for 3.10 and via which tree. I understand the desire to not have an incomplete ABI in a released version, but Linus is already grumbling about how much went into rc3, and you say the hcalls aren't currently used... Are they likely to be used in any timeframe in which we'd reasonably care about 3.10?

Yes. I'd like to have them in. Their implementation is actually fairly trivial and they cannot be emulated by qemu if the rest of the XICS is in the kernel, so it's a problem.

Quoted 3 lines

If so, would the
effect of not having them implemented be such that it would be worse than not having in-kernel XICS at all?

Well, that would mean that running that potential future kernel that uses them (or some other OS that already uses them) would not work without the user explicitly disable in kernel irq chip.

Quoted 17 lines
Quoted 17 lines
arch/powerpc/kvm/book3s_xics.c +1 -0
@@ -787,6 +804,18 @@ int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32  
 req)
 	if (!xics || !vcpu->arch.icp)
 		return H_HARDWARE;
 
+	/* These requests don't have real-mode implementations at  

present */

arch/powerpc/kvm/book3s_xics.c +10 -0
+	switch (req) {
+	case H_XIRR_X:
+		res = kvmppc_h_xirr(vcpu);
+		kvmppc_set_gpr(vcpu, 4, res);
+		kvmppc_set_gpr(vcpu, 5, get_tb());
+		return rc;
+	case H_IPOLL:
+		rc = kvmppc_h_ipoll(vcpu, kvmppc_get_gpr(vcpu, 4));
+		return rc;
+	}
Quoted 8 lines

arch/powerpc/kvm/book3s_xics.c int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 req) · line 821

Quoted 4 lines
arch/powerpc/kvm/book3s_xics.c +1 -0
+
 	/* Check for real mode returning too hard */
 	if (xics->real_mode)
 		return kvmppc_xics_rm_complete(vcpu, req);

Could you explain what's going on here relative to kvmppc_xics_rm_complete()? What does "returning too hard" mean, and why must rm_action not be checked for these hcalls?

This is related to how we handle some hcalls in real mode as a fast path. The real-mode stuff cannot handle cases that require for example a re-emit of the interrupt, a reject, etc... so in some cases, it returns H_TOO_HARD which causes KVM to exit and try to handle the hcall again in kernel virtual mode.

When doing so as the result of a XICS hcall, it sets a bit mask of "tasks" to handle in virtual mode (because it will have already partially done the operation, it cannot just re-play the whole hcall).

So when real-mode is supported we must not just call the normal virtual mode version of the hcalls, we instead go to kvmppc_xics_rm_complete() to handle those "tasks".

However, for those 2 "missing" hcalls, we have no real mode implementation at all (we didn't bother, we will do that later if needed, it's purely a performance issue). So we need to fully handle them in virtual mode, and we know there will be no "tasks" to handle in rm_complete.

We could have done a real mode variant but we wanted to keep the patch as small as possible for 3.10.

Cheers,
Ben.

#3 lore raw
Quoted 12 lines · Benjamin Herrenschmidt
Quoted 1 line · Scott Wood

I believe Alex is staying far away from e-mail on his vacation.

He's

Quoted 3 lines

asked me to fill in for him while he's gone.

The patch itself seems reasonable (though I don't know much about

XICS,

Quoted 1 line

and do have one question...), but I'll leave it up to

Gleb/Marcelo/Ben

Quoted 2 lines

if it should go in for 3.10 and via which tree. I understand the desire to not have an incomplete ABI in a released version, but

Quoted 8 lines

Linus

Quoted 3 lines

is already grumbling about how much went into rc3, and you say the hcalls aren't currently used... Are they likely to be used in any timeframe in which we'd reasonably care about 3.10?

Yes. I'd like to have them in. Their implementation is actually fairly trivial and they cannot be emulated by qemu if the rest of the XICS is in the kernel, so it's a problem.

OK. Does it make more sense for you to take it as Paul suggested, or for Gleb or Marcelo to pick it up directly?

Quoted 39 lines
Quoted 12 lines
Quoted 12 lines
(quoted diff) +1 -0
+	/* These requests don't have real-mode implementations at

present */

arch/powerpc/kvm/book3s_xics.c +10 -0
+	switch (req) {
+	case H_XIRR_X:
+		res = kvmppc_h_xirr(vcpu);
+		kvmppc_set_gpr(vcpu, 4, res);
+		kvmppc_set_gpr(vcpu, 5, get_tb());
+		return rc;
+	case H_IPOLL:
+		rc = kvmppc_h_ipoll(vcpu, kvmppc_get_gpr(vcpu, 4));
+		return rc;
+	}
Quoted 8 lines

arch/powerpc/kvm/book3s_xics.c int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 req) · line 821

Quoted 4 lines
arch/powerpc/kvm/book3s_xics.c +1 -0
+
 	/* Check for real mode returning too hard */
 	if (xics->real_mode)
 		return kvmppc_xics_rm_complete(vcpu, req);

Could you explain what's going on here relative to kvmppc_xics_rm_complete()? What does "returning too hard" mean, and why must rm_action not be checked for these hcalls?

This is related to how we handle some hcalls in real mode as a fast path. The real-mode stuff cannot handle cases that require for example a
re-emit of the interrupt, a reject, etc... so in some cases, it returns
H_TOO_HARD which causes KVM to exit and try to handle the hcall again in
kernel virtual mode.

When doing so as the result of a XICS hcall, it sets a bit mask of "tasks" to handle in virtual mode (because it will have already partially done the operation, it cannot just re-play the whole hcall).

So when real-mode is supported we must not just call the normal virtual
mode version of the hcalls, we instead go to kvmppc_xics_rm_complete() to handle those "tasks".

Quoted 6 lines

However, for those 2 "missing" hcalls, we have no real mode implementation at all (we didn't bother, we will do that later if needed, it's purely a performance issue). So we need to fully handle them in virtual mode, and we know there will be no "tasks" to handle in
rm_complete.

Then rm_action should always be 0 for these hcalls, right? So there's no correctness reason to keep the hcalls in separate switch statements. You shave off a few cycles checking rm_action, at the cost of needing to change kvmppc_xics_hcall() if a real-mode version of these hcalls is ever done.

-Scott

#4 lore raw
Quoted 6 lines · Scott Wood
Quoted 3 lines

Yes. I'd like to have them in. Their implementation is actually fairly trivial and they cannot be emulated by qemu if the rest of the XICS is in the kernel, so it's a problem.

OK. Does it make more sense for you to take it as Paul suggested, or for Gleb or Marcelo to pick it up directly?

I'll take it.

Quoted 5 lines

Then rm_action should always be 0 for these hcalls, right? So there's no correctness reason to keep the hcalls in separate switch statements. You shave off a few cycles checking rm_action, at the cost of needing to change kvmppc_xics_hcall() if a real-mode version of these hcalls is ever done.

No, because rm_action will also be 0 if the hcall was fully done in real mode (which can happen, that's our fast path), in which case we do *NOT* want to to be re-done in virtual mode.

That's why we always return whether rm_action is 0 or not when real-mode is enabled.

Cheers,
Ben.

Acked-by
#5 lore raw
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 1 line · Scott Wood
Quoted 1 line

Yes. I'd like to have them in. Their implementation is actually

fairly

Quoted 1 line
Quoted 1 line

trivial and they cannot be emulated by qemu if the rest of the

Quoted 8 lines

XICS is

Quoted 3 lines
Quoted 1 line

in the kernel, so it's a problem.

OK. Does it make more sense for you to take it as Paul suggested,

or

Quoted 1 line

for Gleb or Marcelo to pick it up directly?

I'll take it.

Quoted 8 lines
Quoted 1 line

Then rm_action should always be 0 for these hcalls, right? So

there's

Quoted 2 lines

no correctness reason to keep the hcalls in separate switch statements. You shave off a few cycles checking rm_action, at the

cost

Quoted 2 lines

of needing to change kvmppc_xics_hcall() if a real-mode version of these hcalls is ever done.

Quoted 6 lines

No, because rm_action will also be 0 if the hcall was fully done in real mode (which can happen, that's our fast path), in which case we do *NOT* want to to be re-done in virtual mode.

That's why we always return whether rm_action is 0 or not when real-mode is enabled.

Oh, I misread the code and thought the decision to return was based on the return value of kvmppc_xics_rm_complete. Sorry about that. :-(

-Scott