KVM: arm/arm64: Allow to use KVM without in-kernel irqchip

Before commit 662d9715840aef44dcb573b0f9fab9e8319c868a ("arm/arm64: KVM: Kill CONFIG_KVM_ARM_{VGIC,TIMER}") is was possible to compile the kernel without vGIC and vTimer support. Commit message says about possibility to detect vGIC support in runtime, but this has never been implemented.

This patch introdices runtime check, restoring the lost functionality. It again allows to use KVM on hardware without vGIC. Interrupt controller has to be emulated in userspace in this case.

(quoted diff) +0 -2

-ENODEV return code from probe function means there's no GIC at all.
-ENXIO happens when, for example, there is GIC node in the device tree,

but it does not specify vGIC resources. Normally this means that vGIC hardware is defunct. Any other error code is still treated as full stop because it might mean some really serious problems.

This patch does not touch any virtual timer code, suggesting that timer hardware is actually in place. Normally on boards in question it is true, however since vGIC is missing, it is impossible to correctly utilize interrupts from the virtual timer. Since virtual timer handling is in

So effectively all we'd need is to set CNTHCTL_EL2.EL1PCEN to 0 for guests that have no in-kernel irqchip, no? We should then trap on all timer accesses and be able to emulate them in user space where we can inject IRQs into an emulated GIC again.

Quoted 4 lines · Alexander Graf

So effectively all we'd need is to set CNTHCTL_EL2.EL1PCEN to 0 for guests that have no in-kernel irqchip, no? We should then trap on all timer accesses and be able to emulate them in user space where we can inject IRQs into an emulated GIC again.

You'd still need to define a new userspace ABI for "we stopped for a system register trap" and the userspace code to emulate the timers as part of KVM rather than as part of TCG, which seems like a lot of effort for a mode that you really don't want to be using...

For GICv3 it gets trickier again because the interface between the interrupt controller and the CPU is no longer as simple as "an FIQ line and an IRQ line". (In particular the interrupt controller needs to know the CPU's current exception level and security state to know if it should raise IRQ or FIQ, which means that it needs to be told every time the CPU changes EL. I haven't yet figured out if I should model this in the QEMU emulated GICv3 by having a backdoor callback function for this or by biting the bullet and putting the GICv3 cpu interface really in the CPU with a properly modelled equivalent of the stream protocol...)

It might make sense to have a generic "system register couldn't get handled" exit code anyway. If nothing else, at least to display unhandled registers in the qemu context where they appear rather than in the kernel log (which a guest could deliberately clutter as it stands today).

Quoted 16 lines · Pavel Fedin

Before commit 662d9715840aef44dcb573b0f9fab9e8319c868a ("arm/arm64: KVM: Kill CONFIG_KVM_ARM_{VGIC,TIMER}") is was possible to compile the kernel without vGIC and vTimer support. Commit message says about possibility to detect vGIC support in runtime, but this has never been implemented.

This patch introdices runtime check, restoring the lost functionality. It again allows to use KVM on hardware without vGIC. Interrupt controller has to be emulated in userspace in this case.

(quoted diff) +0 -2

-ENODEV return code from probe function means there's no GIC at all.
-ENXIO happens when, for example, there is GIC node in the device tree,

but it does not specify vGIC resources. Normally this means that vGIC hardware is defunct. Any other error code is still treated as full stop because it might mean some really serious problems.

Quoted 4 lines

This patch does not touch any virtual timer code, suggesting that timer hardware is actually in place. Normally on boards in question it is true, however since vGIC is missing, it is impossible to correctly utilize interrupts from the virtual timer. Since virtual timer handling is in

So effectively all we'd need is to set CNTHCTL_EL2.EL1PCEN to 0 for guests that have no in-kernel irqchip, no? We should then trap on all timer accesses and be able to emulate them in user space where we can inject IRQs into an emulated GIC again.

Now at least ARM is able to determine whether the machine has virtualization support for irqchip or not at runtime. Obviously, irqfd requires irqchip.

Signed-off-by Pavel Fedin <[email protected]>

---
 virt/kvm/kvm_main.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

virt/kvm/kvm_main.c +0 -0

index 7873d6d..a057d5e 100644
@@ -2716,13 +2716,15 @@ static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
 	case KVM_CAP_INTERNAL_ERROR_DATA:

virt/kvm/kvm_main.c +0 -2

 #endif
-	case KVM_CAP_CHECK_EXTENSION_VM:
-		return 1;
 #ifdef CONFIG_HAVE_KVM_IRQ_ROUTING
 	case KVM_CAP_IRQ_ROUTING:
 		return KVM_MAX_IRQ_ROUTES;

Quoted 6 lines

virt/kvm/kvm_main.c +2 -0

 	case KVM_CAP_INTERNAL_ERROR_DATA:
 #ifdef CONFIG_HAVE_KVM_MSI
 	case KVM_CAP_SIGNAL_MSI:
+		/* Fallthrough */
 #endif
+	case KVM_CAP_CHECK_EXTENSION_VM:

virt/kvm/kvm_main.c static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) · line 2726

Quoted 5 lines

virt/kvm/kvm_main.c +2 -0

+		return 1;
 #ifdef CONFIG_HAVE_KVM_IRQFD
 	case KVM_CAP_IRQFD:
 	case KVM_CAP_IRQFD_RESAMPLE:
+		return kvm_vm_ioctl_check_extension(kvm, KVM_CAP_IRQCHIP);

This won't work for s390, as it doesn't have KVM_CAP_IRQCHIP but KVM_CAP_S390_IRQCHIP (which needs to be enabled).

Hello!

Quoted 6 lines

Quoted 6 lines

virt/kvm/kvm_main.c +2 -0

 	case KVM_CAP_INTERNAL_ERROR_DATA:
 #ifdef CONFIG_HAVE_KVM_MSI
 	case KVM_CAP_SIGNAL_MSI:
+		/* Fallthrough */
 #endif
+	case KVM_CAP_CHECK_EXTENSION_VM:

Quoted 8 lines

virt/kvm/kvm_main.c static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) · line 2726

Quoted 5 lines

virt/kvm/kvm_main.c +2 -0

+		return 1;
 #ifdef CONFIG_HAVE_KVM_IRQFD
 	case KVM_CAP_IRQFD:
 	case KVM_CAP_IRQFD_RESAMPLE:
+		return kvm_vm_ioctl_check_extension(kvm, KVM_CAP_IRQCHIP);

This won't work for s390, as it doesn't have KVM_CAP_IRQCHIP but KVM_CAP_S390_IRQCHIP (which needs to be enabled).

Thank you for the note, i didn't know about irqchip-specific capability codes. There's the same issue with PowerPC, now i
understand why there's no KVM_CAP_IRQCHIP for them. Because they have KVM_CAP_IRQ_MPIC and KVM_CAP_IRQ_XICS, similar to S390.
But isn't it just weird? I understand that perhaps we have some real need to distinguish between different irqchip types, but
shouldn't the kernel also publish KVM_CAP_IRQCHIP, which stands just for "we support some irqchip virtualization"?
May be we should just add this for PowerPC and S390, to make things less ambiguous?

Quoted 9 lines

Thank you for the note, i didn't know about irqchip-specific capability codes. There's the
same issue with PowerPC, now i
understand why there's no KVM_CAP_IRQCHIP for them. Because they have KVM_CAP_IRQ_MPIC and KVM_CAP_IRQ_XICS, similar to S390.
But isn't it just weird? I understand that perhaps we have some real need to distinguish
between different irqchip types, but
shouldn't the kernel also publish KVM_CAP_IRQCHIP, which stands just for "we support some irqchip virtualization"?
May be we should just add this for PowerPC and S390, to make things less ambiguous?

Note that we explicitly need to _enable_ the s390 cap (for compatibility). I'd need to recall the exact details but I came to the conclusion back than that I could not simply enable KVM_CAP_IRQCHIP for s390 (and current qemu would fail to enable the s390 cap if we started advertising KVM_CAP_IRQCHIP now).

Hello!

Quoted 10 lines

Quoted 9 lines

Thank you for the note, i didn't know about irqchip-specific capability codes. There's the
same issue with PowerPC, now i
understand why there's no KVM_CAP_IRQCHIP for them. Because they have KVM_CAP_IRQ_MPIC and KVM_CAP_IRQ_XICS, similar to S390.
But isn't it just weird? I understand that perhaps we have some real need to distinguish
between different irqchip types, but
shouldn't the kernel also publish KVM_CAP_IRQCHIP, which stands just for "we support some irqchip virtualization"?
May be we should just add this for PowerPC and S390, to make things less ambiguous?

Quoted 5 lines

Note that we explicitly need to _enable_ the s390 cap (for compatibility). I'd need to recall the exact details but I came to the conclusion back than that I could not simply enable KVM_CAP_IRQCHIP for s390 (and current qemu would fail to enable the s390 cap if we started advertising KVM_CAP_IRQCHIP now).

OMG... I've looked at the code, what a mess...
If i was implementing this, i'd simply introduce kvm_vm_enable_cap(s, KVM_CAP_IRQCHIP, 0),
which would be allowed to fail with -ENOSYS, so that backwards compatibility is kept and an existing API is reused... But, well, it's already impossible to unscramble an egg... :)
Ok, i think in current situation we could choose one of these ways (both are based on the fact that it's obvious that irqfd require
IRQCHIP).
a) I look for an alternate way to report KVM_CAP_IRQFD dynamically, and maybe PowerPC and S390 follow this way.

b) I simply drop it as it is, because current qemu knows about the dependency and does not try to use irqfd without irqchip,
because there's simply no use for them. But, well, perhaps there would be an exception in vhost, i don't remember testing it.

So what shall we do?

Quoted 1 line

b) I simply drop it as it is, because current qemu knows about the dependency and does not

try to use irqfd without irqchip,

Quoted 1 line

because there's simply no use for them. But, well, perhaps there would be an exception in

vhost, i don't remember testing it.

Wouldn't an irqfd emulation cover vhost?

Quoted 1 line

b) I simply drop it as it is, because current qemu knows about the dependency and does not

try to use irqfd without irqchip,

Quoted 1 line

because there's simply no use for them. But, well, perhaps there would be an exception in

vhost, i don't remember testing it.

Wouldn't an irqfd emulation cover vhost?

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Cornelia Huck
Sent: Monday, November 30, 2015 5:38 PM
To: Pavel Fedin
Cc: [email protected]; [email protected]; 'Marc Zyngier'; 'Christoffer Dall'; 'Gleb Natapov'; 'Paolo Bonzini'
Subject: Re: [PATCH v5 2/2] KVM: Make KVM_CAP_IRQFD dependent on KVM_CAP_IRQCHIP

Quoted 3 lines · On Mon, 30 Nov 2015 15:41:20 +0300 Pavel Fedin <[email protected]> wrote:

Hello!

Quoted 1 line

Quoted 1 line

Thank you for the note, i didn't know about irqchip-specific capability codes. There's

the

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

same issue with PowerPC, now i
understand why there's no KVM_CAP_IRQCHIP for them. Because they have KVM_CAP_IRQ_MPIC

and

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

KVM_CAP_IRQ_XICS, similar to S390.
But isn't it just weird? I understand that perhaps we have some real need to

distinguish

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

between different irqchip types, but
shouldn't the kernel also publish KVM_CAP_IRQCHIP, which stands just for "we support

some

Quoted 9 lines

Quoted 8 lines

Quoted 2 lines

irqchip virtualization"?
May be we should just add this for PowerPC and S390, to make things less ambiguous?

Note that we explicitly need to _enable_ the s390 cap (for compatibility). I'd need to recall the exact details but I came to the conclusion back than that I could not simply enable KVM_CAP_IRQCHIP for s390 (and current qemu would fail to enable the s390 cap if we started advertising KVM_CAP_IRQCHIP now).

Quoted 3 lines

OMG... I've looked at the code, what a mess...
If i was implementing this, i'd simply introduce kvm_vm_enable_cap(s, KVM_CAP_IRQCHIP, 0),
which would be allowed to fail with -ENOSYS, so that backwards compatibility is kept and an

existing API is reused... But, well,

Quoted 2 lines

it's already impossible to unscramble an egg... :)
Ok, i think in current situation we could choose one of these ways (both are based on the

fact that it's obvious that irqfd require

Quoted 2 lines

IRQCHIP).
a) I look for an alternate way to report KVM_CAP_IRQFD dynamically, and maybe PowerPC and

S390 follow this way.

The thing is: _when_ can you report KVM_CAP_IRQFD? It obviously requires an irqchip; but if you need some configuration/enablement beforehand, you'll get different values depending on when you retrieve the cap. So does KVM_CAP_IRQFD mean "irqfds are available in principle" or "everything has been setup for usage of irqfds"? I'd assume the former.

Quoted 1 line

b) I simply drop it as it is, because current qemu knows about the dependency and does not

try to use irqfd without irqchip,

Quoted 1 line

because there's simply no use for them. But, well, perhaps there would be an exception in

vhost, i don't remember testing it.

Wouldn't an irqfd emulation cover vhost?

Quoted 1 line

So what shall we do?

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

It obviously
requires an irqchip; but if you need some configuration/enablement beforehand, you'll get different values depending on when you retrieve the cap. So does KVM_CAP_IRQFD mean "irqfds are available in principle" or "everything has been setup for usage of irqfds"? I'd assume the former.