Handling CP15 timer without in-kernel irqchip
Hello!
Since c2f58514cfb374d5368c9da945f1765cd48eb0da ("arm/arm64: KVM: vgic: Check for
!irqchip_in_kernel() when mapping resources") we can run qemu with kernel_irqchip=off option. The only remaining problem is how to handle CP15 timer in this case.
I have applied my old experimental patches to kernel 4.2.2. Using them, i can run 'virt' machine
with software-emulated GICv2 on GICv3 hardware without v2 backwards compatibility. Also, i studied 4.3 code, and it seems to be feasible to add this support to 4.3 and on.
The only main question now is how would we do it. We can actually use two possible approaches:
1. If there's no in-kernel irqchip, we revert to older timer behavior (ARCH_TIMER_CTRL_IT_MASK
hack), and forward the timer IRQ to userspace using new KVM_EXIT_IRQ return code. Timer IRQ has to be treated as edge-triggered in this case. This is the approach which my current implementation uses.
2. Another possible approach, based on how device tree binding is handled by Linux. It is possible
to remove virtual timer IRQ from the device tree, in this case the kernel reverts to using physical timer. When running under hypervisor, accesses to physical CP15 timer are trapped into HYP, therefore we can forward them to userspace using new exit code, something like KVM_EXIT_REG_ACCESS. In this case the timer would be also emulated by the userspace, which is slower, but allows better emulation. Also, this could be used in order to run some other guests which just expect physical timer to be there.
Both approaches have their own limitations, but anyway this is much better than nothing. What do
you think, and which approach do you like more?
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
Paolo Bonzini
maintainerQuoted 10 lines · Pavel Fedin
2. Another possible approach, based on how device tree binding is handled by Linux. It is possible
to remove virtual timer IRQ from the device tree, in this case the kernel reverts to using physical timer. When running under hypervisor, accesses to physical CP15 timer are trapped into HYP, therefore we can forward them to userspace using new exit code, something likeKVM_EXIT_REG_ACCESS. In this case the timer would be also emulated by the userspace, which is slower, but allows better emulation. Also, this could be used in order to run some other guests which just expect physical timer to be there.Both approaches have their own limitations, but anyway this is much better than nothing. What do
you think, and which approach do you like more?
I like the latter. But I guess one could even do both?
Paolo
Hello!
Quoted 1 line
I like the latter. But I guess one could even do both?
You know, you are right, they absolutely do not contradict each other.
Patch set for #1 is quite not big, if you are interested, and nobody votes strongly against, i can
rebase it to 4.3 and post.
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
Quoted 11 lines · Paolo Bonzini
Quoted 8 lines · Pavel Fedin
2. Another possible approach, based on how device tree binding is handled by Linux. It is possible
to remove virtual timer IRQ from the device tree, in this case the kernel reverts to using physical timer. When running under hypervisor, accesses to physical CP15 timer are trapped into HYP, therefore we can forward them to userspace using new exit code, something likeKVM_EXIT_REG_ACCESS. In this case the timer would be also emulated by the userspace, which is slower, but allows better emulation. Also, this could be used in order to run some other guests which just expect physical timer to be there.
Quoted 4 lines
Quoted 2 lines
Both approaches have their own limitations, but anyway this is much better than nothing. What do
you think, and which approach do you like more?I like the latter. But I guess one could even do both?
I definitely dislike the latter -- userspace ends up having to emulate part of the CPU even though that CPU support is really there in hardware. Also it requires us to edit the device tree, which means it won't work at all on boards other than 'virt' where we use the kernel's device tree rather than creating our own. Better for the kernel to forward the timer interrupts back out to userspace's irq controller.
-- PMM
Paolo Bonzini
maintainerQuoted 17 lines · Peter Maydell
Quoted 11 lines · Paolo Bonzini
Quoted 8 lines · Pavel Fedin
2. Another possible approach, based on how device tree binding is handled by Linux. It is possible
to remove virtual timer IRQ from the device tree, in this case the kernel reverts to using physical timer. When running under hypervisor, accesses to physical CP15 timer are trapped into HYP, therefore we can forward them to userspace using new exit code, something likeKVM_EXIT_REG_ACCESS. In this case the timer would be also emulated by the userspace, which is slower, but allows better emulation. Also, this could be used in order to run some other guests which just expect physical timer to be there.Quoted 4 lines
Quoted 2 lines
Both approaches have their own limitations, but anyway this is much better than nothing. What do
you think, and which approach do you like more?I like the latter. But I guess one could even do both?
Quoted 7 lines
I definitely dislike the latter -- userspace ends up having to emulate part of the CPU even though that CPU support is really there in hardware. Also it requires us to edit the device tree, which means it won't work at all on boards other than 'virt' where we use the kernel's device tree rather than creating our own. Better for the kernel to forward the timer interrupts back out to userspace's irq controller.
How do boards other than 'virt' work when emulated without KVM? It must be possible to emulate the physical timer in QEMU.
Paolo
Quoted 11 lines · Paolo Bonzini
Quoted 7 lines · Peter Maydell
I definitely dislike the latter -- userspace ends up having to emulate part of the CPU even though that CPU support is really there in hardware. Also it requires us to edit the device tree, which means it won't work at all on boards other than 'virt' where we use the kernel's device tree rather than creating our own. Better for the kernel to forward the timer interrupts back out to userspace's irq controller.
How do boards other than 'virt' work when emulated without KVM? It must be possible to emulate the physical timer in QEMU.
Without KVM is easy -- we emulate the physical timer as just one of the parts of the emulated CPU. With KVM, we don't emulate the CPU at all. We don't try to handle a "half TCG half KVM" setup.
thanks
-- PMM
Paolo Bonzini
maintainerQuoted 10 lines · Peter Maydell
Quoted 9 lines · Paolo Bonzini
Quoted 7 lines · Peter Maydell
I definitely dislike the latter -- userspace ends up having to emulate part of the CPU even though that CPU support is really there in hardware. Also it requires us to edit the device tree, which means it won't work at all on boards other than 'virt' where we use the kernel's device tree rather than creating our own. Better for the kernel to forward the timer interrupts back out to userspace's irq controller.
Quoted 6 lines
Quoted 2 lines
How do boards other than 'virt' work when emulated without KVM? It must be possible to emulate the physical timer in QEMU.
Without KVM is easy -- we emulate the physical timer as just one of the parts of the emulated CPU. With KVM, we don't emulate the CPU at all. We don't try to handle a "half TCG half KVM" setup.
I mean in the device tree. Does the boot loader realize it's under a hypervisor, and provide different device trees to the kernel?
Paolo
Hello!
Quoted 2 lines
I mean in the device tree. Does the boot loader realize it's under a hypervisor, and provide different device trees to the kernel?
This has nothing to do with the device tree. Device tree is static. The logic sits inside Linux kernel itself:
a) If we are running in HYP mode (we are host) - use physical timer
b) If there's no virtual timer IRQ specified - use physical timer
c) Otherwise - use virtual vimer.
On systems without virtualization support virtual timer is simply aliased to physical timer, this is architectural requirement. Therefore this logic just works everywhere.
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
Quoted 16 lines · Paolo Bonzini
Quoted 13 lines · Peter Maydell
Quoted 11 lines · Paolo Bonzini
Quoted 7 lines · Peter Maydell
I definitely dislike the latter -- userspace ends up having to emulate part of the CPU even though that CPU support is really there in hardware. Also it requires us to edit the device tree, which means it won't work at all on boards other than 'virt' where we use the kernel's device tree rather than creating our own. Better for the kernel to forward the timer interrupts back out to userspace's irq controller.
How do boards other than 'virt' work when emulated without KVM? It must be possible to emulate the physical timer in QEMU.
Quoted 6 lines
Quoted 3 lines
Without KVM is easy -- we emulate the physical timer as just one of the parts of the emulated CPU. With KVM, we don't emulate the CPU at all. We don't try to handle a "half TCG half KVM" setup.
I mean in the device tree. Does the boot loader realize it's under a hypervisor, and provide different device trees to the kernel?
No. The device tree says "you have both physical and virtual timers", the guest kernel always prefers the virtual timer, and this works for both KVM and TCG.
-- PMM
Hello!
Quoted 1 line
How do boards other than 'virt' work when emulated without KVM?
Under "emulated without KVM" we normally suppose full software emulation (known in qemu as TCG). In this case CPU code is interpreted, and KVM is not used at all.
Some boards (like vexpress) have own platform-specific timer, so they can work under KVM without CP15 timer. In order to make it working one should either disable CP15 timer in guest .config or remove the node from device tree. CP15 timer never worked in this configuration (KVM without irqchip).
Quoted 1 line
It must be possible to emulate the physical timer in QEMU.
Of course it is, and with TCG it works. But we currently cannot use it with KVM because KVM assumes that it emulates all CP15 (or system) registers by itself, and never hands any of them outside.
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
On Fri, 2 Oct 2015 10:28:42 +0300
Pavel Fedin <[email protected]> wrote:
[...]
Quoted 5 lines
1. If there's no in-kernel irqchip, we revert to older timer
behavior (ARCH_TIMER_CTRL_IT_MASKhack), and forward the timer IRQ to userspace using newKVM_EXIT_IRQreturn code. Timer IRQ has to be treated as edge-triggered in this case. This is the approach which my current implementation uses.
No way I'm putting this hack back in. We're working hard enough to make KVM architecture compliant (making the timer LEVEL triggered), and this hack prevents legitimate guests from running properly. So it is gone, never to return.
Quoted 9 lines
2. Another possible approach, based on how device tree binding is
handled by Linux. It is possible to remove virtual timer IRQ from the device tree, in this case the kernel reverts to using physical timer. When running under hypervisor, accesses to physical CP15 timer are trapped into HYP, therefore we can forward them to userspace using new exit code, something likeKVM_EXIT_REG_ACCESS. In this case the timer would be also emulated by the userspace, which is slower, but allows better emulation. Also, this could be used in order to run some other guests which just expect physical timer to be there.
A few things:
- How do you find out that your guest is Linux?
- How do you ensure that the guest will not require the virtual timer?
- Who is going to maintain this new ABI that would exist for the sole benefit of broken hardware?
- What is wrong with exposing another, memory mapped timer and remove the architected timer entirely?
My guess is that this particular ball of hacks will just bitrot as soon as your product has shipped, because the world will have moved on and ditched/reworked the broken HW.
So let me put it another way. The only way I look into this is when we have this particular platform fully supported in mainline.
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny.
Hello!
Quoted 1 line
- How do you find out that your guest is Linux?
Yes, i agree. No way. And it's not host kernel's job anyway.
Quoted 1 line
- How do you ensure that the guest will not require the virtual timer?
Agree again. Except device tree, no way. If the guest has its own hardcoded assumptions, it's
doomed. (*)
Quoted 2 lines
- Who is going to maintain this new ABI that would exist for the sole benefit of broken hardware?
The ABI is simple, just one more exit code. And it's not only for broken hardware. It is quite
legitimate thing to have GICv3 without v2 backwards compatibility.
Quoted 2 lines
- What is wrong with exposing another, memory mapped timer and remove the architected timer entirely?
Well... qemu guys don't want to remove CP15, i already proposed it, and it was very simple thing to
do. :) But, again, this is also device-tree-based, the same concern applies as to (*).
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
Hello!
Quoted 2 lines
So let me put it another way. The only way I look into this is when we have this particular platform fully supported in mainline.
I am sorry for possible misunderstanding. Please give me one more minute to defend myself... So far, we are not putting back timer disable hack. So, i'd like to clarify some things about
variant 2. From kernel's point of view, this is not a hack, but pure feature enhancement. The idea is to add KVM API which would allow to emulate system registers, unhandled by KVM, in userspace. Currently KVM just prints error message about unhandled system register access and feeds guest with "illegal instruction" exception. What i actually propose is to add an API which would allow to handle these things in userspace. This will even be architecture-agnostic, and it can be useful for emulating absolutely any future peripherials which could use system register (or coprocessor, on ARM32) interface.
The rest of timer-related stuff would be needed to be implemented in userspace, and this would have
absolutely nothing to do with kernel. By the way, this would also allow to run under KVM legitimate guests which for some reason expect both timers (are there any? RTOS?)
So, what is your final word? Would you consider this improvement?
Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia
Quoted 5 lines · Pavel Fedin
Hello!
Quoted 2 lines
So let me put it another way. The only way I look into this is when we have this particular platform fully supported in mainline.
Quoted 11 lines
I am sorry for possible misunderstanding. Please give me one more minute to defend myself... So far, we are not putting back timer disable hack. So, i'd like to clarify some things about
variant 2. From kernel's point of view, this is not a hack, but pure feature enhancement. The idea is to add KVM API which would allow to emulate system registers, unhandled by KVM, in userspace. Currently KVM just prints error message about unhandled system register access and feeds guest with "illegal instruction" exception. What i actually propose is to add an API which would allow to handle these things in userspace. This will even be architecture-agnostic, and it can be useful for emulating absolutely any future peripherials which could use system register (or coprocessor, on ARM32) interface.
The rest of timer-related stuff would be needed to be implemented in userspace, and this would have
absolutely nothing to do with kernel.
Except that we now have an ABI that we need to maintain, userspace functionality we need to maintain, we need to ensure that people running VMs don't accidentally use userspace timers with poor performance without noticigin, and there's a high chance that this whole thing will bit-rot; we've seen it happen before.
I appreciate your enthusiasm on this topic, but we are already swamped with a lot of other important things to do, and this is just not very high on the radar at the moment.
Quoted 4 lines
By the way, this would also allow to run under KVM legitimate guests which for some reason expect both timers (are there any? RTOS?)
So, what is your final word? Would you consider this improvement?
I don't want to deny it flat out, and it's hard to say not having seen the code, the ABI definition, and hear what the userspace maintainers think about this.
-Christoffer