VCPU hotplug on KVM/ARM

I hope it is the right forum to post my query.

I am currently looking at the possibility of adding a new VCPU to a running guest VM in KVM/ARM. I see that currently, it is not allowed to add a new VCPU to a guest VM, if it is already initialized. The first check in kvm_arch_vcpu_create() returns failure if it is already initialized.

There was some work done in QEMU to add support for VCPU hotplug:
https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html

But I am looking at the KVM side for enabling adding a new VCPU. If you can point me to any relevant work/resources, which I can refer to then it will help me.

Hi Bhupinder,

Quoted 9 lines · On Tue, Feb 27, 2018 at 03:01:17PM +0530, [email protected] wrote:

I hope it is the right forum to post my query.

I am currently looking at the possibility of adding a new VCPU to a running guest VM in KVM/ARM. I see that currently, it is not allowed to add a new VCPU to a guest VM, if it is already initialized. The first check in kvm_arch_vcpu_create() returns failure if it is already initialized.

This would require a major rework of a lot of logic surrounding the GIC and other parts of KVM initialization.

Quoted 11 lines

There was some work done in QEMU to add support for VCPU hotplug:
https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html

But I am looking at the KVM side for enabling adding a new VCPU. If you can point me to any relevant work/resources, which I can refer to then it will help me.

I don't have any specific pointers, but I was always told that the way we were going to do CPU hotplug would be to instantiate a large number of VCPUs, and hotplug would be equivalent to turning on a VCPU which was previously powered off.

Is this not still a feasible solution?

How does VCPU hotplug work on x86?

Thanks,
-Christoffer

Hi Christoffer,

Thanks for your reply.

Quoted 37 lines · Christoffer Dall

Hi Bhupinder,

Quoted 11 lines · On Tue, Feb 27, 2018 at 03:01:17PM +0530, [email protected] wrote:

I hope it is the right forum to post my query.

I am currently looking at the possibility of adding a new VCPU to a running
guest VM in KVM/ARM. I see that currently, it is not allowed to add a new
VCPU to a guest VM, if it is already initialized. The first check in kvm_arch_vcpu_create() returns failure if it is already initialized.

This would require a major rework of a lot of logic surrounding the GIC and other parts of KVM initialization.

Quoted 7 lines

There was some work done in QEMU to add support for VCPU hotplug:
https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html

Quoted 6 lines

But I am looking at the KVM side for enabling adding a new VCPU. If you can
point me to any relevant work/resources, which I can refer to then it will
help me.

I don't have any specific pointers, but I was always told that the way we were going to do CPU hotplug would be to instantiate a large number of VCPUs, and hotplug would be equivalent to turning on a VCPU which was previously powered off.

Quoted 1 line

Is this not still a feasible solution?

It should be a feasible solution provided the guest VM is not able to control the onlining/offlining of VCPUs. It should be controlled by the Host.

Quoted 2 lines

How does VCPU hotplug work on x86?

On x86, you can add a vcpu through libvirt setvcpu command and it shows up in the guest VM as a new CPU if you do lscpu.

Quoted 42 lines · On Tue, Feb 27, 2018 at 05:34:28PM +0530, [email protected] wrote:

Hi Christoffer,

Thanks for your reply.

Quoted 37 lines · Christoffer Dall

Hi Bhupinder,

Quoted 11 lines · On Tue, Feb 27, 2018 at 03:01:17PM +0530, [email protected] wrote:

I hope it is the right forum to post my query.

I am currently looking at the possibility of adding a new VCPU to a running
guest VM in KVM/ARM. I see that currently, it is not allowed to add a new
VCPU to a guest VM, if it is already initialized. The first check in kvm_arch_vcpu_create() returns failure if it is already initialized.

This would require a major rework of a lot of logic surrounding the GIC and other parts of KVM initialization.

Quoted 7 lines

There was some work done in QEMU to add support for VCPU hotplug:
https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html

Quoted 6 lines

But I am looking at the KVM side for enabling adding a new VCPU. If you can
point me to any relevant work/resources, which I can refer to then it will
help me.

I don't have any specific pointers, but I was always told that the way we were going to do CPU hotplug would be to instantiate a large number of VCPUs, and hotplug would be equivalent to turning on a VCPU which was previously powered off.

Quoted 5 lines

Quoted 1 line

Is this not still a feasible solution?

It should be a feasible solution provided the guest VM is not able to control the onlining/offlining of VCPUs. It should be controlled by the Host.

KVM could simply refuse to turn on some of the CPUs unless given permission from host userspace.

Quoted 1 line

Quoted 1 line

Quoted 4 lines

Quoted 1 line

How does VCPU hotplug work on x86?

On x86, you can add a vcpu through libvirt setvcpu command and it shows up in the guest VM as a new CPU if you do lscpu.

Sure, but what is the mechanism, does x86 qemu actually call KVM_CREATE_VCPU, or is this also a question of turning on already created vcpus ?

Quoted 53 lines · Christoffer Dall

Quoted 42 lines · On Tue, Feb 27, 2018 at 05:34:28PM +0530, [email protected] wrote:

Hi Christoffer,

Thanks for your reply.

Quoted 37 lines · Christoffer Dall

Hi Bhupinder,

Quoted 11 lines · On Tue, Feb 27, 2018 at 03:01:17PM +0530, [email protected] wrote:

I hope it is the right forum to post my query.

I am currently looking at the possibility of adding a new VCPU to a running
guest VM in KVM/ARM. I see that currently, it is not allowed to add a new
VCPU to a guest VM, if it is already initialized. The first check in kvm_arch_vcpu_create() returns failure if it is already initialized.

This would require a major rework of a lot of logic surrounding the GIC and other parts of KVM initialization.

Quoted 7 lines

There was some work done in QEMU to add support for VCPU hotplug:
https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html

Quoted 6 lines

But I am looking at the KVM side for enabling adding a new VCPU. If you can
point me to any relevant work/resources, which I can refer to then it will
help me.

I don't have any specific pointers, but I was always told that the way we were going to do CPU hotplug would be to instantiate a large number of VCPUs, and hotplug would be equivalent to turning on a VCPU which was previously powered off.

Quoted 5 lines

Quoted 1 line

Is this not still a feasible solution?

It should be a feasible solution provided the guest VM is not able to control the onlining/offlining of VCPUs. It should be controlled by the Host.

KVM could simply refuse to turn on some of the CPUs unless given permission from host userspace.

Quoted 1 line

Quoted 1 line

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

How does VCPU hotplug work on x86?

On x86, you can add a vcpu through libvirt setvcpu command and it shows up in the guest VM as a new CPU if you do lscpu.

Sure, but what is the mechanism, does x86 qemu actually call KVM_CREATE_VCPU, or is this also a question of turning on already created vcpus ?

CC'ing Igor and qemu-devel

drew

In QEMU on x86 (and I think ppc, s390 as well), we create vCPUs on demand.

It would be nice if ARM would be able to do that too, so that it could take advantage of the same code.

Quoted 3 lines · Igor Mammedov

In QEMU on x86 (and I think ppc, s390 as well), we create vCPUs on demand.>> It would be nice if ARM would be able to do that too,
so that it could take advantage of the same code.

It's not clear to me how that would work, given that for instance the interrupt controller wants to know up-front how many CPUs it has to deal with.

thanks
-- PMM

Quoted 5 lines · Peter Maydell

Quoted 3 lines · Igor Mammedov

In QEMU on x86 (and I think ppc, s390 as well), we create vCPUs on demand.>> It would be nice if ARM would be able to do that too,
so that it could take advantage of the same code.

Quoted 4 lines

It's not clear to me how that would work, given that for instance the interrupt controller wants to know up-front how many CPUs it has to deal with.

So how is cpu hotplug handled in HW? Or doesn't it even exist there?

(we have max_cpus for the interrupt controller, but not sure if that is what we want)

Quoted 48 lines · Christoffer Dall

Quoted 39 lines · On Tue, Feb 27, 2018 at 05:34:28PM +0530, [email protected] wrote:

Hi Christoffer,

Thanks for your reply.

Quoted 34 lines · Christoffer Dall

Hi Bhupinder,

Quoted 11 lines · On Tue, Feb 27, 2018 at 03:01:17PM +0530, [email protected] wrote:

I hope it is the right forum to post my query.

I am currently looking at the possibility of adding a new VCPU to a running
guest VM in KVM/ARM. I see that currently, it is not allowed to add a new
VCPU to a guest VM, if it is already initialized. The first check in kvm_arch_vcpu_create() returns failure if it is already initialized.

This would require a major rework of a lot of logic surrounding the GIC and other parts of KVM initialization.

Quoted 6 lines

There was some work done in QEMU to add support for VCPU hotplug:
https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html

Quoted 6 lines

But I am looking at the KVM side for enabling adding a new VCPU. If you can
point me to any relevant work/resources, which I can refer to then it will
help me.

I don't have any specific pointers, but I was always told that the way we were going to do CPU hotplug would be to instantiate a large number of VCPUs, and hotplug would be equivalent to turning on a VCPU which was previously powered off.

Quoted 5 lines

Quoted 1 line

Is this not still a feasible solution?

It should be a feasible solution provided the guest VM is not able to control the onlining/offlining of VCPUs. It should be controlled by the Host.

KVM could simply refuse to turn on some of the CPUs unless given permission from host userspace.

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

How does VCPU hotplug work on x86?

On x86, you can add a vcpu through libvirt setvcpu command and it shows up in the guest VM as a new CPU if you do lscpu.

Sure, but what is the mechanism, does x86 qemu actually call KVM_CREATE_VCPU, or is this also a question of turning on already created vcpus ?

CC'ing Igor and qemu-devel

drew
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

It's been a few months since this email thread died off. Has anyone started working on a potential solution that would allow VCPU hotplug on KVM/ARM ? Or is this a project that is still waiting for an owner who has the time and inclination to get started?

Thanks,
-Maran

Quoted 64 lines · Andrew Jones

Quoted 48 lines · Christoffer Dall

Quoted 39 lines · On Tue, Feb 27, 2018 at 05:34:28PM +0530, [email protected] wrote:

Hi Christoffer,

Thanks for your reply.

Quoted 34 lines · Christoffer Dall

Hi Bhupinder,

Quoted 11 lines · On Tue, Feb 27, 2018 at 03:01:17PM +0530, [email protected] wrote:

I hope it is the right forum to post my query.

I am currently looking at the possibility of adding a new VCPU to a running
guest VM in KVM/ARM. I see that currently, it is not allowed to add a new
VCPU to a guest VM, if it is already initialized. The first check in kvm_arch_vcpu_create() returns failure if it is already initialized.

This would require a major rework of a lot of logic surrounding the GIC and other parts of KVM initialization.

Quoted 6 lines

There was some work done in QEMU to add support for VCPU hotplug:
https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html

Quoted 6 lines

But I am looking at the KVM side for enabling adding a new VCPU. If you can
point me to any relevant work/resources, which I can refer to then it will
help me.

I don't have any specific pointers, but I was always told that the way we were going to do CPU hotplug would be to instantiate a large number of VCPUs, and hotplug would be equivalent to turning on a VCPU which was previously powered off.

Quoted 5 lines

Quoted 1 line

Is this not still a feasible solution?

It should be a feasible solution provided the guest VM is not able to control the onlining/offlining of VCPUs. It should be controlled by the Host.

KVM could simply refuse to turn on some of the CPUs unless given permission from host userspace.

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

How does VCPU hotplug work on x86?

On x86, you can add a vcpu through libvirt setvcpu command and it shows up in the guest VM as a new CPU if you do lscpu.

Sure, but what is the mechanism, does x86 qemu actually call KVM_CREATE_VCPU, or is this also a question of turning on already created vcpus ?

CC'ing Igor and qemu-devel

drew
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

It's been a few months since this email thread died off. Has anyone started working on a potential solution that would allow VCPU hotplug on KVM/ARM ? Or is this a project that is still waiting for an owner who has the time and inclination to get started?

Quoted 4 lines · Maran Wilson

It's been a few months since this email thread died off. Has anyone started working on a potential solution that would allow VCPU hotplug on KVM/ARM ? Or is this a project that is still waiting for an owner who has the time and inclination to get started?

This is typically a project for someone who would have this particular itch to scratch, and who has a demonstrable need for this functionality.

Work wise, it would have to include adding physical CPU hotplug support to the arm64 kernel as a precondition, before worrying about doing it in KVM.

For KVM itself, particular area of interests would be:
- Making GICv3 redistributors magically appear in the IPA space
- Live resizing of GICv3 structures
- Dynamic allocation of MPIDR, and mapping with vcpu_id

This should keep someone busy for a good couple of weeks (give or take a few months).

That being said, I'd rather see support in QEMU first, creating all the vcpu/redistributors upfront, and signalling the hotplug event via the virtual firmware. And then post some numbers to show that creating all the vcpus upfront is not acceptable.

Quoted 9 lines · Marc Zyngier

Quoted 4 lines · Maran Wilson

It's been a few months since this email thread died off. Has anyone started working on a potential solution that would allow VCPU hotplug on KVM/ARM ? Or is this a project that is still waiting for an owner who has the time and inclination to get started?

This is typically a project for someone who would have this particular itch to scratch, and who has a demonstrable need for this functionality.

Quoted 8 lines

Work wise, it would have to include adding physical CPU hotplug support to the arm64 kernel as a precondition, before worrying about doing it in KVM.

For KVM itself, particular area of interests would be:
- Making GICv3 redistributors magically appear in the IPA space
- Live resizing of GICv3 structures
- Dynamic allocation of MPIDR, and mapping with vcpu_id

I have CPU topology description patches on the QEMU list now[*]. A next step for me is to this MPIDR work. I probably won't get to it until the end of August though.

[*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html

Quoted 3 lines

This should keep someone busy for a good couple of weeks (give or take a few months).

:-)

Quoted 5 lines

That being said, I'd rather see support in QEMU first, creating all the vcpu/redistributors upfront, and signalling the hotplug event via the virtual firmware. And then post some numbers to show that creating all the vcpus upfront is not acceptable.

I think the upfront allocation, allocating all possible cpus, but only activating all present cpus, was the planned approach. What were the concerns about that approach? Just vcpu memory overhead for too many overly ambitious VM configs?

Quoted 27 lines · Andrew Jones

Quoted 8 lines · Marc Zyngier

Quoted 4 lines · Maran Wilson

It's been a few months since this email thread died off. Has anyone started working on a potential solution that would allow VCPU hotplug on KVM/ARM ? Or is this a project that is still waiting for an owner who has the time and inclination to get started?

This is typically a project for someone who would have this particular itch to scratch, and who has a demonstrable need for this functionality.

Quoted 8 lines

Work wise, it would have to include adding physical CPU hotplug support to the arm64 kernel as a precondition, before worrying about doing it in KVM.

For KVM itself, particular area of interests would be:
- Making GICv3 redistributors magically appear in the IPA space
- Live resizing of GICv3 structures
- Dynamic allocation of MPIDR, and mapping with vcpu_id

I have CPU topology description patches on the QEMU list now[*]. A next step for me is to this MPIDR work. I probably won't get to it until the end of August though.

[*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html

Quoted 2 lines

This should keep someone busy for a good couple of weeks (give or take a few months).

:-)

Quoted 8 lines

Quoted 4 lines

That being said, I'd rather see support in QEMU first, creating all the vcpu/redistributors upfront, and signalling the hotplug event via the virtual firmware. And then post some numbers to show that creating all the vcpus upfront is not acceptable.

I think the upfront allocation, allocating all possible cpus, but only activating all present cpus, was the planned approach. What were the concerns about that approach? Just vcpu memory overhead for too many overly ambitious VM configs?

I don't have any ARM-specific concern about that, and I think this is the right approach. It has the good property of not requiring much change in the kernel (other than actually supporting CPU hotplug).

vcpu memory overhead is a generic concern though, and not only for ARM. We currently allow up to 512 vcpus per VM, which looks like a lot, but really isn't. If we're to allow this to be bumped up significantly, we should start accounting the vcpu-related memory against the user's allowance...

Thanks,

	M.

Quoted 37 lines · Andrew Jones

Quoted 9 lines · Marc Zyngier

Quoted 4 lines · Maran Wilson

It's been a few months since this email thread died off. Has anyone started working on a potential solution that would allow VCPU hotplug on KVM/ARM ? Or is this a project that is still waiting for an owner who has the time and inclination to get started?

This is typically a project for someone who would have this particular itch to scratch, and who has a demonstrable need for this functionality.

Quoted 8 lines

Work wise, it would have to include adding physical CPU hotplug support to the arm64 kernel as a precondition, before worrying about doing it in KVM.

For KVM itself, particular area of interests would be:
- Making GICv3 redistributors magically appear in the IPA space
- Live resizing of GICv3 structures
- Dynamic allocation of MPIDR, and mapping with vcpu_id

I have CPU topology description patches on the QEMU list now[*]. A next step for me is to this MPIDR work. I probably won't get to it until the end of August though.

[*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html

Quoted 3 lines

This should keep someone busy for a good couple of weeks (give or take a few months).

:-)

Quoted 5 lines

That being said, I'd rather see support in QEMU first, creating all the vcpu/redistributors upfront, and signalling the hotplug event via the virtual firmware. And then post some numbers to show that creating all the vcpus upfront is not acceptable.

Quoted 4 lines

I think the upfront allocation, allocating all possible cpus, but only activating all present cpus, was the planned approach. What were the concerns about that approach? Just vcpu memory overhead for too many overly ambitious VM configs?

I don't have any ARM-specific concern about that, and I think this is the right approach. It has the good property of not requiring much change in the kernel (other than actually supporting CPU hotplug).

vcpu memory overhead is a generic concern though, and not only for ARM. We currently allow up to 512 vcpus per VM, which looks like a lot, but really isn't. If we're to allow this to be bumped up significantly, we should start accounting the vcpu-related memory against the user's allowance...

Thanks,

	M.

Quoted 43 lines · On Wed, 25 Jul 2018 14:07:12 +0100 Marc Zyngier <[email protected]> wrote:

Quoted 41 lines · Andrew Jones

Quoted 9 lines · Marc Zyngier

Quoted 4 lines · Maran Wilson

It's been a few months since this email thread died off. Has anyone started working on a potential solution that would allow VCPU hotplug on KVM/ARM ? Or is this a project that is still waiting for an owner who has the time and inclination to get started?

This is typically a project for someone who would have this particular itch to scratch, and who has a demonstrable need for this functionality.

Quoted 8 lines

Work wise, it would have to include adding physical CPU hotplug support to the arm64 kernel as a precondition, before worrying about doing it in KVM.

For KVM itself, particular area of interests would be:
- Making GICv3 redistributors magically appear in the IPA space
- Live resizing of GICv3 structures
- Dynamic allocation of MPIDR, and mapping with vcpu_id

I have CPU topology description patches on the QEMU list now[*]. A next step for me is to this MPIDR work. I probably won't get to it until the end of August though.

[*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html

Quoted 3 lines

This should keep someone busy for a good couple of weeks (give or take a few months).

:-)

Quoted 5 lines

That being said, I'd rather see support in QEMU first, creating all the vcpu/redistributors upfront, and signalling the hotplug event via the virtual firmware. And then post some numbers to show that creating all the vcpus upfront is not acceptable.

I think the upfront allocation, allocating all possible cpus, but only activating all present cpus, was the planned approach. What were the concerns about that approach? Just vcpu memory overhead for too many overly ambitious VM configs?

Quoted 3 lines

I don't have any ARM-specific concern about that, and I think this is the right approach. It has the good property of not requiring much change in the kernel (other than actually supporting CPU hotplug).

for x86 we allocate VCPUs dynamically (both QEMU and KVM) CCing ppc/s390 folks as I don't recall how it's implemented there.

Quoted 5 lines · On Wed, 25 Jul 2018 14:07:12 +0100 Marc Zyngier <[email protected]> wrote:

Quoted 4 lines · Andrew Jones

Quoted 3 lines · Marc Zyngier

Quoted 2 lines · Maran Wilson

It's been a few months since this email thread died off. Has anyone started working on a potential solution that would allow VCPU

hotplug on

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

KVM/ARM ? Or is this a project that is still waiting for an owner

who

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 1 line

has the time and inclination to get started?

This is typically a project for someone who would have this particular itch to scratch, and who has a demonstrable need for this

functionality.

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Work wise, it would have to include adding physical CPU hotplug

support

Quoted 1 line

Quoted 1 line

Quoted 1 line

to the arm64 kernel as a precondition, before worrying about doing it

in

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

KVM.

For KVM itself, particular area of interests would be:
- Making GICv3 redistributors magically appear in the IPA space
- Live resizing of GICv3 structures
- Dynamic allocation of MPIDR, and mapping with vcpu_id

Quoted 5 lines

Quoted 5 lines

I have CPU topology description patches on the QEMU list now[*]. A next step for me is to this MPIDR work. I probably won't get to it until the end of August though.

[*] http://lists.nongnu.org/archive/html/qemu-devel/2018-

07/msg01168.html

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

This should keep someone busy for a good couple of weeks (give or

take a

Quoted 6 lines

Quoted 6 lines

Quoted 1 line

few months).

:-)

Quoted 2 lines

That being said, I'd rather see support in QEMU first, creating all

the

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

vcpu/redistributors upfront, and signalling the hotplug event via the virtual firmware. And then post some numbers to show that creating all the vcpus upfront is not acceptable.

Quoted 8 lines

Quoted 4 lines

I think the upfront allocation, allocating all possible cpus, but only activating all present cpus, was the planned approach. What were the concerns about that approach? Just vcpu memory overhead for too many overly ambitious VM configs?

I don't have any ARM-specific concern about that, and I think this is the right approach. It has the good property of not requiring much change in the kernel (other than actually supporting CPU hotplug).

for x86 we allocate VCPUs dynamically (both QEMU and KVM) CCing ppc/s390 folks as I don't recall how it's implemented there.

but we do not delete vcpus in KVM after they were created (as it deemed to be too complicated), we are just deleting QEMU part of it and keep kvm's vcpu for reuse with future hotplug.