: support more than 32 VCPUs in guests

Jeremy, pv ops is OK as it is on 128 vcpus, but I reworked the xen_vcpu_setup() a little to address more than 32vcpus on xen that doesn't have vcpu placement. Please take a look.

Quoted 4 lines · Mukesh Rathor

Jeremy, pv ops is OK as it is on 128 vcpus, but I reworked the xen_vcpu_setup() a little to address more than 32vcpus on xen that doesn't have vcpu placement. Please take a look.

Why BUG_ON if the number of cpus is too high? Why not just ignore the excess ones?

J

Quoted 3 lines

Why BUG_ON if the number of cpus is too high? Why not just ignore the excess ones?

Yeah, that was my first thought also... but then i realized i couldn't just ignore the excess cpus in that function, but would need to go back and fixup all the cpu_present, cpu_online, etc maps (and any assoc data structs, if any), and it just didn't seem worth it in the 2.6.18* kernels at least. Would have been easier to do if the vcpu setup function returned a value instead of being void.

Quoted 10 lines · Mukesh Rathor

Quoted 3 lines

Why BUG_ON if the number of cpus is too high? Why not just ignore the excess ones?

Yeah, that was my first thought also... but then i realized i couldn't just ignore the excess cpus in that function, but would need to go back and fixup all the cpu_present, cpu_online, etc maps (and any assoc data structs, if any), and it just didn't seem worth it in the 2.6.18* kernels at least. Would have been easier to do if the vcpu setup function returned a value instead of being void.

Yes, but if have_vcpu_info_placement ends up being false (which is tested before any other cpus are brought up) then you can simply fail to online the ones above the limit.

BUG_ON is way too brutal. You need to fail more softly.

J

Well, BUG_ON is only triggered if booting more than 32 VCPUs on a *very old* xen (pre xen 3.1.0).

Looking at code closely, we could just set setup_max_cpus to 32 some where in xen function, perhaps even in xen_vcpu_setup(). That way later in smp_init() it would just be ok.

One thing tho, the per cpus areas are already setup at that point, so that would need to be cleaned. BTW, I don't understand why have_vcpu_info_placement is set to 0 in xen_guest_init()?

What minimum version of xen is required to run pvops kernel?

Quoted 7 lines · Mukesh Rathor

Well, BUG_ON is only triggered if booting more than 32 VCPUs on a *very old* xen (pre xen 3.1.0).

Looking at code closely, we could just set setup_max_cpus to 32 some where in xen function, perhaps even in xen_vcpu_setup(). That way later in smp_init() it would just be ok.

Yes.

Quoted 4 lines

One thing tho, the per cpus areas are already setup at that point, so that would need to be cleaned. BTW, I don't understand why have_vcpu_info_placement is set to 0 in xen_guest_init()?

xen_guest_init is used by the pvhvm path, and hvm domains don't have a notion of vcpu info placement.

Quoted 2 lines

What minimum version of xen is required to run pvops kernel?

In theory it should be back-compatible for all Xen 3, but in practice it tweaks lots of bugs in older Xens (particularly 32-on-64). I don't know that anyone has definitively established an earliest version. I implemented vcpu info placement for use in pvops kernels, but it was never my intention that it be an absolute requirement.

J

arch/x86/xen/enlighten.c +3 -0

+        struct vcpu_register_vcpu_info info;
+        int err;
+        struct vcpu_info *vcpup;

Quoted 31 lines · On Mon, 14 Jun 2010 10:37:30 +0100 Jeremy Fitzhardinge <[email protected]> wrote:

Quoted 8 lines · Mukesh Rathor

Well, BUG_ON is only triggered if booting more than 32 VCPUs on a *very old* xen (pre xen 3.1.0).

Looking at code closely, we could just set setup_max_cpus to 32 some where in xen function, perhaps even in xen_vcpu_setup(). That way later in smp_init() it would just be ok.

Yes.

Quoted 5 lines

One thing tho, the per cpus areas are already setup at that point, so that would need to be cleaned. BTW, I don't understand why have_vcpu_info_placement is set to 0 in xen_guest_init()?

xen_guest_init is used by the pvhvm path, and hvm domains don't have a notion of vcpu info placement.

Quoted 3 lines

What minimum version of xen is required to run pvops kernel?

In theory it should be back-compatible for all Xen 3, but in practice it tweaks lots of bugs in older Xens (particularly 32-on-64). I don't know that anyone has definitively established an earliest version. I implemented vcpu info placement for use in pvops kernels, but it was never my intention that it be an absolute requirement.

Quoted 2 lines

J

Ok, attached patch without BUG_ON. Please feel free to modify to your liking also.

Quoted 36 lines · Mukesh Rathor

Quoted 32 lines · On Mon, 14 Jun 2010 10:37:30 +0100 Jeremy Fitzhardinge <[email protected]> wrote:

Quoted 8 lines · Mukesh Rathor

Well, BUG_ON is only triggered if booting more than 32 VCPUs on a *very old* xen (pre xen 3.1.0).

Looking at code closely, we could just set setup_max_cpus to 32 some where in xen function, perhaps even in xen_vcpu_setup(). That way later in smp_init() it would just be ok.

Yes.

Quoted 5 lines

One thing tho, the per cpus areas are already setup at that point, so that would need to be cleaned. BTW, I don't understand why have_vcpu_info_placement is set to 0 in xen_guest_init()?

xen_guest_init is used by the pvhvm path, and hvm domains don't have a notion of vcpu info placement.

Quoted 3 lines

What minimum version of xen is required to run pvops kernel?

In theory it should be back-compatible for all Xen 3, but in practice it tweaks lots of bugs in older Xens (particularly 32-on-64). I don't know that anyone has definitively established an earliest version. I implemented vcpu info placement for use in pvops kernels, but it was never my intention that it be an absolute requirement.

Quoted 5 lines

Quoted 2 lines

J

Ok, attached patch without BUG_ON. Please feel free to modify to your liking also.

It looks like you smashed all the tabs into spaces so its hard to see what you've changed in the diff. I'll fix it up and give it a look-over.

J

Quoted 43 lines · On Tue, 15 Jun 2010 09:30:35 +0100 Jeremy Fitzhardinge <[email protected]> wrote:

Quoted 36 lines · Mukesh Rathor

Quoted 32 lines · On Mon, 14 Jun 2010 10:37:30 +0100 Jeremy Fitzhardinge <[email protected]> wrote:

Quoted 8 lines · Mukesh Rathor

Well, BUG_ON is only triggered if booting more than 32 VCPUs on a *very old* xen (pre xen 3.1.0).

Looking at code closely, we could just set setup_max_cpus to 32 some where in xen function, perhaps even in xen_vcpu_setup(). That way later in smp_init() it would just be ok.

Yes.

Quoted 5 lines

One thing tho, the per cpus areas are already setup at that point, so that would need to be cleaned. BTW, I don't understand why have_vcpu_info_placement is set to 0 in xen_guest_init()?

xen_guest_init is used by the pvhvm path, and hvm domains don't have a notion of vcpu info placement.

Quoted 3 lines

What minimum version of xen is required to run pvops kernel?

In theory it should be back-compatible for all Xen 3, but in practice it tweaks lots of bugs in older Xens (particularly 32-on-64). I don't know that anyone has definitively established an earliest version. I implemented vcpu info placement for use in pvops kernels, but it was never my intention that it be an absolute requirement.

Quoted 5 lines

Quoted 2 lines

J

Ok, attached patch without BUG_ON. Please feel free to modify to your liking also.

Quoted 5 lines

It looks like you smashed all the tabs into spaces so its hard to see what you've changed in the diff. I'll fix it up and give it a look-over.

J

Sorry, I've tabs turned off because patches I submit to other product I work on must be tab free. Anyways, re attached a new one with tabs.

thanks again,
Mukesh

Signed-off-by Mukesh Rathor <[email protected]>

Quoted 65 lines · On Tue, 15 Jun 2010 11:45:43 -0700 Mukesh Rathor <[email protected]>…

Quoted 52 lines · On Tue, 15 Jun 2010 09:30:35 +0100 Jeremy Fitzhardinge <[email protected]> wrote:

Quoted 43 lines · Mukesh Rathor

Quoted 38 lines · On Mon, 14 Jun 2010 10:37:30 +0100 Jeremy Fitzhardinge <[email protected]> wrote:

Quoted 9 lines · Mukesh Rathor

Well, BUG_ON is only triggered if booting more than 32 VCPUs on a *very old* xen (pre xen 3.1.0).

Looking at code closely, we could just set setup_max_cpus to 32 some where in xen function, perhaps even in xen_vcpu_setup(). That way later in smp_init() it would just be ok.

Yes.

Quoted 6 lines

One thing tho, the per cpus areas are already setup at that point, so that would need to be cleaned. BTW, I don't understand why have_vcpu_info_placement is set to 0 in xen_guest_init()?

xen_guest_init is used by the pvhvm path, and hvm domains don't have a notion of vcpu info placement.

Quoted 4 lines

What minimum version of xen is required to run pvops kernel?

In theory it should be back-compatible for all Xen 3, but in practice it tweaks lots of bugs in older Xens (particularly 32-on-64). I don't know that anyone has definitively established an earliest version. I implemented vcpu info placement for use in pvops kernels, but it was never my intention that it be an absolute requirement.

Quoted 7 lines

Quoted 3 lines

J

Ok, attached patch without BUG_ON. Please feel free to modify to your liking also.

Quoted 6 lines

It looks like you smashed all the tabs into spaces so its hard to see what you've changed in the diff. I'll fix it up and give it a look-over.

J

Sorry, I've tabs turned off because patches I submit to other product I work on must be tab free. Anyways, re attached a new one with tabs.

Quoted 5 lines

thanks again,
Mukesh

Signed-off-by Mukesh Rathor <[email protected]>

Hi Jeremy,

Just curious, did this patch ever make it?

Quoted 77 lines · Mukesh Rathor

Quoted 62 lines · On Tue, 15 Jun 2010 11:45:43 -0700 Mukesh Rathor <[email protected]>…

Quoted 52 lines · On Tue, 15 Jun 2010 09:30:35 +0100 Jeremy Fitzhardinge <[email protected]> wrote:

Quoted 43 lines · Mukesh Rathor

Quoted 38 lines · On Mon, 14 Jun 2010 10:37:30 +0100 Jeremy Fitzhardinge <[email protected]> wrote:

Quoted 9 lines · Mukesh Rathor

Well, BUG_ON is only triggered if booting more than 32 VCPUs on a *very old* xen (pre xen 3.1.0).

Looking at code closely, we could just set setup_max_cpus to 32 some where in xen function, perhaps even in xen_vcpu_setup(). That way later in smp_init() it would just be ok.

Yes.

Quoted 6 lines

One thing tho, the per cpus areas are already setup at that point, so that would need to be cleaned. BTW, I don't understand why have_vcpu_info_placement is set to 0 in xen_guest_init()?

xen_guest_init is used by the pvhvm path, and hvm domains don't have a notion of vcpu info placement.

Quoted 4 lines

What minimum version of xen is required to run pvops kernel?

In theory it should be back-compatible for all Xen 3, but in practice it tweaks lots of bugs in older Xens (particularly 32-on-64). I don't know that anyone has definitively established an earliest version. I implemented vcpu info placement for use in pvops kernels, but it was never my intention that it be an absolute requirement.

Quoted 7 lines

Quoted 3 lines

J

Ok, attached patch without BUG_ON. Please feel free to modify to your liking also.

Quoted 6 lines

It looks like you smashed all the tabs into spaces so its hard to see what you've changed in the diff. I'll fix it up and give it a look-over.

J

Quoted 9 lines

Sorry, I've tabs turned off because patches I submit to other product I work on must be tab free. Anyways, re attached a new one with tabs.

thanks again,
Mukesh

Signed-off-by Mukesh Rathor <[email protected]>

Hi Jeremy,

Quoted 2 lines

Just curious, did this patch ever make it?

Probably not. Looks like I forgot to tag it as "mail containing patch" so it fell through the cracks.

J

Quoted 42 lines · On Tue, 15 Jun 2010 09:30:35 +0100 Jeremy Fitzhardinge<[email protected]> wrote:

Quoted 41 lines · Mukesh Rathor

Quoted 25 lines · On Mon, 14 Jun 2010 10:37:30 +0100 Jeremy Fitzhardinge<[email protected]> wrote:

Quoted 8 lines · Mukesh Rathor

Well, BUG_ON is only triggered if booting more than 32 VCPUs on a *very old* xen (pre xen 3.1.0).

Looking at code closely, we could just set setup_max_cpus to 32 some where in xen function, perhaps even in xen_vcpu_setup(). That way later in smp_init() it would just be ok.

Yes.

Quoted 5 lines

One thing tho, the per cpus areas are already setup at that point, so that would need to be cleaned. BTW, I don't understand why have_vcpu_info_placement is set to 0 in xen_guest_init()?

xen_guest_init is used by the pvhvm path, and hvm domains don't have a notion of vcpu info placement.

Quoted 3 lines

What minimum version of xen is required to run pvops kernel?

Quoted 9 lines

In theory it should be back-compatible for all Xen 3, but in practice it tweaks lots of bugs in older Xens (particularly 32-on-64). I don't know that anyone has definitively established an earliest version. I implemented vcpu info placement for use in pvops kernels, but it was never my intention that it be an absolute requirement.

J

Ok, attached patch without BUG_ON. Please feel free to modify to your liking also.

Quoted 5 lines

It looks like you smashed all the tabs into spaces so its hard to see what you've changed in the diff. I'll fix it up and give it a look-over.

J

Sorry, I've tabs turned off because patches I submit to other product I work on must be tab free. Anyways, re attached a new one with tabs.

Quoted 49 lines · Mukesh Rathor

Quoted 46 lines · On Tue, 15 Jun 2010 09:30:35 +0100 Jeremy Fitzhardinge<[email protected]> wrote:

Quoted 36 lines · Mukesh Rathor

Quoted 32 lines · On Mon, 14 Jun 2010 10:37:30 +0100 Jeremy Fitzhardinge<[email protected]> wrote:

Quoted 8 lines · Mukesh Rathor

Well, BUG_ON is only triggered if booting more than 32 VCPUs on a *very old* xen (pre xen 3.1.0).

Looking at code closely, we could just set setup_max_cpus to 32 some where in xen function, perhaps even in xen_vcpu_setup(). That way later in smp_init() it would just be ok.

Yes.

Quoted 5 lines

One thing tho, the per cpus areas are already setup at that point, so that would need to be cleaned. BTW, I don't understand why have_vcpu_info_placement is set to 0 in xen_guest_init()?

xen_guest_init is used by the pvhvm path, and hvm domains don't have a notion of vcpu info placement.

Quoted 3 lines

What minimum version of xen is required to run pvops kernel?

In theory it should be back-compatible for all Xen 3, but in practice it tweaks lots of bugs in older Xens (particularly 32-on-64). I don't know that anyone has definitively established an earliest version. I implemented vcpu info placement for use in pvops kernels, but it was never my intention that it be an absolute requirement.

Quoted 5 lines

Quoted 2 lines

J

Ok, attached patch without BUG_ON. Please feel free to modify to your liking also.

It looks like you smashed all the tabs into spaces so its hard to see what you've changed in the diff. I'll fix it up and give it a look-over.

Quoted 3 lines

Quoted 1 line

J

Sorry, I've tabs turned off because patches I submit to other product I work on must be tab free. Anyways, re attached a new one with tabs.

This doesn't compile with CONFIG_SMP=n because setup_max_cpus doesn't exist. For now I've just put a couple of #ifdef CONFIG_SMPs in there to avoid compile errors, but could you look at coming up with a cleaner solution?