pv_ops smp support

I'm looking at adding Xen SMP support, so I'm trying to work out what pv_ops we need, and how to cut into the existing smp stuff.

smpboot.c has a mixture of stuff which is generally useful for SMP stuff (the various CPU sets, and presumably the sibling relationships are useful in principle), but also a whole pile of APIC stuff which is irrelevent to Xen. It has these exported symbols, with my first pass comments:

    00000644 T __cpu_die			-- need pv_op
    000008a5 T __cpu_disable		-- need pv_op
    000006aa T __cpu_up			-- need pv_op
    00000000 T cpu_coregroup_map		-- ? ignore
    00000868 T cpu_exit_clear		-- ?
    00000d9f T initialize_secondary		-- Xen no-op
    000008ee T smp_alloc_memory		-- Xen no-op (doesn't matter if it gets called)
    00000057 T smp_cpus_done		-- unwanted for Xen
    00000015 T smp_intr_init		-- need something to set up IPIs, but APIC independent
    0000082a T smp_prepare_boot_cpu		-- looks OK for Xen, I think
    00000066 T smp_prepare_cpus		-- need pv_op

So at first pass, it looks like we need 6-8 new pv_ops for SMP, which isn't very appealing.

Any thoughts about how to come up with a more elegant interface? I'm digging through smpboot.c and friends, but there's a lot of goo in there...

It's not clear to me what we should do with all the topology stuff. Clearly a VCPU won't have any fixed physical relationship with other VCPUs unless they're pinned; I'm wondering if the existing code will confuse itself if it gets basically random topology info from cpuid.

J

Most of the goo is apic related goo, which can be conveniently be tucked out of the way by just ignoring or putting in appropriate hooks. Before we try to dissect this into operations, I think there are two bigger fundamental questions you need to answer -

1) What do you plan to do to address per-cpu data structures?
2) What is your remote TLB shootdown model?

Both of these could have major impacts on how you want to carve up smp.c and smpboot.c.

Zach

Er, what's there at the moment, more or less. The main thing is that the secondary CPU get the PDA set up (init gdt and %gs) before anyone wants to use it (which is generally the first use of smp_processor_id() or current). At some point we'll probably fold the PDA and PER_CPU together. Xen can more or less completely initialize the VCPU state before it is brought up, so there's little or no need for any kind of bootstrap code.

Xen has a hypercall to shoot down a set of CPU's TLBs, so it doesn't need to do an IPI (we'll need to extend the flush_tlb interface to make good use of this). It will still need IPIs for reschedule and remote function calls or course.

No, I don't mean the Linux PDA - how do you access the Xen PDA? Or have they conjoined somehow?

So your invalidate "IPI" is actually a hypercall, and you can use the existing flush_tlb interface for the most part. You just need a paravirt-op then for the IPI itself, which takes a CPU mask - and this seems to match nicely onto your hypercall. I think you might want to optimize this a bit more, however, since in some cases you will issue implicit shootdown IPIs during a pte update hypercall.

This might require some reworking of the MMU interfaces in paravirt-ops to accomodate providing such a record, and perhaps adding a function to flush the tlb_gather interface so that pte updates which have such properties can record the shootdown.

This seems much cleaner than designing the shootdown semantic directly into such PTE updates, which is I believe what some of the older Xen patches did, although I could have misread them.

Zach

You could put it that way I guess. There's a generic Linux PDA; a paravirt patch adds a union for pv use, and a Xen patch adds a Xen-specific element to that union. That's how it has been from the start, so there wasn't really anything to conjoin (there was never a Xen PDA per se).

Yep. There are calls for flushing the whole tlb, and for just a page; both take CPU masks.

Not that I've seen, at least none that doesn't also exist in baseline.

I was starting on SMP with the idea that it would be relatively isolated and simple, but it seems I should probably do the MMU stuff first to see what impact it has on SMP.

J

But if Xen uses direct page tables, the update to a page table through a hypercall can cause other CPUs with that direct page table loaded to require an invalidation. <Pauses>. Never mind, I see that you don't require an immediate invalidation. Instead, you use page type enforcement to guarantee that page remappings can not be used to subvert hypervisor protection, and would either disallow or issue TLB invalidations on page type changes from PT to writable pages. In that case, the guest must use TLB shootdown to guarantee correctness, but no extra TLB shootdowns are required to guarantee hypervisor security, as I had feared.

I think I was imagining some complications which really don't exist, but it certainly doesn't hurt to consider the Xen MMU interfaces as you iron out the SMP issues.

So perhaps SMP will be more straightforward than I expected.

Zach