New CPUID/MSR driver; virtualization hooks

I have finally gotten off the pot and finished writing up my new CPUID/MSR driver, which contains support for registers that need arbitrary GPRs touched. For i386 vs x86-64 compatibility, both use an x86-64 register image (16 64-bit register fields); this allows 32-bit userspace to access the full 64-bit image if the kernel is 64 bits.

Anyway, this presumably requires new paravirtualization hooks. The patch is at:

http://www.kernel.org/pub/linux/kernel/people/hpa/new-cpuid-msr.patch

... and a git tree is at ...

http://git.kernel.org/?p=linux/kernel/git/hpa/linux-2.6-cpuidmsr.git;a=summary

I'm posting this here to give the paravirt maintainers an opportunity to comment. Presumably the functions that need to be paravirtualized are the ones represented by the functions do_cpuid(), do_rdmsr() and do_wrmsr(): they take a cpu number, an input register image, and an output register image, and return either 0 or -EIO (in case of a trap.)

Quoted 2 lines

http://git.kernel.org/?p=linux/kernel/git/hpa/linux-2.6-cpuidmsr.git;a=summary

Bleah, and gitweb is unhappy ATM too.

Without having seen the patch yet, you'll need to make sure that the final point which is issuing asm("cpuid") is wrapped and split to CONFIG_PARAVIRT and non CONFIG_PARAVIRT modes.

It's not *quite* that easy. The assembly code around this is pretty extensive, because it has to stand on its head in order to present the proper register image.

Quoted 1 line · H. Peter Anvin

http://www.kernel.org/pub/linux/kernel/people/hpa/new-cpuid-msr.patch

The requested URL /pub/linux/kernel/people/hpa/new-cpuid-msr.patch

Quoted 3 lines

It's not *quite* that easy. The assembly code around this is pretty extensive, because it has to stand on its head in order to present the proper register image.

Having just stood on my head for 55 breaths, might I suggest we implement a binary equivalent CPUID paravirt-ops wrapper; then the assembly code can just call CPUID and we can redefine it to call to a stub, which makes the pv-ops CPUID call, the puts the outputs back in the proper registers.

The VMI ROM CPUID is binary identical in register format with the native instruction, so it need not do a headstand. For the others,

This code is almost entirely identical to the setgpr_wrapper in the patch (except for the fact that setgpr_wrapper sets and captures *ALL* the GPRs), and it seems rather pointless to use another wrapper. It takes a pointer to an entrypoint (default to "cpuid; ret" in the CPUID case), so it should do what you need.

Quoted 6 lines · H. Peter Anvin

This code is almost entirely identical to the setgpr_wrapper in the patch (except for the fact that setgpr_wrapper sets and captures *ALL* the GPRs), and it seems rather pointless to use another wrapper. It takes a pointer to an entrypoint (default to "cpuid; ret" in the CPUID case), so it should do what you need.

void (*cpuid)(unsigned int *eax, unsigned int *ebx, ...)

Not quite. The paravirt_ops CPUID function is a C function which takes pointers to each GPR as arguments, and returns the values through those pointers. This doesn't allow for an entrypoint compatible with setgpr_wrapper, which expects the same output in both the cpuid; ret case and the paravirt-ops case.

One could argue that the paravirt-ops CPUID should in fact emulate the native instruction semantics, which would make it a non-C function. However, until that bridge is crossed, we would need another wrapper after setgpr to convert the paravirt-ops CPUID back into the same format as native so that setgpr_wrapper can properly store the output fields into the common i386/x86_64 structure. What definitely should be done is hide this secondary wrapper in the paravirt-ops code so that your setgpr wrapper doesn't have to deal with weird issues like this because of paravirt-ops changes.

I guess what I was trying to say was that we'd use setgpr_wrapper in the case where you have an entrypoint with native (non-C) semantics; in the other case we'd use an alternative to setgpr_wrapper. Either way, it sounds like we're talking about implementing paravirtualization *after* CPU selection, i.e. we use IPI to get the thing running on the proper CPU before invoking the paravirtualization function?

Quoted 6 lines · H. Peter Anvin

This code is almost entirely identical to the setgpr_wrapper in the patch (except for the fact that setgpr_wrapper sets and captures *ALL* the GPRs), and it seems rather pointless to use another wrapper. It takes a pointer to an entrypoint (default to "cpuid; ret" in the CPUID case), so it should do what you need.

void (*cpuid)(unsigned int *eax, unsigned int *ebx, ...)

Not quite. The paravirt_ops CPUID function is a C function which takes pointers to each GPR as arguments, and returns the values through those pointers. This doesn't allow for an entrypoint compatible with setgpr_wrapper, which expects the same output in both the cpuid; ret case and the paravirt-ops case.

One could argue that the paravirt-ops CPUID should in fact emulate the native instruction semantics, which would make it a non-C function.

It seems to me that perhaps this is the way to do it:

- Add cpuid_everything, rdmsr_everything and wrmsr_everything entrypoints;

- Provide compatibility wrappers that simply invoke the cpuid, rdmsr, and wrmsr entrypoints with appropriate parameter marshalling;

- On hardware, point them to the functions which invoke setgpr_wrapper.

What do you think?

Quoted 5 lines · H. Peter Anvin

It seems to me that perhaps this is the way to do it:

- Add cpuid_everything, rdmsr_everything and wrmsr_everything entrypoints;

Yes, although I think we can omit rdmsr / wrmsr.

I actually expect rdmsr/wrmsr to have a higher likelihood of weird things. I don't know how many hypervisors can trap and handle those.

I actually expect rdmsr/wrmsr to have a higher likelihood of weird things. I don't know how many hypervisors can trap and handle those.

Or implement anything useful at all ... perhaps it might be useful to simply wrap them with an -ENOSYS in some cases.

Rusty, Jeremy. Chris - any feedback on MSR support?

Quoted 4 lines · Zachary Amsden

Or implement anything useful at all ... perhaps it might be useful to simply wrap them with an -ENOSYS in some cases.

Rusty, Jeremy. Chris - any feedback on MSR support?

I've looked at hpa's patch a bit, but I don't quite get what the point is yet. Is it that rdmsr/wrmsr can use multiple random registers, so its necessary to haul them all around? And that cpuid might use more beyond the current eax-edx?

Xen will trap and emulate any rd/wrmsr instructions, so I don't think it needs anything beyond native support.

Yes, pretty much. There are enough evidence that you can't trust CPU architecture to stay sane. Inside Transmeta it was a constant battle, and we were a small company.

Quoted 4 lines · H. Peter Anvin

Yes, pretty much. There are enough evidence that you can't trust CPU architecture to stay sane. Inside Transmeta it was a constant battle, and we were a small company.

Is there any indication that the msr or cpuid instructions will change in this way? rd/wrmsr is pretty explicitly documented as taking the msr in %ecx, and the value in %edx:eax; do you think that will change? And likewise cpuid?

Quoted 3 lines · H. Peter Anvin

I actually expect rdmsr/wrmsr to have a higher likelihood of weird things. I don't know how many hypervisors can trap and handle those.

Or implement anything useful at all ... perhaps it might be useful to simply wrap them with an -ENOSYS in some cases.

Rusty, Jeremy. Chris - any feedback on MSR support?

I have finally gotten off the pot and finished writing up my new CPUID/MSR driver, which contains support for registers that need arbitrary GPRs touched. For i386 vs x86-64 compatibility, both use an x86-64 register image (16 64-bit register fields); this allows 32-bit userspace to access the full 64-bit image if the kernel is 64 bits.

Anyway, this presumably requires new paravirtualization hooks. The patch is at:

http://www.kernel.org/pub/linux/kernel/people/hpa/new-cpuid-msr.patch

I have finally gotten off the pot and finished writing up my new CPUID/MSR driver, which contains support for registers that need arbitrary GPRs touched. For i386 vs x86-64 compatibility, both use an x86-64 register image (16 64-bit register fields); this allows 32-bit userspace to access the full 64-bit image if the kernel is 64 bits.

Anyway, this presumably requires new paravirtualization hooks. The patch is at:

http://www.kernel.org/pub/linux/kernel/people/hpa/new-cpuid-msr.patch

... and a git tree is at ...

http://git.kernel.org/?p=linux/kernel/git/hpa/linux-2.6-cpuidmsr.git;a=summary

I'm posting this here to give the paravirt maintainers an opportunity to comment. Presumably the functions that need to be paravirtualized are the ones represented by the functions do_cpuid(), do_rdmsr() and