HVMLite / PVHv2 - using x86 EFI boot entry

* You don't need full EFI emulation

Quoted 2 lines · Luis R. Rodriguez

* You don't need full EFI emulation

I think needing any EFI emulation inside Xen (which is where it would need to be for dom0) is not suitable because of the increase in hypervisor ABI.

I also still do not understand your objection to the current tiny stub.

Quoted 2 lines · Luis R. Rodriguez

* You don't need full EFI emulation

I think needing any EFI emulation inside Xen (which is where it would need to be for dom0) is not suitable because of the increase in hypervisor ABI.

I also still do not understand your objection to the current tiny stub.

Quoted 4 lines · David Vrabel

Quoted 2 lines · Luis R. Rodriguez

* You don't need full EFI emulation

Quoted 3 lines

I think needing any EFI emulation inside Xen (which is where it would need to be for dom0) is not suitable because of the increase in hypervisor ABI.

Is this because of timing on architecture / design of HVMLite, or a general position that the complexity to deal with EFI emulation is too much for Xen's taste ?

ARM already went the EFI entry way for domU -- it went the OVMF route, would such a possibility be possible for x86 domU HVMLite ? If not why not, I mean it would seem to make sense to at least mimic the same type of early boot environment, and perhaps there are some lessons to be learned from that effort too.

Are there some lessons to be learned with ARM's effort? What are they? If that could be re-done again with any type of cleaner path, what could that be that could help the x86 side ?

Although emulating EFI may require work, some folks have pointed out that the amount of work may not be that much. If that is done can we instead rely on the same code to replace OVMF to support both Xen ARM and Xen HVMLite on x86 ? What would be the pros / cons of this ?

Quoted 1 line

I also still do not understand your objection to the current tiny stub.

Its more of a hypothetical -- can an EFI entry be used instead given it already does exactly what the new small entry does ? Its also rather odd to add a new entry without evaluating fully a possible alternative that would provide the same exact mechanism.

A full technical unbiased evaluation of the different approaches is what I'd hope we could strive to achieve through discussion and peer review, thinking and prioritizing ultimately what is best to minimize the impact on Linux and also help take advantage of the best features possible through both means. Thinking long term, not immediate short term.

Quoted 4 lines · David Vrabel

Quoted 2 lines · Luis R. Rodriguez

* You don't need full EFI emulation

Quoted 3 lines

I think needing any EFI emulation inside Xen (which is where it would need to be for dom0) is not suitable because of the increase in hypervisor ABI.

Is this because of timing on architecture / design of HVMLite, or a general position that the complexity to deal with EFI emulation is too much for Xen's taste ?

ARM already went the EFI entry way for domU -- it went the OVMF route, would such a possibility be possible for x86 domU HVMLite ? If not why not, I mean it would seem to make sense to at least mimic the same type of early boot environment, and perhaps there are some lessons to be learned from that effort too.

Are there some lessons to be learned with ARM's effort? What are they? If that could be re-done again with any type of cleaner path, what could that be that could help the x86 side ?

Although emulating EFI may require work, some folks have pointed out that the amount of work may not be that much. If that is done can we instead rely on the same code to replace OVMF to support both Xen ARM and Xen HVMLite on x86 ? What would be the pros / cons of this ?

Quoted 1 line

I also still do not understand your objection to the current tiny stub.

Its more of a hypothetical -- can an EFI entry be used instead given it already does exactly what the new small entry does ? Its also rather odd to add a new entry without evaluating fully a possible alternative that would provide the same exact mechanism.

A full technical unbiased evaluation of the different approaches is what I'd hope we could strive to achieve through discussion and peer review, thinking and prioritizing ultimately what is best to minimize the impact on Linux and also help take advantage of the best features possible through both means. Thinking long term, not immediate short term.

Quoted 5 lines · Luis R. Rodriguez

Quoted 4 lines · David Vrabel

Quoted 2 lines · Luis R. Rodriguez

* You don't need full EFI emulation

Quoted 7 lines

Quoted 3 lines

I think needing any EFI emulation inside Xen (which is where it would need to be for dom0) is not suitable because of the increase in hypervisor ABI.

Is this because of timing on architecture / design of HVMLite, or a general position that the complexity to deal with EFI emulation is too much for Xen's taste ?

The Xen hypervisor should be as small as possible. Adding an EFI emulator will be adding quite some code. This should be done after a very thorough evaluation only.

Quoted 5 lines

ARM already went the EFI entry way for domU -- it went the OVMF route, would such a possibility be possible for x86 domU HVMLite ? If not why not, I mean it would seem to make sense to at least mimic the same type of early boot environment, and perhaps there are some lessons to be learned from that effort too.

The final solution must be appropriate for dom0, too. So don't try to limit the discussion to domU. If dom0 isn't going to be acceptable there will no need to discuss domU.

Quoted 10 lines

Are there some lessons to be learned with ARM's effort? What are they? If that could be re-done again with any type of cleaner path, what could that be that could help the x86 side ?

Although emulating EFI may require work, some folks have pointed out that the amount of work may not be that much. If that is done can we instead rely on the same code to replace OVMF to support both Xen ARM and Xen HVMLite on x86 ? What would be the pros / cons of this ?

Quoted 6 lines

Quoted 1 line

I also still do not understand your objection to the current tiny stub.

Its more of a hypothetical -- can an EFI entry be used instead given it already does exactly what the new small entry does ? Its also rather odd to add a new entry without evaluating fully a possible alternative that would provide the same exact mechanism.

The interface isn't the new entry only. It should be evaluated how much of the early EFI boot path would be common to the HVMlite one. What would be gained by using the same entry but having two different boot paths after it? You still need a way to distinguish between bare metal EFI and HVMlite. And Xen needs a way to find out whether a kernel is supporting HVMlite to boot it in the correct mode.

Quoted 5 lines

A full technical unbiased evaluation of the different approaches is what I'd hope we could strive to achieve through discussion and peer review, thinking and prioritizing ultimately what is best to minimize the impact on Linux and also help take advantage of the best features possible through both means. Thinking long term, not immediate short term.

Sure.

Quoted 58 lines · Juergen Gross

Quoted 5 lines · Luis R. Rodriguez

Quoted 4 lines · David Vrabel

Quoted 2 lines · Luis R. Rodriguez

* You don't need full EFI emulation

Quoted 7 lines

Quoted 3 lines

I think needing any EFI emulation inside Xen (which is where it would need to be for dom0) is not suitable because of the increase in hypervisor ABI.

Is this because of timing on architecture / design of HVMLite, or a general position that the complexity to deal with EFI emulation is too much for Xen's taste ?

The Xen hypervisor should be as small as possible. Adding an EFI emulator will be adding quite some code. This should be done after a very thorough evaluation only.

Quoted 5 lines

ARM already went the EFI entry way for domU -- it went the OVMF route, would such a possibility be possible for x86 domU HVMLite ? If not why not, I mean it would seem to make sense to at least mimic the same type of early boot environment, and perhaps there are some lessons to be learned from that effort too.

The final solution must be appropriate for dom0, too. So don't try to limit the discussion to domU. If dom0 isn't going to be acceptable there will no need to discuss domU.

Quoted 10 lines

Are there some lessons to be learned with ARM's effort? What are they? If that could be re-done again with any type of cleaner path, what could that be that could help the x86 side ?

Although emulating EFI may require work, some folks have pointed out that the amount of work may not be that much. If that is done can we instead rely on the same code to replace OVMF to support both Xen ARM and Xen HVMLite on x86 ? What would be the pros / cons of this ?

Quoted 6 lines

Quoted 1 line

I also still do not understand your objection to the current tiny stub.

Its more of a hypothetical -- can an EFI entry be used instead given it already does exactly what the new small entry does ? Its also rather odd to add a new entry without evaluating fully a possible alternative that would provide the same exact mechanism.

The interface isn't the new entry only. It should be evaluated how much of the early EFI boot path would be common to the HVMlite one. What would be gained by using the same entry but having two different boot paths after it? You still need a way to distinguish between bare metal EFI and HVMlite. And Xen needs a way to find out whether a kernel is supporting HVMlite to boot it in the correct mode.

Quoted 5 lines

A full technical unbiased evaluation of the different approaches is what I'd hope we could strive to achieve through discussion and peer review, thinking and prioritizing ultimately what is best to minimize the impact on Linux and also help take advantage of the best features possible through both means. Thinking long term, not immediate short term.

Quoted 1 line

Sure.

FWIW, someone just pointed me to u-boot's EFI implementation. u-boot's lib/efi_loader contains a tiny (<3k LOC, 10kB compiled) UEFI implementation that's sufficient to boot a Linux EFI payload.

An argument against making Xen's default domU entry use UEFI is that it might become unnecessarily awkward to do something like
chainloading to OVMF. But maybe OVMF can be compiled as a UEFI
binary :)

Quoted 58 lines · Juergen Gross

Quoted 5 lines · Luis R. Rodriguez

Quoted 4 lines · David Vrabel

Quoted 2 lines · Luis R. Rodriguez

* You don't need full EFI emulation

Quoted 7 lines

Quoted 3 lines

I think needing any EFI emulation inside Xen (which is where it would need to be for dom0) is not suitable because of the increase in hypervisor ABI.

Is this because of timing on architecture / design of HVMLite, or a general position that the complexity to deal with EFI emulation is too much for Xen's taste ?

The Xen hypervisor should be as small as possible. Adding an EFI emulator will be adding quite some code. This should be done after a very thorough evaluation only.

Quoted 5 lines

ARM already went the EFI entry way for domU -- it went the OVMF route, would such a possibility be possible for x86 domU HVMLite ? If not why not, I mean it would seem to make sense to at least mimic the same type of early boot environment, and perhaps there are some lessons to be learned from that effort too.

The final solution must be appropriate for dom0, too. So don't try to limit the discussion to domU. If dom0 isn't going to be acceptable there will no need to discuss domU.

Quoted 10 lines

Are there some lessons to be learned with ARM's effort? What are they? If that could be re-done again with any type of cleaner path, what could that be that could help the x86 side ?

Although emulating EFI may require work, some folks have pointed out that the amount of work may not be that much. If that is done can we instead rely on the same code to replace OVMF to support both Xen ARM and Xen HVMLite on x86 ? What would be the pros / cons of this ?

Quoted 6 lines

Quoted 1 line

I also still do not understand your objection to the current tiny stub.

Its more of a hypothetical -- can an EFI entry be used instead given it already does exactly what the new small entry does ? Its also rather odd to add a new entry without evaluating fully a possible alternative that would provide the same exact mechanism.

The interface isn't the new entry only. It should be evaluated how much of the early EFI boot path would be common to the HVMlite one. What would be gained by using the same entry but having two different boot paths after it? You still need a way to distinguish between bare metal EFI and HVMlite. And Xen needs a way to find out whether a kernel is supporting HVMlite to boot it in the correct mode.

Quoted 5 lines

A full technical unbiased evaluation of the different approaches is what I'd hope we could strive to achieve through discussion and peer review, thinking and prioritizing ultimately what is best to minimize the impact on Linux and also help take advantage of the best features possible through both means. Thinking long term, not immediate short term.

Quoted 1 line

Sure.

FWIW, someone just pointed me to u-boot's EFI implementation. u-boot's lib/efi_loader contains a tiny (<3k LOC, 10kB compiled) UEFI implementation that's sufficient to boot a Linux EFI payload.

An argument against making Xen's default domU entry use UEFI is that it might become unnecessarily awkward to do something like
chainloading to OVMF. But maybe OVMF can be compiled as a UEFI
binary :)

With my FreeBSD committer hat:

The FreeBSD kernel doesn't contain an EFI entry point, it just contains one single entry point that's used for both legacy BIOS and EFI. Then the FreeBSD loader is the one that contains the different entry points. I would really like to avoid adding an EFI entry point and the PE header to the FreeBSD kernel. The current trampoline in FreeBSD to tie the Xen entry point into the native path contains 96 lines of assembly (half of them are actually comments) and 66 lines of C. I think adding an EFI entry point is going to add a lot more of code than this, and we would probably need changes to the build system in order to assembly the PE header and the ELF headers together.

IMHO, if we want to boot PVH using EFI the right solution is to use OVMF (or any other UEFI firmware) and port it so it's able to run as a PVH guest. I guess it should even be possible to use it for Dom0, although I think this is cumbersome.

With my FreeBSD committer hat:

The FreeBSD kernel doesn't contain an EFI entry point, it just contains one single entry point that's used for both legacy BIOS and EFI. Then the FreeBSD loader is the one that contains the different entry points. I would really like to avoid adding an EFI entry point and the PE header to the FreeBSD kernel. The current trampoline in FreeBSD to tie the Xen entry point into the native path contains 96 lines of assembly (half of them are actually comments) and 66 lines of C. I think adding an EFI entry point is going to add a lot more of code than this, and we would probably need changes to the build system in order to assembly the PE header and the ELF headers together.

IMHO, if we want to boot PVH using EFI the right solution is to use OVMF (or any other UEFI firmware) and port it so it's able to run as a PVH guest. I guess it should even be possible to use it for Dom0, although I think this is cumbersome.

For 1. we'd basically be using the PE/COFF file format with the EFI ABI as an OS agnostic boot protocol, but not as a full firmware runtime environment.

For 1. we'd basically be using the PE/COFF file format with the EFI ABI as an OS agnostic boot protocol, but not as a full firmware runtime environment.

For 1. we'd basically be using the PE/COFF file format with the EFI ABI as an OS agnostic boot protocol, but not as a full firmware runtime environment.

2. is also interesting, though I think less so than 1. I agree that making OVMF work as a PVH guest is probably the right way to go, even for Dom0, not least because you'd have a much cleaner/less buggy implementation than what we see in the real world ;)

For 1. we'd basically be using the PE/COFF file format with the EFI ABI as an OS agnostic boot protocol, but not as a full firmware runtime environment.

2. is also interesting, though I think less so than 1. I agree that making OVMF work as a PVH guest is probably the right way to go, even for Dom0, not least because you'd have a much cleaner/less buggy implementation than what we see in the real world ;)

Quoted 13 lines · Roger Pau Monné

With my FreeBSD committer hat:

The FreeBSD kernel doesn't contain an EFI entry point, it just contains one single entry point that's used for both legacy BIOS and EFI. Then the FreeBSD loader is the one that contains the different entry points. I would really like to avoid adding an EFI entry point and the PE header to the FreeBSD kernel. The current trampoline in FreeBSD to tie the Xen entry point into the native path contains 96 lines of assembly (half of them are actually comments) and 66 lines of C. I think adding an EFI entry point is going to add a lot more of code than this, and we would probably need changes to the build system in order to assembly the PE header and the ELF headers together.

What does the boot flow look like for PVH2 on FreeBSD today? Presumably it doesn't have the same entry point that Boris proposed for Linux?

Does it go, Hypervisor -> FreeBSD loader -> FreeBSD kernel? Or are you able to directly boot the kernel from the hypervisor and skip the middle part by having secondary entry point for Xen marked by the ELF note?

Quoted 4 lines

IMHO, if we want to boot PVH using EFI the right solution is to use OVMF (or any other UEFI firmware) and port it so it's able to run as a PVH guest. I guess it should even be possible to use it for Dom0, although I think this is cumbersome.

There are two levels of EFI boot entry features being discussed,

1. Make the OS kernel a PE/COFF executable
2. Provide some level of EFI service functionality

You can adopt 1. without 2, i.e. without actually providing any EFI services at all, as long as the Xen hypervisor grows a PE/COFF loader (since EFI firmware has to provide you one, for EFI platforms you could use the LoadImage() service in the firmware, but for BIOS platforms you'd need your own in Xen).

On Linux, this has the advantage of deferring the decompression of the bzImage (x86 Linux kernel file format) to the stub on the front of the bzImage. And while I realise that the toolstack already has support for decompressing bzImages, given what Andrew has said about reducing attack surface, having the guest perform the decompression should be a win.

Of course, this is offset somewhat by the fact that you need to audit the PE/COFF loader ;) But decompression in general is notoriously vulnerable to security issues.

Using the in-kernel decompressor is how most (all?) Linux boot loaders work today, so there's the added benefit of reducing the differences between booting on Xen and booting bare metal. For example, you'd probably be able to use CONFIG_RANDOMIZE_BASE (ASLR for kernel image) for Xen if you use the kernel's decompressor. Xen would also get future features in this area for free, and there is a tendency to push boot features into the early stub.

For 1. we'd basically be using the PE/COFF file format with the EFI ABI as an OS agnostic boot protocol, but not as a full firmware runtime environment.

2. is also interesting, though I think less so than 1. I agree that making OVMF work as a PVH guest is probably the right way to go, even for Dom0, not least because you'd have a much cleaner/less buggy implementation than what we see in the real world ;)

Quoted 13 lines · Roger Pau Monné

With my FreeBSD committer hat:

The FreeBSD kernel doesn't contain an EFI entry point, it just contains one single entry point that's used for both legacy BIOS and EFI. Then the FreeBSD loader is the one that contains the different entry points. I would really like to avoid adding an EFI entry point and the PE header to the FreeBSD kernel. The current trampoline in FreeBSD to tie the Xen entry point into the native path contains 96 lines of assembly (half of them are actually comments) and 66 lines of C. I think adding an EFI entry point is going to add a lot more of code than this, and we would probably need changes to the build system in order to assembly the PE header and the ELF headers together.

What does the boot flow look like for PVH2 on FreeBSD today? Presumably it doesn't have the same entry point that Boris proposed for Linux?

Does it go, Hypervisor -> FreeBSD loader -> FreeBSD kernel? Or are you able to directly boot the kernel from the hypervisor and skip the middle part by having secondary entry point for Xen marked by the ELF note?

Quoted 4 lines

IMHO, if we want to boot PVH using EFI the right solution is to use OVMF (or any other UEFI firmware) and port it so it's able to run as a PVH guest. I guess it should even be possible to use it for Dom0, although I think this is cumbersome.

There are two levels of EFI boot entry features being discussed,

1. Make the OS kernel a PE/COFF executable
2. Provide some level of EFI service functionality

You can adopt 1. without 2, i.e. without actually providing any EFI services at all, as long as the Xen hypervisor grows a PE/COFF loader (since EFI firmware has to provide you one, for EFI platforms you could use the LoadImage() service in the firmware, but for BIOS platforms you'd need your own in Xen).

On Linux, this has the advantage of deferring the decompression of the bzImage (x86 Linux kernel file format) to the stub on the front of the bzImage. And while I realise that the toolstack already has support for decompressing bzImages, given what Andrew has said about reducing attack surface, having the guest perform the decompression should be a win.

Of course, this is offset somewhat by the fact that you need to audit the PE/COFF loader ;) But decompression in general is notoriously vulnerable to security issues.

Using the in-kernel decompressor is how most (all?) Linux boot loaders work today, so there's the added benefit of reducing the differences between booting on Xen and booting bare metal. For example, you'd probably be able to use CONFIG_RANDOMIZE_BASE (ASLR for kernel image) for Xen if you use the kernel's decompressor. Xen would also get future features in this area for free, and there is a tendency to push boot features into the early stub.

For 1. we'd basically be using the PE/COFF file format with the EFI ABI as an OS agnostic boot protocol, but not as a full firmware runtime environment.

2. is also interesting, though I think less so than 1. I agree that making OVMF work as a PVH guest is probably the right way to go, even for Dom0, not least because you'd have a much cleaner/less buggy implementation than what we see in the real world ;)

Is this header compatible with the ELF header? Con both co-exist in the same binary without issues?

Is this header compatible with the ELF header? Con both co-exist in the same binary without issues?

Quoted 5 lines · Luis R. Rodriguez

Quoted 4 lines · David Vrabel

Quoted 2 lines · Luis R. Rodriguez

* You don't need full EFI emulation

Quoted 7 lines

Quoted 3 lines

I think needing any EFI emulation inside Xen (which is where it would need to be for dom0) is not suitable because of the increase in hypervisor ABI.

Is this because of timing on architecture / design of HVMLite, or a general position that the complexity to deal with EFI emulation is too much for Xen's taste ?

The Xen hypervisor should be as small as possible. Adding an EFI emulator will be adding quite some code. This should be done after a very thorough evaluation only.

Quoted 5 lines

ARM already went the EFI entry way for domU -- it went the OVMF route, would such a possibility be possible for x86 domU HVMLite ? If not why not, I mean it would seem to make sense to at least mimic the same type of early boot environment, and perhaps there are some lessons to be learned from that effort too.

The final solution must be appropriate for dom0, too. So don't try to limit the discussion to domU. If dom0 isn't going to be acceptable there will no need to discuss domU.

Quoted 10 lines

Are there some lessons to be learned with ARM's effort? What are they? If that could be re-done again with any type of cleaner path, what could that be that could help the x86 side ?

Although emulating EFI may require work, some folks have pointed out that the amount of work may not be that much. If that is done can we instead rely on the same code to replace OVMF to support both Xen ARM and Xen HVMLite on x86 ? What would be the pros / cons of this ?

Quoted 6 lines

Quoted 1 line

I also still do not understand your objection to the current tiny stub.

Its more of a hypothetical -- can an EFI entry be used instead given it already does exactly what the new small entry does ? Its also rather odd to add a new entry without evaluating fully a possible alternative that would provide the same exact mechanism.

The interface isn't the new entry only. It should be evaluated how much of the early EFI boot path would be common to the HVMlite one. What would be gained by using the same entry but having two different boot paths after it? You still need a way to distinguish between bare metal EFI and HVMlite. And Xen needs a way to find out whether a kernel is supporting HVMlite to boot it in the correct mode.

Quoted 5 lines

A full technical unbiased evaluation of the different approaches is what I'd hope we could strive to achieve through discussion and peer review, thinking and prioritizing ultimately what is best to minimize the impact on Linux and also help take advantage of the best features possible through both means. Thinking long term, not immediate short term.

Sure.

Quoted 9 lines · Luis R. Rodriguez

Quoted 7 lines · David Vrabel

Quoted 2 lines · Luis R. Rodriguez

* You don't need full EFI emulation

I think needing any EFI emulation inside Xen (which is where it would need to be for dom0) is not suitable because of the increase in hypervisor ABI.

Quoted 3 lines

Is this because of timing on architecture / design of HVMLite, or a general position that the complexity to deal with EFI emulation is too much for Xen's taste ?

The Xen hypervisor should be as small as possible. Adding an EFI emulator will be adding quite some code. This should be done after a very thorough evaluation only.

Quoted 5 lines

ARM already went the EFI entry way for domU -- it went the OVMF route, would such a possibility be possible for x86 domU HVMLite ? If not why not, I mean it would seem to make sense to at least mimic the same type of early boot environment, and perhaps there are some lessons to be learned from that effort too.

The final solution must be appropriate for dom0, too. So don't try to limit the discussion to domU. If dom0 isn't going to be acceptable there will no need to discuss domU.

Quoted 12 lines

Are there some lessons to be learned with ARM's effort? What are they? If that could be re-done again with any type of cleaner path, what could that be that could help the x86 side ?

Although emulating EFI may require work, some folks have pointed out that the amount of work may not be that much. If that is done can we instead rely on the same code to replace OVMF to support both Xen ARM and Xen HVMLite on x86 ? What would be the pros / cons of this ?

Quoted 1 line

I also still do not understand your objection to the current tiny stub.

Quoted 4 lines

Its more of a hypothetical -- can an EFI entry be used instead given it already does exactly what the new small entry does ? Its also rather odd to add a new entry without evaluating fully a possible alternative that would provide the same exact mechanism.

The interface isn't the new entry only. It should be evaluated how much of the early EFI boot path would be common to the HVMlite one.

What would be gained by using the same entry but having two different boot paths after it?

You still need a way to distinguish between bare metal EFI and HVMlite.

And Xen needs a way to find out whether a kernel is supporting HVMlite to boot it in the correct mode.

Quoted 9 lines · Luis R. Rodriguez

Quoted 7 lines · David Vrabel

Quoted 2 lines · Luis R. Rodriguez

* You don't need full EFI emulation

I think needing any EFI emulation inside Xen (which is where it would need to be for dom0) is not suitable because of the increase in hypervisor ABI.

Quoted 3 lines

Is this because of timing on architecture / design of HVMLite, or a general position that the complexity to deal with EFI emulation is too much for Xen's taste ?

The Xen hypervisor should be as small as possible. Adding an EFI emulator will be adding quite some code. This should be done after a very thorough evaluation only.

Quoted 5 lines

ARM already went the EFI entry way for domU -- it went the OVMF route, would such a possibility be possible for x86 domU HVMLite ? If not why not, I mean it would seem to make sense to at least mimic the same type of early boot environment, and perhaps there are some lessons to be learned from that effort too.

The final solution must be appropriate for dom0, too. So don't try to limit the discussion to domU. If dom0 isn't going to be acceptable there will no need to discuss domU.

Quoted 12 lines

Are there some lessons to be learned with ARM's effort? What are they? If that could be re-done again with any type of cleaner path, what could that be that could help the x86 side ?

Although emulating EFI may require work, some folks have pointed out that the amount of work may not be that much. If that is done can we instead rely on the same code to replace OVMF to support both Xen ARM and Xen HVMLite on x86 ? What would be the pros / cons of this ?

Quoted 1 line

I also still do not understand your objection to the current tiny stub.

Quoted 4 lines

Its more of a hypothetical -- can an EFI entry be used instead given it already does exactly what the new small entry does ? Its also rather odd to add a new entry without evaluating fully a possible alternative that would provide the same exact mechanism.

The interface isn't the new entry only. It should be evaluated how much of the early EFI boot path would be common to the HVMlite one.

What would be gained by using the same entry but having two different boot paths after it?

You still need a way to distinguish between bare metal EFI and HVMlite.

And Xen needs a way to find out whether a kernel is supporting HVMlite to boot it in the correct mode.

Quoted 10 lines · Juergen Gross

Quoted 9 lines · Luis R. Rodriguez

Quoted 7 lines · David Vrabel

Quoted 2 lines · Luis R. Rodriguez

* You don't need full EFI emulation

I think needing any EFI emulation inside Xen (which is where it would need to be for dom0) is not suitable because of the increase in hypervisor ABI.

Quoted 7 lines

Quoted 3 lines

Is this because of timing on architecture / design of HVMLite, or a general position that the complexity to deal with EFI emulation is too much for Xen's taste ?

The Xen hypervisor should be as small as possible. Adding an EFI emulator will be adding quite some code. This should be done after a very thorough evaluation only.

Sure.

Quoted 9 lines

Quoted 5 lines

ARM already went the EFI entry way for domU -- it went the OVMF route, would such a possibility be possible for x86 domU HVMLite ? If not why not, I mean it would seem to make sense to at least mimic the same type of early boot environment, and perhaps there are some lessons to be learned from that effort too.

The final solution must be appropriate for dom0, too. So don't try to limit the discussion to domU. If dom0 isn't going to be acceptable there will no need to discuss domU.

Understood. George noted that on ARM dom0 still uses the ARM native entry point, it seems to accomplish this as it uses a device tree node. I'll chime in on that in another thread.

Quoted 12 lines

Quoted 12 lines

Are there some lessons to be learned with ARM's effort? What are they? If that could be re-done again with any type of cleaner path, what could that be that could help the x86 side ?

Although emulating EFI may require work, some folks have pointed out that the amount of work may not be that much. If that is done can we instead rely on the same code to replace OVMF to support both Xen ARM and Xen HVMLite on x86 ? What would be the pros / cons of this ?

Quoted 1 line

I also still do not understand your objection to the current tiny stub.

Quoted 7 lines

Quoted 4 lines

Its more of a hypothetical -- can an EFI entry be used instead given it already does exactly what the new small entry does ? Its also rather odd to add a new entry without evaluating fully a possible alternative that would provide the same exact mechanism.

The interface isn't the new entry only. It should be evaluated how much of the early EFI boot path would be common to the HVMlite one.

We also have other asm code which can be shared. I'll reply to Boris' original e-mail with what I can identify as perhaps sharable. There is obviously more as you allude.

Quoted 2 lines

What would be gained by using the same entry but having two different boot paths after it?

Its a good question. In summary for me it would be the push for sharing more code and the push for semantics on early boot to address differences proactively, and ultimately it may enable us to help bring closer the old PV boot path closer.

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

The HVMLite boot entry tries to bring the boot entries paths closer as it leverages more of the HVM boot path philosophy to mimic the regular PC boot path.

Is HVMLite supposed to support legacy PV guests as well BTW ?

Reason I'm highlighting Xen ABI as a *reason* alone is that even with today's large discrepancy on the old PV boot path I believe we can bring together the boot paths closer together if the Xen ABI was slightly flexible about this, I've highlighted how I believe that is possible before,

*iff* the Xen ABI would at the very least set 2 things only:

a) Hypervisor type
b) A custom data pointer

This would enable a single boot entry on the guest to handle then:

Pseudo code:

startup_32() startup_64()
| |
| |
V V
pre_hypervisor_stub_32() pre_hypervisor_stub_64()
| |
| |
V V
[existing startup_32()] [existing startup_64()]
| |
| |
V V
post_hypervisor_stub_32() post_hypervisor_stub_64()

If the Xen ABI was flexible about setting a hypervisor type and custom data pointer then we would haven handlers for it, and in it, it can do whatever it thinks is needed for its own guest types. It could also continue to set the zero page on its own as it sees fit.

Again, note that if this is done it could also mean even bringing together the old PV boot path closer together... so this is not just a prospect for HVMLite but also for old PV guests.

2) Because of 1) it has meant we have no formal semantics for early boot code is available and so severe differences can best be addressed also by yet another boot entry. This has meant often times not addressing

or not knowing if we've addressed real differences between the different entries. Case in point, dead code [0]. How do we know we will not run certain code that should not run for the different entries ? Without *any* semantics later in boot code to distinguish where we came from and because we strive to build single kernels with different possible run time environments it means we have tons of code available to execute / run that we may not need.

Because of the lack of semantics we may still have dead code prospects with the new HVMLite entry. How are we sure there is no differences ?

[0] http://www.do-not-panic.com/2015/12/avoiding-dead-code-pvops-not-silver-bullet.html

3) Unikernel / other OS requirements: this is really tied to 2) but even if we tried to evolve the Xen ABI it would mean considering existing solutions out there. Things to consider as an example: FreeBSD doesn't have an EFI entry, unikernels want a simple boot entry.

With this in mind then, that I can think of:

Cons of using the same entry but having two different boot paths:

* Pushes the Xen ABI, needs to make everyone happy, this is hard
* Perhaps harder to implement

Gains of striving to use the same entry but having two different boot:

* Helps to share more code easily
* Reduce attack surface
* Requires us to have semantics for early boot; this has a series of side benefits:
- Means you should try to address differences explicitly rather than implicitly -- case in point Dead Code

Quoted 2 lines

You still need a way to distinguish between bare metal EFI and HVMlite.

Great point! This is the semantics aspect. The new entry for HVMlite approach deals with this by making the differences implicit by the new entry point. My call for addressing this through a hypervisor type was to see if we can get those semantics added explicitly so we can also later address dead code concerns for the new HVMLite guest type.

Part of my own interest in an EFI entry here is that EFI could be used to help expand on the semantics in an OS/agnostic form rather than pushing the x86 boot protocol further. That seems to have its own set of drawbacks though.

Quoted 2 lines

And Xen needs a way to find out whether a kernel is supporting HVMlite to boot it in the correct mode.

How was Xen going to find out if new kernels had HVMlite support with the new entry ? An ELFNOTE() ? If an entry is shared could we note use an

ELFNOTE() also for this though too ?

Luis

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Quoted 5 lines · Luis R. Rodriguez

Quoted 2 lines · Juergen Gross

What would be gained by using the same entry but having two different boot paths after it?

Quoted 4 lines

Its a good question. In summary for me it would be the push for sharing more code and the push for semantics on early boot to address differences proactively, and ultimately it may enable us to help bring closer the old PV boot path closer.

But why? We want to kill PV (eventually).

Quoted 6 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

Quoted 5 lines

The HVMLite boot entry tries to bring the boot entries paths closer as it leverages more of the HVM boot path philosophy to mimic the regular PC boot path.

Is HVMLite supposed to support legacy PV guests as well BTW ?

Gosh no.

Quoted 4 lines

Reason I'm highlighting Xen ABI as a *reason* alone is that even with today's large discrepancy on the old PV boot path I believe we can bring together the boot paths closer together if the Xen ABI was slightly flexible about this, I've highlighted how I believe that is possible before,

<runs away screaming>

Quoted 33 lines

*iff* the Xen ABI would at the very least set 2 things only:

a) Hypervisor type
b) A custom data pointer

This would enable a single boot entry on the guest to handle then:

Pseudo code:

startup_32() startup_64()
| |
| |
V V
pre_hypervisor_stub_32() pre_hypervisor_stub_64()
| |
| |
V V
[existing startup_32()] [existing startup_64()]
| |
| |
V V
post_hypervisor_stub_32() post_hypervisor_stub_64()

If the Xen ABI was flexible about setting a hypervisor type and custom data pointer then we would haven handlers for it, and in it, it can do whatever it thinks is needed for its own guest types. It could also continue to set the zero page on its own as it sees fit.

Again, note that if this is done it could also mean even bringing together the old PV boot path closer together... so this is not just a prospect for HVMLite but also for old PV guests.

Quoted 3 lines

2) Because of 1) it has meant we have no formal semantics for early boot code is available and so severe differences can best be addressed also by yet another boot entry. This has meant often times not addressing

There are semantics written for this new code: http://xenbits.xen.org/docs/unstable/misc/hvmlite.html

All other ones related to low-level operations are described in Intel SDM.

Quoted 7 lines

or not knowing if we've addressed real differences between the different entries. Case in point, dead code [0]. How do we know we will not run certain code that should not run for the different entries ? Without *any* semantics later in boot code to distinguish where we came from and because we strive to build single kernels with different possible run time environments it means we have tons of code available to execute / run that we may not need.

I am not following that. PVH aka HVMLite will pretty much erase the need for the pvops.

Quoted 30 lines

Because of the lack of semantics we may still have dead code prospects with the new HVMLite entry. How are we sure there is no differences ?

[0] http://www.do-not-panic.com/2015/12/avoiding-dead-code-pvops-not-silver-bullet.html

3) Unikernel / other OS requirements: this is really tied to 2) but even if we tried to evolve the Xen ABI it would mean considering existing solutions out there. Things to consider as an example: FreeBSD doesn't have an EFI entry, unikernels want a simple boot entry.

With this in mind then, that I can think of:

Cons of using the same entry but having two different boot paths:

* Pushes the Xen ABI, needs to make everyone happy, this is hard
* Perhaps harder to implement

Gains of striving to use the same entry but having two different boot:

* Helps to share more code easily
* Reduce attack surface
* Requires us to have semantics for early boot; this has a series of side benefits:
- Means you should try to address differences explicitly rather than implicitly -- case in point Dead Code

Quoted 2 lines

You still need a way to distinguish between bare metal EFI and HVMlite.

Quoted 5 lines

Great point! This is the semantics aspect. The new entry for HVMlite approach deals with this by making the differences implicit by the new entry point. My call for addressing this through a hypervisor type was to see if we can get those semantics added explicitly so we can also later address dead code concerns for the new HVMLite guest type.

Right, they are..

Quoted 8 lines

Part of my own interest in an EFI entry here is that EFI could be used to help expand on the semantics in an OS/agnostic form rather than pushing the x86 boot protocol further. That seems to have its own set of drawbacks though.

Quoted 2 lines

And Xen needs a way to find out whether a kernel is supporting HVMlite to boot it in the correct mode.

Quoted 2 lines

How was Xen going to find out if new kernels had HVMlite support with the new entry ? An ELFNOTE() ? If an entry is shared could we note use an

Yeah.

Quoted 1 line

ELFNOTE() also for this though too ?

Not sure what you mean by 'shared'. But you can add multiple Elf PT_NOTEs. See the ELF document.

Quoted 5 lines · Luis R. Rodriguez

Quoted 2 lines · Juergen Gross

What would be gained by using the same entry but having two different boot paths after it?

Quoted 4 lines

Its a good question. In summary for me it would be the push for sharing more code and the push for semantics on early boot to address differences proactively, and ultimately it may enable us to help bring closer the old PV boot path closer.

But why? We want to kill PV (eventually).

Quoted 6 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

Quoted 5 lines

The HVMLite boot entry tries to bring the boot entries paths closer as it leverages more of the HVM boot path philosophy to mimic the regular PC boot path.

Is HVMLite supposed to support legacy PV guests as well BTW ?

Gosh no.

Quoted 4 lines

Reason I'm highlighting Xen ABI as a *reason* alone is that even with today's large discrepancy on the old PV boot path I believe we can bring together the boot paths closer together if the Xen ABI was slightly flexible about this, I've highlighted how I believe that is possible before,

<runs away screaming>

Quoted 33 lines

*iff* the Xen ABI would at the very least set 2 things only:

a) Hypervisor type
b) A custom data pointer

This would enable a single boot entry on the guest to handle then:

Pseudo code:

startup_32() startup_64()
| |
| |
V V
pre_hypervisor_stub_32() pre_hypervisor_stub_64()
| |
| |
V V
[existing startup_32()] [existing startup_64()]
| |
| |
V V
post_hypervisor_stub_32() post_hypervisor_stub_64()

If the Xen ABI was flexible about setting a hypervisor type and custom data pointer then we would haven handlers for it, and in it, it can do whatever it thinks is needed for its own guest types. It could also continue to set the zero page on its own as it sees fit.

Again, note that if this is done it could also mean even bringing together the old PV boot path closer together... so this is not just a prospect for HVMLite but also for old PV guests.

Quoted 3 lines

2) Because of 1) it has meant we have no formal semantics for early boot code is available and so severe differences can best be addressed also by yet another boot entry. This has meant often times not addressing

There are semantics written for this new code: http://xenbits.xen.org/docs/unstable/misc/hvmlite.html

All other ones related to low-level operations are described in Intel SDM.

Quoted 7 lines

or not knowing if we've addressed real differences between the different entries. Case in point, dead code [0]. How do we know we will not run certain code that should not run for the different entries ? Without *any* semantics later in boot code to distinguish where we came from and because we strive to build single kernels with different possible run time environments it means we have tons of code available to execute / run that we may not need.

I am not following that. PVH aka HVMLite will pretty much erase the need for the pvops.

Quoted 30 lines

Because of the lack of semantics we may still have dead code prospects with the new HVMLite entry. How are we sure there is no differences ?

[0] http://www.do-not-panic.com/2015/12/avoiding-dead-code-pvops-not-silver-bullet.html

3) Unikernel / other OS requirements: this is really tied to 2) but even if we tried to evolve the Xen ABI it would mean considering existing solutions out there. Things to consider as an example: FreeBSD doesn't have an EFI entry, unikernels want a simple boot entry.

With this in mind then, that I can think of:

Cons of using the same entry but having two different boot paths:

* Pushes the Xen ABI, needs to make everyone happy, this is hard
* Perhaps harder to implement

Gains of striving to use the same entry but having two different boot:

* Helps to share more code easily
* Reduce attack surface
* Requires us to have semantics for early boot; this has a series of side benefits:
- Means you should try to address differences explicitly rather than implicitly -- case in point Dead Code

Quoted 2 lines

You still need a way to distinguish between bare metal EFI and HVMlite.

Quoted 5 lines

Great point! This is the semantics aspect. The new entry for HVMlite approach deals with this by making the differences implicit by the new entry point. My call for addressing this through a hypervisor type was to see if we can get those semantics added explicitly so we can also later address dead code concerns for the new HVMLite guest type.

Right, they are..

Quoted 8 lines

Part of my own interest in an EFI entry here is that EFI could be used to help expand on the semantics in an OS/agnostic form rather than pushing the x86 boot protocol further. That seems to have its own set of drawbacks though.

Quoted 2 lines

And Xen needs a way to find out whether a kernel is supporting HVMlite to boot it in the correct mode.

Quoted 2 lines

How was Xen going to find out if new kernels had HVMlite support with the new entry ? An ELFNOTE() ? If an entry is shared could we note use an

Yeah.

Quoted 1 line

ELFNOTE() also for this though too ?

Not sure what you mean by 'shared'. But you can add multiple Elf PT_NOTEs. See the ELF document.

Quoted 11 lines · Konrad Rzeszutek Wilk

Quoted 9 lines · Luis R. Rodriguez

Quoted 2 lines · Juergen Gross

What would be gained by using the same entry but having two different boot paths after it?

Its a good question. In summary for me it would be the push for sharing more code and the push for semantics on early boot to address differences proactively, and ultimately it may enable us to help bring closer the old PV boot path closer.

Quoted 1 line

But why? We want to kill PV (eventually).

Yeah yeah, but its still there, and we'll have to live with it for at least minimum 5 years I hear. Part of my interest is to see to it that this path gets less disruption and issues, and we also address dead code issues which pvops simply folded under the rug. The dead code concerns may exist still for hvmlite, so unless someone is willing to make a bold claim there is none, its something to consider.

How we address semantics then is *very* important to me.

Quoted 4 lines

Quoted 4 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

Quoted 6 lines

Quoted 3 lines

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

It can still be OS agnostic and pass on type and custom data pointer.

Would that be reasonable ?

Quoted 4 lines

Quoted 4 lines

The HVMLite boot entry tries to bring the boot entries paths closer as it leverages more of the HVM boot path philosophy to mimic the regular PC boot path.

Quoted 3 lines

Quoted 1 line

Is HVMLite supposed to support legacy PV guests as well BTW ?

Gosh no.

Interesting.. and *everyone* is happy about this?

Quoted 6 lines

Quoted 5 lines

Reason I'm highlighting Xen ABI as a *reason* alone is that even with today's large discrepancy on the old PV boot path I believe we can bring together the boot paths closer together if the Xen ABI was slightly flexible about this, I've highlighted how I believe that is possible before,

Quoted 1 line

<runs away screaming>

Everyone has. If you need to support old PV guests for more than 5 years the work I'm doing should help with that. I'm trying to leverage gains of the work I'm doing for HVMLite, and part of this is trying to address semantics proactively.

Quoted 34 lines

Quoted 34 lines

*iff* the Xen ABI would at the very least set 2 things only:

a) Hypervisor type
b) A custom data pointer

This would enable a single boot entry on the guest to handle then:

Pseudo code:

startup_32() startup_64()
| |
| |
V V
pre_hypervisor_stub_32() pre_hypervisor_stub_64()
| |
| |
V V
[existing startup_32()] [existing startup_64()]
| |
| |
V V
post_hypervisor_stub_32() post_hypervisor_stub_64()

If the Xen ABI was flexible about setting a hypervisor type and custom data pointer then we would haven handlers for it, and in it, it can do whatever it thinks is needed for its own guest types. It could also continue to set the zero page on its own as it sees fit.

Again, note that if this is done it could also mean even bringing together the old PV boot path closer together... so this is not just a prospect for HVMLite but also for old PV guests.

Quoted 5 lines

Quoted 3 lines

2) Because of 1) it has meant we have no formal semantics for early boot code is available and so severe differences can best be addressed also by yet another boot entry. This has meant often times not addressing

There are semantics written for this new code: http://xenbits.xen.org/docs/unstable/misc/hvmlite.html

That only addressed semantics for early boot code implicitly through a new entry...

Quoted 12 lines

All other ones related to low-level operations are described in Intel SDM.

Quoted 7 lines

or not knowing if we've addressed real differences between the different entries. Case in point, dead code [0]. How do we know we will not run certain code that should not run for the different entries ? Without *any* semantics later in boot code to distinguish where we came from and because we strive to build single kernels with different possible run time environments it means we have tons of code available to execute / run that we may not need.

Quoted 2 lines

I am not following that. PVH aka HVMLite will pretty much erase the need for the pvops.

It does not mean there are no dead code concerns with HVMlite.

Quoted 37 lines

Quoted 36 lines

Because of the lack of semantics we may still have dead code prospects with the new HVMLite entry. How are we sure there is no differences ?

[0] http://www.do-not-panic.com/2015/12/avoiding-dead-code-pvops-not-silver-bullet.html

3) Unikernel / other OS requirements: this is really tied to 2) but even if we tried to evolve the Xen ABI it would mean considering existing solutions out there. Things to consider as an example: FreeBSD doesn't have an EFI entry, unikernels want a simple boot entry.

With this in mind then, that I can think of:

Cons of using the same entry but having two different boot paths:

* Pushes the Xen ABI, needs to make everyone happy, this is hard
* Perhaps harder to implement

Gains of striving to use the same entry but having two different boot:

* Helps to share more code easily
* Reduce attack surface
* Requires us to have semantics for early boot; this has a series of side benefits:
- Means you should try to address differences explicitly rather than implicitly -- case in point Dead Code

Quoted 2 lines

You still need a way to distinguish between bare metal EFI and HVMlite.

Great point! This is the semantics aspect. The new entry for HVMlite approach deals with this by making the differences implicit by the new entry point. My call for addressing this through a hypervisor type was to see if we can get those semantics added explicitly so we can also later address dead code concerns for the new HVMLite guest type.

Quoted 1 line

Right, they are..

There is huge merit to address a huge chunks of dead code concerns by sticking more closer to the native booth paths, it doesn't mean you still have no

dead code concerns with HVMlite, nor that HVMLite has no platform quirks, it does and part of some recent work is to pave a *clean* path for setting these differences apart.

Quoted 12 lines

Quoted 11 lines

Part of my own interest in an EFI entry here is that EFI could be used to help expand on the semantics in an OS/agnostic form rather than pushing the x86 boot protocol further. That seems to have its own set of drawbacks though.

Quoted 2 lines

And Xen needs a way to find out whether a kernel is supporting HVMlite to boot it in the correct mode.

How was Xen going to find out if new kernels had HVMlite support with the new entry ? An ELFNOTE() ? If an entry is shared could we note use an

Quoted 5 lines

Yeah.

Quoted 1 line

ELFNOTE() also for this though too ?

Not sure what you mean by 'shared'. But you can add multiple Elf PT_NOTEs. See the ELF document.

OK so even if we used a common/shared entry point we can address letting Xen find out whether or not a kernel supports HVMlite.

Luis

Quoted 12 lines · Luis R. Rodriguez

Quoted 11 lines · Konrad Rzeszutek Wilk

Quoted 9 lines · Luis R. Rodriguez

Quoted 2 lines · Juergen Gross

What would be gained by using the same entry but having two different boot paths after it?

Its a good question. In summary for me it would be the push for sharing more code and the push for semantics on early boot to address differences proactively, and ultimately it may enable us to help bring closer the old PV boot path closer.

Quoted 8 lines

Quoted 1 line

But why? We want to kill PV (eventually).

Yeah yeah, but its still there, and we'll have to live with it for at least minimum 5 years I hear. Part of my interest is to see to it that this path gets less disruption and issues, and we also address dead code issues which pvops simply folded under the rug. The dead code concerns may exist still for hvmlite, so unless someone is willing to make a bold claim there is none, its something to consider.

What is this dead code you speak of? Is it MTRR? Is early path code that PV misses (like KASL or other?)

The entrace point in Linux "proper" is startup_32 or startup_64 - the same path that EFI uses.

If you were to draw this (very simplified):

a)- GRUB2 ---------------------\ (creates an bootparam structure)

                                \
                                 +---- startup_32 or startup_64

b) EFI -> Linux EFI stub -------/
(creates bootparm) /
c) GRUB2-EFI -> Linux EFI----/
stub /
d) HVMLite ----------------/
(creates bootparm)

(I am not sure about the c) - I would have to look in source to be source). There is also LILO in this, but I am not even sure if works anymore.

What you have is that every entry point creates the bootparams and ends up calling startup_X. The startup_64 then hit the rest of the kernel. The startp_X code is the one that would setup the basic pagetables, segments, etc.

Quoted 1 line

How we address semantics then is *very* important to me.

Which semantics? How the CPU is going to be at startup_X ? Or how the CPU is going to be when EFI firmware invokes the EFI stub? Or when GRUB2 loads Linux?

That (those bootloaders) is clearly defined. The URL I provided mentions the HVMLite one. The Documentation/x86/boot.c mentions what the semantics are to expected when providing an bootstrap (which is what HVMLitel stub code in Linux would write against - and what EFI stub code had been written against too).

Quoted 9 lines

Quoted 9 lines

Quoted 7 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Quoted 3 lines

Quoted 1 line

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

It can still be OS agnostic and pass on type and custom data pointer.

Sure. It has that (it MUST otherwise how else would you pass data).
It is documented as well http://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,xen.h.html#incontents_startofday
(see " Start of day structure passed to PVH guests in %ebx.")

Quoted 22 lines · Konrad Rzeszutek Wilk

Quoted 20 lines · Luis R. Rodriguez

Quoted 6 lines · Konrad Rzeszutek Wilk

Quoted 5 lines · Luis R. Rodriguez

Quoted 2 lines · Juergen Gross

What would be gained by using the same entry but having two different boot paths after it?

Quoted 6 lines

Quoted 4 lines

Its a good question. In summary for me it would be the push for sharing more code and the push for semantics on early boot to address differences proactively, and ultimately it may enable us to help bring closer the old PV boot path closer.

But why? We want to kill PV (eventually).

Yeah yeah, but its still there, and we'll have to live with it for at least minimum 5 years I hear. Part of my interest is to see to it that this path gets less disruption and issues, and we also address dead code issues which pvops simply folded under the rug. The dead code concerns may exist still for hvmlite, so unless someone is willing to make a bold claim there is none, its something to consider.

Quoted 2 lines

What is this dead code you speak of? Is it MTRR? Is early path code that PV misses (like KASL or other?)

Kasan is dead code to Xen. If you boot x86 Xen with Kasan enabled

Xen explodes. Quick question, will Kasan not explode with HVMLite ?

MTRR used to be dead code concern but since we have vetted most of that code now we are pretty certain that code should never run now.

KASLR may be -- not sure as I haven't vetted that, but from what I have loosely heard maybe.

VGA code will be dead code for HVMlite for sure as the design doc says it will not run VGA, the ACPI flag will be set but the check for that is not yet on Linux. That means the VGA Linux code will be there but we have no way to ensure it will not run nor that anything will muck with it.

To be clear -- dead code concerns still exist even without virtualization solutions, its just that with virtualization this stuff comes up more and there has been no proactive measures to address this. The question of semantics here is to see to what extent we need earlier boot code annotations to ensure we address semantics proactively.

Quoted 7 lines

The entrace point in Linux "proper" is startup_32 or startup_64 - the same path that EFI uses.

If you were to draw this (very simplified):

a)- GRUB2 ---------------------\ (creates an bootparam structure)

Quoted 8 lines

                                \
                                 +---- startup_32 or startup_64

b) EFI -> Linux EFI stub -------/
(creates bootparm) /
c) GRUB2-EFI -> Linux EFI----/
stub /
d) HVMLite ----------------/
(creates bootparm)

b) and d) might be able to share paths there...

d) still has its own entry, it does more than create boot params.

Quoted 6 lines

(I am not sure about the c) - I would have to look in source to be source). There is also LILO in this, but I am not even sure if works anymore.

Quoted 4 lines

What you have is that every entry point creates the bootparams and ends up calling startup_X. The startup_64 then hit the rest of the kernel. The startp_X code is the one that would setup the basic pagetables, segments, etc.

Sure.. a full diagram should include both sides and how when using a custom entry one runs the risk of skipping a lot of code setup.

There is that and as others have pointed out how certain guests types are assumed to not have certain peripherals, and we have no idea to ensure certain old legacy code may not ever run or be accessed by drivers.

Quoted 6 lines

Quoted 2 lines

How we address semantics then is *very* important to me.

Which semantics? How the CPU is going to be at startup_X ? Or how the CPU is going to be when EFI firmware invokes the EFI stub? Or when GRUB2 loads Linux?

What hypervisor kicked me and what guest type I am.

Let me elaborate more below.

Quoted 16 lines

That (those bootloaders) is clearly defined. The URL I provided mentions the HVMLite one. The Documentation/x86/boot.c mentions what the semantics are to expected when providing an bootstrap (which is what HVMLitel stub code in Linux would write against - and what EFI stub code had been written against too).

Quoted 11 lines

Quoted 10 lines

Quoted 7 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

Quoted 5 lines

Quoted 1 line

It can still be OS agnostic and pass on type and custom data pointer.

Sure. It has that (it MUST otherwise how else would you pass data).
It is documented as well http://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,xen.h.html#incontents_startofday
(see " Start of day structure passed to PVH guests in %ebx.")

The design doc begs for a custom OS entry point though.

If we had a single 'type' and 'custom data' passed to the kernel that should suffice for the default Linux entry point to just pivot off of that and do what it needs without more entry points. Once.

Luis

Quoted 22 lines · Konrad Rzeszutek Wilk

Quoted 20 lines · Luis R. Rodriguez

Quoted 6 lines · Konrad Rzeszutek Wilk

Quoted 5 lines · Luis R. Rodriguez

Quoted 2 lines · Juergen Gross

What would be gained by using the same entry but having two different boot paths after it?

Quoted 6 lines

Quoted 4 lines

Its a good question. In summary for me it would be the push for sharing more code and the push for semantics on early boot to address differences proactively, and ultimately it may enable us to help bring closer the old PV boot path closer.

But why? We want to kill PV (eventually).

Yeah yeah, but its still there, and we'll have to live with it for at least minimum 5 years I hear. Part of my interest is to see to it that this path gets less disruption and issues, and we also address dead code issues which pvops simply folded under the rug. The dead code concerns may exist still for hvmlite, so unless someone is willing to make a bold claim there is none, its something to consider.

Quoted 2 lines

What is this dead code you speak of? Is it MTRR? Is early path code that PV misses (like KASL or other?)

Kasan is dead code to Xen. If you boot x86 Xen with Kasan enabled

Xen explodes. Quick question, will Kasan not explode with HVMLite ?

MTRR used to be dead code concern but since we have vetted most of that code now we are pretty certain that code should never run now.

KASLR may be -- not sure as I haven't vetted that, but from what I have loosely heard maybe.

VGA code will be dead code for HVMlite for sure as the design doc says it will not run VGA, the ACPI flag will be set but the check for that is not yet on Linux. That means the VGA Linux code will be there but we have no way to ensure it will not run nor that anything will muck with it.

To be clear -- dead code concerns still exist even without virtualization solutions, its just that with virtualization this stuff comes up more and there has been no proactive measures to address this. The question of semantics here is to see to what extent we need earlier boot code annotations to ensure we address semantics proactively.

Quoted 7 lines

The entrace point in Linux "proper" is startup_32 or startup_64 - the same path that EFI uses.

If you were to draw this (very simplified):

a)- GRUB2 ---------------------\ (creates an bootparam structure)

Quoted 8 lines

                                \
                                 +---- startup_32 or startup_64

b) EFI -> Linux EFI stub -------/
(creates bootparm) /
c) GRUB2-EFI -> Linux EFI----/
stub /
d) HVMLite ----------------/
(creates bootparm)

b) and d) might be able to share paths there...

d) still has its own entry, it does more than create boot params.

Quoted 6 lines

(I am not sure about the c) - I would have to look in source to be source). There is also LILO in this, but I am not even sure if works anymore.

Quoted 4 lines

What you have is that every entry point creates the bootparams and ends up calling startup_X. The startup_64 then hit the rest of the kernel. The startp_X code is the one that would setup the basic pagetables, segments, etc.

Sure.. a full diagram should include both sides and how when using a custom entry one runs the risk of skipping a lot of code setup.

There is that and as others have pointed out how certain guests types are assumed to not have certain peripherals, and we have no idea to ensure certain old legacy code may not ever run or be accessed by drivers.

Quoted 6 lines

Quoted 2 lines

How we address semantics then is *very* important to me.

Which semantics? How the CPU is going to be at startup_X ? Or how the CPU is going to be when EFI firmware invokes the EFI stub? Or when GRUB2 loads Linux?

What hypervisor kicked me and what guest type I am.

Let me elaborate more below.

Quoted 16 lines

That (those bootloaders) is clearly defined. The URL I provided mentions the HVMLite one. The Documentation/x86/boot.c mentions what the semantics are to expected when providing an bootstrap (which is what HVMLitel stub code in Linux would write against - and what EFI stub code had been written against too).

Quoted 11 lines

Quoted 10 lines

Quoted 7 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

Quoted 5 lines

Quoted 1 line

It can still be OS agnostic and pass on type and custom data pointer.

Sure. It has that (it MUST otherwise how else would you pass data).
It is documented as well http://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,xen.h.html#incontents_startofday
(see " Start of day structure passed to PVH guests in %ebx.")

The design doc begs for a custom OS entry point though.

If we had a single 'type' and 'custom data' passed to the kernel that should suffice for the default Linux entry point to just pivot off of that and do what it needs without more entry points. Once.

Luis

Quoted 10 lines · Luis R. Rodriguez

Quoted 9 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Luis R. Rodriguez

Quoted 1 line · Konrad Rzeszutek Wilk

On Wed, Apr 13, 2016 at 08:29:51PM +0200, Luis R. Rodriguez wrote:

and we also want to address dead code issues which pvops simply folded under the rug. The dead code concerns may exist still for hvmlite, so unless someone is willing to make a bold claim there is none, its something to consider.

Quoted 3 lines

Quoted 1 line

What is this dead code you speak of?

Kasan is dead code to Xen. If you boot x86 Xen with Kasan enabled

For Xen PV guests,

Quoted 1 line

Xen explodes. Quick question, will Kasan not explode with HVMLite ?

.. but for HVMLite of Xen HVM guest Kasan will run.

Quoted 6 lines

MTRR used to be dead code concern but since we have vetted most of that code now we are pretty certain that code should never run now.

KASLR may be -- not sure as I haven't vetted that, but from what I have loosely heard maybe.

Quoted 5 lines

VGA code will be dead code for HVMlite for sure as the design doc says it will not run VGA, the ACPI flag will be set but the check for that is not yet on Linux. That means the VGA Linux code will be there but we have no way to ensure it will not run nor that anything will muck with it.

<shrugs> The worst it will do is try to read non-existent registers.

The VGA code should be able to handle failures like that and not initialize itself when the hardware is dead (or non-existent).

Quoted 6 lines

To be clear -- dead code concerns still exist even without virtualization solutions, its just that with virtualization this stuff comes up more and there has been no proactive measures to address this. The question of semantics here is to see to what extent we need earlier boot code annotations to ensure we address semantics proactively.

I think what you mean by dead code is another word for hardware test coverage?

Quoted 15 lines

Quoted 14 lines

The entrace point in Linux "proper" is startup_32 or startup_64 - the same path that EFI uses.

If you were to draw this (very simplified):

a)- GRUB2 ---------------------\ (creates an bootparam structure)

                                \
                                 +---- startup_32 or startup_64

b) EFI -> Linux EFI stub -------/
(creates bootparm) /
c) GRUB2-EFI -> Linux EFI----/
stub /
d) HVMLite ----------------/
(creates bootparm)

Quoted 1 line

b) and d) might be able to share paths there...

No idea. You would have to look in the assembler code to figure that out.

Quoted 1 line

d) still has its own entry, it does more than create boot params.

d) purpose is to create boot params.

It may do more as nobody likes to muck in assembler and make bootparams from within assembler.

Quoted 10 lines

Quoted 9 lines

(I am not sure about the c) - I would have to look in source to be source). There is also LILO in this, but I am not even sure if works anymore.

What you have is that every entry point creates the bootparams and ends up calling startup_X. The startup_64 then hit the rest of the kernel. The startp_X code is the one that would setup the basic pagetables, segments, etc.

Quoted 2 lines

Sure.. a full diagram should include both sides and how when using a custom entry one runs the risk of skipping a lot of code setup.

But it does not skip a lot of code setup. It starts exactly at the same code startup that _all_ bootstraping code start at.

Quoted 4 lines

There is that and as others have pointed out how certain guests types are assumed to not have certain peripherals, and we have no idea to ensure certain old legacy code may not ever run or be accessed by drivers.

Ok, but that is not at code setup. That is later - when device drivers are initialized. This no different than booting on some hardware with missing functionality. ACPI, PCI and PnP PnP are set there to help OSes discover this.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

How we address semantics then is *very* important to me.

Which semantics? How the CPU is going to be at startup_X ? Or how the CPU is going to be when EFI firmware invokes the EFI stub? Or when GRUB2 loads Linux?

What hypervisor kicked me and what guest type I am.

cpuid software flags have that - and that semantics has been there for eons.

Quoted 20 lines

Let me elaborate more below.

Quoted 18 lines

That (those bootloaders) is clearly defined. The URL I provided mentions the HVMLite one. The Documentation/x86/boot.c mentions what the semantics are to expected when providing an bootstrap (which is what HVMLitel stub code in Linux would write against - and what EFI stub code had been written against too).

Quoted 12 lines

Quoted 10 lines

Quoted 7 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

It can still be OS agnostic and pass on type and custom data pointer.

Quoted 5 lines

Quoted 3 lines

Sure. It has that (it MUST otherwise how else would you pass data).
It is documented as well http://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,xen.h.html#incontents_startofday
(see " Start of day structure passed to PVH guests in %ebx.")

The design doc begs for a custom OS entry point though.

That is what the ELF Note has.

Quoted 3 lines

If we had a single 'type' and 'custom data' passed to the kernel that should suffice for the default Linux entry point to just pivot off of that and do what it needs without more entry points. Once.

And what about ramdisk? What about multiple ramdisks? What about command line? All of that is what bootparams tries to unify on Linux. But 'bootparams' is unique to Linux, it does not exist on FreeBSD. Hence some stub code to transplant OS-agnostic simple data to OS-specific is neccessary.

Quoted 10 lines · Luis R. Rodriguez

Quoted 9 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Luis R. Rodriguez

Quoted 1 line · Konrad Rzeszutek Wilk

On Wed, Apr 13, 2016 at 08:29:51PM +0200, Luis R. Rodriguez wrote:

and we also want to address dead code issues which pvops simply folded under the rug. The dead code concerns may exist still for hvmlite, so unless someone is willing to make a bold claim there is none, its something to consider.

Quoted 3 lines

Quoted 1 line

What is this dead code you speak of?

Kasan is dead code to Xen. If you boot x86 Xen with Kasan enabled

For Xen PV guests,

Quoted 1 line

Xen explodes. Quick question, will Kasan not explode with HVMLite ?

.. but for HVMLite of Xen HVM guest Kasan will run.

Quoted 6 lines

MTRR used to be dead code concern but since we have vetted most of that code now we are pretty certain that code should never run now.

KASLR may be -- not sure as I haven't vetted that, but from what I have loosely heard maybe.

Quoted 5 lines

VGA code will be dead code for HVMlite for sure as the design doc says it will not run VGA, the ACPI flag will be set but the check for that is not yet on Linux. That means the VGA Linux code will be there but we have no way to ensure it will not run nor that anything will muck with it.

<shrugs> The worst it will do is try to read non-existent registers.

The VGA code should be able to handle failures like that and not initialize itself when the hardware is dead (or non-existent).

Quoted 6 lines

To be clear -- dead code concerns still exist even without virtualization solutions, its just that with virtualization this stuff comes up more and there has been no proactive measures to address this. The question of semantics here is to see to what extent we need earlier boot code annotations to ensure we address semantics proactively.

I think what you mean by dead code is another word for hardware test coverage?

Quoted 15 lines

Quoted 14 lines

The entrace point in Linux "proper" is startup_32 or startup_64 - the same path that EFI uses.

If you were to draw this (very simplified):

a)- GRUB2 ---------------------\ (creates an bootparam structure)

                                \
                                 +---- startup_32 or startup_64

b) EFI -> Linux EFI stub -------/
(creates bootparm) /
c) GRUB2-EFI -> Linux EFI----/
stub /
d) HVMLite ----------------/
(creates bootparm)

Quoted 1 line

b) and d) might be able to share paths there...

No idea. You would have to look in the assembler code to figure that out.

Quoted 1 line

d) still has its own entry, it does more than create boot params.

d) purpose is to create boot params.

It may do more as nobody likes to muck in assembler and make bootparams from within assembler.

Quoted 10 lines

Quoted 9 lines

(I am not sure about the c) - I would have to look in source to be source). There is also LILO in this, but I am not even sure if works anymore.

What you have is that every entry point creates the bootparams and ends up calling startup_X. The startup_64 then hit the rest of the kernel. The startp_X code is the one that would setup the basic pagetables, segments, etc.

Quoted 2 lines

Sure.. a full diagram should include both sides and how when using a custom entry one runs the risk of skipping a lot of code setup.

But it does not skip a lot of code setup. It starts exactly at the same code startup that _all_ bootstraping code start at.

Quoted 4 lines

There is that and as others have pointed out how certain guests types are assumed to not have certain peripherals, and we have no idea to ensure certain old legacy code may not ever run or be accessed by drivers.

Ok, but that is not at code setup. That is later - when device drivers are initialized. This no different than booting on some hardware with missing functionality. ACPI, PCI and PnP PnP are set there to help OSes discover this.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

How we address semantics then is *very* important to me.

Which semantics? How the CPU is going to be at startup_X ? Or how the CPU is going to be when EFI firmware invokes the EFI stub? Or when GRUB2 loads Linux?

What hypervisor kicked me and what guest type I am.

cpuid software flags have that - and that semantics has been there for eons.

Quoted 20 lines

Let me elaborate more below.

Quoted 18 lines

That (those bootloaders) is clearly defined. The URL I provided mentions the HVMLite one. The Documentation/x86/boot.c mentions what the semantics are to expected when providing an bootstrap (which is what HVMLitel stub code in Linux would write against - and what EFI stub code had been written against too).

Quoted 12 lines

Quoted 10 lines

Quoted 7 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

It can still be OS agnostic and pass on type and custom data pointer.

Quoted 5 lines

Quoted 3 lines

Sure. It has that (it MUST otherwise how else would you pass data).
It is documented as well http://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,xen.h.html#incontents_startofday
(see " Start of day structure passed to PVH guests in %ebx.")

The design doc begs for a custom OS entry point though.

That is what the ELF Note has.

Quoted 3 lines

If we had a single 'type' and 'custom data' passed to the kernel that should suffice for the default Linux entry point to just pivot off of that and do what it needs without more entry points. Once.

And what about ramdisk? What about multiple ramdisks? What about command line? All of that is what bootparams tries to unify on Linux. But 'bootparams' is unique to Linux, it does not exist on FreeBSD. Hence some stub code to transplant OS-agnostic simple data to OS-specific is neccessary.

Quoted 11 lines · Konrad Rzeszutek Wilk

Quoted 10 lines · Luis R. Rodriguez

Quoted 9 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Luis R. Rodriguez

Quoted 1 line · Konrad Rzeszutek Wilk

On Wed, Apr 13, 2016 at 08:29:51PM +0200, Luis R. Rodriguez wrote:

and we also want to address dead code issues which pvops simply folded under the rug. The dead code concerns may exist still for hvmlite, so unless someone is willing to make a bold claim there is none, its something to consider.

Quoted 5 lines

Quoted 3 lines

Quoted 1 line

What is this dead code you speak of?

Kasan is dead code to Xen. If you boot x86 Xen with Kasan enabled

For Xen PV guests,

That's right. For 5 years this will be a bomb. That went unnoticed and I feel I have to pull hair now to try to get folks to fix this.

How many other issues will go in which will explode during this 5 year time line? How can we proactively address a solution to this now so we avoid this in future ?

Do you believe me now its a real issue?

Fortunately I have a proactive solution for pvops now in my pipeline that should help avoid us having to blow more things up on Xen but also that should cause no headaches on behalf of x86 developers. But, reason I have been so engaged on HVMLite design review is I want to ensure we take the lessons learned from pvops and avoid this for an architecture that will be def-facto Xen on Linux 5 years from now.

Not bringing this up or addressing this now for HVMLite / PVH2 would simply be silly, and since it wasn't addressed in pvops I obviously have to ensure I convince enough people it was a real issue and ensure that we have enough semantics available to address it.

Part of the semantics question, which has made my quest hard, was the use of semantics for virtualization for code on early boot and later in boot has been rather sloppy, so we have recently needed to address some of these gaps. Some of these discussions have however been productive, as I'll explain to George soon regarding his DT questions. The discussion is not over though and we need to ensure that if we need semantics for HVMLite we'll have them available in *clean* way. One of the things where early semantics and design to address these issue help in a proactive manner is to address a clean boot entry -- and that's also why I've been so pedantic over review of the new HVMlite boot entry.

Quoted 4 lines

Quoted 2 lines

Xen explodes. Quick question, will Kasan not explode with HVMLite ?

.. but for HVMLite of Xen HVM guest Kasan will run.

Are you sure? Should that mean that Xen HVM should be fine as well. Does that work? Are we sure?

Quoted 13 lines

Quoted 12 lines

MTRR used to be dead code concern but since we have vetted most of that code now we are pretty certain that code should never run now.

KASLR may be -- not sure as I haven't vetted that, but from what I have loosely heard maybe.

VGA code will be dead code for HVMlite for sure as the design doc says it will not run VGA, the ACPI flag will be set but the check for that is not yet on Linux. That means the VGA Linux code will be there but we have no way to ensure it will not run nor that anything will muck with it.

Quoted 1 line

<shrugs> The worst it will do is try to read non-existent registers.

Really ?

Is that your position on all other possible dead code that may have been possible on old Xen PV guests as well ?

As I hinted, after thinking about this for a while I realized that dead code is likely present on bare metal as well even without virtualization, specially if

you build large single kernels to support a wide array of features which only late at run time can be determined. Virtualization and the pvops design just makes this issue much more prominent. If there are other areas of code exposed that actually may run, but we are not sure may run, I figured some other folks with a bit more security conscience minds might even simply take the position it may be a security risk to leave that code exposed. So to take a position that 'the worst it will do is try to read non-existent registers' -- seems rather shortsighted here.

Anyway for more details on thoughts on this refer to the this wiki:

http://kernelnewbies.org/KernelProjects/kernel-sandboxing

Since this is now getting off topic please send me your feedback on another thread for the non-virtualization aspects of this if that interests you. My point here was rather to highlight the importance of clear semantics due to virtualization in light of possible dead code.

Quoted 3 lines

The VGA code should be able to handle failures like that and not initialize itself when the hardware is dead (or non-existent).

That's right, its through ACPI_FADT_NO_VGA and since its part of the HVMLite design doc we want HVMlite design to address ACPI_FADT_NO_VGA properly. I've paved the way for this to be done cleanly and easily now, but that code should be in place before HVMLite code gets merged.

Does domU for old Xen PV also set ACPI_FADT_NO_VGA as well ? Should it ?

Quoted 8 lines

Quoted 7 lines

To be clear -- dead code concerns still exist even without virtualization solutions, its just that with virtualization this stuff comes up more and there has been no proactive measures to address this. The question of semantics here is to see to what extent we need earlier boot code annotations to ensure we address semantics proactively.

Quoted 2 lines

I think what you mean by dead code is another word for hardware test coverage?

No, no, its very different given that with virtualization the scope of possible dead code is significant and at run time you are certain a huge portion of code should *never ever* run. So for instance we know once we boot bare metal none of the Xen stuff should ever run, likewise on Xen dom0 we know none of the KVM / bare-metal only stuff should never run, when on Xen domU, none of the Xen

domU-only stuff should ever run.

Quoted 16 lines

Quoted 16 lines

Quoted 15 lines

The entrace point in Linux "proper" is startup_32 or startup_64 - the same path that EFI uses.

If you were to draw this (very simplified):

a)- GRUB2 ---------------------\ (creates an bootparam structure)

                                \
                                 +---- startup_32 or startup_64

b) EFI -> Linux EFI stub -------/
(creates bootparm) /
c) GRUB2-EFI -> Linux EFI----/
stub /
d) HVMLite ----------------/
(creates bootparm)

Quoted 4 lines

Quoted 1 line

b) and d) might be able to share paths there...

No idea. You would have to look in the assembler code to figure that out.

And that's a pain, I get it.

I spotted one place already -- will note to Boris. I think Matt may have more ideas ;)

Quoted 3 lines

Quoted 1 line

d) still has its own entry, it does more than create boot params.

d) purpose is to create boot params.

OK good to know that's the only thing we acknowledge it *should* do.

Quoted 3 lines

It may do more as nobody likes to muck in assembler and make bootparams from within assembler.

OK -- it does do more and that's where we'd like to avoid duplication if possible and yet-another-entry (TM).

Quoted 11 lines

Quoted 11 lines

Quoted 10 lines

(I am not sure about the c) - I would have to look in source to be source). There is also LILO in this, but I am not even sure if works anymore.

What you have is that every entry point creates the bootparams and ends up calling startup_X. The startup_64 then hit the rest of the kernel. The startp_X code is the one that would setup the basic pagetables, segments, etc.

Quoted 5 lines

Quoted 2 lines

Sure.. a full diagram should include both sides and how when using a custom entry one runs the risk of skipping a lot of code setup.

But it does not skip a lot of code setup. It starts exactly at the same code startup that _all_ bootstraping code start at.

Its a fair point.

Quoted 5 lines

Quoted 4 lines

There is that and as others have pointed out how certain guests types are assumed to not have certain peripherals, and we have no idea to ensure certain old legacy code may not ever run or be accessed by drivers.

Quoted 4 lines

Ok, but that is not at code setup. That is later - when device drivers are initialized. This no different than booting on some hardware with missing functionality. ACPI, PCI and PnP PnP are set there to help OSes discover this.

To a certain extent this is true, but there may things which are missing still.

We really have no idea what the full list of those things are.

It may be that things may have been running for ages without notice of an issue or that only under certain situations will certain issues or bugs trigger a failure. For instance, just yesterday I was Cc'd on a brand-spanking new legacy conflict [0], caused by upstream commit 8c058b0b9c34d8c ("x86/irq: Probe for PIC presence before allocating descs for legacy IRQs") merged on v4.4 where some new code used nr_legacy_irqs() -- one proposed solution seems to be that for Xen code NR_IRQS_LEGACY should be used instead is as it lacks PCI [1] and another was to peg the legacy requirements as a quirk on the new x86 platform legacy quirk stuff [2]. Are other uses of nr_legacy_irqs() correct ? Are we sure ?

[0] http://lkml.kernel.org/r/[email protected]
[1] https://lkml.org/lkml/2016/4/14/532
[2] http://lkml.kernel.org/r/[email protected]

Quoted 6 lines

Quoted 6 lines

Quoted 5 lines

Quoted 1 line

How we address semantics then is *very* important to me.

Which semantics? How the CPU is going to be at startup_X ? Or how the CPU is going to be when EFI firmware invokes the EFI stub? Or when GRUB2 loads Linux?

Quoted 4 lines

Quoted 1 line

What hypervisor kicked me and what guest type I am.

cpuid software flags have that - and that semantics has been there for eons.

We cannot use cpuid early in asm code, I'm looking for something we

can even use on asm early in boot code, on x86 the best option we have is the boot_params, but I've even have had issues with that early in code, as I can only access it after load_idt() where I described my effort to unify Xen PV and x86_64 init paths [3].

[3] http://lkml.kernel.org/r/CAB=NE6VTCRCazcNpCdJ7pN1eD3=x_fcGOdH37MzVpxkKEN5esw@mail.gmail.com

Quoted 26 lines

Quoted 25 lines

Let me elaborate more below.

Quoted 14 lines

That (those bootloaders) is clearly defined. The URL I provided mentions the HVMLite one. The Documentation/x86/boot.c mentions what the semantics are to expected when providing an bootstrap (which is what HVMLitel stub code in Linux would write against - and what EFI stub code had been written against too).

Quoted 9 lines

Quoted 9 lines

Quoted 7 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Quoted 7 lines

Quoted 3 lines

Quoted 1 line

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

It can still be OS agnostic and pass on type and custom data pointer.

Sure. It has that (it MUST otherwise how else would you pass data).
It is documented as well http://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,xen.h.html#incontents_startofday
(see " Start of day structure passed to PVH guests in %ebx.")

The design doc begs for a custom OS entry point though.

Quoted 1 line

That is what the ELF Note has.

Right, but I'm saying that its rather silly to be adding entry points if all we want the code to do is copy the boot params for us. The design doc requires a new entry, and likewise you'd need yet-another-entry if HVMLite is thrown out the window and come back 5 years later after new hardware solutions are in place and need to redesign HVMLite. Kind of

where we are with PVH today. Likewise if other paravirtualization developers want to support Linux and want to copy your strategy they'd add yet-another-entry-point as well.

This is dumb.

Quoted 10 lines

Quoted 4 lines

If we had a single 'type' and 'custom data' passed to the kernel that should suffice for the default Linux entry point to just pivot off of that and do what it needs without more entry points. Once.

And what about ramdisk? What about multiple ramdisks? What about command line? All of that is what bootparams tries to unify on Linux. But 'bootparams' is unique to Linux, it does not exist on FreeBSD. Hence some stub code to transplant OS-agnostic simple data to OS-specific is neccessary.

If we had a Xen ABI option where *all* that I'm asking is you pass first:

a) hypervisor type

b) custom data pointer

We'd be able to avoid adding *any* entry point and just address the requirements as I noted with pre / post stubs for the type.

This would require an x86 boot protocol bump, but all the issues creeping up randomly I think that's worth putting on the table now.

And maybe we don't want it to be hypervisor specific, perhaps there are other *needs* for custom pre-post startup_32()/startup_64() stubs.

To avoid extending boot_params further I figured perhaps we can look at EFI as another option instead. If we are going to drop all legacy

PV support from the kernel (not the hypervisor) and require hardware virtualization 5 years from now on the Linux kernel, it doesn't seem to me far fetched to at the very least consider using an EFI entry instead, specially since all it does is set boot params and we can make re-use this for HVMLite too.

Luis

Quoted 10 lines · Luis R. Rodriguez

Quoted 8 lines · Konrad Rzeszutek Wilk

Quoted 5 lines · Luis R. Rodriguez

VGA code will be dead code for HVMlite for sure as the design doc says it will not run VGA, the ACPI flag will be set but the check for that is not yet on Linux. That means the VGA Linux code will be there but we have no way to ensure it will not run nor that anything will muck with it.

<shrugs> The worst it will do is try to read non-existent registers.

Quoted 4 lines

Really ?

Is that your position on all other possible dead code that may have been possible on old Xen PV guests as well ?

This is not just with Xen - it with other device drivers that are being invoked on baremetal and are not present in hardware anymore.

Quoted 2 lines

As I hinted, after thinking about this for a while I realized that dead code is likely present on bare metal as well even without virtualization, specially if

Yes!

Quoted 8 lines

you build large single kernels to support a wide array of features which only late at run time can be determined. Virtualization and the pvops design just makes this issue much more prominent. If there are other areas of code exposed that actually may run, but we are not sure may run, I figured some other folks with a bit more security conscience minds might even simply take the position it may be a security risk to leave that code exposed. So to take a position that 'the worst it will do is try to read non-existent registers' -- seems rather shortsighted here.

Security conscious people trim their CONFIG.

Quoted 8 lines

Anyway for more details on thoughts on this refer to the this wiki:

http://kernelnewbies.org/KernelProjects/kernel-sandboxing

Since this is now getting off topic please send me your feedback on another thread for the non-virtualization aspects of this if that interests you. My point here was rather to highlight the importance of clear semantics due to virtualization in light of possible dead code.

Thank you.

Quoted 9 lines

Quoted 3 lines

The VGA code should be able to handle failures like that and not initialize itself when the hardware is dead (or non-existent).

That's right, its through ACPI_FADT_NO_VGA and since its part of the HVMLite design doc we want HVMlite design to address ACPI_FADT_NO_VGA properly. I've paved the way for this to be done cleanly and easily now, but that code should be in place before HVMLite code gets merged.

Quoted 1 line

Does domU for old Xen PV also set ACPI_FADT_NO_VGA as well ? Should it ?

It does not. Not sure - it seems to have worked fine for the last ten years?

Quoted 10 lines

Quoted 9 lines

Quoted 6 lines

To be clear -- dead code concerns still exist even without virtualization solutions, its just that with virtualization this stuff comes up more and there has been no proactive measures to address this. The question of semantics here is to see to what extent we need earlier boot code annotations to ensure we address semantics proactively.

I think what you mean by dead code is another word for hardware test coverage?

Quoted 5 lines

No, no, its very different given that with virtualization the scope of possible dead code is significant and at run time you are certain a huge portion of code should *never ever* run. So for instance we know once we boot bare metal none of the Xen stuff should ever run, likewise on Xen dom0 we know none of the KVM / bare-metal only stuff should never run, when on Xen domU, none of the Xen

What is this 'bare metal only stuff' you speak of? On Xen dom0 most of the baremetal code is running.

In fact that is how the device drivers work. Or are you talking about low level baremetal code? If so, then PVH/HVMLite does that - it skips pvops so that it can run this 'low-level baremetal code'

Quoted 1 line

domU-only stuff should ever run.

You forgot KVM guest support on baremetal. That shouldn't run either.

Quoted 25 lines

Quoted 15 lines

Quoted 15 lines

Quoted 14 lines

The entrace point in Linux "proper" is startup_32 or startup_64 - the same path that EFI uses.

If you were to draw this (very simplified):

a)- GRUB2 ---------------------\ (creates an bootparam structure)

                                \
                                 +---- startup_32 or startup_64

b) EFI -> Linux EFI stub -------/
(creates bootparm) /
c) GRUB2-EFI -> Linux EFI----/
stub /
d) HVMLite ----------------/
(creates bootparm)

Quoted 4 lines

Quoted 1 line

b) and d) might be able to share paths there...

No idea. You would have to look in the assembler code to figure that out.

And that's a pain, I get it.

I spotted one place already -- will note to Boris. I think Matt may have more ideas ;)

Quoted 5 lines

Quoted 3 lines

Quoted 1 line

d) still has its own entry, it does more than create boot params.

d) purpose is to create boot params.

OK good to know that's the only thing we acknowledge it *should* do.

And b), c) purpose is for that too - amongts providing an mechanism to call in EFI firmware.

And I realized that early baremetal boot option also ends up calling C during its startup (see main in arch/x86/boot/main.c) amongst then switching different modes.

Quoted 5 lines

Quoted 2 lines

It may do more as nobody likes to muck in assembler and make bootparams from within assembler.

OK -- it does do more and that's where we'd like to avoid duplication if possible and yet-another-entry (TM).

It does more? EFI stub entry does more than the GRUB2 entry.

If you have some patches to trim the code duplication within those boot paths- please post it.

Quoted 23 lines

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

(I am not sure about the c) - I would have to look in source to be source). There is also LILO in this, but I am not even sure if works anymore.

What you have is that every entry point creates the bootparams and ends up calling startup_X. The startup_64 then hit the rest of the kernel. The startp_X code is the one that would setup the basic pagetables, segments, etc.

Quoted 5 lines

Quoted 2 lines

Sure.. a full diagram should include both sides and how when using a custom entry one runs the risk of skipping a lot of code setup.

But it does not skip a lot of code setup. It starts exactly at the same code startup that _all_ bootstraping code start at.

Its a fair point.

Quoted 5 lines

Quoted 4 lines

There is that and as others have pointed out how certain guests types are assumed to not have certain peripherals, and we have no idea to ensure certain old legacy code may not ever run or be accessed by drivers.

Quoted 6 lines

Quoted 4 lines

Ok, but that is not at code setup. That is later - when device drivers are initialized. This no different than booting on some hardware with missing functionality. ACPI, PCI and PnP PnP are set there to help OSes discover this.

To a certain extent this is true, but there may things which are missing still.

Like?

Quoted 1 line

We really have no idea what the full list of those things are.

Ok, it sounds like you have some homework.

Quoted 10 lines

It may be that things may have been running for ages without notice of an issue or that only under certain situations will certain issues or bugs trigger a failure. For instance, just yesterday I was Cc'd on a brand-spanking new legacy conflict [0], caused by upstream commit 8c058b0b9c34d8c ("x86/irq: Probe for PIC presence before allocating descs for legacy IRQs") merged on v4.4 where some new code used nr_legacy_irqs() -- one proposed solution seems to be that for Xen code NR_IRQS_LEGACY should be used instead is as it lacks PCI [1] and another was to peg the legacy requirements as a quirk on the new x86 platform legacy quirk stuff [2]. Are other uses of nr_legacy_irqs() correct ? Are we sure ?

And how is this example related to 'early bootup' path?

It is not.

It is in fact related to PV codepaths - which PVH/HVMLite and HVM guests do not exercise.

Quoted 10 lines

[0] http://lkml.kernel.org/r/[email protected]
[1] https://lkml.org/lkml/2016/4/14/532
[2] http://lkml.kernel.org/r/[email protected]

Quoted 6 lines

Quoted 6 lines

Quoted 5 lines

Quoted 1 line

How we address semantics then is *very* important to me.

Which semantics? How the CPU is going to be at startup_X ? Or how the CPU is going to be when EFI firmware invokes the EFI stub? Or when GRUB2 loads Linux?

Quoted 6 lines

Quoted 4 lines

Quoted 1 line

What hypervisor kicked me and what guest type I am.

cpuid software flags have that - and that semantics has been there for eons.

We cannot use cpuid early in asm code, I'm looking for something we

?! Why!?

Quoted 4 lines

can even use on asm early in boot code, on x86 the best option we have is the boot_params, but I've even have had issues with that early in code, as I can only access it after load_idt() where I described my effort to unify Xen PV and x86_64 init paths [3].

Well, Xen PV skips x86_64_start_kernel..

Quoted 30 lines

[3] http://lkml.kernel.org/r/CAB=NE6VTCRCazcNpCdJ7pN1eD3=x_fcGOdH37MzVpxkKEN5esw@mail.gmail.com

Quoted 27 lines

Quoted 18 lines

Let me elaborate more below.

Quoted 16 lines

That (those bootloaders) is clearly defined. The URL I provided mentions the HVMLite one. The Documentation/x86/boot.c mentions what the semantics are to expected when providing an bootstrap (which is what HVMLitel stub code in Linux would write against - and what EFI stub code had been written against too).

Quoted 11 lines

Quoted 10 lines

Quoted 7 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

It can still be OS agnostic and pass on type and custom data pointer.

Sure. It has that (it MUST otherwise how else would you pass data).
It is documented as well http://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,xen.h.html#incontents_startofday
(see " Start of day structure passed to PVH guests in %ebx.")

The design doc begs for a custom OS entry point though.

That is what the ELF Note has.

Quoted 5 lines

Right, but I'm saying that its rather silly to be adding entry points if all we want the code to do is copy the boot params for us. The design doc requires a new entry, and likewise you'd need yet-another-entry if HVMLite is thrown out the window and come back 5 years later after new hardware solutions are in place and need to redesign HVMLite. Kind of

Why would you need to redesign HVMLite based on hardware solutions?

The entrace point and the CPU state are pretty well known - it is akin to what GRUB2 bootloader path is (protected mode).

Quoted 5 lines

where we are with PVH today. Likewise if other paravirtualization developers want to support Linux and want to copy your strategy they'd add yet-another-entry-point as well.

This is dumb.

You saying the EFI entry point is dumb? That instead the EFI firmware should understand Linux bootparams and booted that?

Quoted 10 lines

Quoted 9 lines

Quoted 3 lines

If we had a single 'type' and 'custom data' passed to the kernel that should suffice for the default Linux entry point to just pivot off of that and do what it needs without more entry points. Once.

And what about ramdisk? What about multiple ramdisks? What about command line? All of that is what bootparams tries to unify on Linux. But 'bootparams' is unique to Linux, it does not exist on FreeBSD. Hence some stub code to transplant OS-agnostic simple data to OS-specific is neccessary.

Quoted 4 lines

If we had a Xen ABI option where *all* that I'm asking is you pass first:

a) hypervisor type

Why can't you use cpuid.

Quoted 1 line

b) custom data pointer

What is this custom data pointer you speak of?

Quoted 2 lines

We'd be able to avoid adding *any* entry point and just address the requirements as I noted with pre / post stubs for the type.

But you need some entry point to call into Linux. Are you suggesting to use the existing ones? No, the existing one wouldn't understand this.

Quoted 2 lines

This would require an x86 boot protocol bump, but all the issues creeping up randomly I think that's worth putting on the table now.

Aaaah, so you are saying expand the bootparams. In other words make Xen ABI call into Linux using the bootparams structure, similar to how GRUB2 does it.

How is that OS agnostic?

Quoted 2 lines

And maybe we don't want it to be hypervisor specific, perhaps there are other *needs* for custom pre-post startup_32()/startup_64() stubs.

Multiboot?

Quoted 2 lines

To avoid extending boot_params further I figured perhaps we can look at EFI as another option instead. If we are going to drop all legacy

But EFI support is _huge_.

Quoted 5 lines

PV support from the kernel (not the hypervisor) and require hardware virtualization 5 years from now on the Linux kernel, it doesn't seem to me far fetched to at the very least consider using an EFI entry instead, specially since all it does is set boot params and we can make re-use this for HVMLite too.

But to make that work you have to emulate EFI firmware in the hypervisor. Is that work you are signing up for?

Quoted 10 lines · Luis R. Rodriguez

Quoted 8 lines · Konrad Rzeszutek Wilk

Quoted 5 lines · Luis R. Rodriguez

VGA code will be dead code for HVMlite for sure as the design doc says it will not run VGA, the ACPI flag will be set but the check for that is not yet on Linux. That means the VGA Linux code will be there but we have no way to ensure it will not run nor that anything will muck with it.

<shrugs> The worst it will do is try to read non-existent registers.

Quoted 4 lines

Really ?

Is that your position on all other possible dead code that may have been possible on old Xen PV guests as well ?

This is not just with Xen - it with other device drivers that are being invoked on baremetal and are not present in hardware anymore.

Quoted 2 lines

As I hinted, after thinking about this for a while I realized that dead code is likely present on bare metal as well even without virtualization, specially if

Yes!

Quoted 8 lines

you build large single kernels to support a wide array of features which only late at run time can be determined. Virtualization and the pvops design just makes this issue much more prominent. If there are other areas of code exposed that actually may run, but we are not sure may run, I figured some other folks with a bit more security conscience minds might even simply take the position it may be a security risk to leave that code exposed. So to take a position that 'the worst it will do is try to read non-existent registers' -- seems rather shortsighted here.

Security conscious people trim their CONFIG.

Quoted 8 lines

Anyway for more details on thoughts on this refer to the this wiki:

http://kernelnewbies.org/KernelProjects/kernel-sandboxing

Since this is now getting off topic please send me your feedback on another thread for the non-virtualization aspects of this if that interests you. My point here was rather to highlight the importance of clear semantics due to virtualization in light of possible dead code.

Thank you.

Quoted 9 lines

Quoted 3 lines

The VGA code should be able to handle failures like that and not initialize itself when the hardware is dead (or non-existent).

That's right, its through ACPI_FADT_NO_VGA and since its part of the HVMLite design doc we want HVMlite design to address ACPI_FADT_NO_VGA properly. I've paved the way for this to be done cleanly and easily now, but that code should be in place before HVMLite code gets merged.

Quoted 1 line

Does domU for old Xen PV also set ACPI_FADT_NO_VGA as well ? Should it ?

It does not. Not sure - it seems to have worked fine for the last ten years?

Quoted 10 lines

Quoted 9 lines

Quoted 6 lines

To be clear -- dead code concerns still exist even without virtualization solutions, its just that with virtualization this stuff comes up more and there has been no proactive measures to address this. The question of semantics here is to see to what extent we need earlier boot code annotations to ensure we address semantics proactively.

I think what you mean by dead code is another word for hardware test coverage?

Quoted 5 lines

No, no, its very different given that with virtualization the scope of possible dead code is significant and at run time you are certain a huge portion of code should *never ever* run. So for instance we know once we boot bare metal none of the Xen stuff should ever run, likewise on Xen dom0 we know none of the KVM / bare-metal only stuff should never run, when on Xen domU, none of the Xen

What is this 'bare metal only stuff' you speak of? On Xen dom0 most of the baremetal code is running.

In fact that is how the device drivers work. Or are you talking about low level baremetal code? If so, then PVH/HVMLite does that - it skips pvops so that it can run this 'low-level baremetal code'

Quoted 1 line

domU-only stuff should ever run.

You forgot KVM guest support on baremetal. That shouldn't run either.

Quoted 25 lines

Quoted 15 lines

Quoted 15 lines

Quoted 14 lines

The entrace point in Linux "proper" is startup_32 or startup_64 - the same path that EFI uses.

If you were to draw this (very simplified):

a)- GRUB2 ---------------------\ (creates an bootparam structure)

                                \
                                 +---- startup_32 or startup_64

b) EFI -> Linux EFI stub -------/
(creates bootparm) /
c) GRUB2-EFI -> Linux EFI----/
stub /
d) HVMLite ----------------/
(creates bootparm)

Quoted 4 lines

Quoted 1 line

b) and d) might be able to share paths there...

No idea. You would have to look in the assembler code to figure that out.

And that's a pain, I get it.

I spotted one place already -- will note to Boris. I think Matt may have more ideas ;)

Quoted 5 lines

Quoted 3 lines

Quoted 1 line

d) still has its own entry, it does more than create boot params.

d) purpose is to create boot params.

OK good to know that's the only thing we acknowledge it *should* do.

And b), c) purpose is for that too - amongts providing an mechanism to call in EFI firmware.

And I realized that early baremetal boot option also ends up calling C during its startup (see main in arch/x86/boot/main.c) amongst then switching different modes.

Quoted 5 lines

Quoted 2 lines

It may do more as nobody likes to muck in assembler and make bootparams from within assembler.

OK -- it does do more and that's where we'd like to avoid duplication if possible and yet-another-entry (TM).

It does more? EFI stub entry does more than the GRUB2 entry.

If you have some patches to trim the code duplication within those boot paths- please post it.

Quoted 23 lines

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

(I am not sure about the c) - I would have to look in source to be source). There is also LILO in this, but I am not even sure if works anymore.

What you have is that every entry point creates the bootparams and ends up calling startup_X. The startup_64 then hit the rest of the kernel. The startp_X code is the one that would setup the basic pagetables, segments, etc.

Quoted 5 lines

Quoted 2 lines

Sure.. a full diagram should include both sides and how when using a custom entry one runs the risk of skipping a lot of code setup.

But it does not skip a lot of code setup. It starts exactly at the same code startup that _all_ bootstraping code start at.

Its a fair point.

Quoted 5 lines

Quoted 4 lines

There is that and as others have pointed out how certain guests types are assumed to not have certain peripherals, and we have no idea to ensure certain old legacy code may not ever run or be accessed by drivers.

Quoted 6 lines

Quoted 4 lines

Ok, but that is not at code setup. That is later - when device drivers are initialized. This no different than booting on some hardware with missing functionality. ACPI, PCI and PnP PnP are set there to help OSes discover this.

To a certain extent this is true, but there may things which are missing still.

Like?

Quoted 1 line

We really have no idea what the full list of those things are.

Ok, it sounds like you have some homework.

Quoted 10 lines

It may be that things may have been running for ages without notice of an issue or that only under certain situations will certain issues or bugs trigger a failure. For instance, just yesterday I was Cc'd on a brand-spanking new legacy conflict [0], caused by upstream commit 8c058b0b9c34d8c ("x86/irq: Probe for PIC presence before allocating descs for legacy IRQs") merged on v4.4 where some new code used nr_legacy_irqs() -- one proposed solution seems to be that for Xen code NR_IRQS_LEGACY should be used instead is as it lacks PCI [1] and another was to peg the legacy requirements as a quirk on the new x86 platform legacy quirk stuff [2]. Are other uses of nr_legacy_irqs() correct ? Are we sure ?

And how is this example related to 'early bootup' path?

It is not.

It is in fact related to PV codepaths - which PVH/HVMLite and HVM guests do not exercise.

Quoted 10 lines

[0] http://lkml.kernel.org/r/[email protected]
[1] https://lkml.org/lkml/2016/4/14/532
[2] http://lkml.kernel.org/r/[email protected]

Quoted 6 lines

Quoted 6 lines

Quoted 5 lines

Quoted 1 line

How we address semantics then is *very* important to me.

Which semantics? How the CPU is going to be at startup_X ? Or how the CPU is going to be when EFI firmware invokes the EFI stub? Or when GRUB2 loads Linux?

Quoted 6 lines

Quoted 4 lines

Quoted 1 line

What hypervisor kicked me and what guest type I am.

cpuid software flags have that - and that semantics has been there for eons.

We cannot use cpuid early in asm code, I'm looking for something we

?! Why!?

Quoted 4 lines

can even use on asm early in boot code, on x86 the best option we have is the boot_params, but I've even have had issues with that early in code, as I can only access it after load_idt() where I described my effort to unify Xen PV and x86_64 init paths [3].

Well, Xen PV skips x86_64_start_kernel..

Quoted 30 lines

[3] http://lkml.kernel.org/r/CAB=NE6VTCRCazcNpCdJ7pN1eD3=x_fcGOdH37MzVpxkKEN5esw@mail.gmail.com

Quoted 27 lines

Quoted 18 lines

Let me elaborate more below.

Quoted 16 lines

That (those bootloaders) is clearly defined. The URL I provided mentions the HVMLite one. The Documentation/x86/boot.c mentions what the semantics are to expected when providing an bootstrap (which is what HVMLitel stub code in Linux would write against - and what EFI stub code had been written against too).

Quoted 11 lines

Quoted 10 lines

Quoted 7 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

It can still be OS agnostic and pass on type and custom data pointer.

Sure. It has that (it MUST otherwise how else would you pass data).
It is documented as well http://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,xen.h.html#incontents_startofday
(see " Start of day structure passed to PVH guests in %ebx.")

The design doc begs for a custom OS entry point though.

That is what the ELF Note has.

Quoted 5 lines

Right, but I'm saying that its rather silly to be adding entry points if all we want the code to do is copy the boot params for us. The design doc requires a new entry, and likewise you'd need yet-another-entry if HVMLite is thrown out the window and come back 5 years later after new hardware solutions are in place and need to redesign HVMLite. Kind of

Why would you need to redesign HVMLite based on hardware solutions?

The entrace point and the CPU state are pretty well known - it is akin to what GRUB2 bootloader path is (protected mode).

Quoted 5 lines

where we are with PVH today. Likewise if other paravirtualization developers want to support Linux and want to copy your strategy they'd add yet-another-entry-point as well.

This is dumb.

You saying the EFI entry point is dumb? That instead the EFI firmware should understand Linux bootparams and booted that?

Quoted 10 lines

Quoted 9 lines

Quoted 3 lines

If we had a single 'type' and 'custom data' passed to the kernel that should suffice for the default Linux entry point to just pivot off of that and do what it needs without more entry points. Once.

And what about ramdisk? What about multiple ramdisks? What about command line? All of that is what bootparams tries to unify on Linux. But 'bootparams' is unique to Linux, it does not exist on FreeBSD. Hence some stub code to transplant OS-agnostic simple data to OS-specific is neccessary.

Quoted 4 lines

If we had a Xen ABI option where *all* that I'm asking is you pass first:

a) hypervisor type

Why can't you use cpuid.

Quoted 1 line

b) custom data pointer

What is this custom data pointer you speak of?

Quoted 2 lines

We'd be able to avoid adding *any* entry point and just address the requirements as I noted with pre / post stubs for the type.

But you need some entry point to call into Linux. Are you suggesting to use the existing ones? No, the existing one wouldn't understand this.

Quoted 2 lines

This would require an x86 boot protocol bump, but all the issues creeping up randomly I think that's worth putting on the table now.

Aaaah, so you are saying expand the bootparams. In other words make Xen ABI call into Linux using the bootparams structure, similar to how GRUB2 does it.

How is that OS agnostic?

Quoted 2 lines

And maybe we don't want it to be hypervisor specific, perhaps there are other *needs* for custom pre-post startup_32()/startup_64() stubs.

Multiboot?

Quoted 2 lines

To avoid extending boot_params further I figured perhaps we can look at EFI as another option instead. If we are going to drop all legacy

But EFI support is _huge_.

Quoted 5 lines

PV support from the kernel (not the hypervisor) and require hardware virtualization 5 years from now on the Linux kernel, it doesn't seem to me far fetched to at the very least consider using an EFI entry instead, specially since all it does is set boot params and we can make re-use this for HVMLite too.

But to make that work you have to emulate EFI firmware in the hypervisor. Is that work you are signing up for?

Quoted 13 lines · Konrad Rzeszutek Wilk

Quoted 12 lines · Luis R. Rodriguez

Quoted 8 lines · Konrad Rzeszutek Wilk

Quoted 5 lines · Luis R. Rodriguez

VGA code will be dead code for HVMlite for sure as the design doc says it will not run VGA, the ACPI flag will be set but the check for that is not yet on Linux. That means the VGA Linux code will be there but we have no way to ensure it will not run nor that anything will muck with it.

<shrugs> The worst it will do is try to read non-existent registers.

Really ?

Quoted 5 lines

Quoted 2 lines

Is that your position on all other possible dead code that may have been possible on old Xen PV guests as well ?

This is not just with Xen - it with other device drivers that are being invoked on baremetal and are not present in hardware anymore.

Indeed, however virtualization makes this issue much more prominent.

Quoted 14 lines

Quoted 3 lines

As I hinted, after thinking about this for a while I realized that dead code is likely present on bare metal as well even without virtualization, specially if

Yes!

Quoted 8 lines

you build large single kernels to support a wide array of features which only late at run time can be determined. Virtualization and the pvops design just makes this issue much more prominent. If there are other areas of code exposed that actually may run, but we are not sure may run, I figured some other folks with a bit more security conscience minds might even simply take the position it may be a security risk to leave that code exposed. So to take a position that 'the worst it will do is try to read non-existent registers' -- seems rather shortsighted here.

Quoted 1 line

Security conscious people trim their CONFIG.

Not all Linux distributions want to do this, the more binaries the higher the cost to test / vet.

Quoted 20 lines

Quoted 9 lines

Anyway for more details on thoughts on this refer to the this wiki:

http://kernelnewbies.org/KernelProjects/kernel-sandboxing

Since this is now getting off topic please send me your feedback on another thread for the non-virtualization aspects of this if that interests you. My point here was rather to highlight the importance of clear semantics due to virtualization in light of possible dead code.

Thank you.

Quoted 9 lines

Quoted 3 lines

The VGA code should be able to handle failures like that and not initialize itself when the hardware is dead (or non-existent).

That's right, its through ACPI_FADT_NO_VGA and since its part of the HVMLite design doc we want HVMlite design to address ACPI_FADT_NO_VGA properly. I've paved the way for this to be done cleanly and easily now, but that code should be in place before HVMLite code gets merged.

Quoted 4 lines

Quoted 1 line

Does domU for old Xen PV also set ACPI_FADT_NO_VGA as well ? Should it ?

It does not. Not sure - it seems to have worked fine for the last ten years?

Maybe HVMLite will need it enabled then too, just for bug parity.

Quoted 11 lines

Quoted 11 lines

Quoted 10 lines

Quoted 7 lines

To be clear -- dead code concerns still exist even without virtualization solutions, its just that with virtualization this stuff comes up more and there has been no proactive measures to address this. The question of semantics here is to see to what extent we need earlier boot code annotations to ensure we address semantics proactively.

I think what you mean by dead code is another word for hardware test coverage?

Quoted 8 lines

Quoted 5 lines

No, no, its very different given that with virtualization the scope of possible dead code is significant and at run time you are certain a huge portion of code should *never ever* run. So for instance we know once we boot bare metal none of the Xen stuff should ever run, likewise on Xen dom0 we know none of the KVM / bare-metal only stuff should never run, when on Xen domU, none of the Xen

What is this 'bare metal only stuff' you speak of? On Xen dom0 most of the baremetal code is running.

A lot, not all. In the past folks added stubs (used to be paravirt_enabled() checks) to some code, but we are simply not sure of other possible conflicts. This is an known unknown if you will.

Quoted 3 lines

In fact that is how the device drivers work. Or are you talking about low level baremetal code? If so, then PVH/HVMLite does that - it skips pvops so that it can run this 'low-level baremetal code'

Are you telling me that HVMLite has no dead code issues ?

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

There is that and as others have pointed out how certain guests types are assumed to not have certain peripherals, and we have no idea to ensure certain old legacy code may not ever run or be accessed by drivers.

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

Ok, but that is not at code setup. That is later - when device drivers are initialized. This no different than booting on some hardware with missing functionality. ACPI, PCI and PnP PnP are set there to help OSes discover this.

To a certain extent this is true, but there may things which are missing still.

Like?

That's the thing, I had a list of thing to look out for and then things I ran across over code inspection. We need more work to be sure we're really well covered.

Are you *sure* we have no dead code concerns with HVMLite ? If there are dead code concerns are you sure there might not be differences between KVM and HVMLite ? Should cpuid be used to address differences ? Will that enable to distinguish between hybrid versions of HVMLite ? Are we sure ?

Quoted 4 lines

Quoted 2 lines

We really have no idea what the full list of those things are.

Ok, it sounds like you have some homework.

We all do.

Quoted 11 lines

Quoted 10 lines

It may be that things may have been running for ages without notice of an issue or that only under certain situations will certain issues or bugs trigger a failure. For instance, just yesterday I was Cc'd on a brand-spanking new legacy conflict [0], caused by upstream commit 8c058b0b9c34d8c ("x86/irq: Probe for PIC presence before allocating descs for legacy IRQs") merged on v4.4 where some new code used nr_legacy_irqs() -- one proposed solution seems to be that for Xen code NR_IRQS_LEGACY should be used instead is as it lacks PCI [1] and another was to peg the legacy requirements as a quirk on the new x86 platform legacy quirk stuff [2]. Are other uses of nr_legacy_irqs() correct ? Are we sure ?

Quoted 3 lines

And how is this example related to 'early bootup' path?

It is not.

For early boot code -- it is not. HVMLite is not merged, and PHV was never completed.. so how are you sure we won't have any issues there ?

Quoted 3 lines

It is in fact related to PV codepaths - which PVH/HVMLite and HVM guests do not exercise.

Agreed.

Quoted 15 lines

Quoted 15 lines

[0] http://lkml.kernel.org/r/[email protected]
[1] https://lkml.org/lkml/2016/4/14/532
[2] http://lkml.kernel.org/r/[email protected]

Quoted 10 lines

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

How we address semantics then is *very* important to me.

Which semantics? How the CPU is going to be at startup_X ? Or how the CPU is going to be when EFI firmware invokes the EFI stub? Or when GRUB2 loads Linux?

What hypervisor kicked me and what guest type I am.

cpuid software flags have that - and that semantics has been there for eons.

Quoted 3 lines

Quoted 1 line

We cannot use cpuid early in asm code, I'm looking for something we

?! Why!?

What existing code uses it? If there is nothing you are still certain it should work ? Would that work for old PV guest as well BTW ?

Quoted 7 lines

Quoted 5 lines

can even use on asm early in boot code, on x86 the best option we have is the boot_params, but I've even have had issues with that early in code, as I can only access it after load_idt() where I described my effort to unify Xen PV and x86_64 init paths [3].

Well, Xen PV skips x86_64_start_kernel..

Yes, and in doing so often times people skip adding Xen PV specific code, as was the case with Kasan.

Quoted 37 lines

Quoted 36 lines

[3] http://lkml.kernel.org/r/CAB=NE6VTCRCazcNpCdJ7pN1eD3=x_fcGOdH37MzVpxkKEN5esw@mail.gmail.com

Quoted 20 lines

Quoted 20 lines

Let me elaborate more below.

Quoted 18 lines

That (those bootloaders) is clearly defined. The URL I provided mentions the HVMLite one. The Documentation/x86/boot.c mentions what the semantics are to expected when providing an bootstrap (which is what HVMLitel stub code in Linux would write against - and what EFI stub code had been written against too).

Quoted 12 lines

Quoted 10 lines

Quoted 7 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

It can still be OS agnostic and pass on type and custom data pointer.

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

Sure. It has that (it MUST otherwise how else would you pass data).
It is documented as well http://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,xen.h.html#incontents_startofday
(see " Start of day structure passed to PVH guests in %ebx.")

The design doc begs for a custom OS entry point though.

That is what the ELF Note has.

Right, but I'm saying that its rather silly to be adding entry points if all we want the code to do is copy the boot params for us. The design doc requires a new entry, and likewise you'd need yet-another-entry if HVMLite is thrown out the window and come back 5 years later after new hardware solutions are in place and need to redesign HVMLite. Kind of

Quoted 1 line

Why would you need to redesign HVMLite based on hardware solutions?

That's what happened to Xen PV, right ? Are we sure 5 years from now we won't have any new hardware virtualization features that will just obsolete HVMLite?

Quoted 7 lines

The entrace point and the CPU state are pretty well known - it is akin to what GRUB2 bootloader path is (protected mode).

Quoted 4 lines

where we are with PVH today. Likewise if other paravirtualization developers want to support Linux and want to copy your strategy they'd add yet-another-entry-point as well.

Quoted 4 lines

Quoted 1 line

This is dumb.

You saying the EFI entry point is dumb? That instead the EFI firmware should understand Linux bootparams and booted that?

EFI is a standard. Xen is not. And since we are not talking about legacy

hardware in the future, EFI seems like a sensible option to consider for an entry point. Specially given that it may mean that we can ultimately also help unify more entry points on Linux in general. I'd prefer to consider using

EFI configuration tables instead of extending the x86 boot protocol.

Quoted 11 lines

Quoted 11 lines

Quoted 10 lines

Quoted 4 lines

If we had a single 'type' and 'custom data' passed to the kernel that should suffice for the default Linux entry point to just pivot off of that and do what it needs without more entry points. Once.

And what about ramdisk? What about multiple ramdisks? What about command line? All of that is what bootparams tries to unify on Linux. But 'bootparams' is unique to Linux, it does not exist on FreeBSD. Hence some stub code to transplant OS-agnostic simple data to OS-specific is neccessary.

Quoted 6 lines

Quoted 4 lines

If we had a Xen ABI option where *all* that I'm asking is you pass first:

a) hypervisor type

Why can't you use cpuid.

I'll evaluate that.

Quoted 3 lines

Quoted 1 line

b) custom data pointer

What is this custom data pointer you speak of?

For Xen this is the en_start_info, the structure that Xen stuffs in a copy of its version of what we need to fill the boot_params.

Quoted 7 lines

Quoted 3 lines

We'd be able to avoid adding *any* entry point and just address the requirements as I noted with pre / post stubs for the type.

But you need some entry point to call into Linux. Are you suggesting to use the existing ones? No, the existing one wouldn't understand this.

If we used the boot_parms, yes it would be possible.

Quoted 9 lines

Quoted 3 lines

This would require an x86 boot protocol bump, but all the issues creeping up randomly I think that's worth putting on the table now.

Aaaah, so you are saying expand the bootparams. In other words make Xen ABI call into Linux using the bootparams structure, similar to how GRUB2 does it.

How is that OS agnostic?

That's an issue, I understand. EFI is OS agnostic though.

Quoted 4 lines

Quoted 2 lines

And maybe we don't want it to be hypervisor specific, perhaps there are other *needs* for custom pre-post startup_32()/startup_64() stubs.

Multiboot?

Can you elaborate?

Quoted 5 lines

Quoted 3 lines

To avoid extending boot_params further I figured perhaps we can look at EFI as another option instead. If we are going to drop all legacy

But EFI support is _huge_.

I get the sense now. Perhaps we should explore to what extent now really at the Hackathon.

Quoted 9 lines

Quoted 6 lines

PV support from the kernel (not the hypervisor) and require hardware virtualization 5 years from now on the Linux kernel, it doesn't seem to me far fetched to at the very least consider using an EFI entry instead, specially since all it does is set boot params and we can make re-use this for HVMLite too.

But to make that work you have to emulate EFI firmware in the hypervisor. Is that work you are signing up for?

I'll do what is needed, as I have done before. If EFI is on the long term roadmap for ARM perhaps there are a few birds to knock with one stone here. If there is also interest to support other OSes through EFI standard means this also should help make that easier.

Luis

Quoted 13 lines · Konrad Rzeszutek Wilk

Quoted 12 lines · Luis R. Rodriguez

Quoted 8 lines · Konrad Rzeszutek Wilk

Quoted 5 lines · Luis R. Rodriguez

VGA code will be dead code for HVMlite for sure as the design doc says it will not run VGA, the ACPI flag will be set but the check for that is not yet on Linux. That means the VGA Linux code will be there but we have no way to ensure it will not run nor that anything will muck with it.

<shrugs> The worst it will do is try to read non-existent registers.

Really ?

Quoted 5 lines

Quoted 2 lines

Is that your position on all other possible dead code that may have been possible on old Xen PV guests as well ?

This is not just with Xen - it with other device drivers that are being invoked on baremetal and are not present in hardware anymore.

Indeed, however virtualization makes this issue much more prominent.

Quoted 14 lines

Quoted 3 lines

As I hinted, after thinking about this for a while I realized that dead code is likely present on bare metal as well even without virtualization, specially if

Yes!

Quoted 8 lines

you build large single kernels to support a wide array of features which only late at run time can be determined. Virtualization and the pvops design just makes this issue much more prominent. If there are other areas of code exposed that actually may run, but we are not sure may run, I figured some other folks with a bit more security conscience minds might even simply take the position it may be a security risk to leave that code exposed. So to take a position that 'the worst it will do is try to read non-existent registers' -- seems rather shortsighted here.

Quoted 1 line

Security conscious people trim their CONFIG.

Not all Linux distributions want to do this, the more binaries the higher the cost to test / vet.

Quoted 20 lines

Quoted 9 lines

Anyway for more details on thoughts on this refer to the this wiki:

http://kernelnewbies.org/KernelProjects/kernel-sandboxing

Since this is now getting off topic please send me your feedback on another thread for the non-virtualization aspects of this if that interests you. My point here was rather to highlight the importance of clear semantics due to virtualization in light of possible dead code.

Thank you.

Quoted 9 lines

Quoted 3 lines

The VGA code should be able to handle failures like that and not initialize itself when the hardware is dead (or non-existent).

That's right, its through ACPI_FADT_NO_VGA and since its part of the HVMLite design doc we want HVMlite design to address ACPI_FADT_NO_VGA properly. I've paved the way for this to be done cleanly and easily now, but that code should be in place before HVMLite code gets merged.

Quoted 4 lines

Quoted 1 line

Does domU for old Xen PV also set ACPI_FADT_NO_VGA as well ? Should it ?

It does not. Not sure - it seems to have worked fine for the last ten years?

Maybe HVMLite will need it enabled then too, just for bug parity.

Quoted 11 lines

Quoted 11 lines

Quoted 10 lines

Quoted 7 lines

To be clear -- dead code concerns still exist even without virtualization solutions, its just that with virtualization this stuff comes up more and there has been no proactive measures to address this. The question of semantics here is to see to what extent we need earlier boot code annotations to ensure we address semantics proactively.

I think what you mean by dead code is another word for hardware test coverage?

Quoted 8 lines

Quoted 5 lines

No, no, its very different given that with virtualization the scope of possible dead code is significant and at run time you are certain a huge portion of code should *never ever* run. So for instance we know once we boot bare metal none of the Xen stuff should ever run, likewise on Xen dom0 we know none of the KVM / bare-metal only stuff should never run, when on Xen domU, none of the Xen

What is this 'bare metal only stuff' you speak of? On Xen dom0 most of the baremetal code is running.

A lot, not all. In the past folks added stubs (used to be paravirt_enabled() checks) to some code, but we are simply not sure of other possible conflicts. This is an known unknown if you will.

Quoted 3 lines

In fact that is how the device drivers work. Or are you talking about low level baremetal code? If so, then PVH/HVMLite does that - it skips pvops so that it can run this 'low-level baremetal code'

Are you telling me that HVMLite has no dead code issues ?

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

There is that and as others have pointed out how certain guests types are assumed to not have certain peripherals, and we have no idea to ensure certain old legacy code may not ever run or be accessed by drivers.

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

Ok, but that is not at code setup. That is later - when device drivers are initialized. This no different than booting on some hardware with missing functionality. ACPI, PCI and PnP PnP are set there to help OSes discover this.

To a certain extent this is true, but there may things which are missing still.

Like?

That's the thing, I had a list of thing to look out for and then things I ran across over code inspection. We need more work to be sure we're really well covered.

Are you *sure* we have no dead code concerns with HVMLite ? If there are dead code concerns are you sure there might not be differences between KVM and HVMLite ? Should cpuid be used to address differences ? Will that enable to distinguish between hybrid versions of HVMLite ? Are we sure ?

Quoted 4 lines

Quoted 2 lines

We really have no idea what the full list of those things are.

Ok, it sounds like you have some homework.

We all do.

Quoted 11 lines

Quoted 10 lines

It may be that things may have been running for ages without notice of an issue or that only under certain situations will certain issues or bugs trigger a failure. For instance, just yesterday I was Cc'd on a brand-spanking new legacy conflict [0], caused by upstream commit 8c058b0b9c34d8c ("x86/irq: Probe for PIC presence before allocating descs for legacy IRQs") merged on v4.4 where some new code used nr_legacy_irqs() -- one proposed solution seems to be that for Xen code NR_IRQS_LEGACY should be used instead is as it lacks PCI [1] and another was to peg the legacy requirements as a quirk on the new x86 platform legacy quirk stuff [2]. Are other uses of nr_legacy_irqs() correct ? Are we sure ?

Quoted 3 lines

And how is this example related to 'early bootup' path?

It is not.

For early boot code -- it is not. HVMLite is not merged, and PHV was never completed.. so how are you sure we won't have any issues there ?

Quoted 3 lines

It is in fact related to PV codepaths - which PVH/HVMLite and HVM guests do not exercise.

Agreed.

Quoted 15 lines

Quoted 15 lines

[0] http://lkml.kernel.org/r/[email protected]
[1] https://lkml.org/lkml/2016/4/14/532
[2] http://lkml.kernel.org/r/[email protected]

Quoted 10 lines

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

How we address semantics then is *very* important to me.

Which semantics? How the CPU is going to be at startup_X ? Or how the CPU is going to be when EFI firmware invokes the EFI stub? Or when GRUB2 loads Linux?

What hypervisor kicked me and what guest type I am.

cpuid software flags have that - and that semantics has been there for eons.

Quoted 3 lines

Quoted 1 line

We cannot use cpuid early in asm code, I'm looking for something we

?! Why!?

What existing code uses it? If there is nothing you are still certain it should work ? Would that work for old PV guest as well BTW ?

Quoted 7 lines

Quoted 5 lines

can even use on asm early in boot code, on x86 the best option we have is the boot_params, but I've even have had issues with that early in code, as I can only access it after load_idt() where I described my effort to unify Xen PV and x86_64 init paths [3].

Well, Xen PV skips x86_64_start_kernel..

Yes, and in doing so often times people skip adding Xen PV specific code, as was the case with Kasan.

Quoted 37 lines

Quoted 36 lines

[3] http://lkml.kernel.org/r/CAB=NE6VTCRCazcNpCdJ7pN1eD3=x_fcGOdH37MzVpxkKEN5esw@mail.gmail.com

Quoted 20 lines

Quoted 20 lines

Let me elaborate more below.

Quoted 18 lines

That (those bootloaders) is clearly defined. The URL I provided mentions the HVMLite one. The Documentation/x86/boot.c mentions what the semantics are to expected when providing an bootstrap (which is what HVMLitel stub code in Linux would write against - and what EFI stub code had been written against too).

Quoted 12 lines

Quoted 10 lines

Quoted 7 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

It can still be OS agnostic and pass on type and custom data pointer.

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

Sure. It has that (it MUST otherwise how else would you pass data).
It is documented as well http://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,xen.h.html#incontents_startofday
(see " Start of day structure passed to PVH guests in %ebx.")

The design doc begs for a custom OS entry point though.

That is what the ELF Note has.

Right, but I'm saying that its rather silly to be adding entry points if all we want the code to do is copy the boot params for us. The design doc requires a new entry, and likewise you'd need yet-another-entry if HVMLite is thrown out the window and come back 5 years later after new hardware solutions are in place and need to redesign HVMLite. Kind of

Quoted 1 line

Why would you need to redesign HVMLite based on hardware solutions?

That's what happened to Xen PV, right ? Are we sure 5 years from now we won't have any new hardware virtualization features that will just obsolete HVMLite?

Quoted 7 lines

The entrace point and the CPU state are pretty well known - it is akin to what GRUB2 bootloader path is (protected mode).

Quoted 4 lines

where we are with PVH today. Likewise if other paravirtualization developers want to support Linux and want to copy your strategy they'd add yet-another-entry-point as well.

Quoted 4 lines

Quoted 1 line

This is dumb.

You saying the EFI entry point is dumb? That instead the EFI firmware should understand Linux bootparams and booted that?

EFI is a standard. Xen is not. And since we are not talking about legacy

hardware in the future, EFI seems like a sensible option to consider for an entry point. Specially given that it may mean that we can ultimately also help unify more entry points on Linux in general. I'd prefer to consider using

EFI configuration tables instead of extending the x86 boot protocol.

Quoted 11 lines

Quoted 11 lines

Quoted 10 lines

Quoted 4 lines

If we had a single 'type' and 'custom data' passed to the kernel that should suffice for the default Linux entry point to just pivot off of that and do what it needs without more entry points. Once.

And what about ramdisk? What about multiple ramdisks? What about command line? All of that is what bootparams tries to unify on Linux. But 'bootparams' is unique to Linux, it does not exist on FreeBSD. Hence some stub code to transplant OS-agnostic simple data to OS-specific is neccessary.

Quoted 6 lines

Quoted 4 lines

If we had a Xen ABI option where *all* that I'm asking is you pass first:

a) hypervisor type

Why can't you use cpuid.

I'll evaluate that.

Quoted 3 lines

Quoted 1 line

b) custom data pointer

What is this custom data pointer you speak of?

For Xen this is the en_start_info, the structure that Xen stuffs in a copy of its version of what we need to fill the boot_params.

Quoted 7 lines

Quoted 3 lines

We'd be able to avoid adding *any* entry point and just address the requirements as I noted with pre / post stubs for the type.

But you need some entry point to call into Linux. Are you suggesting to use the existing ones? No, the existing one wouldn't understand this.

If we used the boot_parms, yes it would be possible.

Quoted 9 lines

Quoted 3 lines

This would require an x86 boot protocol bump, but all the issues creeping up randomly I think that's worth putting on the table now.

Aaaah, so you are saying expand the bootparams. In other words make Xen ABI call into Linux using the bootparams structure, similar to how GRUB2 does it.

How is that OS agnostic?

That's an issue, I understand. EFI is OS agnostic though.

Quoted 4 lines

Quoted 2 lines

And maybe we don't want it to be hypervisor specific, perhaps there are other *needs* for custom pre-post startup_32()/startup_64() stubs.

Multiboot?

Can you elaborate?

Quoted 5 lines

Quoted 3 lines

To avoid extending boot_params further I figured perhaps we can look at EFI as another option instead. If we are going to drop all legacy

But EFI support is _huge_.

I get the sense now. Perhaps we should explore to what extent now really at the Hackathon.

Quoted 9 lines

Quoted 6 lines

PV support from the kernel (not the hypervisor) and require hardware virtualization 5 years from now on the Linux kernel, it doesn't seem to me far fetched to at the very least consider using an EFI entry instead, specially since all it does is set boot params and we can make re-use this for HVMLite too.

But to make that work you have to emulate EFI firmware in the hypervisor. Is that work you are signing up for?

I'll do what is needed, as I have done before. If EFI is on the long term roadmap for ARM perhaps there are a few birds to knock with one stone here. If there is also interest to support other OSes through EFI standard means this also should help make that easier.

Luis

Quoted 1 line · Luis R. Rodriguez

Are you telling me that HVMLite has no dead code issues ?

You said earlier that baremetal has dead code issue. Then by extensions _any_ execution path has dead code issues.

Quoted 5 lines

Are you *sure* we have no dead code concerns with HVMLite ? If there are dead code concerns are you sure there might not be differences between KVM and HVMLite ? Should cpuid be used to address differences ? Will that enable to distinguish between hybrid versions of HVMLite ? Are we sure ?

HVMLite CPU semantics will be the same as what a baremetal CPU semantics are.

Platform wise it will be different - as in, instead of say having a speaker (to emulated it) or RTC clock (again, another thing to emulate), or say IDE controller (again, another thing to emulate), or Realtek network card (again, another thing to emulate) - it has none of those.

[Keep in mind 'another thing to emulate', means 'another @$@() thing in QEMU that could be a security bug']

So it differs from an consumer x86 platform in that it has none of the 'legacy' stuff. And it requires PV drivers to function. And since it requires PV drivers to function only OSes that have those can use this mode.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

We cannot use cpuid early in asm code, I'm looking for something we

?! Why!?

What existing code uses it? If there is nothing you are still certain it should work ? Would that work for old PV guest as well BTW ?

Yeah. For HVM/HVMLite it traps to the hypervisor.

For old PV guests it is unwise to use it as it goes straight to the hardware (as PV guests run in ring3 - they are considered 'userspace' and the Intel nor AMD do not trap on 'cpuid' in ring3 -unless you run in an VMX container).

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

Right, but I'm saying that its rather silly to be adding entry points if all we want the code to do is copy the boot params for us. The design doc requires a new entry, and likewise you'd need yet-another-entry if HVMLite is thrown out the window and come back 5 years later after new hardware solutions are in place and need to redesign HVMLite. Kind of

Why would you need to redesign HVMLite based on hardware solutions?

Quoted 2 lines

That's what happened to Xen PV, right ? Are we sure 5 years from now we won't have any new hardware virtualization features that will just obsolete HVMLite?

There were no hardware virtualization when Xen PV came about.

If there is hardware virtualization that obsoletes HVMLite that means it would also obsolete KVM and HVM mode

Quoted 11 lines

Quoted 10 lines

The entrace point and the CPU state are pretty well known - it is akin to what GRUB2 bootloader path is (protected mode).

Quoted 5 lines

where we are with PVH today. Likewise if other paravirtualization developers want to support Linux and want to copy your strategy they'd add yet-another-entry-point as well.

This is dumb.

You saying the EFI entry point is dumb? That instead the EFI firmware should understand Linux bootparams and booted that?

Quoted 1 line

EFI is a standard. Xen is not. And since we are not talking about legacy

And is a standard something that has to come out of a committee?

If so, then Linux bootparams is not a standard. Nor is LILO bootup path.

Quoted 3 lines

hardware in the future, EFI seems like a sensible option to consider for an entry point. Specially given that it may mean that we can ultimately also help unify more entry points on Linux in general. I'd prefer to consider using

<chokes>
I can just see that. On non-EFI hardware GRUB2/SYSLINUX would use the EFI entry point and create an fake firmware.

Quoted 1 line

EFI configuration tables instead of extending the x86 boot protocol.

What is that? Are you talking about EFI runtime services? Take a look at the EFI spec and see what you have to implement to emulate this.

Quoted 24 lines

Quoted 11 lines

Quoted 11 lines

Quoted 10 lines

Quoted 4 lines

If we had a single 'type' and 'custom data' passed to the kernel that should suffice for the default Linux entry point to just pivot off of that and do what it needs without more entry points. Once.

And what about ramdisk? What about multiple ramdisks? What about command line? All of that is what bootparams tries to unify on Linux. But 'bootparams' is unique to Linux, it does not exist on FreeBSD. Hence some stub code to transplant OS-agnostic simple data to OS-specific is neccessary.

Quoted 6 lines

Quoted 4 lines

If we had a Xen ABI option where *all* that I'm asking is you pass first:

a) hypervisor type

Why can't you use cpuid.

I'll evaluate that.

Quoted 3 lines

Quoted 1 line

b) custom data pointer

What is this custom data pointer you speak of?

Quoted 2 lines

For Xen this is the xen_start_info, the structure that Xen stuffs in a copy of its version of what we need to fill the boot_params.

Ok, but that is what we do in some way provide.

I am lost here. You seem to saying you want something that is already there?

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

We'd be able to avoid adding *any* entry point and just address the requirements as I noted with pre / post stubs for the type.

But you need some entry point to call into Linux. Are you suggesting to use the existing ones? No, the existing one wouldn't understand this.

If we used the boot_parms, yes it would be possible.

...OS agnostic... they are not.

Quoted 11 lines

Quoted 8 lines

Quoted 2 lines

This would require an x86 boot protocol bump, but all the issues creeping up randomly I think that's worth putting on the table now.

Aaaah, so you are saying expand the bootparams. In other words make Xen ABI call into Linux using the bootparams structure, similar to how GRUB2 does it.

How is that OS agnostic?

That's an issue, I understand. EFI is OS agnostic though.

Quoted 6 lines

Quoted 4 lines

Quoted 2 lines

And maybe we don't want it to be hypervisor specific, perhaps there are other *needs* for custom pre-post startup_32()/startup_64() stubs.

Multiboot?

Can you elaborate?

Google Multiboot specification.

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

To avoid extending boot_params further I figured perhaps we can look at EFI as another option instead. If we are going to drop all legacy

But EFI support is _huge_.

I get the sense now. Perhaps we should explore to what extent now really at the Hackathon.

Print out the EFI spec and carry it on the plane. The plane will tilt to one side when trying to take off.

Quoted 1 line · Luis R. Rodriguez

Are you telling me that HVMLite has no dead code issues ?

You said earlier that baremetal has dead code issue. Then by extensions _any_ execution path has dead code issues.

Quoted 5 lines

Are you *sure* we have no dead code concerns with HVMLite ? If there are dead code concerns are you sure there might not be differences between KVM and HVMLite ? Should cpuid be used to address differences ? Will that enable to distinguish between hybrid versions of HVMLite ? Are we sure ?

HVMLite CPU semantics will be the same as what a baremetal CPU semantics are.

Platform wise it will be different - as in, instead of say having a speaker (to emulated it) or RTC clock (again, another thing to emulate), or say IDE controller (again, another thing to emulate), or Realtek network card (again, another thing to emulate) - it has none of those.

[Keep in mind 'another thing to emulate', means 'another @$@() thing in QEMU that could be a security bug']

So it differs from an consumer x86 platform in that it has none of the 'legacy' stuff. And it requires PV drivers to function. And since it requires PV drivers to function only OSes that have those can use this mode.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

We cannot use cpuid early in asm code, I'm looking for something we

?! Why!?

What existing code uses it? If there is nothing you are still certain it should work ? Would that work for old PV guest as well BTW ?

Yeah. For HVM/HVMLite it traps to the hypervisor.

For old PV guests it is unwise to use it as it goes straight to the hardware (as PV guests run in ring3 - they are considered 'userspace' and the Intel nor AMD do not trap on 'cpuid' in ring3 -unless you run in an VMX container).

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

Right, but I'm saying that its rather silly to be adding entry points if all we want the code to do is copy the boot params for us. The design doc requires a new entry, and likewise you'd need yet-another-entry if HVMLite is thrown out the window and come back 5 years later after new hardware solutions are in place and need to redesign HVMLite. Kind of

Why would you need to redesign HVMLite based on hardware solutions?

Quoted 2 lines

That's what happened to Xen PV, right ? Are we sure 5 years from now we won't have any new hardware virtualization features that will just obsolete HVMLite?

There were no hardware virtualization when Xen PV came about.

If there is hardware virtualization that obsoletes HVMLite that means it would also obsolete KVM and HVM mode

Quoted 11 lines

Quoted 10 lines

The entrace point and the CPU state are pretty well known - it is akin to what GRUB2 bootloader path is (protected mode).

Quoted 5 lines

where we are with PVH today. Likewise if other paravirtualization developers want to support Linux and want to copy your strategy they'd add yet-another-entry-point as well.

This is dumb.

You saying the EFI entry point is dumb? That instead the EFI firmware should understand Linux bootparams and booted that?

Quoted 1 line

EFI is a standard. Xen is not. And since we are not talking about legacy

And is a standard something that has to come out of a committee?

If so, then Linux bootparams is not a standard. Nor is LILO bootup path.

Quoted 3 lines

hardware in the future, EFI seems like a sensible option to consider for an entry point. Specially given that it may mean that we can ultimately also help unify more entry points on Linux in general. I'd prefer to consider using

<chokes>
I can just see that. On non-EFI hardware GRUB2/SYSLINUX would use the EFI entry point and create an fake firmware.

Quoted 1 line

EFI configuration tables instead of extending the x86 boot protocol.

What is that? Are you talking about EFI runtime services? Take a look at the EFI spec and see what you have to implement to emulate this.

Quoted 24 lines

Quoted 11 lines

Quoted 11 lines

Quoted 10 lines

Quoted 4 lines

If we had a single 'type' and 'custom data' passed to the kernel that should suffice for the default Linux entry point to just pivot off of that and do what it needs without more entry points. Once.

And what about ramdisk? What about multiple ramdisks? What about command line? All of that is what bootparams tries to unify on Linux. But 'bootparams' is unique to Linux, it does not exist on FreeBSD. Hence some stub code to transplant OS-agnostic simple data to OS-specific is neccessary.

Quoted 6 lines

Quoted 4 lines

If we had a Xen ABI option where *all* that I'm asking is you pass first:

a) hypervisor type

Why can't you use cpuid.

I'll evaluate that.

Quoted 3 lines

Quoted 1 line

b) custom data pointer

What is this custom data pointer you speak of?

Quoted 2 lines

For Xen this is the xen_start_info, the structure that Xen stuffs in a copy of its version of what we need to fill the boot_params.

Ok, but that is what we do in some way provide.

I am lost here. You seem to saying you want something that is already there?

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

We'd be able to avoid adding *any* entry point and just address the requirements as I noted with pre / post stubs for the type.

But you need some entry point to call into Linux. Are you suggesting to use the existing ones? No, the existing one wouldn't understand this.

If we used the boot_parms, yes it would be possible.

...OS agnostic... they are not.

Quoted 11 lines

Quoted 8 lines

Quoted 2 lines

This would require an x86 boot protocol bump, but all the issues creeping up randomly I think that's worth putting on the table now.

Aaaah, so you are saying expand the bootparams. In other words make Xen ABI call into Linux using the bootparams structure, similar to how GRUB2 does it.

How is that OS agnostic?

That's an issue, I understand. EFI is OS agnostic though.

Quoted 6 lines

Quoted 4 lines

Quoted 2 lines

And maybe we don't want it to be hypervisor specific, perhaps there are other *needs* for custom pre-post startup_32()/startup_64() stubs.

Multiboot?

Can you elaborate?

Google Multiboot specification.

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

To avoid extending boot_params further I figured perhaps we can look at EFI as another option instead. If we are going to drop all legacy

But EFI support is _huge_.

I get the sense now. Perhaps we should explore to what extent now really at the Hackathon.

Print out the EFI spec and carry it on the plane. The plane will tilt to one side when trying to take off.

Quoted 9 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Luis R. Rodriguez

Quoted 1 line · Konrad Rzeszutek Wilk

On Thu, Apr 14, 2016 at 12:23:17AM +0200, Luis R. Rodriguez wrote:

PV support from the kernel (not the hypervisor) and require hardware virtualization 5 years from now on the Linux kernel, it doesn't seem to me far fetched to at the very least consider using an EFI entry instead, specially since all it does is set boot params and we can make re-use this for HVMLite too.

Quoted 2 lines

But to make that work you have to emulate EFI firmware in the hypervisor. Is that work you are signing up for?

I'll do what is needed, as I have done before. If EFI is on the long term roadmap for ARM perhaps there are a few birds to knock with one stone here. If there is also interest to support other OSes through EFI standard means this also should help make that easier.

Quoted 4 lines · Luis R. Rodriguez

Quoted 2 lines · Konrad Rzeszutek Wilk

But to make that work you have to emulate EFI firmware in the hypervisor. Is that work you are signing up for?

Quoted 4 lines

I'll do what is needed, as I have done before. If EFI is on the long term roadmap for ARM perhaps there are a few birds to knock with one stone here. If there is also interest to support other OSes through EFI standard means this also should help make that easier.

We already have a working solution for EFI on ARM which does not require to emulate the firmware in the hypervisor.

On ARM, the EFI stub is communicating with the kernel using device-tree [1]. Once the EFI stub has ended, the native path (i.e non-UEFI) will be executed normally and it won't be possible to use BootServices anymore.

For the guest, we provide a full support of EFI using OVMF.

For DOM0, Xen
will craft the UEFI system table and the UEFI memory map. The locations of those tables will be passed to DOM0 using a tiny device-tree [1] and the kernel will boot using the native path. The runtime services for DOM0 will be provided via hypercall.

The DOM0 approach has been discussed for a long time (see [3]) and I believe this is better than emulating UEFI firmware in Xen. We want to keep Xen on ARM tiny. Adding any sort of emulation will increase the attack surface and require more maintenance from our side.

Quoted 5 lines · Julien Grall

Quoted 4 lines · Luis R. Rodriguez

Quoted 2 lines · Konrad Rzeszutek Wilk

But to make that work you have to emulate EFI firmware in the hypervisor. Is that work you are signing up for?

Quoted 7 lines

Quoted 4 lines

I'll do what is needed, as I have done before. If EFI is on the long term roadmap for ARM perhaps there are a few birds to knock with one stone here. If there is also interest to support other OSes through EFI standard means this also should help make that easier.

We already have a working solution for EFI on ARM which does not require to emulate the firmware in the hypervisor.

I get that.

Quoted 5 lines

On ARM, the EFI stub is communicating with the kernel using device-tree [1]. Once the EFI stub has ended, the native path (i.e non-UEFI) will be executed normally and it won't be possible to use BootServices anymore.

For the guest, we provide a full support of EFI using OVMF.

I get that as well, is this the long term solution ?

That still requires OVMF, will relying on OVMF always be what is used on Xen ARM ?

Was it too much of a burden to require OVMF?

Is the upstream OVMF code pulled by Xen at build time on ARM, or just wget a binary ?

Quoted 4 lines

For DOM0, Xen will craft the UEFI system table and the UEFI memory map. The locations of those tables will be passed to DOM0 using a tiny device-tree [1] and the kernel will boot using the native path. The runtime services for DOM0 will be provided via hypercall.

Thanks this helps!

Quoted 4 lines

The DOM0 approach has been discussed for a long time (see [3]) and I believe this is better than emulating UEFI firmware in Xen. We want to keep Xen on ARM tiny. Adding any sort of emulation will increase the attack surface and require more maintenance from our side.

OK thanks, would re-using OVMF (note, DT perhaps may not be ideal for x86 for the rest though) be a reasonable solution on x86 as an option then?

Quoted 5 lines · Julien Grall

Quoted 4 lines · Luis R. Rodriguez

Quoted 2 lines · Konrad Rzeszutek Wilk

But to make that work you have to emulate EFI firmware in the hypervisor. Is that work you are signing up for?

Quoted 7 lines

Quoted 4 lines

I'll do what is needed, as I have done before. If EFI is on the long term roadmap for ARM perhaps there are a few birds to knock with one stone here. If there is also interest to support other OSes through EFI standard means this also should help make that easier.

We already have a working solution for EFI on ARM which does not require to emulate the firmware in the hypervisor.

I get that.

Quoted 5 lines

On ARM, the EFI stub is communicating with the kernel using device-tree [1]. Once the EFI stub has ended, the native path (i.e non-UEFI) will be executed normally and it won't be possible to use BootServices anymore.

For the guest, we provide a full support of EFI using OVMF.

I get that as well, is this the long term solution ?

That still requires OVMF, will relying on OVMF always be what is used on Xen ARM ?

Was it too much of a burden to require OVMF?

Is the upstream OVMF code pulled by Xen at build time on ARM, or just wget a binary ?

Quoted 4 lines

For DOM0, Xen will craft the UEFI system table and the UEFI memory map. The locations of those tables will be passed to DOM0 using a tiny device-tree [1] and the kernel will boot using the native path. The runtime services for DOM0 will be provided via hypercall.

Thanks this helps!

Quoted 4 lines

The DOM0 approach has been discussed for a long time (see [3]) and I believe this is better than emulating UEFI firmware in Xen. We want to keep Xen on ARM tiny. Adding any sort of emulation will increase the attack surface and require more maintenance from our side.

OK thanks, would re-using OVMF (note, DT perhaps may not be ideal for x86 for the rest though) be a reasonable solution on x86 as an option then?

Quoted 4 lines · Luis R. Rodriguez

Quoted 2 lines · Konrad Rzeszutek Wilk

But to make that work you have to emulate EFI firmware in the hypervisor. Is that work you are signing up for?

Quoted 4 lines

I'll do what is needed, as I have done before. If EFI is on the long term roadmap for ARM perhaps there are a few birds to knock with one stone here. If there is also interest to support other OSes through EFI standard means this also should help make that easier.

We already have a working solution for EFI on ARM which does not require to emulate the firmware in the hypervisor.

On ARM, the EFI stub is communicating with the kernel using device-tree [1]. Once the EFI stub has ended, the native path (i.e non-UEFI) will be executed normally and it won't be possible to use BootServices anymore.

For the guest, we provide a full support of EFI using OVMF.

For DOM0, Xen
will craft the UEFI system table and the UEFI memory map. The locations of those tables will be passed to DOM0 using a tiny device-tree [1] and the kernel will boot using the native path. The runtime services for DOM0 will be provided via hypercall.

The DOM0 approach has been discussed for a long time (see [3]) and I believe this is better than emulating UEFI firmware in Xen. We want to keep Xen on ARM tiny. Adding any sort of emulation will increase the attack surface and require more maintenance from our side.

Quoted 9 lines · Konrad Rzeszutek Wilk

Quoted 7 lines · Luis R. Rodriguez

Quoted 1 line · Konrad Rzeszutek Wilk

On Thu, Apr 14, 2016 at 12:23:17AM +0200, Luis R. Rodriguez wrote:

PV support from the kernel (not the hypervisor) and require hardware virtualization 5 years from now on the Linux kernel, it doesn't seem to me far fetched to at the very least consider using an EFI entry instead, specially since all it does is set boot params and we can make re-use this for HVMLite too.

Quoted 2 lines

But to make that work you have to emulate EFI firmware in the hypervisor. Is that work you are signing up for?

I'll do what is needed, as I have done before. If EFI is on the long term roadmap for ARM perhaps there are a few birds to knock with one stone here. If there is also interest to support other OSes through EFI standard means this also should help make that easier.

Quoted 12 lines · Luis R. Rodriguez

Quoted 11 lines · Konrad Rzeszutek Wilk

Quoted 9 lines · Luis R. Rodriguez

Quoted 2 lines · Juergen Gross

What would be gained by using the same entry but having two different boot paths after it?

Its a good question. In summary for me it would be the push for sharing more code and the push for semantics on early boot to address differences proactively, and ultimately it may enable us to help bring closer the old PV boot path closer.

Quoted 8 lines

Quoted 1 line

But why? We want to kill PV (eventually).

Yeah yeah, but its still there, and we'll have to live with it for at least minimum 5 years I hear. Part of my interest is to see to it that this path gets less disruption and issues, and we also address dead code issues which pvops simply folded under the rug. The dead code concerns may exist still for hvmlite, so unless someone is willing to make a bold claim there is none, its something to consider.

What is this dead code you speak of? Is it MTRR? Is early path code that PV misses (like KASL or other?)

The entrace point in Linux "proper" is startup_32 or startup_64 - the same path that EFI uses.

If you were to draw this (very simplified):

a)- GRUB2 ---------------------\ (creates an bootparam structure)

                                \
                                 +---- startup_32 or startup_64

b) EFI -> Linux EFI stub -------/
(creates bootparm) /
c) GRUB2-EFI -> Linux EFI----/
stub /
d) HVMLite ----------------/
(creates bootparm)

(I am not sure about the c) - I would have to look in source to be source). There is also LILO in this, but I am not even sure if works anymore.

What you have is that every entry point creates the bootparams and ends up calling startup_X. The startup_64 then hit the rest of the kernel. The startp_X code is the one that would setup the basic pagetables, segments, etc.

Quoted 1 line

How we address semantics then is *very* important to me.

Which semantics? How the CPU is going to be at startup_X ? Or how the CPU is going to be when EFI firmware invokes the EFI stub? Or when GRUB2 loads Linux?

That (those bootloaders) is clearly defined. The URL I provided mentions the HVMLite one. The Documentation/x86/boot.c mentions what the semantics are to expected when providing an bootstrap (which is what HVMLitel stub code in Linux would write against - and what EFI stub code had been written against too).

Quoted 9 lines

Quoted 9 lines

Quoted 7 lines

I'll elaborate on this but first let's clarify why a new entry is used for HVMlite to start of with:

1) Xen ABI has historically not wanted to set up the boot params for Linux guests, instead it insists on letting the Linux kernel Xen boot stubs fill that out for it. This sticking point means it has implicated a boot stub.

Quoted 3 lines

Quoted 1 line

Which is b/c it has to be OS agnostic. It has nothing to do 'not wanting'.

It can still be OS agnostic and pass on type and custom data pointer.

Sure. It has that (it MUST otherwise how else would you pass data).
It is documented as well http://xenbits.xen.org/docs/unstable/hypercall/x86_64/include,public,xen.h.html#incontents_startofday
(see " Start of day structure passed to PVH guests in %ebx.")

* You don't need full EFI emulation

A huge summary of the discussion over EFI boot option for HVMLite is now on a wiki [2], below I'll just provide the outline of the discussion. Consider this a request for more public review, feel free to take any of the items below and elaborate on it as you see fit.

* Issues with boot x86 boot entries
* Small x86 zero page stubs

* Points against using EFI
* Nulling the claimed boot loader effect

* Why use EFI for HVMlite
* EFI calling conventions are standardized
* EFI entry generalizes what new HVMLite entry proposes
* Further semantics may be needed
* Match Xen ARM's clean solution
* You don't need full EFI emulation
* Minimal EFI stubs for guests
* GetMemoryMap()
* ExitBootServices()
* EFI stubs which may be needed for guests
* Exit()
* Variable operation functions

* EFI stubs not needed for guests
* GetTime()/SetTime()
* SetVirtualAddressMap()
* ResetSystem()
* dom0 EFI
* domU EFI emulation possibilities
* Xen implements its own EFI environment for guests
* Xen uses Tianocore / OVMF

So rather than make a new entry point which does just the minimal amount of work to run on a software interface (Xen), you want to take an interface designed for hardware (EFI) and put in hacks so that it knows that sometimes some EFI services are not available? That sounds like it's going to make the EFI path just as unmanageable as the current PV path.

Using the EFI entry point would certainly make sense if it was actually simpler than the proposed extra entry point. But it sounds like it's going to be more complicated, not only for Xen, but also for Linux.

So rather than make a new entry point which does just the minimal amount of work to run on a software interface (Xen), you want to take an interface designed for hardware (EFI) and put in hacks so that it knows that sometimes some EFI services are not available? That sounds like it's going to make the EFI path just as unmanageable as the current PV path.

Using the EFI entry point would certainly make sense if it was actually simpler than the proposed extra entry point. But it sounds like it's going to be more complicated, not only for Xen, but also for Linux.

Quoted 7 lines · George Dunlap

So rather than make a new entry point which does just the minimal amount of work to run on a software interface (Xen), you want to take an interface designed for hardware (EFI) and put in hacks so that it knows that sometimes some EFI services are not available? That sounds like it's going to make the EFI path just as unmanageable as the current PV path.

Requiring code in the new entry point to manipulate control registers and do the switch to long-mode does not seem like a minimal amount of code to me,

http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg00134.html

What's likely to happen in the future is that startup_(32|64) will be entered with different settings depending on whether coming from HVMlite or bare metal, due to the natural tendency for these kinds of code paths to diverge.

Sometimes EFI runtime services are not available on bare metal hardware too, for example, when booting 32-bit kernels on 64-bit EFI or 64-bit kernels on 32-bit EFI without CONFIG_EFI_MIXED. Or when booting with the "noefi" kernel command line parameter. That's how things work today when booting Xen, we disable the runtime services.

EFI boot services are a different story however, and the EFI boot stub would need to be changed to handle that. Though honestly, it would make more sense to provide EFI services stubs in the kernel image itself that are implemented using hypercalls, and assuming you can run hypercalls that early in boot.

One place that struck me as suitable for this "hypercall in an EFI service stub" approach is the trouble with doing ACPI reboot as documented here,

http://lists.xen.org/archives/html/xen-devel/2016-02/msg01609.html

Performing the reset hypercall from within HVMlite's custom EfiReset() service would avoid having to touch ACPICA at all, and would be indistinguishable from bare metal.

Quoted 4 lines

Using the EFI entry point would certainly make sense if it was actually simpler than the proposed extra entry point. But it sounds like it's going to be more complicated, not only for Xen, but also for Linux.

Until someone sits down and writes the code I think we're going to be arguing back and forth over this particular point.

Quoted 13 lines · Matt Fleming

Quoted 7 lines · George Dunlap

So rather than make a new entry point which does just the minimal amount of work to run on a software interface (Xen), you want to take an interface designed for hardware (EFI) and put in hacks so that it knows that sometimes some EFI services are not available? That sounds like it's going to make the EFI path just as unmanageable as the current PV path.

Requiring code in the new entry point to manipulate control registers and do the switch to long-mode does not seem like a minimal amount of code to me,

Quoted 6 lines

http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg00134.html

What's likely to happen in the future is that startup_(32|64) will be entered with different settings depending on whether coming from HVMlite or bare metal, due to the natural tendency for these kinds of code paths to diverge.

I hope they do not have the same churn as the rest of Linux code.

The startup_(32|64) are to be called from divergent bootloaders - and they are responsible to set the stage. Or in other words - startup_(32|64) has some expectations of what the world will look like. Changing those means the bootloaders stub have to change too.

But if there is churn it surely is less than what the PV code paths are enforcing now in x86 code.

Quoted 23 lines · Konrad Rzeszutek Wilk

Quoted 13 lines · Matt Fleming

Quoted 7 lines · George Dunlap

So rather than make a new entry point which does just the minimal amount of work to run on a software interface (Xen), you want to take an interface designed for hardware (EFI) and put in hacks so that it knows that sometimes some EFI services are not available? That sounds like it's going to make the EFI path just as unmanageable as the current PV path.

Requiring code in the new entry point to manipulate control registers and do the switch to long-mode does not seem like a minimal amount of code to me,

Quoted 6 lines

http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg00134.html

What's likely to happen in the future is that startup_(32|64) will be entered with different settings depending on whether coming from HVMlite or bare metal, due to the natural tendency for these kinds of code paths to diverge.

I hope they do not have the same churn as the rest of Linux code.

Quoted 8 lines

The startup_(32|64) are to be called from divergent bootloaders - and they are responsible to set the stage. Or in other words - startup_(32|64) has some expectations of what the world will look like. Changing those means the bootloaders stub have to change too.

But if there is churn it surely is less than what the PV code paths are enforcing now in x86 code.

Let me expand on that since I was not sure if I was clear.

Currently Boris tirelessly ends up fixing on almost every merge window Xen related fallout. That is new functionality that breaks Xen. He has been doing this for years and before him I was doing it.

This is what an maintainer does - and with the HVMLite/PVH stub paths that will still continue - that is fallout from the startup_(32|64) code changes will be handled as before.

However the bigger goals are that:
- This churn will be much much lower than the existing one,

- baremetal won't have to deal with some rather odd semantics placed by the pvops paths that are funky and drive x86 maintainers to lose hair (amongts other things).

Quoted 23 lines · Konrad Rzeszutek Wilk

Quoted 13 lines · Matt Fleming

Quoted 7 lines · George Dunlap

So rather than make a new entry point which does just the minimal amount of work to run on a software interface (Xen), you want to take an interface designed for hardware (EFI) and put in hacks so that it knows that sometimes some EFI services are not available? That sounds like it's going to make the EFI path just as unmanageable as the current PV path.

Requiring code in the new entry point to manipulate control registers and do the switch to long-mode does not seem like a minimal amount of code to me,

Quoted 6 lines

http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg00134.html

What's likely to happen in the future is that startup_(32|64) will be entered with different settings depending on whether coming from HVMlite or bare metal, due to the natural tendency for these kinds of code paths to diverge.

I hope they do not have the same churn as the rest of Linux code.

Quoted 8 lines

The startup_(32|64) are to be called from divergent bootloaders - and they are responsible to set the stage. Or in other words - startup_(32|64) has some expectations of what the world will look like. Changing those means the bootloaders stub have to change too.

But if there is churn it surely is less than what the PV code paths are enforcing now in x86 code.

Let me expand on that since I was not sure if I was clear.

Currently Boris tirelessly ends up fixing on almost every merge window Xen related fallout. That is new functionality that breaks Xen. He has been doing this for years and before him I was doing it.

This is what an maintainer does - and with the HVMLite/PVH stub paths that will still continue - that is fallout from the startup_(32|64) code changes will be handled as before.

However the bigger goals are that:
- This churn will be much much lower than the existing one,

- baremetal won't have to deal with some rather odd semantics placed by the pvops paths that are funky and drive x86 maintainers to lose hair (amongts other things).

One place that struck me as suitable for this "hypercall in an EFI service stub" approach is the trouble with doing ACPI reboot as documented here,

http://lists.xen.org/archives/html/xen-devel/2016-02/msg01609.html

Performing the reset hypercall from within HVMlite's custom EfiReset() service would avoid having to touch ACPICA at all, and would be indistinguishable from bare metal.

I don't get this, the "reset/shutdown" hypercall requires the following steps from Dom0 (it's not as simple as calling a hypercall):

The way to perform a full system power off from Dom0 is different than what's done in a DomU guest. In order to perform a power off from Dom0 the native ACPI path should be followed, but the guest should not write the SLP_EN bit to the Pm1Control register. Instead the XENPF_enter_acpi_sleep hypercall should be used, filling the following data in the xen_platform_op struct:

cmd = XENPF_enter_acpi_sleep interface_version = XENPF_INTERFACE_VERSION u.enter_acpi_sleep.pm1a_cnt_val = Pm1aControlValue u.enter_acpi_sleep.pm1b_cnt_val = Pm1bControlValue

At which point it means that we are either going to duplicate ACPICA code into the HVMlite's custom EfiReset() service, or we are going to call into ACPICA, which is what we already do now.

I don't get this, the "reset/shutdown" hypercall requires the following steps from Dom0 (it's not as simple as calling a hypercall):

The way to perform a full system power off from Dom0 is different than what's done in a DomU guest. In order to perform a power off from Dom0 the native ACPI path should be followed, but the guest should not write the SLP_EN bit to the Pm1Control register. Instead the XENPF_enter_acpi_sleep hypercall should be used, filling the following data in the xen_platform_op struct:

cmd = XENPF_enter_acpi_sleep interface_version = XENPF_INTERFACE_VERSION u.enter_acpi_sleep.pm1a_cnt_val = Pm1aControlValue u.enter_acpi_sleep.pm1b_cnt_val = Pm1bControlValue

At which point it means that we are either going to duplicate ACPICA code into the HVMlite's custom EfiReset() service, or we are going to call into ACPICA, which is what we already do now.

One place that struck me as suitable for this "hypercall in an EFI service stub" approach is the trouble with doing ACPI reboot as documented here,

http://lists.xen.org/archives/html/xen-devel/2016-02/msg01609.html

Performing the reset hypercall from within HVMlite's custom EfiReset() service would avoid having to touch ACPICA at all, and would be indistinguishable from bare metal.

Quoted 4 lines · Luis R. Rodriguez

A huge summary of the discussion over EFI boot option for HVMLite is now on a wiki [2], below I'll just provide the outline of the discussion. Consider this a request for more public review, feel free to take any of the items below and elaborate on it as you see fit.

[snip]

Quoted 2 lines

* Issues with boot x86 boot entries
* Small x86 zero page stubs

[snip]

Quoted 2 lines

* Points against using EFI
* Nulling the claimed boot loader effect

I'm a bit confused about this. You list exactly two arguments against the proposed stub in the "con" section:
1. Bootloaders may not be able to use the extra entry point
2. It's an extra entry point

And then later, in another section, you actually list the reason #1 is
irrelevant: bootloaders don't matter because the stub is there to boot from the Xen hypervisor.

So the only actual argument you have against the proposed PVH stub in the linked document is that it's an extra entry point.

Quoted 13 lines

* Why use EFI for HVMlite
* EFI calling conventions are standardized
* EFI entry generalizes what new HVMLite entry proposes
* Further semantics may be needed
* Match Xen ARM's clean solution
* You don't need full EFI emulation
* Minimal EFI stubs for guests
* GetMemoryMap()
* ExitBootServices()
* EFI stubs which may be needed for guests
* Exit()
* Variable operation functions

Quoted 8 lines

* EFI stubs not needed for guests
* GetTime()/SetTime()
* SetVirtualAddressMap()
* ResetSystem()
* dom0 EFI
* domU EFI emulation possibilities
* Xen implements its own EFI environment for guests
* Xen uses Tianocore / OVMF

So rather than make a new entry point which does just the minimal amount of work to run on a software interface (Xen), you want to take an interface designed for hardware (EFI) and put in hacks so that it knows that sometimes some EFI services are not available?

That sounds like it's going to make the EFI path just as unmanageable as the current PV path.

Using the EFI entry point would certainly make sense if it was actually simpler than the proposed extra entry point. But it sounds like it's going to be more complicated, not only for Xen, but also for Linux.

While Andrew's position is right in that perhaps only Xen tools have to deal with the HVMLite specific entry, it would also still mean diverging from ARM's own EFI entry only position, which I'd like to clarify that ARM has no custom Xen entry, we should strive to match that. Anything far from that to me really deserves an explanation, specially if we are going to argue that HVMLite is the best that x86 Xen can do.

Ultimately unifying entry approaches for Xen in a streamlined fashion seems like a sensible thing to strive for. Anything we push in the other direction, as small as it can be, should deserve at least a 'hey, wait a minute'...

Quoted 10 lines · Luis R. Rodriguez

While Andrew's position is right in that perhaps only Xen tools have to deal with the HVMLite specific entry, it would also still mean diverging from ARM's own EFI entry only position, which I'd like to clarify that ARM has no custom Xen entry, we should strive to match that. Anything far from that to me really deserves an explanation, specially if we are going to argue that HVMLite is the best that x86 Xen can do.

Ultimately unifying entry approaches for Xen in a streamlined fashion seems like a sensible thing to strive for. Anything we push in the other direction, as small as it can be, should deserve at least a 'hey, wait a minute'...

Quick factual correction here.

"Since ARM guests only use the EFI entry point, x86 guests should also only use the EFI entry point" is certainly a reasonable argument to make.

However, dom0 on ARM does not use the EFI entry point. When starting dom0, Xen uses the native entry point (the one that UBoot uses) and hands dom0 a device-tree node. The reason this is possible on ARM is that there are no assumptions made about what hardware is or is not present on the system -- everything that needs to be communicated about what is or is not present can be passed in DT.

So it is incorrect to say that ARM has an "EFI entry only" position.

(On ACPI systems, it does apparently generate some UEFI informational tables, which it passes to the dom0 kernel via DT; and the kernel unpacks and puts in the right place. Normal Xen ARM guests can use EFI, but that's because we start OVMF in the guest context to provide the EFI services. These may be where the idea that ARM guests use only the UEFI entry point came from.)

Obviously it would be nice if we could use the native entry point on x86 as well, but there's decades of legacy hardware and backwards compatibility to deal with there.

Quoted 10 lines · Luis R. Rodriguez

While Andrew's position is right in that perhaps only Xen tools have to deal with the HVMLite specific entry, it would also still mean diverging from ARM's own EFI entry only position, which I'd like to clarify that ARM has no custom Xen entry, we should strive to match that. Anything far from that to me really deserves an explanation, specially if we are going to argue that HVMLite is the best that x86 Xen can do.

Ultimately unifying entry approaches for Xen in a streamlined fashion seems like a sensible thing to strive for. Anything we push in the other direction, as small as it can be, should deserve at least a 'hey, wait a minute'...

Quick factual correction here.

"Since ARM guests only use the EFI entry point, x86 guests should also only use the EFI entry point" is certainly a reasonable argument to make.

However, dom0 on ARM does not use the EFI entry point. When starting dom0, Xen uses the native entry point (the one that UBoot uses) and hands dom0 a device-tree node. The reason this is possible on ARM is that there are no assumptions made about what hardware is or is not present on the system -- everything that needs to be communicated about what is or is not present can be passed in DT.

So it is incorrect to say that ARM has an "EFI entry only" position.

(On ACPI systems, it does apparently generate some UEFI informational tables, which it passes to the dom0 kernel via DT; and the kernel unpacks and puts in the right place. Normal Xen ARM guests can use EFI, but that's because we start OVMF in the guest context to provide the EFI services. These may be where the idea that ARM guests use only the UEFI entry point came from.)

Obviously it would be nice if we could use the native entry point on x86 as well, but there's decades of legacy hardware and backwards compatibility to deal with there.

Quoted 33 lines · George Dunlap

Quoted 10 lines · Luis R. Rodriguez

While Andrew's position is right in that perhaps only Xen tools have to deal with the HVMLite specific entry, it would also still mean diverging from ARM's own EFI entry only position, which I'd like to clarify that ARM has no custom Xen entry, we should strive to match that. Anything far from that to me really deserves an explanation, specially if we are going to argue that HVMLite is the best that x86 Xen can do.

Ultimately unifying entry approaches for Xen in a streamlined fashion seems like a sensible thing to strive for. Anything we push in the other direction, as small as it can be, should deserve at least a 'hey, wait a minute'...

Quick factual correction here.

"Since ARM guests only use the EFI entry point, x86 guests should also only use the EFI entry point" is certainly a reasonable argument to make.

However, dom0 on ARM does not use the EFI entry point. When starting dom0, Xen uses the native entry point (the one that UBoot uses) and hands dom0 a device-tree node. The reason this is possible on ARM is that there are no assumptions made about what hardware is or is not present on the system -- everything that needs to be communicated about what is or is not present can be passed in DT.

So it is incorrect to say that ARM has an "EFI entry only" position.

(On ACPI systems, it does apparently generate some UEFI informational tables, which it passes to the dom0 kernel via DT; and the kernel unpacks and puts in the right place. Normal Xen ARM guests can use EFI, but that's because we start OVMF in the guest context to provide the EFI services. These may be where the idea that ARM guests use only the UEFI entry point came from.)

Quoted 3 lines

Obviously it would be nice if we could use the native entry point on x86 as well, but there's decades of legacy hardware and backwards compatibility to deal with there.

OK thanks for the clarification -- still no custom entries for Xen! We should strive for that, at the very least.

You do have a point about the legacy stuff. There are two options there:

* Fold legacy support under HVMLite -- which seems to be what we currently want to do (we should evaluate the implications and requirements here for that); or

* Leave legacy stuff on the old PV path; this may be something to bring to the table if we had in place a proactive solution to avoid further fallout from the architecture of the huge differences on the entries. The work I'm doing should help with that. (We should also evaluate the implications and requirements here for that as well).

Also, x86 does have a history of short DT use. Just pointing that its there as an option as well. I'll Cc you on some thread about that.

Quoted 2 lines · Luis R. Rodriguez

Also, x86 does have a history of short DT use. Just pointing that its there as an option as well. I'll Cc you on some thread about that.

I'm not sure how this is relevant to anything.

What we're talking about is how to get from Xen to a point in the Linux kernel where everything can Just Work. The proposed feature is a mini trampoline that (as I understand it):
1. Tells Xen where to jump to (via ELF note)
2. Sets up some basic modes and pagetables and then jumps to the zero page so Linux can just carry on.

Quoted 5 lines · George Dunlap

Quoted 2 lines · Luis R. Rodriguez

Also, x86 does have a history of short DT use. Just pointing that its there as an option as well. I'll Cc you on some thread about that.

I'm not sure how this is relevant to anything.

You brought DT as a reason why ARM was able to use the native point. I'm clarifying DT has nothing to do as a restriction on x86.

One. CE4100.

arch/x86/platform/ce4100/falconfalls.dt

Quoted 6 lines

What we're talking about is how to get from Xen to a point in the Linux kernel where everything can Just Work. The proposed feature is a mini trampoline that (as I understand it):
1. Tells Xen where to jump to (via ELF note)
2. Sets up some basic modes and pagetables and then jumps to the zero page so Linux can just carry on.

Right, and the my goal is to see to it we do enough homework to ensure we reviewed all possibilities to share as much code as possible already and looked at all options before saying we certainly need yet another entry point. I am not convinced yet this has been done.

Quoted 5 lines · George Dunlap

Quoted 2 lines · Luis R. Rodriguez

Also, x86 does have a history of short DT use. Just pointing that its there as an option as well. I'll Cc you on some thread about that.

I'm not sure how this is relevant to anything.

You brought DT as a reason why ARM was able to use the native point. I'm clarifying DT has nothing to do as a restriction on x86.

One. CE4100.

arch/x86/platform/ce4100/falconfalls.dt

Quoted 6 lines

What we're talking about is how to get from Xen to a point in the Linux kernel where everything can Just Work. The proposed feature is a mini trampoline that (as I understand it):
1. Tells Xen where to jump to (via ELF note)
2. Sets up some basic modes and pagetables and then jumps to the zero page so Linux can just carry on.

Right, and the my goal is to see to it we do enough homework to ensure we reviewed all possibilities to share as much code as possible already and looked at all options before saying we certainly need yet another entry point. I am not convinced yet this has been done.

Quoted 9 lines · Luis R. Rodriguez

Quoted 5 lines · George Dunlap

Quoted 2 lines · Luis R. Rodriguez

Also, x86 does have a history of short DT use. Just pointing that its there as an option as well. I'll Cc you on some thread about that.

I'm not sure how this is relevant to anything.

You brought DT as a reason why ARM was able to use the native point. I'm clarifying DT has nothing to do as a restriction on x86.

No, DT isn't the reason Xen is able to use the native entry point on ARM. The reason is, to quote myself: "there are no assumptions made about what hardware is or is not present on the system -- everything that needs to be communicated about what is or is not present can be passed in DT."

So that's three things:
1. DT is available to be used
2. DT is expected as the main thing that entry point accepts
3. There are no assumptions about what hardware is or is not present in the system
4. Everything that needs to be communicated about what is or is not present can be passed in DT.

Are #2, #3, and #4 true on x86? If not then #1 is irrelevant.

[snip from another thread]

Quoted 3 lines

One. CE4100.

arch/x86/platform/ce4100/falconfalls.dt

You CC'd me on some patches related to that. I don't know anything about the code, but it looked like CE4100 is a subarch, and in response to that thread Ingo specifically asked you to add a comment saying basically "Don't add any more subarches".

And not only that, but the ugly, nasty legacy PV boot path we're trying to get rid of IS ALSO A SUBARCH. So instead of a quick stub with an extra EFI flag, you're proposing we consider add yet another Xen PV subarch?

Quoted 7 lines

Quoted 6 lines

What we're talking about is how to get from Xen to a point in the Linux kernel where everything can Just Work. The proposed feature is a mini trampoline that (as I understand it):
1. Tells Xen where to jump to (via ELF note)
2. Sets up some basic modes and pagetables and then jumps to the zero page so Linux can just carry on.

Quoted 4 lines

Right, and the my goal is to see to it we do enough homework to ensure we reviewed all possibilities to share as much code as possible already and looked at all options before saying we certainly need yet another entry point. I am not convinced yet this has been done.

I think we have different ideas about what an appropriate amount of homework is. :-) Everything you've put forward has been given consideration and judged unlikely to be promising;

and your suggestions for further possibilities (like this one) keep getting more and more obviously unsuitable.

We shouldn't be required to actually post code
for every single other option just to prove how ugly they are, particularly when there's nothing particularly wrong with the code we have.

Quoted 9 lines · Luis R. Rodriguez

Quoted 5 lines · George Dunlap

Quoted 2 lines · Luis R. Rodriguez

Also, x86 does have a history of short DT use. Just pointing that its there as an option as well. I'll Cc you on some thread about that.

I'm not sure how this is relevant to anything.

You brought DT as a reason why ARM was able to use the native point. I'm clarifying DT has nothing to do as a restriction on x86.

No, DT isn't the reason Xen is able to use the native entry point on ARM. The reason is, to quote myself: "there are no assumptions made about what hardware is or is not present on the system -- everything that needs to be communicated about what is or is not present can be passed in DT."

So that's three things:
1. DT is available to be used
2. DT is expected as the main thing that entry point accepts
3. There are no assumptions about what hardware is or is not present in the system
4. Everything that needs to be communicated about what is or is not present can be passed in DT.

Are #2, #3, and #4 true on x86? If not then #1 is irrelevant.

[snip from another thread]

Quoted 3 lines

One. CE4100.

arch/x86/platform/ce4100/falconfalls.dt

You CC'd me on some patches related to that. I don't know anything about the code, but it looked like CE4100 is a subarch, and in response to that thread Ingo specifically asked you to add a comment saying basically "Don't add any more subarches".

And not only that, but the ugly, nasty legacy PV boot path we're trying to get rid of IS ALSO A SUBARCH. So instead of a quick stub with an extra EFI flag, you're proposing we consider add yet another Xen PV subarch?

Quoted 7 lines

Quoted 6 lines

What we're talking about is how to get from Xen to a point in the Linux kernel where everything can Just Work. The proposed feature is a mini trampoline that (as I understand it):
1. Tells Xen where to jump to (via ELF note)
2. Sets up some basic modes and pagetables and then jumps to the zero page so Linux can just carry on.

Quoted 4 lines

Right, and the my goal is to see to it we do enough homework to ensure we reviewed all possibilities to share as much code as possible already and looked at all options before saying we certainly need yet another entry point. I am not convinced yet this has been done.

I think we have different ideas about what an appropriate amount of homework is. :-) Everything you've put forward has been given consideration and judged unlikely to be promising;

and your suggestions for further possibilities (like this one) keep getting more and more obviously unsuitable.

We shouldn't be required to actually post code
for every single other option just to prove how ugly they are, particularly when there's nothing particularly wrong with the code we have.