ARM/FDT: passing multiple binaries to a kernel

Quoted 4 lines · Andre Przywara

I am about to write up a more elaborate technical rationale describing the problems with multiboot on ARM:
https://wiki.linaro.org/AndrePrzywara/Multiboot

Doesn't seem to exist? A search for "mulitboot" doesn't seem to throw up the one you meant either.

Quoted 7 lines

Quoted 1 line
So, is having a more generic solution really needed?
Not necessarily needed, but useful, I think. As described above I don't see any technical obstacles of doing it in a more generic way, so we could as well go ahead with this. On x86 from time to time the need for additional binaries pops up (early microcode loading, for instance), so why not be be prepared.

Ian.

Rob Herring

2013-09-05 13:23 UTC #7 lore raw

Quoted 9 lines · Andre Przywara

Quoted 6 lines · Rob Herring
On Tue, Sep 3, 2013 at 10:53 AM, Andre Przywara
But surely there would be some advantage of reuse by using the multi-boot protocol since Xen, grub, and OS tools already support it for x86.

Quoted 7 lines

Yes, but that is x86 only and multiboot is it's nature quite architecture specific. The current(?) multiboot v2 spec has no official ARM support (only x86 and MIPS), so this would need to be "invented" first. While this is technically easy, ARM software currently has no support for multiboot at
all: not in u-boot and not in Xen.
Multiboot support in Xen lives entirely in the x86 directory, and big parts of it are even in assembly.

If so, I cannot find the assembly and most of it seems to be moved out of x86 code now.

Rob

Andre Przywara

author

2013-09-06 13:30 UTC #8 lore raw

Quoted 19 lines · Rob Herring

Quoted 9 lines · Andre Przywara
Quoted 6 lines · Rob Herring
On Tue, Sep 3, 2013 at 10:53 AM, Andre Przywara
But surely there would be some advantage of reuse by using the multi-boot protocol since Xen, grub, and OS tools already support it for x86.
Quoted 7 lines
Yes, but that is x86 only and multiboot is it's nature quite architecture specific. The current(?) multiboot v2 spec has no official ARM support (only x86 and MIPS), so this would need to be "invented" first. While this is technically easy, ARM software currently has no support for multiboot at
all: not in u-boot and not in Xen.
Multiboot support in Xen lives entirely in the x86 directory, and big parts of it are even in assembly.

Quoted 2 lines

If so, I cannot find the assembly and most of it seems to be moved out of x86 code now.

There is some code that relocates and maps the multiboot structure: in files in the xen/arch/x86/boot directory, including head.S, reloc.c and setup.c.
That would need to be completely rewritten, not only because it deals with multiboot v1, but also because it is scattered throughout the x86 setup code and the initial page setup is different from ARM.

All these things are also done on ARM - but for the device tree already. Of course proper multiboot v2 support is doable - but please compare the effort needed for this against this one:
http://lists.xen.org/archives/html/xen-devel/2013-09/msg00359.html

Regards,
Andre.

Andre Przywara

author

2013-09-06 13:30 UTC #9 lore raw

Quoted 19 lines · Rob Herring

Quoted 9 lines · Andre Przywara
Quoted 6 lines · Rob Herring
On Tue, Sep 3, 2013 at 10:53 AM, Andre Przywara
But surely there would be some advantage of reuse by using the multi-boot protocol since Xen, grub, and OS tools already support it for x86.
Quoted 7 lines
Yes, but that is x86 only and multiboot is it's nature quite architecture specific. The current(?) multiboot v2 spec has no official ARM support (only x86 and MIPS), so this would need to be "invented" first. While this is technically easy, ARM software currently has no support for multiboot at
all: not in u-boot and not in Xen.
Multiboot support in Xen lives entirely in the x86 directory, and big parts of it are even in assembly.

Quoted 2 lines

If so, I cannot find the assembly and most of it seems to be moved out of x86 code now.

All these things are also done on ARM - but for the device tree already.
Of course proper multiboot v2 support is doable - but please compare the effort needed for this against this one:
http://lists.xen.org/archives/html/xen-devel/2013-09/msg00359.html

Regards,
Andre.

Rob Herring

2013-09-05 13:23 UTC #10 lore raw

Quoted 9 lines · Andre Przywara

Quoted 6 lines · Rob Herring
On Tue, Sep 3, 2013 at 10:53 AM, Andre Przywara
But surely there would be some advantage of reuse by using the multi-boot protocol since Xen, grub, and OS tools already support it for x86.

Quoted 7 lines

Yes, but that is x86 only and multiboot is it's nature quite architecture specific. The current(?) multiboot v2 spec has no official ARM support (only x86 and MIPS), so this would need to be "invented" first. While this is technically easy, ARM software currently has no support for multiboot at
all: not in u-boot and not in Xen.
Multiboot support in Xen lives entirely in the x86 directory, and big parts of it are even in assembly.

If so, I cannot find the assembly and most of it seems to be moved out of x86 code now.

Rob

Andre Przywara

author

2013-09-04 08:43 UTC #11 lore raw

https://wiki.linaro.org/AndrePrzywara/Multiboot

Quoted 1 line · Rob Herring

On Tue, Sep 3, 2013 at 10:53 AM, Andre Przywara

Hi Rob,

Quoted 10 lines

Quoted 10 lines
a normal Linux kernel currently supports reading the start and end address of a single binary blob via the FDT's /chosen node.
This will be interpreted as the location of an initial RAM disk.
The Xen hypervisor itself is a kernel, but needs up to _two_ binaries for proper operation: a Dom0 Linux kernel and it's associated initrd. On x86 this is solved via the multiboot protocol used by the Grub bootloader, which supports to pass an arbitrary number of binary modules to any kernel.

Quoted 6 lines

Quoted 2 lines
Since in the ARM world we have the versatile device tree, we don't need to implement the mulitboot protocol.
But surely there would be some advantage of reuse by using the multi-boot protocol since Xen, grub, and OS tools already support it for x86.

I am about to write up a more elaborate technical rationale describing the problems with multiboot on ARM:

Quoted 10 lines

Quoted 9 lines
So I'd like to propose a new binding which denotes binary modules a kernel can use at it's own discretion.
The need is triggered by the Xen hypervisor (which already uses a very similar scheme), but the approach is deliberately chosen to be as generic as possible to allow future uses (like passing firmware blobs for devices or the like).
Credits for this go to Ian Campbell, who started something very similar [1] for the Xen hypervisor. The intention of this proposal is to make this generic and publicly documented.

Quoted 4 lines

Can you describe how you see the boot flow working starting with OS installer writes kernel, initrd, xen and ??? to disk. How does the bootloader know what to load? The OS may not have access to the dtb, so this has to be described to the bootloader as well.

The idea is to use bootscripts (for instance in u-boot) to tackle this. See for an example below.

Quoted 46 lines

Quoted 31 lines
Looking forward to any comments!
Thanks,
Andre.
[1]
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/misc/arm/device-tree/booting.txt;h=94cd3f18a4e1317a35e1255bf5c6e1e091001d1a;hb=HEAD
----------------------------
* Multiple boot modules device tree bindings
Boot loaders wanting to pass multiple additional binaries to a kernel shall add a node "module" for each binary blob under the /chosen node with the following properties:
- compatible:
     compatible = "boot,module";
A bootloader may add names to more specifically describe the module, e.g. Xen may use "xen,dom0-kernel" or "xen,dom0-ramdisk". If possible a kernel should be able to use modules even without a descriptive naming, by enumerating them in order and using hard-coded meanings for each module (e.g. first is kernel, second is initrd).
- reg: specifies the base physical address and size of a region in memory where the bootloader loaded the respective binary data to.
- bootargs:
An optional property describing arguments to use for this module. Could be a command line or configuration data.
Example:
Quoted 14 lines
/chosen {
     #size-cells = <0x1>;
     #address-cells = <0x1>;
     module@0 {
         compatible = "xen,linux-zimage", "xen,multiboot-module",
"boot,module";
         reg = <0x80000000 0x003dcff8>;
         bootargs = "console=hvc0 earlyprintk ro root=/dev/sda1 nosmp";
     };
     module@1 {
         compatible = "xen,linux-initrd", "xen,multiboot-module",
"boot,module";
         reg = <0x08000000 0x00123456>;
     };

Quoted 4 lines

This has to be created and parsed typically in FDT format by early boot code, and I worry about the complexity this has. Being future proof and extensible is good, but we could meet today's needs with something simple like this:

Generating is also part of libfdt, in fact this whole subtree above has been generated on the command line of a stock Calxeda U-Boot:
dom0kernel=fdt addr ${fdt_addr}; fdt resize; fdt mknod /chosen module@0; fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module" "boot,module"; fdt set /chosen/module@0 reg <${dom0_addr_r} 0x${filesize}>; fdt set /chosen/module@0 bootargs "console=hvc0 earlyprintk ro root=/dev/sda1 nosmp"

Quoted 4 lines

bootargs = "xen args --- linux args";
xen,linux-image = <start size>;
So, is having a more generic solution really needed?

Not necessarily needed, but useful, I think. As described above I don't see any technical obstacles of doing it in a more generic way, so we could as well go ahead with this. On x86 from time to time the need for additional binaries pops up (early microcode loading, for instance), so why not be be prepared. Also this approach avoids hard-coding the Xen name into the bootloader, as said in the proposal the meaning could be derived from the order of the modules (as on x86), so a bootloader does not need to know anything about Xen at all.

Regards,
Andre.

Ian Campbell

2013-09-04 08:44 UTC #12 lore raw

http://www.syslinux.org/wiki/index.php/Doc/mboot

Quoted 14 lines · Rob Herring

Quoted 12 lines · Andre Przywara
Hi,
a normal Linux kernel currently supports reading the start and end address of a single binary blob via the FDT's /chosen node.
This will be interpreted as the location of an initial RAM disk.
The Xen hypervisor itself is a kernel, but needs up to _two_ binaries for proper operation: a Dom0 Linux kernel and it's associated initrd. On x86 this is solved via the multiboot protocol used by the Grub bootloader, which supports to pass an arbitrary number of binary modules to any kernel.

Quoted 6 lines

Quoted 2 lines
Since in the ARM world we have the versatile device tree, we don't need to implement the mulitboot protocol.
But surely there would be some advantage of reuse by using the multi-boot protocol since Xen, grub, and OS tools already support it for x86.

Multiboot is pretty x86 specific (although MB2 has a MIPS port) and covers more stuff than we strictly require (e.g. on x86 it has requirements around which processor mode you enter in, has paging enabled etc).

Quoted 10 lines

Quoted 9 lines
So I'd like to propose a new binding which denotes binary modules a kernel can use at it's own discretion.
The need is triggered by the Xen hypervisor (which already uses a very similar scheme), but the approach is deliberately chosen to be as generic as possible to allow future uses (like passing firmware blobs for devices or the like).
Credits for this go to Ian Campbell, who started something very similar [1] for the Xen hypervisor. The intention of this proposal is to make this generic and publicly documented.

Quoted 2 lines

Can you describe how you see the boot flow working starting with OS installer writes kernel, initrd, xen and ??? to disk.

Kernel and initrd are written to /boot in the usual way (probably from kernel.deb or whatever). Xen would also normally come from a distro package (also in /boot).

Quoted 1 line

How does the bootloader know what to load?

It's in the bootloader config, e.g. boot.scr or grub.cfg, which are either hand written or produced by the distros tooling.

grub on ARM could consume the same stanzas as are used by grub on x86 to
boot Xen (which are produced by update-grub):
echo 'Loading Xen 4.1-amd64 ...'
multiboot /xen-4.1-amd64.gz placeholder
echo 'Loading Linux 3.10-2-amd64 ...'
module /vmlinuz-3.10-2-amd64 placeholder root=/dev/mapper/disks-root ro resume=/dev/mapper/disks-swap quiet
echo 'Loading initial ramdisk ...'
module /initrd.img-3.10-2-amd64

Since there is no multiboot on ARM (and never will be) this is safe.

If multiboot ever does come to ARM it will necessarily be multiboot2 which uses a different keyword.

For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)

Quoted 2 lines

The OS may not have access to the dtb,
so this has to be described to the bootloader as well.

Right. The proposal is not for the OS to frob the DTB to support this protocol but to configure the bootloader to do it.

Ian.

Rob Herring

2013-09-04 16:41 UTC #13 lore raw

Adding Dennis for a distro perspective.

Quoted 58 lines · Ian Campbell

Quoted 14 lines · Rob Herring
Quoted 12 lines · Andre Przywara
Hi,
a normal Linux kernel currently supports reading the start and end address of a single binary blob via the FDT's /chosen node.
This will be interpreted as the location of an initial RAM disk.
The Xen hypervisor itself is a kernel, but needs up to _two_ binaries for proper operation: a Dom0 Linux kernel and it's associated initrd. On x86 this is solved via the multiboot protocol used by the Grub bootloader, which supports to pass an arbitrary number of binary modules to any kernel.
Quoted 6 lines
Quoted 2 lines
Since in the ARM world we have the versatile device tree, we don't need to implement the mulitboot protocol.
But surely there would be some advantage of reuse by using the multi-boot protocol since Xen, grub, and OS tools already support it for x86.
Multiboot is pretty x86 specific (although MB2 has a MIPS port) and covers more stuff than we strictly require (e.g. on x86 it has requirements around which processor mode you enter in, has paging enabled etc).
Quoted 10 lines
Quoted 9 lines
So I'd like to propose a new binding which denotes binary modules a kernel can use at it's own discretion.
The need is triggered by the Xen hypervisor (which already uses a very similar scheme), but the approach is deliberately chosen to be as generic as possible to allow future uses (like passing firmware blobs for devices or the like).
Credits for this go to Ian Campbell, who started something very similar [1] for the Xen hypervisor. The intention of this proposal is to make this generic and publicly documented.
Quoted 2 lines
Can you describe how you see the boot flow working starting with OS installer writes kernel, initrd, xen and ??? to disk.
Kernel and initrd are written to /boot in the usual way (probably from kernel.deb or whatever). Xen would also normally come from a distro package (also in /boot).
Quoted 1 line
How does the bootloader know what to load?
It's in the bootloader config, e.g. boot.scr or grub.cfg, which are either hand written or produced by the distros tooling.
grub on ARM could consume the same stanzas as are used by grub on x86 to
boot Xen (which are produced by update-grub):
echo 'Loading Xen 4.1-amd64 ...'
multiboot /xen-4.1-amd64.gz placeholder
echo 'Loading Linux 3.10-2-amd64 ...'
module /vmlinuz-3.10-2-amd64 placeholder root=/dev/mapper/disks-root ro resume=/dev/mapper/disks-swap quiet
echo 'Loading initial ramdisk ...'
module /initrd.img-3.10-2-amd64

Quoted 4 lines

Since there is no multiboot on ARM (and never will be) this is safe.
If multiboot ever does come to ARM it will necessarily be multiboot2 which uses a different keyword.

Right, this is just a text file with a list of binaries. It is not really the multiboot spec. There is no reason for this part to be different for grub on ARM. There is a big advantage to reusing the distro side tooling. If there isn't really much reuse on the bootloader side, then I'm fine with a different bootloader to Xen interface. I would like to hear that from folks working on grub though.

Quoted 5 lines

For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)

And being machine specific is a PITA. flash-kernel is certainly not something we want to expand on. There is not much love for boot.scr either. There is work to address what are not really machine differences, but largely vendor u-boot differences:

http://www.mail-archive.com/u-boot at lists.denx.de/msg119025.html

One option for u-boot which already supports syslinux style menu files is to adopt the syslinux multiboot parsing support:

We need to back-up and consider what this looks like in the end for all the pieces and get input from folks on grub, UEFI, and armv8. The UEFI answer may be this is a grub problem. For armv8, this proposal does match up well as the kernel boot interface for v8 is DT. Despite some claims, ACPI will not completely replace DT because of this.

Rob

Grant Likely

2013-09-13 10:13 UTC #14 lore raw

Quoted 11 lines · Rob Herring

Quoted 5 lines
For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)
And being machine specific is a PITA. flash-kernel is certainly not something we want to expand on. There is not much love for boot.scr either. There is work to address what are not really machine differences, but largely vendor u-boot differences:

Quoted 6 lines

http://www.mail-archive.com/u-boot at lists.denx.de/msg119025.html
One option for u-boot which already supports syslinux style menu files is to adopt the syslinux multiboot parsing support:
http://www.syslinux.org/wiki/index.php/Doc/mboot

Even building it into U-Boot is problematic because it leaves older machines out in the cold. Leif's port of Grub to U-Boot is far more interesting since the distro can now be in control of the code that loads the images and jumps into the kernel/hypervisor.

Quoted 5 lines

We need to back-up and consider what this looks like in the end for all the pieces and get input from folks on grub, UEFI, and armv8. The UEFI answer may be this is a grub problem. For armv8, this proposal does match up well as the kernel boot interface for v8 is DT. Despite some claims, ACPI will not completely replace DT because of this.

Yes, for UEFI it is absolutely an OS loader problem. UEFI provides an API and runtime environment. Grub is in general moving towards a boot menu system and a tool for loading images. Actual booting however should be done by a separate OS loader application. For Linux, this will be an in-kernel UEFI Stub. For Xen I would recommend taking the Linux EFI stub code and doing the same thing. There really isn't a need for a multiboot spec when you can rely on a runtime execution environment for setting things up exactly as you want them.

Multiboot only makes sense to me when you need to rely on firmware or something else out of your control to load the images in a particular way.

Ian Campbell

2013-09-13 11:22 UTC #15 lore raw

Quoted 17 lines · Grant Likely

Quoted 16 lines · Rob Herring
Quoted 5 lines
For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)
And being machine specific is a PITA. flash-kernel is certainly not something we want to expand on. There is not much love for boot.scr either. There is work to address what are not really machine differences, but largely vendor u-boot differences:
http://www.mail-archive.com/u-boot at lists.denx.de/msg119025.html
One option for u-boot which already supports syslinux style menu files is to adopt the syslinux multiboot parsing support:

Quoted 6 lines

Quoted 1 line
http://www.syslinux.org/wiki/index.php/Doc/mboot
Even building it into U-Boot is problematic because it leaves older machines out in the cold. Leif's port of Grub to U-Boot is far more interesting since the distro can now be in control of the code that loads the images and jumps into the kernel/hypervisor.

AIUI this is not being developed any further?

Quoted 11 lines

Quoted 5 lines
We need to back-up and consider what this looks like in the end for all the pieces and get input from folks on grub, UEFI, and armv8. The UEFI answer may be this is a grub problem. For armv8, this proposal does match up well as the kernel boot interface for v8 is DT. Despite some claims, ACPI will not completely replace DT because of this.
Yes, for UEFI it is absolutely an OS loader problem. UEFI provides an API and runtime environment. Grub is in general moving towards a boot menu system and a tool for loading images. Actual booting however should be done by a separate OS loader application. For Linux, this will be an in-kernel UEFI Stub.

I'm not sure I follow your distinction between loading the images and "actual booting". If grub loads the images and Linux stub does the actual booting how does this stub locate the images which grub loaded?

Or are you saying the stub should load the initramfs itself? How does it know where to find it? Having the kernel in one config file (grub's) and the initramfs in another (the Linux UEFI stub's) seems likely to result in things getting out of sync. Having Linux's stub parse the grub CFG is even less likely to work well IMHO.

Quoted 4 lines

For Xen I would recommend taking the
Linux EFI stub code and doing the same thing. There really isn't a need for a multiboot spec when you can rely on a runtime execution environment for setting things up exactly as you want them.

If this works for Linux on EFI then I see no reason it could work for Xen on EFI (assuming my questions above are just a result of my misunderstanding something)

But... Xen also wants to support non-server and therefore non-EFI systems i.e. u-boot. We also want to support v7 where EFI is not a given even for servers AIUI.

Given that I think it is a given that Xen will have some sort of protocol along these lines, for use in these environments even if it does the EFI stub thing on EFI systems. The question is shall we make it more general and useful to others or just go our own way? I'd prefer to do the former.

Quoted 3 lines

Multiboot only makes sense to me when you need to rely on firmware or something else out of your control to load the images in a particular way.

I think in some cases do end up needing to rely on that though.

Ian.

Grant Likely

2013-09-15 12:17 UTC #16 lore raw

Quoted 18 lines · Ian Campbell

Quoted 17 lines · Grant Likely
Quoted 16 lines · Rob Herring
Quoted 5 lines
For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)
And being machine specific is a PITA. flash-kernel is certainly not something we want to expand on. There is not much love for boot.scr either. There is work to address what are not really machine differences, but largely vendor u-boot differences:
http://www.mail-archive.com/u-boot at lists.denx.de/msg119025.html
One option for u-boot which already supports syslinux style menu files is to adopt the syslinux multiboot parsing support:

Quoted 8 lines

Quoted 6 lines
Quoted 1 line
http://www.syslinux.org/wiki/index.php/Doc/mboot
Even building it into U-Boot is problematic because it leaves older machines out in the cold. Leif's port of Grub to U-Boot is far more interesting since the distro can now be in control of the code that loads the images and jumps into the kernel/hypervisor.
AIUI this is not being developed any further?

Last I checked the patches have been posted for merging, but it is stalled on the Grub maintainer. I believe he wanted to fix a bug on the raspberry pi before merging. Leif would know more. LEG isn't actively working on it anymore since UEFI is the priority, but I wouldn't call it abandoned.

Quoted 12 lines

Quoted 11 lines
Quoted 5 lines
We need to back-up and consider what this looks like in the end for all the pieces and get input from folks on grub, UEFI, and armv8. The UEFI answer may be this is a grub problem. For armv8, this proposal does match up well as the kernel boot interface for v8 is DT. Despite some claims, ACPI will not completely replace DT because of this.
Yes, for UEFI it is absolutely an OS loader problem. UEFI provides an API and runtime environment. Grub is in general moving towards a boot menu system and a tool for loading images. Actual booting however should be done by a separate OS loader application. For Linux, this will be an in-kernel UEFI Stub.

Quoted 3 lines

I'm not sure I follow your distinction between loading the images and "actual booting". If grub loads the images and Linux stub does the actual booting how does this stub locate the images which grub loaded?

(Note: this isn't implemented yet, but is in progress) The DTB will need to be passed via the UEFI system table. Initrd is passed by modifying the dtb with linux,initrd-* properties in the device tree.

In the Xen case, I think the original proposal is conceptually sound. I'll quibble about a couple of the details, but I'll address those with a reply to the original proposal. It makes perfect sense to use the same mechanism as the initrd properties to tell the kernel about additional blobs.

Quoted 5 lines

Or are you saying the stub should load the initramfs itself? How does it know where to find it? Having the kernel in one config file (grub's) and the initramfs in another (the Linux UEFI stub's) seems likely to result in things getting out of sync. Having Linux's stub parse the grub CFG is even less likely to work well IMHO.

The stub has the ability to load both the dtb and initrd itself by adding "dtb=" and "initrd=" arguments, but it is currently limited to loading images from the same filesystem that the kernel was loaded from. GRUB (or gummiboot/refind/etc) is potentially more flexible.

Quoted 5 lines

Quoted 4 lines
For Xen I would recommend taking the
Linux EFI stub code and doing the same thing. There really isn't a need for a multiboot spec when you can rely on a runtime execution environment for setting things up exactly as you want them.

Quoted 7 lines

If this works for Linux on EFI then I see no reason it could work for Xen on EFI (assuming my questions above are just a result of my misunderstanding something)
But... Xen also wants to support non-server and therefore non-EFI systems i.e. u-boot. We also want to support v7 where EFI is not a given even for servers AIUI.

I was only responding to the UEFI question, but that aside, I don't see a problem with adding properties to the DT for telling Xen where they are.

Quoted 5 lines

Given that I think it is a given that Xen will have some sort of protocol along these lines, for use in these environments even if it does the EFI stub thing on EFI systems. The question is shall we make it more general and useful to others or just go our own way? I'd prefer to do the former.

Yes, make it general. GRUB and the EFI stub will use it.

Ian Campbell

2013-09-13 11:22 UTC #17 lore raw

Quoted 17 lines · Grant Likely

Quoted 16 lines · Rob Herring
Quoted 5 lines
For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)
And being machine specific is a PITA. flash-kernel is certainly not something we want to expand on. There is not much love for boot.scr either. There is work to address what are not really machine differences, but largely vendor u-boot differences:
http://www.mail-archive.com/[email protected]/msg119025.html
One option for u-boot which already supports syslinux style menu files is to adopt the syslinux multiboot parsing support:

Quoted 6 lines

Quoted 1 line
http://www.syslinux.org/wiki/index.php/Doc/mboot
Even building it into U-Boot is problematic because it leaves older machines out in the cold. Leif's port of Grub to U-Boot is far more interesting since the distro can now be in control of the code that loads the images and jumps into the kernel/hypervisor.

AIUI this is not being developed any further?

Quoted 11 lines

Quoted 5 lines
We need to back-up and consider what this looks like in the end for all the pieces and get input from folks on grub, UEFI, and armv8. The UEFI answer may be this is a grub problem. For armv8, this proposal does match up well as the kernel boot interface for v8 is DT. Despite some claims, ACPI will not completely replace DT because of this.
Yes, for UEFI it is absolutely an OS loader problem. UEFI provides an API and runtime environment. Grub is in general moving towards a boot menu system and a tool for loading images. Actual booting however should be done by a separate OS loader application. For Linux, this will be an in-kernel UEFI Stub.

Quoted 4 lines

For Xen I would recommend taking the
Linux EFI stub code and doing the same thing. There really isn't a need for a multiboot spec when you can rely on a runtime execution environment for setting things up exactly as you want them.

If this works for Linux on EFI then I see no reason it could work for Xen on EFI (assuming my questions above are just a result of my misunderstanding something)

But... Xen also wants to support non-server and therefore non-EFI systems i.e. u-boot. We also want to support v7 where EFI is not a given even for servers AIUI.

Quoted 3 lines

Multiboot only makes sense to me when you need to rely on firmware or something else out of your control to load the images in a particular way.

I think in some cases do end up needing to rely on that though.

Ian.

Rob Herring

2013-09-14 01:40 UTC #18 lore raw

Quoted 17 lines · Grant Likely

Quoted 16 lines · Rob Herring
Quoted 5 lines
For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)
And being machine specific is a PITA. flash-kernel is certainly not something we want to expand on. There is not much love for boot.scr either. There is work to address what are not really machine differences, but largely vendor u-boot differences:
http://www.mail-archive.com/u-boot at lists.denx.de/msg119025.html
One option for u-boot which already supports syslinux style menu files is to adopt the syslinux multiboot parsing support:

Quoted 6 lines

Quoted 1 line
http://www.syslinux.org/wiki/index.php/Doc/mboot
Even building it into U-Boot is problematic because it leaves older machines out in the cold. Leif's port of Grub to U-Boot is far more interesting since the distro can now be in control of the code that loads the images and jumps into the kernel/hypervisor.

Considering there is no distro support for grub on ARM yet, it may be more interesting in the long run, but it is not for the short term. So there needs to be something that is supportable on both u-boot and grub (or any other bootloader).

Quoted 7 lines

Quoted 6 lines
We need to back-up and consider what this looks like in the end for all the pieces and get input from folks on grub, UEFI, and armv8. The UEFI answer may be this is a grub problem. For armv8, this proposal does match up well as the kernel boot interface for v8 is DT. Despite some claims, ACPI will not completely replace DT because of this.

Quoted 8 lines

Yes, for UEFI it is absolutely an OS loader problem. UEFI provides an API and runtime environment. Grub is in general moving towards a boot menu system and a tool for loading images. Actual booting however should be done by a separate OS loader application. For Linux, this will be an in-kernel UEFI Stub. For Xen I would recommend taking the Linux EFI stub code and doing the same thing. There really isn't a need for a multiboot spec when you can rely on a runtime execution environment for setting things up exactly as you want them.

You've lost me as well. How do you see the flow working with UEFI for a user running bare metal OS, installing Xen, and rebooting running Xen.

Rob

Grant Likely

2013-09-15 12:23 UTC #19 lore raw

Quoted 25 lines · Rob Herring

Quoted 17 lines · Grant Likely
Quoted 16 lines · Rob Herring
Quoted 5 lines
For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)
And being machine specific is a PITA. flash-kernel is certainly not something we want to expand on. There is not much love for boot.scr either. There is work to address what are not really machine differences, but largely vendor u-boot differences:
http://www.mail-archive.com/u-boot at lists.denx.de/msg119025.html
One option for u-boot which already supports syslinux style menu files is to adopt the syslinux multiboot parsing support:
Quoted 6 lines
Quoted 1 line
http://www.syslinux.org/wiki/index.php/Doc/mboot
Even building it into U-Boot is problematic because it leaves older machines out in the cold. Leif's port of Grub to U-Boot is far more interesting since the distro can now be in control of the code that loads the images and jumps into the kernel/hypervisor.

Quoted 4 lines

Considering there is no distro support for grub on ARM yet, it may be more interesting in the long run, but it is not for the short term. So there needs to be something that is supportable on both u-boot and grub (or any other bootloader)

True.

Quoted 17 lines

Quoted 8 lines
Quoted 7 lines
We need to back-up and consider what this looks like in the end for all the pieces and get input from folks on grub, UEFI, and armv8. The UEFI answer may be this is a grub problem. For armv8, this proposal does match up well as the kernel boot interface for v8 is DT. Despite some claims, ACPI will not completely replace DT because of this.
Quoted 8 lines
Yes, for UEFI it is absolutely an OS loader problem. UEFI provides an API and runtime environment. Grub is in general moving towards a boot menu system and a tool for loading images. Actual booting however should be done by a separate OS loader application. For Linux, this will be an in-kernel UEFI Stub. For Xen I would recommend taking the Linux EFI stub code and doing the same thing. There really isn't a need for a multiboot spec when you can rely on a runtime execution environment for setting things up exactly as you want them.

Quoted 3 lines

You've lost me as well. How do you see the flow working with UEFI for a user running bare metal OS, installing Xen, and rebooting running Xen.

I see I've caused confusion. I wasn't disputing the proposed binding, but rather stating that none of the Xen or Linux specific boot code should be built directly into UEFI. It should be contained in a separate UEFI OS Loader applications (GRUB + stub).

As for the question above, I see the Xen installer adding stanzas to the Grub configuration to boo the xen kernel instead of the Linux kernel. Here there are two options (the Xen folks can decide which is best)
a) The Xen kernel has an EFI stub which uses a "linux=" argument to specify the Linux kernel image to load.
b) GRUB is Xen-aware and adds the extra images to the dtb.

In either case, the Xen loader is completely separate from UEFI and therefore under our own control.

Rob Herring

2013-09-14 01:40 UTC #20 lore raw

Quoted 17 lines · Grant Likely

Quoted 16 lines · Rob Herring
Quoted 5 lines
For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)
And being machine specific is a PITA. flash-kernel is certainly not something we want to expand on. There is not much love for boot.scr either. There is work to address what are not really machine differences, but largely vendor u-boot differences:
http://www.mail-archive.com/[email protected]/msg119025.html
One option for u-boot which already supports syslinux style menu files is to adopt the syslinux multiboot parsing support:

Quoted 6 lines

Quoted 1 line
http://www.syslinux.org/wiki/index.php/Doc/mboot
Even building it into U-Boot is problematic because it leaves older machines out in the cold. Leif's port of Grub to U-Boot is far more interesting since the distro can now be in control of the code that loads the images and jumps into the kernel/hypervisor.

Quoted 7 lines

Quoted 6 lines
We need to back-up and consider what this looks like in the end for all the pieces and get input from folks on grub, UEFI, and armv8. The UEFI answer may be this is a grub problem. For armv8, this proposal does match up well as the kernel boot interface for v8 is DT. Despite some claims, ACPI will not completely replace DT because of this.

Quoted 8 lines

Yes, for UEFI it is absolutely an OS loader problem. UEFI provides an API and runtime environment. Grub is in general moving towards a boot menu system and a tool for loading images. Actual booting however should be done by a separate OS loader application. For Linux, this will be an in-kernel UEFI Stub. For Xen I would recommend taking the Linux EFI stub code and doing the same thing. There really isn't a need for a multiboot spec when you can rely on a runtime execution environment for setting things up exactly as you want them.

You've lost me as well. How do you see the flow working with UEFI for a user running bare metal OS, installing Xen, and rebooting running Xen.

Rob

Grant Likely

2013-09-13 10:13 UTC #21 lore raw

Quoted 11 lines · Rob Herring

Quoted 5 lines
For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)
And being machine specific is a PITA. flash-kernel is certainly not something we want to expand on. There is not much love for boot.scr either. There is work to address what are not really machine differences, but largely vendor u-boot differences:

Quoted 6 lines

http://www.mail-archive.com/[email protected]/msg119025.html
One option for u-boot which already supports syslinux style menu files is to adopt the syslinux multiboot parsing support:
http://www.syslinux.org/wiki/index.php/Doc/mboot

Quoted 5 lines

We need to back-up and consider what this looks like in the end for all the pieces and get input from folks on grub, UEFI, and armv8. The UEFI answer may be this is a grub problem. For armv8, this proposal does match up well as the kernel boot interface for v8 is DT. Despite some claims, ACPI will not completely replace DT because of this.

Multiboot only makes sense to me when you need to rely on firmware or something else out of your control to load the images in a particular way.

Dennis Gilmore

2013-09-14 04:06 UTC #22 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

El Wed, 4 Sep 2013 11:41:01 -0500
Rob Herring <[email protected]> escribi?:

Quoted 82 lines

Adding Dennis for a distro perspective.
Quoted 55 lines · Ian Campbell
Quoted 14 lines · Rob Herring
Quoted 12 lines · Andre Przywara
Hi,
a normal Linux kernel currently supports reading the start and end address of a single binary blob via the FDT's /chosen node. This will be interpreted as the location of an initial RAM disk.
The Xen hypervisor itself is a kernel, but needs up to _two_ binaries for proper operation: a Dom0 Linux kernel and it's associated initrd. On x86 this is solved via the multiboot protocol used by the Grub bootloader, which supports to pass an arbitrary number of binary modules to any kernel.
Quoted 6 lines
Quoted 2 lines
Since in the ARM world we have the versatile device tree, we don't need to implement the mulitboot protocol.
But surely there would be some advantage of reuse by using the multi-boot protocol since Xen, grub, and OS tools already support it for x86.
Multiboot is pretty x86 specific (although MB2 has a MIPS port) and covers more stuff than we strictly require (e.g. on x86 it has requirements around which processor mode you enter in, has paging enabled etc).
Quoted 10 lines
Quoted 9 lines
So I'd like to propose a new binding which denotes binary modules a kernel can use at it's own discretion.
The need is triggered by the Xen hypervisor (which already uses a very similar scheme), but the approach is deliberately chosen to be as generic as possible to allow future uses (like passing firmware blobs for devices or the like).
Credits for this go to Ian Campbell, who started something very similar [1] for the Xen hypervisor. The intention of this proposal is to make this generic and publicly documented.
Quoted 2 lines
Can you describe how you see the boot flow working starting with OS installer writes kernel, initrd, xen and ??? to disk.
Kernel and initrd are written to /boot in the usual way (probably from kernel.deb or whatever). Xen would also normally come from a distro package (also in /boot).
Quoted 1 line
How does the bootloader know what to load?
It's in the bootloader config, e.g. boot.scr or grub.cfg, which are either hand written or produced by the distros tooling.
grub on ARM could consume the same stanzas as are used by grub on x86 to boot Xen (which are produced by update-grub):
        echo    'Loading Xen 4.1-amd64 ...'
        multiboot       /xen-4.1-amd64.gz placeholder
        echo    'Loading Linux 3.10-2-amd64 ...'
        module  /vmlinuz-3.10-2-amd64 placeholder
Quoted 8 lines
root=/dev/mapper/disks-root ro resume=/dev/mapper/disks-swap quiet
echo 'Loading initial ramdisk ...'
module /initrd.img-3.10-2-amd64
Since there is no multiboot on ARM (and never will be) this is safe.
If multiboot ever does come to ARM it will necessarily be multiboot2 which uses a different keyword.
Right, this is just a text file with a list of binaries. It is not really the multiboot spec. There is no reason for this part to be different for grub on ARM. There is a big advantage to reusing the distro side tooling. If there isn't really much reuse on the bootloader side, then I'm fine with a different bootloader to Xen interface. I would like to hear that from folks working on grub though.
Quoted 5 lines
For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)

Quoted 6 lines

And being machine specific is a PITA. flash-kernel is certainly not something we want to expand on. There is not much love for boot.scr either. There is work to address what are not really machine differences, but largely vendor u-boot differences:
http://www.mail-archive.com/u-boot at lists.denx.de/msg119025.html

u-boot still has quite a bit of work that needs to be done to make things standard across the board. I hope we will get there. but it will require vendors to update their u-boot builds. even for grub to be a viable option u-boot needs updated.

Quoted 4 lines

One option for u-boot which already supports syslinux style menu files is to adopt the syslinux multiboot parsing support:
http://www.syslinux.org/wiki/index.php/Doc/mboot

I would like to have u-boot support more of syslinux than it does today.

Quoted 8 lines

We need to back-up and consider what this looks like in the end for all the pieces and get input from folks on grub, UEFI, and armv8. The UEFI answer may be this is a grub problem. For armv8, this proposal does match up well as the kernel boot interface for v8 is DT. Despite some claims, ACPI will not completely replace DT because of this.
Rob

grub on arm still needs a bit of work. as does u-boot. ultimately having things look the same between x86 and arm for the user is the best option. so if u-boot supported syslinux's mboot spec and grub supported the same syntax in both cases, regardless of if the implementation varied wildly, the API for lack of a better word should be consistent.

Dennis
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.21 (GNU/Linux)

iEYEARECAAYFAlIz4NMACgkQkSxm47BaWffX8QCgtSFFHfgHongecz9AESG43RsW /48An131lBoSmjwcJOlWNkQUxI3LXC5f
=fQqh
-----END PGP SIGNATURE-----

Dennis Gilmore

2013-09-14 04:06 UTC #23 lore raw

http://www.mail-archive.com/[email protected]/msg119025.html

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 82 lines · El Wed, 4 Sep 2013 11:41:01 -0500 Rob Herring <[email protected]> escribió:

Adding Dennis for a distro perspective.
Quoted 55 lines · Ian Campbell
Quoted 14 lines · Rob Herring
Quoted 12 lines · Andre Przywara
Hi,
a normal Linux kernel currently supports reading the start and end address of a single binary blob via the FDT's /chosen node. This will be interpreted as the location of an initial RAM disk.
The Xen hypervisor itself is a kernel, but needs up to _two_ binaries for proper operation: a Dom0 Linux kernel and it's associated initrd. On x86 this is solved via the multiboot protocol used by the Grub bootloader, which supports to pass an arbitrary number of binary modules to any kernel.
Quoted 6 lines
Quoted 2 lines
Since in the ARM world we have the versatile device tree, we don't need to implement the mulitboot protocol.
But surely there would be some advantage of reuse by using the multi-boot protocol since Xen, grub, and OS tools already support it for x86.
Multiboot is pretty x86 specific (although MB2 has a MIPS port) and covers more stuff than we strictly require (e.g. on x86 it has requirements around which processor mode you enter in, has paging enabled etc).
Quoted 10 lines
Quoted 9 lines
So I'd like to propose a new binding which denotes binary modules a kernel can use at it's own discretion.
The need is triggered by the Xen hypervisor (which already uses a very similar scheme), but the approach is deliberately chosen to be as generic as possible to allow future uses (like passing firmware blobs for devices or the like).
Credits for this go to Ian Campbell, who started something very similar [1] for the Xen hypervisor. The intention of this proposal is to make this generic and publicly documented.
Quoted 2 lines
Can you describe how you see the boot flow working starting with OS installer writes kernel, initrd, xen and ??? to disk.
Kernel and initrd are written to /boot in the usual way (probably from kernel.deb or whatever). Xen would also normally come from a distro package (also in /boot).
Quoted 1 line
How does the bootloader know what to load?
It's in the bootloader config, e.g. boot.scr or grub.cfg, which are either hand written or produced by the distros tooling.
grub on ARM could consume the same stanzas as are used by grub on x86 to boot Xen (which are produced by update-grub):
        echo    'Loading Xen 4.1-amd64 ...'
        multiboot       /xen-4.1-amd64.gz placeholder
        echo    'Loading Linux 3.10-2-amd64 ...'
        module  /vmlinuz-3.10-2-amd64 placeholder
Quoted 8 lines
root=/dev/mapper/disks-root ro resume=/dev/mapper/disks-swap quiet
echo 'Loading initial ramdisk ...'
module /initrd.img-3.10-2-amd64
Since there is no multiboot on ARM (and never will be) this is safe.
If multiboot ever does come to ARM it will necessarily be multiboot2 which uses a different keyword.
Right, this is just a text file with a list of binaries. It is not really the multiboot spec. There is no reason for this part to be different for grub on ARM. There is a big advantage to reusing the distro side tooling. If there isn't really much reuse on the bootloader side, then I'm fine with a different bootloader to Xen interface. I would like to hear that from folks working on grub though.
Quoted 5 lines
For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)

Quoted 6 lines

And being machine specific is a PITA. flash-kernel is certainly not something we want to expand on. There is not much love for boot.scr either. There is work to address what are not really machine differences, but largely vendor u-boot differences:
http://www.mail-archive.com/[email protected]/msg119025.html

Quoted 4 lines

One option for u-boot which already supports syslinux style menu files is to adopt the syslinux multiboot parsing support:
http://www.syslinux.org/wiki/index.php/Doc/mboot

I would like to have u-boot support more of syslinux than it does today.

Quoted 8 lines

We need to back-up and consider what this looks like in the end for all the pieces and get input from folks on grub, UEFI, and armv8. The UEFI answer may be this is a grub problem. For armv8, this proposal does match up well as the kernel boot interface for v8 is DT. Despite some claims, ACPI will not completely replace DT because of this.
Rob

Dennis
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.21 (GNU/Linux)

iEYEARECAAYFAlIz4NMACgkQkSxm47BaWffX8QCgtSFFHfgHongecz9AESG43RsW /48An131lBoSmjwcJOlWNkQUxI3LXC5f
=fQqh
-----END PGP SIGNATURE-----

Rob Herring

2013-09-04 16:41 UTC #24 lore raw

In reply to Ian Campbell

Adding Dennis for a distro perspective.

Quoted 58 lines · Ian Campbell

Quoted 14 lines · Rob Herring
Quoted 12 lines · Andre Przywara
Hi,
a normal Linux kernel currently supports reading the start and end address of a single binary blob via the FDT's /chosen node.
This will be interpreted as the location of an initial RAM disk.
The Xen hypervisor itself is a kernel, but needs up to _two_ binaries for proper operation: a Dom0 Linux kernel and it's associated initrd. On x86 this is solved via the multiboot protocol used by the Grub bootloader, which supports to pass an arbitrary number of binary modules to any kernel.
Quoted 6 lines
Quoted 2 lines
Since in the ARM world we have the versatile device tree, we don't need to implement the mulitboot protocol.
But surely there would be some advantage of reuse by using the multi-boot protocol since Xen, grub, and OS tools already support it for x86.
Multiboot is pretty x86 specific (although MB2 has a MIPS port) and covers more stuff than we strictly require (e.g. on x86 it has requirements around which processor mode you enter in, has paging enabled etc).
Quoted 10 lines
Quoted 9 lines
So I'd like to propose a new binding which denotes binary modules a kernel can use at it's own discretion.
The need is triggered by the Xen hypervisor (which already uses a very similar scheme), but the approach is deliberately chosen to be as generic as possible to allow future uses (like passing firmware blobs for devices or the like).
Credits for this go to Ian Campbell, who started something very similar [1] for the Xen hypervisor. The intention of this proposal is to make this generic and publicly documented.
Quoted 2 lines
Can you describe how you see the boot flow working starting with OS installer writes kernel, initrd, xen and ??? to disk.
Kernel and initrd are written to /boot in the usual way (probably from kernel.deb or whatever). Xen would also normally come from a distro package (also in /boot).
Quoted 1 line
How does the bootloader know what to load?
It's in the bootloader config, e.g. boot.scr or grub.cfg, which are either hand written or produced by the distros tooling.
grub on ARM could consume the same stanzas as are used by grub on x86 to
boot Xen (which are produced by update-grub):
echo 'Loading Xen 4.1-amd64 ...'
multiboot /xen-4.1-amd64.gz placeholder
echo 'Loading Linux 3.10-2-amd64 ...'
module /vmlinuz-3.10-2-amd64 placeholder root=/dev/mapper/disks-root ro resume=/dev/mapper/disks-swap quiet
echo 'Loading initial ramdisk ...'
module /initrd.img-3.10-2-amd64

Quoted 4 lines

Since there is no multiboot on ARM (and never will be) this is safe.
If multiboot ever does come to ARM it will necessarily be multiboot2 which uses a different keyword.

Quoted 5 lines

For u-boot Andre has proposed some syntactic sugar over the "fdt" command to make boot.scr more trivial to use. We would of course need to implement support for using it in the relevant distro tools (but they tend to be very distro/machine specific already, e.g. Debian's flash-kernel)

One option for u-boot which already supports syslinux style menu files is to adopt the syslinux multiboot parsing support:

http://www.syslinux.org/wiki/index.php/Doc/mboot

Rob

Ian Campbell

2013-09-04 08:44 UTC #25 lore raw

Quoted 14 lines · Rob Herring

Quoted 12 lines · Andre Przywara
Hi,
a normal Linux kernel currently supports reading the start and end address of a single binary blob via the FDT's /chosen node.
This will be interpreted as the location of an initial RAM disk.
The Xen hypervisor itself is a kernel, but needs up to _two_ binaries for proper operation: a Dom0 Linux kernel and it's associated initrd. On x86 this is solved via the multiboot protocol used by the Grub bootloader, which supports to pass an arbitrary number of binary modules to any kernel.

Quoted 6 lines

Quoted 2 lines
Since in the ARM world we have the versatile device tree, we don't need to implement the mulitboot protocol.
But surely there would be some advantage of reuse by using the multi-boot protocol since Xen, grub, and OS tools already support it for x86.

Quoted 10 lines

Quoted 9 lines
So I'd like to propose a new binding which denotes binary modules a kernel can use at it's own discretion.
The need is triggered by the Xen hypervisor (which already uses a very similar scheme), but the approach is deliberately chosen to be as generic as possible to allow future uses (like passing firmware blobs for devices or the like).
Credits for this go to Ian Campbell, who started something very similar [1] for the Xen hypervisor. The intention of this proposal is to make this generic and publicly documented.

Quoted 2 lines

Can you describe how you see the boot flow working starting with OS installer writes kernel, initrd, xen and ??? to disk.

Kernel and initrd are written to /boot in the usual way (probably from kernel.deb or whatever). Xen would also normally come from a distro package (also in /boot).

Quoted 1 line

How does the bootloader know what to load?

It's in the bootloader config, e.g. boot.scr or grub.cfg, which are either hand written or produced by the distros tooling.

Since there is no multiboot on ARM (and never will be) this is safe.

If multiboot ever does come to ARM it will necessarily be multiboot2 which uses a different keyword.

Quoted 2 lines

The OS may not have access to the dtb,
so this has to be described to the bootloader as well.

Right. The proposal is not for the OS to frob the DTB to support this protocol but to configure the bootloader to do it.

Ian.

Rob Herring

2013-09-03 22:00 UTC #26 lore raw

Quoted 6 lines · Andre Przywara

Hi,
a normal Linux kernel currently supports reading the start and end address of a single binary blob via the FDT's /chosen node.
This will be interpreted as the location of an initial RAM disk.

Quoted 8 lines

The Xen hypervisor itself is a kernel, but needs up to _two_ binaries for proper operation: a Dom0 Linux kernel and it's associated initrd. On x86 this is solved via the multiboot protocol used by the Grub bootloader, which supports to pass an arbitrary number of binary modules to any kernel.
Since in the ARM world we have the versatile device tree, we don't need to implement the mulitboot protocol.

But surely there would be some advantage of reuse by using the multi-boot protocol since Xen, grub, and OS tools already support it for x86.

Quoted 9 lines

So I'd like to propose a new binding which denotes binary modules a kernel can use at it's own discretion.
The need is triggered by the Xen hypervisor (which already uses a very similar scheme), but the approach is deliberately chosen to be as generic as possible to allow future uses (like passing firmware blobs for devices or the like).
Credits for this go to Ian Campbell, who started something very similar [1] for the Xen hypervisor. The intention of this proposal is to make this generic and publicly documented.

Quoted 31 lines

Looking forward to any comments!
Thanks,
Andre.
[1]
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/misc/arm/device-tree/booting.txt;h=94cd3f18a4e1317a35e1255bf5c6e1e091001d1a;hb=HEAD
----------------------------
* Multiple boot modules device tree bindings
Boot loaders wanting to pass multiple additional binaries to a kernel shall add a node "module" for each binary blob under the /chosen node with the following properties:
- compatible:
    compatible = "boot,module";
A bootloader may add names to more specifically describe the module, e.g. Xen may use "xen,dom0-kernel" or "xen,dom0-ramdisk". If possible a kernel should be able to use modules even without a descriptive naming, by enumerating them in order and using hard-coded meanings for each module (e.g. first is kernel, second is initrd).
- reg: specifies the base physical address and size of a region in memory where the bootloader loaded the respective binary data to.
- bootargs:
An optional property describing arguments to use for this module. Could be a command line or configuration data.
Example:

Quoted 14 lines

/chosen {
    #size-cells = <0x1>;
    #address-cells = <0x1>;
    module@0 {
        compatible = "xen,linux-zimage", "xen,multiboot-module",
"boot,module";
        reg = <0x80000000 0x003dcff8>;
        bootargs = "console=hvc0 earlyprintk ro root=/dev/sda1 nosmp";
    };
    module@1 {
        compatible = "xen,linux-initrd", "xen,multiboot-module",
"boot,module";
        reg = <0x08000000 0x00123456>;
    };

bootargs = "xen args --- linux args";
xen,linux-image = <start size>;

So, is having a more generic solution really needed?

Rob

Grant Likely

2013-09-15 12:37 UTC #27 lore raw

Quoted 55 lines · Andre Przywara

Hi,
a normal Linux kernel currently supports reading the start and end address of a single binary blob via the FDT's /chosen node. This will be interpreted as the location of an initial RAM disk.
The Xen hypervisor itself is a kernel, but needs up to _two_ binaries for proper operation: a Dom0 Linux kernel and it's associated initrd. On x86 this is solved via the multiboot protocol used by the Grub bootloader, which supports to pass an arbitrary number of binary modules to any kernel.
Since in the ARM world we have the versatile device tree, we don't need to implement the mulitboot protocol.
So I'd like to propose a new binding which denotes binary modules a kernel can use at it's own discretion.
The need is triggered by the Xen hypervisor (which already uses a very similar scheme), but the approach is deliberately chosen to be as generic as possible to allow future uses (like passing firmware blobs for devices or the like).
Credits for this go to Ian Campbell, who started something very similar
[1] for the Xen hypervisor. The intention of this proposal is to make this generic and publicly documented.
Looking forward to any comments!
Thanks,
Andre.
[1]
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/misc/arm/device-tree/booting.txt;h=94cd3f18a4e1317a35e1255bf5c6e1e091001d1a;hb=HEAD
----------------------------
* Multiple boot modules device tree bindings
Boot loaders wanting to pass multiple additional binaries to a kernel shall add a node "module" for each binary blob under the /chosen node with the following properties:
- compatible:
     compatible = "boot,module";
A bootloader may add names to more specifically describe the module, e.g. Xen may use "xen,dom0-kernel" or "xen,dom0-ramdisk". If possible a kernel should be able to use modules even without a descriptive naming, by enumerating them in order and using hard-coded meanings for each module (e.g. first is kernel, second is initrd).
- reg: specifies the base physical address and size of a region in memory where the bootloader loaded the respective binary data to.
- bootargs:
An optional property describing arguments to use for this module. Could be a command line or configuration data.
Example:

Quoted 14 lines

/chosen {
     #size-cells = <0x1>;
     #address-cells = <0x1>;
     module at 0 {
         compatible = "xen,linux-zimage", "xen,multiboot-module", 
"boot,module";
         reg = <0x80000000 0x003dcff8>;
         bootargs = "console=hvc0 earlyprintk ro root=/dev/sda1 nosmp";
     };
     module at 1 {
         compatible = "xen,linux-initrd", "xen,multiboot-module", 
"boot,module";
         reg = <0x08000000 0x00123456>;
     };

The moment you pull in a 'reg' property, your required to property parse #size-cells and #address-cells and also 'ranges' which is missing from the above. I don't necessarily object to that, but it can be a lot to ask for during early boot. It also looks really weird in the chosen node. For boot time artifacts like this, I'd rather see a duplication of the linux,initrd- pattern. Something like:

/chosen {
	xen,linux-zimage-start = <0x80000000>;
	xen,linux-zimage-end = <0x803dcff8>;
	linux,initrd-start = <0x8000000>;
	linux,initrd-end = <0x8123456>;
}

If have two reason for this; consistency and simplicity. Consistent because it matches what Linux already uses, and really easy to parse without mucking with ranges or #address/size-cells. The assumption is that during early boot the system is dealing with a 1:1 mapped physical address and there isn't any bus translations that need to be processed.

Question: what is the need for the xen,linux-initrd node? Why can't the existing "linux,initrd-*" properties continue to be used?

Andre Przywara

author

2013-09-16 17:01 UTC #28 lore raw

Quoted 55 lines · Grant Likely

Quoted 54 lines · Andre Przywara
Hi,
a normal Linux kernel currently supports reading the start and end address of a single binary blob via the FDT's /chosen node. This will be interpreted as the location of an initial RAM disk.
The Xen hypervisor itself is a kernel, but needs up to _two_ binaries for proper operation: a Dom0 Linux kernel and it's associated initrd. On x86 this is solved via the multiboot protocol used by the Grub bootloader, which supports to pass an arbitrary number of binary modules to any kernel.
Since in the ARM world we have the versatile device tree, we don't need to implement the mulitboot protocol.
So I'd like to propose a new binding which denotes binary modules a kernel can use at it's own discretion.
The need is triggered by the Xen hypervisor (which already uses a very similar scheme), but the approach is deliberately chosen to be as generic as possible to allow future uses (like passing firmware blobs for devices or the like).
Credits for this go to Ian Campbell, who started something very similar
[1] for the Xen hypervisor. The intention of this proposal is to make this generic and publicly documented.
Looking forward to any comments!
Thanks,
Andre.
[1]
http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/misc/arm/device-tree/booting.txt;h=94cd3f18a4e1317a35e1255bf5c6e1e091001d1a;hb=HEAD
----------------------------
* Multiple boot modules device tree bindings
Boot loaders wanting to pass multiple additional binaries to a kernel shall add a node "module" for each binary blob under the /chosen node with the following properties:
- compatible:
compatible = "boot,module";
A bootloader may add names to more specifically describe the module, e.g. Xen may use "xen,dom0-kernel" or "xen,dom0-ramdisk". If possible a kernel should be able to use modules even without a descriptive naming, by enumerating them in order and using hard-coded meanings for each module (e.g. first is kernel, second is initrd).
- reg: specifies the base physical address and size of a region in memory where the bootloader loaded the respective binary data to.
- bootargs:
An optional property describing arguments to use for this module. Could be a command line or configuration data.

Quoted 15 lines

Example:

/chosen {
      #size-cells = <0x1>;
      #address-cells = <0x1>;
      module at 0 {
          compatible = "xen,linux-zimage", "xen,multiboot-module",
"boot,module";
          reg = <0x80000000 0x003dcff8>;
          bootargs = "console=hvc0 earlyprintk ro root=/dev/sda1 nosmp";
      };
      module at 1 {
          compatible = "xen,linux-initrd", "xen,multiboot-module",
"boot,module";
          reg = <0x08000000 0x00123456>;
      };

Grant,

thanks for looking at this. Appreciate your comments.

Quoted 3 lines

The moment you pull in a 'reg' property, your required to property parse #size-cells and #address-cells and also 'ranges' which is missing from the above.

You mean that early boot code has to read #size-cells and #address-cells and act accordingly when parsing the reg property? So just more code in early boot? But that would easily open the door for 64bit addresses for instance, right?

Quoted 2 lines

I don't necessarily object to that, but it can be a lot to ask for during early boot.

I don't think Xen in particular has a lot of problems with that. The DT implementation is quite capable (see Julien's latest updates).

Quoted 1 line

It also looks really weird in the chosen node.

TBH this initrd syntax looks a bit weird to me in general, not following the nice and well defined DT way (no compatible, separate properties for start and length, ...)

Quoted 8 lines

For boot time artifacts like this, I'd rather see a duplication of the linux,initrd- pattern. Something like:
/chosen {
	xen,linux-zimage-start = <0x80000000>;
	xen,linux-zimage-end = <0x803dcff8>;
	linux,initrd-start = <0x8000000>;
	linux,initrd-end = <0x8123456>;
}

To me that sounds like a hack to avoid more coding work.
The problem with that is that it would introduce a Xen specific property (actually two).
So from a user's perspective you want:

a) generic properties to support future OSes without much hassle (using the DT's compatible semantic)
b) specifying load address and length in the normal way
c) adding command lines
d) potentially adding more arbitrary properties
e) allowing an arbitrary number of modules

To just go ahead with another hard-coded property to make life easier for us kernel level developers doesn't sound very convincing to me. And especially supporting multiple modules would be very hackish then.

Quoted 2 lines

If have two reason for this; consistency and simplicity. Consistent because it matches what Linux already uses,

But this is actually not consistent with the rest of the DT architecture. Defining an own node for that looks like a perfect match to me in the DT world.

Quoted 4 lines

and really easy to parse
without mucking with ranges or #address/size-cells. The assumption is that during early boot the system is dealing with a 1:1 mapped physical address and there isn't any bus translations that need to be processed.

I see your point with early boot code constraints. But my idea was to create a flexible and generic solution, so that there will be a framework for bootloaders to specify blobs without needing updates every time someone comes up with a need for one. On x86 there was once such a need for microcode updates which couldn't be easily satisfied because Linux/x86 itself does not support the multiboot protocol.

Quoted 2 lines

Question: what is the need for the xen,linux-initrd node? Why can't the existing "linux,initrd-*" properties continue to be used?

The main driver for this was to avoid patching bootloaders just to boot Xen specifically. Rather we want to patch bootloaders to support some kind of "multiboot" protocol once and let Xen piggy-back on this. I think that is somewhat driven by experiences in the x86 world, were especially kernel people are reluctant to support something just for Xen's need.

Regards,
Andre.

Andre Przywara

author

2013-09-16 17:01 UTC #29 lore raw