arm64 boot requirements

In commit bd00cd5f8c8c3c282bb1e1eac6a6679a4f808091, the idmap_pg_dir and swapper_pg_dir where moved from before the kernel to after it.

The problem is that these symbols fall outside the range covered by the ELF file - outside of any section.

A bootloader which loads the kernel ELF file and dynamically determines where to place the DTB, may try place it after the kernel. We've just run into this problem and the DTB gets overwritten as soon as the pagetables are created.

I'd suggest that the kernel either:
A. document this boot requirement for where not to load a DTB

B. update the vmlinux.lds.S such that these symbols (and _end) are
properly covered by a section in the ELF, and thus preventing this issue.

Hi,

Quoted 10 lines · Carl van Schaik

In commit bd00cd5f8c8c3c282bb1e1eac6a6679a4f808091, the idmap_pg_dir and swapper_pg_dir where moved from before the kernel to after it.

The problem is that these symbols fall outside the range covered by the ELF file - outside of any section.

A bootloader which loads the kernel ELF file and dynamically determines where to place the DTB, may try place it after the kernel. We've just run into this problem and the DTB gets overwritten as soon as the pagetables are created.

We had similar issues with the BSS when booting Image files prior to this and commit a2c1d73b94ed49f5 ("arm64: Update the Image header"). Since then, the image_size field in the Image header tells you how much memory the kernel may clobber (including the BSS and page tables).

Prior to that, the page tables were below the kernel, and also not described in any ELF section.

Others booting the kernel vmlinux haven't reported similar issues, so I assume that either they are parsing the Image header, or getting lucky. Parsing the header is necessary to get the correct text offset, too...

Pratyush, Geoff, I understood you were loading the kernel vmlinux for kexec. Do you parse the Image header to figure out where to place things?

Quoted 2 lines

I'd suggest that the kernel either:
A. document this boot requirement for where not to load a DTB

Do you have any particular suggestion?

We already describe the Image footprint (including BSS and page tables) by the image_size in the Image header, which is sufficient. The size of the BSS and page tables is effectively unbound, so we can't define some upper bound that will always be true.

The documentation is written on the assumption that an Image file is being used rather than a vmlinux. Perhaps that is something to consider.

Quoted 3 lines

B. update the vmlinux.lds.S such that these symbols (and _end) are
properly covered by a section in the ELF, and thus preventing this issue.

I'm worried that this only solves this one case, and it means that there are two (potentially conflicting) sources of information that a bootloader might be using -- the ELF or the Image header. I don't want to have to duplicate text_offset and so on, which implies that parsing the Image header is necessary anyway.

That's something we can discuss if you send a patch (inline rather than attached).

Pratyush, Geoff, I understood you were loading the kernel vmlinux for kexec. Do you parse the Image header to figure out where to place things?

Quoted 13 lines · Mark Rutland

Hi,

Quoted 10 lines · Carl van Schaik

In commit bd00cd5f8c8c3c282bb1e1eac6a6679a4f808091, the idmap_pg_dir and swapper_pg_dir where moved from before the kernel to after it.

The problem is that these symbols fall outside the range covered by the ELF file - outside of any section.

A bootloader which loads the kernel ELF file and dynamically determines where to place the DTB, may try place it after the kernel. We've just run into this problem and the DTB gets overwritten as soon as the pagetables are created.

Could you explain why you are using the ELF file and not the binary image file? This is not future proof: currently, the Image is a straight binary objcopy of vmlinux, but that is not guaranteed to remain that way. Things like KASLR may require post build steps that mangle vmlinux or Image afterwards.

Quoted 34 lines

We had similar issues with the BSS when booting Image files prior to this and commit a2c1d73b94ed49f5 ("arm64: Update the Image header"). Since then, the image_size field in the Image header tells you how much memory the kernel may clobber (including the BSS and page tables).

Prior to that, the page tables were below the kernel, and also not described in any ELF section.

Others booting the kernel vmlinux haven't reported similar issues, so I assume that either they are parsing the Image header, or getting lucky. Parsing the header is necessary to get the correct text offset, too...

Pratyush, Geoff, I understood you were loading the kernel vmlinux for kexec. Do you parse the Image header to figure out where to place things?

Quoted 2 lines

I'd suggest that the kernel either:
A. document this boot requirement for where not to load a DTB

Do you have any particular suggestion?

We already describe the Image footprint (including BSS and page tables) by the image_size in the Image header, which is sufficient. The size of the BSS and page tables is effectively unbound, so we can't define some upper bound that will always be true.

The documentation is written on the assumption that an Image file is being used rather than a vmlinux. Perhaps that is something to consider.

Quoted 3 lines

B. update the vmlinux.lds.S such that these symbols (and _end) are
properly covered by a section in the ELF, and thus preventing this issue.

Quoted 9 lines

I'm worried that this only solves this one case, and it means that there are two (potentially conflicting) sources of information that a bootloader might be using -- the ELF or the Image header. I don't want to have to duplicate text_offset and so on, which implies that parsing the Image header is necessary anyway.

That's something we can discuss if you send a patch (inline rather than attached).

I think updating the linker script to put the page tables into a .pgdir section is reasonable, since it is part of the static footprint of the kernel.

However, I strongly feel that the Image header should remain the authoritative source of information regarding the nature (big/little endian, page size) and the static footprint of the Image.

Quoted 3 lines · Ard Biesheuvel

However, I strongly feel that the Image header should remain the authoritative source of information regarding the nature (big/little endian, page size) and the static footprint of the Image.

I find readelf -a | less quite handy. Is there anything comparable for the AArch64 Image format?

Please forgive my ignorance, but is the EFI stub another possible source for sort of information?

Quoted 4 lines · Mark Rutland

Hi,

Quoted 1 line · Carl van Schaik

In commit bd00cd5f8c8c3c282bb1e1eac6a6679a4f808091, the

idmap_pg_dir

Quoted 9 lines

Quoted 9 lines

and swapper_pg_dir where moved from before the kernel to after it.

The problem is that these symbols fall outside the range covered by the ELF file - outside of any section.

A bootloader which loads the kernel ELF file and dynamically determines where to place the DTB, may try place it after the kernel. We've just run into this problem and the DTB gets overwritten as soon as the pagetables are created.

Could you explain why you are using the ELF file and not the binary image file?
This is not future proof: currently, the Image is a straight binary objcopy of vmlinux, but that is not guaranteed to remain that way. Things like KASLR may require post build steps that mangle vmlinux or Image afterwards.

Quoted 40 lines

We had similar issues with the BSS when booting Image files prior to this and commit a2c1d73b94ed49f5 ("arm64: Update the Image header"). Since then, the image_size field in the Image header tells you how much memory the kernel may clobber (including the BSS and page tables).

Prior to that, the page tables were below the kernel, and also not described in any ELF section.

Others booting the kernel vmlinux haven't reported similar issues, so I assume that either they are parsing the Image header, or getting lucky. Parsing the header is necessary to get the correct text offset, too...

Pratyush, Geoff, I understood you were loading the kernel vmlinux for kexec. Do you parse the Image header to figure out where to place things?

Quoted 2 lines

I'd suggest that the kernel either:
A. document this boot requirement for where not to load a DTB

Do you have any particular suggestion?

We already describe the Image footprint (including BSS and page tables) by the image_size in the Image header, which is sufficient. The size of the BSS and page tables is effectively unbound, so we can't define some upper bound that will always be true.

The documentation is written on the assumption that an Image file is being used rather than a vmlinux. Perhaps that is something to consider.

Quoted 3 lines

B. update the vmlinux.lds.S such that these symbols (and _end) are
properly covered by a section in the ELF, and thus preventing this issue.

I'm worried that this only solves this one case, and it means that there are two (potentially conflicting) sources of information that a bootloader might be using -- the ELF or the Image header. I don't want to have to duplicate text_offset and so on, which implies that parsing the Image header is necessary anyway.

Quoted 3 lines

That's something we can discuss if you send a patch (inline rather than attached).

I think updating the linker script to put the page tables into a .pgdir section is reasonable, since it is part of the static footprint of the kernel.

However, I strongly feel that the Image header should remain the authoritative source of information regarding the nature (big/little endian, page size) and the static footprint of the Image .

Pratyush, Geoff, I understood you were loading the kernel vmlinux for kexec. Do you parse the Image header to figure out where to place things?

Hi,

On 1 December 2015 Ard Biesheuvel <[email protected]> wote:

Quoted 17 lines

Quoted 4 lines · Mark Rutland

Hi,

Quoted 1 line · Carl van Schaik

In commit bd00cd5f8c8c3c282bb1e1eac6a6679a4f808091, the

idmap_pg_dir

Quoted 9 lines

Quoted 9 lines

and swapper_pg_dir where moved from before the kernel to after it.

The problem is that these symbols fall outside the range covered by the ELF file - outside of any section.

A bootloader which loads the kernel ELF file and dynamically determines where to place the DTB, may try place it after the kernel. We've just run into this problem and the DTB gets overwritten as soon as the pagetables are created.

Quoted 6 lines

Could you explain why you are using the ELF file and not the binary image file?
This is not future proof: currently, the Image is a straight binary objcopy of vmlinux, but that is not guaranteed to remain that way. Things like KASLR may require post build steps that mangle vmlinux or Image afterwards.

The reason we've been using ELF files is mostly to do with legacy virtualization related reasons in our systems, we used to patch symbols in the ELFs for example pre device-tree. However, since it hadn't caused problems until now we had continued to use it. We haven't yet added Aarch64 Linux boot image header parsing but it should be trivial.

The other area we are looking into is optimized multi-VM static boot images by constructing hypervisor-bundle images containing de-duplicated Linux sections, allowing an ELF bootloader to populate multiple Linux VMs from a smaller boot image - resulting in faster boot.

Quoted 27 lines

Quoted 26 lines

Quoted 2 lines

I'd suggest that the kernel either:
A. document this boot requirement for where not to load a DTB

Do you have any particular suggestion?

We already describe the Image footprint (including BSS and page tables) by the image_size in the Image header, which is sufficient. The size of the BSS and page tables is effectively unbound, so we can't define some upper bound that will always be true.

The documentation is written on the assumption that an Image file is being used rather than a vmlinux. Perhaps that is something to consider.

Quoted 3 lines

B. update the vmlinux.lds.S such that these symbols (and _end) are
properly covered by a section in the ELF, and thus preventing this issue.

I'm worried that this only solves this one case, and it means that there are two (potentially conflicting) sources of information that a bootloader might be using -- the ELF or the Image header. I don't want to have to duplicate text_offset and so on, which implies that parsing the Image header is necessary anyway.

That's something we can discuss if you send a patch (inline rather than attached).

Quoted 3 lines

I think updating the linker script to put the page tables into a .pgdir section is reasonable, since it is part of the static footprint of the kernel.

I agree

Quoted 3 lines

However, I strongly feel that the Image header should remain the authoritative source of information regarding the nature (big/little endian, page size) and the static footprint of the Image .

Agreed, and there are other ways to de-duplicate which will still work with binary image inputs.

Hi,

Quoted 3 lines · Mark Rutland

Pratyush, Geoff, I understood you were loading the kernel vmlinux for kexec. Do you parse the Image header to figure out where to place things?

Yes, in the kexec user tools we use text_offset to make enough room for the kernel, but there is also the need for page_offset.

We need to know the page_offset to be able to do virtual to physical address conversions. We can calculate the page_offset for a vmlinux image as page_offset = phdr->p_vaddr - text_offset.

The binary Image currently has no info about page_offset or virtual addressing. We have a kexec-tools option for the user to specify a page_offset. If that option is not provided we try to look at the running kernel's symbols, and if that fails, fall back to a default page_offset. This is less than ideal, and certainly makes the binary Image less appealing to use with kexec.