ieee1275: find and claim more memory
cover ieee1275: find and claim more memory
I've posted a couple of RFC patches that attempted to claim some more memory on ieee1275 platforms, which is necessary for the verifiers to do their work. The RFCs were a bit unsatisfactory and also it turns out they broke in some corner cases.
This, I believe, is a proper, upstream-worthy solution for getting more memory on ieee1275 platforms and powerpc-ieee1275 in particular. It is also, sadly, an set of quite complex solutions, as the commit messages for patches 2 and 3 show! It seems this is the cost of broad platform support and backwards compatibility.
There's no reason for this to go into 2.06.
Kind regards,
Daniel
Daniel Axtens (3):
ieee1275: drop HEAP_MAX_ADDR, HEAP_MIN_SIZE
ieee1275: claim more memory
ieee1275: request memory with ibm,client-architecture-support
docs/grub-dev.texi | 6 +- grub-core/kern/ieee1275/cmain.c | 5 +- grub-core/kern/ieee1275/init.c | 238 ++++++++++++++++++++++++++----- include/grub/ieee1275/ieee1275.h | 6 + 4 files changed, 219 insertions(+), 36 deletions(-)
Signature · 1 line
2.27.0
PATCH 1/3 ieee1275: drop HEAP_MAX_ADDR, HEAP_MIN_SIZE
HEAP_MAX_ADDR is confusing. Currently it is set to 32MB, except on ieee1275 on x86, where it is 64MB.
There is a comment which purports to explain it:
/* If possible, we will avoid claiming heap above this address, because it
seems to cause relocation problems with OSes that link at 4 MiB */
This doesn't make a lot of sense when the constants are well above 4MB already. It was not always this way. Prior to
commit 7b5d0fe4440c ("Increase heap limit") in 2010, HEAP_MAX_SIZE and HEAP_MAX_ADDR were indeed 4MB. However, when the constants were increased the comment was left unchanged.
It's been over a decade. It doesn't seem like we have problems with claims over 4MB on powerpc or x86 ieee1275. (sparc does things completely differently and never used the constant.)
Drop the constant and the check.
The only use of HEAP_MIN_SIZE was to potentially override the HEAP_MAX_ADDR check. It is now unused. Remove it.
--- grub-core/kern/ieee1275/init.c | 17 ----------------- 1 file changed, 17 deletions(-)
grub-core/kern/ieee1275/init.c +0 -17
@@ -45,9 +45,6 @@#include <grub/machine/kernel.h> #endif -/* The minimal heap size we can live with. */ -#define HEAP_MIN_SIZE (unsigned long) (2 * 1024 * 1024) -/* The maximum heap size we're going to claim */ #ifdef __i386__#define HEAP_MAX_SIZE (unsigned long) (64 * 1024 * 1024)
@@ -55,14 +52,6 @@#define HEAP_MAX_SIZE (unsigned long) (32 * 1024 * 1024)#endif -/* If possible, we will avoid claiming heap above this address, because it -seems to cause relocation problems with OSes that link at 4 MiB */ -#ifdef __i386__ -#define HEAP_MAX_ADDR (unsigned long) (64 * 1024 * 1024) -#else -#define HEAP_MAX_ADDR (unsigned long) (32 * 1024 * 1024) -#endif -extern char _start[];extern char _end[];
@@ -183,12 +172,6 @@ heap_init (grub_uint64_t addr, grub_uint64_t len, grub_memory_type_t type,if (*total + len >HEAP_MAX_SIZE ) len =HEAP_MAX_SIZE - *total; -/* Avoid claiming anything above HEAP_MAX_ADDR, if possible. */ -if ((addr <HEAP_MAX_ADDR ) &&/* if it's too late, don't bother */ - (addr + len >HEAP_MAX_ADDR ) &&/* if it wasn't available anyway, don't bother */ - (*total + (HEAP_MAX_ADDR - addr) >HEAP_MIN_SIZE ))/* only limit ourselves when we can afford to */ - len =HEAP_MAX_ADDR - addr; -/* In theory, firmware should already prevent this from happening by not listing our own image in /memory/available. The check below is intended as a safeguard in case that doesn't happen. However, it doesn't protect
Signature · 1 line
2.27.0
PATCH 2/3 ieee1275: claim more memory
In reply to Daniel AxtensOn powerpc-ieee1275, we are running out of memory trying to verify anything. This is because:
- we have to load an entire file into memory to verify it. This is extremely difficult to change with appended signatures.
- We only have 32MB of heap.
- Distro kernels are now often around 30MB.
So we want to claim more memory from OpenFirmware for our heap.
There are some complications:
- The grub mm code isn't the only thing that will make claims on memory from OpenFirmware:
* PFW/SLOF will have claimed some for their own use.
* The ieee1275 loader will try to find other bits of memory that we haven't claimed to place the kernel and initrd when we go to boot.
* Once we load Linux, it will also try to claim memory. It claims memory without any reference to /memory/available, it just starts at min(top of RMO, 768MB) and works down. So we need to avoid this area. See arch/powerpc/kernel/prom_init.c as of v5.11.
- The smallest amount of memory a ppc64 KVM guest can have is 256MB. It doesn't work with distro kernels but can work with custom kernels. We should maintain support for that. (ppc32 can boot with even less, and we shouldn't break that either.)
- Even if a VM has more memory, the memory OpenFirmware makes available as Real Memory Area can be restricted. A freshly created LPAR on a PowerVM machine is likely to have only 256MB available to OpenFirmware even if it has many gigabytes of memory allocated.
EFI systems will attempt to allocate 1/4th of the available memory, clamped to between 1M and 1600M. That seems like a good sort of approach, we just need to figure out if 1/4 is the right fraction for us.
We don't know in advance how big the kernel and initrd are going to be, which makes figuring out how much memory we can take a bit tricky.
To figure out how much memory we should leave unused, I looked at:
- an Ubuntu 20.04.1 ppc64le pseries KVM guest:
vmlinux: ~30MB
initrd: ~50MB- a RHEL8.2 ppc64le pseries KVM guest:
vmlinux: ~30MB
initrd: ~30MBUbuntu VMs struggle to boot with just 256MB under SLOF.
RHEL likewise has a higher minimum supported memory figure. So lets first consider a distro kernel and 512MB of addressible memory. (This is the default case for anything booting under PFW.) Say we lose 131MB to PFW (based on some tests). This leaves us 381MB. 1/4 of 381MB is ~95MB. That should be enough to verify a 30MB vmlinux and should leave plenty of space to load Linux and the initrd.
If we consider 256MB of RMA under PFW, we have just 125MB remaining. 1/4 of that is a smidge under 32MB, which gives us very poor odds of verifying a distro-sized kernel. However, if we need 80MB just to put the kernel and initrd in memory, we can't claim any more than 45MB anyway. So 1/4 will do. We'll come back to this later.
grub is always built as a 32-bit binary, even if it's loading a ppc64 kernel. So we can't address memory beyond 4GB. This gives a natural cap of 1GB for powerpc-ieee1275.
Also apply this 1/4 approach to i386-ieee1275, but keep the 32MB cap.
make check still works for both i386 and powerpc and I've booted powerpc grub with this change under SLOF and PFW.
--- docs/grub-dev.texi | 6 ++- grub-core/kern/ieee1275/init.c | 81 +++++++++++++++++++++++++++------- 2 files changed, 69 insertions(+), 18 deletions(-)
docs/grub-dev.texi +4 -2
@@ -1047,7 +1047,9 @@ space is limited to 4GiB. GRUB allocates pages from EFI for its heap, at most 1.6 GiB. On i386-ieee1275 and powerpc-ieee1275 GRUB uses same stack as IEEE1275. -It allocates at most 32MiB for its heap. + +On i386-ieee1275, GRUB allocates at most 32MiB for its heap. On +powerpc-ieee1275, GRUB allocates up to 1GiB. On sparc64-ieee1275 stack is 256KiB and heap is 2MiB.
@@ -1075,7 +1077,7 @@ In short: @item i386-qemu @tab 60 KiB @tab < 4 GiB @item *-efi @tab ? @tab < 1.6 GiB @item i386-ieee1275 @tab ? @tab < 32 MiB -@item powerpc-ieee1275 @tab ? @tab < 32 MiB +@item powerpc-ieee1275 @tab ? @tab < 1 GiB @item sparc64-ieee1275 @tab 256KiB @tab 2 MiB @item arm-uboot @tab 256KiB @tab 2 MiB @item mips(el)-qemu_mips @tab 2MiB @tab 253 MiB
grub-core/kern/ieee1275/init.c +65 -16
131 lines not on this page · raw message
Signature · 1 line
2.27.0
PATCH 3/3 ieee1275: request memory with ibm, client-architecture-support
In reply to Daniel AxtensOn PowerVM, the first time we boot a Linux partition, we may only get 256MB of real memory area, even if the partition has more memory.
This isn't really enough. Fortunately, the Power Architecture Platform Reference (PAPR) defines a method we can call to ask for more memory. This is part of the broad and powerful ibm,client-architecture-support (CAS) method.
CAS can do an enormous amount of things on a PAPR platform: as well as asking for memory, you can set the supported processor level, the interrupt controller, hash vs radix mmu, and so on. We want to touch as little of this as possible because we don't want to step on the toes of the future OS.
If:
- we are running under what we think is PowerVM (compatible property of / begins with "IBM"), and
- the full amount of RMA is less than 512MB (as determined by the reg property of /memory)
then call CAS as follows: (refer to the Linux on Power Architecture Reference, LoPAR, which is public, at B.5.2.3):
- Use the "any" PVR value and supply 2 option vectors.
- Set option vector 1 (PowerPC Server Processor Architecture Level) to "ignore".
- Set option vector 2 with default or Linux-like options, including a min-rma-size of 512MB.
This will cause a CAS reboot and the partition will restart with 512MB of RMA. Grub will notice the 512MB and not call CAS again.
(A partition can be configured with only 256MB of memory, which would mean this request couldn't be satisfied, but PFW refuses to load with only 256MB of memory, so it's a bit moot. SLOF will run fine with 256MB, but we will never call CAS under qemu/SLOF because /compatible won't begin with "IBM".)
One of the first things Linux does while still running under OpenFirmware is to call CAS with a much fuller set of options (including asking for 512MB of memory). This includes a much more restrictive set of PVR values and processor support levels, and this will induce another reboot. On this reboot grub will again notice the higher RMA, and not call CAS. We will get to Linux, Linux will call CAS but because the values are now set for Linux this will not induce another CAS reboot and we will finally boot.
On all subsequent boots, everything will be configured with 512MB of RMA and all the settings Linux likes, so there will be no further CAS reboots.
(phyp is super sticky with the RMA size - it persists even on cold boots. So if you've ever booted Linux in a partition, you'll probably never have grub call CAS. It'll only ever fire the first time a partition loads grub, or if you deliberately lower the amount of memory your partition has below 512MB.)
--- grub-core/kern/ieee1275/cmain.c | 5 +- grub-core/kern/ieee1275/init.c | 144 ++++++++++++++++++++++++++++++- include/grub/ieee1275/ieee1275.h | 6 ++ 3 files changed, 152 insertions(+), 3 deletions(-)
grub-core/kern/ieee1275/cmain.c +4 -1
@@ -90,7 +90,10 @@ grub_ieee1275_find_options (void) }if (rc >=0 &&grub_strncmp (tmp,"IBM" ,3 ) ==0 ) -grub_ieee1275_set_flag (GRUB_IEEE1275_FLAG_NO_TREE_SCANNING_FOR_DISKS ); + { +grub_ieee1275_set_flag (GRUB_IEEE1275_FLAG_NO_TREE_SCANNING_FOR_DISKS ); +grub_ieee1275_set_flag (GRUB_IEEE1275_FLAG_CAN_TRY_CAS_FOR_MORE_MEMORY ); + }/* Old Macs have no key repeat, newer ones have fully working one. The ones inbetween when repeated key generates an escaoe sequence
grub-core/kern/ieee1275/init.c +142 -2
159 lines not on this page · raw message
include/grub/ieee1275/ieee1275.h +6 -0
@@ -148,6 +148,12 @@ enum grub_ieee1275_flagGRUB_IEEE1275_FLAG_CURSORONOFF_ANSI_BROKEN ,GRUB_IEEE1275_FLAG_RAW_DEVNAMES , + +/* On PFW, the first time we boot a Linux partition, we may only get 256MB +of real memory area, even if the partition has more memory. Set this flag +if we think we're running under PFW. Then, if this flag is set, and the +RMA is only 256MB in size, try asking for more with CAS. */ + GRUB_IEEE1275_FLAG_CAN_TRY_CAS_FOR_MORE_MEMORY, };extern int EXPORT_FUNC (grub_ieee1275_test_flag ) (enum grub_ieee1275_flag flag);
Signature · 1 line
2.27.0
Re: [PATCH 3/3] ieee1275: request memory with ibm, client-architecture-support
grub-core/kern/ieee1275/cmain.c grub_ieee1275_find_options (void) · line 96
Quoted 13 lines
grub-core/kern/ieee1275/cmain.c +4
-1@@ -90,7 +90,10 @@ grub_ieee1275_find_options (void) }if (rc >=0 &&grub_strncmp (tmp,"IBM" ,3 ) ==0 ) -grub_ieee1275_set_flag (GRUB_IEEE1275_FLAG_NO_TREE_SCANNING_FOR_DISKS ); + { +grub_ieee1275_set_flag (GRUB_IEEE1275_FLAG_NO_TREE_SCANNING_FOR_DISKS ); +grub_ieee1275_set_flag (GRUB_IEEE1275_FLAG_CAN_TRY_CAS_FOR_MORE_MEMORY ); + }
There's a bug here, I use /model to set the flag, not /compatible. The test just has to move down in the function a bit. I'll send a new spin in a couple of days once anyone interested has a chance to look at the patch. Sorry about that, thought I had checked it, not quite sure what went wrong.
Daniel