x86 Boot enhancements, boot protocol 2.04 7/9

In imitation of the arm and ppc ports a CONFIG_CMDLINE option is also implemented.

+0240/4 2.04+ real_base 0x90000
+0244/4 2.04+ real_memsz Memory used @ 0x90000
+0248/4 2.04+ real_filesz Precomputed data for @ 0x90000

Documentation/i386/boot.txt +35 -0

+
+  low_base, low_memsz, low_filesz, 
+  real_base, real_memsz, real_filesz, 
+  high_base, high_memsz, high_filesz:
+	Up to this point building a zImage or a bzImage has been a very lossy
+	process.  The introduction of these six variables attempts to
+	rectify that situation.  They document exactly which pieces
+	of memory, the kernel uses during the boot process, and they
+	indicate exactly how large the various data segments of the
+	kernel are.  It is now possible to create a lossless
+	transformation to and from a static ELF executable.
+
+	For a bzImage the low program segment describes the memory
+	from 4KB - 572KB the kernel decompressors uses as a temorary
+	buffer.  For a zImage the low program segment describes the
+	memory from 4KB - 572KB where the compressed kernel is loaded. 
+
+	For a bzImage the high program segment describes the memory
+	from 1MB on where the compressed kernel is loaded, where
+	decompression takes place, where the kernel initially runs,
+	and where the kernels bss segment is.  For a zImage the high
+	program segment describes the memory from 1MB on where the
+	kernel is decompressed to, where the kernel initial runs, and
+	where the kernels bss segment is located.
+
+	The real program segment describes the memory from 572KB
+	(0x90000) to 640KB (0xa0000) that the real mode kernel uses.
+	This region may be moved lower in memory if the BIOS has
+	reserved region for some other purpose.  When doing so
+	the following considerations should be applied.
+	
+		For a zImage you may move the real mode kernel now
+		lower than low_base + low_memsz.
+
+		For a bzImage if you move real_base below (low_base +

Quoted 2 lines · Eric W. Biederman

In imitation of the arm and ppc ports a CONFIG_CMDLINE option is also implemented.

Just wondering, why didn't you do it with a CONFIG_CMDLINE_BOOL/CONFIG_CMDLINE set of options? The way you did it, I _think_ you can't actually get a help msg from 'config' or 'oldconfig', you'll just set the commandline to '?'.

Also, on current PPC, if we have a compiled-in commandline we put it in arch/ppc/kernel/setup.c and allow it to be overridden. This even makes it semi-useful outside of the self-containted {b,}zImage situation.

Quoted 5 lines · Tom Rini

Quoted 2 lines · Eric W. Biederman

In imitation of the arm and ppc ports a CONFIG_CMDLINE option is also implemented.

Quoted 4 lines

Just wondering, why didn't you do it with a CONFIG_CMDLINE_BOOL/CONFIG_CMDLINE set of options? The way you did it, I _think_ you can't actually get a help msg from 'config' or 'oldconfig', you'll just set the commandline to '?'.

I just tested it and oldconfig at least works.

Quoted 3 lines

Also, on current PPC, if we have a compiled-in commandline we put it in arch/ppc/kernel/setup.c and allow it to be overridden. This even makes it semi-useful outside of the self-containted {b,}zImage situation.

I currently allow a compiled in command line to be appended to. lilo also does this when you specify a command line, and to my knowledge all boot options prefer the last value specified so that should be good enough. As the decision happens in C code it isn't to hard to change either way.

Quoted 7 lines · Eric W. Biederman

Quoted 5 lines · Tom Rini

Quoted 2 lines · Eric W. Biederman

In imitation of the arm and ppc ports a CONFIG_CMDLINE option is also implemented.

Quoted 6 lines

Quoted 4 lines

Just wondering, why didn't you do it with a CONFIG_CMDLINE_BOOL/CONFIG_CMDLINE set of options? The way you did it, I _think_ you can't actually get a help msg from 'config' or 'oldconfig', you'll just set the commandline to '?'.

I just tested it and oldconfig at least works.

That's sort of supprising.

The way you're doing it now, you're sticking it into final image itself tho, and passing it along. If you're not going to be able to change the commandline, why not handle it all in C? eg:
strcpy(cmd_line, CONFIG_CMDLINE);
if ( passed a new commandline )
strcpy(cmd_line, new_cmdline);

Followup to:  <[email protected]>
By author:    [email protected] (Eric W. Biederman)
In newsgroup: linux.dev.kernel

Quoted 5 lines

+0240/4 2.04+ real_base 0x90000
+0244/4 2.04+ real_memsz Memory used @ 0x90000
+0248/4 2.04+ real_filesz Precomputed data for @ 0x90000

[...]

Quoted 42 lines

Documentation/i386/boot.txt +42 -0

+
+  low_base, low_memsz, low_filesz, 
+  real_base, real_memsz, real_filesz, 
+  high_base, high_memsz, high_filesz:
+	Up to this point building a zImage or a bzImage has been a very lossy
+	process.  The introduction of these six variables attempts to
+	rectify that situation.  They document exactly which pieces
+	of memory, the kernel uses during the boot process, and they
+	indicate exactly how large the various data segments of the
+	kernel are.  It is now possible to create a lossless
+	transformation to and from a static ELF executable.
+
+	For a bzImage the low program segment describes the memory
+	from 4KB - 572KB the kernel decompressors uses as a temorary
+	buffer.  For a zImage the low program segment describes the
+	memory from 4KB - 572KB where the compressed kernel is loaded. 
+
+	For a bzImage the high program segment describes the memory
+	from 1MB on where the compressed kernel is loaded, where
+	decompression takes place, where the kernel initially runs,
+	and where the kernels bss segment is.  For a zImage the high
+	program segment describes the memory from 1MB on where the
+	kernel is decompressed to, where the kernel initial runs, and
+	where the kernels bss segment is located.
+
+	The real program segment describes the memory from 572KB
+	(0x90000) to 640KB (0xa0000) that the real mode kernel uses.
+	This region may be moved lower in memory if the BIOS has
+	reserved region for some other purpose.  When doing so
+	the following considerations should be applied.
+	
+		For a zImage you may move the real mode kernel now
+		lower than low_base + low_memsz.
+
+		For a bzImage if you move real_base below (low_base +
+		low_memsz) the following are the values of the other
+		variables.
+			low_memsz -= (low_base + low_memsz) - real_base
+			high_memsz += (low_base + low_memsz) - real_base
+		
+	
+	With this information it becomes safe to to statically

Documentation/i386/boot.txt line 302

Quoted 4 lines

Documentation/i386/boot.txt +3 -0

+	relocate the real mode kernel as well as dynamically relocate
+	it.  real_base should not be > 0x90000.
+

Whereas this is technically correct, it is dangerously misleading. Please rephrase this to remove any unnecessary references to the legacy address 0x90000.

It's not legacy it is still the optimal address. The amount of memory required to load the kernel is directly related to how much low memory the decompresser can use. If you move it lower you need more memory above 1MB to load the kernel.

To load the real mode code at < 0x90000 requires a fairly sophisticated bootloader. You have already convinced me how it is bad for bootloaders to make BIOS calls on behalf of the kernel, and how bad it is to need bootloaders to change. So unless the real mode code does the int 12 call itself and relocates itself as high as it can go I really don't see the default load address changing. And I am documenting the default load address.

I'd like to change the way we do this, so I'm going to stare at this problem a little bit more. Changing the default load address and still being able to compute how much memory the kernel is going to use is a challenge.

Eric W. Biederman wrote:

No, that's not how that works in reality. In practice, the boot loader picks the lowest address it can practically use, in order to minimize the conventional memory ceiling.

For example, PXELINUX always loads at 0x50000, simply because odds that you have a PXE stack and can use 0x90000 is about zero to none.

In
fact, these days there are enough BIOSes that load stuff in the high part of memory that using 0x90000 is actively dangerous and **needs to be avoided**. Theoretically, you're right; it adds a very small amount of memory to the decompression. If that matters, there is actually a very easy way to deal with
it: for any boot loader there is a Lowest Usable Address (conventional memory ceiling). You can use INT 12h and adjust the load address all the way up to 0x90000 if the conventional memory ceiling permits; this usually is something like five lines of assembly.

Quoted 4 lines

I'd like to change the way we do this, so I'm going to stare at this problem a little bit more. Changing the default load address and still being able to compute how much memory the kernel is going to use is a challenge.

There can't be a "default load address". 0x90000 is actively dangerous and trying to encourage it for anything than legacy kernels is WRONG. If you can't handle this, then you need to go back to the drawing board.

Quoted 4 lines

There can't be a "default load address". 0x90000 is actively dangerous and trying to encourage it for anything than legacy kernels is WRONG. If you can't handle this, then you need to go back to the drawing board.

I agree. But I do think being able to hard code the load address is a very good thing.

After digesting the requirements I plan on having setup.S call int 12h (so the information is available), and then having misc.c relocate the real mode code, and the command line, out of the way, of it's decompression buffer. This removes the need for bootloaders to make a tradeoff between memory use efficiency and reliability.

This should give me about 630KB on machines designed to run DOS, where
this matters. Better than the current best of 572KB, with the real
mode code @ 0x90000.

And when your total size is 1-4MB. +-640KB is a significant change.

Agreed. Note that so far putting the real mode code *above* 0x90000 is completely untested. It *should* work with boot protocol 2.02 support; it almost certainly *does not* work with earlier boot protocols (due to the "move it back to 0x90000" braindamage.)

Agreed. Note that so far putting the real mode code *above* 0x90000 is completely untested. It *should* work with boot protocol 2.02 support; it almost certainly *does not* work with earlier boot protocols (due to the "move it back to 0x90000" braindamage.)

Quoted 5 lines

Agreed. Note that so far putting the real mode code *above* 0x90000 is completely untested. It *should* work with boot protocol 2.02 support; it almost certainly *does not* work with earlier boot protocols (due to the "move it back to 0x90000" braindamage.)

Having misc.c move the real mode code and the command line above 0x90000 avoids this issue. I relocate the command line so cmd_line_ptr must be written. This looses track of exactly which protocol version the bootloader was using, but it doesn't matter as all the kernel cares about is being able to find it's command line, and the command line can still be found. For code coming in the new 32bit entry point we are above protocol version 2.02 when the problem was fixed.

I have now solved the space/reliability tradeoff with belt and suspenders.

I have modified misc.c to do an inplace decompression. This means I use approximately 78KB of memory below 1MB and 8 bytes more than the decompressesed kernel above 1MB. And if I have to except for the 5KB of real mode code I can put everything above 1MB. The 78KB is 5KB real mode code 10KB decompressor code 61KB bss.

The change is especially nice because now in my worst case of only using 5KB real mode data, I do better than the best case with previous kernels (assuming it isn't a zImage). And if I ever get the bootmem bootmap fixed I can put initrds down at 2.6MB and not have to worry about them getting stomped :)