mm/memblock: Add "reserve_mem" to reserved named memory at boot up
Subsystems (6)
Steven Rostedt
maintainerv6 cover mm/memblock: Add "reserve_mem" to reserved named memory at boot up
Reserve unspecified location of physical memory from kernel command line
Background:
In ChromeOS, we have 1 MB of pstore ramoops reserved so that we can extract dmesg output and some other information when a crash happens in the field. (This is only done when the user selects "Allow Google to collect data for
improving the system"). But there are cases when there's a bug that
requires more data to be retrieved to figure out what is happening. We would like to increase the pstore size, either temporarily, or maybe even permanently. The pstore on these devices are at a fixed location in RAM (as the RAM is not cleared on soft reboots nor crashes). The location is chosen by the BIOS (coreboot) and passed to the kernel via ACPI tables on x86. There's a driver that queries for this to initialize the pstore for ChromeOS:
See drivers/platform/chrome/chromeos_pstore.c
Problem:
The problem is that, even though there's a process to change the kernel on these systems, and is done regularly to install updates, the firmware is updated much less frequently. Choosing the place in RAM also takes special care, and may be in a different address for different boards. Updating the size via firmware is a large effort and not something that many are willing to do for a temporary pstore size change.
Requirement:
Need a way to reserve memory that will be at a consistent location for every boot, if the kernel and system are the same. Does not need to work if rebooting to a different kernel, or if the system can change the memory layout between boots.
The reserved memory can not be an hard coded address, as the same kernel / command line needs to run on several different machines. The picked memory reservation just needs to be the same for a given machine, but may be different for different machines.
Solution:
The solution I have come up with is to introduce a new "reserve_mem=" kernel command line. This parameter takes the following format:
reserve_mem=nn:align:label
Where nn is the size of memory to reserve, the align is the alignment of that memory, and label is the way for other sub-systems to find that memory. This way the kernel command line could have:
reserve_mem=12M:4096:oops ramoops.mem_name=oops
At boot up, the kernel will search for 12 megabytes in usable memory regions with an alignment of 4096. It will start at the highest regions and work its way down (for those old devices that want access to lower address DMA). When it finds a region, it will save it off in a small table and mark it with the "oops" label. Then the pstore ramoops sub-system could ask for that memory and location, and it will map itself there.
This prototype allows for 8 different mappings (which may be overkill, 4 is probably plenty) with 16 byte size to store the label.
I have tested this and it works for us to solve the above problem. We can update the kernel and command line and increase the size of pstore without needing to update the firmware, or knowing every memory layout of each board. I only tested this locally, it has not been tested in the field.
Changes since v5: https://lore.kernel.org/all/[email protected]/
[ patch at bottom showing differences ]
- Stressed more that this is a best effort use case
- Updated ramoops.rst to document this new feature
- Used a new variable "tmp" to use in reserve_mem_find_by_name() instead of using "size" and possibly corrupting it.
Changes since v4: https://lore.kernel.org/all/[email protected]/
- Add all checks about reserve_mem before allocation. This means reserved_mem_add() is now a void function.
- Check for name duplications.
- Fix compare of align to SMP_CACHE_BYTES ("<" instead of "<=")
Changes since v3: https://lore.kernel.org/all/[email protected]/
- Changed table type of start and size from unsigned long to phys_addr_t (as well as the parameters to the functions that use them)
- Changed old reference to "early_reserve_mem" to "reserve_mem"
- Check before reservering memory:
o Size is non-zero
o name has text in it
- If align is less than SMP_CACHE_BYTES, make it SMP_CACHE_BYTES
- Remove the silly check of testing *p == '\0' after a p += strlen(p)
Changes since v2: https://lore.kernel.org/all/[email protected]/
- Fixed typo of "reserver"
- Added EXPORT_SYMBOL_GPL() for reserve_mem_find_by_name()
- Removed "built-in" from module description that was changed from v1.
Changes since v1: https://lore.kernel.org/all/[email protected]/
- Updated the change log of the first patch as well as added an entry into kernel-parameters.txt about how reserve_mem is for soft reboots and may not be reliable.
Steven Rostedt (Google) (2):
mm/memblock: Add "reserve_mem" to reserved named memory at boot up pstore/ramoops: Add ramoops.mem_name= command line option
----
Documentation/admin-guide/kernel-parameters.txt | 22 +++++ Documentation/admin-guide/ramoops.rst | 13 +++ fs/pstore/ram.c | 14 +++ include/linux/mm.h | 2 + mm/memblock.c | 117 ++++++++++++++++++++++++ 5 files changed, 168 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +5 -3
@@ -5717,9 +5717,11 @@ used for systems that do not wipe the RAM, and this command line will try to reserve the same physical memory on soft reboots. Note, it is not guaranteed to be the same - location. For example, if KASLR places the kernel at the - location of where the RAM reservation was from a previous - boot, the new reservation will be at a different location. + location. For example, if anything about the system changes + or if booting a different kernel. It can also fail if KASLR + places the kernel at the location of where the RAM reservation + was from a previous boot, the new reservation will be at a + different location. Any subsystem using this feature must add a way to verify that the contents of the physical memory is from a previous boot, as there may be cases where the memory will not be
Documentation/admin-guide/ramoops.rst +13 -0
@@ -23,6 +23,8 @@ and type of the memory area are set using three variables: * ``mem_size`` for the size. The memory size will be rounded down to a power of two. * ``mem_type`` to specify if the memory type (default is pgprot_writecombine). + * ``mem_name`` to specify a memory region defined by ``reserve_mem`` command + line parameter. Typically the default value of ``mem_type=0`` should be used as that sets the pstore mapping to pgprot_writecombine. Setting ``mem_type=1`` attempts to use
@@ -118,6 +120,17 @@ Setting the ramoops parameters can be done in several different manners: return ret; } + D. Using a region of memory reserved via ``reserve_mem`` command line + parameter. The address and size will be defined by the ``reserve_mem`` + parameter. Note, that ``reserve_mem`` may not always allocate memory + in the same location, and cannot be relied upon. Testing will need + to be done, and it may not work on every machine, nor every kernel. + Consider this a "best effort" approach. The ``reserve_mem`` option + takes a size, alignment and name as arguments. The name is used + to map the memory to a label that can be retrieved by ramoops. + + reserver_mem=2M:4096:oops ramoops.mem_name=oops + You can specify either RAM memory or peripheral devices' memory. However, when specifying RAM, be sure to reserve the memory by issuing memblock_reserve() very early in the architecture code, e.g.::
mm/memblock.c +3 -3
@@ -2301,7 +2301,7 @@ EXPORT_SYMBOL_GPL(reserve_mem_find_by_name);*/ static int __init reserve_mem (char *p) { -phys_addr_t start, size, align; +phys_addr_t start, size, align, tmp;char *name;char *oldp;int len;
@@ -2347,8 +2347,8 @@ static int __init reserve_mem(char *p)if (!*p)return -EINVAL ; -/* Make sure the name is not already used (size is only updated if found) */ -if (reserve_mem_find_by_name (name, &start, &size)) +/* Make sure the name is not already used */ +if (reserve_mem_find_by_name (name, &start, &tmp))return -EBUSY ; start =memblock_phys_alloc (size, align);
Steven Rostedt
maintainerPATCH v6 1/2 mm/memblock: Add "reserve_mem" to reserved named memory at boot up
From: "Steven Rostedt (Google)" <[email protected]>
In order to allow for requesting a memory region that can be used for things like pstore on multiple machines where the memory layout is not the same, add a new option to the kernel command line called "reserve_mem".
The format is: reserve_mem=nn:align:name
Where it will find nn amount of memory at the given alignment of align. The name field is to allow another subsystem to retrieve where the memory was found. For example:
reserve_mem=12M:4096:oops ramoops.mem_name=oops
Where ramoops.mem_name will tell ramoops that memory was reserved for it via the reserve_mem option and it can find it by calling:
if (reserve_mem_find_by_name("oops", &start, &size)) {
// start holds the start address and size holds the size givenThis is typically used for systems that do not wipe the RAM, and this command line will try to reserve the same physical memory on soft reboots. Note, it is not guaranteed to be the same location. For example, if KASLR places the kernel at the location of where the RAM reservation was from a previous boot, the new reservation will be at a different location. Any subsystem using this feature must add a way to verify that the contents of the physical memory is from a previous boot, as there may be cases where the memory will not be located at the same location.
Not all systems may work either. There could be bit flips if the reboot goes through the BIOS. Using kexec to reboot the machine is likely to have better results in such cases.
--- .../admin-guide/kernel-parameters.txt | 22 ++++ include/linux/mm.h | 2 + mm/memblock.c | 117 ++++++++++++++++++ 3 files changed, 141 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +22 -0
@@ -5710,6 +5710,28 @@ them. If <base> is less than 0x10000, the region is assumed to be I/O ports; otherwise it is memory. + reserve_mem= [RAM] + Format: nn[KNG]:<align>:<label> + Reserve physical memory and label it with a name that + other subsystems can use to access it. This is typically + used for systems that do not wipe the RAM, and this command + line will try to reserve the same physical memory on + soft reboots. Note, it is not guaranteed to be the same + location. For example, if anything about the system changes + or if booting a different kernel. It can also fail if KASLR + places the kernel at the location of where the RAM reservation + was from a previous boot, the new reservation will be at a + different location. + Any subsystem using this feature must add a way to verify + that the contents of the physical memory is from a previous + boot, as there may be cases where the memory will not be + located at the same location. + + The format is size:align:label for example, to request + 12 megabytes of 4096 alignment for ramoops: + + reserve_mem=12M:4096:oops ramoops.mem_name=oops + reservetop= [X86-32,EARLY] Format: nn[KMG] Reserves a hole at the top of the kernel virtual
include/linux/mm.h +2 -0
@@ -4263,4 +4263,6 @@ static inline bool pfn_is_unaccepted_memory(unsigned long pfn)void vma_pgtable_walk_begin (struct vm_area_struct *vma);void vma_pgtable_walk_end (struct vm_area_struct *vma); +int reserve_mem_find_by_name (const char *name,phys_addr_t *start,phys_addr_t *size); +#endif /* _LINUX_MM_H */
mm/memblock.c +117 -0
123 lines not on this page · raw message
Signature · 1 line
2.43.0
Quoted 200 lines · Steven Rostedt
From: "Steven Rostedt (Google)" <[email protected]>
In order to allow for requesting a memory region that can be used for things like pstore on multiple machines where the memory layout is not the same, add a new option to the kernel command line called "
reserve_mem".The format is: reserve_mem=nn:align:name
Where it will find nn amount of memory at the given alignment of align. The name field is to allow another subsystem to retrieve where the memory was found. For example:
reserve_mem=12M:4096:oops ramoops.mem_name=oopsWhere ramoops.mem_name will tell ramoops that memory was reserved for it via the
reserve_memoption and it can find it by calling:if (reserve_mem_find_by_name("oops", &start, &size)) { // start holds the start address and size holds the size givenThis is typically used for systems that do not wipe the RAM, and this command line will try to reserve the same physical memory on soft reboots. Note, it is not guaranteed to be the same location. For example, if KASLR places the kernel at the location of where the RAM reservation was from a previous boot, the new reservation will be at a different location. Any subsystem using this feature must add a way to verify that the contents of the physical memory is from a previous boot, as there may be cases where the memory will not be located at the same location.
Not all systems may work either. There could be bit flips if the reboot goes through the BIOS. Using kexec to reboot the machine is likely to have better results in such cases.
Suggested-by Mike Rapoport <[email protected]>Tested-by Guilherme G. Piccoli <[email protected]>Signed-off-by Steven Rostedt (Google) <[email protected]>--- .../admin-guide/kernel-parameters.txt | 22 ++++ include/linux/mm.h | 2 + mm/memblock.c | 117 ++++++++++++++++++ 3 files changed, 141 insertions(+)Documentation/admin-guide/kernel-parameters.txt +22
-0@@ -5710,6 +5710,28 @@ them. If <base> is less than 0x10000, the region is assumed to be I/O ports; otherwise it is memory. + reserve_mem= [RAM] + Format: nn[KNG]:<align>:<label> + Reserve physical memory and label it with a name that + other subsystems can use to access it. This is typically + used for systems that do not wipe the RAM, and this command + line will try to reserve the same physical memory on + soft reboots. Note, it is not guaranteed to be the same + location. For example, if anything about the system changes + or if booting a different kernel. It can also fail if KASLR + places the kernel at the location of where the RAM reservation + was from a previous boot, the new reservation will be at a + different location. + Any subsystem using this feature must add a way to verify + that the contents of the physical memory is from a previous + boot, as there may be cases where the memory will not be + located at the same location. + + The format is size:align:label for example, to request + 12 megabytes of 4096 alignment for ramoops: + + reserve_mem=12M:4096:oops ramoops.mem_name=oops + reservetop= [X86-32,EARLY] Format: nn[KMG] Reserves a hole at the top of the kernel virtualinclude/linux/mm.h +2
-0@@ -4263,4 +4263,6 @@ static inline bool pfn_is_unaccepted_memory(unsigned long pfn)void vma_pgtable_walk_begin (struct vm_area_struct *vma);void vma_pgtable_walk_end (struct vm_area_struct *vma); +int reserve_mem_find_by_name (const char *name,phys_addr_t *start,phys_addr_t *size); +#endif /* _LINUX_MM_H */ mm/memblock.c +104
-0@@ -2244,6 +2244,123 @@ void __init memblock_free_all(void)totalram_pages_add (pages); } +/* Keep a table to reserve named memory */ +#define RESERVE_MEM_MAX_ENTRIES 8 +#define RESERVE_MEM_NAME_SIZE 16 +struct reserve_mem_table { +char name[RESERVE_MEM_NAME_SIZE ]; +phys_addr_t start; +phys_addr_t size; +}; +static struct reserve_mem_table reserved_mem_table[RESERVE_MEM_MAX_ENTRIES ]; +static int reserved_mem_count; + +/* Add wildcard region with a lookup name */ +static void __init reserved_mem_add (phys_addr_t start,phys_addr_t size, +const char *name) +{ +struct reserve_mem_table *map; + + map = &reserved_mem_table[reserved_mem_count++]; + map->start = start; + map->size = size; +strscpy (map->name, name); +} + +/** +* reserve_mem_find_by_name - Find reserved memory region with a given name +* @name: The name that is attached to a reserved memory region +* @start: If found, holds the start address +* @size: If found, holds the size of the address. +* +* @start and @size are only updated if @name is found. +* +* Returns: 1 if found or 0 if not found. +*/ +int reserve_mem_find_by_name (const char *name,phys_addr_t *start,phys_addr_t *size) +{ +struct reserve_mem_table *map; +int i; + +for (i =0 ; i < reserved_mem_count; i++) { + map = &reserved_mem_table[i]; +if (!map->size) +continue ; +if (strcmp (name, map->name) ==0 ) { + *start = map->start; + *size = map->size; +return 1 ; + } + } +return 0 ; +} +EXPORT_SYMBOL_GPL (reserve_mem_find_by_name); + +/* +* Parse reserve_mem=nn:align:name +*/ +static int __init reserve_mem (char *p) +{ +phys_addr_t start, size, align, tmp; +char *name; +char *oldp; +int len; + +if (!p) +return -EINVAL ; + +/* Check if there's room for more reserved memory */ +if (reserved_mem_count >=RESERVE_MEM_MAX_ENTRIES ) +return -EBUSY ; + + oldp = p; + size =memparse (p, &p); +if (!size || p == oldp) +return -EINVAL ; + +if (*p !=':' ) +return -EINVAL ; + + align =memparse (p+1 , &p); +if (*p !=':' ) +return -EINVAL ; + +/* +* memblock_phys_alloc() doesn't like a zero size align, +* but it is OK for this command to have it. +*/ +if (align <SMP_CACHE_BYTES ) + align =SMP_CACHE_BYTES ; + + name = p +1 ; + len =strlen (name); + +/* name needs to have length but not too big */ +if (!len || len >=RESERVE_MEM_NAME_SIZE ) +return -EINVAL ; + +/* Make sure that name has text */ +for (p = name; *p; p++) { +if (!isspace (*p)) +break ; + } +if (!*p) +return -EINVAL ; + +/* Make sure the name is not already used */
mm/memblock.c void __init memblock_free_all(void) · line 2360
Quoted 10 lines
mm/memblock.c +10
-0+if (reserve_mem_find_by_name (name, &start, &tmp)) +return -EBUSY ; + + start =memblock_phys_alloc (size, align); +if (!start) +return -ENOMEM ; + +reserved_mem_add (start, size, name); + +return 0 ;
Hi, steve, should here return 1 ? Other __setup functions return 1 when it success.
Quoted 6 lines
mm/memblock.c +3
-0+} +__setup ("reserve_mem=" , reserve_mem); +#if defined(CONFIG_DEBUG_FS ) && defined(CONFIG_ARCH_KEEP_MEMBLOCK )static const char *const flagname[] = { [ilog2(MEMBLOCK_HOTPLUG)] = "HOTPLUG",
Steven Rostedt
maintainerQuoted 10 lines · On Tue, 18 Jun 2024 20:55:17 +0800 Zhengyejian <[email protected]>…
mm/memblock.c void __init memblock_free_all(void) · line 2360
Quoted 7 lines
mm/memblock.c +7
-0+ start =memblock_phys_alloc (size, align); +if (!start) +return -ENOMEM ; + +reserved_mem_add (start, size, name); + +return 0 ;Hi, steve, should here return 1 ? Other
__setupfunctions return 1 when it success.
Ug, you're correct.
Mike, want me to send a v7?
-- Steve
Mike Rapoport
maintainerQuoted 14 lines · Steven Rostedt
Quoted 10 lines · On Tue, 18 Jun 2024 20:55:17 +0800 Zhengyejian <[email protected]>…
mm/memblock.c void __init memblock_free_all(void) · line 2360
Quoted 7 lines
mm/memblock.c +7
-0+ start =memblock_phys_alloc (size, align); +if (!start) +return -ENOMEM ; + +reserved_mem_add (start, size, name); + +return 0 ;Hi, steve, should here return 1 ? Other
__setupfunctions return 1 when it success.
Quoted 3 lines
Ug, you're correct.
Mike, want me to send a v7?
I can fix it up while applying.
Quoted 1 line
-- Steve
Signature · 2 lines
Sincerely yours, Mike.
Steven Rostedt
maintainerPATCH v6 2/2 pstore/ramoops: Add ramoops.mem_name= command line option
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Add a method to find a region specified by reserve_mem=nn:align:name for ramoops. Adding a kernel command line parameter:
reserve_mem=12M:4096:oops ramoops.mem_name=oops
Will use the size and location defined by the memmap parameter where it finds the memory and labels it "oops". The "oops" in the ramoops option is used to search for it.
This allows for arbitrary RAM to be used for ramoops if it is known that the memory is not cleared on kernel crashes or soft reboots.
--- Documentation/admin-guide/ramoops.rst | 13 +++++++++++++ fs/pstore/ram.c | 14 ++++++++++++++ 2 files changed, 27 insertions(+)
Documentation/admin-guide/ramoops.rst +13 -0
@@ -23,6 +23,8 @@ and type of the memory area are set using three variables: * ``mem_size`` for the size. The memory size will be rounded down to a power of two. * ``mem_type`` to specify if the memory type (default is pgprot_writecombine). + * ``mem_name`` to specify a memory region defined by ``reserve_mem`` command + line parameter. Typically the default value of ``mem_type=0`` should be used as that sets the pstore mapping to pgprot_writecombine. Setting ``mem_type=1`` attempts to use
@@ -118,6 +120,17 @@ Setting the ramoops parameters can be done in several different manners: return ret; } + D. Using a region of memory reserved via ``reserve_mem`` command line + parameter. The address and size will be defined by the ``reserve_mem`` + parameter. Note, that ``reserve_mem`` may not always allocate memory + in the same location, and cannot be relied upon. Testing will need + to be done, and it may not work on every machine, nor every kernel. + Consider this a "best effort" approach. The ``reserve_mem`` option + takes a size, alignment and name as arguments. The name is used + to map the memory to a label that can be retrieved by ramoops. + + reserver_mem=2M:4096:oops ramoops.mem_name=oops + You can specify either RAM memory or peripheral devices' memory. However, when specifying RAM, be sure to reserve the memory by issuing memblock_reserve() very early in the architecture code, e.g.::
fs/pstore/ram.c +14 -0
@@ -50,6 +50,10 @@ module_param_hw(mem_address, ullong, other, 0400);MODULE_PARM_DESC (mem_address,"start of reserved RAM used to store oops/panic logs" ); +static char *mem_name; +module_param_named (mem_name, mem_name, charp,0400 ); +MODULE_PARM_DESC (mem_name,"name of kernel param that holds addr" ); +static ulong mem_size;module_param (mem_size, ulong,0400 ); MODULE_PARM_DESC(mem_size,
@@ -914,6 +918,16 @@ static void __init ramoops_register_dummy(void) {struct ramoops_platform_data pdata; +if (mem_name) { +phys_addr_t start; +phys_addr_t size; + +if (reserve_mem_find_by_name (mem_name, &start, &size)) { + mem_address = start; + mem_size = size; + } + } +/* * Prepare a dummy platform data structure to carry the module * parameters. If mem_size isn't set, then there are no module
Signature · 1 line
2.43.0
Guilherme G. Piccoli
reviewerRe: [PATCH v6 2/2] pstore/ramoops: Add ramoops.mem_name= command line option
Thanks Steve, re-tested in my VM and it's working fine. Just a minor below...
Documentation/admin-guide/ramoops.rst Setting the ramoops parameters can be done in several different manners: · line 133
Quoted 12 lines · Steven Rostedt
[...]
+ D. Using a region of memory reserved via ``reserve_mem`` command lineDocumentation/admin-guide/ramoops.rst +10
-0+ parameter. The address and size will be defined by the ``reserve_mem`` + parameter. Note, that ``reserve_mem`` may not always allocate memory + in the same location, and cannot be relied upon. Testing will need + to be done, and it may not work on every machine, nor every kernel. + Consider this a "best effort" approach. The ``reserve_mem`` option + takes a size, alignment and name as arguments. The name is used + to map the memory to a label that can be retrieved by ramoops. + + reserver_mem=2M:4096:oops ramoops.mem_name=oops +
Likely this could be fixed on merge, to avoid another version, but...
s/reserver_mem/reserve_mem
Cheers!
Steven Rostedt
maintainerRe: [PATCH v6 2/2] pstore/ramoops: Add ramoops.mem_name= command line option
Quoted 7 lines · On Thu, 13 Jun 2024 15:19:50 -0300 "Guilherme G. Piccoli"…
Documentation/admin-guide/ramoops.rst Setting the ramoops parameters can be done in several different manners: · line 133
Quoted 3 lines
Documentation/admin-guide/ramoops.rst +3
-0+ + reserver_mem=2M:4096:oops ramoops.mem_name=oops +Likely this could be fixed on merge, to avoid another version, but...
s/reserver_mem/reserve_mem
That 'r' is my nemesis! Almost every time I type "reserve" I write it as "reserver" and have to go back and delete it. :-p
Just to make patchwork work nicely, I'll send a v7. But I'll wait for other comments or acks and reviews.
-- Steve
Guilherme G. Piccoli
reviewerRe: [PATCH v6 2/2] pstore/ramoops: Add ramoops.mem_name= command line option
Quoted 9 lines · Steven Rostedt
Quoted 6 lines · On Thu, 13 Jun 2024 15:19:50 -0300 "Guilherme G. Piccoli"…
Documentation/admin-guide/ramoops.rst Setting the ramoops parameters can be done in several different manners: · line 133
Quoted 3 lines
Documentation/admin-guide/ramoops.rst +3
-0+ + reserver_mem=2M:4096:oops ramoops.mem_name=oops +Likely this could be fixed on merge, to avoid another version, but...
Quoted 4 lines
Quoted 1 line
s/reserver_mem/reserve_mem
That 'r' is my nemesis! Almost every time I type "reserve" I write it as "reserver" and have to go back and delete it. :-p
LOL, I thought the same! And I even wondered if you're not a vim user that makes use of 'r' for replacing text and double typed that - I do that all the time, with r and i.
Anyway, thanks for fixing that.
Cheers!
Hey Steve,
Quoted 18 lines · Steven Rostedt
Reserve unspecified location of physical memory from kernel command line
Background:
In ChromeOS, we have 1 MB of pstore ramoops reserved so that we can extract dmesg output and some other information when a crash happens in the field. (This is only done when the user selects "Allow Google to collect data for
improving the system"). But there are cases when there's a bug that
requires more data to be retrieved to figure out what is happening. We would like to increase the pstore size, either temporarily, or maybe even permanently. The pstore on these devices are at a fixed location in RAM (as the RAM is not cleared on soft reboots nor crashes). The location is chosen by the BIOS (coreboot) and passed to the kernel via ACPI tables on x86. There's a driver that queries for this to initialize the pstore for ChromeOS:See
drivers/platform/chrome/chromeos_pstore.c
Quoted 8 lines
Problem:
The problem is that, even though there's a process to change the kernel on these systems, and is done regularly to install updates, the firmware is updated much less frequently. Choosing the place in RAM also takes special care, and may be in a different address for different boards. Updating the size via firmware is a large effort and not something that many are willing to do for a temporary pstore size change.
(sorry for not commenting on earlier versions, I didn't see v1-v5 in my inbox)
Do you have a "real" pstore on these systems that you could store non-volatile variables in, such as persistent UEFI variables? If so, you could create an actually persistent mapping for your trace pstore even across kernel version updates as a general mechanism to create reserved memblocks at fixed offsets.
Quoted 10 lines
Requirement:
Need a way to reserve memory that will be at a consistent location for every boot, if the kernel and system are the same. Does not need to work if rebooting to a different kernel, or if the system can change the memory layout between boots.
The reserved memory can not be an hard coded address, as the same kernel / command line needs to run on several different machines. The picked memory reservation just needs to be the same for a given machine, but may be
With KASLR is enabled, doesn't this approach break too often to be reliable enough for the data you want to extract?
Picking up the idea above, with a persistent variable we could even make KASLR avoid that reserved pstore region in its search for a viable KASLR offset.
Alex
Amazon Web Services Development Center Germany GmbH Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597
Steven Rostedt
maintainerQuoted 6 lines · On Thu, 13 Jun 2024 18:54:12 +0200 Alexander Graf <[email protected]> wrote:
Do you have a "real" pstore on these systems that you could store non-volatile variables in, such as persistent UEFI variables? If so, you could create an actually persistent mapping for your trace pstore even across kernel version updates as a general mechanism to create reserved memblocks at fixed offsets.
After implementing all this, I don't think I can use pstore for my purpose. pstore is a generic interface for persistent storage, and requires an interface to access it. From what I understand, it's not the place to just ask for an area of RAM.
For this, I have a single patch that allows the tracing instance to use an area reserved by reserve_mem.
reserve_mem=12M:4096:trace trace_instance=boot_mapped@trace
I've already tested this on qemu and a couple of chromebooks. It works well.
Quoted 14 lines
Quoted 4 lines
Requirement:
Quoted 8 lines
Need a way to reserve memory that will be at a consistent location for every boot, if the kernel and system are the same. Does not need to work if rebooting to a different kernel, or if the system can change the memory layout between boots.
The reserved memory can not be an hard coded address, as the same kernel / command line needs to run on several different machines. The picked memory reservation just needs to be the same for a given machine, but may be
Quoted 6 lines
With KASLR is enabled, doesn't this approach break too often to be reliable enough for the data you want to extract?
Picking up the idea above, with a persistent variable we could even make KASLR avoid that reserved pstore region in its search for a viable KASLR offset.
I think I was hit by it once in all my testing. For our use case, the few times it fails to map is not going to affect what we need this for at all.
-- Steve
Hey Steve,
Quoted 13 lines · Steven Rostedt
Quoted 5 lines · On Thu, 13 Jun 2024 18:54:12 +0200 Alexander Graf <[email protected]> wrote:
Do you have a "real" pstore on these systems that you could store non-volatile variables in, such as persistent UEFI variables? If so, you could create an actually persistent mapping for your trace pstore even across kernel version updates as a general mechanism to create reserved memblocks at fixed offsets.
After implementing all this, I don't think I can use pstore for my purpose. pstore is a generic interface for persistent storage, and requires an interface to access it. From what I understand, it's not the place to just ask for an area of RAM.
Quoted 7 lines
For this, I have a single patch that allows the tracing instance to use an area reserved by
reserve_mem.reserve_mem=12M:4096:trace trace_instance=boot_mapped@trace
I've already tested this on qemu and a couple of chromebooks. It works well.
I believe we're talking about 2 different things :). Let me rephrase a bit and make a concrete example.
Imagine you have passed the "reserve_mem=12M:4096:trace" kernel command line option. The kernel now comes up and allocates a random chunk of memory that - by (admittedly good) chance - may be at the same physical location as before. Let's assume it deemed 0x1000000 as a good offset.
Let's now assume you're running on a UEFI system. There, you always have non-volatile storage available to you even in the pre-boot phase. That means the kernel could create a UEFI variable that says "12M:4096:trace -> 0x1000000". The pre-boot phase takes all these UEFI variables and marks them as reserved. When you finally reach your command line parsing logic for reserve_mem=, you can flip all reservations that were not on the command line back to normal memory.
That way you have pretty much guaranteed persistent memory regions, even with KASLR changing your memory layout across boots.
The nice thing is that the above is an extension of what you've already built: Systems with UEFI simply get better guarantees that their regions persist.
Quoted 16 lines
Quoted 16 lines
Quoted 4 lines
Requirement:
Quoted 8 lines
Need a way to reserve memory that will be at a consistent location for every boot, if the kernel and system are the same. Does not need to work if rebooting to a different kernel, or if the system can change the memory layout between boots.
The reserved memory can not be an hard coded address, as the same kernel / command line needs to run on several different machines. The picked memory reservation just needs to be the same for a given machine, but may be
With KASLR is enabled, doesn't this approach break too often to be reliable enough for the data you want to extract?
Quoted 6 lines
Quoted 3 lines
Picking up the idea above, with a persistent variable we could even make KASLR avoid that reserved pstore region in its search for a viable KASLR offset.
I think I was hit by it once in all my testing. For our use case, the few times it fails to map is not going to affect what we need this for at all.
Once is pretty good. Do you know why? Also once out of how many runs? Is the randomness source not as random as it should be or are the number of bits for KASLR maybe so few on your target architecture that the odds of hitting anything become low? Do these same constraints hold true outside of your testing environment?
Alex
Amazon Web Services Development Center Germany GmbH Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597
Steven Rostedt
maintainerQuoted 10 lines · On Mon, 17 Jun 2024 09:07:29 +0200 Alexander Graf <[email protected]> wrote:
Hey Steve,
I believe we're talking about 2 different things :). Let me rephrase a bit and make a concrete example.
Imagine you have passed the "reserve_mem=12M:4096:trace" kernel command line option. The kernel now comes up and allocates a random chunk of memory that - by (admittedly good) chance - may be at the same physical location as before. Let's assume it deemed
0x1000000as a good offset.
Note, it's not random. It picks from the top of available memory every time. But things can mess with it (see below).
Quoted 9 lines
Let's now assume you're running on a UEFI system. There, you always have non-volatile storage available to you even in the pre-boot phase. That means the kernel could create a UEFI variable that says "12M:4096:trace ->
0x1000000". The pre-boot phase takes all these UEFI variables and marks them as reserved. When you finally reach your command line parsing logic for reserve_mem=, you can flip all reservations that were not on the command line back to normal memory.
Quoted 6 lines
That way you have pretty much guaranteed persistent memory regions, even with KASLR changing your memory layout across boots.
The nice thing is that the above is an extension of what you've already
built: Systems with UEFI simply get better guarantees that their regions persist.
This could be an added feature, but it is very architecture specific, and would likely need architecture specific updates.
Quoted 26 lines
Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
Requirement:
Quoted 8 lines
Need a way to reserve memory that will be at a consistent location for every boot, if the kernel and system are the same. Does not need to work if rebooting to a different kernel, or if the system can change the memory layout between boots.
The reserved memory can not be an hard coded address, as the same kernel / command line needs to run on several different machines. The picked memory reservation just needs to be the same for a given machine, but may be
With KASLR is enabled, doesn't this approach break too often to be reliable enough for the data you want to extract?
Quoted 6 lines
Quoted 3 lines
Picking up the idea above, with a persistent variable we could even make KASLR avoid that reserved pstore region in its search for a viable KASLR offset.
I think I was hit by it once in all my testing. For our use case, the few times it fails to map is not going to affect what we need this for at all.
Quoted 5 lines
Once is pretty good. Do you know why? Also once out of how many runs? Is the randomness source not as random as it should be or are the number of bits for KASLR maybe so few on your target architecture that the odds of hitting anything become low? Do these same constraints hold true outside of your testing environment?
So I just ran it a hundred times in a loop. I added a patch to print the location of "_text". The loop was this:
for i in seq 100; do
ssh root@debiantesting-x86-64 "dmesg | grep -e 'text starts' -e 'mapped boot' >> text; grub-reboot '1>0'; sleep 0.5; reboot" sleep 25
done
It searches dmesg for my added printk as well as the print of were the ring buffer was loaded in physical memory.
It takes about 15 seconds to reboot, so I waited 25. The results are attached. I found that it was consistent 76 times, which means 1 out of 4 it's not. Funny enough, it broke whenever it loaded the kernel below 0x100000000. And then it would be off by a little.
It was consistently at:
0x27d000000
And when it failed, it was at 0x27ce00000.
Note, when I used the e820 tables to do this, I never saw a failure. My assumption is that when it is below 0x100000000, something else gets allocated causing this to get pushed down.
As this code relies on memblock_phys_alloc() being consistent, if something gets allocated before it differently depending on where the kernel is, it can also move the location. A plugin to UEFI would mean that it would need to reserve the memory, and the code here will need to know where it is. We could always make the function reserve_mem() global and weak so that architectures can override it.
-- Steve
[resend because Thunderbird decided to send the previous version as HTML :(]
Quoted 24 lines · Steven Rostedt
Quoted 10 lines · On Mon, 17 Jun 2024 09:07:29 +0200 Alexander Graf<[email protected]> wrote:
Hey Steve,
I believe we're talking about 2 different things :). Let me rephrase a bit and make a concrete example.
Imagine you have passed the "reserve_mem=12M:4096:trace" kernel command line option. The kernel now comes up and allocates a random chunk of memory that - by (admittedly good) chance - may be at the same physical location as before. Let's assume it deemed
0x1000000as a good offset.Note, it's not random. It picks from the top of available memory every time. But things can mess with it (see below).
Quoted 8 lines
Let's now assume you're running on a UEFI system. There, you always have non-volatile storage available to you even in the pre-boot phase. That means the kernel could create a UEFI variable that says "12M:4096:trace ->
0x1000000". The pre-boot phase takes all these UEFI variables and marks them as reserved. When you finally reach your command line parsing logic for reserve_mem=, you can flip all reservations that were not on the command line back to normal memory.
Quoted 8 lines
Quoted 6 lines
That way you have pretty much guaranteed persistent memory regions, even with KASLR changing your memory layout across boots.
The nice thing is that the above is an extension of what you've already
built: Systems with UEFI simply get better guarantees that their regions persist.This could be an added feature, but it is very architecture specific, and would likely need architecture specific updates.
It definitely would be an added feature, yes. But one that allows you to ensure persistence a lot more safely :).
Quoted 42 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Requirement:
Need a way to reserve memory that will be at a consistent location for every boot, if the kernel and system are the same. Does not need to work if rebooting to a different kernel, or if the system can change the memory layout between boots.
The reserved memory can not be an hard coded address, as the same kernel / command line needs to run on several different machines. The picked memory reservation just needs to be the same for a given machine, but may be
With KASLR is enabled, doesn't this approach break too often to be reliable enough for the data you want to extract?
Picking up the idea above, with a persistent variable we could even make KASLR avoid that reserved pstore region in its search for a viable KASLR offset.
Quoted 8 lines
Quoted 3 lines
I think I was hit by it once in all my testing. For our use case, the few times it fails to map is not going to affect what we need this for at all.
Once is pretty good. Do you know why? Also once out of how many runs? Is the randomness source not as random as it should be or are the number of bits for KASLR maybe so few on your target architecture that the odds of hitting anything become low? Do these same constraints hold true outside of your testing environment?
So I just ran it a hundred times in a loop. I added a patch to print the location of "
_text". The loop was this:for i in
seq 100; do
ssh root@debiantesting-x86-64 "dmesg | grep -e 'text starts' -e 'mapped boot' >> text; grub-reboot '1>0'; sleep 0.5; reboot" sleep 25
doneIt searches dmesg for my added printk as well as the print of were the ring buffer was loaded in physical memory.
It takes about 15 seconds to reboot, so I waited 25. The results are attached. I found that it was consistent 76 times, which means 1 out of 4 it's not. Funny enough, it broke whenever it loaded the kernel below
0x100000000. And then it would be off by a little.It was consistently at:
Quoted 7 lines
0x27d000000And when it failed, it was at
0x27ce00000.Note, when I used the e820 tables to do this, I never saw a failure. My assumption is that when it is below
0x100000000, something else gets allocated causing this to get pushed down.
Thinking about it again: What if you run the allocation super early (see arch/x86/boot/compressed/kaslr.c:handle_mem_options())? If you stick to allocating only from top, you're effectively kernel version independent for your allocations because none of the kernel code ran yet and definitely KASLR independent because you're running deterministically before KASLR even gets allocated.
Quoted 6 lines
As this code relies on
memblock_phys_alloc()being consistent, if something gets allocated before it differently depending on where the kernel is, it can also move the location. A plugin to UEFI would mean that it would need to reserve the memory, and the code here will need to know where it is. We could always make the functionreserve_mem()global and weak so that architectures can override it.
Yes, the in-kernel UEFI loader (efi-stub) could simply populate a new type of memblock with the respective reservations and you later call memblock_find_in_range_node() instead of memblock_phys_alloc() to pass in flags that you want to allocate only from the new MEMBLOCK_RESERVE_MEM type. The same model would work for BIOS boots through the handle_mem_options() path above. In fact, if the BIOS way works fine, we don't even need UEFI variables: The same way allocations will be identical during BIOS execution, they should stay identical across UEFI launches.
As cherry on top, kexec also works seamlessly with the special memblock approach because kexec (at least on x86) hands memblocks as is to the next kernel. So the new kernel will also automatically use the same ranges for its allocations.
Alex
Amazon Web Services Development Center Germany GmbH Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597
Steven Rostedt
maintainerQuoted 6 lines · On Mon, 17 Jun 2024 23:01:12 +0200 Alexander Graf <[email protected]> wrote:
Quoted 2 lines
This could be an added feature, but it is very architecture specific, and would likely need architecture specific updates.
It definitely would be an added feature, yes. But one that allows you to ensure persistence a lot more safely :).
Sure.
Quoted 26 lines
Thinking about it again: What if you run the allocation super early (see arch/x86/boot/compressed/kaslr.c:handle_mem_options())? If you stick to allocating only from top, you're effectively kernel version independent for your allocations because none of the kernel code ran yet and definitely KASLR independent because you're running deterministically before KASLR even gets allocated.
Quoted 6 lines
As this code relies on
memblock_phys_alloc()being consistent, if something gets allocated before it differently depending on where the kernel is, it can also move the location. A plugin to UEFI would mean that it would need to reserve the memory, and the code here will need to know where it is. We could always make the functionreserve_mem()global and weak so that architectures can override it.Yes, the in-kernel UEFI loader (efi-stub) could simply populate a new type of memblock with the respective reservations and you later call
memblock_find_in_range_node()instead ofmemblock_phys_alloc()to pass in flags that you want to allocate only from the newMEMBLOCK_RESERVE_MEMtype. The same model would work for BIOS boots through the handle_mem_options() path above. In fact, if the BIOS way works fine, we don't even need UEFI variables: The same way allocations will be identical during BIOS execution, they should stay identical across UEFI launches.
Quoted 4 lines
As cherry on top, kexec also works seamlessly with the special memblock approach because kexec (at least on x86) hands memblocks as is to the next kernel. So the new kernel will also automatically use the same ranges for its allocations.
I'm all for expanding this. But I would just want to get this in for now as is. It theoretically works on all architectures. If someone wants to make in more robust and accurate on a specific architecture, I'm all for it. Like I said, we could make the reserver_mem() function global and weak, and then if an architecture has a better way to handle this, it could use that.
Hmm, x86 could do this with the e820 code like I did in my first versions. Like I said, it didn't fail at all with that.
And we can have an UEFI version as well.
-- Steve
Quoted 86 lines · Alexander Graf
Quoted 24 lines · Steven Rostedt
Quoted 10 lines · On Mon, 17 Jun 2024 09:07:29 +0200 Alexander Graf<[email protected]> wrote:
Hey Steve,
I believe we're talking about 2 different things :). Let me rephrase a bit and make a concrete example.
Imagine you have passed the "reserve_mem=12M:4096:trace" kernel command line option. The kernel now comes up and allocates a random chunk of memory that - by (admittedly good) chance - may be at the same physical location as before. Let's assume it deemed
0x1000000as a good offset.Note, it's not random. It picks from the top of available memory every time. But things can mess with it (see below).
Quoted 8 lines
Let's now assume you're running on a UEFI system. There, you always have non-volatile storage available to you even in the pre-boot phase. That means the kernel could create a UEFI variable that says "12M:4096:trace ->
0x1000000". The pre-boot phase takes all these UEFI variables and marks them as reserved. When you finally reach your command line parsing logic for reserve_mem=, you can flip all reservations that were not on the command line back to normal memory.Quoted 8 lines
Quoted 6 lines
That way you have pretty much guaranteed persistent memory regions, even with KASLR changing your memory layout across boots.
The nice thing is that the above is an extension of what you've already
built: Systems with UEFI simply get better guarantees that their regions persist.This could be an added feature, but it is very architecture specific, and would likely need architecture specific updates.
It definitely would be an added feature, yes. But one that allows you to ensure persistence a lot more safely :).
Quoted 46 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Requirement:
Need a way to reserve memory that will be at a consistent location for every boot, if the kernel and system are the same. Does not need to work if rebooting to a different kernel, or if the system can change the memory layout between boots.
The reserved memory can not be an hard coded address, as the same kernel / command line needs to run on several different machines. The picked memory reservation just needs to be the same for a given machine, but may be
With KASLR is enabled, doesn't this approach break too often to be reliable enough for the data you want to extract?
Picking up the idea above, with a persistent variable we could even make KASLR avoid that reserved pstore region in its search for a viable KASLR offset.
Quoted 8 lines
Quoted 3 lines
I think I was hit by it once in all my testing. For our use case, the few times it fails to map is not going to affect what we need this for at all.
Once is pretty good. Do you know why? Also once out of how many runs? Is the randomness source not as random as it should be or are the number of bits for KASLR maybe so few on your target architecture that the odds of hitting anything become low? Do these same constraints hold true outside of your testing environment?
So I just ran it a hundred times in a loop. I added a patch to print the location of "
_text". The loop was this:for i in
seq 100; do
ssh root@debiantesting-x86-64 "dmesg | grep -e 'text starts' -e 'mapped boot' >> text; grub-reboot '1>0'; sleep 0.5; reboot" sleep 25
doneIt searches dmesg for my added printk as well as the print of were the ring buffer was loaded in physical memory.
It takes about 15 seconds to reboot, so I waited 25. The results are attached. I found that it was consistent 76 times, which means 1 out of 4 it's not. Funny enough, it broke whenever it loaded the kernel below
0x100000000. And then it would be off by a little.It was consistently at:
0x27d000000And when it failed, it was at
0x27ce00000.
Quoted 7 lines
Quoted 3 lines
Note, when I used the e820 tables to do this, I never saw a failure. My assumption is that when it is below
0x100000000, something else gets allocated causing this to get pushed down.Thinking about it again: What if you run the allocation super early (see arch/x86/boot/compressed/kaslr.c:handle_mem_options())?
That code is not used by EFI boot anymore.
In general, I would recommend (and have recommended) against these kinds of hacks in mainline, because -especially on x86- there is always someone that turns up with some kind of convoluted use case that gets broken if we try to change anything in the boot code.
I spent considerable time over the past year making the EFI/x86 boot code compatible with the new MS imposed requirements on PC boot firmware (related to secure boot and NX restrictions on memory mappings). This involved some radical refactoring of the boot sequence, including the KASLR logic. Adding fragile code there that will result in regressions observable to end users when it gets broken is really not what I'd like to see.
So I would personally prefer for this code not to go in at all. But if it does go in (and Steven has already agreed to this), it needs a giant disclaimer that it is best effort and may get broken inadvertently by changes that may seem unrelated.
Quoted 6 lines
If you stick to
allocating only from top, you're effectively kernel version independent for your allocations because none of the kernel code ran yet and definitely KASLR independent because you're running deterministically before KASLR even gets allocated.
Allocating top down under EFI is almost guaranteed to result in problems, because that is how the EFI page allocator works as well. This means that allocations will move around depending on, e.g., whether some USB stick was inserted on the first boot and removed on the second, or whether your external display was on or off.
Quoted 8 lines
Quoted 6 lines
As this code relies on
memblock_phys_alloc()being consistent, if something gets allocated before it differently depending on where the kernel is, it can also move the location. A plugin to UEFI would mean that it would need to reserve the memory, and the code here will need to know where it is. We could always make the functionreserve_mem()global and weak so that architectures can override it.
Quoted 5 lines
Yes, the in-kernel UEFI loader (efi-stub) could simply populate a new type of memblock with the respective reservations and you later call
memblock_find_in_range_node()instead ofmemblock_phys_alloc()to pass in flags that you want to allocate only from the newMEMBLOCK_RESERVE_MEMtype.
This would require the introduction of a new kind of memory map for the handover between EFI stub and core kernel. Currently, x86 relies on E820 and other architectures pass the EFI memory map unmodified.
Quoted 106 lines · Ard Biesheuvel
Quoted 82 lines · Alexander Graf
Quoted 24 lines · Steven Rostedt
Quoted 10 lines · On Mon, 17 Jun 2024 09:07:29 +0200 Alexander Graf<[email protected]> wrote:
Hey Steve,
I believe we're talking about 2 different things :). Let me rephrase a bit and make a concrete example.
Imagine you have passed the "reserve_mem=12M:4096:trace" kernel command line option. The kernel now comes up and allocates a random chunk of memory that - by (admittedly good) chance - may be at the same physical location as before. Let's assume it deemed
0x1000000as a good offset.Note, it's not random. It picks from the top of available memory every time. But things can mess with it (see below).
Quoted 8 lines
Let's now assume you're running on a UEFI system. There, you always have non-volatile storage available to you even in the pre-boot phase. That means the kernel could create a UEFI variable that says "12M:4096:trace ->
0x1000000". The pre-boot phase takes all these UEFI variables and marks them as reserved. When you finally reach your command line parsing logic for reserve_mem=, you can flip all reservations that were not on the command line back to normal memory.Quoted 8 lines
Quoted 6 lines
That way you have pretty much guaranteed persistent memory regions, even with KASLR changing your memory layout across boots.
The nice thing is that the above is an extension of what you've already
built: Systems with UEFI simply get better guarantees that their regions persist.This could be an added feature, but it is very architecture specific, and would likely need architecture specific updates.
It definitely would be an added feature, yes. But one that allows you to ensure persistence a lot more safely :).
Quoted 44 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 10 lines
Requirement:
Need a way to reserve memory that will be at a consistent location for every boot, if the kernel and system are the same. Does not need to work if rebooting to a different kernel, or if the system can change the memory layout between boots.
The reserved memory can not be an hard coded address, as the same kernel / command line needs to run on several different machines. The picked memory reservation just needs to be the same for a given machine, but may be
With KASLR is enabled, doesn't this approach break too often to be reliable enough for the data you want to extract?
Picking up the idea above, with a persistent variable we could even make KASLR avoid that reserved pstore region in its search for a viable KASLR offset.
Quoted 8 lines
Quoted 3 lines
I think I was hit by it once in all my testing. For our use case, the few times it fails to map is not going to affect what we need this for at all.
Once is pretty good. Do you know why? Also once out of how many runs? Is the randomness source not as random as it should be or are the number of bits for KASLR maybe so few on your target architecture that the odds of hitting anything become low? Do these same constraints hold true outside of your testing environment?
So I just ran it a hundred times in a loop. I added a patch to print the location of "
_text". The loop was this:for i in
seq 100; do
ssh root@debiantesting-x86-64 "dmesg | grep -e 'text starts' -e 'mapped boot' >> text; grub-reboot '1>0'; sleep 0.5; reboot" sleep 25
doneIt searches dmesg for my added printk as well as the print of were the ring buffer was loaded in physical memory.
It takes about 15 seconds to reboot, so I waited 25. The results are attached. I found that it was consistent 76 times, which means 1 out of 4 it's not. Funny enough, it broke whenever it loaded the kernel below
0x100000000. And then it would be off by a little.It was consistently at:
0x27d000000Quoted 8 lines
Quoted 5 lines
And when it failed, it was at
0x27ce00000.Note, when I used the e820 tables to do this, I never saw a failure. My assumption is that when it is below
0x100000000, something else gets allocated causing this to get pushed down.Thinking about it again: What if you run the allocation super early (see arch/x86/boot/compressed/kaslr.c:handle_mem_options())?
That code is not used by EFI boot anymore.
In general, I would recommend (and have recommended) against these kinds of hacks in mainline, because -especially on x86- there is always someone that turns up with some kind of convoluted use case that gets broken if we try to change anything in the boot code.
I spent considerable time over the past year making the EFI/x86 boot code compatible with the new MS imposed requirements on PC boot firmware (related to secure boot and NX restrictions on memory mappings). This involved some radical refactoring of the boot sequence, including the KASLR logic. Adding fragile code there that will result in regressions observable to end users when it gets broken is really not what I'd like to see.
Quoted 4 lines
So I would personally prefer for this code not to go in at all. But if it does go in (and Steven has already agreed to this), it needs a giant disclaimer that it is best effort and may get broken inadvertently by changes that may seem unrelated.
Alright, happy to rest my case about making it more reliable for now then :).
IMHO the big fat disclaimer should be in the argument name. "reserve_mem" to me sounds like it actually guarantees a reservation - which it doesn't. Can we name it more along the lines of "debug" (to indicate it's not for production data) or "phoenix" (usually gets reborn out of ashes, but you can never know for sure): "debug_mem", / "phoenix_mem"?
Quoted 7 lines
Quoted 7 lines
If you stick to
allocating only from top, you're effectively kernel version independent for your allocations because none of the kernel code ran yet and definitely KASLR independent because you're running deterministically before KASLR even gets allocated.
Quoted 5 lines
Allocating top down under EFI is almost guaranteed to result in problems, because that is how the EFI page allocator works as well. This means that allocations will move around depending on, e.g., whether some USB stick was inserted on the first boot and removed on the second, or whether your external display was on or off.
I believe most UEFI implementations only allocate top down in the lower 32bits. But yes, it's fragile, I hear you. Let's embrace the flaky nature of the beast then :).
Alex
Amazon Web Services Development Center Germany GmbH Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597
Steven Rostedt
maintainerQuoted 6 lines · On Tue, 18 Jun 2024 13:47:49 +0200 Alexander Graf <[email protected]> wrote:
IMHO the big fat disclaimer should be in the argument name. "
reserve_mem" to me sounds like it actually guarantees a reservation - which it doesn't. Can we name it more along the lines of "debug" (to indicate it's not for production data) or "phoenix" (usually gets reborn out of ashes, but you can never know for sure): "debug_mem", / "phoenix_mem"?
I don't see any reason it will not reserve memory. That doesn't seem to be the issue. What is not guaranteed is that it will be in the same location as last time. So the name is correct. It's not "reserve_consistent_memory" ;-)
-- Steve
Mike Rapoport
maintainerQuoted 6 lines · Ard Biesheuvel
On Mon, 17 Jun 2024 at 23:01, Alexander Graf <[email protected]> wrote:
So I would personally prefer for this code not to go in at all. But if it does go in (and Steven has already agreed to this), it needs a giant disclaimer that it is best effort and may get broken inadvertently by changes that may seem unrelated.
I think that reserve_mem= is way less intrusive than memmap= we anyway have.
It will reserve memory and the documentation adequately warns that the location of that memory might be at different physical addresses.
Signature · 2 lines
Sincerely yours, Mike.
From: Mike Rapoport (IBM) <[email protected]>
Quoted 16 lines · Steven Rostedt
Reserve unspecified location of physical memory from kernel command line
Background:
In ChromeOS, we have 1 MB of pstore ramoops reserved so that we can extract dmesg output and some other information when a crash happens in the field. (This is only done when the user selects "Allow Google to collect data for
improving the system"). But there are cases when there's a bug that
requires more data to be retrieved to figure out what is happening. We would like to increase the pstore size, either temporarily, or maybe even permanently. The pstore on these devices are at a fixed location in RAM (as the RAM is not cleared on soft reboots nor crashes). The location is chosen by the BIOS (coreboot) and passed to the kernel via ACPI tables on x86. There's a driver that queries for this to initialize the pstore for ChromeOS:
Quoted 1 line
[...]
Applied to for-next branch of memblock.git tree, thanks!
[0/2] mm/memblock: Add "reserve_mem" to reserved named memory at boot up
commit: 1e4c64b71c9bf230b25fde12cbcceacfdc8b3332
[1/2] mm/memblock: Add "reserve_mem" to reserved named memory at boot up
commit: 1e4c64b71c9bf230b25fde12cbcceacfdc8b3332
[2/2] pstore/ramoops: Add ramoops.mem_name= command line option
commit: d9d814eebb1ae9742e7fd7f39730653b16326bd4tree: https://git.kernel.org/pub/scm/linux/kernel/git/rppt/memblock
branch: for-next
--
Sincerely yours,
Mike.