Skip to contentSkip to conversation

memblock: add no-map alloc functions

PATCH Superseded [email protected] 3 participants 16 messages Apr 16, 2024 – Apr 28, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 memblock: add no-map alloc functions

Quoted 9 lines
Quoted 6 lines

From: Daero Lee <[email protected]>

Like reserved-memory with the no-map property, there are memory regions need to be allocated in memblock.memory marked with the MEMBLOCK_NOMAP flag, but sholud not be allocated in memblock.reserved.

Can you please explain your use case?
Why do you need this functionality?

Thank you for your comments.
I added a example to the commit message.

Quoted 18 lines
Quoted 18 lines

So, functions were added that find the required memory area in memblock.memory, but do not allocate it to memblock.reserved.

The early_init_dt_alloc_reserved_memory_arch function was modified using the no-map alloc function.

---
 drivers/of/of_reserved_mem.c |  9 +++--
 mm/memblock.c                | 78 ++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+), 3 deletions(-)
drivers/of/of_reserved_mem.c +0 -0
index 8236ecae2953..504f2f60689c 100644
@@ -40,15 +40,18 @@ static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
 
Quoted 12 lines

drivers/of/of_reserved_mem.c static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size, · line 48

Quoted 9 lines
drivers/of/of_reserved_mem.c +6 -1
 	end = !end ? MEMBLOCK_ALLOC_ANYWHERE : end;
 	align = !align ? SMP_CACHE_BYTES : align;
-	base = memblock_phys_alloc_range(size, align, start, end);
+	if (nomap) {
+		base = memblock_phys_alloc_range_nomap(size, align, start, end);
+	} else {
+		base = memblock_phys_alloc_range(size, align, start, end);
+	}
+	

This changes behaviour of internal function, what effect will it have on the users?

I added explanation about this to the commit message, too.

Thank you.

#1 lore raw

PATCH v2 1/1 memblock: add no-map alloc functions

From: Daero Lee <[email protected]>

Like reserved-memory with the 'no-map' property and only 'size' property (w/o 'reg' property), there are memory regions need to be allocated in memblock.memory marked with the MEMBLOCK_NOMAP flag, but should not be allocated in memblock.reserved.

example : arch/arm64/boot/dts/freescale/fsl-ls1043a.dtsi

        reserved-memory {
                #address-cells = <2>;
                #size-cells = <2>;
                ranges;

                bman_fbpr: bman-fbpr {
                        compatible = "shared-dma-pool";
                        size = <0 0x1000000>;
                        alignment = <0 0x1000000>;
                        no-map;
                };

                qman_fqd: qman-fqd {
                        compatible = "shared-dma-pool";
                        size = <0 0x400000>;
                        alignment = <0 0x400000>;
                        no-map;
                };

                qman_pfdr: qman-pfdr {
                        compatible = "shared-dma-pool";
                        size = <0 0x2000000>;
                        alignment = <0 0x2000000>;
                        no-map;
                };
        };

So, functions were added that find the required memory area in memblock.memory, but do not allocate it to memblock.reserved.

In previous patch(a7259df), early_init_dt_alloc_reserved_memory was modified to use memblock_phys_alloc_range allocating memory in memblock.reserved, instead of memblock_find_in_range that just find the available region. But if there is a 'no-map' property, memory region should not be allocated to memblock.reserved.

So, the early_init_dt_alloc_reserved_memory_arch function was modified using the no-map alloc function.

---
 drivers/of/of_reserved_mem.c |  9 +++--
 mm/memblock.c                | 78 ++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+), 3 deletions(-)
drivers/of/of_reserved_mem.c +6 -3
index 8236ecae2953..504f2f60689c 100644
@@ -40,15 +40,18 @@ static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
 
 	end = !end ? MEMBLOCK_ALLOC_ANYWHERE : end;
 	align = !align ? SMP_CACHE_BYTES : align;
-	base = memblock_phys_alloc_range(size, align, start, end);
+	if (nomap) {
+		base = memblock_phys_alloc_range_nomap(size, align, start, end);
+	} else {
+		base = memblock_phys_alloc_range(size, align, start, end);
+	}
+	
 	if (!base)
 		return -ENOMEM;
 
 	*res_base = base;
 	if (nomap) {
 		err = memblock_mark_nomap(base, size);
-		if (err)
-			memblock_phys_free(base, size);
 	}
 
 	kmemleak_ignore_phys(base);
mm/memblock.c +78 -0

90 lines not on this page · raw message

Signature · 1 line
2.25.1
#2 lore raw
Quoted 6 lines · On Tue, Apr 16, 2024 at 09:06:35PM +0900, [email protected] wrote:

From: Daero Lee <[email protected]>

Like reserved-memory with the 'no-map' property and only 'size' property (w/o 'reg' property), there are memory regions need to be allocated in memblock.memory marked with the MEMBLOCK_NOMAP flag, but should not be allocated in memblock.reserved.

We don't "allocate" memory from memblock.memory directly.

We present memory in memblock.memory and "allocate" a memory by marking it in memblock.reserved.

Quoted 29 lines

example : arch/arm64/boot/dts/freescale/fsl-ls1043a.dtsi

       reserved-memory {
               #address-cells = <2>;
               #size-cells = <2>;
               ranges;

               bman_fbpr: bman-fbpr {
                       compatible = "shared-dma-pool";
                       size = <0 0x1000000>;
                       alignment = <0 0x1000000>;
                       no-map;
               };

               qman_fqd: qman-fqd {
                       compatible = "shared-dma-pool";
                       size = <0 0x400000>;
                       alignment = <0 0x400000>;
                       no-map;
               };

               qman_pfdr: qman-pfdr {
                       compatible = "shared-dma-pool";
                       size = <0 0x2000000>;
                       alignment = <0 0x2000000>;
                       no-map;
               };
       };
Quoted 4 lines

So, functions were added that find the required memory area in memblock.memory, but do not allocate it to memblock.reserved.

In previous patch(a7259df), early_init_dt_alloc_reserved_memory was

You want to say this patch introduced a regression?

Quoted 4 lines

modified to use memblock_phys_alloc_range allocating memory in memblock.reserved, instead of memblock_find_in_range that just find the available region. But if there is a 'no-map' property, memory region should not be allocated to memblock.reserved.

If my understanding is correct, memblock_phys_free() is called if 'no-map' property is set. This would release the "allocation" in memblock.reserved.

Quoted 23 lines

So, the early_init_dt_alloc_reserved_memory_arch function was modified using the no-map alloc function.

---
drivers/of/of_reserved_mem.c | 9 +++--
mm/memblock.c | 78 ++++++++++++++++++++++++++++++++++++

2 files changed, 84 insertions(+), 3 deletions(-)
drivers/of/of_reserved_mem.c +4 -1
index 8236ecae2953..504f2f60689c 100644
@@ -40,15 +40,18 @@ static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
 
 	end = !end ? MEMBLOCK_ALLOC_ANYWHERE : end;
 	align = !align ? SMP_CACHE_BYTES : align;
-	base = memblock_phys_alloc_range(size, align, start, end);
+	if (nomap) {
+		base = memblock_phys_alloc_range_nomap(size, align, start, end);
+	} else {
+		base = memblock_phys_alloc_range(size, align, start, end);

drivers/of/of_reserved_mem.c static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size, · line 51

Quoted 10 lines
drivers/of/of_reserved_mem.c +2 -2
+	}
+	
 	if (!base)
 		return -ENOMEM;
 
 	*res_base = base;
 	if (nomap) {
 		err = memblock_mark_nomap(base, size);
-		if (err)
-			memblock_phys_free(base, size);

I see you removed it here, seems does the same as before.

Quoted 102 lines
drivers/of/of_reserved_mem.c +0 -0
 	}
 
 	kmemleak_ignore_phys(base);
mm/memblock.c +78 -0
index d09136e040d3..f103f1ecbfad 100644
@@ -1506,6 +1506,72 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size,
 	return found;
 }
 
+phys_addr_t __init memblock_alloc_range_nid_nomap(phys_addr_t size,
+                                        phys_addr_t align, phys_addr_t start,
+                                        phys_addr_t end, int nid,
+                                        bool exact_nid)
+{
+        enum memblock_flags flags = choose_memblock_flags();
+        phys_addr_t found;
+
+        if (WARN_ONCE(nid == MAX_NUMNODES, "Usage of MAX_NUMNODES is deprecated. Use NUMA_NO_NODE instead\n"))
+                nid = NUMA_NO_NODE;
+
+        if (!align) {
+                /* Can't use WARNs this early in boot on powerpc */
+                dump_stack();
+                align = SMP_CACHE_BYTES;
+        }
+
+again:
+        found = memblock_find_in_range_node(size, align, start, end, nid,
+                                            flags);
+        if (found)
+                goto done;
+
+        if (nid != NUMA_NO_NODE && !exact_nid) {
+                found = memblock_find_in_range_node(size, align, start,
+                                                    end, NUMA_NO_NODE,
+                                                    flags);
+                if (found)
+                        goto done;
+        }
+
+        if (flags & MEMBLOCK_MIRROR) {
+                flags &= ~MEMBLOCK_MIRROR;
+                pr_warn_ratelimited("Could not allocate %pap bytes of mirrored memory\n",
+                        &size);
+                goto again;
+        }
+
+        return 0;
+
+done:
+        /*
+         * Skip kmemleak for those places like kasan_init() and
+         * early_pgtable_alloc() due to high volume.
+         */
+        if (end != MEMBLOCK_ALLOC_NOLEAKTRACE)
+                /*
+                 * Memblock allocated blocks are never reported as
+                 * leaks. This is because many of these blocks are
+                 * only referred via the physical address which is
+                 * not looked up by kmemleak.
+                 */
+                kmemleak_alloc_phys(found, size, 0);
+
+        /*
+         * Some Virtual Machine platforms, such as Intel TDX or AMD SEV-SNP,
+         * require memory to be accepted before it can be used by the
+         * guest.
+         *
+         * Accept the memory of the allocated buffer.
+         */
+        accept_memory(found, found + size);
+
+        return found;
+}
+
 /**
 * memblock_phys_alloc_range - allocate a memory block inside specified range
 * @size: size of memory block to be allocated in bytes
@@ -1530,6 +1596,18 @@ phys_addr_t __init memblock_phys_alloc_range(phys_addr_t size,
 					false);
 }
 
+phys_addr_t __init memblock_phys_alloc_range_nomap(phys_addr_t size,
+                                                   phys_addr_t align,
+                                                   phys_addr_t start,
+                                                   phys_addr_t end)
+{
+        memblock_dbg("%s: %llu bytes align=0x%llx from=%pa max_addr=%pa %pS\n",
+                     __func__, (u64)size, (u64)align, &start, &end,
+                     (void *)_RET_IP_);
+        return memblock_alloc_range_nid_nomap(size, align, start, end, 
+					      NUMA_NO_NODE, false);
+}
+
 /**
 * memblock_phys_alloc_try_nid - allocate a memory block from specified NUMA node
 * @size: size of memory block to be allocated in bytes
Signature · 1 line
2.25.1
Signature · 2 lines
Wei Yang
Help you, Help me
maintainer
#3 lore raw
In reply to [email protected]
Quoted 6 lines · On Tue, Apr 16, 2024 at 09:06:35PM +0900, [email protected] wrote:

From: Daero Lee <[email protected]>

Like reserved-memory with the 'no-map' property and only 'size' property (w/o 'reg' property), there are memory regions need to be allocated in memblock.memory marked with the MEMBLOCK_NOMAP flag, but should not be allocated in memblock.reserved.

This still does not explain why you need such regions.

As Wei Yang explained, memblock does not allocate memory from memblock.reserved. The memblock.reserved array represents memory that is in use by firmware or by early kernel allocations and cannot be freed to page allocator.

If you have a region that's _NOMAP in memblock.memory and is absent in memblock.reserved it will not be mapped by the kernel page tables, but it will be considered as free memory by the core mm.

Is this really what you want?

Quoted 28 lines

example : arch/arm64/boot/dts/freescale/fsl-ls1043a.dtsi

        reserved-memory {
                #address-cells = <2>;
                #size-cells = <2>;
                ranges;

                bman_fbpr: bman-fbpr {
                        compatible = "shared-dma-pool";
                        size = <0 0x1000000>;
                        alignment = <0 0x1000000>;
                        no-map;
                };

                qman_fqd: qman-fqd {
                        compatible = "shared-dma-pool";
                        size = <0 0x400000>;
                        alignment = <0 0x400000>;
                        no-map;
                };

                qman_pfdr: qman-pfdr {
                        compatible = "shared-dma-pool";
                        size = <0 0x2000000>;
                        alignment = <0 0x2000000>;
                        no-map;
                };
        };
Signature · 2 lines
Sincerely yours,
Mike.
author
#4 lore raw

2024년 4월 17일 (수) 오후 3:03, Mike Rapoport <[email protected]>님이 작성:

Quoted 9 lines · On Tue, Apr 16, 2024 at 09:06:35PM +0900, [email protected] wrote:
Quoted 6 lines

From: Daero Lee <[email protected]>

Like reserved-memory with the 'no-map' property and only 'size' property (w/o 'reg' property), there are memory regions need to be allocated in memblock.memory marked with the MEMBLOCK_NOMAP flag, but should not be allocated in memblock.reserved.

Quoted 6 lines

This still does not explain why you need such regions.

As Wei Yang explained, memblock does not allocate memory from memblock.reserved. The memblock.reserved array represents memory that is in use by firmware or by early kernel allocations and cannot be freed to page allocator.

Thank you for your comments. I used the wrong word.
When I use 'allocate', I mean that the region 'adds' to the memblock.reserved.

Quoted 6 lines

If you have a region that's _NOMAP in memblock.memory and is absent in memblock.reserved it will not be mapped by the kernel page tables, but it will be considered as free memory by the core mm.

Is this really what you want?

If my understanding is right, before freeing (memory && !reserved) area, we marked the memblock.reserved regions and memblock.memory regions with no-map flag. And when we free (memory && !reserved) area, we skip the memblock.memory regions with no-map(see should_skip_region). So, I think that the memory regions with no-map flag will not be considered as free memory.

If there is anything I think is wrong, feel free to correct me.

Regards,
DaeRo Lee

maintainer
#5 lore raw
Quoted 24 lines · DaeRo Lee

2024년 4월 17일 (수) 오후 3:03, Mike Rapoport <[email protected]>님이 작성:

Quoted 9 lines · On Tue, Apr 16, 2024 at 09:06:35PM +0900, [email protected] wrote:
Quoted 6 lines

From: Daero Lee <[email protected]>

Like reserved-memory with the 'no-map' property and only 'size' property (w/o 'reg' property), there are memory regions need to be allocated in memblock.memory marked with the MEMBLOCK_NOMAP flag, but should not be allocated in memblock.reserved.

Quoted 6 lines

This still does not explain why you need such regions.

As Wei Yang explained, memblock does not allocate memory from memblock.reserved. The memblock.reserved array represents memory that is in use by firmware or by early kernel allocations and cannot be freed to page allocator.

Thank you for your comments. I used the wrong word.
When I use 'allocate', I mean that the region 'adds' to the memblock.reserved.

Quoted 5 lines

If you have a region that's _NOMAP in memblock.memory and is absent in memblock.reserved it will not be mapped by the kernel page tables, but it will be considered as free memory by the core mm.

Quoted 7 lines
Quoted 1 line

Is this really what you want?

If my understanding is right, before freeing (memory && !reserved) area, we marked the memblock.reserved regions and memblock.memory regions with no-map flag. And when we free (memory && !reserved) area, we skip the memblock.memory regions with no-map(see should_skip_region). So, I think that the memory regions with no-map flag will not be considered as free memory.

You are right here.

But I still don't understand *why* do you want to change the way early_init_dt_alloc_reserved_memory_arch() works.

Quoted 2 lines

Regards,
DaeRo Lee

Signature · 2 lines
Sincerely yours,
Mike.
author
#6 lore raw

2024년 4월 19일 (금) 오전 3:04, Mike Rapoport <[email protected]>님이 작성:

Quoted 34 lines · DaeRo Lee
Quoted 24 lines

2024년 4월 17일 (수) 오후 3:03, Mike Rapoport <[email protected]>님이 작성:

Quoted 9 lines · On Tue, Apr 16, 2024 at 09:06:35PM +0900, [email protected] wrote:
Quoted 6 lines

From: Daero Lee <[email protected]>

Like reserved-memory with the 'no-map' property and only 'size' property (w/o 'reg' property), there are memory regions need to be allocated in memblock.memory marked with the MEMBLOCK_NOMAP flag, but should not be allocated in memblock.reserved.

Quoted 6 lines

This still does not explain why you need such regions.

As Wei Yang explained, memblock does not allocate memory from memblock.reserved. The memblock.reserved array represents memory that is in use by firmware or by early kernel allocations and cannot be freed to page allocator.

Thank you for your comments. I used the wrong word.
When I use 'allocate', I mean that the region 'adds' to the memblock.reserved.

Quoted 5 lines

If you have a region that's _NOMAP in memblock.memory and is absent in memblock.reserved it will not be mapped by the kernel page tables, but it will be considered as free memory by the core mm.

Quoted 7 lines
Quoted 1 line

Is this really what you want?

If my understanding is right, before freeing (memory && !reserved) area, we marked the memblock.reserved regions and memblock.memory regions with no-map flag. And when we free (memory && !reserved) area, we skip the memblock.memory regions with no-map(see should_skip_region). So, I think that the memory regions with no-map flag will not be considered as free memory.

Quoted 4 lines

You are right here.

But I still don't understand *why* do you want to change the way early_init_dt_alloc_reserved_memory_arch() works.

In memmap_init_reserved_pages, we mark memblock.reserved as PageReserved first and mark the memblock.reserved with nomap flag also.
-> Isn't this duplicated work? (If we add no-map region to memblock.reserved 'and' mark in memblock.memory..)
So, I think that for the no-map region, we don't need to add to the memblock.reserved.
This is what we do now in early_init_dt_reserve_memory. the nomap region is not added to the memblock.reserved.

In early_init_dt_alloc_reserved_memory_arch, if 'nomap' is true, we mark the memblock.memory region as _NOMAP. And if the return value 'err' is not zero(which is '-ENOMEM' from memblock_isolate_range), we free the region.
- 'nomap' is true -> memblock_mark_nomap : success -> not free the region

: fail -> free the region
And it can be said that we add the region to the memblock.reserved using memblock_phys_alloc_range and if the region is nomap, then we can free the region from memblock.reserved. But is it necessary to add it to memblock.reserved? We just need the region in memblock.memory to mark nomap.

So, here is what I think:
- reserved-memory w/ nomap region -> mark only to memblock.memory
- reserved-memory w/o nomap region -> add to the memblock.reserved

Regards,
DaeRo Lee

author
#7 lore raw

2024년 4월 19일 (금) 오전 10:46, DaeRo Lee <[email protected]>님이 작성:

Quoted 36 lines

2024년 4월 19일 (금) 오전 3:04, Mike Rapoport <[email protected]>님이 작성:

Quoted 34 lines · DaeRo Lee
Quoted 31 lines

2024년 4월 17일 (수) 오후 3:03, Mike Rapoport <[email protected]>님이 작성:

Quoted 9 lines · On Tue, Apr 16, 2024 at 09:06:35PM +0900, [email protected] wrote:
Quoted 6 lines

From: Daero Lee <[email protected]>

Like reserved-memory with the 'no-map' property and only 'size' property (w/o 'reg' property), there are memory regions need to be allocated in memblock.memory marked with the MEMBLOCK_NOMAP flag, but should not be allocated in memblock.reserved.

Quoted 6 lines

This still does not explain why you need such regions.

As Wei Yang explained, memblock does not allocate memory from memblock.reserved. The memblock.reserved array represents memory that is in use by firmware or by early kernel allocations and cannot be freed to page allocator.

Thank you for your comments. I used the wrong word.
When I use 'allocate', I mean that the region 'adds' to the memblock.reserved.

Quoted 6 lines

If you have a region that's _NOMAP in memblock.memory and is absent in memblock.reserved it will not be mapped by the kernel page tables, but it will be considered as free memory by the core mm.

Is this really what you want?

If my understanding is right, before freeing (memory && !reserved) area, we marked the memblock.reserved regions and memblock.memory regions with no-map flag. And when we free (memory && !reserved) area, we skip the memblock.memory regions with no-map(see should_skip_region). So, I think that the memory regions with no-map flag will not be considered as free memory.

Quoted 8 lines
Quoted 4 lines

You are right here.

But I still don't understand *why* do you want to change the way early_init_dt_alloc_reserved_memory_arch() works.

In memmap_init_reserved_pages, we mark memblock.reserved as PageReserved first and mark the memblock.reserved with nomap flag also.

Sorry. This is my mistake. 'memblock.memory with nomap flag' is right.

Quoted 26 lines

-> Isn't this duplicated work? (If we add no-map region to memblock.reserved 'and' mark in memblock.memory..)
So, I think that for the no-map region, we don't need to add to the memblock.reserved.
This is what we do now in early_init_dt_reserve_memory. the nomap region is not added to the memblock.reserved.

In early_init_dt_alloc_reserved_memory_arch, if 'nomap' is true, we mark the memblock.memory region as _NOMAP. And if the return value 'err' is not zero(which is '-ENOMEM' from memblock_isolate_range), we free the region.
- 'nomap' is true -> memblock_mark_nomap : success -> not free the region

: fail -> free the region
And it can be said that we add the region to the memblock.reserved using memblock_phys_alloc_range and if the region is nomap, then we can free the region from memblock.reserved. But is it necessary to add it to memblock.reserved? We just need the region in memblock.memory to mark nomap.

So, here is what I think:
- reserved-memory w/ nomap region -> mark only to memblock.memory
- reserved-memory w/o nomap region -> add to the memblock.reserved

Regards,
DaeRo Lee

author
#8 lore raw

2024년 4월 19일 (금) 오전 10:59, DaeRo Lee <[email protected]>님이 작성:

Quoted 48 lines

2024년 4월 19일 (금) 오전 10:46, DaeRo Lee <[email protected]>님이 작성:

Quoted 10 lines

2024년 4월 19일 (금) 오전 3:04, Mike Rapoport <[email protected]>님이 작성:

Quoted 8 lines · DaeRo Lee
Quoted 6 lines

2024년 4월 17일 (수) 오후 3:03, Mike Rapoport <[email protected]>님이 작성:

Quoted 5 lines · On Tue, Apr 16, 2024 at 09:06:35PM +0900, [email protected] wrote:
Quoted 3 lines

From: Daero Lee <[email protected]>

Like reserved-memory with the 'no-map' property and only 'size'

property

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

(w/o 'reg' property), there are memory regions need to be

allocated in

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

memblock.memory marked with the MEMBLOCK_NOMAP flag, but should

not be

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

allocated in memblock.reserved.

This still does not explain why you need such regions.

As Wei Yang explained, memblock does not allocate memory from memblock.reserved. The memblock.reserved array represents memory

that is in

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

use by firmware or by early kernel allocations and cannot be freed

to page

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

allocator.

Thank you for your comments. I used the wrong word. When I use 'allocate', I mean that the region 'adds' to the

memblock.reserved.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

If you have a region that's _NOMAP in memblock.memory and is

absent in

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

memblock.reserved it will not be mapped by the kernel page tables,

but it

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

will be considered as free memory by the core mm.

Is this really what you want?

If my understanding is right, before freeing (memory && !reserved) area, we marked the memblock.reserved regions and memblock.memory regions with no-map flag. And when we free (memory && !reserved)

area,

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

we skip the memblock.memory regions with no-map(see should_skip_region). So, I think that the memory regions with no-map flag will not be considered as free memory.

You are right here.

Quoted 35 lines
Quoted 6 lines
Quoted 2 lines

But I still don't understand *why* do you want to change the way early_init_dt_alloc_reserved_memory_arch() works.

In memmap_init_reserved_pages, we mark memblock.reserved as PageReserved first and mark the memblock.reserved with nomap flag also.

Sorry. This is my mistake. 'memblock.memory with nomap flag' is right.

Quoted 26 lines

-> Isn't this duplicated work? (If we add no-map region to memblock.reserved 'and' mark in memblock.memory..)
So, I think that for the no-map region, we don't need to add to the memblock.reserved.
This is what we do now in early_init_dt_reserve_memory. the nomap region is not added to the memblock.reserved.

In early_init_dt_alloc_reserved_memory_arch, if 'nomap' is true, we mark the memblock.memory region as _NOMAP. And if the return value 'err' is not zero(which is '-ENOMEM' from memblock_isolate_range), we free the region.
- 'nomap' is true -> memblock_mark_nomap : success -> not free the region

: fail -> free the region
And it can be said that we add the region to the memblock.reserved using memblock_phys_alloc_range and if the region is nomap, then we can free the region from memblock.reserved. But is it necessary to add it to memblock.reserved? We just need the region in memblock.memory to mark nomap.

So, here is what I think:
- reserved-memory w/ nomap region -> mark only to memblock.memory
- reserved-memory w/o nomap region -> add to the memblock.reserved

Regards,
DaeRo Lee

Hello
Can I get your opinions about this?
It will be very helpful to improve my understanding of memblock and reserved region.

Thank you

Quoted 1 line
author
#9 lore raw

2024년 4월 27일 (토) 오전 11:42, DaeRo Lee <[email protected]>님이 작성:

Quoted 90 lines

2024년 4월 19일 (금) 오전 10:59, DaeRo Lee <[email protected]>님이 작성:

Quoted 64 lines

2024년 4월 19일 (금) 오전 10:46, DaeRo Lee <[email protected]>님이 작성:

Quoted 7 lines

2024년 4월 19일 (금) 오전 3:04, Mike Rapoport <[email protected]>님이 작성:

Quoted 5 lines · DaeRo Lee
Quoted 3 lines

2024년 4월 17일 (수) 오후 3:03, Mike Rapoport <[email protected]>님이 작성:

Quoted 2 lines

On Tue, Apr 16, 2024 at 09:06:35PM +0900, [email protected]

Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

From: Daero Lee <[email protected]>

Like reserved-memory with the 'no-map' property and only 'size'

property

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

(w/o 'reg' property), there are memory regions need to be

allocated in

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

memblock.memory marked with the MEMBLOCK_NOMAP flag, but should

not be

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

allocated in memblock.reserved.

This still does not explain why you need such regions.

As Wei Yang explained, memblock does not allocate memory from memblock.reserved. The memblock.reserved array represents memory

that is in

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

use by firmware or by early kernel allocations and cannot be

freed to page

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

allocator.

Thank you for your comments. I used the wrong word. When I use 'allocate', I mean that the region 'adds' to the

memblock.reserved.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

If you have a region that's _NOMAP in memblock.memory and is

absent in

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

memblock.reserved it will not be mapped by the kernel page

tables, but it

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

will be considered as free memory by the core mm.

Is this really what you want?

If my understanding is right, before freeing (memory && !reserved) area, we marked the memblock.reserved regions and memblock.memory regions with no-map flag. And when we free (memory && !reserved)

area,

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

we skip the memblock.memory regions with no-map(see should_skip_region). So, I think that the memory regions with no-map flag will not be considered as free memory.

Quoted 8 lines
Quoted 4 lines

You are right here.

But I still don't understand *why* do you want to change the way early_init_dt_alloc_reserved_memory_arch() works.

In memmap_init_reserved_pages, we mark memblock.reserved as PageReserved first and mark the memblock.reserved with nomap flag also.

Sorry. This is my mistake. 'memblock.memory with nomap flag' is right.

Quoted 7 lines

-> Isn't this duplicated work? (If we add no-map region to memblock.reserved 'and' mark in memblock.memory..)
So, I think that for the no-map region, we don't need to add to the memblock.reserved.
This is what we do now in early_init_dt_reserve_memory. the nomap region is not added to the memblock.reserved.

Quoted 21 lines
Quoted 5 lines

In early_init_dt_alloc_reserved_memory_arch, if 'nomap' is true, we mark the memblock.memory region as _NOMAP. And if the return value 'err' is not zero(which is '-ENOMEM' from memblock_isolate_range), we free the region.
- 'nomap' is true -> memblock_mark_nomap : success -> not free the

region

Quoted 14 lines

: fail -> free the region
And it can be said that we add the region to the memblock.reserved using memblock_phys_alloc_range and if the region is nomap, then we can free the region from memblock.reserved. But is it necessary to add it to memblock.reserved? We just need the region in memblock.memory to mark nomap.

So, here is what I think:
- reserved-memory w/ nomap region -> mark only to memblock.memory
- reserved-memory w/o nomap region -> add to the memblock.reserved

Regards,
DaeRo Lee

Quoted 7 lines

Hello
Can I get your opinions about this?
It will be very helpful to improve my understanding of memblock and reserved region.

Thank you

Sorry for wrong email format.
I just re-send this for kernel mailing loop.

Quoted 1 line
maintainer
#10 lore raw
In reply to DaeRo Lee
Quoted 30 lines · DaeRo Lee

2024년 4월 19일 (금) 오전 10:46, DaeRo Lee <[email protected]>님이 작성:

Quoted 4 lines

In memmap_init_reserved_pages, we mark memblock.reserved as PageReserved first and mark the memblock.reserved with nomap flag also.

Sorry. This is my mistake. 'memblock.memory with nomap flag' is right.

Quoted 23 lines

-> Isn't this duplicated work? (If we add no-map region to memblock.reserved 'and' mark in memblock.memory..)
So, I think that for the no-map region, we don't need to add to the memblock.reserved.
This is what we do now in early_init_dt_reserve_memory. the nomap region is not added to the memblock.reserved.

In early_init_dt_alloc_reserved_memory_arch, if 'nomap' is true, we mark the memblock.memory region as _NOMAP. And if the return value 'err' is not zero(which is '-ENOMEM' from memblock_isolate_range), we free the region.
- 'nomap' is true -> memblock_mark_nomap : success -> not free the region

: fail -> free the region
And it can be said that we add the region to the memblock.reserved using memblock_phys_alloc_range and if the region is nomap, then we can free the region from memblock.reserved. But is it necessary to add it to memblock.reserved? We just need the region in memblock.memory to mark nomap.

So, here is what I think:
- reserved-memory w/ nomap region -> mark only to memblock.memory
- reserved-memory w/o nomap region -> add to the memblock.reserved

NOMAP and memblock.reserved are semantically different, and at makes sense to have a "reserved nomap" node in fdt recorded in both memblock.memory and memblock.reserved.

memblock.reserved represents the memory that is used by firmware or early kernel allocation, so reserved memory in fdt should be reserved in memblock as well. I believe it's an oversight that early_init_dt_reserve_memory() does not call memblock_reserve() for nomap memory.

NOMAP is a property of a memory region that says that that region should not be mapped in the linear map, it's not necessarily in use.

Quoted 2 lines
Quoted 2 lines

Regards,
DaeRo Lee

Signature · 2 lines
Sincerely yours,
Mike.
author
#11 lore raw

2024년 4월 27일 (토) 오후 5:50, Mike Rapoport <[email protected]>님이 작성:

Quoted 37 lines · DaeRo Lee
Quoted 30 lines

2024년 4월 19일 (금) 오전 10:46, DaeRo Lee <[email protected]>님이 작성:

Quoted 4 lines

In memmap_init_reserved_pages, we mark memblock.reserved as PageReserved first and mark the memblock.reserved with nomap flag also.

Sorry. This is my mistake. 'memblock.memory with nomap flag' is right.

Quoted 23 lines

-> Isn't this duplicated work? (If we add no-map region to memblock.reserved 'and' mark in memblock.memory..)
So, I think that for the no-map region, we don't need to add to the memblock.reserved.
This is what we do now in early_init_dt_reserve_memory. the nomap region is not added to the memblock.reserved.

In early_init_dt_alloc_reserved_memory_arch, if 'nomap' is true, we mark the memblock.memory region as _NOMAP. And if the return value 'err' is not zero(which is '-ENOMEM' from memblock_isolate_range), we free the region.
- 'nomap' is true -> memblock_mark_nomap : success -> not free the region

: fail -> free the region
And it can be said that we add the region to the memblock.reserved using memblock_phys_alloc_range and if the region is nomap, then we can free the region from memblock.reserved. But is it necessary to add it to memblock.reserved? We just need the region in memblock.memory to mark nomap.

So, here is what I think:
- reserved-memory w/ nomap region -> mark only to memblock.memory
- reserved-memory w/o nomap region -> add to the memblock.reserved

NOMAP and memblock.reserved are semantically different, and at makes sense to have a "reserved nomap" node in fdt recorded in both memblock.memory and memblock.reserved.

Quoted 7 lines

memblock.reserved represents the memory that is used by firmware or early kernel allocation, so reserved memory in fdt should be reserved in memblock as well. I believe it's an oversight that early_init_dt_reserve_memory() does not call memblock_reserve() for nomap memory.

NOMAP is a property of a memory region that says that that region should not be mapped in the linear map, it's not necessarily in use.

I agree that the NOMAP region should be added to memblock.reserved.

So, I think we need to clean-up memmap_init_reserved_pages, because in this function we call reserve_bootmem_region for memblock.reserved and memblock.memory with nomap. We don't need to call reserve_bootmem_region for nomap.

Regards,.
DaeRo Lee

maintainer
#12 lore raw
Quoted 46 lines · DaeRo Lee

2024년 4월 27일 (토) 오후 5:50, Mike Rapoport <[email protected]>님이 작성:

Quoted 37 lines · DaeRo Lee
Quoted 30 lines

2024년 4월 19일 (금) 오전 10:46, DaeRo Lee <[email protected]>님이 작성:

Quoted 4 lines

In memmap_init_reserved_pages, we mark memblock.reserved as PageReserved first and mark the memblock.reserved with nomap flag also.

Sorry. This is my mistake. 'memblock.memory with nomap flag' is right.

Quoted 23 lines

-> Isn't this duplicated work? (If we add no-map region to memblock.reserved 'and' mark in memblock.memory..)
So, I think that for the no-map region, we don't need to add to the memblock.reserved.
This is what we do now in early_init_dt_reserve_memory. the nomap region is not added to the memblock.reserved.

In early_init_dt_alloc_reserved_memory_arch, if 'nomap' is true, we mark the memblock.memory region as _NOMAP. And if the return value 'err' is not zero(which is '-ENOMEM' from memblock_isolate_range), we free the region.
- 'nomap' is true -> memblock_mark_nomap : success -> not free the region

: fail -> free the region
And it can be said that we add the region to the memblock.reserved using memblock_phys_alloc_range and if the region is nomap, then we can free the region from memblock.reserved. But is it necessary to add it to memblock.reserved? We just need the region in memblock.memory to mark nomap.

So, here is what I think:
- reserved-memory w/ nomap region -> mark only to memblock.memory
- reserved-memory w/o nomap region -> add to the memblock.reserved

NOMAP and memblock.reserved are semantically different, and at makes sense to have a "reserved nomap" node in fdt recorded in both memblock.memory and memblock.reserved.

Quoted 7 lines

memblock.reserved represents the memory that is used by firmware or early kernel allocation, so reserved memory in fdt should be reserved in memblock as well. I believe it's an oversight that early_init_dt_reserve_memory() does not call memblock_reserve() for nomap memory.

NOMAP is a property of a memory region that says that that region should not be mapped in the linear map, it's not necessarily in use.

Quoted 6 lines

I agree that the NOMAP region should be added to memblock.reserved.

So, I think we need to clean-up memmap_init_reserved_pages, because in this function we call reserve_bootmem_region for memblock.reserved and memblock.memory with nomap. We don't need to call reserve_bootmem_region for nomap.

Read the comment about memblock_mark_nomap()

Quoted 2 lines

Regards,.
DaeRo Lee

Signature · 2 lines
Sincerely yours,
Mike.
author
#13 lore raw

2024년 4월 28일 (일) 오후 3:35, Mike Rapoport <[email protected]>님이 작성:

Quoted 48 lines · DaeRo Lee
Quoted 46 lines

2024년 4월 27일 (토) 오후 5:50, Mike Rapoport <[email protected]>님이 작성:

Quoted 44 lines · DaeRo Lee
Quoted 30 lines

2024년 4월 19일 (금) 오전 10:46, DaeRo Lee <[email protected]>님이 작성:

Quoted 4 lines

In memmap_init_reserved_pages, we mark memblock.reserved as PageReserved first and mark the memblock.reserved with nomap flag also.

Sorry. This is my mistake. 'memblock.memory with nomap flag' is right.

Quoted 23 lines

-> Isn't this duplicated work? (If we add no-map region to memblock.reserved 'and' mark in memblock.memory..)
So, I think that for the no-map region, we don't need to add to the memblock.reserved.
This is what we do now in early_init_dt_reserve_memory. the nomap region is not added to the memblock.reserved.

In early_init_dt_alloc_reserved_memory_arch, if 'nomap' is true, we mark the memblock.memory region as _NOMAP. And if the return value 'err' is not zero(which is '-ENOMEM' from memblock_isolate_range), we free the region.
- 'nomap' is true -> memblock_mark_nomap : success -> not free the region

: fail -> free the region
And it can be said that we add the region to the memblock.reserved using memblock_phys_alloc_range and if the region is nomap, then we can free the region from memblock.reserved. But is it necessary to add it to memblock.reserved? We just need the region in memblock.memory to mark nomap.

So, here is what I think:
- reserved-memory w/ nomap region -> mark only to memblock.memory
- reserved-memory w/o nomap region -> add to the memblock.reserved

NOMAP and memblock.reserved are semantically different, and at makes sense to have a "reserved nomap" node in fdt recorded in both memblock.memory and memblock.reserved.

memblock.reserved represents the memory that is used by firmware or early kernel allocation, so reserved memory in fdt should be reserved in memblock as well. I believe it's an oversight that early_init_dt_reserve_memory() does not call memblock_reserve() for nomap memory.

NOMAP is a property of a memory region that says that that region should not be mapped in the linear map, it's not necessarily in use.

Quoted 8 lines
Quoted 6 lines

I agree that the NOMAP region should be added to memblock.reserved.

So, I think we need to clean-up memmap_init_reserved_pages, because in this function we call reserve_bootmem_region for memblock.reserved and memblock.memory with nomap. We don't need to call reserve_bootmem_region for nomap.

Read the comment about memblock_mark_nomap()

I read the comment about memblock_mark_nomap() and understood that regions with nomap flags should be treated as PageReserved. But, if we add this nomap region to memblock.reserved, the region with nomap flag will be processed in the first for-loop in memmap_init_reserved_pages.

Am I thinking wrong?

Regards,
DaeRo Lee

maintainer
#14 lore raw
Quoted 28 lines · DaeRo Lee

2024년 4월 28일 (일) 오후 3:35, Mike Rapoport <[email protected]>님이 작성:

Quoted 27 lines · DaeRo Lee
Quoted 24 lines

2024년 4월 27일 (토) 오후 5:50, Mike Rapoport <[email protected]>님이 작성:

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines

So, here is what I think:
- reserved-memory w/ nomap region -> mark only to memblock.memory
- reserved-memory w/o nomap region -> add to the memblock.reserved

NOMAP and memblock.reserved are semantically different, and at makes sense to have a "reserved nomap" node in fdt recorded in both memblock.memory and memblock.reserved.

Quoted 7 lines

memblock.reserved represents the memory that is used by firmware or early kernel allocation, so reserved memory in fdt should be reserved in memblock as well. I believe it's an oversight that early_init_dt_reserve_memory() does not call memblock_reserve() for nomap memory.

NOMAP is a property of a memory region that says that that region should not be mapped in the linear map, it's not necessarily in use.

I agree that the NOMAP region should be added to memblock.reserved.

So, I think we need to clean-up memmap_init_reserved_pages, because in this function we call reserve_bootmem_region for memblock.reserved and memblock.memory with nomap. We don't need to call reserve_bootmem_region for nomap.

Quoted 6 lines
Quoted 1 line

Read the comment about memblock_mark_nomap()

I read the comment about memblock_mark_nomap() and understood that regions with nomap flags should be treated as PageReserved. But, if we add this nomap region to memblock.reserved, the region with nomap flag will be processed in the first for-loop in memmap_init_reserved_pages.

memblock still must make sure that pages in nomap regions get PG_Reserved to be robust against potential errors and bugs in firmware parsing.

Quoted 4 lines

Am I thinking wrong?

Regards,
DaeRo Lee

Signature · 2 lines
Sincerely yours,
Mike.
author
#15 lore raw

2024년 4월 28일 (일) 오후 9:01, Mike Rapoport <[email protected]>님이 작성:

Quoted 31 lines · DaeRo Lee
Quoted 29 lines

2024년 4월 28일 (일) 오후 3:35, Mike Rapoport <[email protected]>님이 작성:

Quoted 28 lines · DaeRo Lee
Quoted 18 lines

2024년 4월 27일 (토) 오후 5:50, Mike Rapoport <[email protected]>님이 작성:

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines

So, here is what I think:
- reserved-memory w/ nomap region -> mark only to memblock.memory
- reserved-memory w/o nomap region -> add to the memblock.reserved

NOMAP and memblock.reserved are semantically different, and at makes sense to have a "reserved nomap" node in fdt recorded in both memblock.memory and memblock.reserved.

Quoted 7 lines

memblock.reserved represents the memory that is used by firmware or early kernel allocation, so reserved memory in fdt should be reserved in memblock as well. I believe it's an oversight that early_init_dt_reserve_memory() does not call memblock_reserve() for nomap memory.

NOMAP is a property of a memory region that says that that region should not be mapped in the linear map, it's not necessarily in use.

Quoted 6 lines

I agree that the NOMAP region should be added to memblock.reserved.

So, I think we need to clean-up memmap_init_reserved_pages, because in this function we call reserve_bootmem_region for memblock.reserved and memblock.memory with nomap. We don't need to call reserve_bootmem_region for nomap.

Read the comment about memblock_mark_nomap()

Quoted 8 lines
Quoted 5 lines

I read the comment about memblock_mark_nomap() and understood that regions with nomap flags should be treated as PageReserved. But, if we add this nomap region to memblock.reserved, the region with nomap flag will be processed in the first for-loop in memmap_init_reserved_pages.

memblock still must make sure that pages in nomap regions get PG_Reserved to be robust against potential errors and bugs in firmware parsing.

Got it.
Thank you for your comments. I'll make another patch for just clean-up early_init_dt_reserve_memory()

Thank you.

Regards,
DaeRo Lee