Skip to contentSkip to conversation

ARM: dts: rainier,everest: Move reserved memory regions

PATCH Superseded Adriana Kobylak 2 participants 5 messages Sep 15, 2022 – Sep 21, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ARM: dts: rainier,everest: Move reserved memory regions

From: Adriana Kobylak <[email protected]>

Move the reserved regions to account for a decrease in DRAM when ECC is enabled. ECC takes 1/9th of memory, bringing the size down from 1024MiB to 912MiB (minus 16MiB for VGA) for a total of 896MiB available DRAM.

Move the regions by 128MiB since the flash_memory region needs to be aligned to 64MiB. This change doesn't affect the host if ECC is not enabled.

---
 arch/arm/boot/dts/aspeed-bmc-ibm-everest.dts | 11 ++++++-----
 arch/arm/boot/dts/aspeed-bmc-ibm-rainier.dts | 12 ++++++++----
 2 files changed, 14 insertions(+), 9 deletions(-)
arch/arm/boot/dts/aspeed-bmc-ibm-everest.dts +6 -5
index 1bba5ad7378e..9a09301dd79e 100644
@@ -163,15 +163,16 @@ reserved-memory {
 		ranges;
 
 		/* LPC FW cycle bridge region requires natural alignment */
-		flash_memory: region@b8000000 {
+		flash_memory: region@b0000000 {
 			no-map;
-			reg = <0xb8000000 0x04000000>; /* 64M */
+			reg = <0xb0000000 0x04000000>; /* 64M */
 		};
 
-		/* 48MB region from the end of flash to start of vga memory */
-		ramoops@bc000000 {
+		/* 176MB region from the end of flash to start of vga memory */
+
+		ramoops@b4000000 {
 			compatible = "ramoops";
-			reg = <0xbc000000 0x200000>; /* 16 * (4 * 0x8000) */
+			reg = <0xb4000000 0x200000>; /* 16 * (4 * 0x8000) */
 			record-size = <0x8000>;
 			console-size = <0x8000>;
 			ftrace-size = <0x8000>;
arch/arm/boot/dts/aspeed-bmc-ibm-rainier.dts +8 -4
index 8bd2f441b159..01e7d8f46dc7 100644
@@ -95,14 +95,17 @@ reserved-memory {
 		#size-cells = <1>;
 		ranges;
 
-		flash_memory: region@b8000000 {
+		/* LPC FW cycle bridge region requires natural alignment */
+		flash_memory: region@b0000000 {
 			no-map;
-			reg = <0xb8000000 0x04000000>; /* 64M */
+			reg = <0xb0000000 0x04000000>; /* 64M */
 		};
 
-		ramoops@bc000000 {
+		/* 176MB region from the end of flash to start of vga memory */
+
+		ramoops@b4000000 {
 			compatible = "ramoops";
-			reg = <0xbc000000 0x200000>; /* 16 * (4 * 0x8000) */
+			reg = <0xb4000000 0x200000>; /* 16 * (4 * 0x8000) */
 			record-size = <0x8000>;
 			console-size = <0x8000>;
 			ftrace-size = <0x8000>;
@@ -110,6 +113,7 @@ ramoops@bc000000 {
 			max-reason = <3>; /* KMSG_DUMP_EMERG */
 		};
 
+		/* VGA region is dictated by hardware strapping */
 		vga_memory: region@bf000000 {
 			no-map;
 			compatible = "shared-dma-pool";
Signature · 1 line
2.25.1
#1 lore raw

Hi Adriana,

Quoted 5 lines · Adriana Kobylak

From: Adriana Kobylak <[email protected]>

Move the reserved regions to account for a decrease in DRAM when ECC is enabled. ECC takes 1/9th of memory, bringing the size down from 1024MiB to 912MiB (minus 16MiB for VGA) for a total of 896MiB available DRAM.

The 912MiB mentioned here struck me, so I poked at the numbers:

ECC(1024MiB)
= ECC(1024 * 1024 * 1024)
= 1024 * 1024 * 1024 * 8 / 9
= 954437176.888888889
= 910.222222222MiB

From there subtracting 16MiB gets us 894.222222222MiB, which isn't a number we see in practice.

On the other hand:

ECC(1024 - 16)MiB)
= ECC((1024 - 16) * 1024 * 1024)
= (1024 - 16) * 1024 * 1024 * 8 / 9
= 939524096
= 896MiB

We do see 896MiB in practice which implies that MCR54 is configured for ECC to be bounded at the bottom of a 16MiB VGA memory region. 912MiB doesn't fall out of the numbers, so I think we need to rework the words to avoid confusion?

Quoted 4 lines

Move the regions by 128MiB since the flash_memory region needs to be aligned to 64MiB. This change doesn't affect the host if ECC is not enabled.

So I was curious about the 128MiB relocation as well. More on this below:

Quoted 13 lines
---
 arch/arm/boot/dts/aspeed-bmc-ibm-everest.dts | 11 ++++++-----
 arch/arm/boot/dts/aspeed-bmc-ibm-rainier.dts | 12 ++++++++----
 2 files changed, 14 insertions(+), 9 deletions(-)
arch/arm/boot/dts/aspeed-bmc-ibm-everest.dts +0 -0
index 1bba5ad7378e..9a09301dd79e 100644

arch/arm/boot/dts/aspeed-bmc-ibm-everest.dts reserved-memory { · line 169

Quoted 9 lines
arch/arm/boot/dts/aspeed-bmc-ibm-everest.dts +2 -2
@@ -163,15 +163,16 @@ reserved-memory {
 		ranges;
 
 		/* LPC FW cycle bridge region requires natural alignment */
-		flash_memory: region@b8000000 {
+		flash_memory: region@b0000000 {
 			no-map;
-			reg = <0xb8000000 0x04000000>; /* 64M */
+			reg = <0xb0000000 0x04000000>; /* 64M */
 		};

0xb8000000 is an offset 2944MiB into the address space. The RAM base address is in the middle of the address space, so 2944MiB - 2048MiB, which means the buffer is offset 896MiB into RAM. This drives the need for the patch with ECC, as our RAM address space is now only 896MiB in size and basing the buffer at that offset is obviously invalid. So change is necessary.

We know that the flash_memory buffer needs to be naturally aligned, so the fact that it was where it is we know that 896MiB is valid alignment (as it already works). We also have the ramoops buffer below, which I initially put after the flash_memory buffer to reduce the "hole" in between vga_memory and flash_memory. However there's no other reason to order it this way, and we can shift it as we see fit. Specifically, we're free to order it before the flash_memory buffer.

Focusing back on the flash_memory buffer, if 896MiB is 64MiB aligned, then (896-64)MiB is also aligned. This means we can butt the flash_memory buffer up against the "end" of RAM, and with the observation that we can put the ramoops buffer before it, we can butt the ramoops buffer up against the start of flash_memory. This minimises the address-space fragmentation.

So maybe we can go with this arrangement?

ramoops@b3e00000 {
    compatible = "ramoops";
    reg = <0xb3e00000 0x200000>;
    ...
}
flash_memory: region@b4000000 {
    no-map;
    reg = <0xb4000000 0x04000000>;
    ...
};

Thoughts?

Andrew

#2 lore raw

PATCH v2 1/1 ARM: dts: rainier,everest: Move reserved memory regions

From: Adriana Kobylak <[email protected]>

Move the reserved regions to account for a decrease in DRAM when ECC is enabled. ECC takes 1/9th of memory.

Running on HW with ECC off, u-boot prints:
DRAM: already initialized, 1008 MiB (capacity:1024 MiB, VGA:16 MiB, ECC:off)

And with ECC on, u-boot prints:
DRAM: already initialized, 896 MiB (capacity:1024 MiB, VGA:16 MiB, ECC:on, ECC size:896 MiB)

This implies that MCR54 is configured for ECC to be bounded at the bottom of a 16MiB VGA memory region:

1024MiB - 16MiB (VGA) = 1008MiB
1008MiB / 9 (for ECC) = 112MiB
1008MiB - 112MiB = 896MiB (available DRAM)

The flash_memory region currently starts at offset 896MiB:
0xb8000000 (flash_memory offset) - 0x80000000 (base memory address) = 0x38000000 = 896MiB

This is the end of the available DRAM with ECC enabled and therefore it needs to be moved.

Since the flash_memory is 64MiB in size and needs to be 64MiB aligned, it can just be moved up by 64MiB and would sit right at the end of the available DRAM buffer.

The ramoops region currently follows the flash_memory, but it can be moved to sit above flash_memory which would minimize the address-space fragmentation.

---
 arch/arm/boot/dts/aspeed-bmc-ibm-everest.dts | 17 ++++++++---------
 arch/arm/boot/dts/aspeed-bmc-ibm-rainier.dts | 16 +++++++++-------
 2 files changed, 17 insertions(+), 16 deletions(-)
arch/arm/boot/dts/aspeed-bmc-ibm-everest.dts +8 -9
index 1bba5ad7378e..3bdd79506704 100644
@@ -162,16 +162,9 @@ reserved-memory {
 		#size-cells = <1>;
 		ranges;
 
-		/* LPC FW cycle bridge region requires natural alignment */
-		flash_memory: region@b8000000 {
-			no-map;
-			reg = <0xb8000000 0x04000000>; /* 64M */
-		};
-
-		/* 48MB region from the end of flash to start of vga memory */
-		ramoops@bc000000 {
+		ramoops@b3e00000 {
 			compatible = "ramoops";
-			reg = <0xbc000000 0x200000>; /* 16 * (4 * 0x8000) */
+			reg = <0xb3e00000 0x200000>; /* 16 * (4 * 0x8000) */
 			record-size = <0x8000>;
 			console-size = <0x8000>;
 			ftrace-size = <0x8000>;
@@ -179,6 +172,12 @@ ramoops@bc000000 {
 			max-reason = <3>; /* KMSG_DUMP_EMERG */
 		};
 
+		/* LPC FW cycle bridge region requires natural alignment */
+		flash_memory: region@b4000000 {
+			no-map;
+			reg = <0xb4000000 0x04000000>; /* 64M */
+		};
+
 		/* VGA region is dictated by hardware strapping */
 		vga_memory: region@bf000000 {
 			no-map;
arch/arm/boot/dts/aspeed-bmc-ibm-rainier.dts +9 -7
index 8bd2f441b159..e1cb3c88368a 100644
@@ -95,14 +95,9 @@ reserved-memory {
 		#size-cells = <1>;
 		ranges;
 
-		flash_memory: region@b8000000 {
-			no-map;
-			reg = <0xb8000000 0x04000000>; /* 64M */
-		};
-
-		ramoops@bc000000 {
+		ramoops@b3e00000 {
 			compatible = "ramoops";
-			reg = <0xbc000000 0x200000>; /* 16 * (4 * 0x8000) */
+			reg = <0xb3e00000 0x200000>; /* 16 * (4 * 0x8000) */
 			record-size = <0x8000>;
 			console-size = <0x8000>;
 			ftrace-size = <0x8000>;
@@ -110,6 +105,13 @@ ramoops@bc000000 {
 			max-reason = <3>; /* KMSG_DUMP_EMERG */
 		};
 
+		/* LPC FW cycle bridge region requires natural alignment */
+		flash_memory: region@b4000000 {
+			no-map;
+			reg = <0xb4000000 0x04000000>; /* 64M */
+		};
+
+		/* VGA region is dictated by hardware strapping */
 		vga_memory: region@bf000000 {
 			no-map;
 			compatible = "shared-dma-pool";
Signature · 1 line
2.25.1
Reviewed-by on v2 1/1
#3 lore raw
Quoted 30 lines · Adriana Kobylak

From: Adriana Kobylak <[email protected]>

Move the reserved regions to account for a decrease in DRAM when ECC is enabled. ECC takes 1/9th of memory.

Running on HW with ECC off, u-boot prints:
DRAM: already initialized, 1008 MiB (capacity:1024 MiB, VGA:16 MiB, ECC:off)

And with ECC on, u-boot prints:
DRAM: already initialized, 896 MiB (capacity:1024 MiB, VGA:16 MiB, ECC:on, ECC size:896 MiB)

This implies that MCR54 is configured for ECC to be bounded at the bottom of a 16MiB VGA memory region:

1024MiB - 16MiB (VGA) = 1008MiB
1008MiB / 9 (for ECC) = 112MiB
1008MiB - 112MiB = 896MiB (available DRAM)

The flash_memory region currently starts at offset 896MiB:
0xb8000000 (flash_memory offset) - 0x80000000 (base memory address) = 0x38000000 = 896MiB

This is the end of the available DRAM with ECC enabled and therefore it needs to be moved.

Since the flash_memory is 64MiB in size and needs to be 64MiB aligned, it can just be moved up by 64MiB and would sit right at the end of the available DRAM buffer.

Quoted 5 lines

The ramoops region currently follows the flash_memory, but it can be moved to sit above flash_memory which would minimize the address-space fragmentation.

#4 lore raw
In reply to Andrew Jeffery
Quoted 98 lines · Andrew Jeffery

Hi Adriana,

Quoted 5 lines · Adriana Kobylak

From: Adriana Kobylak <[email protected] <mailto:[email protected]>>

Move the reserved regions to account for a decrease in DRAM when ECC is enabled. ECC takes 1/9th of memory, bringing the size down from 1024MiB to 912MiB (minus 16MiB for VGA) for a total of 896MiB available DRAM.

The 912MiB mentioned here struck me, so I poked at the numbers:

ECC(1024MiB)
= ECC(1024 * 1024 * 1024)
= 1024 * 1024 * 1024 * 8 / 9
= 954437176.888888889
= 910.222222222MiB

From there subtracting 16MiB gets us 894.222222222MiB, which isn't a number we see in practice.

On the other hand:

ECC(1024 - 16)MiB)
= ECC((1024 - 16) * 1024 * 1024)
= (1024 - 16) * 1024 * 1024 * 8 / 9
= 939524096
= 896MiB

We do see 896MiB in practice which implies that MCR54 is configured for ECC to be bounded at the bottom of a 16MiB VGA memory region. 912MiB doesn't fall out of the numbers, so I think we need to rework the words to avoid confusion?

Quoted 4 lines

Move the regions by 128MiB since the flash_memory region needs to be aligned to 64MiB. This change doesn't affect the host if ECC is not enabled.

So I was curious about the 128MiB relocation as well. More on this below:

Quoted 13 lines

---
arch/arm/boot/dts/aspeed-bmc-ibm-everest.dts | 11 ++++++----- arch/arm/boot/dts/aspeed-bmc-ibm-rainier.dts | 12 ++++++++----

2 files changed, 14 insertions(+), 9 deletions(-)
arch/arm/boot/dts/aspeed-bmc-ibm-everest.dts +0 -0
index 1bba5ad7378e..9a09301dd79e 100644

arch/arm/boot/dts/aspeed-bmc-ibm-everest.dts reserved-memory { · line 169

Quoted 9 lines
arch/arm/boot/dts/aspeed-bmc-ibm-everest.dts +2 -2
@@ -163,15 +163,16 @@ reserved-memory {
 		ranges;
 
 		/* LPC FW cycle bridge region requires natural alignment */
-		flash_memory: region@b8000000 {
+		flash_memory: region@b0000000 {
 			no-map;
-			reg = <0xb8000000 0x04000000>; /* 64M */
+			reg = <0xb0000000 0x04000000>; /* 64M */
 		};

0xb8000000 is an offset 2944MiB into the address space. The RAM base address is in the middle of the address space, so 2944MiB - 2048MiB, which means the buffer is offset 896MiB into RAM. This drives the need for the patch with ECC, as our RAM address space is now only 896MiB in size and basing the buffer at that offset is obviously invalid. So change is necessary.

We know that the flash_memory buffer needs to be naturally aligned, so the fact that it was where it is we know that 896MiB is valid alignment (as it already works). We also have the ramoops buffer below, which I initially put after the flash_memory buffer to reduce the "hole" in between vga_memory and flash_memory. However there's no other reason to order it this way, and we can shift it as we see fit. Specifically, we're free to order it before the flash_memory buffer.

Focusing back on the flash_memory buffer, if 896MiB is 64MiB aligned, then (896-64)MiB is also aligned. This means we can butt the flash_memory buffer up against the "end" of RAM, and with the observation that we can put the ramoops buffer before it, we can butt the ramoops buffer up against the start of flash_memory. This minimises the address-space fragmentation.

So maybe we can go with this arrangement?

ramoops@b3e00000 {
   compatible = "ramoops";
   reg = <0xb3e00000 0x200000>;
   ...
}
Quoted 7 lines
flash_memory: region@b4000000 {
   no-map;
   reg = <0xb4000000 0x04000000>;
   ...
};

Thoughts?

Thanks Andrew for the background and explanation. This would be even better like you mentioned to minimize the fragmentation. I’ll send a v2 with those changes.

Quoted 2 lines

Andrew