Skip to contentSkip to conversation

x86: fsp: Use mtrr_set_next_var() for graphics memory

PATCH In review Bin Meng 2 participants 17 messages Jul 21, 2023 – Jul 31, 2023

Subsystems (2)
lore
Share & tools
lore

Bin Meng

author
#0 lore raw

PATCH 1/5 x86: fsp: Use mtrr_set_next_var() for graphics memory

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/lib/fsp/fsp_graphics.c +1 -2
index 2bcc49f605..09d5da8c84 100644
@@ -110,8 +110,7 @@ static int fsp_video_probe(struct udevice *dev)
 	if (ret)
 		goto err;
 
-	mtrr_add_request(MTRR_TYPE_WRCOMB, vesa->phys_base_ptr, 256 << 20);
-	mtrr_commit(true);
+	mtrr_set_next_var(MTRR_TYPE_WRCOMB, vesa->phys_base_ptr, 256 << 20);
 
 	printf("%dx%dx%d @ %x\n", uc_priv->xsize, uc_priv->ysize,
 	       vesa->bits_per_pixel, vesa->phys_base_ptr);
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 2/5 video: broadwell: Use mtrr_set_next_var() for graphics memory

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Correct this to use mtrr_set_next_var() instead.

---

 drivers/video/broadwell_igd.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)
drivers/video/broadwell_igd.c +3 -7
index 6aa4e27071..83b6c908a8 100644
@@ -693,13 +693,9 @@ static int broadwell_igd_probe(struct udevice *dev)
 
 	/* Use write-combining for the graphics memory, 256MB */
 	fbbase = IS_ENABLED(CONFIG_VIDEO_COPY) ? plat->copy_base : plat->base;
-	ret = mtrr_add_request(MTRR_TYPE_WRCOMB, fbbase, 256 << 20);
-	if (!ret)
-		ret = mtrr_commit(true);
-	if (ret && ret != -ENOSYS) {
-		printf("Failed to add MTRR: Display will be slow (err %d)\n",
-		       ret);
-	}
+	ret = mtrr_set_next_var(MTRR_TYPE_WRCOMB, fbbase, 256 << 20);
+	if (ret)
+		printf("Failed to add MTRR: Display will be slow (err %d)\n", ret);
 
 	debug("fb=%lx, size %x, display size=%d %d %d\n", plat->base,
 	      plat->size, uc_priv->xsize, uc_priv->ysize, uc_priv->bpix);
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH 3/5 video: ivybridge: Use mtrr_set_next_var() for graphics memory

In reply to Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Correct this to use mtrr_set_next_var() instead.

---

 drivers/video/ivybridge_igd.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/video/ivybridge_igd.c +1 -2
index 9264dd6770..c2cc976618 100644
@@ -774,8 +774,7 @@ static int bd82x6x_video_probe(struct udevice *dev)
 
 	/* Use write-combining for the graphics memory, 256MB */
 	fbbase = IS_ENABLED(CONFIG_VIDEO_COPY) ? plat->copy_base : plat->base;
-	mtrr_add_request(MTRR_TYPE_WRCOMB, fbbase, 256 << 20);
-	mtrr_commit(true);
+	mtrr_set_next_var(MTRR_TYPE_WRCOMB, fbbase, 256 << 20);
 
 	return 0;
 }
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH 4/5 video: vesa: Use mtrr_set_next_var() for graphics memory

In reply to Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Correct this to use mtrr_set_next_var() instead.

---

 drivers/video/vesa.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/video/vesa.c +1 -2
index cac3bb0c33..50912c5c8b 100644
@@ -23,8 +23,7 @@ static int vesa_video_probe(struct udevice *dev)
 
 	/* Use write-combining for the graphics memory, 256MB */
 	fbbase = IS_ENABLED(CONFIG_VIDEO_COPY) ? plat->copy_base : plat->base;
-	mtrr_add_request(MTRR_TYPE_WRCOMB, fbbase, 256 << 20);
-	mtrr_commit(true);
+	mtrr_set_next_var(MTRR_TYPE_WRCOMB, fbbase, 256 << 20);
 
 	return 0;
 }
Signature · 1 line
2.34.1
author
#4 lore raw

PATCH 5/5 x86: Return mtrr_add_request() to its old purpose

In reply to Bin Meng

From: Simon Glass <[email protected]>

This function used to be for adding a list of requests to be actioned on relocation. Revert it back to this purpose, to avoid problems with boards which need control of their MTRRs (i.e. those which don't use FSP).

The mtrr_set_next_var() function is available when the next free variable-MTRR must be set, so this can be used instead.

Fixes 3bcd6cf89ef ("x86: mtrr: Skip MSRs that were already programmed..")
Fixes 596bd0589ad ("x86: mtrr: Do not clear the unused ones..")

---

 arch/x86/cpu/mtrr.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/cpu/mtrr.c +5 -1
index d57fcface0..9c24ae984e 100644
@@ -166,8 +166,12 @@ int mtrr_commit(bool do_caches)
 	debug("open done\n");
 	qsort(req, gd->arch.mtrr_req_count, sizeof(*req), h_comp_mtrr);
 	for (i = 0; i < gd->arch.mtrr_req_count; i++, req++)
-		mtrr_set_next_var(req->type, req->start, req->size);
+		set_var_mtrr(i, req->type, req->start, req->size);
 
+	/* Clear the ones that are unused */
+	debug("clear\n");
+	for (; i < mtrr_get_var_count(); i++)
+		wrmsrl(MTRR_PHYS_MASK_MSR(i), 0);
 	debug("close\n");
 	mtrr_close(&state, do_caches);
 	debug("mtrr done\n");
Signature · 1 line
2.34.1
#5 lore raw
In reply to Bin Meng

Hi Bin,

Quoted 16 lines · Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Quoted 7 lines

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Thanks for looking into this. The series works fine on link. I suspect it will be find on samus too, but I cannot test right now. Sadly minnowmax is also dead right now but I hope to fix it soon. I don't expect any problems there.

However, for coral, this first patch breaks the mtrrs. With master we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

with this patch on coral we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

At present coral expects to handle the MTRRs itself, and it seems that perhaps the APL FSPv2 does not? Do we need a new Kconfig for dealing with FSPv2 perhaps?

Regards,
Simon

author
#6 lore raw

Hi Simon,

Quoted 59 lines · Simon Glass

Hi Bin,

Quoted 16 lines · Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Quoted 7 lines

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Thanks for looking into this. The series works fine on link. I suspect it will be find on samus too, but I cannot test right now. Sadly minnowmax is also dead right now but I hope to fix it soon. I don't expect any problems there.

However, for coral, this first patch breaks the mtrrs. With master we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

with this patch on coral we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
Quoted 3 lines

At present coral expects to handle the MTRRs itself, and it seems that perhaps the APL FSPv2 does not? Do we need a new Kconfig for dealing with FSPv2 perhaps?

I am a little bit confused. The comment in arch/x86/lib/fsp/fsp_dram.c::
dram_init_banksize() says:

/*
* For FSP1, the system memory and reserved memory used by FSP are
* already programmed in the MTRR by FSP. Also it is observed that
* FSP on Intel Queensbay platform reports the TSEG memory range
* that has the same RES_MEM_RESERVED resource type whose address
* is programmed by FSP to be near the top of 4 GiB space, which is
* not what we want for DRAM.
*
* However it seems FSP2's behavior is different. We need to add the
* DRAM range in MTRR otherwise the boot process goes very slowly,
* which was observed on Chromebook Coral with FSP2.
*/

So on Coral with FSP2, U-Boot programs the MTTR by itself.

In this dram_init_banksize(), it calls mtrr_add_request() 3 times, 2 of which should be what you were seeing as 2 and 4 below:

Quoted 3 lines
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000

The #3 should be the FSP graphics frame buffer. But I failed to understand how the FSP graphics programs a MTRR register in between the 2 memory regions programmed by dram_init_banksize() on u-boot/master, how could that happen?

On the other hand, with this patch, how could the FSP graphics memory programs a MTRR register that should be programmed by dram_init_banksize()?

Regards,
Bin

#7 lore raw

Hi Bin,

Quoted 88 lines · Bin Meng

Hi Simon,

Quoted 59 lines · Simon Glass

Hi Bin,

Quoted 16 lines · Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Quoted 7 lines

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Thanks for looking into this. The series works fine on link. I suspect it will be find on samus too, but I cannot test right now. Sadly minnowmax is also dead right now but I hope to fix it soon. I don't expect any problems there.

However, for coral, this first patch breaks the mtrrs. With master we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

with this patch on coral we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
Quoted 3 lines

At present coral expects to handle the MTRRs itself, and it seems that perhaps the APL FSPv2 does not? Do we need a new Kconfig for dealing with FSPv2 perhaps?

I am a little bit confused. The comment in arch/x86/lib/fsp/fsp_dram.c::
dram_init_banksize() says:

/*
* For FSP1, the system memory and reserved memory used by FSP are
* already programmed in the MTRR by FSP. Also it is observed that
* FSP on Intel Queensbay platform reports the TSEG memory range
* that has the same RES_MEM_RESERVED resource type whose address
* is programmed by FSP to be near the top of 4 GiB space, which is
* not what we want for DRAM.
*
* However it seems FSP2's behavior is different. We need to add the
* DRAM range in MTRR otherwise the boot process goes very slowly,
* which was observed on Chromebook Coral with FSP2.
*/

So on Coral with FSP2, U-Boot programs the MTTR by itself.

In this dram_init_banksize(), it calls mtrr_add_request() 3 times, 2 of which should be what you were seeing as 2 and 4 below:

Quoted 8 lines
Quoted 3 lines
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000

The #3 should be the FSP graphics frame buffer. But I failed to understand how the FSP graphics programs a MTRR register in between the 2 memory regions programmed by dram_init_banksize() on u-boot/master, how could that happen?

Remember that the MTRRs are sorted, so the order or mtrr_add_request() calls does not matter.

Quoted 4 lines

On the other hand, with this patch, how could the FSP graphics memory programs a MTRR register that should be programmed by dram_init_banksize()?

I believe this adds a new mtrr and then commits the result.

Regards,
Simon

author
#8 lore raw

Hi Simon,

Quoted 96 lines · Simon Glass

Hi Bin,

Quoted 92 lines · Bin Meng

Hi Simon,

Quoted 59 lines · Simon Glass

Hi Bin,

Quoted 16 lines · Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Quoted 7 lines

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Thanks for looking into this. The series works fine on link. I suspect it will be find on samus too, but I cannot test right now. Sadly minnowmax is also dead right now but I hope to fix it soon. I don't expect any problems there.

However, for coral, this first patch breaks the mtrrs. With master we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

with this patch on coral we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
Quoted 3 lines

At present coral expects to handle the MTRRs itself, and it seems that perhaps the APL FSPv2 does not? Do we need a new Kconfig for dealing with FSPv2 perhaps?

I am a little bit confused. The comment in arch/x86/lib/fsp/fsp_dram.c::
dram_init_banksize() says:

/*
* For FSP1, the system memory and reserved memory used by FSP are
* already programmed in the MTRR by FSP. Also it is observed that
* FSP on Intel Queensbay platform reports the TSEG memory range
* that has the same RES_MEM_RESERVED resource type whose address
* is programmed by FSP to be near the top of 4 GiB space, which is
* not what we want for DRAM.
*
* However it seems FSP2's behavior is different. We need to add the
* DRAM range in MTRR otherwise the boot process goes very slowly,
* which was observed on Chromebook Coral with FSP2.
*/

So on Coral with FSP2, U-Boot programs the MTTR by itself.

In this dram_init_banksize(), it calls mtrr_add_request() 3 times, 2 of which should be what you were seeing as 2 and 4 below:

Quoted 3 lines
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 8 lines
Quoted 4 lines

The #3 should be the FSP graphics frame buffer. But I failed to understand how the FSP graphics programs a MTRR register in between the 2 memory regions programmed by dram_init_banksize() on u-boot/master, how could that happen?

Remember that the MTRRs are sorted, so the order or mtrr_add_request() calls does not matter.

Still cannot explain.

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000

After we sort the mtrr memory range, #2 whose base is 0x0 should have been put to the first entry, then followed by #3 whose base is 0xb0000000.

Regards,
Bin

#9 lore raw

Hi Bin,

Quoted 117 lines · Bin Meng

Hi Simon,

Quoted 96 lines · Simon Glass

Hi Bin,

Quoted 92 lines · Bin Meng

Hi Simon,

Quoted 59 lines · Simon Glass

Hi Bin,

Quoted 16 lines · Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Quoted 7 lines

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Thanks for looking into this. The series works fine on link. I suspect it will be find on samus too, but I cannot test right now. Sadly minnowmax is also dead right now but I hope to fix it soon. I don't expect any problems there.

However, for coral, this first patch breaks the mtrrs. With master we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

with this patch on coral we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
Quoted 3 lines

At present coral expects to handle the MTRRs itself, and it seems that perhaps the APL FSPv2 does not? Do we need a new Kconfig for dealing with FSPv2 perhaps?

I am a little bit confused. The comment in arch/x86/lib/fsp/fsp_dram.c::
dram_init_banksize() says:

/*
* For FSP1, the system memory and reserved memory used by FSP are
* already programmed in the MTRR by FSP. Also it is observed that
* FSP on Intel Queensbay platform reports the TSEG memory range
* that has the same RES_MEM_RESERVED resource type whose address
* is programmed by FSP to be near the top of 4 GiB space, which is
* not what we want for DRAM.
*
* However it seems FSP2's behavior is different. We need to add the
* DRAM range in MTRR otherwise the boot process goes very slowly,
* which was observed on Chromebook Coral with FSP2.
*/

So on Coral with FSP2, U-Boot programs the MTTR by itself.

In this dram_init_banksize(), it calls mtrr_add_request() 3 times, 2 of which should be what you were seeing as 2 and 4 below:

Quoted 3 lines
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 8 lines
Quoted 4 lines

The #3 should be the FSP graphics frame buffer. But I failed to understand how the FSP graphics programs a MTRR register in between the 2 memory regions programmed by dram_init_banksize() on u-boot/master, how could that happen?

Remember that the MTRRs are sorted, so the order or mtrr_add_request() calls does not matter.

Still cannot explain.

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 3 lines

After we sort the mtrr memory range, #2 whose base is 0x0 should have been put to the first entry, then followed by #3 whose base is 0xb0000000.

Right, but the thing is, your first patch does not revert the behaviour of mtrr_add_request(). It is still just adding to the end.

i.e. mtrr_commt() adds new ones but does not overwrite those at the back.

Looking at your full series, this is what I see on coral:

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000

But I see that do_mtrr is wrong for coral in init_cache_f_r():

do_mtrr &= !IS_ENABLED(CONFIG_SPL) &&

So with coral the mtrrs are never written?

Regards,
Simon

author
#10 lore raw

Hi Simon,

Quoted 136 lines · Simon Glass

Hi Bin,

Quoted 117 lines · Bin Meng

Hi Simon,

Quoted 96 lines · Simon Glass

Hi Bin,

Quoted 92 lines · Bin Meng

Hi Simon,

Quoted 59 lines · Simon Glass

Hi Bin,

Quoted 16 lines · Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Quoted 7 lines

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Thanks for looking into this. The series works fine on link. I suspect it will be find on samus too, but I cannot test right now. Sadly minnowmax is also dead right now but I hope to fix it soon. I don't expect any problems there.

However, for coral, this first patch breaks the mtrrs. With master we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

with this patch on coral we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
Quoted 3 lines

At present coral expects to handle the MTRRs itself, and it seems that perhaps the APL FSPv2 does not? Do we need a new Kconfig for dealing with FSPv2 perhaps?

I am a little bit confused. The comment in arch/x86/lib/fsp/fsp_dram.c::
dram_init_banksize() says:

/*
* For FSP1, the system memory and reserved memory used by FSP are
* already programmed in the MTRR by FSP. Also it is observed that
* FSP on Intel Queensbay platform reports the TSEG memory range
* that has the same RES_MEM_RESERVED resource type whose address
* is programmed by FSP to be near the top of 4 GiB space, which is
* not what we want for DRAM.
*
* However it seems FSP2's behavior is different. We need to add the
* DRAM range in MTRR otherwise the boot process goes very slowly,
* which was observed on Chromebook Coral with FSP2.
*/

So on Coral with FSP2, U-Boot programs the MTTR by itself.

In this dram_init_banksize(), it calls mtrr_add_request() 3 times, 2 of which should be what you were seeing as 2 and 4 below:

Quoted 3 lines
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 8 lines
Quoted 4 lines

The #3 should be the FSP graphics frame buffer. But I failed to understand how the FSP graphics programs a MTRR register in between the 2 memory regions programmed by dram_init_banksize() on u-boot/master, how could that happen?

Remember that the MTRRs are sorted, so the order or mtrr_add_request() calls does not matter.

Still cannot explain.

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 3 lines

After we sort the mtrr memory range, #2 whose base is 0x0 should have been put to the first entry, then followed by #3 whose base is 0xb0000000.

Right, but the thing is, your first patch does not revert the behaviour of mtrr_add_request(). It is still just adding to the end.

i.e. mtrr_commt() adds new ones but does not overwrite those at the back.

Looking at your full series, this is what I see on coral:

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
Quoted 5 lines

But I see that do_mtrr is wrong for coral in init_cache_f_r():

do_mtrr &= !IS_ENABLED(CONFIG_SPL) &&

So with coral the mtrrs are never written?

Yes, it seems this place is the culprit. The comment says:

* Note: if there is an SPL, then it has already set up MTRRs so we
* don't need to do that here

So on Coral, the assumption of SPL programming MTRRs is wrong.

Maybe we should do:

        bool do_mtrr = CONFIG_IS_ENABLED(X86_32BIT_INIT);

do_mtrr &= (!IS_ENABLED(CONFIG_SPL) || IS_ENABLED(CONFIG_FSP_VERSION2)) &&

        !IS_ENABLED(CONFIG_FSP_VERSION1) &&
        !IS_ENABLED(CONFIG_SYS_SLIMBOOTLOADER);

Will this work?

Regards,
Bin

#11 lore raw

Hi Bin,

Quoted 155 lines · Bin Meng

Hi Simon,

Quoted 136 lines · Simon Glass

Hi Bin,

Quoted 117 lines · Bin Meng

Hi Simon,

Quoted 96 lines · Simon Glass

Hi Bin,

Quoted 92 lines · Bin Meng

Hi Simon,

Quoted 59 lines · Simon Glass

Hi Bin,

Quoted 16 lines · Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Quoted 7 lines

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Thanks for looking into this. The series works fine on link. I suspect it will be find on samus too, but I cannot test right now. Sadly minnowmax is also dead right now but I hope to fix it soon. I don't expect any problems there.

However, for coral, this first patch breaks the mtrrs. With master we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

with this patch on coral we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
Quoted 3 lines

At present coral expects to handle the MTRRs itself, and it seems that perhaps the APL FSPv2 does not? Do we need a new Kconfig for dealing with FSPv2 perhaps?

I am a little bit confused. The comment in arch/x86/lib/fsp/fsp_dram.c::
dram_init_banksize() says:

/*
* For FSP1, the system memory and reserved memory used by FSP are
* already programmed in the MTRR by FSP. Also it is observed that
* FSP on Intel Queensbay platform reports the TSEG memory range
* that has the same RES_MEM_RESERVED resource type whose address
* is programmed by FSP to be near the top of 4 GiB space, which is
* not what we want for DRAM.
*
* However it seems FSP2's behavior is different. We need to add the
* DRAM range in MTRR otherwise the boot process goes very slowly,
* which was observed on Chromebook Coral with FSP2.
*/

So on Coral with FSP2, U-Boot programs the MTTR by itself.

In this dram_init_banksize(), it calls mtrr_add_request() 3 times, 2 of which should be what you were seeing as 2 and 4 below:

Quoted 3 lines
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 8 lines
Quoted 4 lines

The #3 should be the FSP graphics frame buffer. But I failed to understand how the FSP graphics programs a MTRR register in between the 2 memory regions programmed by dram_init_banksize() on u-boot/master, how could that happen?

Remember that the MTRRs are sorted, so the order or mtrr_add_request() calls does not matter.

Still cannot explain.

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 3 lines

After we sort the mtrr memory range, #2 whose base is 0x0 should have been put to the first entry, then followed by #3 whose base is 0xb0000000.

Right, but the thing is, your first patch does not revert the behaviour of mtrr_add_request(). It is still just adding to the end.

i.e. mtrr_commt() adds new ones but does not overwrite those at the back.

Looking at your full series, this is what I see on coral:

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
Quoted 5 lines

But I see that do_mtrr is wrong for coral in init_cache_f_r():

do_mtrr &= !IS_ENABLED(CONFIG_SPL) &&

So with coral the mtrrs are never written?

Yes, it seems this place is the culprit. The comment says:

* Note: if there is an SPL, then it has already set up MTRRs so we
* don't need to do that here

So on Coral, the assumption of SPL programming MTRRs is wrong.

Maybe we should do:

Quoted 7 lines
        bool do_mtrr = CONFIG_IS_ENABLED(X86_32BIT_INIT);

do_mtrr &= (!IS_ENABLED(CONFIG_SPL) || IS_ENABLED(CONFIG_FSP_VERSION2)) &&

        !IS_ENABLED(CONFIG_FSP_VERSION1) &&
        !IS_ENABLED(CONFIG_SYS_SLIMBOOTLOADER);

Will this work?

Unfortunately not. In fact I don't think we need to change this function.

For coral the sequence is:
SPL manually messes with MTRRs to add two MTRRs for SPI flash SPL jumps to U-Boot proper
now we start the global_data with 0 MTRR requests fsp_init_banksize() runs and adds two MTRR requests (uncommitted) init_cache_f_r() runs, but does not call mtrr_commit() fsp_graphics_probe() adds one MTRR request, making 5 in total fsp_graphics_probe() calls mtrr_commit()

So I think if we adjust fsp_graphics to either add a request and commit (for FSP2) or just add next var (for other things) we might be OK

Here is a run on coral with my modifications at
https://github.com/sjg20/u-boot/tree/for-bin

U-Boot 2023.10-rc1-00014-g37d6c899d4b (Jul 28 2023 - 09:58:37 -0600)

CPU:   Intel(R) Celeron(R) CPU N3450 @ 1.10GHz
DRAM:  dram

- add req
- add req
dram done
4 GiB (effective 3.9 GiB)
do_mtrr 0

Core:  67 devices, 35 uclasses, devicetree: separate
MMC:   sdmmc@1b,0: 1, emmc@1c,0: 0
Loading Environment from nowhere... OK
Video: graphics
Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
4   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
- add req
commit, do_caches=1
Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         0000000000000000 0000007f80000000 0000000080000000
1   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
2   Y     Back         0000000100000000 0000007f80000000 0000000080000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
4   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
graphics done
1024x768x32 @ b0000000
Model: Google Coral
Net:         eth_initialize() No ethernet found.
SF: Detected w25q128fw with page size 256 Bytes, erase size 4 KiB, total 16 MiB
Hit any key to stop autoboot:  0
=> mtrr
CPU 0:
Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         0000000000000000 0000007f80000000 0000000080000000
1   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
2   Y     Back         0000000100000000 0000007f80000000 0000000080000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
4   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

=>

Regards,
Simon

author
#12 lore raw

Hi Simon,

Quoted 167 lines · Simon Glass

Hi Bin,

Quoted 155 lines · Bin Meng

Hi Simon,

Quoted 136 lines · Simon Glass

Hi Bin,

Quoted 117 lines · Bin Meng

Hi Simon,

Quoted 96 lines · Simon Glass

Hi Bin,

Quoted 92 lines · Bin Meng

Hi Simon,

Quoted 59 lines · Simon Glass

Hi Bin,

Quoted 16 lines · Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Quoted 7 lines

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Thanks for looking into this. The series works fine on link. I suspect it will be find on samus too, but I cannot test right now. Sadly minnowmax is also dead right now but I hope to fix it soon. I don't expect any problems there.

However, for coral, this first patch breaks the mtrrs. With master we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

with this patch on coral we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
Quoted 3 lines

At present coral expects to handle the MTRRs itself, and it seems that perhaps the APL FSPv2 does not? Do we need a new Kconfig for dealing with FSPv2 perhaps?

I am a little bit confused. The comment in arch/x86/lib/fsp/fsp_dram.c::
dram_init_banksize() says:

/*
* For FSP1, the system memory and reserved memory used by FSP are
* already programmed in the MTRR by FSP. Also it is observed that
* FSP on Intel Queensbay platform reports the TSEG memory range
* that has the same RES_MEM_RESERVED resource type whose address
* is programmed by FSP to be near the top of 4 GiB space, which is
* not what we want for DRAM.
*
* However it seems FSP2's behavior is different. We need to add the
* DRAM range in MTRR otherwise the boot process goes very slowly,
* which was observed on Chromebook Coral with FSP2.
*/

So on Coral with FSP2, U-Boot programs the MTTR by itself.

In this dram_init_banksize(), it calls mtrr_add_request() 3 times, 2 of which should be what you were seeing as 2 and 4 below:

Quoted 3 lines
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 8 lines
Quoted 4 lines

The #3 should be the FSP graphics frame buffer. But I failed to understand how the FSP graphics programs a MTRR register in between the 2 memory regions programmed by dram_init_banksize() on u-boot/master, how could that happen?

Remember that the MTRRs are sorted, so the order or mtrr_add_request() calls does not matter.

Still cannot explain.

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 3 lines

After we sort the mtrr memory range, #2 whose base is 0x0 should have been put to the first entry, then followed by #3 whose base is 0xb0000000.

Right, but the thing is, your first patch does not revert the behaviour of mtrr_add_request(). It is still just adding to the end.

i.e. mtrr_commt() adds new ones but does not overwrite those at the back.

Looking at your full series, this is what I see on coral:

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
Quoted 5 lines

But I see that do_mtrr is wrong for coral in init_cache_f_r():

do_mtrr &= !IS_ENABLED(CONFIG_SPL) &&

So with coral the mtrrs are never written?

Yes, it seems this place is the culprit. The comment says:

* Note: if there is an SPL, then it has already set up MTRRs so we
* don't need to do that here

So on Coral, the assumption of SPL programming MTRRs is wrong.

Maybe we should do:

Quoted 7 lines
        bool do_mtrr = CONFIG_IS_ENABLED(X86_32BIT_INIT);

do_mtrr &= (!IS_ENABLED(CONFIG_SPL) || IS_ENABLED(CONFIG_FSP_VERSION2)) &&

        !IS_ENABLED(CONFIG_FSP_VERSION1) &&
        !IS_ENABLED(CONFIG_SYS_SLIMBOOTLOADER);

Will this work?

Quoted 8 lines

Unfortunately not. In fact I don't think we need to change this function.

For coral the sequence is:
SPL manually messes with MTRRs to add two MTRRs for SPI flash SPL jumps to U-Boot proper
now we start the global_data with 0 MTRR requests fsp_init_banksize() runs and adds two MTRR requests (uncommitted) init_cache_f_r() runs, but does not call mtrr_commit()

But with my proposed change, mtrr_commit() should be called here. Why does it not work?

Quoted 66 lines

fsp_graphics_probe() adds one MTRR request, making 5 in total fsp_graphics_probe() calls mtrr_commit()

So I think if we adjust fsp_graphics to either add a request and commit (for FSP2) or just add next var (for other things) we might be OK

Here is a run on coral with my modifications at
https://github.com/sjg20/u-boot/tree/for-bin

U-Boot 2023.10-rc1-00014-g37d6c899d4b (Jul 28 2023 - 09:58:37 -0600)

CPU:   Intel(R) Celeron(R) CPU N3450 @ 1.10GHz
DRAM:  dram

- add req
- add req
dram done
4 GiB (effective 3.9 GiB)
do_mtrr 0

Core:  67 devices, 35 uclasses, devicetree: separate
MMC:   sdmmc@1b,0: 1, emmc@1c,0: 0
Loading Environment from nowhere... OK
Video: graphics
Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
4   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
- add req
commit, do_caches=1
Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         0000000000000000 0000007f80000000 0000000080000000
1   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
2   Y     Back         0000000100000000 0000007f80000000 0000000080000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
4   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
graphics done
1024x768x32 @ b0000000
Model: Google Coral
Net:         eth_initialize() No ethernet found.
SF: Detected w25q128fw with page size 256 Bytes, erase size 4 KiB, total 16 MiB
Hit any key to stop autoboot:  0
=> mtrr
CPU 0:
Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         0000000000000000 0000007f80000000 0000000080000000
1   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
2   Y     Back         0000000100000000 0000007f80000000 0000000080000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
4   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
Quoted 1 line

=>

Regards,
Bin

#13 lore raw

Hi Bin,

Quoted 180 lines · Bin Meng

Hi Simon,

Quoted 167 lines · Simon Glass

Hi Bin,

Quoted 155 lines · Bin Meng

Hi Simon,

Quoted 136 lines · Simon Glass

Hi Bin,

Quoted 117 lines · Bin Meng

Hi Simon,

Quoted 96 lines · Simon Glass

Hi Bin,

Quoted 92 lines · Bin Meng

Hi Simon,

Quoted 59 lines · Simon Glass

Hi Bin,

Quoted 16 lines · Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Quoted 7 lines

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Thanks for looking into this. The series works fine on link. I suspect it will be find on samus too, but I cannot test right now. Sadly minnowmax is also dead right now but I hope to fix it soon. I don't expect any problems there.

However, for coral, this first patch breaks the mtrrs. With master we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

with this patch on coral we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
Quoted 3 lines

At present coral expects to handle the MTRRs itself, and it seems that perhaps the APL FSPv2 does not? Do we need a new Kconfig for dealing with FSPv2 perhaps?

I am a little bit confused. The comment in arch/x86/lib/fsp/fsp_dram.c::
dram_init_banksize() says:

/*
* For FSP1, the system memory and reserved memory used by FSP are
* already programmed in the MTRR by FSP. Also it is observed that
* FSP on Intel Queensbay platform reports the TSEG memory range
* that has the same RES_MEM_RESERVED resource type whose address
* is programmed by FSP to be near the top of 4 GiB space, which is
* not what we want for DRAM.
*
* However it seems FSP2's behavior is different. We need to add the
* DRAM range in MTRR otherwise the boot process goes very slowly,
* which was observed on Chromebook Coral with FSP2.
*/

So on Coral with FSP2, U-Boot programs the MTTR by itself.

In this dram_init_banksize(), it calls mtrr_add_request() 3 times, 2 of which should be what you were seeing as 2 and 4 below:

Quoted 3 lines
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 8 lines
Quoted 4 lines

The #3 should be the FSP graphics frame buffer. But I failed to understand how the FSP graphics programs a MTRR register in between the 2 memory regions programmed by dram_init_banksize() on u-boot/master, how could that happen?

Remember that the MTRRs are sorted, so the order or mtrr_add_request() calls does not matter.

Still cannot explain.

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 3 lines

After we sort the mtrr memory range, #2 whose base is 0x0 should have been put to the first entry, then followed by #3 whose base is 0xb0000000.

Right, but the thing is, your first patch does not revert the behaviour of mtrr_add_request(). It is still just adding to the end.

i.e. mtrr_commt() adds new ones but does not overwrite those at the back.

Looking at your full series, this is what I see on coral:

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
Quoted 5 lines

But I see that do_mtrr is wrong for coral in init_cache_f_r():

do_mtrr &= !IS_ENABLED(CONFIG_SPL) &&

So with coral the mtrrs are never written?

Yes, it seems this place is the culprit. The comment says:

* Note: if there is an SPL, then it has already set up MTRRs so we
* don't need to do that here

So on Coral, the assumption of SPL programming MTRRs is wrong.

Maybe we should do:

Quoted 7 lines
        bool do_mtrr = CONFIG_IS_ENABLED(X86_32BIT_INIT);

do_mtrr &= (!IS_ENABLED(CONFIG_SPL) || IS_ENABLED(CONFIG_FSP_VERSION2)) &&

        !IS_ENABLED(CONFIG_FSP_VERSION1) &&
        !IS_ENABLED(CONFIG_SYS_SLIMBOOTLOADER);

Will this work?

Quoted 8 lines

Unfortunately not. In fact I don't think we need to change this function.

For coral the sequence is:
SPL manually messes with MTRRs to add two MTRRs for SPI flash SPL jumps to U-Boot proper
now we start the global_data with 0 MTRR requests fsp_init_banksize() runs and adds two MTRR requests (uncommitted) init_cache_f_r() runs, but does not call mtrr_commit()

Quoted 2 lines

But with my proposed change, mtrr_commit() should be called here. Why does it not work?

Firstly it is still running from SPI flash, so the commit makes everything run very slow from this point, since it drops those two MTRRs. So we don't want to commit the MTRRs yet.

But yes it does work, in that we end up with three MTRRs (two DRAM and one graphics). It's just very slow getting to that point. That's why I think we should stick with fsp_graphics_probe() doing the commit, for FSPv2.

Quoted 60 lines
Quoted 60 lines

fsp_graphics_probe() adds one MTRR request, making 5 in total fsp_graphics_probe() calls mtrr_commit()

So I think if we adjust fsp_graphics to either add a request and commit (for FSP2) or just add next var (for other things) we might be OK

Here is a run on coral with my modifications at
https://github.com/sjg20/u-boot/tree/for-bin

U-Boot 2023.10-rc1-00014-g37d6c899d4b (Jul 28 2023 - 09:58:37 -0600)

CPU:   Intel(R) Celeron(R) CPU N3450 @ 1.10GHz
DRAM:  dram

- add req
- add req
dram done
4 GiB (effective 3.9 GiB)
do_mtrr 0

Core:  67 devices, 35 uclasses, devicetree: separate
MMC:   sdmmc@1b,0: 1, emmc@1c,0: 0
Loading Environment from nowhere... OK
Video: graphics
Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
4   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
- add req
commit, do_caches=1
Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         0000000000000000 0000007f80000000 0000000080000000
1   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
2   Y     Back         0000000100000000 0000007f80000000 0000000080000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
4   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
graphics done
1024x768x32 @ b0000000
Model: Google Coral
Net:         eth_initialize() No ethernet found.
SF: Detected w25q128fw with page size 256 Bytes, erase size 4 KiB, total 16 MiB
Hit any key to stop autoboot:  0
=> mtrr
CPU 0:
Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         0000000000000000 0000007f80000000 0000000080000000
1   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
2   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 9 lines
Quoted 8 lines
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
4   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

=>

Regards,
Simon

author
#14 lore raw

Hi Simon,

Quoted 187 lines · Simon Glass

Hi Bin,

Quoted 180 lines · Bin Meng

Hi Simon,

Quoted 167 lines · Simon Glass

Hi Bin,

Quoted 155 lines · Bin Meng

Hi Simon,

Quoted 136 lines · Simon Glass

Hi Bin,

Quoted 117 lines · Bin Meng

Hi Simon,

Quoted 96 lines · Simon Glass

Hi Bin,

Quoted 92 lines · Bin Meng

Hi Simon,

Quoted 59 lines · Simon Glass

Hi Bin,

Quoted 16 lines · Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Quoted 7 lines

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Thanks for looking into this. The series works fine on link. I suspect it will be find on samus too, but I cannot test right now. Sadly minnowmax is also dead right now but I hope to fix it soon. I don't expect any problems there.

However, for coral, this first patch breaks the mtrrs. With master we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

with this patch on coral we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
Quoted 3 lines

At present coral expects to handle the MTRRs itself, and it seems that perhaps the APL FSPv2 does not? Do we need a new Kconfig for dealing with FSPv2 perhaps?

I am a little bit confused. The comment in arch/x86/lib/fsp/fsp_dram.c::
dram_init_banksize() says:

/*
* For FSP1, the system memory and reserved memory used by FSP are
* already programmed in the MTRR by FSP. Also it is observed that
* FSP on Intel Queensbay platform reports the TSEG memory range
* that has the same RES_MEM_RESERVED resource type whose address
* is programmed by FSP to be near the top of 4 GiB space, which is
* not what we want for DRAM.
*
* However it seems FSP2's behavior is different. We need to add the
* DRAM range in MTRR otherwise the boot process goes very slowly,
* which was observed on Chromebook Coral with FSP2.
*/

So on Coral with FSP2, U-Boot programs the MTTR by itself.

In this dram_init_banksize(), it calls mtrr_add_request() 3 times, 2 of which should be what you were seeing as 2 and 4 below:

Quoted 3 lines
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 8 lines
Quoted 4 lines

The #3 should be the FSP graphics frame buffer. But I failed to understand how the FSP graphics programs a MTRR register in between the 2 memory regions programmed by dram_init_banksize() on u-boot/master, how could that happen?

Remember that the MTRRs are sorted, so the order or mtrr_add_request() calls does not matter.

Still cannot explain.

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 3 lines

After we sort the mtrr memory range, #2 whose base is 0x0 should have been put to the first entry, then followed by #3 whose base is 0xb0000000.

Right, but the thing is, your first patch does not revert the behaviour of mtrr_add_request(). It is still just adding to the end.

i.e. mtrr_commt() adds new ones but does not overwrite those at the back.

Looking at your full series, this is what I see on coral:

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
Quoted 5 lines

But I see that do_mtrr is wrong for coral in init_cache_f_r():

do_mtrr &= !IS_ENABLED(CONFIG_SPL) &&

So with coral the mtrrs are never written?

Yes, it seems this place is the culprit. The comment says:

* Note: if there is an SPL, then it has already set up MTRRs so we
* don't need to do that here

So on Coral, the assumption of SPL programming MTRRs is wrong.

Maybe we should do:

Quoted 7 lines
        bool do_mtrr = CONFIG_IS_ENABLED(X86_32BIT_INIT);

do_mtrr &= (!IS_ENABLED(CONFIG_SPL) || IS_ENABLED(CONFIG_FSP_VERSION2)) &&

        !IS_ENABLED(CONFIG_FSP_VERSION1) &&
        !IS_ENABLED(CONFIG_SYS_SLIMBOOTLOADER);

Will this work?

Quoted 8 lines

Unfortunately not. In fact I don't think we need to change this function.

For coral the sequence is:
SPL manually messes with MTRRs to add two MTRRs for SPI flash SPL jumps to U-Boot proper
now we start the global_data with 0 MTRR requests fsp_init_banksize() runs and adds two MTRR requests (uncommitted) init_cache_f_r() runs, but does not call mtrr_commit()

Quoted 2 lines

But with my proposed change, mtrr_commit() should be called here. Why does it not work?

Quoted 8 lines

Firstly it is still running from SPI flash, so the commit makes everything run very slow from this point, since it drops those two MTRRs. So we don't want to commit the MTRRs yet.

But yes it does work, in that we end up with three MTRRs (two DRAM and one graphics). It's just very slow getting to that point. That's why I think we should stick with fsp_graphics_probe() doing the commit, for FSPv2.

It's possible that someone does not include FSP_GRAPHICS on Coral so you rely on fsp_graphics_probe() doing the commit is not going to work.

Besides, the logic of doing mtrr commit in fsp_graphics_probe() does not keep everything in consistency.

This Coral issue, it sounds like we should fix arch/x86/cpu/apollolake/cpu_spl.c::arch_cpu_init_spl() to call mtrr_commit() for the cache of SPI flash in the SPL phase.

Quoted 73 lines
Quoted 61 lines
Quoted 61 lines

fsp_graphics_probe() adds one MTRR request, making 5 in total fsp_graphics_probe() calls mtrr_commit()

So I think if we adjust fsp_graphics to either add a request and commit (for FSP2) or just add next var (for other things) we might be OK

Here is a run on coral with my modifications at
https://github.com/sjg20/u-boot/tree/for-bin

U-Boot 2023.10-rc1-00014-g37d6c899d4b (Jul 28 2023 - 09:58:37 -0600)

CPU:   Intel(R) Celeron(R) CPU N3450 @ 1.10GHz
DRAM:  dram

- add req
- add req
dram done
4 GiB (effective 3.9 GiB)
do_mtrr 0

Core:  67 devices, 35 uclasses, devicetree: separate
MMC:   sdmmc@1b,0: 1, emmc@1c,0: 0
Loading Environment from nowhere... OK
Video: graphics
Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
4   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
- add req
commit, do_caches=1
Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         0000000000000000 0000007f80000000 0000000080000000
1   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
2   Y     Back         0000000100000000 0000007f80000000 0000000080000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
4   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
graphics done
1024x768x32 @ b0000000
Model: Google Coral
Net:         eth_initialize() No ethernet found.
SF: Detected w25q128fw with page size 256 Bytes, erase size 4 KiB, total 16 MiB
Hit any key to stop autoboot:  0
=> mtrr
CPU 0:
Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         0000000000000000 0000007f80000000 0000000080000000
1   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
2   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 9 lines
Quoted 8 lines
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
4   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

=>

Regards,
Simon

author
#15 lore raw

Hi Simon,

Quoted 204 lines · Bin Meng

Hi Simon,

Quoted 187 lines · Simon Glass

Hi Bin,

Quoted 180 lines · Bin Meng

Hi Simon,

Quoted 167 lines · Simon Glass

Hi Bin,

Quoted 155 lines · Bin Meng

Hi Simon,

Quoted 136 lines · Simon Glass

Hi Bin,

Quoted 117 lines · Bin Meng

Hi Simon,

Quoted 96 lines · Simon Glass

Hi Bin,

Quoted 92 lines · Bin Meng

Hi Simon,

Quoted 59 lines · Simon Glass

Hi Bin,

Quoted 16 lines · Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Quoted 7 lines

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Thanks for looking into this. The series works fine on link. I suspect it will be find on samus too, but I cannot test right now. Sadly minnowmax is also dead right now but I hope to fix it soon. I don't expect any problems there.

However, for coral, this first patch breaks the mtrrs. With master we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

with this patch on coral we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
Quoted 3 lines

At present coral expects to handle the MTRRs itself, and it seems that perhaps the APL FSPv2 does not? Do we need a new Kconfig for dealing with FSPv2 perhaps?

I am a little bit confused. The comment in arch/x86/lib/fsp/fsp_dram.c::
dram_init_banksize() says:

/*
* For FSP1, the system memory and reserved memory used by FSP are
* already programmed in the MTRR by FSP. Also it is observed that
* FSP on Intel Queensbay platform reports the TSEG memory range
* that has the same RES_MEM_RESERVED resource type whose address
* is programmed by FSP to be near the top of 4 GiB space, which is
* not what we want for DRAM.
*
* However it seems FSP2's behavior is different. We need to add the
* DRAM range in MTRR otherwise the boot process goes very slowly,
* which was observed on Chromebook Coral with FSP2.
*/

So on Coral with FSP2, U-Boot programs the MTTR by itself.

In this dram_init_banksize(), it calls mtrr_add_request() 3 times, 2 of which should be what you were seeing as 2 and 4 below:

Quoted 3 lines
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 8 lines
Quoted 4 lines

The #3 should be the FSP graphics frame buffer. But I failed to understand how the FSP graphics programs a MTRR register in between the 2 memory regions programmed by dram_init_banksize() on u-boot/master, how could that happen?

Remember that the MTRRs are sorted, so the order or mtrr_add_request() calls does not matter.

Still cannot explain.

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 3 lines

After we sort the mtrr memory range, #2 whose base is 0x0 should have been put to the first entry, then followed by #3 whose base is 0xb0000000.

Right, but the thing is, your first patch does not revert the behaviour of mtrr_add_request(). It is still just adding to the end.

i.e. mtrr_commt() adds new ones but does not overwrite those at the back.

Looking at your full series, this is what I see on coral:

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
Quoted 5 lines

But I see that do_mtrr is wrong for coral in init_cache_f_r():

do_mtrr &= !IS_ENABLED(CONFIG_SPL) &&

So with coral the mtrrs are never written?

Yes, it seems this place is the culprit. The comment says:

* Note: if there is an SPL, then it has already set up MTRRs so we
* don't need to do that here

So on Coral, the assumption of SPL programming MTRRs is wrong.

Maybe we should do:

Quoted 7 lines
        bool do_mtrr = CONFIG_IS_ENABLED(X86_32BIT_INIT);

do_mtrr &= (!IS_ENABLED(CONFIG_SPL) || IS_ENABLED(CONFIG_FSP_VERSION2)) &&

        !IS_ENABLED(CONFIG_FSP_VERSION1) &&
        !IS_ENABLED(CONFIG_SYS_SLIMBOOTLOADER);

Will this work?

Quoted 8 lines

Unfortunately not. In fact I don't think we need to change this function.

For coral the sequence is:
SPL manually messes with MTRRs to add two MTRRs for SPI flash SPL jumps to U-Boot proper
now we start the global_data with 0 MTRR requests fsp_init_banksize() runs and adds two MTRR requests (uncommitted) init_cache_f_r() runs, but does not call mtrr_commit()

Quoted 2 lines

But with my proposed change, mtrr_commit() should be called here. Why does it not work?

Quoted 8 lines

Firstly it is still running from SPI flash, so the commit makes everything run very slow from this point, since it drops those two MTRRs. So we don't want to commit the MTRRs yet.

But yes it does work, in that we end up with three MTRRs (two DRAM and one graphics). It's just very slow getting to that point. That's why I think we should stick with fsp_graphics_probe() doing the commit, for FSPv2.

It's possible that someone does not include FSP_GRAPHICS on Coral so you rely on fsp_graphics_probe() doing the commit is not going to work.

Quoted 7 lines

Besides, the logic of doing mtrr commit in fsp_graphics_probe() does not keep everything in consistency.

This Coral issue, it sounds like we should fix arch/x86/cpu/apollolake/cpu_spl.c::arch_cpu_init_spl() to call mtrr_commit() for the cache of SPI flash in the SPL phase.

Another possible place to insert the mtrr_commit() is ich_spi_probe().

Currently it programmed the MTRR for TPL, but not for SPL.

I suspect the TPL phase is duplicate since it is already programmed in the arch/x86/cpu/apollolake/cpu_spl.c::arch_cpu_init_tpl().

Regards,
Bin

#16 lore raw

Hi Bin,

Quoted 216 lines · Bin Meng

Hi Simon,

Quoted 204 lines · Bin Meng

Hi Simon,

Quoted 187 lines · Simon Glass

Hi Bin,

Quoted 180 lines · Bin Meng

Hi Simon,

Quoted 167 lines · Simon Glass

Hi Bin,

Quoted 155 lines · Bin Meng

Hi Simon,

Quoted 136 lines · Simon Glass

Hi Bin,

Quoted 117 lines · Bin Meng

Hi Simon,

Quoted 96 lines · Simon Glass

Hi Bin,

Quoted 92 lines · Bin Meng

Hi Simon,

Quoted 59 lines · Simon Glass

Hi Bin,

Quoted 16 lines · Bin Meng

At present this uses mtrr_add_request() & mtrr_commit() combination to program the MTRR for graphics memory. This usage has two major issues as below:

- mtrr_commit() will re-initialize all MTRR registers from index 0, using the settings previously added by mtrr_add_request() and saved in gd->arch.mtrr_req[], which won't cause any issue but is unnecessary
- The way such combination works is based on the assumption that U-Boot has full control with MTRR programming (e.g.: U-Boot without any blob that does all low-level initialization on its own, or using FSP2 which does not touch MTRR), but this is not the case with FSP. FSP programs some MTRRs during its execution but U-Boot does not have the settings saved in gd->arch.mtrr_req[] and when doing mtrr_commit() it will corrupt what was already programmed previously.

Quoted 7 lines

Correct this to use mtrr_set_next_var() instead.

---

 arch/x86/lib/fsp/fsp_graphics.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Thanks for looking into this. The series works fine on link. I suspect it will be find on samus too, but I cannot test right now. Sadly minnowmax is also dead right now but I hope to fix it soon. I don't expect any problems there.

However, for coral, this first patch breaks the mtrrs. With master we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
5   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
6   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
7   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
8   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
9   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000

with this patch on coral we get:

=> mtrr
CPU 0:

Reg Valid Write-type   Base   ||        Mask   ||        Size   ||
0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
3   N     Uncacheable  0000000000000000 0000000000000000 0000008000000000
Quoted 3 lines

At present coral expects to handle the MTRRs itself, and it seems that perhaps the APL FSPv2 does not? Do we need a new Kconfig for dealing with FSPv2 perhaps?

I am a little bit confused. The comment in arch/x86/lib/fsp/fsp_dram.c::
dram_init_banksize() says:

/*
* For FSP1, the system memory and reserved memory used by FSP are
* already programmed in the MTRR by FSP. Also it is observed that
* FSP on Intel Queensbay platform reports the TSEG memory range
* that has the same RES_MEM_RESERVED resource type whose address
* is programmed by FSP to be near the top of 4 GiB space, which is
* not what we want for DRAM.
*
* However it seems FSP2's behavior is different. We need to add the
* DRAM range in MTRR otherwise the boot process goes very slowly,
* which was observed on Chromebook Coral with FSP2.
*/

So on Coral with FSP2, U-Boot programs the MTTR by itself.

In this dram_init_banksize(), it calls mtrr_add_request() 3 times, 2 of which should be what you were seeing as 2 and 4 below:

Quoted 3 lines
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 8 lines
Quoted 4 lines

The #3 should be the FSP graphics frame buffer. But I failed to understand how the FSP graphics programs a MTRR register in between the 2 memory regions programmed by dram_init_banksize() on u-boot/master, how could that happen?

Remember that the MTRRs are sorted, so the order or mtrr_add_request() calls does not matter.

Still cannot explain.

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Back         0000000000000000 0000007f80000000 0000000080000000
3   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
4   Y     Back         0000000100000000 0000007f80000000 0000000080000000
Quoted 3 lines

After we sort the mtrr memory range, #2 whose base is 0x0 should have been put to the first entry, then followed by #3 whose base is 0xb0000000.

Right, but the thing is, your first patch does not revert the behaviour of mtrr_add_request(). It is still just adding to the end.

i.e. mtrr_commt() adds new ones but does not overwrite those at the back.

Looking at your full series, this is what I see on coral:

0   Y     Back         00000000fef00000 0000007ffff80000 0000000000080000
1   Y     Back         00000000fef80000 0000007ffffc0000 0000000000040000
2   Y     Combine      00000000b0000000 0000007ff0000000 0000000010000000
Quoted 5 lines

But I see that do_mtrr is wrong for coral in init_cache_f_r():

do_mtrr &= !IS_ENABLED(CONFIG_SPL) &&

So with coral the mtrrs are never written?

Yes, it seems this place is the culprit. The comment says:

* Note: if there is an SPL, then it has already set up MTRRs so we
* don't need to do that here

So on Coral, the assumption of SPL programming MTRRs is wrong.

Maybe we should do:

Quoted 7 lines
        bool do_mtrr = CONFIG_IS_ENABLED(X86_32BIT_INIT);

do_mtrr &= (!IS_ENABLED(CONFIG_SPL) || IS_ENABLED(CONFIG_FSP_VERSION2)) &&

        !IS_ENABLED(CONFIG_FSP_VERSION1) &&
        !IS_ENABLED(CONFIG_SYS_SLIMBOOTLOADER);

Will this work?

Quoted 8 lines

Unfortunately not. In fact I don't think we need to change this function.

For coral the sequence is:
SPL manually messes with MTRRs to add two MTRRs for SPI flash SPL jumps to U-Boot proper
now we start the global_data with 0 MTRR requests fsp_init_banksize() runs and adds two MTRR requests (uncommitted) init_cache_f_r() runs, but does not call mtrr_commit()

Quoted 2 lines

But with my proposed change, mtrr_commit() should be called here. Why does it not work?

Quoted 8 lines

Firstly it is still running from SPI flash, so the commit makes everything run very slow from this point, since it drops those two MTRRs. So we don't want to commit the MTRRs yet.

But yes it does work, in that we end up with three MTRRs (two DRAM and one graphics). It's just very slow getting to that point. That's why I think we should stick with fsp_graphics_probe() doing the commit, for FSPv2.

It's possible that someone does not include FSP_GRAPHICS on Coral so you rely on fsp_graphics_probe() doing the commit is not going to work.

Quoted 7 lines

Besides, the logic of doing mtrr commit in fsp_graphics_probe() does not keep everything in consistency.

This Coral issue, it sounds like we should fix arch/x86/cpu/apollolake/cpu_spl.c::arch_cpu_init_spl() to call mtrr_commit() for the cache of SPI flash in the SPL phase.

Quoted 6 lines

Another possible place to insert the mtrr_commit() is ich_spi_probe().

Currently it programmed the MTRR for TPL, but not for SPL.

I suspect the TPL phase is duplicate since it is already programmed in the arch/x86/cpu/apollolake/cpu_spl.c::arch_cpu_init_tpl().

Yes there is probably some other stuff going on in SPL/TPL.

But with your v2 series I found that if we change the condition at the top of init_cache_f_r() it works. I will send the updated patch.

Regards,
Simon