Skip to contentSkip to conversation

Disable HS400 for eMMC on Radxa ROCK 4 SBCs

PATCH Applied Christopher Obbard 4 participants 7 messages Jul 5, 2023 – Jul 16, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Disable HS400 for eMMC on Radxa ROCK 4 SBCs

There is some instablity with some eMMC modules on ROCK Pi 4 SBCs running in HS400 mode. This ends up resulting in some block errors after a while or after a "heavy" operation utilising the eMMC (e.g. resizing a filesystem). An example of these errors is as follows:

    [  289.171014] mmc1: running CQE recovery
    [  290.048972] mmc1: running CQE recovery
    [  290.054834] mmc1: running CQE recovery
    [  290.060817] mmc1: running CQE recovery
    [  290.061337] blk_update_request: I/O error, dev mmcblk1, sector 1411072 op 0x1:(WRITE) flags 0x800 phys_seg 36 prio class 0
    [  290.061370] EXT4-fs warning (device mmcblk1p1): ext4_end_bio:348: I/O error 10 writing to inode 29547 starting block 176466)
    [  290.061484] Buffer I/O error on device mmcblk1p1, logical block 172288
    [  290.061531] Buffer I/O error on device mmcblk1p1, logical block 172289
    [  290.061551] Buffer I/O error on device mmcblk1p1, logical block 172290
    [  290.061574] Buffer I/O error on device mmcblk1p1, logical block 172291
    [  290.061592] Buffer I/O error on device mmcblk1p1, logical block 172292
    [  290.061615] Buffer I/O error on device mmcblk1p1, logical block 172293
    [  290.061632] Buffer I/O error on device mmcblk1p1, logical block 172294
    [  290.061654] Buffer I/O error on device mmcblk1p1, logical block 172295
    [  290.061673] Buffer I/O error on device mmcblk1p1, logical block 172296
    [  290.061695] Buffer I/O error on device mmcblk1p1, logical block 172297

Disabling the Command Queue seems to stop the CQE recovery from running, but doesn't seem to improve the I/O errors. Until this can be investigated further, disable HS400 mode on the ROCK Pi 4 SBCs to at least stop I/O errors from occurring.

Christopher Obbard (2):
arm64: dts: rockchip: Disable HS400 for eMMC on ROCK Pi 4
arm64: dts: rockchip: Disable HS400 for eMMC on ROCK 4C+

 arch/arm64/boot/dts/rockchip/rk3399-rock-4c-plus.dts | 3 +--
 arch/arm64/boot/dts/rockchip/rk3399-rock-pi-4.dtsi   | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)
Signature · 7 lines
2.40.1


_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip
#1 lore raw

PATCH 1/2 arm64: dts: rockchip: Disable HS400 for eMMC on ROCK Pi 4

There is some instablity with some eMMC modules on ROCK Pi 4 SBCs running in HS400 mode. This ends up resulting in some block errors after a while or after a "heavy" operation utilising the eMMC (e.g. resizing a filesystem). An example of these errors is as follows:

    [  289.171014] mmc1: running CQE recovery
    [  290.048972] mmc1: running CQE recovery
    [  290.054834] mmc1: running CQE recovery
    [  290.060817] mmc1: running CQE recovery
    [  290.061337] blk_update_request: I/O error, dev mmcblk1, sector 1411072 op 0x1:(WRITE) flags 0x800 phys_seg 36 prio class 0
    [  290.061370] EXT4-fs warning (device mmcblk1p1): ext4_end_bio:348: I/O error 10 writing to inode 29547 starting block 176466)
    [  290.061484] Buffer I/O error on device mmcblk1p1, logical block 172288
    [  290.061531] Buffer I/O error on device mmcblk1p1, logical block 172289
    [  290.061551] Buffer I/O error on device mmcblk1p1, logical block 172290
    [  290.061574] Buffer I/O error on device mmcblk1p1, logical block 172291
    [  290.061592] Buffer I/O error on device mmcblk1p1, logical block 172292
    [  290.061615] Buffer I/O error on device mmcblk1p1, logical block 172293
    [  290.061632] Buffer I/O error on device mmcblk1p1, logical block 172294
    [  290.061654] Buffer I/O error on device mmcblk1p1, logical block 172295
    [  290.061673] Buffer I/O error on device mmcblk1p1, logical block 172296
    [  290.061695] Buffer I/O error on device mmcblk1p1, logical block 172297

Disabling the Command Queue seems to stop the CQE recovery from running, but doesn't seem to improve the I/O errors. Until this can be investigated further, disable HS400 mode on the ROCK Pi 4 SBCs to at least stop I/O errors from occurring.

While we are here, set the eMMC maximum clock frequency to 1.5MHz to follow the ROCK 4C+.

Fixes 1b5715c602fd ("arm64: dts: rockchip: add ROCK Pi 4 DTS support")

---

 arch/arm64/boot/dts/rockchip/rk3399-rock-pi-4.dtsi | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/boot/dts/rockchip/rk3399-rock-pi-4.dtsi +2 -2
index 907071d4fe80..95efee311ece 100644
@@ -645,9 +645,9 @@ &saradc {
 };
 
 &sdhci {
+	max-frequency = <150000000>;
 	bus-width = <8>;
-	mmc-hs400-1_8v;
-	mmc-hs400-enhanced-strobe;
+	mmc-hs200-1_8v;
 	non-removable;
 	status = "okay";
 };
Signature · 7 lines
2.40.1


_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip
Tested-by on v1 1/2
#2 lore raw

Re: [PATCH v1 1/2] arm64: dts: rockchip: Disable HS400 for eMMC on ROCK Pi 4

Quoted 43 lines · Christopher Obbard

There is some instablity with some eMMC modules on ROCK Pi 4 SBCs running in HS400 mode. This ends up resulting in some block errors after a while or after a "heavy" operation utilising the eMMC (e.g. resizing a filesystem). An example of these errors is as follows:

    [  289.171014] mmc1: running CQE recovery
    [  290.048972] mmc1: running CQE recovery
    [  290.054834] mmc1: running CQE recovery
    [  290.060817] mmc1: running CQE recovery
    [  290.061337] blk_update_request: I/O error, dev mmcblk1, sector 1411072 op 0x1:(WRITE) flags 0x800 phys_seg 36 prio class 0
    [  290.061370] EXT4-fs warning (device mmcblk1p1): ext4_end_bio:348: I/O error 10 writing to inode 29547 starting block 176466)
    [  290.061484] Buffer I/O error on device mmcblk1p1, logical block 172288
    [  290.061531] Buffer I/O error on device mmcblk1p1, logical block 172289
    [  290.061551] Buffer I/O error on device mmcblk1p1, logical block 172290
    [  290.061574] Buffer I/O error on device mmcblk1p1, logical block 172291
    [  290.061592] Buffer I/O error on device mmcblk1p1, logical block 172292
    [  290.061615] Buffer I/O error on device mmcblk1p1, logical block 172293
    [  290.061632] Buffer I/O error on device mmcblk1p1, logical block 172294
    [  290.061654] Buffer I/O error on device mmcblk1p1, logical block 172295
    [  290.061673] Buffer I/O error on device mmcblk1p1, logical block 172296
    [  290.061695] Buffer I/O error on device mmcblk1p1, logical block 172297

Disabling the Command Queue seems to stop the CQE recovery from running, but doesn't seem to improve the I/O errors. Until this can be investigated further, disable HS400 mode on the ROCK Pi 4 SBCs to at least stop I/O errors from occurring.

While we are here, set the eMMC maximum clock frequency to 1.5MHz to follow the ROCK 4C+.

Fixes 1b5715c602fd ("arm64: dts: rockchip: add ROCK Pi 4 DTS support")

---

 arch/arm64/boot/dts/rockchip/rk3399-rock-pi-4.dtsi | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/boot/dts/rockchip/rk3399-rock-pi-4.dtsi +0 -0
index 907071d4fe80..95efee311ece 100644
@@ -645,9 +645,9 @@ &saradc {
 };

arch/arm64/boot/dts/rockchip/rk3399-rock-pi-4.dtsi &saradc { · line 653

Quoted 10 lines
arch/arm64/boot/dts/rockchip/rk3399-rock-pi-4.dtsi +2 -2
 
 &sdhci {
+	max-frequency = <150000000>;
 	bus-width = <8>;
-	mmc-hs400-1_8v;
-	mmc-hs400-enhanced-strobe;
+	mmc-hs200-1_8v;
 	non-removable;
 	status = "okay";
 };

Works as advertised on a RockPi 4b v1.3 with kernel 6.1.37.

Folker

#3 lore raw

Re: [PATCH v1 1/2] arm64: dts: rockchip: Disable HS400 for eMMC on ROCK Pi 4

In reply to Christopher Obbard

Le mercredi 05 juillet 2023 à 15:42 +0100, Christopher Obbard a écrit :

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

There is some instablity with some eMMC modules on ROCK Pi 4 SBCs
running
in HS400 mode. This ends up resulting in some block errors after a
while
or after a "heavy" operation utilising the eMMC (e.g. resizing a
filesystem). An example of these errors is as follows:

I did not report my finding to the Linux upstream back then (due to using a non vanilla Linux kernel) but with my Armbian install I had bisected this issue to 06653ebc0ad2e0b7d799cd71a5c2933ed2fb7a66 as the first bad commit.
I believe it was released in 5.10.60 (the first broken version to reach armbian was 5.10.63 from a working 5.10.43.
Since then all rk3399 I have checked have disabled hs400 (down to hs200 which is stable even with the above commits).

commit 06653ebc0ad2e0b7d799cd71a5c2933ed2fb7a66

Author: Dmitry Baryshkov <[email protected]>
Date:   Thu May 20 01:12:23 2021 +0300

regulator: core: resolve supply for boot-on/always-on regulators

commit 98e48cd9283dbac0e1445ee780889f10b3d1db6a upstream.

For the boot-on/always-on regulators the set_machine_constrainst() is called before resolving rdev->supply. Thus the code would try to enable rdev before enabling supplying regulator. Enforce resolving supply regulator before enabling rdev.

Fixes aea6cb99703e ("regulator: resolve supply after creating regulator")
 drivers/regulator/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/regulator/core.c +6 -0
index f192bf19492ed..e20e77e4c159d 100644
@@ -1425,6 +1425,12 @@ static int set_machine_constraints(struct regulator_dev *rdev)
 	 * and we have control then make sure it is enabled.
 	 */
 	if (rdev->constraints->always_on || rdev->constraints->boot_on) {
+		/* If we want to enable this regulator, make sure that we know
+		 * the supplying regulator.
+		 */
+		if (rdev->supply_name && !rdev->supply)
+			return -EPROBE_DEFER;
+
 		if (rdev->supply) {
 			ret = regulator_enable(rdev->supply);
 			if (ret < 0) {

My findings here:
https://forum.armbian.com/topic/18855-upgrading-to-bullseye-troubleshooting-armbian-21081/?do=findComment&comment=128793
this on a kobol helios64 rk3399 board.

I told a user to try this fix (revert commits 06653ebc0ad2e0b7d799cd71a5c2933ed2fb7a66
and aea6cb99703e17019e025aa71643b4d3e0a24413) also for an armbian kernel on a Nanopc-T4
and it fixes the issue https://forum.armbian.com/topic/20002-nanopc-t4-new-kernel-2202-generates-issues-on-mmc2-and-makes-system-not-properly-working/?do=findComment&comment=138052
This above 5.16.8.

I had high expectations that the commit that fixed double init would fix the issue for good, but sadly not. I believe this would have been the only required fix for 5.16 kernels but nowadays it is not enough a revert.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/regulator/core.c?id=8a866d527ac0441c0eb14a991fa11358b476b11d

regulator: core: Resolve supply name earlier to prevent double-init Previously, an unresolved regulator supply reference upon calling regulator_register on an always-on or boot-on regulator caused set_machine_constraints to be called twice.

This in turn may initialize the regulator twice, leading to voltage glitches that are timing-dependent. A simple, unrelated configuration change may be enough to hide this problem, only to be surfaced by chance.

One such example is the SD-Card voltage regulator in a NanoPI R4S that would not initialize reliably unless the registration flow was just complex enough to allow the regulator to properly reset between calls.

Fix this by re-arranging regulator_register, trying resolve the regulator's supply early enough that set_machine_constraints does not need to be called twice.

"
story behing this patch https://kohlschuetter.github.io/blog/posts/2022/10/28/linux-nanopi-r4s/

It should have worked because basically this patch is a revert of commit aea6cb99703e17019e025aa71643b4d3e0a24413 "regulator: resolve supply after creating regulator" except it keep what I believe is now dead code (ie the second set_machine_constains in "if (ret == - EPROBE_DEFER) " is of no use now that the regulator supply is resolved before the first set_machine_constraints call in regilator_registers. The only code left from the 5.10.60 breakage is the EPROBE_DEFER if regulator supply is not registered in set_machine_constrains. But even after removing this leftover and the new EPROBE_DEFER that was added to set_machine_constraints for "regulator that have no direct control", I cannot get rid of the Filesystem corruption and errors with hs400 with 6.3.

Still I have no clue why emmc regulators double init is fine on most SoC but not rk3399.

Cheers,
Alban

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#4 lore raw

PATCH 2/2 arm64: dts: rockchip: Disable HS400 for eMMC on ROCK 4C+

In reply to Christopher Obbard

There is some instablity with some eMMC modules on ROCK Pi 4 SBCs running in HS400 mode. This ends up resulting in some block errors after a while or after a "heavy" operation utilising the eMMC (e.g. resizing a filesystem). An example of these errors is as follows:

    [  289.171014] mmc1: running CQE recovery
    [  290.048972] mmc1: running CQE recovery
    [  290.054834] mmc1: running CQE recovery
    [  290.060817] mmc1: running CQE recovery
    [  290.061337] blk_update_request: I/O error, dev mmcblk1, sector 1411072 op 0x1:(WRITE) flags 0x800 phys_seg 36 prio class 0
    [  290.061370] EXT4-fs warning (device mmcblk1p1): ext4_end_bio:348: I/O error 10 writing to inode 29547 starting block 176466)
    [  290.061484] Buffer I/O error on device mmcblk1p1, logical block 172288
    [  290.061531] Buffer I/O error on device mmcblk1p1, logical block 172289
    [  290.061551] Buffer I/O error on device mmcblk1p1, logical block 172290
    [  290.061574] Buffer I/O error on device mmcblk1p1, logical block 172291
    [  290.061592] Buffer I/O error on device mmcblk1p1, logical block 172292
    [  290.061615] Buffer I/O error on device mmcblk1p1, logical block 172293
    [  290.061632] Buffer I/O error on device mmcblk1p1, logical block 172294
    [  290.061654] Buffer I/O error on device mmcblk1p1, logical block 172295
    [  290.061673] Buffer I/O error on device mmcblk1p1, logical block 172296
    [  290.061695] Buffer I/O error on device mmcblk1p1, logical block 172297

Disabling the Command Queue seems to stop the CQE recovery from running, but doesn't seem to improve the I/O errors. Until this can be investigated further, disable HS400 mode on the ROCK Pi 4 SBCs to at least stop I/O errors from occurring.

Fixes 246450344dad ("arm64: dts: rockchip: rk3399: Radxa ROCK 4C+")

---

 arch/arm64/boot/dts/rockchip/rk3399-rock-4c-plus.dts | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm64/boot/dts/rockchip/rk3399-rock-4c-plus.dts +1 -2
index 028eb508ae30..8bfd5f88d1ef 100644
@@ -548,9 +548,8 @@ &saradc {
 &sdhci {
 	max-frequency = <150000000>;
 	bus-width = <8>;
-	mmc-hs400-1_8v;
+	mmc-hs200-1_8v;
 	non-removable;
-	mmc-hs400-enhanced-strobe;
 	status = "okay";
 };
 
Signature · 7 lines
2.40.1


_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip
#5 lore raw
In reply to Christopher Obbard

Hi,

Quoted 22 lines · Christopher Obbard

There is some instablity with some eMMC modules on ROCK Pi 4 SBCs running in HS400 mode. This ends up resulting in some block errors after a while or after a "heavy" operation utilising the eMMC (e.g. resizing a filesystem). An example of these errors is as follows:

    [  289.171014] mmc1: running CQE recovery
    [  290.048972] mmc1: running CQE recovery
    [  290.054834] mmc1: running CQE recovery
    [  290.060817] mmc1: running CQE recovery
    [  290.061337] blk_update_request: I/O error, dev mmcblk1, sector 1411072 op 0x1:(WRITE) flags 0x800 phys_seg 36 prio class 0
    [  290.061370] EXT4-fs warning (device mmcblk1p1): ext4_end_bio:348: I/O error 10 writing to inode 29547 starting block 176466)
    [  290.061484] Buffer I/O error on device mmcblk1p1, logical block 172288
    [  290.061531] Buffer I/O error on device mmcblk1p1, logical block 172289
    [  290.061551] Buffer I/O error on device mmcblk1p1, logical block 172290
    [  290.061574] Buffer I/O error on device mmcblk1p1, logical block 172291
    [  290.061592] Buffer I/O error on device mmcblk1p1, logical block 172292
    [  290.061615] Buffer I/O error on device mmcblk1p1, logical block 172293
    [  290.061632] Buffer I/O error on device mmcblk1p1, logical block 172294
    [  290.061654] Buffer I/O error on device mmcblk1p1, logical block 172295
    [  290.061673] Buffer I/O error on device mmcblk1p1, logical block 172296
    [  290.061695] Buffer I/O error on device mmcblk1p1, logical block 172297
Quoted 4 lines

Disabling the Command Queue seems to stop the CQE recovery from running, but doesn't seem to improve the I/O errors. Until this can be investigated further, disable HS400 mode on the ROCK Pi 4 SBCs to at least stop I/O errors from occurring.

Thanks for the patches! This issue lately got some attention on the Radxa forums so thanks for bringing it to the kernel lists.

As a user who was hitting this issue some time ago I'd like to share the observations I made during my testing:

I've seen these EMMC issues on several RockPi4 boards (4b v1.4, 4b v1.5, 4SE and 4b+ v1.73), with different EMMC modules (3 Foresee, 1 Samsung) and throughout different kernels (v6.1.8 through 6.1.37, 6.3, 6.3.0-rc7-next-20230421).

However, with this vendor image [1] that uses kernel 4.4.154-116-rockchip-g86a614bc15b3 all of the EMMC modules work just fine. The same holds true for another vendor image with kernel 4.4.154-00039-g00fccd37c63c-dirty.

The interesting thing here is, that both 4.4 kernels have HS400 enabled (and max-frequency=<150000000> set) in their respective device-trees. This can be checked in the vendor kernel repo for 4.4.154-116-rockchip-g86a614bc15b3 [2] and for 4.4.154-00039-g00fccd37c63c [3].

Kind regards

Folker

[1]: https://github.com/radxa-build/rock-pi-4b/releases/download/main-df04b3af/rockpi-4b-debian-buster-xfce4-arm64-20220401-0335-gpt.img.xz
[2]: https://github.com/radxa/kernel/blob/86a614bc15b3b1aeb3a9a9e395aedd088c70e35e/arch/arm64/boot/dts/rockchip/rk3399-rock-pi-4.dtsi
[3]: https://github.com/radxa/kernel/blob/00fccd37c63cd51b2ae9b3af965f975c561674b1/arch/arm64/boot/dts/rockchip/rk3399-rock-pi-4.dtsi

Quoted 8 lines

Christopher Obbard (2):
arm64: dts: rockchip: Disable HS400 for eMMC on ROCK Pi 4
arm64: dts: rockchip: Disable HS400 for eMMC on ROCK 4C+

 arch/arm64/boot/dts/rockchip/rk3399-rock-4c-plus.dts | 3 +--
 arch/arm64/boot/dts/rockchip/rk3399-rock-pi-4.dtsi   | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)

_______________________________________________
Linux-rockchip mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-rockchip

maintainer Applied
#6 lore raw
In reply to Christopher Obbard
Quoted 22 lines · Christopher Obbard

There is some instablity with some eMMC modules on ROCK Pi 4 SBCs running in HS400 mode. This ends up resulting in some block errors after a while or after a "heavy" operation utilising the eMMC (e.g. resizing a filesystem). An example of these errors is as follows:

    [  289.171014] mmc1: running CQE recovery
    [  290.048972] mmc1: running CQE recovery
    [  290.054834] mmc1: running CQE recovery
    [  290.060817] mmc1: running CQE recovery
    [  290.061337] blk_update_request: I/O error, dev mmcblk1, sector 1411072 op 0x1:(WRITE) flags 0x800 phys_seg 36 prio class 0
    [  290.061370] EXT4-fs warning (device mmcblk1p1): ext4_end_bio:348: I/O error 10 writing to inode 29547 starting block 176466)
    [  290.061484] Buffer I/O error on device mmcblk1p1, logical block 172288
    [  290.061531] Buffer I/O error on device mmcblk1p1, logical block 172289
    [  290.061551] Buffer I/O error on device mmcblk1p1, logical block 172290
    [  290.061574] Buffer I/O error on device mmcblk1p1, logical block 172291
    [  290.061592] Buffer I/O error on device mmcblk1p1, logical block 172292
    [  290.061615] Buffer I/O error on device mmcblk1p1, logical block 172293
    [  290.061632] Buffer I/O error on device mmcblk1p1, logical block 172294
    [  290.061654] Buffer I/O error on device mmcblk1p1, logical block 172295
    [  290.061673] Buffer I/O error on device mmcblk1p1, logical block 172296
    [  290.061695] Buffer I/O error on device mmcblk1p1, logical block 172297
Quoted 1 line

[...]

Applied, thanks!

[1/2] arm64: dts: rockchip: Disable HS400 for eMMC on ROCK Pi 4

      commit: cee572756aa2cb46e959e9797ad4b730b78a050b
[2/2] arm64: dts: rockchip: Disable HS400 for eMMC on ROCK 4C+
      commit: 2bd1d2dd808c60532283e9cf05110bf1bf2f9079

Best regards,

Signature · 1 line
Heiko Stuebner <[email protected]>