Skip to contentSkip to conversation

sunxi: Fix gmac not working reliable on the Bananapi

PATCH In review Hans de Goede 6 participants 9 messages Sep 28, 2014 – Sep 30, 2014 2 messages not in the archive Arm/Allwinner sunXi SoC support

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sunxi: Fix gmac not working reliable on the Bananapi

In order for the gmac nic to work reliable on the Bananapi, we need to poke these 2 undocumented bits in the gmac clk register. Since these are undocumented, this commit only sets these bits on the Bananapi for now.

I'll contact Allwinner to try and get these bits documented, once they are documented we can hopefully replace this hack with a better patch.

---
 board/sunxi/gmac.c | 9 +++++++++
 1 file changed, 9 insertions(+)
board/sunxi/gmac.c +9 -0
index e7ff952..f58c963 100644
@@ -24,6 +24,15 @@ int sunxi_gmac_initialize(bd_t *bis)
 		CCM_GMAC_CTRL_GPIT_MII);
 #endif
 
+	/*
+	 * HdG: this is necessary to get GMAC to work reliable on the
+	 * Bananapi. We don't know what these undocumented bits do, so this
+	 * is a Bananapi specific hack for now.
+	 */
+#ifdef CONFIG_BANANAPI
+	setbits_le32(&ccm->gmac_clk_cfg, 0x3 << 10);
+#endif
+
 	/* Configure pin mux settings for GMAC */
 	for (pin = SUNXI_GPA(0); pin <= SUNXI_GPA(16); pin++) {
 #ifdef CONFIG_RGMII
Signature · 1 line
2.1.0

A message that is not in the archive (sent off-list or not yet received): [email protected]

Tested-by
#2 lore raw
Quoted 5 lines · Karsten Merker
Quoted 4 lines · Hans de Goede

In order for the gmac nic to work reliable on the Bananapi, we need to poke these 2 undocumented bits in the gmac clk register. Since these are undocumented, this commit only sets these bits on the Bananapi for now.

Quoted 7 lines
Quoted 5 lines

I'll contact Allwinner to try and get these bits documented, once they are documented we can hopefully replace this hack with a better patch.

Quoted 34 lines
Quoted 25 lines
---
 board/sunxi/gmac.c | 9 +++++++++
 1 file changed, 9 insertions(+)
board/sunxi/gmac.c +9 -1
index e7ff952..f58c963 100644
@@ -24,6 +24,15 @@ int sunxi_gmac_initialize(bd_t *bis)
 		CCM_GMAC_CTRL_GPIT_MII);
 #endif
 
+	/*
+	 * HdG: this is necessary to get GMAC to work reliable on the
+	 * Bananapi. We don't know what these undocumented bits do, so this
+	 * is a Bananapi specific hack for now.
+	 */
+#ifdef CONFIG_BANANAPI
+	setbits_le32(&ccm->gmac_clk_cfg, 0x3 << 10);
+#endif
+
 	/* Configure pin mux settings for GMAC */
 	for (pin = SUNXI_GPA(0); pin <= SUNXI_GPA(16); pin++) {
 #ifdef CONFIG_RGMII
--
Quoted 1 line

2.1.0

Regards,
Karsten

Signature · 3 lines
Gem. Par. 28 Abs. 4 Bundesdatenschutzgesetz widerspreche ich der Nutzung
sowie der Weitergabe meiner personenbezogenen Daten f?r Zwecke der
Werbung sowie der Markt- oder Meinungsforschung.
#3 lore raw

PATCH 1/1 sunxi: Fix gmac not working reliable on the Bananapi

In reply to Hans de Goede
Quoted 6 lines · On Sun, 28 Sep 2014 20:13:21 +0200 Hans de Goede <[email protected]> wrote:

In order for the gmac nic to work reliable on the Bananapi, we need to poke these 2 undocumented bits in the gmac clk register. Since these are undocumented, this commit only sets these bits on the Bananapi for now.

I'll contact Allwinner to try and get these bits documented, once they are documented we can hopefully replace this hack with a better patch.

Could you please provide a bit more details in the commit message? What are the symptoms of the problem? How did you come to the idea to poke these bits? Does the GMAC driver in the linux kernel need a similar fix/workaround? And if you apply it in the linux kernel, does it provide any visible changes in performance or anything else?

Also as mentioned in another e-mail

    http://lists.denx.de/pipermail/u-boot/2014-September/190096.html

u-boot configures the "912MHz @1.4V" CPU clock frequency/voltage setup for sun7i hardware. And according to the information from Tony Zhang, this is supposed to be unreliable for the Banana Pi. So what happens to this GMAC bug if you just increase the dcdc2 voltage in u-boot (or reduce the CPU clock frequency)?

I remember that some users suffered from GMAC problems, which turned out to be caused by just overclocking the CPU to 1.2GHz (a "helful" SD card image creator decided that the users would like to enjoy better performance):

    http://www.cubieforums.com/index.php/topic,2590.0.html

So it looks like GMAC might be one of those canaries, which die first on wrong CPU clock frequency/voltage settings.

Quoted 26 lines
---
 board/sunxi/gmac.c | 9 +++++++++
 1 file changed, 9 insertions(+)
board/sunxi/gmac.c +9 -0
index e7ff952..f58c963 100644
@@ -24,6 +24,15 @@ int sunxi_gmac_initialize(bd_t *bis)
 		CCM_GMAC_CTRL_GPIT_MII);
 #endif
 
+	/*
+	 * HdG: this is necessary to get GMAC to work reliable on the
+	 * Bananapi. We don't know what these undocumented bits do, so this
+	 * is a Bananapi specific hack for now.
+	 */
+#ifdef CONFIG_BANANAPI
+	setbits_le32(&ccm->gmac_clk_cfg, 0x3 << 10);
+#endif
+
 	/* Configure pin mux settings for GMAC */
 	for (pin = SUNXI_GPA(0); pin <= SUNXI_GPA(16); pin++) {
 #ifdef CONFIG_RGMII
Signature · 2 lines
Best regards,
Siarhei Siamashka

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 21 lines · Karsten Merker
Quoted 7 lines · Siarhei Siamashka
Quoted 4 lines · On Sun, 28 Sep 2014 20:13:21 +0200 Hans de Goede <[email protected]> wrote:

In order for the gmac nic to work reliable on the Bananapi, we need to poke these 2 undocumented bits in the gmac clk register. Since these are undocumented, this commit only sets these bits on the Bananapi for now.

Quoted 5 lines
Quoted 2 lines

I'll contact Allwinner to try and get these bits documented, once they are documented we can hopefully replace this hack with a better patch.

Could you please provide a bit more details in the commit message? What are the symptoms of the problem?

Without the patch, u-boot on the BananaPi shows massive packet loss on every network activity. The packet loss is so extreme that it is actually impossible to boot a kernel by tftp because u-boot gives up. The mainline kernel has (once booted) shown similar problems on the BananaPi.

Quoted 7 lines
Quoted 1 line

How did you come to the idea to poke these bits?

The linux 3.4 kernel from https://github.com/LeMaker/linux-bananapi
(which is a patched linux-sunxi.org 3.4 kernel) does not show this problem, so we started looking into what is different there, and the relevant change that was found was setting these particular undocumented bits.

File at issue here asking for the documentation
https://github.com/allwinner-zh/documents

Quoted 30 lines
Quoted 3 lines

Does the GMAC driver in the linux kernel need a similar fix/workaround?

Not as far as I can see - once u-boot has set these two bits during system initialization, the mainline kernel works fine without any changes.

Quoted 7 lines

Also as mentioned in another e-mail

    http://lists.denx.de/pipermail/u-boot/2014-September/190096.html

u-boot configures the "912MHz @1.4V" CPU clock frequency/voltage setup for sun7i hardware. And according to the information from Tony Zhang, this is supposed to be unreliable for the Banana Pi. So what happens to this GMAC bug if you just increase the dcdc2 voltage in u-boot (or reduce the CPU clock frequency)?

I can test that if you could tell me what would have to be changed in u-boot to do that.

Regards,
Karsten
--
Gem. Par. 28 Abs. 4 Bundesdatenschutzgesetz widerspreche ich der Nutzung sowie der Weitergabe meiner personenbezogenen Daten f?r Zwecke der Werbung sowie der Markt- oder Meinungsforschung.

--
You received this message because you are subscribed to the Google Groups "linux-sunxi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to linux-sunxi+unsubscribe at googlegroups.com. For more options, visit https://groups.google.com/d/optout.

Signature · 2 lines
Jon Smirl
jonsmirl at gmail.com
#6 lore raw

Hi,

Quoted 4 lines · jonsmirl at gmail.com
Quoted 3 lines · Karsten Merker
Quoted 1 line · Siarhei Siamashka

How did you come to the idea to poke these bits?

Quoted 8 lines
Quoted 5 lines

The linux 3.4 kernel from https://github.com/LeMaker/linux-bananapi
(which is a patched linux-sunxi.org 3.4 kernel) does not show this problem, so we started looking into what is different there, and the relevant change that was found was setting these particular undocumented bits.

File at issue here asking for the documentation
https://github.com/allwinner-zh/documents

I'm already talking to Allwinner about this, it seems that bits 10-11, or more likely 10-12 of the gmac clk register are the tx equivalent of bits 5-7, "Configure GMAC receive clock delay chain". I think this is there to deal with the pcb data and clk traces not having the same length, so that this is highly board specific. I'm waiting on confirmation on this from Allwinner, as well as some info on the unit of this 0-7 value.

Regards,

Hans

#7 lore raw
Quoted 16 lines · Hans de Goede

Hi,

Quoted 4 lines · jonsmirl at gmail.com
Quoted 3 lines · Karsten Merker
Quoted 1 line · Siarhei Siamashka

How did you come to the idea to poke these bits?

Quoted 8 lines
Quoted 5 lines

The linux 3.4 kernel from https://github.com/LeMaker/linux-bananapi
(which is a patched linux-sunxi.org 3.4 kernel) does not show this problem, so we started looking into what is different there, and the relevant change that was found was setting these particular undocumented bits.

File at issue here asking for the documentation
https://github.com/allwinner-zh/documents

Quoted 7 lines

I'm already talking to Allwinner about this, it seems that bits 10-11, or more likely 10-12 of the gmac clk register are the tx equivalent of bits 5-7, "Configure GMAC receive clock delay chain". I think this is there to deal with the pcb data and clk traces not having the same length, so that this is highly board specific. I'm waiting on confirmation on this from Allwinner, as well as some info on the unit of this 0-7 value.

Encourage them to update the manual and push a new copy to git hub immediately. Too much info like this is getting buried in mail archives. Electronic publishing is free - update as often as you want.

The more they fix stuff like this in the manual, the fewer questions they will need to deal with in person. When it goes into the manual it won't get lost.

Quoted 4 lines

Regards,

Hans

Signature · 2 lines
Jon Smirl
jonsmirl at gmail.com
#8 lore raw

PATCH 1/1 sunxi: Fix gmac not working reliable on the Bananapi

In reply to Hans de Goede

Op 30 sep. 2014, om 14:02 heeft Hans de Goede <[email protected]> het volgende geschreven:

Quoted 16 lines

Hi,

Quoted 4 lines · jonsmirl at gmail.com
Quoted 3 lines · Karsten Merker
Quoted 1 line · Siarhei Siamashka

How did you come to the idea to poke these bits?

Quoted 8 lines
Quoted 5 lines

The linux 3.4 kernel from https://github.com/LeMaker/linux-bananapi
(which is a patched linux-sunxi.org 3.4 kernel) does not show this problem, so we started looking into what is different there, and the relevant change that was found was setting these particular undocumented bits.

File at issue here asking for the documentation
https://github.com/allwinner-zh/documents

Quoted 7 lines

I'm already talking to Allwinner about this, it seems that bits 10-11, or more likely 10-12 of the gmac clk register are the tx equivalent of bits 5-7, "Configure GMAC receive clock delay chain". I think this is there to deal with the pcb data and clk traces not having the same length, so that this is highly board specific. I'm waiting on confirmation on this from Allwinner, as well as some info on the unit of this 0-7 value.

Gbit ethernet requires a delay between the lines, which you can implement by adding 10cm or so extra trace on the PCB or tell the MAC or PHY to do it in software. Very board specific, but so far I've only seen Intel reference designs use the extra trace length method most arm/ppc/mips gbit boards use a PHY that handles it in hardware on request like the GMAC is doing above.

regards,

Koen

#9 lore raw
In reply to a message not in the archive
Quoted 21 lines · On Mon, 29 Sep 2014 20:16:35 +0200 Karsten Merker <[email protected]> wrote:
Quoted 7 lines · Siarhei Siamashka
Quoted 4 lines · On Sun, 28 Sep 2014 20:13:21 +0200 Hans de Goede <[email protected]> wrote:

In order for the gmac nic to work reliable on the Bananapi, we need to poke these 2 undocumented bits in the gmac clk register. Since these are undocumented, this commit only sets these bits on the Bananapi for now.

Quoted 5 lines
Quoted 2 lines

I'll contact Allwinner to try and get these bits documented, once they are documented we can hopefully replace this hack with a better patch.

Could you please provide a bit more details in the commit message? What are the symptoms of the problem?

Without the patch, u-boot on the BananaPi shows massive packet loss on every network activity. The packet loss is so extreme that it is actually impossible to boot a kernel by tftp because u-boot gives up. The mainline kernel has (once booted) shown similar problems on the BananaPi.

Quoted 7 lines
Quoted 1 line

How did you come to the idea to poke these bits?

The linux 3.4 kernel from https://github.com/LeMaker/linux-bananapi
(which is a patched linux-sunxi.org 3.4 kernel) does not show this problem, so we started looking into what is different there, and the relevant change that was found was setting these particular undocumented bits.

Thanks for the detailed explanations. They would look really great in the commit message.

Quoted 6 lines
Quoted 2 lines

Does the GMAC driver in the linux kernel need a similar fix/workaround?

Not as far as I can see - once u-boot has set these two bits during system initialization, the mainline kernel works fine without any changes.

OK. I just wonder if it is a good idea for the kernel to rely on these bits being set appropriately by the bootloader. Not that it has anything to do with u-boot.

Quoted 10 lines
Quoted 7 lines

Also as mentioned in another e-mail

    http://lists.denx.de/pipermail/u-boot/2014-September/190096.html

u-boot configures the "912MHz @1.4V" CPU clock frequency/voltage setup for sun7i hardware. And according to the information from Tony Zhang, this is supposed to be unreliable for the Banana Pi. So what happens to this GMAC bug if you just increase the dcdc2 voltage in u-boot (or reduce the CPU clock frequency)?

I can test that if you could tell me what would have to be changed in u-boot to do that.

The dcdc2 voltage is set here (under the CONFIG_AXP209_POWER ifdef):
http://git.denx.de/?p=u-boot.git;a=blob;f=board/sunxi/board.c;h=2179e234e21d67b0fec064de792fca175db90ca5;hb=be9f643ae6aa9044c60fe80e3a2c10be8371c692#l140
Right now it is set to 1400, which means 1.4V (and the voltage can be changed in 0.025V steps).

The CPU clock frequency for sun7i is set here:
http://git.denx.de/?p=u-boot.git;a=blob;f=include/configs/sun7i.h;h=a902b845744707e64fb5a064c6039bc93b0150d9;hb=be9f643ae6aa9044c60fe80e3a2c10be8371c692#l16
Right now it is set to 912MHz

This CPU clock frequency/voltage pair is then just used unmodified by the mainline kernel (until we get proper cpufreq support later). So it makes a lot of sense to be sure that it is sufficiently reliable on every Allwinner A20 device. You can try to increase the dcdc2 voltage or reduce the CPU clock frequency and check if the Banana Pi reliability problems disappear.

By the way, the dcdc2 voltage is already set to the maximum, which is specified in the datasheet. You can have a look at VDD-CPU limits in
https://github.com/allwinner-zh/documents/blob/master/A20/A20%20Datasheet%20V1.4%2020131230.pdf
And VDD-CPU is provided from dcdc2, as can be seen, for example, in the A20-OLINUXINO-MICRO schematics (the Banana Pi does not seem to have its schematics document in public access, but all A20 devices have it designed in the same way):
https://github.com/OLIMEX/OLINUXINO/blob/master/HARDWARE/A10-OLinuXino-MICRO/A20-OLINUXINO-MICRO_4GB_Rev_F1.pdf

Signature · 2 lines
Best regards,
Siarhei Siamashka
#10 lore raw

PATCH 1/1 sunxi: Fix gmac not working reliable on the Bananapi

In reply to Hans de Goede
Quoted 9 lines · Hans de Goede

In order for the gmac nic to work reliable on the Bananapi, we need to poke these 2 undocumented bits in the gmac clk register. Since these are undocumented, this commit only sets these bits on the Bananapi for now.

I'll contact Allwinner to try and get these bits documented, once they are documented we can hopefully replace this hack with a better patch.

It would be good to include some of Karsten's more detailed explanation (see his reply downthread) of what's going on, but other than that:

and ack to pushing it for v2014.10.

Quoted 24 lines
---
 board/sunxi/gmac.c | 9 +++++++++
 1 file changed, 9 insertions(+)
board/sunxi/gmac.c +9 -0
index e7ff952..f58c963 100644
@@ -24,6 +24,15 @@ int sunxi_gmac_initialize(bd_t *bis)
 		CCM_GMAC_CTRL_GPIT_MII);
 #endif
 
+	/*
+	 * HdG: this is necessary to get GMAC to work reliable on the
+	 * Bananapi. We don't know what these undocumented bits do, so this
+	 * is a Bananapi specific hack for now.
+	 */
+#ifdef CONFIG_BANANAPI
+	setbits_le32(&ccm->gmac_clk_cfg, 0x3 << 10);
+#endif
+
 	/* Configure pin mux settings for GMAC */
 	for (pin = SUNXI_GPA(0); pin <= SUNXI_GPA(16); pin++) {
 #ifdef CONFIG_RGMII