OMAP baseline test results for v4.1-rc5

Here are some basic OMAP test results for Linux v4.1-rc5. Logs and other details at:

    http://www.pwsan.com/omap/testlogs/test_v4.1-rc5/20150529162206/

Test summary
------------

Build: uImage:

    Pass ( 3/ 3): omap1_defconfig, omap1_defconfig_1510innovator_only,
		  omap1_defconfig_5912osk_only

Build: uImage+dtb:

    Pass (13/13): omap2plus_defconfig_am33xx_only/am335x-bone,
		  omap2plus_defconfig/omap4-panda,
		  omap2plus_defconfig/omap4-panda-es,
		  omap2plus_defconfig/omap4-var-stk-om44,
		  omap2plus_defconfig/omap3-evm-37xx,
		  omap2plus_defconfig_n800_only_a/omap2420-n800,
		  omap2plus_defconfig/omap2430-sdp,
		  omap2plus_defconfig/am3517-evm,
		  omap2plus_defconfig/omap3-beagle,
		  omap2plus_defconfig/omap3-beagle-xm,
		  omap2plus_defconfig/omap3-sbc-t3517,
		  omap2plus_defconfig/omap5-uevm,
		  omap2plus_defconfig/omap5-sbc-t54

Build: zImage:

    Pass (17/17): omap2plus_defconfig, omap2plus_defconfig_am33xx_only,
		  omap2plus_defconfig_n800_only_a,
		  omap2plus_defconfig_n800_multi_omap2xxx,
		  omap2plus_defconfig_2430sdp_only,
		  omap2plus_defconfig_cpupm, omap2plus_defconfig_no_pm,
		  omap2plus_defconfig_omap2_4_only,
		  omap2plus_defconfig_omap3_4_only,
		  omap2plus_defconfig_omap5_only,
		  omap2plus_defconfig_dra7xx_only,
		  omap2plus_defconfig_am43xx_only,
		  rmk_omap3430_ldp_oldconfig,
		  rmk_omap3430_ldp_allnoconfig,
		  rmk_omap4430_sdp_oldconfig,
		  rmk_omap4430_sdp_allnoconfig, multi_v7_defconfig

Build warnings from toolchain: uImage:

    FAIL ( 3/ 3): omap1_defconfig, omap1_defconfig_1510innovator_only,
		  omap1_defconfig_5912osk_only

Build warnings from toolchain: uImage+dtb:

    (none)

Build warnings from toolchain: zImage:

    FAIL (14/17): omap2plus_defconfig, omap2plus_defconfig_am33xx_only,
		  omap2plus_defconfig_n800_only_a,
		  omap2plus_defconfig_n800_multi_omap2xxx,
		  omap2plus_defconfig_2430sdp_only,
		  omap2plus_defconfig_cpupm, omap2plus_defconfig_no_pm,
		  omap2plus_defconfig_omap2_4_only,
		  omap2plus_defconfig_omap3_4_only,
		  omap2plus_defconfig_omap5_only,
		  omap2plus_defconfig_dra7xx_only,
		  omap2plus_defconfig_am43xx_only,
		  rmk_omap3430_ldp_oldconfig,
		  rmk_omap4430_sdp_oldconfig

Boot to userspace:

    FAIL ( 2/17): 2430sdp, 5430es2sbct54
    skip ( 2/17): 5912osk, 3517evm
    Pass (13/17): am335xbonelt, am335xbone, 4430es2panda, 4460pandaes,
		  4460varsomom, 37xxevm, 3530es3beagle, 3530es31beagle,
		  3730beaglexm, 3730es12beaglexm, cmt3517, 5430es2uevm,
		  2420n800

Kernel warnings during boot to userspace:

    FAIL ( 2/17): 4430es2panda, cmt3517

PM: chip retention via suspend:

    FAIL ( 6/12): am335xbonelt, 4430es2panda, 4460varsomom, 2430sdp,
		  5430es2uevm, 5430es2sbct54
    Pass ( 6/12): 4460pandaes, 37xxevm, 3530es3beagle, 3530es31beagle,
		  3730beaglexm, 3730es12beaglexm

PM: chip retention via dynamic idle:

    FAIL ( 6/12): am335xbonelt, 4430es2panda, 4460varsomom, 2430sdp,
		  5430es2uevm, 5430es2sbct54
    Pass ( 6/12): 4460pandaes, 37xxevm, 3530es3beagle, 3530es31beagle,
		  3730beaglexm, 3730es12beaglexm

PM: chip off (except CORE, due to errata) via suspend:

    Pass ( 1/ 1): 3730beaglexm

PM: chip off (except CORE, due to errata) via dynamic idle:

    Pass ( 1/ 1): 3730beaglexm

PM: chip off via suspend:

    Pass ( 4/ 4): 37xxevm, 3530es3beagle, 3530es31beagle,
		  3730es12beaglexm

PM: chip off via dynamic idle:

    Pass ( 4/ 4): 37xxevm, 3530es3beagle, 3530es31beagle,
		  3730es12beaglexm

Kernel warnings during PM test:

    FAIL ( 1/17): 4430es2panda

Obsolete Kconfig symbols:

    FAIL ( 1/20): multi_v7_defconfig

vmlinux object size
(delta in bytes from test_v4.1-rc4 (e26081808edadfd257c6c9d81014e3b25e9a6118)):

   text     data      bss    total  kernel
  +1658        0        0    +1658  omap1_defconfig
  +1658        0        0    +1658  omap1_defconfig_1510innovator_only
  +1658        0        0    +1658  omap1_defconfig_5912osk_only
 +28262   +52928        0   +81190  multi_v7_defconfig
  +2678        0        0    +2678  omap2plus_defconfig
  +2130        0       -8    +2122  omap2plus_defconfig_2430sdp_only
  +2550      +64        0    +2614  omap2plus_defconfig_am33xx_only
  +2550        0        0    +2550  omap2plus_defconfig_am43xx_only
  +2678        0        0    +2678  omap2plus_defconfig_cpupm
  +2614        0        0    +2614  omap2plus_defconfig_dra7xx_only
  +1226        0        0    +1226  omap2plus_defconfig_n800_multi_omap2xxx
  +1230        0        0    +1230  omap2plus_defconfig_n800_only_a
  +2042        0        0    +2042  omap2plus_defconfig_no_pm
  +2614        0        0    +2614  omap2plus_defconfig_omap2_4_only
  +2554        0        0    +2554  omap2plus_defconfig_omap3_4_only
  +2550        0        0    +2550  omap2plus_defconfig_omap5_only
   +632        0      -80     +552  rmk_omap3430_ldp_allnoconfig
  +2078        0        0    +2078  rmk_omap3430_ldp_oldconfig
   +664        0      -80     +584  rmk_omap4430_sdp_allnoconfig
  +2978        0        0    +2978  rmk_omap4430_sdp_oldconfig

Boot-time memory difference
(delta in bytes from test_v4.1-rc4 (e26081808edadfd257c6c9d81014e3b25e9a6118))

  avail  rsrvd   high  freed  board          kconfig
    -8k     8k      .     4k  am335xbone     omap2plus_defconfig_am33xx_only

Thanks to help from Tony, Suman, Nishanth, and Tero, some of the earlier test failures in the v4.1-rc series have been tracked down. The 4460 VAR-SOM-OM was booting with an older DT file since the point in time last year when the DT file was renamed; this has now been fixed. Most of the other failures were due to timeout problems after commit cc4a5fe972ad7834e8662b49b3a5fdb597e9e15e ("arm: config:
omap2plus_defconfig: switch over to LZMA compression") which pushed the time interval from the point at which the "Starting kernel" message is emitted to the point at which the kernel starts to log serial output to the console over 5 seconds on many OMAP3 and earlier platforms. From my point of view this commit is a mistake. Anyone integrating a kernel into a consumer device will need to be sure to disable LZMA compression if reasonable boot times are desired. This is not something that the vast majority of integrators will be aware of.

The 5430es2sbct54 is still not booting to userspace; this is on the list here to bisect.

Hello Paul,

The cmt3517 seems to have these some In-Band errors. Do you happen to know where these are coming from?

Regards,
Jeroen

[    0.476773] In-band Error seen by MPU  at address 0
[    0.476802] ------------[ cut here ]------------
[    0.476848] WARNING: CPU: 0 PID: 1 at drivers/bus/omap_l3_smx.c:166 
omap3_l3_app_irq+0xcc/0x124()
[    0.476865] Modules linked in:
[    0.476899] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 
4.1.0-rc5-143184-gba155e2 #1
[    0.476916] Hardware name: Generic AM3517 (Flattened Device Tree)
[    0.476964] [<c0016504>] (unwind_backtrace) from [<c0012a44>] 
(show_stack+0x10/0x14)
[    0.476997] [<c0012a44>] (show_stack) from [<c05ddbd4>] 
(dump_stack+0x80/0x9c)
[    0.477028] [<c05ddbd4>] (dump_stack) from [<c004000c>] 
(warn_slowpath_common+0x78/0xb4)
[    0.477056] [<c004000c>] (warn_slowpath_common) from [<c0040064>] 
(warn_slowpath_null+0x1c/0x24)
[    0.477083] [<c0040064>] (warn_slowpath_null) from [<c0370ed0>] 
(omap3_l3_app_irq+0xcc/0x124)
[    0.477121] [<c0370ed0>] (omap3_l3_app_irq) from [<c009804c>] 
(handle_irq_event_percpu+0x64/0x204)
[    0.477150] [<c009804c>] (handle_irq_event_percpu) from [<c0098228>] 
(handle_irq_event+0x3c/0x5c)
[    0.477181] [<c0098228>] (handle_irq_event) from [<c009b09c>] 
(handle_level_irq+0xb4/0x13c)
[    0.477209] [<c009b09c>] (handle_level_irq) from [<c00978f0>] 
(generic_handle_irq+0x20/0x30)
[    0.477235] [<c00978f0>] (generic_handle_irq) from [<c0097a08>] 
(__handle_domain_irq+0x68/0xdc)
[    0.477263] [<c0097a08>] (__handle_domain_irq) from [<c0009480>] 
(omap_intc_handle_irq+0xb4/0xc4)
[    0.477299] [<c0009480>] (omap_intc_handle_irq) from [<c05e48a4>] 
(__irq_svc+0x44/0x5c)
[    0.477317] Exception stack(0xce0a3d00 to 0xce0a3d48)
[    0.477342] 3d00: 00000001 ce0a1138 00000000 ce0a0b80 60000153 
ce013c60 ce013c60 00000000
[    0.477365] 3d20: 0000001a 60000153 ce013c38 00000000 00000000 
ce0a3d48 c008dde8 c05e4424
[    0.477382] 3d40: 20000153 ffffffff

Hi,

git bisect + some workarounds seem to indicate:

d744ce37b721d6678f420ba0fb058f615eb015b6 is the first bad commit commit d744ce37b721d6678f420ba0fb058f615eb015b6

Author: Tero Kristo <[email protected]>
Date:   Tue Feb 24 16:22:45 2015 +0200

ARM: dts: omap3: add minimal l4 bus layout with control module support

This patch creates an l4_core interconnect for OMAP3, and moves some of the generic peripherals under it. System control module nodes are moved under this new interconnect also, and the SCM clock layout is changed to use the renamed SCM node as the clock provider.

I haven't looked further into it yet,

Regards, Jeroen

+ Tero

Hello Jeroen,

Interesting; thanks for the bisect. In the mainline kernel, this appears to be commit b8845074cfbbd1d1b46720a1b563d7b4240dac21.

I took a quick look at the control module offsets in that patch, and they appear to match what's in the SPRUGR0B PDF. Will try a few test boots here to confirm your findings.

Tero, care to take a look?

- Paul

Yes, seems I have introduced a bug with this patch on am35xx only. I missed updating part of the am35xx related dts files.

Will post a fix in a bit.

-Tero

New system control module layout for omap3 overlooked parts of the am35xx configuration. Basically the am35xx clocks were not converted to use the changed offsets, which caused weird boot warnings. The errors were not fatal so far, so they were not caught earlier. Fixed by applying the proper offsets for the AM35xx scm clocks.

---
 arch/arm/boot/dts/am35xx-clocks.dtsi |   14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

index 518b8fd..18cc826 100644
@@ -12,7 +12,7 @@
 		#clock-cells = <0>;
 		compatible = "ti,am35xx-gate-clock";
 		clocks = <&ipss_ick>;
-		reg = <0x059c>;
+		reg = <0x032c>;
 		ti,bit-shift = <1>;
 	};
 

@@ -20,7 +20,7 @@
 		#clock-cells = <0>;
 		compatible = "ti,gate-clock";
 		clocks = <&rmii_ck>;
-		reg = <0x059c>;
+		reg = <0x032c>;
 		ti,bit-shift = <9>;
 	};
 

@@ -28,7 +28,7 @@
 		#clock-cells = <0>;
 		compatible = "ti,am35xx-gate-clock";
 		clocks = <&ipss_ick>;
-		reg = <0x059c>;
+		reg = <0x032c>;
 		ti,bit-shift = <2>;
 	};
 

@@ -36,7 +36,7 @@
 		#clock-cells = <0>;
 		compatible = "ti,gate-clock";
 		clocks = <&pclk_ck>;
-		reg = <0x059c>;
+		reg = <0x032c>;
 		ti,bit-shift = <10>;
 	};
 

@@ -44,7 +44,7 @@
 		#clock-cells = <0>;
 		compatible = "ti,am35xx-gate-clock";
 		clocks = <&ipss_ick>;
-		reg = <0x059c>;
+		reg = <0x032c>;
 		ti,bit-shift = <0>;
 	};
 

@@ -52,7 +52,7 @@
 		#clock-cells = <0>;
 		compatible = "ti,gate-clock";
 		clocks = <&sys_ck>;
-		reg = <0x059c>;
+		reg = <0x032c>;
 		ti,bit-shift = <8>;
 	};
 

@@ -60,7 +60,7 @@
 		#clock-cells = <0>;
 		compatible = "ti,am35xx-gate-clock";
 		clocks = <&sys_ck>;
-		reg = <0x059c>;
+		reg = <0x032c>;
 		ti,bit-shift = <3>;
 	};
 };

1.7.9.5

Hello Tero,

With this patch the error interrupt / stack dumps are no longer present.

Thanks,

Thanks, I'm suprised this was not caught earlier with all the automated boot testing going on?

Anyways, applying into omap-for-v4.1/fixes.

Regards,

Tony

At least speaking in terms the testbed results that I post, the warnings get reported. But not many people seem to act on them. (Jeroen is a pleasant exception.)

See for example the "Build warnings from toolchain", "Kernel warnings during boot to userspace", "Kernel warnings during PM test", and "Obsolete Kconfig symbols" sections here:

http://www.pwsan.com/omap/testlogs/test_v4.1-rc6/20150601012139/README.txt

The best way to make this work IMHO would be for us not to accept any new feature addition patches as long as there are warnings reported in the test results. The only real exception that I would foresee is if those warnings are due to something outside of our control, e.g., a crappy bootloader, as I suspect the USB_OTG initiator warnings are for the CM-T3517.

- Paul

OK somehow 3517evm is listed under "skip" there?

Right. Also seems we should diff dmesg output too (after stripping out the timestamps). Pretty much the only things that should change are the sysfs paths for devices in the dmesg output unless some warnings get fixed. That output probably still also needs to be manually looked at though..

Regards,

Tony

Oh I see you have two 3517 devices there, never mind.

Hmm now I'm wondering what the panda es warnings listed there are..

Tony

http://www.pwsan.com/omap/testlogs/test_v4.1-rc6/20150601012139/boot/4430es2panda/4430es2panda_log.txt

Looked to me like an OMAP4430 ES2.2 bug. I recall discussing it with someone with an ES2.3 Pandaboard and they said it didn't show up. So I asked TI at the time if there was an erratum for it; apparently not. So I think we may need to add in another hardware bug workaround flag to the OMAP integration code...

- Paul

Don't know about pandaboard ES2.2, but this doesn't show up on SDP4430 es2.3 at least. Do you know which clockdomain is in question there? It could just be a race someplace in the usecounting system that shows up on that specific SoC.

-Tero

Yep that board is down right now; something's wrong with it and I haven't had the time to figure out whether it's fixable, or if it's fried.

Yep, there are still some other minor loose ends to be tied up too that don't show up as full warnings, such as the broken UART4 idle protocol integration on AM35xx...

- Paul

I added some extra logic into my test setup today for detecting this. Previously the dumps were pretty much hidden as there are existing dumps so I kind of ignored the new ones semi-blindly. >.<

-Tero

Hello Paul,

I doubt this is related to the bootloader. I have the suspicion that is actually
a bug in linux but only triggered depending on whether the ROMcode setup the USB OTG or not. Here is some data to backup my statement:

Linux booting without USB_OTG error trap
md 480022F0 1
480022f0: 0000032f /...
md 48002580 1
48002580: 0f00b7a2 ....

bit USBOTG_PHY_RESET is 0 -> out of reset

USB_OTG sees memory hole
md 480022F0 1
480022f0: 0000030f ....
md 48002580 1
48002580: 0f00c71e ....

USBOTG_PHY_RESET is 1 -> still in reset when booting linux.

Does that match with how your am3517 boards boot?

Regards,
Jeroen

ps. the dumped register are CONTROL.CONTROL_STATUS and CONTROL.CONTROL_DEVCONF2.

Hello Paul,

Turns out my suspicion was wrong. This is what I know at the moment, depending on the bootpins, u-boot will trigger a bad access when loading a file over ethernet, but only the first time. Clearing the pending interrupt
before booting linux make the "USB_OTG address hole seen" go away.

Regards,
Jeroen

U-Boot 2015.04-00098-g487ee34-dirty (Jun 05 2015 - 13:14:48)

AM35XX-GP ES2.0, CPU-OPP2, L3-165MHz, Max CPU Clock 600 Mhz ccgx + LPDDR/NAND

I2C:   ready
DRAM:  256 MiB
NAND:  512 MiB
MMC:   OMAP SD/MMC: 0
In:    serial
Out:   serial
Err:   serial

Die ID #2276000100000000014e0fb21500b024
Net: DaVinci-EMAC
Hit any key to stop autoboot: 0
ccgx=> echo $clear_l3_int
mw 68004428 0x11000000; mw 6800442C 0x00000000; mw 68004458 0xFFFFFFFF; mw 6800445C 0xFFFFFFFF
ccgx=> md 0x68000510 2
68000510: 00000000 00000000 ........
ccgx=> tftp ccgx/zImage; tftp 80000000 ccgx/am3517-ccgx.dtb; Using DaVinci-EMAC device
TFTP from server 10.0.0.103; our IP address is 10.0.0.250 Filename 'ccgx/zImage'.
Load address: 0x80300000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################

          ##########
          863.3 KiB/s

done
Bytes transferred = 4040072 (3da588 hex)
Using DaVinci-EMAC device
TFTP from server 10.0.0.103; our IP address is 10.0.0.250 Filename 'ccgx/am3517-ccgx.dtb'.
Load address: 0x80000000
Loading: ###########

          825.2 KiB/s

done
Bytes transferred = 54112 (d360 hex)
ccgx=> md 0x68000510 2
68000510: 04000000 00000000 ........
ccgx=> run clear_l3_int
ccgx=> md 0x68000510 2
68000510: 00000000 00000000 ........
ccgx=> boot

# USB_OTG error is gone...

Hi Jeroen,

Oh, too bad. I had been hoping that you were right and that I was wrong ;-) I'll try this on the CM-T3517 here.

I'd like to solicit your opinion on what to do here. I wonder if, in the L3 bus drivers, we should simply report, in a line or two, if any bus errors were logged before the kernel starts? That would help discriminate between problems that the kernel is responsible for, vs. problems that occur earlier in the boot process. Any thoughts?

Thanks for all your investigation, by the way, it's much appreciated.

regards,

- Paul

Hi Jeroen,

I used your debugging technique here and was able to reproduce your results - with one difference:

http://www.pwsan.com/omap/testlogs/test_v4.1-rc6/20150607194102/boot/cmt3517/cmt3517_log.txt

The interconnect error was logged upon the first interaction with the network. In my case this was with the U-boot 'dhcp' command. The pending interrupt bit was cleared before loading the kernel via tftp, and the interrupt bit was not set again, even after a tftp load.

regards,

- Paul

Hello Paul, +Menon (since you asked about the USB_OTG trap),

I sent a patch to u-boot to disable the offending line, see [1]. It would be interesting to know if it can also result in valid, accidental memory adjustments,
before the invalid one, but I haven't checked that yet.

Regards,
Jeroen

[1] https://patchwork.ozlabs.org/patch/481751/

p.s. the USB_OTG trap is actually a musb / emac / camera trap on an am3517.

Hello Paul,

I am not sure this can be easily done in a general manner. Since in general linux doesn't know the bootloader, it can't rely on what peripherals are setup,
so I doubt it is in general possible to store a copy of the interrupt registers
really early. Besides that, when not hacking around, I guess during the really
early boot stage it is unknown what the interrupt registers are in the first place
and which clocks are needed. And even if it could be done, if there is a bug in
that code, it would lead to bigger problems than it is trying to solve (a bit weird
backlogs).

I guess it would be easier to check these flags in u-boot right before the jump
to linux (there is a cleanup_before_linux() or something name like that). [ Or fake
the boot and run the check as a separate command]. Unfortunately u-boot does not have a complete driver model, so a implementation of that in todays u-boot
will lead to a complete #ifdef CONFIG_foo mesh.

So, if you ask me, no don't add it to linux, but to u-boot instead. But likely as feature
for a later release which can query all drivers. (and doubt this should be limited to
interrupt flags, runtime_test() could be called on all of them e.g. when entering a
command like runtime_test).

Regards,
Jeroen