Overo serial problems after resume, vs. Beagleboard

Hey there,

I am seeing some discrepancies between the Overo (I believe I have a Water) and the Beagleboard (I have a Rev. B5) when resuming after a suspend to RAM.

The setup: My kernel is built from the latest commit (commit ID 305f453e897e4673dd4c2b52ec7e2c4be2e2b035 [1]) of the branch named pm of Kevin Hilman's linux-omap-pm [2].

I take the omap3_pm_defconfig, and change the below options to Y as they seem to be needed to boot from SD card (Names given are names in menuconfig under Device Drivers-->MMC/SD/SDIO card support. Config symbols given in parentheses)
- MMC block device driver (MMC_BLOCK)
- TI OMAP High Speed Multimedia Card Interface support (MMC_OMAP_HS)
- Secure Digital Host Controller Interface support (MMC_SDHC)

My filesystem is omap3-console-image dated May 23 from Steve Sakoman's binaries [3]. I do need to note that I made a few changes: symlinked /sbin/init to /bin/busybox instead of /sbin/init.sysvinit and changed /etc/init.d/rcS and /etc/inittab to suit Busybox.

My uBoot is also taken from Steve Sakoman, and also dated May 23.

I use the same card for both the Beagle and the Overo. This reduces sources of confounding variables, as now the only difference is the hardware (and that I need a micro-SD adapter for the Beagle).

For Beagle:
Kernel command line: console=ttyS2,115200n8 mpurate=500 vram=12M omapfb.mode=dvi:1024x768MR-16@60 omapfb.debug=y omapdss.def_disp=dvi root=/dev/mmcblk0p2 rw rootfstype=ext3 rootwait

For Overo:
Kernel command line: console=ttyS2,115200n8 vram=12M omapfb.mode=dvi:1024x768MR-16@60 omapfb.debug=y omapdss.def_disp=dvi root=/dev/mmcblk0p2 rw rootfstype=ext3 rootwait

The Overo connects to my computer through USB, on the port on the Summit board labeled CONSOLE. The Beagle connects to my computer through RS232, a null modem cable, and a USB to serial adapter. I use screen.

The problem comes when I use echo mem > /sys/power/state to suspend to ram, and then press some key to wake up the system. The Beagleboard comes up just fine. On the other hand, the Overo initially looks like it comes up fine, but I can no longer type anything into the prompt that appears. This looks suspiciously similar to Han Wang's problem, but I did a little more exploring via a quick test script to see how far the system gets.

root@overo:~# cat test.sh
echo "I went to sleep" > log
echo mem > /sys/power/state
echo "HEY I WOKE UP"
echo "HEY I WOKE UP" >> log
sync
root@overo:~#

Interestingly enough, when I run this script and resume on the Overo, I can see HEY I WOKE UP both in the console and the file (after a restart so I can type something to read the file)! So the problem is simply that I cannot type anything, not that the system isn't waking up, otherwise it wouldn't even get to the echoes. (Blindly typing in things afterward, including echoing to a file, does not seem to have an effect, unfortunately)

Any ideas on what might be causing this discrepancy and whether there's anything I can do to fix? I appreciate your time.

Peter Tseng

[1]
http://git.kernel.org/?p=linux/kernel/git/khilman/linux-omap-pm.git;a=commit;h=305f453e897e4673dd4c2b52ec7e2c4be2e2b035
[2]
http://git.kernel.org/?p=linux/kernel/git/khilman/linux-omap-pm.git;a=summary
[3] http://www.sakoman.com/feeds/omap3/glibc/images/overo/

Not that it is much comfort, but I have the same problem on Overo but don't see it on any other OMAP3 board.

I have yet to debug this particular issue further.

Kevin

I've reported this issue earlier this month, unfortunately I have no clue what is happening. When you telnet (or ssh) to the board via Ethernet-LAN you can see that the board is suspending and resuming. Output to the serial console works fine. If you do:

from the ssh/telnet session, you'll see the message being printed to the serial console.

Also, if you have configured the serial console as a wakeup source:

then sending a character over the (USB) console link does resume the Overo.

Any character send over the (USB) console link *after* the Overo has resumed will lock up the system in one of two ways:

1) immediate

The system is locked completely, and the parallel telnet/ssh session cannot make any progress

2) gradual failure

The system becomes sluggish, especially noticable in the parallel telnet/ssh session that makes progress but only slowly. That situation gets worse progressively until there is no noticable progress anymore.

The fact that the serial console does produce output should be proof that there is no problem with the UART clocks. In case (2), 'top' running in the parallel telnet/ssh session shows virtually no system load, so the lockup/worsening performance seem to have a kernel origin.

I haven't found an effective way to debug this on the Overo.

Cheers,

Robert

I had the same issue on CM-T35 and when I've disabled CONFIG_CPU_IDLE in the kernel configuration the issue disappeared. Maybe it'll be of some help.

Sincerely yours,
Mike.

I have that disabled as well on my debug builds, but only in order not to have 1 char swallowed when the serial wakes the cpu... Still might be worth a try on overo/summit

I just tried that. It helps. I no longer experience this lockup of the console. Thanks.

I'll make a note to try to figure out why this actually helps.

Regards,

Robert

That just masks the problem by not auto-gating the UART clocks.

Kevin

Agree, but having this observation may help to identify the real problem :)

Sincerely yours,
Mike.

OK, I think I finally figured out this problem. A similar thing was reported on the n900 recently and got me to thinking that it must be something fishy going on with the PER powerdomain since both n900 and Overo are using a UART2 console, and UART2 is in PER.

Anyways, I just posted a patch[1] that should fix this. I'd appreciate if anyone else with an Overo could confirm that things are back to working as expected with CONFIG_CPU_IDLE.

Thanks,

Kevin

[1] https://patchwork.kernel.org/patch/118709/