Skip to contentSkip to conversation

TK1 System Freeze

DISCUSSION Marcel Ziswiler 4 participants 6 messages Nov 19, 2017 – Feb 7, 2018 3 messages not in the archive Tegra architecture support

lore
Share & tools
lore
#0 lore raw

Hi there

I lately was tasked to run some legacy Qt4e [1] application on Apalis TK1. It should really only draw directly to the Linux frame buffer /dev/fb0. Strangely as soon as that application is started the whole system freezes. Running it with the VNC backend or a pure frame buffer emulation it works just fine. So it must have something to do with the particular way the frame buffer is done on TK1. So far no attempt in debugging this any further bear any fruit. Whether stracing what the application is doing nor tracing the linux kernel side of things revealed where exactly the freeze happens. As I feared some kind of a configuration issue on Apalis TK1 I also tried the same on Jetson TK1 with latest stock Linux kernel 4.14. However the same freeze happens.

Has anybody ever seen a similar complete system freeze? Any ideas what it may be or how we could debug this any further?

[1] https://layers.openembedded.org/layerindex/recipe/43622/

Cheers

Marcel

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

Hi Marcel, Hi Thierry,

Quoted 13 lines · Marcel Ziswiler

Hi there

I lately was tasked to run some legacy Qt4e [1] application on Apalis TK1. It should really only draw directly to the Linux frame buffer /dev/fb0. Strangely as soon as that application is started the whole system freezes. Running it with the VNC backend or a pure frame buffer emulation it works just fine. So it must have something to do with the particular way the frame buffer is done on TK1. So far no attempt in debugging this any further bear any fruit. Whether stracing what the application is doing nor tracing the linux kernel side of things revealed where exactly the freeze happens. As I feared some kind of a configuration issue on Apalis TK1 I also tried the same on Jetson TK1 with latest stock Linux kernel 4.14. However the same freeze happens.

This is still the case with 4.15-rc7.

While dd is not enough to reproduce it, using a simple fbdev application such as ts_calibrate seems to be sufficient. Qt and ts_calibrate use mmap for the framebuffer. The system seems to freeze when it tries to write into the mapped area (at the memset line):
https://github.com/kergoth/tslib/blob/master/tests/fbutils-linux.c#L160

Is this a known problem? Any idea?

--
Stefan

Quoted 9 lines

Has anybody ever seen a similar complete system freeze? Any ideas what it may be or how we could debug this any further?

[1] https://layers.openembedded.org/layerindex/recipe/43622/

Cheers

Marcel

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 19 lines · Stefan Agner

Hi Marcel, Hi Thierry,

Quoted 13 lines · Marcel Ziswiler

Hi there

I lately was tasked to run some legacy Qt4e [1] application on Apalis TK1. It should really only draw directly to the Linux frame buffer /dev/fb0. Strangely as soon as that application is started the whole system freezes. Running it with the VNC backend or a pure frame buffer emulation it works just fine. So it must have something to do with the particular way the frame buffer is done on TK1. So far no attempt in debugging this any further bear any fruit. Whether stracing what the application is doing nor tracing the linux kernel side of things revealed where exactly the freeze happens. As I feared some kind of a configuration issue on Apalis TK1 I also tried the same on Jetson TK1 with latest stock Linux kernel 4.14. However the same freeze happens.

This is still the case with 4.15-rc7.

Quoted 7 lines

While dd is not enough to reproduce it, using a simple fbdev application such as ts_calibrate seems to be sufficient. Qt and ts_calibrate use mmap for the framebuffer. The system seems to freeze when it tries to write into the mapped area (at the memset line):
https://github.com/kergoth/tslib/blob/master/tests/fbutils-linux.c#L160

Is this a known problem? Any idea?

Thierry, anybody else, any idea? This should be easily reproducible on Jetson TK1.

--
Stefan

Quoted 13 lines
Quoted 9 lines

Has anybody ever seen a similar complete system freeze? Any ideas what it may be or how we could debug this any further?

[1] https://layers.openembedded.org/layerindex/recipe/43622/

Cheers

Marcel

--
To unsubscribe from this list: send the line "unsubscribe linux-tegra" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

Hi Stefan,

Quoted 26 lines · On Tue, 06 Feb 2018 08:51:19 +0100 stefan-XLVq0VzYD2Y-XMD5yJDbdMReXY1tMh2IBg@pu…
Quoted 25 lines · Stefan Agner

Hi Marcel, Hi Thierry,

Quoted 13 lines · Marcel Ziswiler

Hi there

I lately was tasked to run some legacy Qt4e [1] application on Apalis TK1. It should really only draw directly to the Linux frame buffer /dev/fb0. Strangely as soon as that application is started the whole system freezes. Running it with the VNC backend or a pure frame buffer emulation it works just fine. So it must have something to do with the particular way the frame buffer is done on TK1. So far no attempt in debugging this any further bear any fruit. Whether stracing what the application is doing nor tracing the linux kernel side of things revealed where exactly the freeze happens. As I feared some kind of a configuration issue on Apalis TK1 I also tried the same on Jetson TK1 with latest stock Linux kernel 4.14. However the same freeze happens.

This is still the case with 4.15-rc7.

While dd is not enough to reproduce it, using a simple fbdev application such as ts_calibrate seems to be sufficient. Qt and ts_calibrate use mmap for the framebuffer. The system seems to freeze when it tries to write into the mapped area (at the memset line):
https://github.com/kergoth/tslib/blob/master/tests/fbutils-linux.c#L160

Quoted 5 lines
Quoted 1 line

Is this a known problem? Any idea?

Thierry, anybody else, any idea? This should be easily reproducible on Jetson TK1.

Some time ago I was able to cause a similar-sounding hang by running kmscon[1] on the Jetson TK1. I remember that I tried the debug the problem with a Lauterbach JTAG debugger at some point, but whatever steps that I did to make the Lauterbach usable on mainline also made the problem disappear! IIRC, it was disabling cpuidle (either via cmdline or sysfs, can't remember) that made the hang go away.

[1]: https://github.com/dvdhrm/kmscon

- Tuomas

#7 lore raw
In reply to a message not in the archive
Quoted 26 lines · On Tue, Feb 06, 2018 at 08:51:19AM +0100, [email protected]…
Quoted 25 lines · Stefan Agner

Hi Marcel, Hi Thierry,

Quoted 13 lines · Marcel Ziswiler

Hi there

I lately was tasked to run some legacy Qt4e [1] application on Apalis TK1. It should really only draw directly to the Linux frame buffer /dev/fb0. Strangely as soon as that application is started the whole system freezes. Running it with the VNC backend or a pure frame buffer emulation it works just fine. So it must have something to do with the particular way the frame buffer is done on TK1. So far no attempt in debugging this any further bear any fruit. Whether stracing what the application is doing nor tracing the linux kernel side of things revealed where exactly the freeze happens. As I feared some kind of a configuration issue on Apalis TK1 I also tried the same on Jetson TK1 with latest stock Linux kernel 4.14. However the same freeze happens.

This is still the case with 4.15-rc7.

While dd is not enough to reproduce it, using a simple fbdev application such as ts_calibrate seems to be sufficient. Qt and ts_calibrate use mmap for the framebuffer. The system seems to freeze when it tries to write into the mapped area (at the memset line):
https://github.com/kergoth/tslib/blob/master/tests/fbutils-linux.c#L160

Quoted 4 lines
Quoted 1 line

Is this a known problem? Any idea?

Thierry, anybody else, any idea? This should be easily reproducible on Jetson TK1.

I was able to reproduce this on Venice2 (just because I was using it for unrelated work, I'm pretty certain this can be reproduced on Jetson TK1) and came up with a fix. I think the problem is that fbdev has an mmap() implementation that conflicts with what we do in DRM/KMS and therefore causes a hang. I've sent out a series with the fix. It'd be great if you two can check that this indeed fixes the issue for you on Apalis. It did fix the issue for me on Venice2.

Tested using fbdev.c from here:

	https://patchwork.kernel.org/patch/743682/

This seems like it would have been broken since the dawn of time. Have you never encountered this before? Or has it only recently broken with newer kernels? We probably want this backported to all stable releases.

Thierry

#8 lore raw
Quoted 32 lines · Thierry Reding
Quoted 26 lines · On Tue, Feb 06, 2018 at 08:51:19AM +0100, [email protected]…
Quoted 25 lines · Stefan Agner

Hi Marcel, Hi Thierry,

Quoted 13 lines · Marcel Ziswiler

Hi there

I lately was tasked to run some legacy Qt4e [1] application on Apalis TK1. It should really only draw directly to the Linux frame buffer /dev/fb0. Strangely as soon as that application is started the whole system freezes. Running it with the VNC backend or a pure frame buffer emulation it works just fine. So it must have something to do with the particular way the frame buffer is done on TK1. So far no attempt in debugging this any further bear any fruit. Whether stracing what the application is doing nor tracing the linux kernel side of things revealed where exactly the freeze happens. As I feared some kind of a configuration issue on Apalis TK1 I also tried the same on Jetson TK1 with latest stock Linux kernel 4.14. However the same freeze happens.

This is still the case with 4.15-rc7.

While dd is not enough to reproduce it, using a simple fbdev application such as ts_calibrate seems to be sufficient. Qt and ts_calibrate use mmap for the framebuffer. The system seems to freeze when it tries to write into the mapped area (at the memset line):
https://github.com/kergoth/tslib/blob/master/tests/fbutils-linux.c#L160

Quoted 4 lines
Quoted 1 line

Is this a known problem? Any idea?

Thierry, anybody else, any idea? This should be easily reproducible on Jetson TK1.

Quoted 7 lines

I was able to reproduce this on Venice2 (just because I was using it for unrelated work, I'm pretty certain this can be reproduced on Jetson TK1) and came up with a fix. I think the problem is that fbdev has an mmap() implementation that conflicts with what we do in DRM/KMS and therefore causes a hang. I've sent out a series with the fix. It'd be great if you two can check that this indeed fixes the issue for you on Apalis. It did fix the issue for me on Venice2.

Thanks for looking into this!

Yeah I suspected that missing mmap in fbdev emulation is the issue since Tegra was pretty much the only "higher end" graphics driver which did not overwrite the fbdev mmap callback.

FWIW, tested our use-case (Qt4e) on a patched 4.14, works like a charm!

Quoted 8 lines

Tested using fbdev.c from here:

	https://patchwork.kernel.org/patch/743682/

This seems like it would have been broken since the dawn of time. Have you never encountered this before? Or has it only recently broken with newer kernels? We probably want this backported to all stable releases.

Yeah we discussed that too, we realized that only now when we tried to use mainline with this legacy application.

Marcel remembered that he used to observe crashes when using fbdev & X.org but never really bothered and just used modesetting...

It is kind of a nice DoS (as in Denial of System :-)). Agreed, this should be backported.

--
Stefan