Skip to contentSkip to conversation

rt-preempt: problem compiling rt-preempt 2.6.23.1-rt11 on MIPS

DISCUSSION Tim Bird 4 participants 11 messages Nov 15, 2007 – Nov 16, 2007

lore
Share & tools
lore

Tim Bird

author
#0 lore raw

I applied the patches in patch-2.6.23.1-rt11-broken-out.tar.bz2 to a Linux kernel version 2.6.23.1 (along with a few other board specific patches).

I got the following compilation error:

  GEN     /home/tbird/work/rt-preempt/build/tx49/Makefile
  CHK     include/linux/version.h
  CHK     include/linux/utsrelease.h
  CALL    /home/tbird/work/rt-preempt/linux-2.6.23.1-rt11/scripts/checksyscalls.sh
  CHK     include/linux/compile.h
  CC      kernel/latency_trace.o

/home/tbird/work/rt-preempt/linux-2.6.23.1-rt11/kernel/latency_trace.c:28:21: error: asm/rtc.h: No such file or directory make[2]: *** [kernel/latency_trace.o] Error 1
make[1]: *** [kernel] Error 2
make: *** [vmlinux] Error 2

Indeed, there is no include/asm-mips/rtc.h.

I commented out the include line in latency_trace.c, and everything compiled fine. I'm not sure what is needed in an arch-specific rtc.h, but compiling without it for the mips arch caused no problems. Should I create a patch with a stub for rtc.h for mips?

As an aside, this has me worried. Is anyone else doing any RT Preempt testing or work on MIPS platforms, or am I forging new ground? :-)

-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#1 lore raw

On Wed, 14 Nov 2007, Tim Bird wrote:

Added Steven and John to CC

Quoted 17 lines

I applied the patches in patch-2.6.23.1-rt11-broken-out.tar.bz2 to a Linux kernel version 2.6.23.1 (along with a few other board specific patches).

I got the following compilation error:

  GEN     /home/tbird/work/rt-preempt/build/tx49/Makefile
  CHK     include/linux/version.h
  CHK     include/linux/utsrelease.h
  CALL    /home/tbird/work/rt-preempt/linux-2.6.23.1-rt11/scripts/checksyscalls.sh
  CHK     include/linux/compile.h
  CC      kernel/latency_trace.o

/home/tbird/work/rt-preempt/linux-2.6.23.1-rt11/kernel/latency_trace.c:28:21: error: asm/rtc.h: No such file or directory make[2]: *** [kernel/latency_trace.o] Error 1
make[1]: *** [kernel] Error 2
make: *** [vmlinux] Error 2

Quoted 6 lines

Indeed, there is no include/asm-mips/rtc.h.

I commented out the include line in latency_trace.c, and everything compiled fine. I'm not sure what is needed in an arch-specific rtc.h, but compiling without it for the mips arch caused no problems. Should I create a patch with a stub for rtc.h for mips?

Hm, dunno why the rtc.h include was added.

Quoted 3 lines

As an aside, this has me worried. Is anyone else doing any RT Preempt testing or work on MIPS platforms, or am I forging new ground? :-)

I fear you are the one who is in charge to get mips working again :) But as always, there are bad news and good news: As far as I heard last week John Cooper is looking into this as well.

Thanks,

tglx

#2 lore raw
Quoted 22 lines · Thomas Gleixner

On Wed, 14 Nov 2007, Tim Bird wrote:

Added Steven and John to CC

Quoted 18 lines

I applied the patches in patch-2.6.23.1-rt11-broken-out.tar.bz2 to a Linux kernel version 2.6.23.1 (along with a few other board specific patches).

I got the following compilation error:

  GEN     /home/tbird/work/rt-preempt/build/tx49/Makefile
  CHK     include/linux/version.h
  CHK     include/linux/utsrelease.h
  CALL    /home/tbird/work/rt-preempt/linux-2.6.23.1-rt11/scripts/checksyscalls.sh
  CHK     include/linux/compile.h
  CC      kernel/latency_trace.o

/home/tbird/work/rt-preempt/linux-2.6.23.1-rt11/kernel/latency_trace.c:28:21: error: asm/rtc.h: No such file or directory make[2]: *** [kernel/latency_trace.o] Error 1
make[1]: *** [kernel] Error 2
make: *** [vmlinux] Error 2

Indeed, there is no include/asm-mips/rtc.h.

Looks like that may be just an artifact from older days.

I'll do some more tests, and it it truely is, I'll go and remove the added headers.

Quoted 8 lines
Quoted 5 lines

I commented out the include line in latency_trace.c, and everything compiled fine. I'm not sure what is needed in an arch-specific rtc.h, but compiling without it for the mips arch caused no problems. Should I create a patch with a stub for rtc.h for mips?

Hm, dunno why the rtc.h include was added.

Quoted 7 lines
Quoted 3 lines

As an aside, this has me worried. Is anyone else doing any RT Preempt testing or work on MIPS platforms, or am I forging new ground? :-)

I fear you are the one who is in charge to get mips working again :) But as always, there are bad news and good news: As far as I heard last week John Cooper is looking into this as well.

I'm just about to release 2.6.24-rc2-rt1 and I'm sure mips as well as powrepc is badly broken. Any help in getting these back up and working would be greatly appreciated.

-- Steve

author
#3 lore raw
Quoted 4 lines · Steven Rostedt

Looks like that may be just an artifact from older days.

I'll do some more tests, and it it truely is, I'll go and remove the added headers.

Thanks.

Quoted 3 lines

I'm just about to release 2.6.24-rc2-rt1 and I'm sure mips as well as powrepc is badly broken. Any help in getting these back up and working would be greatly appreciated.

I'll probably have some pretty basic questions.
If you don't mind an RT newbie helping, I'll do what I can. :-)

-- Tim

BTW - I was just trying to cross-compile the IBM test programs (rt-test-0.6) today, and had some problems. I didn't want to bug anyone until I took a closer look at it, but would it be best to report problems with that here on LKML, on linux-rt-users, or just directly to Darren Hart?
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#4 lore raw
Quoted 6 lines · Tim Bird
Quoted 3 lines

I'm just about to release 2.6.24-rc2-rt1 and I'm sure mips as well as powrepc is badly broken. Any help in getting these back up and working would be greatly appreciated.

I'll probably have some pretty basic questions.
If you don't mind an RT newbie helping, I'll do what I can. :-)

We can use all the help we can get :-)

Quoted 5 lines

BTW - I was just trying to cross-compile the IBM test programs (rt-test-0.6) today, and had some problems. I didn't want to bug anyone until I took a closer look at it, but would it be best to report problems with that here on LKML, on linux-rt-users, or just directly to Darren Hart?

Things related to rt-test should go to Darren Hart and to linux-rt-users. We only CC LKML on RT kernel issues.

-- Steve

#5 lore raw
In reply to Thomas Gleixner
Quoted 3 lines · Thomas Gleixner

I fear you are the one who is in charge to get mips working again :) But as always, there are bad news and good news: As far as I heard last week John Cooper is looking into this as well.

I'm not actively working on it but AFAIK I may have been the last one to touch it when I did the mips version back at Timesys. Although I was able to get a functional port of the work there were gremlins I never had sufficient time to address. That is a relatively minor issue.

The more daunting problem stems from limitations in the MIPS ABI which makes the latency trace support problematic. Rather than rehash the issue:

    http://lists.linuxcoding.com/kernel/2005-q4/msg10163.html

Until we have a usable instrumentation solution in place, characterization, debug, and support of PREEMPT_RT for MIPS is going to be a challenge.

-john

Signature · 1 line
[email protected]
author
#6 lore raw

MIPS RT debug support

Quoted 9 lines · john cooper

The more daunting problem stems from limitations in the MIPS ABI which makes the latency trace support problematic. Rather than rehash the issue:

    http://lists.linuxcoding.com/kernel/2005-q4/msg10163.html

Until we have a usable instrumentation solution in place, characterization, debug, and support of PREEMPT_RT for MIPS is going to be a challenge.

Agreed. I have been using KFT (Kernel Function Trace) on MIPS, and it has decent support for function traceback reporting, but it's not currently integrated with latency-trace
at all. We should discuss if this could possibly be
used to debug RT-preempt. It is much heavier weight than the mcount stuff, but uses similar (but not identical) gcc profiling instrumentation. I'm not sure if the two can be turned on together, or how hard it would be to move latency-trace onto -finstrument_functions.

But it's probably worth researching a little. We'll need something to give insight into the problem paths.
-- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================

#7 lore raw

Re: MIPS RT debug support

Quoted 11 lines · Tim Bird
Quoted 9 lines · john cooper

The more daunting problem stems from limitations in the MIPS ABI which makes the latency trace support problematic. Rather than rehash the issue:

    http://lists.linuxcoding.com/kernel/2005-q4/msg10163.html

Until we have a usable instrumentation solution in place, characterization, debug, and support of PREEMPT_RT for MIPS is going to be a challenge.

Quoted 9 lines

Agreed. I have been using KFT (Kernel Function Trace) on MIPS, and it has decent support for function traceback reporting, but it's not currently integrated with latency-trace
at all. We should discuss if this could possibly be
used to debug RT-preempt. It is much heavier weight than the mcount stuff, but uses similar (but not identical) gcc profiling instrumentation. I'm not sure if the two can be turned on together, or how hard it would be to move latency-trace onto -finstrument_functions.

I'm not familiar with the KFT but I'm sure it would be easy to port latency_trace to it. Really, all the mcount does is make a wrapper to pass to the trace calls.

Here's the code for mcount in arch/x86/kernel/entry_64.S:

ENTRY(mcount)

	cmpl $0, mcount_enabled
	jz out

	push %rbp
	mov %rsp,%rbp

	push %r11
	push %r10
	push %r9
	push %r8
	push %rdi
	push %rsi
	push %rdx
	push %rcx
	push %rax

	mov 0x0(%rbp),%rax
	mov 0x8(%rbp),%rdi
	mov 0x8(%rax),%rsi

	call   __trace

	pop %rax
	pop %rcx
	pop %rdx
	pop %rsi
	pop %rdi
	pop %r8
	pop %r9
	pop %r10
	pop %r11

	pop %rbp

out:
ret

Which simply passes to __trace the rip that jumped here, and (if possible) the rip of that caller. The parent rip is not necessary.

Quoted 3 lines

But it's probably worth researching a little. We'll need something to give insight into the problem paths.

If the KFT could do the above, it should be trivial to adapt.

Hmm, if someone is willing to send me a free mips box, I may do it myself ;-)

-- Steve

#8 lore raw

Re: MIPS RT debug support

Quoted 22 lines · Steven Rostedt
Quoted 10 lines · Tim Bird
Quoted 9 lines · john cooper

The more daunting problem stems from limitations in the MIPS ABI which makes the latency trace support problematic. Rather than rehash the issue:

    http://lists.linuxcoding.com/kernel/2005-q4/msg10163.html

Until we have a usable instrumentation solution in place, characterization, debug, and support of PREEMPT_RT for MIPS is going to be a challenge.

Quoted 9 lines

Agreed. I have been using KFT (Kernel Function Trace) on MIPS, and it has decent support for function traceback reporting, but it's not currently integrated with latency-trace
at all. We should discuss if this could possibly be
used to debug RT-preempt. It is much heavier weight than the mcount stuff, but uses similar (but not identical) gcc profiling instrumentation. I'm not sure if the two can be turned on together, or how hard it would be to move latency-trace onto -finstrument_functions.

Quoted 3 lines

I'm not familiar with the KFT but I'm sure it would be easy to port latency_trace to it. Really, all the mcount does is make a wrapper to pass to the trace calls.

It isn't an issue of getting a hook into the FUNCTION_PROLOGUE (mcount() here) but rather of emulating the CALLER_ADDR[0123] defs which map onto the gcc internal __builtin_return_address(). Doing so using the affectionately dubbed "Three Stooges Algorithm" called out in the MIPS ABI is both complex and nondeterministic.

The entry FUNCTION_PROLOGUE hook provides a means to log the path traveled thus far. __builtin_return_address() gives a way to snapshot a stub of the stack invocation. Together they provide somewhat complimentary but useful debug information.

We could log the stack invocation progress in the latency instrumentation itself by noting a new invocation in the FUNCTION_PROLOGUE and unwind of the same in the currently unused FUNCTION_EPILOGUE hook. So we're just shadowing the actual runtime stack in effect with a data structure which can be traversed far more easily.

-john

Signature · 1 line
[email protected]
#9 lore raw

Re: MIPS RT debug support

Quoted 12 lines · john cooper

It isn't an issue of getting a hook into the FUNCTION_PROLOGUE (mcount() here) but rather of emulating the CALLER_ADDR[0123] defs which map onto the gcc internal __builtin_return_address(). Doing so using the affectionately dubbed "Three Stooges Algorithm" called out in the MIPS ABI is both complex and nondeterministic.

The entry FUNCTION_PROLOGUE hook provides a means to log the path traveled thus far. __builtin_return_address() gives a way to snapshot a stub of the stack invocation. Together they provide somewhat complimentary but useful debug information.

Quoted 7 lines

We could log the stack invocation progress in the latency instrumentation itself by noting a new invocation in the FUNCTION_PROLOGUE and unwind of the same in the currently unused FUNCTION_EPILOGUE hook. So we're just shadowing the actual runtime stack in effect with a data structure which can be traversed far more easily.

IIRC, only CALLER_ADDR0 is actually used (I've found that the others are mostly unreliable). I thought that mips always has one register that holds the return address of the function. It's been several years since I've worked on mips (I'd love to do it again, if someone sends me a box!, hint hint). So the __builtin_return_address(0) should simply return that register. Now I'm being a bit lazy, and I'm not pulling out my MIPS books, so I could be completely wrong on this.

-- Steve

#10 lore raw

Re: MIPS RT debug support

Quoted 2 lines · Steven Rostedt

IIRC, only CALLER_ADDR0 is actually used (I've found that the others are mostly unreliable).

I seem to recall more use was made of __builtin_return_address(n) for 0 < n in the past compared to the current code. Likely for shallow frames the 0 < n calls were potentially returning invalid data which is why they currently appear to be commented out.

Rather than lobbying to fix __builtin_return_address(), replacing it with a primitive which works in all cases seems an easier solution. That stub stack frame really does provide useful debug information. In the case of ARM I simply walked the stack. Worst case in that vintage of the code was four iterations so the overhead was in the noise. That same solution could easily be applied to other architectures.

Quoted 3 lines

It's been several years since
I've worked on mips (I'd love to do it again, if someone sends me a box!, hint hint).

Concerning a MIPS board, if no freebies should arrive you might want to dig up a linux footprint friendly version of a Linksys WRT54GS.

-john

Signature · 1 line
[email protected]