Debugging the MIPS processor using GDB

The questions are many, and I don't really know where to begin, but I shall make a try.

I am developing software to debug a MIPS processor in an embedded system using an EJTAG probe. Currently I have software to tell the processor to perform certain instructions, and read/write from/to certain register addresses. Now, I don't really know how to continue from here; I'm trying to find out how to make GDB benefit from these functions in order to be able to debug the processor.

I know that GDB needs a gdb-stub, but I don't really know what it is and what it does, or why GDB needs it. The GDB manual suggests taking a look at sparc-stub.c coming along with GDB, since it's the best organized. Not finding any main function in it, I'm having a hard time figuring out to what it does. How should this stub be implemented and how do I know what to write in the stub file I'm going to write?

There is also something called gdb-hook, which I don't really know what it is either. Do I need it?

Finally, there's a program called gdbserver, which comes with GDB. If I write a remote stub, do I need this program? Where should it be run? Where should my program be run? Since the stub is a c file, but lacks of a main function, how do I compile it?

There are just so many unanswered questions, I don't know where to start working or where to start looking for information. I haven't used GDB before either, possibly contributing to some of the confusion; I hope I'll get this straight soon, though.

/Kristofer

View this message in context: http://www.nabble.com/Debugging-the-MIPS-processor-using-GDB-tp18944199p18944199.html
Sent from the linux-mips main mailing list archive at Nabble.com.

At least this I can answer. In short, you need to call gdbserver on your target machine in order to do remote debugging from your workstation. The first Google match for "using gdbserver" reveals this:

http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/gdb/server.html

HTH

Martin

To the best of my knowledge, ‘gdbserver’ itself is not normally
used with (E)JTAG probes; it's mostly for debugging userland code on a remote target machine whilst running ‘gdb’ itself on a local host workstation.

I'm using the commercial FS² (First Silicon Systems, now owned
by MIPS) EJTAG probe. The local ‘gdb’ on the workstation talks to the local FS² software on the workstation, which talks to the probe (in my case, over USB, but there is also an Ethernet model). There is no ‘gdbserver’ in this setup per se, albeit I suppose the protocol between ‘gdb’ and the FS² software (which is called something like ‘jnetserver’?) might be similar/identical (I have no idea!).

cheers!

	-blf-

“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com

Not really -- it uses a C API called MDI -- the spec is available from
MIPS Technologies. I am happy to read somebody finds it useful. :) Debugging the Linux kernel with GDB and this piece of hardware is certainly a lot of fun.

Maciej

Maciej,

Thanks for the clarification. I didn't know if MDI
was related to the remote-‘gdbserver’ stuff or not.

Re the FS²: When it works, my (somewhat limited)
experience to-date is it works Ok. And the use of TCL on the Host workstation side allows some neat tricks. However, at least one thing doesn't work reliably for me, albeit I've never investigated:
Breakpoints in the Linux kernel. They do detonate. Then, sometimes, I can ‘c’(ontinue) or ‘s’(tep) Ok. But other times, when I ‘c’ or ‘s’, the breakpoint detonates again and I'm stuck. I cannot proceed. (The same breakpoint might even work once or twice
and then fail.) Any ideas? AFAICR, this can also
happen if I try to use the ‘sysnav’ console instead of ‘gdb’.

I understand my predecessor in my job I gave up on
the FS² (very possibly because of this breakpoint issue?) and used a competing (E?)JTAG probe.

Weirdly, I've only seen this effect with the Linux
kernel — other kernel-mode software (e.g., the trivial custom bootloader) — doesn't seem to suffer from these “flakey FS² breakpoints”?

cheers!

	-blf-

“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com

Hmm, odd. It looks like a cache coherence issue. It could be a bug in
your version of FS2 software -- did you raise the issue with them? Anyway, as a workaround try setting "coherent=on" (quoting from memory) in fs2.ini (just an idea -- it may not work and you will lose some performance though) or use hardware breakpoints.

Maciej

Maciej,

That would be my guess also.

I've been trying to. (I cannot say more on this ATM, sorry!)

I'm using the most recent FS² (2.4.4) with the most recent
SDE-Lite from MIPS (V6.06.01). Older versions of FS²/SDE had the same(?) issue. (This is with a 4KSd core, running Little Endian.)

What _might_ be an issue/cause is we're using our own
home-grown ‘gdb’ scripts (to init the memory, load the kernel, etc.). I didn't write them, but I have looked them over, and they _look_ Ok to me.

As it turns out, I _have_ been running Coherent On!
So I tried turning if Off, just to see what would happen. No difference.

The behavior I saw was:

1. gdb b xxx_open
2. target cat /dev/xxx
3. (breakpoint detonates)
4. gdb x/i $pc (all is Ok)
5. gdb c (Ok)
6. target cat /dev/xxx
7. (breakpoint detonates)
8. gdb x/i $pc (wrong! instruction is ‘sdbbp’.)

I'm now stuck. Any attempt to ‘c’ or ‘s’ just hits the
breakpoint again.

I tried some “mdi cacheflush” at some plausible-seeming
points, all to no effect. I also tried deleting the breakpoint (after step 8), which was a disaster: (From memory) when I then ‘c’(ontinued), ‘gdb’ hung, and the ‘sysnav’ went into an infinite loop of reporting a breakpoint. ;-(

( I seem to recall also having an issue with hardware
breakpoints, but cannot recall for certain ATM; tests will have to wait until later .... ;-\ )

All ideas and suggestions are very welcome!

cheers!

	-blf-

“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com

That should not matter. For example you could boot your system in the
usual way provided by the firmware and attach to an already running kernel. The probe does not know or care about that.

Your situation looks pretty miserable -- you should definitely pester
FS2.

Maciej

Hello!

Just in case someone recognizes these symptoms (or has any ideas) ....

I tried hardware breakpoints from ‘gdb’ and got:

h1. gdb hb xxx_open
h2. gdb says:
warning: Cannot query hardware breakpoint/watchpoint resources.
warning: Available functionality may be limited.
h3. target cat /dev/xxx
h4. Nothing happens! (The breakpoint does not detonate.)

The warnings from ‘sde-gdb’ (h2) have me puzzled. As I read the VHDL synthesis definitions for our implementation of the 4KSd core — an advantage of working for a HW company! — we should have 4 I and 2 D hardware breakpoints in Debug (EJTAG)
Mode. Related, and maybe significantly, ‘info reg’ from
‘sde-gdb’ reports “xxxxxxxx” for the ‘debug’ register (WTF?).

I also tried the ‘sysnav’ command ‘bkpt’ (via the ‘monitor’
command in ‘gdb’). It works! ‘set’, ‘setsw’, and ‘sethw’
all work as expected in my simple test (below, 0xVIRTUAL is the virtual address of xxx_open()):

m1. gdb monitor bkpt set 0xVIRTUAL
m2. target cat /dev/xxx
m3. (breakpoint detonates)
m4. gdb x/i $pc (all is Ok)
m5. gdb c (Ok)
m6. target cat /dev/xxx
m7. (breakpoint detonates)
m8. gdb x/i $pc (all is Ok)
m9. gdb c (Ok)
(... and repeat ad infinitum (‘s’ also works) ...)

I presume I can cook up some ‘gdb’ macros to can (simplify) those horrible ‘monitor bkpt ...’ commands, but I shouldn't have to do that, should I? ;-\

Any ideas? (And yes, I'm attempting to pursue this with
FS²/MIPS, but .... .)

cheers!

	-blf-

“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com

This warning is issued because FS2's MDIlib does not implement the
MDIHwBpQuery() call, which tells the MDI client what hardware breakpoint capabilities the given implementation of MDIlib provides. They generally correspond to what the debug port provides, but there is room for some abstraction to be provided by MDIlib, for example a given library implementation may emulate ways to specify address ranges different to what hardware implements.

Therefore GDB cannot determine whether the intended breakpoint type will
or will not fail at the time MDISetBp() is called to actually insert the breakpoint into the target, which only happens when execution is resumed. There is a heuristic implemented in GDB that tries to figure out what the MDIlib supports by trial and error, but in the end the target may not really support what GDB requests and you'll get a failure when you try to start execution. The message therefore serves as a precautionary warning.

They really ought to implement this call -- I suppose if customers like
you keep pushing, they may eventually decide it is worth the effort.

Anyway, the warning is normal and happened for all releases of FS2
software I have ever used.

You should be able to see a hardware breakpoint set by GDB from their
System Navigator console as soon as execution is resumed. I don't remember the exact command to use -- it's in their reference manual.

I suggest you verify this way the breakpoint is indeed set in the target
and then whether it is really at an address that guarantees it to trigger. A good way of determining whether hardware breakpoints work is to set one at the reset vector i.e. 0xffffffffbfc00000 (remove "f" digits as appropriate for your platform), resume execution and hit the NMI or soft reset button.

You won't get proper handling from GDB side (condition checking,
breakpoint commands, etc.) if you set breakpoints behind GDB's back anyway -- all you will get are unrecognised traps. And resumption under these circumstances is rather cumbersome as you have to perform all the actions GDB normally does itself manually.

Well, it's their product and they should be able to sort out your
problem. It's not a problem with Linux nor GDB even, so it's quite unlikely anybody on this list is able to provide you with any advice beyond what has been written already, so I suggest we stop bothering people with the noise.

Maciej

FYI,

I use the FS2 probe with sde-gdb on a nearly daily basis.
Are you compiling your kernel with -g -O1 ?

You can also try 'hbreak' instead of 'break' in sde-gdb.

Jon

Jon,

‘-g’, yes. ‘-O1’? Not sure, I will have to check when
I get a chance. (But I don't see how either would have the effect on breakpoints I'm suffering from?)

I'll have to (re-)try that when I get a chance. I did
try it once, and have (very! vague!!) memories of some issue, but now cannot recall anything useful. ;-\

cheers!

	-blf-

“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com