lan based kgdb
Is there a source level remote kernel debugger that communicates over an ethernet interface? The debugger kgdb from kgdb.sourceforge.net works only with serial port.
Quoted 3 lines · Kallol Biswas
Is there a source level remote kernel debugger that communicates over an ethernet interface? The debugger kgdb from kgdb.sourceforge.net works only with serial port.
With some hacking to produce a polling network driver, it should very well be possible to provide a communication back-end for a gdb stub that uses an ethernet NIC.
gdb itself is already perfectly capable of communicating with such targets, using the specifier
target remote otherhost:1234
where 1234 is the port number the gdb stub on the target reserves for itself to communicate with the host. It's probably best to use a dedicated network interface, though it's not absolutely required. (With a dedicated interface otherhost will then be the interface's hostname.)
A serious issue is that the communication protocol is somewhat more complex, so there is a possibility the debug runtime will not be robust.
Bill
Quoted 3 lines
Is there a source level remote kernel debugger that communicates over an ethernet interface? The debugger kgdb from kgdb.sourceforge.net works only with serial port.
A cheap terminal server might work ...
M.
In article <334960000.1037397999@flay>,
Quoted 5 lines · Martin J. Bligh
Quoted 3 lines
Is there a source level remote kernel debugger that communicates over an ethernet interface? The debugger kgdb from kgdb.sourceforge.net works only with serial port.
A cheap terminal server might work ...
Well, apart from the fact that a lot of machines don't even _have_ serial ports..
I dunno. I might even be willing to apply kgdb patches to my tree if it just could use the regular network card I already have connected on all my machines. None of my laptops have a serial line, for example, but they all have networking.
Soon even _desktops_ probably won't have serial lines any more, only USB.
Linus
Quoted 12 lines · Linus Torvalds
In article <334960000.1037397999@flay>,
Quoted 5 lines · Martin J. Bligh
Quoted 3 lines
Is there a source level remote kernel debugger that communicates over an ethernet interface? The debugger kgdb from kgdb.sourceforge.net works only with serial port.
A cheap terminal server might work ...
Well, apart from the fact that a lot of machines don't even _have_ serial ports..
Quoted 7 lines
I dunno. I might even be willing to apply kgdb patches to my tree if it just could use the regular network card I already have connected on all my machines. None of my laptops have a serial line, for example, but they all have networking.
Soon even _desktops_ probably won't have serial lines any more, only USB.
The only real work which has ever been done on this was by San Mehat earlier this year. When he had the advantage of sharing an office with me and being repeatedly harangued to do it ;)
He did have it working - it was basically the same idea as Ingo's netconsole code, using a little polling stub in each driver. He extended the concept to support Rx as well as Tx. You provide a whole bunch of parameters to the kernel and to the debugger, right down to the MAC address.
But San became quite unwell some months ago and vanished. As far as I know, the code is lost.
He may have sent a copy to Amit?
But as far as integrating the stub goes: IMO it would need quite some cleanup work first. Great tool, but the implementation is quite straggly.
An ethernet version could never be as robust as something which spins on a uart port though. Anyone who seriously wants to use the facility would just need to get themselves a 16550.
Many of our embedded motherboards do not have a serial port. May be I should consider adding ethernet interface support to kgdb.
Quoted 27 lines · Linus Torvalds
In article <334960000.1037397999@flay>,
Quoted 5 lines · Martin J. Bligh
Quoted 3 lines
Is there a source level remote kernel debugger that communicates over an ethernet interface? The debugger kgdb from kgdb.sourceforge.net works only with serial port.
A cheap terminal server might work ...
Well, apart from the fact that a lot of machines don't even _have_ serial ports..
I dunno. I might even be willing to apply kgdb patches to my tree if it just could use the regular network card I already have connected on all my machines. None of my laptops have a serial line, for example, but they all have networking.
Soon even _desktops_ probably won't have serial lines any more, only USB.
Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines · Linus Torvalds
I dunno. I might even be willing to apply kgdb patches to my tree if it just could use the regular network card I already have connected on all my machines. None of my laptops have a serial line, for example, but they all have networking.
Soon even _desktops_ probably won't have serial lines any more, only USB.
Using USB instead of the serial line or the network card would be the best IMHO, because:
* many machines have network cards, but all machines have USB (and it's gonna stay this way for some time)
* the USB stack seems simpler than the net stack + (eventualy) pcmcia + network card driver.
Maybe the 'simpler' USB protocols (usbkbd and usbmouse) could be used for this, I don't know...
Stelian, which has a Vaio Picturebook without a serial port.
Signature · 2 lines
Stelian Pop <[email protected]> Alcove - http://www.alcove.com
Quoted 11 lines · Stelian Pop
Using USB instead of the serial line or the network card would be the best IMHO, because:
* many machines have network cards, but all machines have USB (and it's gonna stay this way for some time)
* the USB stack seems simpler than the net stack + (eventualy) pcmcia + network card driver.
Maybe the 'simpler' USB protocols (usbkbd and usbmouse) could be used for this, I don't know...
USB hardware and protocols are master-slave, meaning that you can not connect another computer to this one directly. What USB *device* would you want to see connected?
Of course, a USB-Serial adapter would work, and you can connect any serial terminal, but then we are back to using serial ports; it's just you will need a different driver for that.
Dmitri
Quoted 2 lines · Dmitri
Quoted 2 lines
Using USB instead of the serial line or the network card would be the best IMHO, because:
[...]
Quoted 3 lines
USB hardware and protocols are master-slave, meaning that you can not connect another computer to this one directly.
Of course, you are correct :(
Quoted 2 lines
What USB *device* would
you want to see connected?
Well, I could say 'iPAQ' here but I guess it's getting out of topic.
Quoted 3 lines
Of course, a USB-Serial adapter would work, and you can connect any serial terminal, but then we are back to using serial ports; it's just you will need a different driver for that.
Well, I have one of those, and it could be a good replacement because it needs serial only on the development box, not on the target. It would be acceptable for me in any case (because I already have the adapter), but it probably won't be in the general case, given the price ratio between a network card and the USB-Serial adapter.
Stelian.
Signature · 2 lines
Stelian Pop <[email protected]> Alcove - http://www.alcove.com
* Dmitri <[email protected]> on Fri, Nov 15, 2002:
Quoted 11 lines · Stelian Pop
Quoted 10 lines
Using USB instead of the serial line or the network card would be the best IMHO, because:
* many machines have network cards, but all machines have USB (and it's gonna stay this way for some time)
* the USB stack seems simpler than the net stack + (eventualy) pcmcia + network card driver.
Quoted 7 lines
Quoted 2 lines
Maybe the 'simpler' USB protocols (usbkbd and usbmouse) could be used for this, I don't know...
USB hardware and protocols are master-slave, meaning that you can not connect another computer to this one directly. What USB *device* would you want to see connected?
Not true. Just grab any USB master<->master cable. There's P2P network drivers already in the kernel that support most brands of cables.
M. R.
Quoted 8 lines · Stelian Pop
Quoted 5 lines · Linus Torvalds
I dunno. I might even be willing to apply kgdb patches to my tree if it just could use the regular network card I already have connected on all my machines. None of my laptops have a serial line, for example, but they all have networking.
Quoted 5 lines
Quoted 1 line
Soon even _desktops_ probably won't have serial lines any more, only USB.
Using USB instead of the serial line or the network card would be the best IMHO, because:
Here is the kgdb stub's "send a byte" function:
static voidwrite_char(int chr)
{
while (!(inb(gdb_port + UART_LSR) & UART_LSR_THRE)) ;
outb(chr, gdb_port + UART_TX);
}Need I say more?
Quoted 7 lines · Andrew Morton
Quoted 2 lines
Using USB instead of the serial line or the network card would be the best IMHO, because:
Here is the kgdb stub's "send a byte" function:
static void
write_char(int chr)
Quoted 7 lines
{ while (!(inb(gdb_port + UART_LSR) & UART_LSR_THRE)) ; outb(chr, gdb_port + UART_TX); }Need I say more?
I already know that, but this is not the point. The point is that more and more boxes have no serial (or paralel) ports.
But even on those boxes, sometimes I'd just love to be able to use kgdb. And I can't.
Ok, it will have to be at a higher level than the inb/outb serial transport implementation (with possible bad effects on what can and what cannot be debugged), but still, I feel there is a need for that.
USB (with USB-to-serial adapter), network, ieee1394 would be acceptable replacements for me.
Stelian.
Signature · 2 lines
Stelian Pop <[email protected]> Alcove - http://www.alcove.com
Quoted 23 lines · Stelian Pop
Quoted 7 lines · Andrew Morton
Quoted 2 lines
Using USB instead of the serial line or the network card would be the best IMHO, because:
Here is the kgdb stub's "send a byte" function:
static void
write_char(int chr)Quoted 7 lines
{ while (!(inb(gdb_port + UART_LSR) & UART_LSR_THRE)) ; outb(chr, gdb_port + UART_TX); }Need I say more?
I already know that, but this is not the point. The point is that more and more boxes have no serial (or paralel) ports.
But even on those boxes, sometimes I'd just love to be able to use kgdb. And I can't.
Quoted 7 lines
Ok, it will have to be at a higher level than the inb/outb serial transport implementation (with possible bad effects on what can and what cannot be debugged), but still, I feel there is a need for that.
USB (with USB-to-serial adapter), network, ieee1394 would be acceptable replacements for me.
Have you ever looked at a USB or 1394 driver? The nice thing about serial is that the software to make it work is trivial. A debugger that relies on a 5000 line driver is quite suspect.
Quoted 12 lines
Stelian.
Signature · 7 lines
Stelian Pop <[email protected]> Alcove - http://www.alcove.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Signature · 5 lines
--------------------------------------------------------- Victor Yodaiken Finite State Machine Labs: The RTLinux Company. www.fsmlabs.com www.rtlinux.com 1+ 505 838 9109
Quoted 1 line · On Sat, Nov 16, 2002 at 09:23:41AM -0700, [email protected] wrote:
Quoted 1 line
Quoted 1 line
Here is the kgdb stub's "send a byte" function:
[...]
Quoted 5 lines
Quoted 3 lines
USB (with USB-to-serial adapter), network, ieee1394 would be acceptable replacements for me.
Have you ever looked at a USB or 1394 driver?
As a matter of fact, I did.
Quoted 3 lines
The nice thing about
serial is that the software to make it work is trivial. A debugger that relies on a 5000 line driver is quite suspect.
Agreed. But even a suspect debugger is preferable to no debugger at all.
Look, serial ports are becoming obsolete. We (not everybody but many people) need kgdb.
The general oppinion seems to be that a mini network stack + a simple network card driver is the easiest thing to implement (mostly because already done). That's fine for me.
Stelian.
Signature · 2 lines
Stelian Pop <[email protected]> Alcove - http://www.alcove.com
Quoted 5 lines · Stelian Pop
Agreed. But even a suspect debugger is preferable to no debugger at all.
Look, serial ports are becoming obsolete. We (not everybody but many people) need kgdb.
Machines may not ship with serial ports anymore, but theoretically they should have debug ports.
(A Debug Port being a 16550 with an arbitrary IO port, an ACPI table entry, and a connector on the motherboard. There's a spec on the Microsoft web site with the details, but there's a click-through license agreement and IANAL, so read at your own risk.)
Signature · 1 line
Nicholas Miell <[email protected]>
Quoted 19 lines · Andrew Morton
Quoted 8 lines · Stelian Pop
Quoted 5 lines · Linus Torvalds
I dunno. I might even be willing to apply kgdb patches to my tree if it just could use the regular network card I already have connected on all my machines. None of my laptops have a serial line, for example, but they all have networking.
Quoted 5 lines
Quoted 1 line
Soon even _desktops_ probably won't have serial lines any more, only USB.
Using USB instead of the serial line or the network card would be the best IMHO, because:
Here is the kgdb stub's "send a byte" function:
static void
write_char(int chr)
Quoted 7 lines
{ while (!(inb(gdb_port + UART_LSR) & UART_LSR_THRE)) ; outb(chr, gdb_port + UART_TX); }Need I say more?
netdump has polled eepro100 handlers that will plug nicely into this. Of course you still want a protocol on top of it, but there are some tiny tcp implementations that are GPL (eg the Linux 8086 TCP by Harry K)
Quoted 4 lines · Alan Cox
netdump has polled eepro100 handlers that will plug nicely into this. Of course you still want a protocol on top of it, but there are some tiny tcp implementations that are GPL (eg the Linux 8086 TCP by Harry K)
TCP is complete overkill. You don't want to run a debugging protocol over a WAN in any case. Let's assume LAN, 1ms or less round-trip, let's assume polling ACKs per packet so we don't have to keep a window, UDP with hand-specified src/dst/gateway/MACs/ports.
Signature · 1 line
"Love the dolphins," she advised him. "Write by W.A.S.T.E.."
In article <[email protected]>,
Quoted 3 lines · Stelian Pop
* the USB stack seems simpler than the net stack + (eventualy) pcmcia + network card driver.
What drugs are you on? The USB stack is extremely complex and fragile, and depends on a lot more working than just about any network driver out there. We're still debugging basic USB functionality.
Yes, if you're comparing to a full TCP implementation, plain USB serial lines may be simpler (ignoring for the moment the fact that there isn't even a standard USB serial line protocol, and they may be going the same way as the hardware serial lines - the way of the dodo).
But it should be possible to do a really simple UDP-packets-only thing for kgdb. Sure, it may lose packets. Tough. Don't debug over a WAN, and try to keep a clean direct network connection if you are worried about it. But we want kernel printk's to be synchronous anyway, without timeouts etc.
And I suspect you're better off losing packets (very rarely over any normal local network) if that means that your debugger needs only minimal support. You can always re-type.
Linus
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · On Sat, 16 Nov 2002 05:30:05 +0100, [email protected] (Linus Torvalds)…
Linus> Yes, if you're comparing to a full TCP implementation, plain Linus> USB serial lines may be simpler (ignoring for the moment the Linus> fact that there isn't even a standard USB serial line Linus> protocol, and they may be going the same way as the hardware Linus> serial lines - the way of the dodo).
Linus> But it should be possible to do a really simple Linus> UDP-packets-only thing for kgdb. Sure, it may lose packets. Linus> Tough. Don't debug over a WAN, and try to keep a clean Linus> direct network connection if you are worried about it. But Linus> we want kernel printk's to be synchronous anyway, without Linus> timeouts etc.
Linus> And I suspect you're better off losing packets (very rarely Linus> over any normal local network) if that means that your Linus> debugger needs only minimal support. You can always re-type.
I did this a couple of years ago for my research OS (Scout) and it worked great. It did UDP over Ethernet and was about 300 lines of code. Rather than using the normal UDP stack, the kernel gdb I/O ran directly on top of the network driver to reduce the risk of getting hit by a breakpoint that is in the way of the gdb I/O. The code is almost too trivial to mention, but for anyone interested, it can still be found at:
http://www.cs.arizona.edu/scout/software.html
The relevant files are scout/sys/ai/kgdb_net.c and scout/router/tulip/tulip.c (the latter contains the Ethernet driver portion).
--david
PS: IIRC, the kgdb protocol has a simple checksumming protocol so it
can deal with packet losses (perhaps not very gracefully, but on a LAN it's not going to be a problem anyhow).
Quoted 27 lines · David Mosberger-Tang
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · On Sat, 16 Nov 2002 05:30:05 +0100, [email protected] (Linus Torvalds)…
Linus> Yes, if you're comparing to a full TCP implementation, plain Linus> USB serial lines may be simpler (ignoring for the moment the Linus> fact that there isn't even a standard USB serial line Linus> protocol, and they may be going the same way as the hardware Linus> serial lines - the way of the dodo).
Linus> But it should be possible to do a really simple Linus> UDP-packets-only thing for kgdb. Sure, it may lose packets. Linus> Tough. Don't debug over a WAN, and try to keep a clean Linus> direct network connection if you are worried about it. But Linus> we want kernel printk's to be synchronous anyway, without Linus> timeouts etc.
Linus> And I suspect you're better off losing packets (very rarely Linus> over any normal local network) if that means that your Linus> debugger needs only minimal support. You can always re-type.
I did this a couple of years ago for my research OS (Scout) and it worked great. It did UDP over Ethernet and was about 300 lines of code. Rather than using the normal UDP stack, the kernel gdb I/O ran directly on top of the network driver to reduce the risk of getting hit by a breakpoint that is in the way of the gdb I/O. The code is almost too trivial to mention, but for anyone interested, it can still be found at:
Quoted 5 lines
http://www.cs.arizona.edu/scout/software.html
The relevant files are
scout/sys/ai/kgdb_net.candscout/router/tulip/tulip.c(the latter contains the Ethernet driver portion).
And I'm 95% sure that someone here (MontaVista) has done the same thing for Linux... ARM if my memory serves me correctly. You'll also notice that UDP debugging support is now in FSF GDB 5.3 snapshots. Sure, it's a little lossy, but it works in the useful cases.
I'm not quite sure what happened to the kernel side... but it sounds like this could be a nice candidate for the network dumping stack, too.
Quoted 3 lines
PS: IIRC, the kgdb protocol has a simple checksumming protocol so it
can deal with packet losses (perhaps not very gracefully, but on a LAN it's not going to be a problem anyhow).
Yes. The protocol also has explicit ACKs and has the basis for retransmission; it's not extensive but it's enough to get by over even a lossy serial cable.
Signature · 2 lines
Daniel Jacobowitz MontaVista Software Debian GNU/Linux Developer
Quoted 9 lines · David Mosberger-Tang
hit by a breakpoint that is in the way of the gdb I/O. The code is almost too trivial to mention, but for anyone interested, it can still be found at:
http://www.cs.arizona.edu/scout/software.html
The relevant files are
scout/sys/ai/kgdb_net.candscout/router/tulip/tulip.c(the latter contains the Ethernet driver portion).
Very nice. That solves the debugger question elegantly. I'd still rather have tcp for a console control though 8)
Quoted 5 lines · Linus Torvalds
But it should be possible to do a really simple UDP-packets-only thing for kgdb. Sure, it may lose packets. Tough. Don't debug over a WAN, and try to keep a clean direct network connection if you are worried about it. But we want kernel printk's to be synchronous anyway, without timeouts etc.
LAN latencies should be low enough that waiting on an ACK for each packet will do just fine for error correction. If someone wants to do remote debugging, they can ssh into a debugging machine on the same LAN.
Signature · 1 line
"Love the dolphins," she advised him. "Write by W.A.S.T.E.."
Quoted 4 lines · Oliver Xymoron
LAN latencies should be low enough that waiting on an ACK for each packet will do just fine for error correction. If someone wants to do remote debugging, they can ssh into a debugging machine on the same LAN.
I agree in theory on a technical level, yet at the same time it's clearly advantageous _not_ to wait, since it would allow you to just universally enable the LAN as the console on all your machines when you maintain them, and then not have that LAN console be a maintenance problem.
Basically, I don't personally care too much for kgdb itself, but I see a asynchronous LAN console as a more generic tool for just doing not just kernel debugging, but management in general. syslogd is fine for when the machines work, but it tends to lose important information just when you need it the most, ie when a machine starts having serious problems.
So if you see the LAN interface more as a simple console (that _also_ gives you access to kgdb), then it's a whole lot more usable for much more than just kernel hacking.
A synchronous interface would make logging basically useless - it would basically crash the machine if there are logger problems, and it would make log output slow things down a lot.
Linus
Quoted 11 lines · Linus Torvalds
Quoted 4 lines · Oliver Xymoron
LAN latencies should be low enough that waiting on an ACK for each packet will do just fine for error correction. If someone wants to do remote debugging, they can ssh into a debugging machine on the same LAN.
I agree in theory on a technical level, yet at the same time it's clearly advantageous _not_ to wait, since it would allow you to just universally enable the LAN as the console on all your machines when you maintain them, and then not have that LAN console be a maintenance problem.
Definitely agreed on the usefulness of LAN console. Being able to just run netcat for all the boxes in your datacenter is a huge win. Cuts your cable count by a third to a half and eliminates a bunch of term servers and KVMs.
Other folks have pointed out that the GDB stub protocol includes checksumming and retries up at the 'application' layer so we don't actually have to worry about it.
Signature · 1 line
"Love the dolphins," she advised him. "Write by W.A.S.T.E.."
On Sat, 2002-11-16 10:33:57 -0800, Linus Torvalds <[email protected]> wrote in message <[email protected]>:
Quoted 9 lines · Oliver Xymoron
Quoted 4 lines
LAN latencies should be low enough that waiting on an ACK for each packet will do just fine for error correction. If someone wants to do remote debugging, they can ssh into a debugging machine on the same LAN.
Basically, I don't personally care too much for kgdb itself, but I see a asynchronous LAN console as a more generic tool for just doing not just kernel debugging, but management in general. syslogd is fine for when the
...which reminds me to DEC's MOP (Maintainence and Operator's Protocol), which is ethernet (but not IP) based remote console and a mixture of bootp/tftp. Sure, we won't (yet) go as far as sending the next kernel to boot via our new console protocol to kexec(), but wait for the very first S-Records to arrive:-p
MfG, JBG
Signature · 4 lines
Jan-Benedict Glaw [email protected] . +49-172-7608481 "Eine Freie Meinung in einem Freien Kopf | Gegen Zensur fuer einen Freien Staat voll Freier Bürger" | im Internet! Shell Script APT-Proxy: http://lug-owl.de/~jbglaw/software/ap2/
Quoted 5 lines · Jan-Benedict Glaw
...which reminds me to DEC's MOP (Maintainence and Operator's Protocol), which is ethernet (but not IP) based remote console and a mixture of bootp/tftp. Sure, we won't (yet) go as far as sending the next kernel to boot via our new console protocol to
kexec(), but wait for the very first S-Records to arrive:-p
We have working DECnet and MOP server protocol support. That doesn't mean its a good idea
On Sun, 2002-11-17 14:50:32 +0000, Alan Cox <[email protected]> wrote in message <[email protected]>:
Quoted 9 lines · Jan-Benedict Glaw
Quoted 5 lines
...which reminds me to DEC's MOP (Maintainence and Operator's Protocol), which is ethernet (but not IP) based remote console and a mixture of bootp/tftp. Sure, we won't (yet) go as far as sending the next kernel to boot via our new console protocol to
kexec(), but wait for the very first S-Records to arrive:-pWe have working DECnet and MOP server protocol support. That doesn't mean its a good idea
Really? Do we? Well, sure for the server part (-> booting. Half a dozend implementations from which only one ot possibly two are really useable), but the client side (remote console) was never implemented IIRC. And for sure there's no MOP console server on linux:-)
MfG, JBG
Signature · 4 lines
Jan-Benedict Glaw [email protected] . +49-172-7608481 "Eine Freie Meinung in einem Freien Kopf | Gegen Zensur fuer einen Freien Staat voll Freier Bürger" | im Internet! Shell Script APT-Proxy: http://lug-owl.de/~jbglaw/software/ap2/
Looks like there is a lot of interset in lan based kgdb. I've got the kgdb patch with lan written by San, thanks to Samuel Flory.
I'll integrate the patch into mainline kgdb (http://kgdb.sourceforge.net/) and release it asap.
Signature · 2 lines
Amit Kale Veritas Software India ( http://www.veritasindia.com/ )
Quoted 3 lines · Linus Torvalds
And I suspect you're better off losing packets (very rarely over any normal local network) if that means that your debugger needs only minimal support. You can always re-type.
Of course, if your virtual LAN has several machines that all think that congestion control only makes sense in a WAN setting, loss may get quite common :-) I still have fond memories of shared Ethernet melting down under the load of dozens of diskless clients rebooting after some problem hit the cluster ...
Loss recovery doesn't have to be painful, and it can still be real time. When sending, include the packet sequence number, and the sequence number of the earliest packet buffered. Buffer up to N packets after sending. Packets older than current_sequence-N drop off the buffer and cannot be recovered. Receiver can request retransmission of specific buffered packets. Retransmitted packets carry an up to date earliest packet buffered sequence number.
This way, it's up to the receiver to decide what to do with lost packets, e.g. request retransmission, and wait up to a certain deadline, just flag the loss and proceed, of maybe flag the loss, request retransmission, and when receiving the missing packet, redraw the output.
Congestion control would be very good to have, though. As soon as you put something on IP, it's almost guaranteed to eventually end up crossing a WAN.
Crazy idea: maybe one could combine the ideas from MCORE and RT
Linux: use a dispatcher for basic system resources (interrupts, etc.), and run two kernels. One of them would just run the debugger and serve its communication needs. That would leave things still relatively close to hardware (unlike just using UML).
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Quoted 5 lines · Linus Torvalds
But it should be possible to do a really simple UDP-packets-only thing for kgdb. Sure, it may lose packets. Tough. Don't debug over a WAN, and try to keep a clean direct network connection if you are worried about it. But we want kernel printk's to be synchronous anyway, without timeouts etc.
And in the real world you end up back with TCP. Been there, done that with network debugger tools before.
Alan
Quoted 3 lines · Alan Cox
And in the real world you end up back with TCP. Been there, done that with network debugger tools before.
.. and we have another "been there, done that" that says UDP works fine.
I also dislike overdesign with a passion. I'll believe we have to go to TCP when I see it - and even if that happens, I think we should do the UDP case first just to avoid having a monster ("start off simple, and evolve" as opposed to "try to get it right the first time").
Linus
Quoted 10 lines · Linus Torvalds
Quoted 2 lines · Alan Cox
And in the real world you end up back with TCP. Been there, done that with network debugger tools before.
.. and we have another "been there, done that" that says UDP works fine.
I also dislike overdesign with a passion. I'll believe we have to go to TCP when I see it - and even if that happens, I think we should do the UDP case first just to avoid having a monster ("start off simple, and evolve" as opposed to "try to get it right the first time").
I've done this sort of "reliable UDP" on local networks before, it works just fine, no TCP is needed. TCP is a really well designed protocol, it handles an amazing range of problems gracefully. It's overkill for a lan remote gdb.
Signature · 2 lines
--- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
As long as the network console/debug interface includes basic a basic check to verify that the packets it accepts are from the local network. And it's outgoing packets have a ttl of one. I don't have a problem.
Otherwise the concept gives me security nightmares.
Eric
Quoted 3 lines · Eric W. Biederman
As long as the network console/debug interface includes basic a basic check to verify that the packets it accepts are from the local network. And it's outgoing packets have a ttl of one. I don't have a problem.
Is there a working network console? It would be _great_ to have a network console to my _remote_ server, far far away on the internet.
-- Jamie
Quoted 7 lines · Jamie Lokier
Quoted 3 lines · Eric W. Biederman
As long as the network console/debug interface includes basic a basic check to verify that the packets it accepts are from the local network. And it's outgoing packets have a ttl of one. I don't have a problem.
Is there a working network console? It would be _great_ to have a network console to my _remote_ server, far far away on the internet.
There are bits and pieces, and a lan based kgdb is basically the same problem security wise. When you allow any kind of control the security cannot be in the console protocol. Therefore it can only be used on a trusted lan and be only talk to local addresses. At the same time if you have two remote machines on that trusted lan you can use one to control the other. Knowing that a root exploit on one likely becomes a root exploit on both.
And weather or not we have one at the moment, it is an active area of research. We just need to get a useable security model. And I think enforcing that the console be on a secure lan where every connected machine is trusted is a good first draft, at the latter.
So I do not think this is the kind of thing that will help if you only have one _remote_ server, far far away on the internet, but when you start getting a collection of them it may help.
Eric
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 2 lines · Eric W. Biederman
As long as the network console/debug interface includes basic a basic check to verify that the packets it accepts are from the local network.
This is pretty hard to do in some configurations. You essentially have to do this at the router, not at destination.
Quoted 1 line
And it's outgoing packets have a ttl of one. I don't have a problem.
Recent IETF work on link-local has used TTL=255 outgoing, and it has to be 255 at the receive end too. That is a reasonable way to ensure that is is link-local, since even the most brain-dead routers will at least decrement TTL.
Quoted 1 line
Otherwise the concept gives me security nightmares.
Computing should give you security nightmares :)
Brad
- --
http://linux.conf.au. 22-25Jan2003. Perth, Aust. I'm registered. Are you? -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE91/s3W6pHgIdAuOMRAsCNAJ42Fc7CCclgD+zMbraiHYFydMcKJACfegyr JcZ8T8+1nIe2f8G3eerNwVs=
=uGdb
-----END PGP SIGNATURE-----
Quoted 5 lines · Brad Hards
Quoted 2 lines · Eric W. Biederman
As long as the network console/debug interface includes basic a basic check to verify that the packets it accepts are from the local network.
This is pretty hard to do in some configurations. You essentially have to do this at the router, not at destination.
I agree that you cannot do a perfect job. The goal is to get something that is good enough so that it can be enabled and not give an automatic root exploit if someone accidentally leaves it on at the wrong time.
Quoted 5 lines
Quoted 1 line
And it's outgoing packets have a ttl of one. I don't have a problem.
Recent IETF work on link-local has used TTL=255 outgoing, and it has to be 255 at the receive end too. That is a reasonable way to ensure that is is link-local, since even the most brain-dead routers will at least decrement TTL.
Nice. And then on the transmit end I would still use a TTL=1 so that routers will refuse to route the packets. A bit asymmetric but I only care about security in one direction.
But in what kind of configurations is checking the ip against the current netmask insufficient? Checking for a TTL of 255 will trivially make the check stronger.
Having a network console for various debugging tasks could be very useful. The question is how to implement it simply and reliably.
Eric
a couple quick questions from an end-user
1. will an interface be dedicated to this use, or will it share an interface with other traffic.
2. if it is dedicated how do you inform the rest of the system that this card is off limits? (and can you use one port on a quad card, or do you require a dedicated card?)
3. if we really want to make this limited to the local wire why not use something other then UDP? either another IP protocol number (more likly to be blocked by routers) or somethign not IP compatable (so that routers couldn't forward it if they wanted to). especially if you are talking about useing a special (aka stripped down, simplified) stack for this interface instead of the full-blown version
normally I would agree that standards are good, becouse they let you interoperate with other equipment, but in this case I'm not sure that's really what we want. All communications is not IP :-)
as someone managing 60 or so remote boxes, this sounds really nice, if it can be made to work securely.
David Lang
Quoted 42 lines · On 17 Nov 2002 [email protected] wrote:
Date: 17 Nov 2002 14:30:00 -0700
From: [email protected]
To: Brad Hards <[email protected]>
Cc: Linux Kernel Mailing List <[email protected]>
Subject: Re: lan based kgdbQuoted 5 lines · Brad Hards
Quoted 2 lines · Eric W. Biederman
As long as the network console/debug interface includes basic a basic check to verify that the packets it accepts are from the local network.
This is pretty hard to do in some configurations. You essentially have to do this at the router, not at destination.
I agree that you cannot do a perfect job. The goal is to get something that is good enough so that it can be enabled and not give an automatic root exploit if someone accidentally leaves it on at the wrong time.
Quoted 5 lines
Quoted 1 line
And it's outgoing packets have a ttl of one. I don't have a problem.
Recent IETF work on link-local has used TTL=255 outgoing, and it has to be 255 at the receive end too. That is a reasonable way to ensure that is is link-local, since even the most brain-dead routers will at least decrement TTL.
Nice. And then on the transmit end I would still use a TTL=1 so that routers will refuse to route the packets. A bit asymmetric but I only care about security in one direction.
But in what kind of configurations is checking the ip against the current netmask insufficient? Checking for a TTL of 255 will trivially make the check stronger.
Having a network console for various debugging tasks could be very useful. The question is how to implement it simply and reliably.
Eric
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 4 lines · David Lang
a couple quick questions from an end-user
1. will an interface be dedicated to this use, or will it share an interface with other traffic.
I imagined that it would have to be shared. The world is not a PC, and you can't trivially add extra connectivtity to that embedded ARM board...
<snip>
Quoted 6 lines
3. if we really want to make this limited to the local wire why not use something other then UDP? either another IP protocol number (more likly to be blocked by routers) or somethign not IP compatable (so that routers couldn't forward it if they wanted to). especially if you are talking about useing a special (aka stripped down, simplified) stack for this interface instead of the full-blown version
If you really want link-local, use the link-local IP addresses (169.254/16). See http://files.zeroconf.org/draft-ietf-zeroconf-ipv4-linklocal.txt
(I actually have an implementation of this for linux, alas it is only in userspace :)
Quoted 3 lines
normally I would agree that standards are good, becouse they let you interoperate with other equipment, but in this case I'm not sure that's really what we want. All communications is not IP :-)
All _good_ communication is IP :)
Quoted 2 lines
as someone managing 60 or so remote boxes, this sounds really nice, if it can be made to work securely.
OK, I'm confused again. Do you want remote, or to you want link-local?
Brad
- --
http://linux.conf.au. 22-25Jan2003. Perth, Aust. I'm registered. Are you? -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE92A64W6pHgIdAuOMRArrEAJ9Yp6w85APLudklNBVxfY6nAF066ACfeLc7 z00tKrs79Ri3k7UbFNCQzJo=
=VXrF
-----END PGP SIGNATURE-----
Quoted 7 lines · Brad Hards
Quoted 4 lines · David Lang
a couple quick questions from an end-user
1. will an interface be dedicated to this use, or will it share an interface with other traffic.
I imagined that it would have to be shared. The world is not a PC, and you can't trivially add extra connectivtity to that embedded ARM board...
I can see that, it will add more complexity thought
Quoted 4 lines
<snip>
Quoted 2 lines
as someone managing 60 or so remote boxes, this sounds really nice, if it can be made to work securely.
OK, I'm confused again. Do you want remote, or to you want link-local?
I want link-local. as has been discused once you have two boxes in one location link-local is good enough and I always deploy boxes in HA pairs
David Lang
Quoted 11 lines · David Lang
Quoted 7 lines · Brad Hards
Quoted 4 lines · David Lang
a couple quick questions from an end-user
1. will an interface be dedicated to this use, or will it share an interface with other traffic.
I imagined that it would have to be shared. The world is not a PC, and you can't trivially add extra connectivtity to that embedded ARM board...
I can see that, it will add more complexity thought
There are only three cases I can see supporting.
- Transmit a kernel message
- Process a MAGIC-SYSRQ packet
- Remote GDB.
For transmitting a kernel message, sharing the nic is really not more complicated. It simply preempts the normal driver transmits it's packet and then goes away.
For processing a MAGIC-SYSRQ that is almost certainly going to be interrupt driven just a special debug path that can safely execute inside of the interrupt handler. For conditions that make the box unpingable this likely will not help, but very will at that point.
For a remote GDB transmitting a packet is the same as for kernel messages. If a packet is expected it just waits on the nic throwing out the rest of the packets until the one it is expecting comes in. And if it isn't particularly expecting a packet it can work just like the MAGIC-SYSRQ case.
So while there is some small additional complexity to sharing the link I do not think it is significant. But just because you can share the link does not mean you have to.
Quoted 8 lines
Quoted 4 lines
<snip>
Quoted 2 lines
as someone managing 60 or so remote boxes, this sounds really nice, if it can be made to work securely.
OK, I'm confused again. Do you want remote, or to you want link-local?
I want link-local. as has been discused once you have two boxes in one location link-local is good enough and I always deploy boxes in HA pairs
I am still trying to hammer down the protocol enough to suit me. IPv4 Link local IP addresses are not generally appropriate, because claim-and-defend is potentially a pain. IPv6 link-local addresses look better as everything can be configured statically. Ideally everything can be kept simple enough that retransmissions do not need to at all. Nor extraneous traffic processing like arp.
Configuration needed:
Interface: (This should default to none) Local-ip address: (For ipv6 we can default to the link-local address
derived from the MAC address)
Remote-ip address: (By default this should probably be a multi-cast IP
address, derived from the local-ip address)
The important things:
- The ability to have everything hard coded, so you can find the network console/network gdb session.
- A trivial implementation that can fire and forget packets. No retransmit should be needed on the machine being debugged. This is over a mostly reliable interface so that should not be a problem.
- A setup that does not loose a connection when you reboot a machine. This rules out TCP.
- The ability to have good defaults so you can setup a cluster without manual configuration.
The way I envision it, on the debugged machine:
- Transmit:
TTL=1 to some address (ideally this would be a multicast ip)
- Receive:
verify TTL=255 ip on the same subnet. Packet is not fragmented. Dest IP is my local ip (Ipv6 link-local ip?)
It will be a little bit before I scratch this itch but I have a good feeling about how to do it now.
Eric
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 11 lines · Eric W. Biederman
Quoted 6 lines · Brad Hards
Quoted 2 lines · Eric W. Biederman
As long as the network console/debug interface includes basic a basic check to verify that the packets it accepts are from the local network.
This is pretty hard to do in some configurations. You essentially have to do this at the router, not at destination.
I agree that you cannot do a perfect job. The goal is to get something that is good enough so that it can be enabled and not give an automatic root exploit if someone accidentally leaves it on at the wrong time.
if you put it on the network, the best you can do is restrict the problem to being link-local. If you exclude wireless, then the problem is basically then a matter of physical security, which isn't much different to having a console running on the box itself.
Quoted 10 lines
Quoted 6 lines
Quoted 1 line
And it's outgoing packets have a ttl of one. I don't have a problem.
Recent IETF work on link-local has used TTL=255 outgoing, and it has to be 255 at the receive end too. That is a reasonable way to ensure that is is link-local, since even the most brain-dead routers will at least decrement TTL.
Nice. And then on the transmit end I would still use a TTL=1 so that routers will refuse to route the packets. A bit asymmetric but I only care about security in one direction.
I was completely confused when I first read this.
I take it that you mean:
On machine under test (the target in the debug context) you make TTL=1 on outgoing packets, and drop incoming packets that don't have TTL=255 On machine being used to view console (the host in the debug context) you make TTL=255 on outgoing packets.
That doesn't ensure that the machine you are monitoring (or debugging) is really the machine you think it is, but it might solve enough of the problem.
Quoted 3 lines
But in what kind of configurations is checking the ip against the current netmask insufficient? Checking for a TTL of 255 will trivially make the check stronger.
It prevents source field spoofing. I don't know how much defence you get from this on fielded routers, but faking the source field in a UDP packet doesn't seem too hard.
TTL=255 ensures that the machine you are talking to is _really_ on the same link. Any other number is subject to an attacker knowing how many hops it is, and setting TTL appropriately.
Brad
- --
http://linux.conf.au. 22-25Jan2003. Perth, Aust. I'm registered. Are you? -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE92A07W6pHgIdAuOMRAhmBAJ9O//ObT/e2pq6lrbiKUDhkNNJslwCdHYEN mXWWIAkZW7SDUSRhY7AvN0s=
=EA7y
-----END PGP SIGNATURE-----
Quoted 1 line · Eric W. Biederman
Otherwise the concept gives me security nightmares.
Bah, there are a few Fundamental Truths of Networking Simplification that we can absolutely rely on:
- it is perfectly safe to require that a given port be connected only to a secure and trusted network
- if you design a protocol not to work over a WAN, everybody will respect this, and deploy it only on LANs
- if any of the above constraints is no longer tenable, people will carefully redesign the protocol in question, and replace the installed base
- all LANs have simple, well-understood characteristics - now and forever
After all, SNMP, FSP, LANE, NAT, WEP, etc. can't be wrong (-:C
And yes, I can vividly imagine users of dedicated hosts rush to switch on that remote console the very moment it becomes available, and use it across half of the planet.
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, Buenos Aires, Argentina [email protected] / /_http://www.almesberger.net/____________________________________________/
Quoted 5 lines · Stelian Pop
Using USB instead of the serial line or the network card would be the best IMHO, because:
* many machines have network cards, but all machines have USB (and it's gonna stay this way for some time)
Um, no; _many_ machines have USB.
[The boards I work with all have a UART _on the CPU chip_, so it's pretty cheap to give them a serial connection, and many of them have a network connection because it's needed -- but none of them have USB...
Even if you're talking about just PCs, most of the PCs I use are slightly old, and don't have USB; maybe I'm just from mars or something, but it's always kind of annoying to see people talking as if USB were universal...]
-Miles
Signature · 1 line
80% of success is just showing up. --Woody Allen