In-kernel gdbstub based on utrace Infrastructure.
RFC PATCH 1/1 In-kernel gdbstub based on utrace Infrastructure.
Hi,
This patch implements an in-kernel gdb stub.
It provides an interface between gdb and Linux Kernel by implementing the remote serial protocol. This gdbstub uses utrace infrastructure. This patch provides register set access, signal mapping, process event handling, input/output operations.
/proc/<pid>/gdb was chosen as file for gdb to interact with the process through remote serial protocol.
Hence users would have to use "target remote /proc/<pid>/gdb" command on gdb prompt to start using this infrastructure.
For Breakpointing support, gdbstub needs User space breakpointing layer and uprobes layer which will be posted later.
Here is an illustration of what all this can do.
$ pgrep zsh 8865 $ gdb /bin/zsh GNU gdb (GDB) 7.0.0.20091030 ..... <snipped gdb header > ............... For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /bin/zsh...Reading symbols from /usr/lib/debug/bin/zsh.debug...done. (no debugging symbols found)...done. (gdb) target remote /proc/8865/gdb Remote debugging using /proc/8865/gdb .... <snipped several "Reading symbols" and "Loaded symbols" > ...... Loaded symbols for /usr/lib64/zsh/4.2.6/zsh/zle.so 0x0000003f1dec5f00 in __read_nocancel () from /lib64/libc.so.6 (gdb) c Continuing. ^C Program received signal SIGINT, Interrupt. 0x0000003f1dec5f00 in __read_nocancel () from /lib64/libc.so.6 (gdb) info registers rax 0xfffffffffffffe00 -512 rbx 0x6a01c8 6947272 rcx 0xffffffffffffffff -1 rdx 0x1 1 rsi 0x7fff507ad5ef 140734543615471 rdi 0xa 10 rbp 0x0 0x0 rsp 0x7fff507ad3e8 0x7fff507ad3e8 r8 0xffffffff 4294967295 r9 0xffffffff 4294967295 r10 0x7d 125 r11 0x246 582 r12 0x7f43b87a9398 139928834577304 r13 0x0 0 r14 0x0 0 r15 0xd 13 rip 0x3f1dec5f00 0x3f1dec5f00 <__read_nocancel+7> eflags 0x246 [ PF ZF IF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 fctrl 0x0 0 fstat 0x0 0 ftag 0x0 0 fiseg 0x0 0 fioff 0x0 0 foseg 0x0 0 fooff 0x0 0 fop 0x0 0 mxcsr 0x0 [ ] (gdb) bt #0 0x0000003f1dec5f00 in __read_nocancel () from /lib64/libc.so.6 #1 0x00007f43b858a39f in raw_getkey (keytmout=0) at /usr/include/bits/unistd.h:35 #2 getkey (keytmout=0) at zle_main.c:610 #3 0x00007f43b8584bf1 in getkeybuf (km=0x0, funcp=0x7fff507ad718, strp=0x7fff507ad710) at zle_keymap.c:1313 #4 getkeymapcmd (km=0x0, funcp=0x7fff507ad718, strp=0x7fff507ad710) at zle_keymap.c:1275 #5 0x00007f43b8584ef0 in getkeycmd () at zle_keymap.c:1349 #6 0x00007f43b8588281 in zlecore () at zle_main.c:695 #7 0x00007f43b85888f8 in zleread (lp=<value optimized out>, rp=<value optimized out>, flags=<value optimized out>, context=<value optimized out>) at zle_main.c:863 #8 0x00000000004388a0 in inputline () at input.c:278 #9 ingetc () at input.c:214 #10 0x000000000043339d in ihgetc () at hist.c:241 #11 0x000000000044126b in gettok () at lex.c:631 #12 0x0000000000441a88 in yylex () at lex.c:347 #13 0x000000000045cec7 in parse_event () at parse.c:449 #14 0x0000000000437328 in loop (toplevel=1, justonce=0) at init.c:128 #15 0x0000000000438021 in zsh_main (argc=<value optimized out>, argv=0x7fff507ae048) at init.c:1280 #16 0x0000003f1de1d994 in __libc_start_main (main=<value optimized out>, argc=<value optimized out>, ubp_av=<value optimized out>, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=Could not find the frame base for "__libc_start_main". ) at libc-start.c:231 #17 0x000000000040cc09 in _start () (gdb) c Continuing. ^C Program received signal SIGINT, Interrupt. 0x0000003f1dec5f00 in __read_nocancel () from /lib64/libc.so.6 (gdb) bt #0 0x0000003f1dec5f00 in __read_nocancel () from /lib64/libc.so.6 #1 0x00007f43b858a39f in raw_getkey (keytmout=0) at /usr/include/bits/unistd.h:35 #2 getkey (keytmout=0) at zle_main.c:610 #3 0x00007f43b8584bf1 in getkeybuf (km=0x0, funcp=0x7fff507ad718, strp=0x7fff507ad710) at zle_keymap.c:1313 #4 getkeymapcmd (km=0x0, funcp=0x7fff507ad718, strp=0x7fff507ad710) at zle_keymap.c:1275 #5 0x00007f43b8584ef0 in getkeycmd () at zle_keymap.c:1349 #6 0x00007f43b8588281 in zlecore () at zle_main.c:695 #7 0x00007f43b85888f8 in zleread (lp=<value optimized out>, rp=<value optimized out>, flags=<value optimized out>, context=<value optimized out>) at zle_main.c:863 #8 0x00000000004388a0 in inputline () at input.c:278 #9 ingetc () at input.c:214 #10 0x000000000043339d in ihgetc () at hist.c:241 #11 0x000000000044126b in gettok () at lex.c:631 #12 0x0000000000441a88 in yylex () at lex.c:347 #13 0x000000000045cec7 in parse_event () at parse.c:449 #14 0x0000000000437328 in loop (toplevel=1, justonce=0) at init.c:128 #15 0x0000000000438021 in zsh_main (argc=<value optimized out>, argv=0x7fff507ae048) at init.c:1280 #16 0x0000003f1de1d994 in __libc_start_main (main=<value optimized out>, argc=<value optimized out>, ubp_av=<value optimized out>, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=Could not find the frame base for "__libc_start_main". ) at libc-start.c:231 #17 0x000000000040cc09 in _start () (gdb) b zrefresh Breakpoint 1 at 0x7f43b858f600: file zle_refresh.c, line 278. ### Note: This is (gdb) c Continuing. ^C Program received signal SIGINT, Interrupt. 0x0000003f1dec5f00 in __read_nocancel () from /lib64/libc.so.6 (gdb) info b Num Type Disp Enb Address What 1 breakpoint keep y <PENDING> in zrefresh at zle_refresh.c:278 #### HERE breakpoint is still pending because, gdbstub needs more support for #### breakpointing. (gdb) (gdb) q A debugging session is active.
Inferior 1 [Remote target] will be killed.
Quit anyway? (y or n) y
$ pgrep zsh 8865 $
I request you to please review this and lets us know your comments.
--- fs/proc/base.c | 4 + include/linux/utrace.h | 4 + init/Kconfig | 26 +- kernel/Makefile | 1 + kernel/utrace-gdb.c | 1314 ++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 1340 insertions(+), 9 deletions(-)
fs/proc/base.c +4 -0
@@ -77,6 +77,7 @@#include <linux/audit.h> #include <linux/poll.h> #include <linux/nsproxy.h> +#include <linux/utrace.h> #include <linux/oom.h> #include <linux/elf.h> #include <linux/pid_namespace.h>
@@ -2562,6 +2563,9 @@ static const struct pid_entry tgid_base_stuff[] = {#ifdef CONFIG_TASK_IO_ACCOUNTING INF("io", S_IRUGO, proc_tgid_io_accounting),#endif +#ifdef CONFIG_UTRACE_GDB + REG("gdb", S_IRUSR|S_IWUSR, proc_gdb_operations), +#endif }; static int proc_tgid_base_readdir(struct file * filp,
include/linux/utrace.h +4 -0
@@ -726,4 +726,8 @@ static inline __must_check int utrace_barrier_pid(struct pid *pid,#endif /* CONFIG_UTRACE */ +#ifdef CONFIG_UTRACE_GDB +extern const struct file_operations proc_gdb_operations; +#endif +#endif /* linux/utrace.h */
init/Kconfig +17 -9
@@ -295,15 +295,6 @@ config AUDIT logging of avc messages output). Does not do system-call auditing without CONFIG_AUDITSYSCALL. -config UTRACE - bool "Infrastructure for tracing and debugging user processes" - depends on EXPERIMENTAL - depends on HAVE_ARCH_TRACEHOOK - help - Enable the utrace process tracing interface. This is an internal - kernel interface exported to kernel modules, to track events in - user threads, extract and change user thread state. - config AUDITSYSCALL bool "Enable system-call auditing support" depends on AUDIT && (X86 || PPC || S390 || IA64 || UML || SPARC64 || SUPERH)
@@ -1214,6 +1205,23 @@ config STOP_MACHINE help Need stop_machine() primitive. +menuconfig UTRACE + bool "Infrastructure for tracing and debugging user processes" + depends on EXPERIMENTAL + depends on HAVE_ARCH_TRACEHOOK + help + Enable the utrace process tracing interface. This is an internal + kernel interface exported to kernel modules, to track events in + user threads, extract and change user thread state. + +config UTRACE_GDB + bool "/proc/<pid>/gdb file for gdb remote connection" + select UTRACE + default y + help + Enable the utrace-based /proc/<pid>/gdb process debugging + interface, for connection using the gdb remote protocol. + source "block/Kconfig" config PREEMPT_NOTIFIERS
kernel/Makefile +1 -0
@@ -69,6 +69,7 @@ obj-$(CONFIG_RESOURCE_COUNTERS) += res_counter.o obj-$(CONFIG_STOP_MACHINE) += stop_machine.o obj-$(CONFIG_KPROBES_SANITY_TEST) += test_kprobes.o obj-$(CONFIG_UTRACE) += utrace.o +obj-$(CONFIG_UTRACE_GDB) += utrace-gdb.o obj-$(CONFIG_AUDIT) += audit.o auditfilter.o audit_watch.o obj-$(CONFIG_AUDITSYSCALL) += auditsc.o obj-$(CONFIG_GCOV_KERNEL) += gcov/
kernel/utrace-gdb.c new file +1314 -0
1,314 lines not on this page · raw message
Quoted 6 lines · Srikar Dronamraju
This patch implements an in-kernel gdb stub.
It provides an interface between gdb and Linux Kernel by implementing the remote serial protocol. This gdbstub uses utrace infrastructure. This patch provides register set access, signal mapping, process event handling, input/output operations.
Quoted 8 lines
/proc/<pid>/gdb was chosen as file for gdb to interact with the process through remote serial protocol.
Hence users would have to use "target remote /proc/<pid>/gdb" command on gdb prompt to start using this infrastructure.
For Breakpointing support, gdbstub needs User space breakpointing layer and uprobes layer which will be posted later.
How does this compare to kgdb and related efforts?
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Srikar Dronamraju
This patch implements an in-kernel gdb stub.
It provides an interface between gdb and Linux Kernel by implementing the remote serial protocol. This gdbstub uses utrace infrastructure. This patch provides register set access, signal mapping, process event handling, input/output operations./proc/<pid>/gdb was chosen as file for gdb to interact with the process through remote serial protocol.
Quoted 7 lines
Quoted 5 lines
Hence users would have to use "target remote /proc/<pid>/gdb" command on gdb prompt to start using this infrastructure.
For Breakpointing support, gdbstub needs User space breakpointing layer and uprobes layer which will be posted later.
How does this compare to kgdb and related efforts?
This is a In-kernel gdbstub to debug user space programs. This stub doesnt help in debugging kernel.
Hence I am not sure how to compare kgdb gdbstub with this gdbstub. Can you please provide more pointers on what you were referring to?
--
Thanks and Regards
Srikar
Quoted 18 lines · Srikar Dronamraju
Quoted 16 lines · Peter Zijlstra
Quoted 14 lines · Srikar Dronamraju
This patch implements an in-kernel gdb stub.
It provides an interface between gdb and Linux Kernel by implementing the remote serial protocol. This gdbstub uses utrace infrastructure. This patch provides register set access, signal mapping, process event handling, input/output operations./proc/<pid>/gdb was chosen as file for gdb to interact with the process through remote serial protocol.
Hence users would have to use "target remote /proc/<pid>/gdb" command on gdb prompt to start using this infrastructure.
For Breakpointing support, gdbstub needs User space breakpointing layer and uprobes layer which will be posted later.
Quoted 7 lines
Quoted 1 line
How does this compare to kgdb and related efforts?
This is a In-kernel gdbstub to debug user space programs. This stub doesnt help in debugging kernel.
Hence I am not sure how to compare kgdb gdbstub with this gdbstub. Can you please provide more pointers on what you were referring to?
Well, not even that much was clear from your changelog, so I wasn't really sure wtf I was looking at. All it says was an in-kernel gdb stub, what other than to debug the kernel would you need in-kernel stubs for?
So now my question is, what do you need an in-kernel stub to debug userspace for?
In general, tell me about this patch thing, what does it do, why does it do it, and how does it improve on the current situation.
Your changelog doesn't address any of those things, so wth are we supposed to think?
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines
This is a In-kernel gdbstub to debug user space programs. This stub doesnt help in debugging kernel.
Hence I am not sure how to compare kgdb gdbstub with this gdbstub. Can you please provide more pointers on what you were referring to?
Quoted 7 lines
Well, not even that much was clear from your changelog, so I wasn't really sure wtf I was looking at. All it says was an in-kernel gdb stub, what other than to debug the kernel would you need in-kernel stubs for?
So now my question is, what do you need an in-kernel stub to debug userspace for?
This stub would allow users use features provided by utrace but through a gdb interface. This idea was brought up in this year's Tracing roundtable at the Linux Foundation Collaboration summit, April 8-10 in San Francisco. Here is the link to the minutes of the meeting sent by Christoph Hellwig.
http://www.mail-archive.com/[email protected]/msg00830.html
Quoted 2 lines
In general, tell me about this patch thing, what does it do, why does it do it, and how does it improve on the current situation.
This is suppose to be one of the interfaces to use utrace, i.e Allow gdb to use utrace features without having to change gdb itself. Though there are not enough features in this patch, intentions include support multi-threaded debugging, concurrent debugger support for same process, syscall tracing.
For Breakpoint support(not yet submitted to LKML), it would use execution out of line rather than the conventional inline-single stepping.
I guess Christoph, Roland and Frank would be able to explain in a better fashion the rational and advantages of this stub over convential gdb.
Quoted 3 lines
Your changelog doesn't address any of those things, so wth are we supposed to think?
Thanks for pointing out.
--
Thanks and Regards
Srikar
Quoted 6 lines · Srikar Dronamraju
This is suppose to be one of the interfaces to use utrace, i.e Allow gdb to use utrace features without having to change gdb itself. Though there are not enough features in this patch, intentions include support multi-threaded debugging, concurrent debugger support for same process, syscall tracing.
Quoted 6 lines
For Breakpoint support(not yet submitted to LKML), it would use execution out of line rather than the conventional inline-single stepping.
I guess Christoph, Roland and Frank would be able to explain in a better fashion the rational and advantages of this stub over convential gdb.
Hmm,. wouldn't it make much more sense to extend the current kgdb stub to include userspace debugging, providing an all-in-one solution?
I think it would be much more powerful to be able to observe the full software stack and not be limited by this user<->kernel barrier.
(Provided the user has sufficient privileges of course).
Quoted 6 lines
Quoted 3 lines
I guess Christoph, Roland and Frank would be able to explain in a better fashion the rational and advantages of this stub over convential gdb.
Hmm,. wouldn't it make much more sense to extend the current kgdb stub to include userspace debugging, providing an all-in-one solution?
I see two limitations but I guess there could be ways to get over it.
1. gdb requiring file that needs to be debugged. I always thought it can either be a user program or a vmlinux file. gdb makes most of the information(i.e registers) from the remote protocol to display the backtrace, variable values and the like by reading this file.
2. Also I am not sure if gdb has a way to tell the remote to switch the context and provide information(registers) pertaining to the user mode/ kernel mode.
There could be other limitations too that I may not be aware.
Quoted 5 lines
I think it would be much more powerful to be able to observe the full software stack and not be limited by this user<->kernel barrier.
(Provided the user has sufficient privileges of course).
In this implementation, the current user can debug his/her own processes. May be if we can debug both the context from same gdb then we might have to place restrictions..
--
Thanks and Regards
Srikar
Quoted 1 line
Quoted 4 lines · peterz
Hmm,. wouldn't it make much more sense to extend the current kgdb stub to include userspace debugging, providing an all-in-one solution? I think it would be much more powerful to be able to observe the full software stack and not be limited by this user<->kernel barrier.
There exist other tools for this broad a scope (systemtap being one), and present gdb is not well suited for this. That makes this idea an exciting potential for the future, but not a practical short-term goal.
- FChE
Quoted 11 lines · Frank Ch. Eigler
Quoted 4 lines · peterz
Hmm,. wouldn't it make much more sense to extend the current kgdb stub to include userspace debugging, providing an all-in-one solution? I think it would be much more powerful to be able to observe the full software stack and not be limited by this user<->kernel barrier.
There exist other tools for this broad a scope (systemtap being one), and present gdb is not well suited for this. That makes this idea an exciting potential for the future, but not a practical short-term goal.
Who's talking short-term? And if gdb is unsuited, it'll simply have to get fixed :-)
Quoted 11 lines · Frank Ch. Eigler
Quoted 4 lines · peterz
Hmm,. wouldn't it make much more sense to extend the current kgdb stub to include userspace debugging, providing an all-in-one solution? I think it would be much more powerful to be able to observe the full software stack and not be limited by this user<->kernel barrier.
There exist other tools for this broad a scope (systemtap being one), and present gdb is not well suited for this. That makes this idea an exciting potential for the future, but not a practical short-term goal.
Well, but Peter's suggestion is the obvious next step - or even a necessary first step in my view.
kgdb exists here and today in the kernel and you cannot just build a facility that doesnt replace it and doesnt integrate well with it.
So if a unified user/kernel model for debugging is a 'long term' feature in your view then perhaps this framework (which introduces _extensive_ hooks all around the kernel) is not designed/approached in the right way and should not be merged in this form.
Concentrating on 'other tools' just generates extensive dependencies on something that is lacking - making it even harder to implement unified debugging down the line.
Thanks,
Ingo
Quoted 3 lines · Ingo Molnar
[...]
kgdb exists here and today in the kernel and you cannot just build a facility that doesnt replace it and doesnt integrate well with it.
Surely you don't mean that: every non-kgdb facility in the kernel meets that definition, even all debugging-related facilities such as perf and ftrace.
Quoted 4 lines
So if a unified user/kernel model for debugging is a 'long term' feature in your view then perhaps this framework (which introduces _extensive_ hooks all around the kernel) is not designed/approached in the right way and should not be merged in this form.
Which "this framework" are you talking about? Please clarify what exactly you're trying to say.
- FChE
Quoted 8 lines · Frank Ch. Eigler
Quoted 3 lines · Ingo Molnar
[...]
kgdb exists here and today in the kernel and you cannot just build a facility that doesnt replace it and doesnt integrate well with it.Surely you don't mean that: every non-kgdb facility in the kernel meets that definition, even all debugging-related facilities such as perf and ftrace.
Those facilities are not overlapping with kgdb though so my point doesnt apply to them. An in-kernel gdb server sure overlaps/extends kgdb though.
Btw., perf does meet that definition: it functionally replaces all facilities that it overlaps/extends - such as Oprofile. 'What about Oprofile?' was one of the first questions asked when we posted perf. 'What about kgdb?' is the first question to the in-kernel gdb stub submission here, for similar reasons.
Ingo
Hi -
Quoted 3 lines · Ingo Molnar
Those facilities are not overlapping with kgdb though so my point doesnt apply to them. An in-kernel gdb server sure overlaps/extends kgdb though.
Only in name. One is highly invasive, for debugging the kernel across serial consoles. The other is highly noninvasive, for debugging user processes across normal userspace channels. They both happen to talk to gdb, but that's the end of the natural "overlap".
Even if kgdb was extended to be able to manage userspace, and if gdb itself was extended to be able to use that same single channel, this would still not duplicate the use scenario for an ordinary user debugging his own processes.
(Plus, in the future where at least gdb is applied toward kernel+user debugging, it is unlikely to be the case that this would need to be done *over a single channel*. A separate channel for kernel and separate channels for userspace programs are no less likely.)
Quoted 2 lines
Btw., perf does meet that definition: it functionally replaces all facilities that it overlaps/extends - such as Oprofile. [...]
(And they currently separately coexist.)
- FChE
Quoted 18 lines · Frank Ch. Eigler
Hi -
Quoted 3 lines · Ingo Molnar
Those facilities are not overlapping with kgdb though so my point doesnt apply to them. An in-kernel gdb server sure overlaps/extends kgdb though.
Only in name. One is highly invasive, for debugging the kernel across serial consoles. The other is highly noninvasive, for debugging user processes across normal userspace channels. They both happen to talk to gdb, but that's the end of the natural "overlap".
Even if kgdb was extended to be able to manage userspace, and if gdb itself was extended to be able to use that same single channel, this would still not duplicate the use scenario for an ordinary user debugging his own processes.
Quoted 4 lines
(Plus, in the future where at least gdb is applied toward kernel+user debugging, it is unlikely to be the case that this would need to be done *over a single channel*. A separate channel for kernel and separate channels for userspace programs are no less likely.)
Well nothing that you mention here changes our obvious suggestion that an in-kernel gdb stub should obviously either be a kgdb extension, or a replacement of it. We dont want to separate facilities for the same conceptual thing: examining application state (be that in user-space and kernel-space).
Quoted 4 lines
Quoted 2 lines
Btw., perf does meet that definition: it functionally replaces all facilities that it overlaps/extends - such as Oprofile. [...]
(And they currently separately coexist.)
You didnt get my point apparently. Keeping the overlapped facility for compatibility (and general user inertia) is fine. Creating a new facility that doesnt do everything that the existing facility does, and not integrating it either, is not fine.
Which was both Peter's and my point really.
Ingo
Hi -
Quoted 9 lines
Quoted 4 lines
Only in name. One is highly invasive, for debugging the kernel across serial consoles. The other is highly noninvasive, for debugging user processes across normal userspace channels. They both happen to talk to gdb, but that's the end of the natural "overlap".
[...]
Well nothing that you mention here changes our obvious suggestion that an in-kernel gdb stub should obviously either be a kgdb extension, or a replacement of it.
Help me out here: by "kgdb extension" do you imagine "something new that an unprivileged user can use to debug his own process"? Or do you imagine a new userspace facility that single-steps the kernel?
Quoted 3 lines
We dont want to separate facilities for the same conceptual thing:
examining application state (be that in user-space and kernel-space).
This seems like a shallow sort of consistency. kgdb was added after ptrace existed -- why not extend ptrace instead to target the kernel? After all, it's "examining application state". The answer is that it doesn't make a heck of a lot of sense.
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Btw., perf does meet that definition: it functionally replaces all facilities that it overlaps/extends - such as Oprofile. [...]
(And they currently separately coexist.)
You didnt get my point apparently. Keeping the overlapped facility for compatibility (and general user inertia) is fine. Creating a new facility that doesnt do everything that the existing facility does, and not integrating it either, is not fine.
oprofile and perfctr are closer in concept than kgdb and ptrace, yet AFAIK perfctr doesn't "interface" to oprofile, except perhaps to the extent of resolving contention over the underlying physical resources. In any case this is not a great analogy.
- FChE
Quoted 8 lines · Frank Ch. Eigler
Hi -
Quoted 5 lines
Quoted 4 lines
Only in name. One is highly invasive, for debugging the kernel across serial consoles. The other is highly noninvasive, for debugging user processes across normal userspace channels. They both happen to talk to gdb, but that's the end of the natural "overlap".
[...]
Quoted 7 lines
Quoted 3 lines
Well nothing that you mention here changes our obvious suggestion that an in-kernel gdb stub should obviously either be a kgdb extension, or a replacement of it.
Help me out here: by "kgdb extension" do you imagine "something new that an unprivileged user can use to debug his own process"? Or do you imagine a new userspace facility that single-steps the kernel?
Is this a trick question? Single-stepping the kernel on the same system [especially if it's an UP system] would certainly be a challenge ;-)
What i mean is what i said: if you provide a new framework (especially if it's user visible - which both kgdb and the gdb stub is) you should either fully replace existing functionality or extend it. Overlapping it in an incomplete way is not useful to anyone.
Extending kgdb to allow the use of it as if we used gdb locally would certainly be interesting - and then you could drop into the kernel anytime as well. But i'm not siding with any particular solution - i'm just seconding Peter's point that there's very clear overlap and inconsistency, and that ought to be resolved one way or another.
Quoted 8 lines
Quoted 3 lines
We dont want to separate facilities for the same conceptual thing:
examining application state (be that in user-space and kernel-space).This seems like a shallow sort of consistency. kgdb was added after ptrace existed -- why not extend ptrace instead to target the kernel? After all, it's "examining application state". The answer is that it doesn't make a heck of a lot of sense.
kgdb simply used gdb's preferred way of remote debugging. That's certainly the ugliest bit of it btw - but it's an externality to kgdb.
Had it extended ptrace it wouldnt have gdb compatibility.
So i think this example of yours is inapposite as well.
Having said all that, i certainly subscribe to the view that neither kgdb nor ptrace is particularly cleanly done. So i wouldnt mind if something new existed that had a modern, flexible, extensible and generally pleasant interface and implementation. If you are heading in that direction, please let me know.
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Btw., perf does meet that definition: it functionally replaces all facilities that it overlaps/extends - such as Oprofile. [...]
(And they currently separately coexist.)
You didnt get my point apparently. Keeping the overlapped facility for compatibility (and general user inertia) is fine. Creating a new facility that doesnt do everything that the existing facility does, and not integrating it either, is not fine.
Quoted 4 lines
oprofile and perfctr are closer in concept than kgdb and ptrace, yet AFAIK perfctr doesn't "interface" to oprofile, except perhaps to the extent of resolving contention over the underlying physical resources. In any case this is not a great analogy.
(FYI, 'perfctr' is a different project that has existed for years, i suspect you meant perf events?)
perf replaces oprofile functionally. If the in-kernel gdb stub replaced kgdb functionally you'd hear no complaints from me.
Ingo
Hi -
Quoted 11 lines
Quoted 3 lines
Help me out here: by "kgdb extension" do you imagine "something new that an unprivileged user can use to debug his own process"? Or do you imagine a new userspace facility that single-steps the kernel?
Is this a trick question? Single-stepping the kernel on the same system [especially if it's an UP system] would certainly be a challenge ;-)
What i mean is what i said: if you provide a new framework (especially if it's user visible - which both kgdb and the gdb stub is) you should either fully replace existing functionality or extend it. Overlapping it in an incomplete way is not useful to anyone.
But there is no "overlap" beyond the name. The functional scope of the two interfaces is totally non-overlapping, and are consistent with the current chasms between kernel- and user-side debugging.
Sure, in the future, it may make sense to teach the kernel-side (kgdb serial console) interface to manipulate userspace. But that will require a gdb extension. And it would not satisfy an unprivileged user's need to debug pure userspace (in a better way than current ptrace can).
This is why I keep asking for specificity as to this "new framework" you imagine. Just sharing definitions such as kgdb_arch/kgdb_io but otherwise completely disconnected (separate channels)?
Quoted 3 lines
Extending kgdb to allow the use of it as if we used gdb locally would certainly be interesting - and then you could drop into the kernel anytime as well.
(Is this a restatement of the "trick question" idea?)
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
We dont want to separate facilities for the same conceptual thing:
examining application state (be that in user-space and kernel-space).
Quoted 8 lines
Quoted 4 lines
This seems like a shallow sort of consistency. kgdb was added after ptrace existed -- why not extend ptrace instead to target the kernel? After all, it's "examining application state". The answer is that it doesn't make a heck of a lot of sense.
kgdb simply used gdb's preferred way of remote debugging. That's certainly the ugliest bit of it btw - but it's an externality to kgdb. Had it extended ptrace it wouldnt have gdb compatibility.
So, because of a constraint for gdb compatibility, you built a separate interface for kgdb vs. ptrace. Fine. Do you accept that, even if a hypothetical single channel existed for which kernel- and user-space debugging could occur, current gdb is not compatible with this? So by your own reasoning, such a facility should not be mandated as a "necessary first step".
Quoted 1 line
[...] perf replaces oprofile functionally.
(I'm told that it's not a strict superset from a functional point of view, FWIW, something about a larger selection of low level hardware counters.)
Quoted 2 lines
If the in-kernel gdb stub replaced kgdb functionally you'd hear no complaints from me.
Let's leave it as an idea for the future.
- FChE
Quoted 6 lines · Frank Ch. Eigler
Quoted 4 lines
[...]
If the in-kernel gdb stub replaced kgdb functionally you'd hear no complaints from me.
Let's leave it as an idea for the future.
We came a full circle - that's the argument. We say overlap, duplication and incomplete implementation in this area is a problem. Since the speed of development in this area is truly glacial at the moment and the practical advantages that i can experience personally (directly as a Linux user and indirectly as a maintainer) are miniscule so far, caution is warranted IMO. You say there is no problem. We'll have to agree to disagree i guess.
Ingo
Quoted 4 lines · Ingo Molnar
[...] Since the speed of development in this area is truly glacial at the moment and the practical advantages that i can experience personally (directly as a Linux user and indirectly as a maintainer) are miniscule so far, caution is warranted IMO. [...]
If the "caution" you suggest is operationally equivalent to discouraging even miniscule improvements, is it any wonder that progress is glacial?
The gdbstub prototype was constructed for two reasons: to demonstrate utrace usage now, and in the future to be incrementally useful (over ptrace, by moving into fast kernel-space operations like multithreading control, gdb-tracepoint support, other stuff). #1 is about done. With respect to #2, we can certainly commit to ongoing work on improvements, provided the community shows interest and goodwill.
- FChE
Quoted 11 lines · Frank Ch. Eigler
Quoted 4 lines · Ingo Molnar
[...] Since the speed of development in this area is truly glacial at the moment and the practical advantages that i can experience personally (directly as a Linux user and indirectly as a maintainer) are miniscule so far, caution is warranted IMO. [...]
If the "caution" you suggest is operationally equivalent to discouraging even miniscule improvements, is it any wonder that progress is glacial?
I think you might be mixing up cause and causation ;-)
Quoted 7 lines
The gdbstub prototype was constructed for two reasons: to demonstrate utrace usage now, and in the future to be incrementally useful (over ptrace, by moving into fast kernel-space operations like multithreading control, gdb-tracepoint support, other stuff). #1 is about done. With respect to #2, we can certainly commit to ongoing work on improvements, provided the community shows interest and goodwill.
What i'd like to see is measurable benefits to users, developers and maintainers. I'd like to see the same for SystemTap too btw.
Ingo
Hi -
Quoted 8 lines · Ingo Molnar
[...]
Quoted 4 lines
The gdbstub prototype was constructed for two reasons: to demonstrate utrace usage now, and in the future to be incrementally useful (over ptrace, by moving into fast kernel-space operations like multithreading control, gdb-tracepoint support, other stuff). [...]
What i'd like to see is measurable benefits to users, developers and maintainers.
OK, it's clear that in the gdb-stub's case, some positive motivation beyond it being an api-educational example would be appropriate before merging. Note that it was only an RFC at the time.
Quoted 1 line
I'd like to see the same for SystemTap too btw.
systemtap's benefits are beyond question to its users. (Others are worried about systemtap problems, but that wasn't your question.) In what form do you expect to see such measurements? It would help if you could point out prior examples of such "measurable benefit" analyses that perchance accompanied other then-new kernel features, say in the tracing/debugging area.
- FChE