fix kallsyms to allow discrimination of local symbols
RFC 1/1 fix kallsyms to allow discrimination of local symbols
The problem is that local symbols, being hidden from the linker, might not be unique. Thus, they don't make good anchors for the symbol relative addressing used by kprobes (it takes the first occurrence it finds). Likewise, when they appear in stack traces, it's sometimes not obvious which local symbol it is (although context usually allows an easy guess).
Fix all of this by prefixing local symbols with the actual C file name they occur in separated by '|' (I had to use '|' since ':' is already in use for module prefixes in kallsyms lookups.
I also had to rewrite mksysmap in perl because the necessary text formatting changes in shell are painfully slow.
Comments?
James
---
Makefile +2 -2
@@ -685,7 +685,7 @@ quiet_cmd_vmlinux_version = GEN .version # Generate System.map quiet_cmd_sysmap = SYSMAP - cmd_sysmap = $(CONFIG_SHELL) $(srctree)/scripts/mksysmap + cmd_sysmap = $(PERL) $(srctree)/scripts/mksysmap # Link of vmlinux # If CONFIG_KALLSYMS is set .version is already updated
@@ -759,7 +759,7 @@ endef # Generate .S file with all kernel symbols quiet_cmd_kallsyms = KSYM $@ - cmd_kallsyms = $(NM) -n $< | $(KALLSYMS) \ + cmd_kallsyms = $(NM) -n -l $< | sed "s|`pwd`/||" | $(KALLSYMS) \ $(if $(CONFIG_KALLSYMS_ALL),--all-symbols) > $@ .tmp_kallsyms1.o .tmp_kallsyms2.o .tmp_kallsyms3.o: %.o: %.S scripts FORCE
scripts/kallsyms.c +20 -4
@@ -63,11 +63,12 @@ static inline int is_arm_mapping_symbol(const char *str)static int read_symbol (FILE *in,struct sym_entry *s) { -char str[500 ]; +char str[500 ], file[500 ];char *sym, stype; -int rc; +int rc, c, line; - rc =fscanf (in,"%llx %c %499s\n" , &s->addr, &stype, str); + file[0 ] ='\0' ; + rc =fscanf (in,"%llx %c %499s" , &s->addr, &stype, str);if (rc !=3 ) {if (rc !=EOF ) {/* skip line */
@@ -75,6 +76,12 @@ static int read_symbol(FILE *in, struct sym_entry *s) }return -1 ; } + c =fgetc (in); +if (c !='\n' ) { + rc =fscanf (in,"%499[^:]:%d\n" , file, &line); +if (rc !=2 ) + file[0 ] ='\0' ; + } sym = str;/* skip prefix char */
@@ -115,13 +122,22 @@ static int read_symbol(FILE *in, struct sym_entry *s)/* include the type field in the symbol name, so that it gets * compressed together */ s->len =strlen (str) +1 ; +if (islower (stype)) + s->len +=strlen (file) +1 ; s->sym =malloc (s->len +1 );if (!s->sym) {fprintf (stderr,"kallsyms failure: " "unable to allocate required amount of memory\n" );exit (EXIT_FAILURE ); } -strcpy ((char *)s->sym +1 , str); +if (islower (stype)) { +char *ss = (char *)s->sym +1 ; + +strcpy (ss, file); +strcat (ss,"|" ); +strcat (ss, str); + }else +strcpy ((char *)s->sym +1 , str); s->sym[0 ] = stype;return 0 ;
scripts/mksysmap +18 -2
@@ -1,4 +1,4 @@ -#!/bin/sh -x +#!/usr/bin/perl # Based on the vmlinux file create the System.map file # System.map is used by module-init tools and some debugging # tools to retrieve the actual addresses of symbols in the kernel.
@@ -41,5 +41,21 @@ # so we just ignore them to let readprofile continue to work. # (At least sparc64 has __crc_ in the middle). -$NM -n $1 | grep -v '\( [aNUw] \)\|\(__crc_\)\|\( \$[adt]\)' > $2 +chomp($cwd = `pwd`); +open(I, "nm -n -l $ARGV[0]|") || die; +open(O, ">$ARGV[1]") || die; +foreach(<I>) { + chomp; + ($addr, $type, $symbol, $file_and_line) = split(/[ ]/, $_); + next if ($type =~ /[aNUw]/ || $type =~ /\$[adt]/); + next if ($symbol=~ /__crc_/); + if ($type =~ /[a-z]/ && $file_and_line) { + ($_) = split(/:/, $file_and_line); + (undef, $file) = split(/^$cwd\//, $_); + $symbol = $file."|".$symbol; + } + print O "$addr $type $symbol\n"; +} + +
Quoted 4 lines · James Bottomley
[...] Fix all of this by prefixing local symbols with the actual C file name they occur in separated by '|' (I had to use '|' since ':' is already in use for module prefixes in kallsyms lookups. [...] Comments?
Can we take some time to review how we got here?
- You disprefer systemtap's use of an established, non-deprecated API for placing kernel probes. (We calculate addresses by a mixture of elf-analysis and runtime user-space lookup means. That's partly since kallsyms_lookup was unexported over our objections.) There is nothing outright broken (e.g. incorrect numbers) with what systemtap has been doing for years.
- You argue that symbols+offset kprobing is better. We can see that, in some sense, but ...
- I explain that we are used to final address calculating, as we'll have to do that regardless for user-space probes. Plus we need to work with kernels that predate the symbol+offset kprobe api extension. So this change would not simplify systemtap in any way. You do not respond.
- I offer _stext+offset (for the kernel) and (.text*)+offset (for modules) kprobes: basically to use the "better" symbol+offset kprobes api, but use the same single reference addresses we already do, and leaving just the final addition to the kernel. You do not respond materially.
- You argue that it cannot only be any symbol+offset ... but the actual nearest symbol+offset. But that doesn't work for local symbols. So you fix that to the nearest globally visible symbol+offset. But this requires:
- yet more extra work and code from systemtap
- extension to the kernel build system, and kallsyms runtime data to fix the current local-symbol-ambiguity problem
- storage of all that new file name data in permanent unswappable kernel data (>>100kB, if done simply prefixing local symbol names file file names).
- possible further complications related to filename string matching
- You have yet to invent a scheme to allow offloading *data* address calculations to the kernel. Without that (and perhaps more), systemtap will *still* have to fetch same base _stext etc. addresses at run time that it currently does -- even if it did not use them to compute kprobes addresses.
In total, this path would end up with both systemtap and the kernel more complex, larger and a bit slower too. Does that still seem an acceptable cost, just to get systemtap to change its preferred kprobes api?
- FChE
Quoted 10 lines · Frank Ch. Eigler
Quoted 4 lines · James Bottomley
[...] Fix all of this by prefixing local symbols with the actual C file name they occur in separated by '|' (I had to use '|' since ':' is already in use for module prefixes in kallsyms lookups. [...] Comments?
Can we take some time to review how we got here?
Quoted 6 lines
- You disprefer systemtap's use of an established, non-deprecated API for placing kernel probes. (We calculate addresses by a mixture of elf-analysis and runtime user-space lookup means. That's partly since
kallsyms_lookupwas unexported over our objections.) There is nothing outright broken (e.g. incorrect numbers) with what systemtap has been doing for years.
You mean embedding half a megabyte of symbols simply so you can avoid the inconvenience of using a kernel API? yes, I think it's ... suboptimal.
Quoted 8 lines
- You argue that symbols+offset kprobing is better. We can see that, in some sense, but ...
- I explain that we are used to final address calculating, as we'll have to do that regardless for user-space probes. Plus we need to work with kernels that predate the symbol+offset kprobe api extension. So this change would not simplify systemtap in any way. You do not respond.
There is no current userspace infrastructure, since utrace still isn't in the kernel, so you're predicating this argument on an event which hasn't happened.
Even assuming utrace is accepted, embedding the symbol table of every user space process in the probes is still daft. It's this constant assertion that "it must be done my way" that's causing such a drag on the open source process. For instance, the obvious way to me of doing this would be to map the user space stack into the systemtap runtime and unwind it from there instead of vectoring it into the kernel.
Quoted 5 lines
- I offer _stext+offset (for the kernel) and (.text*)+offset (for modules) kprobes: basically to use the "better" symbol+offset kprobes api, but use the same single reference addresses we already do, and leaving just the final addition to the kernel. You do not respond materially.
I thought this and subsequent emails addressed the points pretty well:
http://marc.info/?l=linux-kernel&m=121632572409118
Quoted 5 lines
- You argue that it cannot only be any symbol+offset ... but the actual nearest symbol+offset. But that doesn't work for local symbols. So you fix that to the nearest globally visible symbol+offset. But this requires:
- yet more extra work and code from systemtap
I'm afraid that's how open source development works ... you iterate to find the best solution
Quoted 2 lines
- extension to the kernel build system, and kallsyms runtime data to fix the current local-symbol-ambiguity problem
Finding weaknesses in APIs and fixing them is what it's all about.
Quoted 3 lines
- storage of all that new file name data in permanent unswappable kernel data (>>100kB, if done simply prefixing local symbol names file file names).
I'd check my facts before making assertions. The kernel symbol table is stored in a compressed form that actually eliminates most of these repetitions.
Quoted 1 line
- possible further complications related to filename string matching
Any substantiation of that?
Quoted 5 lines
- You have yet to invent a scheme to allow offloading *data* address calculations to the kernel. Without that (and perhaps more), systemtap will *still* have to fetch same base
_stextetc. addresses at run time that it currently does -- even if it did not use them to compute kprobes addresses.
That would be because I haven't actually started looking at this one yet. Of course, that would make it a great starting point for others who wished to help.
Quoted 2 lines
In total, this path would end up with both systemtap and the kernel more complex, larger and a bit slower too.
Really? I count the reduction of the probe modules from 500kb to 50kb a worthwhile saving. I don't even see where anything became larger.
Quoted 3 lines
Does that still seem an
acceptable cost, just to get systemtap to change its preferred kprobes api?
James
Hi -
Quoted 7 lines · James Bottomley
[...]
Quoted 2 lines
- You disprefer systemtap's use of an established, non-deprecated API for placing kernel probes. [...]
You mean embedding half a megabyte of symbols simply so you can avoid the inconvenience of using a kernel API? yes, I think it's ... suboptimal.
It has been explained already that the symbol table you saw in stap-symbols.h has nothing to do with the kprobe addressing issue.
http://www.gossamer-threads.com/lists/linux/kernel/947365#957365
Quoted 9 lines
Quoted 8 lines
- You argue that symbols+offset kprobing is better. We can see that, in some sense, but ...
- I explain that we are used to final address calculating, as we'll have to do that regardless for user-space probes. Plus we need to work with kernels that predate the symbol+offset kprobe api extension. So this change would not simplify systemtap in any way. You do not respond.
Quoted 3 lines
There is no current userspace infrastructure, since utrace still isn't in the kernel, so you're predicating this argument on an event which hasn't happened.
We exercise professional foresight. And the backward compatibility issue remains even without that.
Quoted 2 lines
Even assuming utrace is accepted, embedding the symbol table of every user space process in the probes is still daft. [....]
It would take space, no question, though we're not talking about "every" process, just designated ones.
Quoted 3 lines
For instance, the obvious way to me of doing this would be to map the user space stack into the systemtap runtime and unwind it from there instead of vectoring it into the kernel.
Please elaborate. What does mapping a stack into the runtime mean? Do you mean to suggest having the userspace program unwind itself? Or relying on the userspace programs' possibly-paged-out unwind data? That would be intrusive.
Quoted 9 lines
Quoted 5 lines
- I offer _stext+offset (for the kernel) and (.text*)+offset (for modules) kprobes: basically to use the "better" symbol+offset kprobes api, but use the same single reference addresses we already do, and leaving just the final addition to the kernel. You do not respond materially.
I thought this and subsequent emails addressed the points pretty well:
No, they didn't. Every time I explained about how it does work, you just claimed "not", without even a single worked-out substantiating example.
Quoted 8 lines
[...]
Quoted 3 lines
- storage of all that new file name data in permanent unswappable kernel data (>>100kB, if done simply prefixing local symbol names file file names).
I'd check my facts before making assertions. The kernel symbol table is stored in a compressed form that actually eliminates most of these repetitions.
A careful reader will notice the "if" in my sentence. Anyway, that or a superior compression scheme could apply to systemtap's various tables too.
Quoted 3 lines
Quoted 1 line
- possible further complications related to filename string matching
Any substantiation of that?
We have had reported problems with differences between kernels hand-built with long absolute source path names versus the smallest "kernel/foo.c" names. If such canonicalization takes place but inconsistently by the different tools, we will have a problem.
Quoted 6 lines
[...]
Quoted 2 lines
In total, this path would end up with both systemtap and the kernel more complex, larger and a bit slower too.
Really? I count the reduction of the probe modules from 500kb to 50kb a worthwhile saving.
The red clupea harengus again.
Quoted 1 line
I don't even see where anything became larger.
Even with ksymtab compression, there is still new data to be stored in the kernel, and it is extra for each systemtap probe datum.
Quoted 4 lines
Quoted 2 lines
Does that still seem an acceptable cost, just to get systemtap to change its preferred kprobes api?
[no answer]
Indeed.
- FChE
Quoted 7 lines · Frank Ch. Eigler
Hi -
Quoted 4 lines · James Bottomley
[...]
Quoted 2 lines
- You disprefer systemtap's use of an established, non-deprecated API for placing kernel probes. [...]
Quoted 8 lines
Quoted 3 lines
You mean embedding half a megabyte of symbols simply so you can avoid the inconvenience of using a kernel API? yes, I think it's ... suboptimal.
It has been explained already that the symbol table you saw in
stap-symbols.hhas nothing to do with the kprobe addressing issue.http://www.gossamer-threads.com/lists/linux/kernel/947365#957365
You're confusing issues. I said embedding half a megabyte of symbol table that the kernel already has is a bad idea full stop. The ultimate think I'm looking to do is to evolve kernel APIs that makes this practice unnecessary.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
- You argue that symbols+offset kprobing is better. We can see that, in some sense, but ...
- I explain that we are used to final address calculating, as we'll have to do that regardless for user-space probes. Plus we need to work with kernels that predate the symbol+offset kprobe api extension. So this change would not simplify systemtap in any way. You do not respond.
Quoted 6 lines
Quoted 3 lines
There is no current userspace infrastructure, since utrace still isn't in the kernel, so you're predicating this argument on an event which hasn't happened.
We exercise professional foresight. And the backward compatibility issue remains even without that.
No ... you're trying to constrain the open source process to a pre-conceived design which is unrealised by in-kernel code. This is directly producing an impasse.
Backwards compatibility is the province of distros. The job of upstream is to produce the best tool and environment we can. After this, the backwards portability of desirable pieces can be discussed.
Quoted 11 lines
Quoted 2 lines
Even assuming utrace is accepted, embedding the symbol table of every user space process in the probes is still daft. [....]
It would take space, no question, though we're not talking about "every" process, just designated ones.
Quoted 3 lines
For instance, the obvious way to me of doing this would be to map the user space stack into the systemtap runtime and unwind it from there instead of vectoring it into the kernel.
Please elaborate. What does mapping a stack into the runtime mean?
It means that the systemtap runtime and the process would share a mapping for the process stacks Obviously the process would have to be quiesced to poke about in it, but it obviates the need to vector megabytes of stack information through the kernel.
Quoted 1 line
Do you mean to suggest having the userspace program unwind itself?
It's possible ... but more likely that the stap runtime would do the unwinding. Which is more efficient won't really be known until someone actually tries coding it.
Quoted 2 lines
Or relying on the userspace programs' possibly-paged-out unwind data?
That would be intrusive.
I think you'll find doing it in user space is an advantage for paged out data. It's much more complex to get to it in the kernel because you have to be careful of context while you're asking for it to be paged back in.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
- I offer _stext+offset (for the kernel) and (.text*)+offset (for modules) kprobes: basically to use the "better" symbol+offset kprobes api, but use the same single reference addresses we already do, and leaving just the final addition to the kernel. You do not respond materially.
Quoted 7 lines
Quoted 3 lines
I thought this and subsequent emails addressed the points pretty well:
No, they didn't. Every time I explained about how it does work, you just claimed "not", without even a single worked-out substantiating example.
Really? The mutability of _stext vs _text; the problem probing init sections I think they're real issues.
Quoted 14 lines
Quoted 8 lines
[...]
Quoted 3 lines
- storage of all that new file name data in permanent unswappable kernel data (>>100kB, if done simply prefixing local symbol names file file names).
I'd check my facts before making assertions. The kernel symbol table is stored in a compressed form that actually eliminates most of these repetitions.
A careful reader will notice the "if" in my sentence. Anyway, that or a superior compression scheme could apply to systemtap's various tables too.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
- possible further complications related to filename string matching
Any substantiation of that?
We have had reported problems with differences between kernels hand-built with long absolute source path names versus the smallest "
kernel/foo.c" names. If such canonicalization takes place but inconsistently by the different tools, we will have a problem.
What it currently does is add tree relative names, so, for example this is a cut and paste from System.map:
c0100000 T _textc0100000 T startup_32c0100054 t arch/x86/kernel/head_32.S|default_entry c01000a8 T startup_32_smp
c010012a t arch/x86/kernel/head_32.S|checkCPUtype c01001ab t arch/x86/kernel/head_32.S|is486 c01001b2 t arch/x86/kernel/head_32.S|is386
c0100220 T initial_codec0100224 t arch/x86/kernel/head_32.S|check_x87
There's no ambiguity about how the path is constructed.
Quoted 8 lines
Quoted 6 lines
[...]
Quoted 2 lines
In total, this path would end up with both systemtap and the kernel more complex, larger and a bit slower too.
Really? I count the reduction of the probe modules from 500kb to 50kb a worthwhile saving.
The red clupea harengus again.
only if alio populo leges tantum adhiberent
Quoted 4 lines
Quoted 1 line
I don't even see where anything became larger.
Even with ksymtab compression, there is still new data to be stored in the kernel, and it is extra for each systemtap probe datum.
I think the requirement that your huge data problem be solved at absolutely no cost is possibly a bit ambitious. However, it can be done at little cost to the kernel.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Does that still seem an acceptable cost, just to get systemtap to change its preferred kprobes api?
[no answer]
Indeed.
Perhaps because that's the wrong question. It's not about trying to change a "preferred" API (and the concept of preferred means according to your preset design). It's about trying to get systemtap actually to engage with the kernel and iterate to an actual solution.
James
Quoted 9 lines · James Bottomley
[...]
Quoted 8 lines
Quoted 7 lines
[...]
Quoted 2 lines
- You disprefer systemtap's use of an established, non-deprecated API for placing kernel probes. [...]
You mean embedding half a megabyte of symbols simply so you can avoid the inconvenience of using a kernel API? yes, I think it's ... suboptimal.
Quoted 6 lines
Quoted 3 lines
It has been explained already that the symbol table you saw in
stap-symbols.hhas nothing to do with the kprobe addressing issue. [...]You're confusing issues. I said embedding half a megabyte of symbol table that the kernel already has is a bad idea full stop.
Be that as it may, but it is not appropriate to knowingly bring up a topic that is irrelevant to the patch series you're actually proposing.
Quoted 2 lines
The ultimate think I'm looking to do is to evolve kernel APIs that makes this practice unnecessary.
If by "this practice" you mean "stap-symbols.h tables", then you're worrying on the wrong area of code (anything in/near kprobes). I am happy to suggest more appropriate areas to help with that.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
There is no current userspace infrastructure, since utrace still isn't in the kernel, so you're predicating this argument on an event which hasn't happened.
We exercise professional foresight. And the backward compatibility issue remains even without that.
No ... you're trying to constrain the open source process to a pre-conceived design which is unrealised by in-kernel code.
The "open source process" does not entail welcoming unnecessary changes.
Quoted 1 line
This is directly producing an impasse.
I don't recognize an impasse. This aspect of systemtap has been working just fine with the kernel as has been. We are not blocked on a resolution to the current issue.
Quoted 11 lines
Quoted 6 lines
[...]
Quoted 3 lines
For instance, the obvious way to me of doing this would be to map the user space stack into the systemtap runtime and unwind it from there instead of vectoring it into the kernel.
Please elaborate. What does mapping a stack into the runtime mean?
It means that the systemtap runtime and the process would share a mapping for the process stacks Obviously the process would have to be quiesced to poke about in it, but it obviates the need to vector megabytes of stack information through the kernel.
I still don't understand. No one has seriously proposed "vectoring megabytes of stack information through the kernel", if by that you mean copying it all to userspace. Even frame-pointer-based dtrace doesn't do that.
Accessing the process stack from systemtap modules is not the problem in the first place: rather, it's instant access to the unwind & symbol data needed to decode it.
Quoted 5 lines
Quoted 1 line
Do you mean to suggest having the userspace program unwind itself?
It's possible ... but more likely that the stap runtime would do the unwinding. Which is more efficient won't really be known until someone actually tries coding it.
It's not just a matter of efficiency but a matter of non-disruptiveness.
Quoted 7 lines
Quoted 2 lines
Or relying on the userspace programs' possibly-paged-out unwind data?
That would be intrusive.I think you'll find doing it in user space is an advantage for paged out data. It's much more complex to get to it in the kernel because you have to be careful of context while you're asking for it to be paged back in.
We would not ask for anything to be paged back in. That is one kind of disruption to system state that we strain to avoid.
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
- I offer _stext+offset (for the kernel) and (.text*)+offset (for modules) kprobes [...]
I thought this and subsequent emails addressed the points pretty well:
http://marc.info/?l=linux-kernel&m=121632572409118No, they didn't. Every time I explained about how it does work, you just claimed "not", without even a single worked-out substantiating example.
Really? The mutability of
_stextvs_text; the problem probing init sections I think they're real issues.
Then please do me the courtesy of replying to the points already made on those items.
(Recap: we don't care whether the vmlinux reference symbol is _text or _stext or something else. Whatever works, and so far _stext does everywhere we've needed it.)
For init sections in general, they are irrelevant for systemtap at this time as kprobes blocks them, and we'd need extra kernel/module.c infrastructure to let us hook module-loaded init-not-yet-run events (and the corresponding exit transitions). Once those are solved, systemtap can trivially calculate .init addresses relative to _stext or whatever -- and it does today!)
So, they do not appear to be real issues.
Quoted 9 lines
[...]
Quoted 4 lines
We have had reported problems with differences between kernels hand-built with long absolute source path names versus the smallest "
kernel/foo.c" names. If such canonicalization takes place but inconsistently by the different tools, we will have a problem.What it currently does is add tree relative names, so, for example this is a cut and paste from System.map: [...]
There's no ambiguity about how the path is constructed.
Perhaps, as long as the dwarf-parsing side canonicalizes them the same way.
Quoted 5 lines
Quoted 2 lines
Even with ksymtab compression, there is still new data to be stored in the kernel, and it is extra for each systemtap probe datum.
I think the requirement that your huge data problem be solved at absolutely no cost is possibly a bit ambitious.
What "huge data problem"? Forget the stap-symbols.h already.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
Does that still seem an acceptable cost, just to get systemtap to change its preferred kprobes api?
[no answer]
Indeed.
Perhaps because that's the wrong question. It's not about trying to change a "preferred" API (and the concept of preferred means according to your preset design).
Preferred, as in working, stable, not in any way deprecated, backward compatible -- and more. It ain't broke and it don't need fixin'.
Quoted 2 lines
It's about trying to get systemtap actually to engage with the kernel and iterate to an actual solution.
It is unfortunate that this technical trivium wants to turn into jabs about "engaging with the kernel" or "constraining the open source process" or something. Let the idea stand on its own merits.
- FChE
Quoted 6 lines · Frank Ch. Eigler
Quoted 2 lines
You're confusing issues. I said embedding half a megabyte of symbol table that the kernel already has is a bad idea full stop.
Be that as it may, but it is not appropriate to knowingly bring up a topic that is irrelevant to the patch series you're actually proposing.
Quoted 6 lines
Quoted 2 lines
The ultimate think I'm looking to do is to evolve kernel APIs that makes this practice unnecessary.
If by "this practice" you mean "
stap-symbols.htables", then you're worrying on the wrong area of code (anything in/near kprobes). I am happy to suggest more appropriate areas to help with that.
I think there is a certain amount of talking past each other which is going on here.
Let me see if I can summarize what is going on. My understanding of the situation is as follows:
1) Right now, when trying to resolve the address to install a kprobe, Systemtap is using the DWARF information from the vmlinux file and/or the module's debuginfo files.
2) Kprobe can support setting probes based on raw addresses (which is what Systemtap is doing now) as well as a text symbol looked up from kallsyms plus an offset. The latter was introduced two years ago, but Systemtap does not take advantage of it.
3) James is offended by the fact that Systemtap is not utilizing this interface, and has offerred up patches which adds the capability of using the symbol+offset feature of the kprobes interface. There seems to be some question as to whether his patch has all of the functionality of the existing code which determines addresses via pawing through DWARF headers (especially where a function may have been inlined one or more times) but presumably James' patches could be enhanced to deal with this issues, if he hasn't done so already.
Am I missing anything?
So the main arguments against this seems to be that it by itself this doesn't actually reduce any complexity or code in Systemtap because there are other places (kernel data segment, user space tracing if it ever manages to get merged into mainline, etc.) which needs to be able to paw through DWARF headers.
James' argument that this Systemtap is cramming over half a megabyte is regarded by is somewhat of a red herring with respect to this specific patch, since systemtap does not calculate probe addresses at runtime. This 600k or so of symbol tables is being used for something else. (What, exactly?!?) James clearly in the long term wants to make this go away, which seems reasonable. He views changing how kprobes are placed is the first in a series of changes that he would like to make. So perhaps the resistance in accepting his first patch troubles him since it appears that Systemtap folks aren't willing to take his suggestions about how things should be improved.
May I make a suggestion and not try to come to a conclusion about the big picture question for a moment and focus on the very short-term of whether it is better that when I implement a probe such as:
probe module("ext4dev").function("ext4_fill_super")
{
printk("here am I!\n");
}This should be done via passing a hard-coded address, or via using the kprobes function+offset facility? It would seem that there are advantages to James' patch all by itself, in that it will will work even if the debuginfo information for the ext4dev module can't be found, since the kallsyms information would be used instead. It also means that the resulting systemtap probe modules will be easier to make more kernel independent, since it won't be using a hard-coded address, but rather a symbolic name.
So what is the good reason *not* to do things the way James has suggested? The kprobes kallsyms facility has been around for a while now. Is it that we need to make changes?
Maybe if things are focused on somewhat more concrete questions, we can make progress.
Regards,
- Ted
Hi, Ted -
Quoted 4 lines · Theodore Tso
[...]
1) Right now, when trying to resolve the address to install a kprobe, Systemtap is using the DWARF information from the vmlinux file and/or the module's debuginfo files.
Or as a fallback, it can use textual symbol tables, such as a System.map or /proc/kallsyms dump.
Quoted 4 lines
2) Kprobe can support setting probes based on raw addresses (which is what Systemtap is doing now) as well as a text symbol looked up from kallsyms plus an offset. The latter was introduced two years ago, but Systemtap does not take advantage of it.
Right.
Quoted 5 lines
3) James is offended by the fact that Systemtap is not utilizing this interface, and has offerred up patches which adds the capability of using the symbol+offset feature of the kprobes interface. [...]
Am I missing anything?
I also proposed a compromise where systemtap would use the symbol+offset interface, but choose a single convenient symbol as base for all probes in a particular elf file (/section).
Quoted 5 lines
So the main arguments against this seems to be that it by itself this doesn't actually reduce any complexity or code in Systemtap because there are other places (kernel data segment, user space tracing if it ever manages to get merged into mainline, etc.) which needs to be able to paw through DWARF headers.
Right.
Quoted 4 lines
James' argument that this Systemtap is cramming over half a megabyte is regarded by is somewhat of a red herring with respect to this specific patch, since systemtap does not calculate probe addresses at runtime.
Or more precisely, not from that table.
Quoted 2 lines
This 600k or so of symbol tables is being used for something else. (What, exactly?!?)
They are there to enable scripts to perform address-to-symbol and symbol-to-address mappings on demand. This comes up in several contexts, one of which is symbolic stack unwinding, another is a set of explicit utility functions that are/will be available.
It is possible that through analysis of any particular script's contents, systemtap will be able to determine that this data is unused, and thus compile it out.
Quoted 2 lines
James clearly in the long term wants to make this go away, which seems reasonable.
Right, though this is an active development area whose ultimate costs/benefits we do not yet know.
Quoted 2 lines
He views changing how kprobes are placed is the first in a series of changes that he would like to make.
Right, keeping in mind that this change does not advance that prior goal.
Quoted 3 lines
So perhaps the resistance in accepting his first patch troubles him since it appears that Systemtap folks aren't willing to take his suggestions about how things should be improved.
Actually, this is not James' first patch; I am grateful for several that are in systemtap already and others are in the queue. But point taken.
Quoted 8 lines
May I make a suggestion and not try to come to a conclusion about the big picture question for a moment and focus on the very short-term of whether it is better that when I implement a probe such as:
probe module("ext4dev").function("ext4_fill_super")
{ printk("here am I!\n"); }
Sure.
Quoted 5 lines
This should be done via passing a hard-coded address, or via using the kprobes function+offset facility? It would seem that there are advantages to James' patch all by itself, in that it will will work even if the debuginfo information for the ext4dev module can't be found, since the kallsyms information would be used instead.
As a quality-of-implementation matter, systemtap checks at translation time that such probes make sense -- that "ext4_fill_super" even exists. (That is needed also to expand wildcards.) The only way it can do that is if it has dwarf or separate textual symbol table data (see above). Both of those carry addresses as well, so we might as well use them.
Quoted 3 lines
It also means that the resulting systemtap probe modules will be easier to make more kernel independent, since it won't be using a hard-coded address, but rather a symbolic name.
This is true, but I think it would apply to only a small class of scripts. Such a script would have to access no $context variables (since those require dwarf location/type data), and cannot include statement probes at function interiors (since those would require dwarf-derived non-zero symbol offsets), and cannot use any other runtime facility that involves arbitrary symbol<->address lookups.
Even then, kernel-independence in the sense of being able to copy and reuse a compiled probe module amongst separate kernel builds seems like a faint hope anyway, considering modversions and our own internal version-matching checks. But maybe there is an opportunity here.
Quoted 3 lines
So what is the good reason *not* to do things the way James has suggested? The kprobes kallsyms facility has been around for a while now. Is it that we need to make changes?
One reason is that James' proposed code breaks systemtap for pre-kallsyms-kprobes kernels, and those kernels too that have kallsyms-kprobes but not the RFC'd new one that has the source file names encoded within them. It could instead use e.g. our autoconf* facility to generate code compatible with them all.
Another reason is that it likely breaks systemtap for impending code for user-space by replacing rather than extending various internal data structures that deal with this.
I have seen little sympathy expressed for either of these concerns, which means that the new code would primarily allay some offense (but not constitute a bug fix or "usability for kernel developers" matter), and leave us to pick up the pieces. We can't make a habit out of that sort of thing, but maybe as a one-off in the interest of mutual goodwill, we should work out a way to get it done.
Quoted 2 lines
Maybe if things are focused on somewhat more concrete questions, we can make progress.
Thank you.
- FChE
Quoted 3 lines · Frank Ch. Eigler
I also proposed a compromise where systemtap would use the symbol+offset interface, but choose a single convenient symbol as base for all probes in a particular elf file (/section).
I guess I don't see the value of that over just using the address directly. James' point wasn't just to use the symbol+offset feature just for the sake of using it, but rather as a better way of specifying how to insert a probe into a kernel. For example, it may be that by allowing the kernel to have a bit more semantic knowledge of where a probe is going, it could more easily do various safety-related checks that can't be done if all it is given is a a raw address.
Quoted 11 lines
Quoted 10 lines
probe module("ext4dev").function("ext4_fill_super")
{ printk("here am I!\n"); }This should be done via passing a hard-coded address, or via using the kprobes function+offset facility? It would seem that there are advantages to James' patch all by itself, in that it will will work even if the debuginfo information for the ext4dev module can't be found, since the kallsyms information would be used instead.
Quoted 6 lines
As a quality-of-implementation matter, systemtap checks at translation time that such probes make sense -- that "
ext4_fill_super" even exists. (That is needed also to expand wildcards.) The only way it can do that is if it has dwarf or separate textual symbol table data (see above). Both of those carry addresses as well, so we might as well use them.
True, though I'll note for modern kernels, with /proc/kallsyms, we should hopefully be able to do this (offset=0 probes) without DWARF headers. BTW, one of the things which I have wondered is whether DWARF was really the right approach after all, given how bloated and space-inefficient it seems to be. Needing to keep 600 megs of module debuginfo for each kernel that I build (and yes, I build a fairly complete module-rich kernel, but having 59 megs of modules expand into 606 megs of debuginfo files is more than a little bit obnoxious.)
Maybe the kernel could be one of the places where we experiment with using something like CTF. Apparently CTF is compact enough that the Solaris folks are willing to keep the CTF information for the currently booted kernel in unswappable kernel memory --- which I know is one of those things we wouldn't want to do with DWARF information!
Quoted 6 lines
I have seen little sympathy expressed for either of these concerns, which means that the new code would primarily allay some offense (but not constitute a bug fix or "usability for kernel developers" matter), and leave us to pick up the pieces. We can't make a habit out of that sort of thing, but maybe as a one-off in the interest of mutual goodwill, we should work out a way to get it done.
This may have been a communication disconnect. I suspect if the answer was, "please make the following changes and we will accept the patch", that James may have been willing to make the changes --- especially if it is for the sake of backwards compatibility with older kernels. Most kernel developers do care very much about backwards compatibility between userspace and older kernel versions. (Some of the folks who weren't as careful have been flamed pretty heavily about that; I'm still not a great fan of the sysfs interface from a userspace usability perspective, but we haven't had a major breakage there in quite a while.)
So there is a big difference between "please do X, Y, and Z first and the patch would then be acceptable" versus "for reasons A, B and C this patch is totally unacceptable and in fact what you are trying to do is pointless".
Regards,
- Ted
Quoted 32 lines · Theodore Tso
Quoted 3 lines · Frank Ch. Eigler
I also proposed a compromise where systemtap would use the symbol+offset interface, but choose a single convenient symbol as base for all probes in a particular elf file (/section).
I guess I don't see the value of that over just using the address directly. James' point wasn't just to use the symbol+offset feature just for the sake of using it, but rather as a better way of specifying how to insert a probe into a kernel. For example, it may be that by allowing the kernel to have a bit more semantic knowledge of where a probe is going, it could more easily do various safety-related checks that can't be done if all it is given is a a raw address.
Quoted 11 lines
Quoted 10 lines
probe module("ext4dev").function("ext4_fill_super")
{ printk("here am I!\n"); }This should be done via passing a hard-coded address, or via using the kprobes function+offset facility? It would seem that there are advantages to James' patch all by itself, in that it will will work even if the debuginfo information for the ext4dev module can't be found, since the kallsyms information would be used instead.
Quoted 6 lines
As a quality-of-implementation matter, systemtap checks at translation time that such probes make sense -- that "
ext4_fill_super" even exists. (That is needed also to expand wildcards.) The only way it can do that is if it has dwarf or separate textual symbol table data (see above). Both of those carry addresses as well, so we might as well use them.
Quoted 8 lines
True, though I'll note for modern kernels, with
/proc/kallsyms, we should hopefully be able to do this (offset=0 probes) without DWARF headers. BTW, one of the things which I have wondered is whether DWARF was really the right approach after all, given how bloated and space-inefficient it seems to be. Needing to keep 600 megs of module debuginfo for each kernel that I build (and yes, I build a fairly complete module-rich kernel, but having 59 megs of modules expand into 606 megs of debuginfo files is more than a little bit obnoxious.)
Yes ... the Sun solution to this was something called CTF which is basically compressed dwarf with duplicates eliminated (as you say below). We could go a long way with simple duplicate elimination in dwarf. For instance dwarf tends to deposit a definition of a structure wherever it's used. For popular structures (like struct task) this can lead to hundreds of duplicates in the debug info.
However, as has been rightly pointed out, most of what people want is simple entry and exit notification on functions ... this we can do with kprobes without any dwarf at all (so it's something systemtap should be able to do). In theory, we can also get at the actual function call arguments without resorting to dwarf, although that requires someone actually knowing the order and type (without dwarf we can't cross check unless we do something like parse the kernel headers).
Where it gets tricky is if you want access to variables in the middle of the routine. This is very useful to kernel developers and for static tracepoints. Note that the current markers project has a solution to this which could be done dwarfless ... the problem is that it has to perturb the optimiser at that point to spill the variables of interest to locations we know, so the price we pay is zero impact when disabled. The approach I was taking more or less relies on dwarf (or CTF) to be zero impact because the variable could be aynwhere ... including in a register or temporary stack location.
Quoted 5 lines
Maybe the kernel could be one of the places where we experiment with using something like CTF. Apparently CTF is compact enough that the Solaris folks are willing to keep the CTF information for the currently booted kernel in unswappable kernel memory --- which I know is one of those things we wouldn't want to do with DWARF information!
I'm not sure we'd want to do it with CTF either ... the only reason to have it in the kernel is if the kernel wants to use it. Right at the moment with the dwarf, systemtap userspace is the only consumer, so it doesn't need to be in the kernel; I don't really see that changing with CTF. I think dtrace does it because the D runtime is in-kernel so some of the run time typing needs the CTF.
James
Quoted 6 lines · James Bottomley
I'm not sure we'd want to do it with CTF either ... the only reason to have it in the kernel is if the kernel wants to use it. Right at the moment with the dwarf, systemtap userspace is the only consumer, so it doesn't need to be in the kernel; I don't really see that changing with CTF. I think dtrace does it because the D runtime is in-kernel so some of the run time typing needs the CTF.
As far as I understand things, all of that is done in userspace and byte-compiled down to run in their simple DTrace VM. The reason for CTF being in the Open Slaris is for their kernel debugger (kmdb) and so that a crash dump can be guaranteed to be easily analyzed.[1]
- Ted