Extending coredump note section to contain filenames

Hi Roland, Jan,

While working with coredump analysis, it struck me how much PITA is caused merely by the fact that names of loaded binary and libraries are not known.

gdb retrieves loaded library names by examining dynamic loader's data stored in the coredump's data segments. It uses intimate knowledge how and where dynamic loader keeps the list of loaded libraries. (Meaning that it will break if non-standard loader is used).

And, as Jan explained to me, it depends on knowing
where the linked list of libs starts, which requires knowing binary which was running. IIRC there is no easy and reasonably foolproof way to determine binary's name. (Looking at argv[0] on stack is not reasonably foolproof).

Which is *ridiculous*. We *know* the list of mapped files at the coredump generation time. It can even be accessed as /proc/PID/maps.

I propose to save this information in coredump.

(For people not very familiar with coredump format:
coredumps have a NOTE segment which contains register values, PID of the crashed process, and other such info. It looks like this:

Program Headers:

  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  NOTE           0x000254 0x00000000 0x00000000 0x00814 0x00000     0
  LOAD           0x001000 0x00172000 0x00000000 0x01000 0x01000 R E 0x1000
...
Notes at offset 0x00000254 with length 0x00000814:
  Owner                 Data size       Description
  CORE                 0x00000090       NT_PRSTATUS (prstatus structure)
  CORE                 0x0000007c       NT_PRPSINFO (prpsinfo structure)
  CORE                 0x000000a0       NT_AUXV (auxiliary vector)
  CORE                 0x0000006c       NT_FPREGSET (floating point registers)
  LINUX                0x00000200       NT_PRXFPREG (user_xfpregs structure)
  LINUX                0x00000340       NT_X86_XSTATE (x86 XSAVE extended state)
  LINUX                0x00000030       Unknown note type: (0x00000200)

and I propose to add a new note to this segment)

Do you think such addition would be useful?

What format this note should have? Hmmm. How about this:

Elf_Word count // how many files are mapped
array of [count] elements of
Elf_Addr start
Elf_Addr end
Elf_Addr file_ofs
followed by filenames in ASCII: "FILE1" NUL "FILE2" NUL "FILE3" NUL...

The rationale for not saving some other attributes is that the list of all mapped files can be somewhat big. For example:

$ cat /proc/`pidof firefox`/maps | wc -c
41553

Thus, we probably may want to make it smaller.
We may save a bit by coalescing the adjacent mappings to the same file which only differ in attributes. Example from my firefox's /proc/pid/maps file:

b6fa0000-b6fb4000 r-xp 00000000 fd:01 671717 /usr/lib/xulrunner-2/libmozjs.so
b6fb4000-b6fb6000 ---p 00014000 fd:01 671717 /usr/lib/xulrunner-2/libmozjs.so

In fact these two mappings map contiguos area of the file to a contiguous area of memory, so in coredump we can represent it as one item:

start:b6fa0000 end:b6fb6000 ofs:00000000 name:/usr/lib/xulrunner-2/libmozjs.so

The information about memory attributes is present in coredump anyway, in program header, so it can be restored by coredump analysis tools.

Maybe we also would want to be able to compress filenames by saying "take N chars from previous name, then append this suffix". This means that instead of storing
"/usr/lib/xulrunner-2/libxul.so" NUL "/usr/lib/xulrunner-2/libxul.so" NUL "/usr/lib/xulrunner-2/libmozjs.so" NUL
we'd store

    "/usr/lib/xulrunner-2/libxul.so" NUL
    <30> NUL
    <24> "mozjs.so" NUL

But is it worth the pain in the coredump parsers?

Another question is detection of deleted files.
If /usr/lib/xulrunner-2/libmozjs.so was updated while program ran and now file mapped into process address space does not correspond to the same-named file on disk, can we help users to detect this? How? By saving maj/min/inode? Hash thereof?
File size?
File's md5sum (probably not, way too expensive. But nicely robust...)?

vda

Hi Denys,

this is the backward compatible way and it is no longer the right one with build-ids.

GDB should scan the address space for mapped build-ids and map symbol files accordingly. The shared library list may be even corrupted in crashes with memory overruns. But it has not been implemented yet, I plan it for years.

(It can use the shared library list as a hint as in some cases the symbol files could overlap if you load the first page of an ELF file etc., this is very improbable possibility.)

build-id is already being saved. This is all that matters. Filename does not say anything - as you noticed it can be even already deleted, it can have unknown content etc. I do not see what problems you target here.

Regards,
Jan

Build-ids are useful, but they still don't map directly to the names of loaded files. You need to rely on /usr/lib/debug/.build-id/XX/YYYYYYYYYY symlinks to translate build-ids to names.

For example, on my home machine (linux-from-scratch style) I don't have /usr/lib/debug/.build-id/* directory at all. So build-ids can't be used to find the binary and libraries there.

Why we don't save library names in coredump? I see no logical reason not to do so. Even if those names sometimes won't be reliable ("deleted files" problem), it's not a good reason to shoot ourself in the food and deprive ourself from this information 100% of the time.

Yes, the file can be deleted/updated-via-rename. That's the case I want to be possible to detect.

I don't understand. *What* can have unknown content?

I'm thinking whether we should supply some mechanism for detecting "deleted/updated file" problem. Even if this would be a heuristic. I'll be satisfied with 99.9999% success rate instead of 100% :)

vda

s/food/foot/ :)

There is draft of https://fedoraproject.org/wiki/Darkserver but it is not yet deployed. This will give you distro and file version / URL for the specified build-id. Without any local symlinks/files at all.

Reasons why it has not yet been finished + deployed I find out of the scope of this thread.

Because they are useless. If you have a filename there how to find which content it should match? Even if you verify the file is still there with the same content there is a race it can no longer be true when you read the core file 5 seconds later.

The build-id mapping server above always works and without races.

You will save there "/lib64/libc-2.14.90.so". But the next day you have no idea which compilation or build the core file was generated for, that virtual machine can be either already updated or even reinstalled from scratch etc. "/lib64/libc-2.14.90.so" does not say anything about the build.

I am not, I prefer 100% build-id server.

Regards,
Jan

They may be useless in some situations. Not in every situation, by a long shot. Here is a live example from my system:

$ ulimit -c unlimited
$ md5sum </dev/zero &
$ pid=$!
$ sleep 1
$ kill -ABRT $pid
$ gdb -ex "core core.12977"
GNU gdb (GDB) Fedora (7.3.50.20110722-10.fc16)
...
Missing separate debuginfo for the main executable file
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/ec/1fd70dbee0db36eff9527254d9d2bbfd260f13
[New LWP 12977]
Core was generated by `md5sum'.
Program terminated with signal 6, Aborted.
#0  0x0804b2b0 in ?? ()
(gdb) bt
#0  0x0804b2b0 in ?? ()
Backtrace stopped: Not enough registers or memory available to unwind further

No backtrace at all.
Let's tell it which binary was that:

(gdb) file /usr/bin/md5sum
Reading symbols from /usr/bin/md5sum...(no debugging symbols found)...done. Missing separate debuginfos, use: debuginfo-install coreutils-8.12-6.fc16.i686 (gdb) bt

#0  0x0804b2b0 in ?? ()
#1  0x0804bdd8 in ?? ()
#2  0x0804a093 in ?? ()
#3  0x08049659 in ?? ()
#4  0xb760f6b3 in ?? ()

Backtrace stopped: Not enough registers or memory available to unwind further

This is better, isn't it?
Wouldn't it be nice if gdb would retrieve binary's name by itself? (BTW: nothing prevents it from checking build ids and refusing to use it if they don't match.)

And maybe root will run "rm -rf /*" in parallel. By this logic, we should just give up on using computers.

But it is not always available. Some people don't want to be connected to internet; other can't be connected.

Does it follow from the above that filenames are *never* useful? I don't think so.

vda

Yes, that yum should have been executed automatically, instead of just suggested by that:

	Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/ec/1fd70dbee0db36eff9527254d9d2bbfd260f13

But this is a user interface issue, it was discussed with PackageKit people etc. but it went nowhere.

Sorry but I cannot code the whole OS myself including all the Gnome UI interfaces.

And if they do not match it will need to run the yum command above anyway. So why to code two ways where the first one (by filename) works only sometimes while the second way works always? Isn't it easier to do it always just the second way?

That 'yum' command above will run in some conditions without any Internet connectivity. But in some cases it will have more bandwidth requirements than a build-id server query.

This is about package management vs. network servers connectivity, this is also partially distro dependent.

They can be sometimes useful but they are superseded by build-ids; with build-ids they can be safely ignored.

Regards,
Jan

My network is down. "yum install" won't work.

My network is down. "yum install" won't work.

I just gave you an example where filename is useful.

vda

You need to fix yum first on each distro so that it works like apt-get does:
echo 'metadata_expire=never' >>/etc/yum.conf
echo "yum --enablerepo='*' makecache" >/etc/cron.daily/yumupdate chmod +x /etc/cron.daily/yumupdate

Then all the examples I have given work fine.

Regards,
Jan

Now I think you mean that you want to just find /usr/bin/md5sum without ever using /usr/lib/debug/usr/bin/md5sum.debug, right?

Then you can create your local build-id -> filename database specific to your machine in some /var/lib/build-id/ . Such database has nothing to do with the core file itself. Core file can be easily transferred to other hosts as it is machine independent. Core file is dependent only on the specific binaries _content_ (not _names_) which is uniquely identified by their build-id.

/var/lib/build-id/ can be created/updated either during 'rpm -i' time or from '/proc/sys/kernel/core_pattern' hook or via some inotify daemon watching /bin, /usr/bin and other directories updates etc. etc.

Regards,
Jan

It seems to me that there would be value in having *both*.

In particular, for libraries which aren't installed in standard system directories (because they are test versions or mapped with dlopen()) it would be good to have a hint of where to find them. The build-id can then tell you if you have the right version.

-hpa

I believe "/proc/sys/kernel/core_pattern" was created so that Linux kernel does not have to contain code dumping of many associated info, the filenames seem to me to belong in this category.

Regards,
Jan

That's quite absurd if you think about it... we're talking about each individual library, not necessarily the root binary.

-hpa

H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Denys is also working on ABRT which already dumps a lot of useful info associated with a core file:
abrt_version analyzer architecture cmdline component coredump count dso_list environ executable hostname kernel maps os_release package pid pwd reason time uid username uuid var_log_messages

I find the local copy instances of the files of specific build the same or even less less useful info than those files above.

One should start implementing dumping all the info above info Linux kernel before the filenames are worth it.

The problem with filenames is they work in the most common end-user host cases. Which means tools get built on top of it that way - like what AFAIK happened in the Ubuntu Apport case. The resulting toolchain is then not applicable for the longterm running server setups, it is that 99% working solution which needs to be rewritten from scratch for that final 1%.

Regards,
Jan

This is basically providing the dso_list and executable information. Several of the other things you list above (cmdline, environ, pid, reason, maps) are already part of a core file. The rest sounds like an RFE to me.

-hpa

dso_list and executable identification (their build-id) are also part of the core file. It is only filesystem defect it can map filename -> content but it cannot map build-id -> content. There are many indexing solutions for it, including slocate, when you have started to talk about not 100% reliable solutions.

If you really mean all that info does belong to a core file then I no longer have any objections, Linux kernel can put a lot of redundant info into the core file.

Regards,
Jan

There is no 100% reliable solution possible -- you have no guarantee of any kind that the library executable still exists.

-hpa

H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

I have guarantee that the library binary mapped in memory identified by build-id can be found out there in the could. There is no other guarantee. And this guarantee fails with other solutions.

If you say that it is _additional_ info to build-id then yes, one can always use build-id if everything else fails. But then the non-build-id information is redundant and it can just lead to wrong toolchain solutions - which has already happened (Apport).

Regards,
Jan

You're thinking of a particular use case which isn't necessarily the only one that matters. It might be the only one that matters to *you*, but that's very different to what matters to a developer, for example.

And yes of course the build-id should be included.

-hpa

H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Another example. abrt project wants to do better duplicate detection for coredumps, where duplicate is defined as "this crash is probably caused by the same bug".

This needs to be done quickly. Downloading and installing debuginfos are definitely out of the question.

Karel decided to do it by walking the stack and creating a simplified backtrace of this form:

BUILD_ID OFFSET SYMBOL MODNAME

where
BUILD_ID: build id of the binary file the address is mapped to.
OFFSET: offset from the start of the executable section of the file
the stored instruction pointer points to.
SYMBOL: name of the function if it is known.
MODNAME: name of the binary or library.

We wrote a small standalone tool which generates such trace. The code we have now frequently generates something like this:

ec1fd70dbee0db36eff9527254d9d2bbfd260f13 0x34a7 close_stdout [exe] ec1fd70dbee0db36eff9527254d9d2bbfd260f13 0x3dd8 close_stdout [exe] ec1fd70dbee0db36eff9527254d9d2bbfd260f13 0x2093 - [exe] ec1fd70dbee0db36eff9527254d9d2bbfd260f13 0x1659 - [exe]

Naturally, "[exe]" is there because we don't know executable names. I have a new code which uses saved /proc/pid/maps and it generates:

ec1fd70dbee0db36eff9527254d9d2bbfd260f13 0x34a7 close_stdout /usr/bin/md5sum ec1fd70dbee0db36eff9527254d9d2bbfd260f13 0x3dd8 close_stdout /usr/bin/md5sum ec1fd70dbee0db36eff9527254d9d2bbfd260f13 0x2093 - /usr/bin/md5sum ec1fd70dbee0db36eff9527254d9d2bbfd260f13 0x1659 - /usr/bin/md5sum

If anyone would want to use this tool on a large collection of coredumps with the intent of fishing out similar crashes (this is not a theoretical assumption, we *do* have people who badly need this feature), they won't get nice names of binaries, they'll get non-informative "[exe]" things - because /proc/pid/maps info is not saved in coredump, and won't be available. This will impair ability to perform crash similarity analysis.

vda

In such moment it is already on public Internet and the build-id -> name mapping database is available.

Or it rather would be available if the Darkserver would be finished. So far I have only always seen every project for build-id targeted at getting rid of the build-id and map it back to the ambiguous filenames / package names.

In such case there could filename / package name already in the ELF header. build-id should be there as a feature, not as a complication.

If someone does 'cp /usr/bin/md5sum ~/bin/mysum' then you either get unidentifiable names or even misleading names.

Regards,
Jan