Skip to contentSkip to conversation

emit logging when a process receives a fatal signal

PATCH In review Folkert van Heusden 5 participants 14 messages Nov 18, 2006 – Nov 21, 2006

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 emit logging when a process receives a fatal signal

Hi,

I found that sometimes processes disappear on some heavily used system of mine without any logging. So I've written a patch against 2.6.18.2 which emits logging when a process emits a fatal signal.

kernel/signal.c +9 -0
@@ -706,6 +706,15 @@
 	struct sigqueue * q = NULL;
 	int ret = 0;
 
+	if (sig == SIGQUIT || sig == SIGILL  || sig == SIGTRAP ||
+	    sig == SIGABRT || sig == SIGBUS  || sig == SIGFPE  ||
+	    sig == SIGSEGV || sig == SIGXCPU || sig == SIGXFSZ ||
+	    sig == SIGSYS  || sig == SIGSTKFLT)
+	{
+		printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
+			sig, t -> pid, t -> uid, t -> gid, t -> comm);
+	}
+
 	/*
 	 * fast-pathed signals for kernel-internal things like SIGSTOP
 	 * or SIGKILL.

Folkert van Heusden

www.vanheusden.com/multitail - multitail is tail on steroids. multiple
windows, filtering, coloring, anything you can think of
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

#1 lore raw
Quoted 5 lines · Folkert van Heusden

Hi,

I found that sometimes processes disappear on some heavily used system of mine without any logging. So I've written a patch against 2.6.18.2 which emits logging when a process emits a fatal signal.

Why not to patch default signal handlers in glibc, to have not only stderr, but syslog, or /dev/kmsg copy of fatal messages?

Quoted 28 lines
kernel/signal.c +9 -0
@@ -706,6 +706,15 @@
 	struct sigqueue * q = NULL;
 	int ret = 0;
 
+	if (sig == SIGQUIT || sig == SIGILL  || sig == SIGTRAP ||
+	    sig == SIGABRT || sig == SIGBUS  || sig == SIGFPE  ||
+	    sig == SIGSEGV || sig == SIGXCPU || sig == SIGXFSZ ||
+	    sig == SIGSYS  || sig == SIGSTKFLT)
+	{
+		printk(KERN_WARNING "Sig %d send to %d owned by %d.%d (%s)\n",
+			sig, t -> pid, t -> uid, t -> gid, t -> comm);
+	}
+
 	/*
 	 * fast-pathed signals for kernel-internal things like SIGSTOP
 	 * or SIGKILL.

Folkert van Heusden

www.vanheusden.com/multitail - multitail is tail on steroids. multiple
windows, filtering, coloring, anything you can think of
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com

#2 lore raw

Hi,

Quoted 5 lines
Quoted 3 lines

I found that sometimes processes disappear on some heavily used system of mine without any logging. So I've written a patch against 2.6.18.2 which emits logging when a process emits a fatal signal.

Why not to patch default signal handlers in glibc, to have not only stderr, but syslog, or /dev/kmsg copy of fatal messages?

Afaik when a proces gets shot because of a segfault, also the libraries it used are shot so to say. iirc some of the more fatal signals are handled directly by the kernel.

Folkert van Heusden

Signature · 3 lines
www.biglumber.com <- site where one can exchange PGP key signatures 
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com
#3 lore raw
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I found that sometimes processes disappear on some heavily used system of mine without any logging. So I've written a patch against 2.6.18.2 which emits logging when a process emits a fatal signal.

Why not to patch default signal handlers in glibc, to have not only stderr, but syslog, or /dev/kmsg copy of fatal messages?

Afaik when a proces gets shot because of a segfault, also the libraries it used are shot so to say. iirc some of the more fatal signals are handled directly by the kernel.

Also: what about statically build programs?

Folkert van Heusden

Signature · 3 lines
Feeling generous? -> http://www.vanheusden.com/wishlist.php
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com
#4 lore raw
Quoted 8 lines · Folkert van Heusden
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I found that sometimes processes disappear on some heavily used system of mine without any logging. So I've written a patch against 2.6.18.2 which emits logging when a process emits a fatal signal.

Why not to patch default signal handlers in glibc, to have not only stderr, but syslog, or /dev/kmsg copy of fatal messages?

Afaik when a proces gets shot because of a segfault, also the libraries it used are shot so to say. iirc some of the more fatal signals are handled directly by the kernel.

Kernel sends signals, no doubt.

Then, who you think prints that "Killed" or "Segmentation fault" messages in *stderr*?
[Hint: libc's default signal handler (man 2 signal).]

Quoted 1 line

Also: what about statically build programs?

"-lc" embeds libc in static binary, no?

IMHO it's not a lkml issue. Here are many who would say you, that userspace preblems are userspace problems.
____

#5 lore raw

In article <[email protected]>,

Quoted 10 lines · Oleg Verych
Quoted 8 lines · Folkert van Heusden
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I found that sometimes processes disappear on some heavily used system of mine without any logging. So I've written a patch against 2.6.18.2 which emits logging when a process emits a fatal signal.

Why not to patch default signal handlers in glibc, to have not only stderr, but syslog, or /dev/kmsg copy of fatal messages?

Afaik when a proces gets shot because of a segfault, also the libraries it used are shot so to say. iirc some of the more fatal signals are handled directly by the kernel.

Quoted 5 lines

Kernel sends signals, no doubt.

Then, who you think prints that "Killed" or "Segmentation fault" messages in *stderr*?
[Hint: libc's default signal handler (man 2 signal).]

There is no such thing as a "libc default signal handler". [Hint: waitpid (man 2 waitpid).]

Mike.

#6 lore raw

Nice to meet you, Miquel!

Quoted 12 lines · Miquel van Smoorenburg

In article <[email protected]>,

Quoted 10 lines · Oleg Verych
Quoted 8 lines · Folkert van Heusden
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I found that sometimes processes disappear on some heavily used system of mine without any logging. So I've written a patch against 2.6.18.2 which emits logging when a process emits a fatal signal.

Why not to patch default signal handlers in glibc, to have not only stderr, but syslog, or /dev/kmsg copy of fatal messages?

Afaik when a proces gets shot because of a segfault, also the libraries it used are shot so to say. iirc some of the more fatal signals are handled directly by the kernel.

Quoted 7 lines
Quoted 5 lines

Kernel sends signals, no doubt.

Then, who you think prints that "Killed" or "Segmentation fault" messages in *stderr*?
[Hint: libc's default signal handler (man 2 signal).]

There is no such thing as a "libc default signal handler".

By that i mean SIG_DFL, even if that means signal masks, shell/debuger/tracer/lib/whatever installed *actual* functions.

Maybe there isn't one for actual patching (if someone really wants to patch something ;). One may add, just like in libSegFault.so. There are many in-userspace solutions, that problem isn't kernel's one.

Quoted 1 line

[Hint: waitpid (man 2 waitpid).]

Thanks.

--

-o--=O`C  info emacs : not found  /. .\ (is there any reason to live?)
 #oo'L O  info make  : not found      o (           R.I.P            )
<___=E M  man gcc    : not found    .-- (  Debian Operating System   )
#7 lore raw
In reply to Oleg Verych
Quoted 10 lines · Oleg Verych
Quoted 8 lines · Folkert van Heusden
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I found that sometimes processes disappear on some heavily used system of mine without any logging. So I've written a patch against 2.6.18.2 which emits logging when a process emits a fatal signal.

Why not to patch default signal handlers in glibc, to have not only stderr, but syslog, or /dev/kmsg copy of fatal messages?

Afaik when a proces gets shot because of a segfault, also the libraries it used are shot so to say. iirc some of the more fatal signals are handled directly by the kernel.

Quoted 5 lines

Kernel sends signals, no doubt.

Then, who you think prints that "Killed" or "Segmentation fault" messages in *stderr*?
[Hint: libc's default signal handler (man 2 signal).]

Please enlighten us on how you plan to catch the uncatchable SIGKILL.

	-`J'
#8 lore raw
Quoted 12 lines · Jan Engelhardt
Quoted 10 lines · Oleg Verych
Quoted 8 lines · Folkert van Heusden
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I found that sometimes processes disappear on some heavily used system of mine without any logging. So I've written a patch against 2.6.18.2 which emits logging when a process emits a fatal signal.

Why not to patch default signal handlers in glibc, to have not only stderr, but syslog, or /dev/kmsg copy of fatal messages?

Afaik when a proces gets shot because of a segfault, also the libraries it used are shot so to say. iirc some of the more fatal signals are handled directly by the kernel.

Quoted 8 lines
Quoted 5 lines

Kernel sends signals, no doubt.

Then, who you think prints that "Killed" or "Segmentation fault" messages in *stderr*?
[Hint: libc's default signal handler (man 2 signal).]

Please enlighten us on how you plan to catch the uncatchable SIGKILL.

Here's question of getting information. Collecting information is possible by `waitpid()' from parent process as Miquel noted.

That man above, gave me impression, that SIG_DFL can not be changed in case of KILL and STOP signals, what yields to "The signals SIGKILL and SIGSTOP cannot be caught or ignored." Implementation of such no-action can be different. In case if kernel just stops processing of task with STOP, breaks with KILL, without giving a chance to flush any pending data OK, if this is an assembler prorgam with just data segment and no infrastructure at all. But i think (didn't read anything), it is bad, if there's libc with standard stream I/O buffers and no callback is possible.

Quoted 3 lines
	-`J'
#9 lore raw
Quoted 9 lines · Oleg Verych
Quoted 5 lines · Jan Engelhardt
Quoted 3 lines

Then, who you think prints that "Killed" or "Segmentation fault" messages in *stderr*?
[Hint: libc's default signal handler (man 2 signal).]

Please enlighten us on how you plan to catch the uncatchable SIGKILL.

Here's question of getting information. Collecting information is possible by `waitpid()' from parent process as Miquel noted.

Yes, that is true. However that would involve adding support for This Situation to the parent process. Which is where it becomes tricky. Patch /sbin/init, in case the daemon runs like everything else. Or patch xinetd, in case it is run from within that. Or, ...
The 'problem' with the waitpid solution is that you would need to
patch every possible parent that may become the owner of The Sigkilled Target.

Quoted 8 lines

That man above, gave me impression, that SIG_DFL can not be changed in case of KILL and STOP signals, what yields to "The signals SIGKILL and SIGSTOP cannot be caught or ignored." Implementation of such no-action can be different. In case if kernel just stops processing of task with STOP, breaks with KILL, without giving a chance to flush any pending data OK, if this is an assembler prorgam with just data segment and no infrastructure at all. But i think (didn't read anything), it is bad, if there's libc with standard stream I/O buffers and no callback is possible.

	-`J'
#10 lore raw
Quoted 12 lines · Jan Engelhardt
Quoted 9 lines · Oleg Verych
Quoted 5 lines · Jan Engelhardt
Quoted 3 lines

Then, who you think prints that "Killed" or "Segmentation fault" messages in *stderr*?
[Hint: libc's default signal handler (man 2 signal).]

Please enlighten us on how you plan to catch the uncatchable SIGKILL.

Here's question of getting information. Collecting information is possible by `waitpid()' from parent process as Miquel noted.

Quoted 7 lines

Yes, that is true. However that would involve adding support for This Situation to the parent process. Which is where it becomes tricky. Patch /sbin/init, in case the daemon runs like everything else. Or patch xinetd, in case it is run from within that. Or, ...
The 'problem' with the waitpid solution is that you would need to
patch every possible parent that may become the owner of The Sigkilled Target.

I think this is pure userspace admin issue, one wrapper shell script for not programmers.

I'm not sure about init, you've told. For example, in Debian daemons are run by start-stop-daemon function in LSB package. And in proposed LSB standard <http://refspecs.freestandards.org/LSB_3.1.0/LSB-Core-generic/>
portable start_daemon, killproc, pidofproc funcions are described.
____

#11 lore raw
Quoted 20 lines · Oleg Verych
Quoted 11 lines · Jan Engelhardt
Quoted 9 lines · Oleg Verych
Quoted 5 lines · Jan Engelhardt
Quoted 3 lines

Then, who you think prints that "Killed" or "Segmentation fault" messages in *stderr*?
[Hint: libc's default signal handler (man 2 signal).]

Please enlighten us on how you plan to catch the uncatchable SIGKILL.

Here's question of getting information. Collecting information is possible by `waitpid()' from parent process as Miquel noted.

Quoted 7 lines

Yes, that is true. However that would involve adding support for This Situation to the parent process. Which is where it becomes tricky. Patch /sbin/init, in case the daemon runs like everything else. Or patch xinetd, in case it is run from within that. Or, ...
The 'problem' with the waitpid solution is that you would need to
patch every possible parent that may become the owner of The Sigkilled Target.

Quoted 7 lines

I think this is pure userspace admin issue, one wrapper shell script for not programmers.

I'm not sure about init, you've told. For example, in Debian daemons are run by start-stop-daemon function in LSB package. And in proposed LSB standard <http://refspecs.freestandards.org/LSB_3.1.0/LSB-Core-generic/>
portable start_daemon, killproc, pidofproc funcions are described.

But usually the start wrapper will succeed, and the daemon will eventually get reparented to init. At least this is the case in LSB-compliant distros like opensuse and fedora.

	-`J'
#12 lore raw
In reply to Folkert van Heusden

Folkert van Heusden <[email protected]> writes on Sat, 18 Nov 2006 03:02:
00 +0100
> Hi,
>
> > > I found that sometimes processes disappear on some heavily used system > > > of mine without any logging. So I've written a patch against 2.6.18.2 > > > which emits logging when a process emits a fatal signal. > > Why not to patch default signal handlers in glibc, to have not only > > stderr, but syslog, or /dev/kmsg copy of fatal messages?
>
> Afaik when a proces gets shot because of a segfault, also the libraries > it used are shot so to say. iirc some of the more fatal signals are > handled directly by the kernel.
>
>
> Folkert van Heusden

This is a user-space issue, not kernel (carrying this forward, we can say the "kernel should complain when programs have bugs").

Newer glibc has catchsegv (haven't found any documentation, but its interesting) --

: leisner@gateway 10:28:16;catchsegv sleep 1d
*** Segmentation fault
Register dump:

 EAX: fffffffc   EBX: bfc302e4   ECX: 00000000   EDX: b7f0c690
 ESI: bfc302e4   EDI: 00003800   EBP: bfc302f8   ESP: bfc302b8

EIP: 00c5a402 EFLAGS: 00200246

CS: 0073 DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b

Trap: 00000000 Error: 00000000 OldMask: 00000000
ESP/signal: bfc302b8 CR2: 00000000

FPUCW: ffff037f FPUSW: ffff4000 TAG: ffffffff
IPOFF: 080492e9 CSSEL: 0073 DATAOFF: bfc302bc DATASEL: 007b

 ST(0) 0000 0000000000000000   ST(1) 0000 0000000000000000
 ST(2) 0000 0000000000000000   ST(3) 0000 a8c0000000000000
 ST(4) 0000 0000000000000000   ST(5) 0000 0000000000000000
 ST(6) 0000 0000000000000000   ST(7) 0000 0000000000000000
Backtrace:

/lib/libSegFault.so[0x2e21ff]
??:0(??)[0xc5a420]
sleep[0x8048ddf]
/lib/libc.so.6(__libc_start_main+0xdc)[0xc8b7e4]
sleep[0x8048af1]

Memory map:

002e0000-002e3000 r-xp 00000000 08:15 91703 /lib/libSegFault.so 002e3000-002e4000 r-xp 00002000 08:15 91703 /lib/libSegFault.so 002e4000-002e5000 rwxp 00003000 08:15 91703 /lib/libSegFault.so 0036f000-0037a000 r-xp 00000000 08:15 244694 /lib/libgcc_s-4.1.0-20060304.so.1 0037a000-0037b000 rwxp 0000a000 08:15 244694 /lib/libgcc_s-4.1.0-20060304.so.1 003e9000-00402000 r-xp 00000000 08:15 183695 /lib/ld-2.4.so 00402000-00403000 r-xp 00018000 08:15 183695 /lib/ld-2.4.so 00403000-00404000 rwxp 00019000 08:15 183695 /lib/ld-2.4.so 00c5a000-00c5b000 r-xp 00c5a000 00:00 0 [vdso]
00c76000-00da2000 r-xp 00000000 08:15 244689 /lib/libc-2.4.so 00da2000-00da5000 r-xp 0012b000 08:15 244689 /lib/libc-2.4.so 00da5000-00da6000 rwxp 0012e000 08:15 244689 /lib/libc-2.4.so 00da6000-00da9000 rwxp 00da6000 00:00 0
08048000-0804b000 r-xp 00000000 08:18 1354883 /usr/local/gnu/coreutils-6.5/bin/sleep 0804b000-0804c000 rw-p 00003000 08:18 1354883 /usr/local/gnu/coreutils-6.5/bin/sleep 09c1b000-09c40000 rw-p 09c1b000 00:00 0 [heap]
b7f0c000-b7f0d000 rw-p b7f0c000 00:00 0
b7f31000-b7f32000 rw-p b7f31000 00:00 0
bfc1d000-bfc32000 rw-p bfc1d000 00:00 0 [stack]

Processes don't "disappear" -- the parent can track this...

Also, do you have core dumps turned on?

marty

#13 lore raw

On Mon, Nov 20, 2006 at 10:48:59PM -0500, Marty Leisner wrote:
[]

Quoted 5 lines
Quoted 1 line

This is a user-space issue, not kernel (carrying this forward, we can

say the "kernel should complain when programs have bugs").

Newer glibc has catchsegv (haven't found any documentation, but its interesting) --

It's just LD_PRELOAD (man ld.so) and libSegFault.so, which installs (among others) SIGSEGV handler, see "glibc/sysdeps/generic/segfault.c".
____