[2.6.27.24] Kernel coredump to a pipe is failing
Subsystems (2)
Sorry for not following up to my previous message to get the threading headers correct: the original is on another system I don't have access to and I can't find a way to reply from any of the web archived versions. Anyway, this is a link to the original FYI:
http://marc.info/?l=linux-kernel&m=124299629611706
In that post I observed that my short cores were being caused by dump_write() returning 0; taking another look at dump_write():
static int dump_write(struct file *file, const void *addr, int nr)
{
return file->f_op->write(file, addr, nr, &file->f_pos) == nr;
}If the write operation returns an error of any kind, or it fails to write the complete set of bytes (nr here is always PAGE_SIZE, as this function is called when it fails), then dump_write() returns 0 and we get a short core.
So I annotated dump_write() to printk() if this operation is false, and I get:
file ffff8803b95d0180: dump_write: -512 < 4096
Well, -512 is ERESTARTSYS. That, to me, seems like a reasonable error code to get when we're trying to dump core to a pipe. Yes? No?
Shouldn't we be doing some kind of error handling here, at least for basic things like signals? Should a process that's dumping core be set to ignore signals? Should dump_write() try again on ERESTARTSYS?
Any advice or comments would be greatly appreciated!
Quoted 15 lines · Paul Smith
Sorry for not following up to my previous message to get the threading headers correct: the original is on another system I don't have access to and I can't find a way to reply from any of the web archived versions. Anyway, this is a link to the original FYI:
http://marc.info/?l=linux-kernel&m=124299629611706
In that post I observed that my short cores were being caused by
dump_write()returning 0; taking another look atdump_write():static int dump_write(struct file *file, const void *addr, int nr) { return file->f_op->write(file, addr, nr, &file->f_pos) == nr; }
Quoted 4 lines
If the write operation returns an error of any kind, or it fails to write the complete set of bytes (nr here is always
PAGE_SIZE, as this function is called when it fails), thendump_write()returns 0 and we get a short core.
I have verified that fs/pipe.c:pipe_write() is returning ERESTARTSYS as a result of this code:
if (signal_pending(current)) {
if (!ret)
ret = -ERESTARTSYS;
break;
}I'm not exactly sure where to go from here, without knowing what the design SHOULD be for signals that are received during core dumps.
Quoted 8 lines · Paul Smith
So I annotated
dump_write()toprintk()if this operation is false, and I get:file ffff8803b95d0180: dump_write: -512 < 4096Well, -512 is ERESTARTSYS. That, to me, seems like a reasonable error code to get when we're trying to dump core to a pipe. Yes? No?
Which signal is it? SIGPIPE?
Quoted 4 lines
Shouldn't we be doing some kind of error handling here, at least for basic things like signals? Should a process that's dumping core be set to ignore signals? Should
dump_write()try again on ERESTARTSYS?
I think it should block signals. Here's a untested patch.
It has the disadvantage that it reports the incorrect blocked mask in the ELF corefile, but that's probably better than truncated coredumps.
-Andi
---
Block signals during core dump
When a signal happens during core dump the core dump to a pipe can fail, because the write returns short, but the ELF core dumpers cannot handle that.
There's no reason to handle signals during core dumping, so just block them all.
Open issue: ELF puts blocked signals into the core dump and that will be always fully blocked now. Need to save it somewhere?
Based on debugging by Paul Smith.
--- fs/exec.c | 6 ++++++ 1 file changed, 6 insertions(+)
fs/exec.c +6 -0
@@ -1760,6 +1760,12 @@ goto fail; } +/* block all signals */ +spin_lock_irq (¤t->sighand->siglock); +sigfillset (¤t->blocked); +/* No recalc sigpending */ +spin_unlock_irq (¤t->sighand->siglock); +down_write (&mm->mmap_sem);/* * If another thread got here first, or we are not dumpable, bail out.
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 5 lines · Andi Kleen
Quoted 2 lines · Paul Smith
Well, -512 is ERESTARTSYS. That, to me, seems like a reasonable error code to get when we're trying to dump core to a pipe. Yes? No?
Which signal is it? SIGPIPE?
I'm not sure; I'll have to dig in a little further. I'm not sure offhand how to determine which signal it was from inside the kernel but it shouldn't be hard to find.
Quoted 10 lines
Quoted 4 lines
Shouldn't we be doing some kind of error handling here, at least for basic things like signals? Should a process that's dumping core be set to ignore signals? Should
dump_write()try again on ERESTARTSYS?I think it should block signals. Here's a untested patch.
It has the disadvantage that it reports the incorrect blocked mask in the ELF corefile, but that's probably better than truncated coredumps.
As a quick test I changed dump_write() to retry on ERESTARTSYS after disabling the pending signal, like:
static int dump_write(struct file *file, const void *addr, int nr)
{
while (1) {
int r = file->f_op->write(file, addr, nr, &file->f_pos);
if (r != -ERESTARTSYS)
return r == nr;
/* We don't handle signals while dumping core. */
clear_thread_flag(TIF_SIGPENDING);
}
}I don't know if this is right, but in some quick tests I ran it did
work: my cores were full size. I haven't finished testing (and I have to go to soccer practice right now).
This obviously doesn't reset the signal mask in the dumping process, but it makes the dump_write() more complex and it may cause other issues so I can't say whether this is the way to go.
Quoted 2 lines
-
Block signals during core dump
Cool, I'll test this one as well.
Quoted 9 lines · On Tue, 26 May 2009 22:31:41 +0200 Andi Kleen <[email protected]> wrote:
Block signals during core dump
When a signal happens during core dump the core dump to a pipe can fail, because the write returns short, but the ELF core dumpers cannot handle that.
There's no reason to handle signals during core dumping, so just block them all.
Quoted 6 lines
Open issue: ELF puts blocked signals into the core dump and that will be always fully blocked now. Need to save it somewhere?
Based on debugging by Paul Smith.
Signed-off-by Andi Kleen <[email protected]>
dump_write() doesn't seem right, either. If ->write() returns, say, 100 then the dump should keep on going. At present it treats this return as an error.
I wonder why the signal problem has just turned up now - did we change the pipe code or something?
Quoted 3 lines · Andrew Morton
dump_write()doesn't seem right, either. If ->write() returns, say, 100 then the dump should keep on going. At present it treats this return as an error.
I think that's correct actually. Short write typically means serious issue like disk full or broken pipe, so stopping is good.
Quoted 2 lines
I wonder why the signal problem has just turned up now - did we change the pipe code or something?
No idea.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 7 lines · On Wed, 27 May 2009 01:14:28 +0200 Andi Kleen <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
dump_write()doesn't seem right, either. If ->write() returns, say, 100 then the dump should keep on going. At present it treats this return as an error.I think that's correct actually. Short write typically means serious issue like disk full or broken pipe, so stopping is good.
But we shouldn't assume that. It could be that the ->write implementation is perfectly able to absorb the remaining data.
We should only error out of the write() returned zero or -EFOO. The current code is simply buggy, but got lucky.
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Wed, 27 May 2009 01:14:28 +0200 Andi Kleen <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
dump_write()doesn't seem right, either. If ->write() returns, say, 100 then the dump should keep on going. At present it treats this return as an error.
Quoted 5 lines
Quoted 2 lines
I think that's correct actually. Short write typically means serious issue like disk full or broken pipe, so stopping is good.
But we shouldn't assume that. It could be that the ->write implementation is perfectly able to absorb the remaining data.
Maybe in theory, but in practice that's unlikely isn't it? Disk is full or pipe is blocking etc.
Quoted 2 lines
We should only error out of the
write()returned zero or-EFOO. The current code is simply buggy, but got lucky.
Maybe very pedantically, but I would argue that most programs don't do what you're saying (retry on any short write) and it's actually not very nice to always write a loop for each write.
Also any IO device who relies on that would likely find that it won't work with a lot of software.
So I think the current behaviour is ok, just need to get rid of the signals.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 9 lines · On Wed, 27 May 2009 01:41:09 +0200 Andi Kleen <[email protected]> wrote:
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Wed, 27 May 2009 01:14:28 +0200 Andi Kleen <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
dump_write()doesn't seem right, either. If ->write() returns, say, 100 then the dump should keep on going. At present it treats this return as an error.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I think that's correct actually. Short write typically means serious issue like disk full or broken pipe, so stopping is good.
But we shouldn't assume that. It could be that the ->write implementation is perfectly able to absorb the remaining data.
Maybe in theory, but in practice that's unlikely isn't it?
I dunno. Is this true of all linux filesystems in all cases? Maybe.
Quoted 9 lines
Disk is full or pipe is blocking etc.
Quoted 2 lines
We should only error out of the
write()returned zero or-EFOO. The current code is simply buggy, but got lucky.Maybe very pedantically, but I would argue that most programs don't do what you're saying (retry on any short write) and it's actually not very nice to always write a loop for each write.
Quoted 5 lines
Also any IO device who relies on that would likely find that it won't work with a lot of software.
So I think the current behaviour is ok, just need to get rid of the signals.
It's buggy!
Quoted 1 line
I dunno. Is this true of all linux filesystems in all cases? Maybe.
Assuming one of them is not would you rather want to fix that file system or 10 zillion user programs (including the kernel core dumper) that get it wrong? @)
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 6 lines · On Wed, 27 May 2009 02:11:04 +0200 Andi Kleen <[email protected]> wrote:
Quoted 1 line
I dunno. Is this true of all linux filesystems in all cases? Maybe.
Assuming one of them is not would you rather want to fix that file system or 10 zillion user programs (including the kernel core dumper) that get it wrong? @)
I think that removing one bug is better than adding one.
Many filesystems will return a short write if they hit a memory allocation failure, for example. pipe_write() sure will. Retrying is appropriate in such a case.
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Wed, 27 May 2009 02:11:04 +0200 Andi Kleen <[email protected]> wrote:
Quoted 1 line
I dunno. Is this true of all linux filesystems in all cases? Maybe.
Assuming one of them is not would you rather want to fix that file system or 10 zillion user programs (including the kernel core dumper) that get it wrong? @)
Quoted 5 lines
I think that removing one bug is better than adding one.
Many filesystems will return a short write if they hit a memory allocation failure, for example.
pipe_write()sure will. Retrying is appropriate in such a case.
As a mainly userspace guy maybe I'm missing some details for kernel behavior, but I know I would never write a program that used write(2) and assumed it would never return a short write. The documentation for write(2) is very clear that short writes are possible and any reasonably robust program will handle this. Consider things like NFS filesystems, etc. where who knows what behavior is found.
I'm more concerned with the loss of the signal mask settings in the core dump in Andi's patch. This seems to be losing important information. Andi, why did you prefer that to clearing the pending signal and retrying the write? I'm definitely not familiar enough with signal management in the kernel to know what side-effects there might be from just clearing the pending flag without doing anything else: I did it that way because fs/exec.c:do_coredump() does this before it runs the ->core_dump function.
I wonder whether dump_write() shouldn't be rewritten along the lines of a normal, robust userspace writer, where we handle EAGAIN and EINTR (can we ever get these at this level, or do we ever just get ERESTARTSYS?), short writes, etc.
PS. I have a thought about why this happens for me; I doubt I'm getting SIGPIPE. In our system it's almost certain that these worker processes will get a signal (SIGUSR1 or something: I forget exactly which one) if they are still alive after a few seconds. I suspect that the core dump takes long enough that this signal is received in the middle of the core dump. It may be that this problem hasn't been noticed before because it's unlikely you'll receive a signal in the middle of dumping core, and if you do get one every now and then, and get a short core, it's not easily reproducible. I left my debugging in the kernel and I get exactly one instance of signal_pending() per process, so having the signal be SIGPIPE seems unlikely.
Quoted 3 lines · Andrew Morton
Many filesystems will return a short write if they hit a memory allocation failure, for example.
pipe_write()sure will. Retrying is appropriate in such a case.
Here's a patch that "works for me" and tries to address the various issues. I've no idea what landmines I might have stepped on here. I also have no git-fu so this uses simple diff -u format.
Open issues: is it possible to get -EAGAIN or -EINTR at this level of the kernel? Or will it always be just -ERESTARTSYS? Is there evil in simply running clear_thread_flag(TIF_SIGPENDING) without "handling" the signal in any way?
---
Retry core dump writes where appropriate
Core dump write operations can be incomplete due to signal reception or possibly recoverable partial writes.
Previously any incomplete write in the ELF core dumper caused the core dump to stop, giving short cores in these cases. Modify the core dumper to retry the write where appropriate.
---
fs/binfmt_elf.c +19 -1
@@ -1128,7 +1128,25 @@*/ static int dump_write (struct file *file,const void *addr,int nr) { -return file->f_op->write (file, addr, nr, &file->f_pos) == nr; +const char *p = addr; +while (1 ) { +int r = file->f_op->write (file, p, nr, &file->f_pos); + +if (likely (r == nr)) +return 1 ; + +if (r == -ERESTARTSYS || r == -EAGAIN || r == -EINTR ) +/* Ignore signals during coredump. */ +clear_thread_flag (TIF_SIGPENDING ); +else if (r >0 ) { +/* Partial write: try again with the rest. */ + p += r; + nr -= r; + } +else +/* Lose! */ +return 0 ; + } }static int dump_seek (struct file *file,loff_t off)
Quoted 9 lines · Andrew Morton
Quoted 6 lines · On Wed, 27 May 2009 02:11:04 +0200 Andi Kleen <[email protected]> wrote:
Quoted 1 line
I dunno. Is this true of all linux filesystems in all cases? Maybe.
Assuming one of them is not would you rather want to fix that file system or 10 zillion user programs (including the kernel core dumper) that get it wrong? @)
Quoted 5 lines
I think that removing one bug is better than adding one.
Many filesystems will return a short write if they hit a memory allocation failure, for example.
pipe_write()sure will. Retrying is appropriate in such a case.
Sorry but are you really suggesting every program in the world that uses write() anywhere should put it into a loop? That seems just like really bad API design to me, requiring such contortions in a fundamental system call just to work around kernel deficiencies.
I can just imagine the programmers putting nasty comments about the Linux kernel on top of those loops and they would be fully deserved.
And the same applies to in-kernel users really.
The memory allocation case more sounds like a bug in these fs and in pipe.
e.g. the network stack sleeps waiting for memory, perhaps these file systems should too.
Or it should just always return -ENOMEM. Typically when the system is badly out of memory you're gonna lose anyways because a lot of things start failing.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 16 lines · On Wed, 27 May 2009 09:31:36 +0200 Andi Kleen <[email protected]> wrote:
Quoted 9 lines · Andrew Morton
Quoted 6 lines · On Wed, 27 May 2009 02:11:04 +0200 Andi Kleen <[email protected]> wrote:
Quoted 1 line
I dunno. Is this true of all linux filesystems in all cases? Maybe.
Assuming one of them is not would you rather want to fix that file system or 10 zillion user programs (including the kernel core dumper) that get it wrong? @)
Quoted 5 lines
I think that removing one bug is better than adding one.
Many filesystems will return a short write if they hit a memory allocation failure, for example.
pipe_write()sure will. Retrying is appropriate in such a case.
Quoted 9 lines
Sorry but are you really suggesting every program in the world that uses
write()anywhere should put it into a loop? That seems just like really bad API design to me, requiring such contortions in a fundamental system call just to work around kernel deficiencies.I can just imagine the programmers putting nasty comments about the Linux kernel on top of those loops and they would be fully deserved.
Hey, don't look at me - blame Brian Kernighan or George Bush or someone.
Quoted 1 line
And the same applies to in-kernel users really.
We could delete a rather nice amount of tricky VFS code if we were to make this assumption. But of course we daren't do that.
And as long as we're attempting to correctly handle partial writes all over the kernel, it's a bit dopey to deliberately avoid doing this at one particular codesite.
I bet glibc handles partial writes...
Quoted 9 lines
The memory allocation case more sounds like a bug in these fs and in pipe.
e.g. the network stack sleeps waiting for memory, perhaps these file systems should too.
Or it should just always return
-ENOMEM. Typically when the system is badly out of memory you're gonna lose anyways because a lot of things start failing.
The kernel should only fail if it has no other option.
Quoted 2 lines
Hey, don't look at me - blame Brian Kernighan or George Bush or someone.
Heh.
What I meant is: if it makes sense to retry the kernel should do that on its own. It has better information about the circumstances anyways. And it would make sense to put all such logic into a single place instead of all programs.
If it doesn't we shouldn't try to force that to user space. That would be similar to signal handling without SA_RESTART which I think nearly everyone agrees was one of the worst APIs in Unix ever.
In most cases (e.g. out of memory) it likely doesn't make much sense to retry anyways.
So short write always means error.
Quoted 4 lines
Quoted 1 line
And the same applies to in-kernel users really.
We could delete a rather nice amount of tricky VFS code if we were to make this assumption. But of course we daren't do that.
What do you mean?
Quoted 6 lines
And as long as we're attempting to correctly handle partial writes all over the kernel, it's a bit dopey to deliberately avoid doing this at one particular codesite.
I bet glibc handles partial writes...
I just spent some time nagivating through the glibc stdio me^wmaze and I don't see any retry loops.
Also even if it did there would be lots of other users (google codesearch has >23 million hits for "write")
Quoted 3 lines
Quoted 1 line
a lot of things start failing.
The kernel should only fail if it has no other option.
Typically short write means no other option.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 20 lines · On Wed, 27 May 2009 10:52:38 +0200 Andi Kleen <[email protected]> wrote:
Quoted 2 lines
Hey, don't look at me - blame Brian Kernighan or George Bush or someone.
Heh.
What I meant is: if it makes sense to retry the kernel should do that on its own. It has better information about the circumstances anyways. And it would make sense to put all such logic into a single place instead of all programs.
If it doesn't we shouldn't try to force that to user space. That would be similar to signal handling without
SA_RESTARTwhich I think nearly everyone agrees was one of the worst APIs in Unix ever.In most cases (e.g. out of memory) it likely doesn't make much sense to retry anyways.
So short write always means error.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
And the same applies to in-kernel users really.
We could delete a rather nice amount of tricky VFS code if we were to make this assumption. But of course we daren't do that.
What do you mean?
There's code all over the VFS IO paths which correctly recognises, handles and propagates short reads and writes.
Actually, by far the most common case here is that the short read/write will be followed by a -ENOSPC or -EFAULT or whatever, so forget I said that ;)
Quoted 9 lines
Quoted 6 lines
And as long as we're attempting to correctly handle partial writes all over the kernel, it's a bit dopey to deliberately avoid doing this at one particular codesite.
I bet glibc handles partial writes...
I just spent some time nagivating through the glibc stdio me^wmaze and I don't see any retry loops.
Perhaps it's usually propagated back, dunno.
Quoted 8 lines
Also even if it did there would be lots of other users (google codesearch has >23 million hits for "write")
Quoted 3 lines
Quoted 1 line
a lot of things start failing.
The kernel should only fail if it has no other option.
Typically short write means no other option.
typically != always. It would certainly be peculiar behaviour and perhaps it was specified that way mainly on behalf of UART drivers and such, dunno.
Quoted 7 lines · Andi Kleen
When a signal happens during core dump the core dump to a pipe can fail, because the write returns short, but the ELF core dumpers cannot handle that.
There's no reason to handle signals during core dumping, so just block them all.
Actually, I think there is a strong reason to handle signals during core dumping. The coredump can take a lot of time/resources, not good it looks like unkillable procees to users.
Please look at
killable/interruptible coredumps http://marc.info/?l=linux-kernel&m=121665710711931
at least, I think SIGKILL should terminate core dumping.
Quoted 6 lines
Open issue: ELF puts blocked signals into the core dump and that will be always fully blocked now. Need to save it somewhere?
fs/exec.c line 1767
Quoted 8 lines
fs/exec.c +5
-0@@ -1760,6 +1760,12 @@ goto fail; } +/* block all signals */ +spin_lock_irq (¤t->sighand->siglock); +sigfillset (¤t->blocked); +/* No recalc sigpending */ +spin_unlock_irq (¤t->sighand->siglock);
Perhaps it makes sense to do
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -644,6 +644,7 @@ static int prepare_signal(int sig, struc
/*
* The process is in the middle of dying, nothing to do.
*/
+ return 0;
} else if (sig_kernel_stop(sig)) {
/*
* This is a stop signal. Remove SIGCONT from all queues.instead, this was discussed before. This way the exiting/coredumping task ignores all signals, and we cab simplify complete_signal() a bit.
This all needs more discussion, but imho for now something like Paul's patch http://marc.info/?l=linux-kernel&m=124340506200729
is the best workaround. Note that we have the same dump_write() in binfmt_elf.c and binfmt_aout.c, perhaps it makes sense to create coredump_file_write() helper in fs/exec.c.
Oleg.
Quoted 3 lines
Actually, I think there is a strong reason to handle signals during core dumping. The coredump can take a lot of time/resources, not good it looks like unkillable procees to users.
One problem with that is if you send a process a string of signals that cause a core dump and then kill. In the old case you would just get a full core dump on the first signal and be done. With your change it would process the second signal too and stop the dumping and you get none or a partial core dump. That might well break existing setups.
Quoted 2 lines
Perhaps it makes sense to do
Probably, my patch is not usable as is anyways because it would need to save the mask somewhere so that the ELF dumper can build the correct mask. Your version would at least avoid that problem.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 10 lines · Andi Kleen
Quoted 4 lines
Actually, I think there is a strong reason to handle signals during core dumping. The coredump can take a lot of time/resources, not good it looks like unkillable procees to users.
One problem with that is if you send a process a string of signals that cause a core dump and then kill. In the old case you would just get a full core dump on the first signal and be done. With your change it would process the second signal too and stop the dumping and you get none or a partial core dump. That might well break existing setups.
I don't think we should worry about this particular case. Suppose a user does
kill(pid, SIGQUIT); kill(pid, SIGKILL);
In this case, most likely the core dump will not start. Because SIGKILL will set SIGNAL_GROUP_EXIT before the process dequeues SIGQUIT and start do_coredump() which checks signal_group_exit() in zap_threads().
But yes, I agree, this change is user-visible and should be discussed.
Oleg.
Quoted 12 lines · Oleg Nesterov
Quoted 10 lines · Andi Kleen
Quoted 4 lines
Actually, I think there is a strong reason to handle signals during core dumping. The coredump can take a lot of time/resources, not good it looks like unkillable procees to users.
One problem with that is if you send a process a string of signals that cause a core dump and then kill. In the old case you would just get a full core dump on the first signal and be done. With your change it would process the second signal too and stop the dumping and you get none or a partial core dump. That might well break existing setups.
Quoted 5 lines
I don't think we should worry about this particular case. Suppose a user does
kill(pid, SIGQUIT); kill(pid, SIGKILL);
I'm not sure about this. Why even bother with SIGQUIT (or anything else) if you're just going to immediately SIGKILL afterwards? What people do all the time, and I think should be supported, is something like this:
<do 5 times> kill(pid, SIGINT); sleep(1); <if pid is dead break> kill(pid, SIGKILL);
Often with other signals in the mix like SIGHUP or whatever. The idea is to give the process a chance to do "whatever it does" to clean up and then, if it's still there we consider it too wedged to respond and send a SIGKILL. If the cleanup operations invoked by receiving the SIGINT caused a core dump, then you wouldn't want the SIGKILL to stop the core dump.
On the other hand I do agree that it would be nice to be able to smash a core dump that was taking a long time or trying to write to an unavailable resource like a stalled NFS mount or whatever. Sigh.
Quoted 14 lines · Oleg Nesterov
Quoted 7 lines · Andi Kleen
When a signal happens during core dump the core dump to a pipe can fail, because the write returns short, but the ELF core dumpers cannot handle that.
There's no reason to handle signals during core dumping, so just block them all.
Actually, I think there is a strong reason to handle signals during core dumping. The coredump can take a lot of time/resources, not good it looks like unkillable procees to users.
Quoted 6 lines
Please look at
killable/interruptible coredumps http://marc.info/?l=linux-kernel&m=121665710711931at least, I think SIGKILL should terminate core dumping.
Forgot to mention, and we have problems with OOM. Not only the coredumping task can't be killed (and it can populate the memory via get_user_pages). The coredump just disables OOM, if select_bad_process() sees the PF_EXITING task with ->mm == NULL it returns -1.
Quoted 5 lines
This all needs more discussion, but imho for now something like Paul's patch http://marc.info/?l=linux-kernel&m=124340506200729
is the best workaround. Note that we have the samedump_write()inbinfmt_elf.candbinfmt_aout.c, perhaps it makes sense to createcoredump_file_write()helper infs/exec.c.
But I didn't notice Paul also reports the kernel panic:
page:ffffe20010d63d00 flags:0x8000000000000001 mapping:0000000000000000 mapcount:0 \ count:0 Trying to fix it up, but a reboot is needed Backtrace: Pid: 3346, comm: worker Tainted: P 2.6.27.24-worker #4 Call Trace: [<ffffffff80284fd4>] bad_page+0x74/0xc0 [<ffffffff80286168>] free_hot_cold_page+0x248/0x2f0 [<ffffffff802f4096>] free_wr_note_data+0x56/0x70 [<ffffffff802a95c6>] kfree+0x86/0x100 [<ffffffff802f4096>] free_wr_note_data+0x56/0x70 [<ffffffff802f0991>] elf_core_dump+0x611/0x1160
At first glance, this looks like a bug outside of coredump.c, we are trying to free PG_locked page?
Oleg.
Quoted 26 lines · Oleg Nesterov
Forgot to mention, and we have problems with OOM. Not only the coredumping task can't be killed (and it can populate the memory via
get_user_pages). The coredump just disables OOM, ifselect_bad_process()sees thePF_EXITINGtask with ->mm == NULL it returns -1.Quoted 5 lines
This all needs more discussion, but imho for now something like Paul's patch http://marc.info/?l=linux-kernel&m=124340506200729
is the best workaround. Note that we have the samedump_write()inbinfmt_elf.candbinfmt_aout.c, perhaps it makes sense to createcoredump_file_write()helper infs/exec.c.But I didn't notice Paul also reports the kernel panic:
page:ffffe20010d63d00 flags:0x8000000000000001 mapping:0000000000000000 mapcount:0 \ count:0 Trying to fix it up, but a reboot is needed Backtrace: Pid: 3346, comm: worker Tainted: P 2.6.27.24-worker #4 Call Trace: [<ffffffff80284fd4>] bad_page+0x74/0xc0 [<ffffffff80286168>] free_hot_cold_page+0x248/0x2f0 [<ffffffff802f4096>] free_wr_note_data+0x56/0x70 [<ffffffff802a95c6>] kfree+0x86/0x100 [<ffffffff802f4096>] free_wr_note_data+0x56/0x70 [<ffffffff802f0991>] elf_core_dump+0x611/0x1160
Quoted 2 lines
At first glance, this looks like a bug outside of
coredump.c, we are trying to freePG_lockedpage?
This might be something different, or a side-effect that's not understood; I haven't seen this happen again since I applied my change, and I used to be able to make it happen every time within 2 or 3 invocations of my "failing" core dump procedure. Now I have dumped core using my "failing" procedure 10-15 times in a row with no ill-effects.
I'll keep an eye out for this one though.
Quoted 18 lines · Andi Kleen
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Wed, 27 May 2009 01:14:28 +0200 Andi Kleen <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
dump_write()doesn't seem right, either. If ->write() returns, say, 100 then the dump should keep on going. At present it treats this return as an error.Quoted 5 lines
Quoted 2 lines
I think that's correct actually. Short write typically means serious issue like disk full or broken pipe, so stopping is good.
But we shouldn't assume that. It could be that the ->write implementation is perfectly able to absorb the remaining data.
Maybe in theory, but in practice that's unlikely isn't it? Disk is full or pipe is blocking etc.
Quoted 7 lines
Quoted 2 lines
We should only error out of the
write()returned zero or-EFOO. The current code is simply buggy, but got lucky.Maybe very pedantically, but I would argue that most programs don't do what you're saying (retry on any short write) and it's actually not very nice to always write a loop for each write.
My experience, from userspace, both with the codebases I currently work on professionally and my own hobby projects, is that people usually write a "write wrapper" function that deals with short writes, interrupted system call etc and then just call the wrapper so one does not have to open-code a loop everywhere one wants to call write() - one just calls the wrapper function that does the right thing.
Signature · 3 lines
Jesper Juhl <[email protected]> http://www.chaosbits.net/ Plain text mails only, please http://www.expita.com/nomime.html Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Quoted 14 lines · Paul Smith
Quoted 12 lines · Oleg Nesterov
Quoted 10 lines · Andi Kleen
Quoted 4 lines
Actually, I think there is a strong reason to handle signals during core dumping. The coredump can take a lot of time/resources, not good it looks like unkillable procees to users.
One problem with that is if you send a process a string of signals that cause a core dump and then kill. In the old case you would just get a full core dump on the first signal and be done. With your change it would process the second signal too and stop the dumping and you get none or a partial core dump. That might well break existing setups.
Quoted 8 lines
Quoted 5 lines
I don't think we should worry about this particular case. Suppose a user does
kill(pid, SIGQUIT); kill(pid, SIGKILL);I'm not sure about this. Why even bother with SIGQUIT (or anything else) if you're just going to immediately SIGKILL afterwards?
Probably I misunderstood what Andi meant.
Quoted 10 lines
What
people do all the time, and I think should be supported, is something like this:<do 5 times> kill(pid, SIGINT); sleep(1); <if pid is dead break> kill(pid, SIGKILL);
Quoted 6 lines
Often with other signals in the mix like SIGHUP or whatever. The idea is to give the process a chance to do "whatever it does" to clean up and then, if it's still there we consider it too wedged to respond and send a SIGKILL. If the cleanup operations invoked by receiving the SIGINT caused a core dump, then you wouldn't want the SIGKILL to stop the core dump.
Yes. Once again, this change is user-visble and it can confuse/break existing setups, I agree. As almost any user-visible change ;)
Quoted 3 lines
On the other hand I do agree that it would be nice to be able to smash a core dump that was taking a long time or trying to write to an unavailable resource like a stalled NFS mount or whatever. Sigh.
Yes.
Oleg.
Quoted 25 lines · Andi Kleen
Quoted 8 lines · Andrew Morton
Quoted 5 lines · On Wed, 27 May 2009 01:14:28 +0200 Andi Kleen <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
dump_write()doesn't seem right, either. If ->write() returns, say, 100 then the dump should keep on going. At present it treats this return as an error.Quoted 5 lines
Quoted 2 lines
I think that's correct actually. Short write typically means serious issue like disk full or broken pipe, so stopping is good.
But we shouldn't assume that. It could be that the ->write implementation is perfectly able to absorb the remaining data.
Maybe in theory, but in practice that's unlikely isn't it? Disk is full or pipe is blocking etc.
Quoted 2 lines
We should only error out of the
write()returned zero or-EFOO. The current code is simply buggy, but got lucky.Maybe very pedantically, but I would argue that most programs don't do what you're saying (retry on any short write) and it's actually not very nice to always write a loop for each write.
Quoted 5 lines
Also any IO device who relies on that would likely find that it won't work with a lot of software.
So I think the current behaviour is ok, just need to get rid of the signals.
Short writes are normal at least for pipes and sockets... better fix the sw.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html