select() setting ERESTARTNOHAND (514).
I've been looking at an issue in Python where a "time.sleep(1)" will sporadically raise an IOError exception with errno=514. time.sleep() is implemented with select(), to get sub-second resolution.
In looking at the Linux code for ERESTARTNOHAND, I see that include/linux/errno.h says this errno should never make it to the user. However, in this instance we ARE seeing it. Looking around on google shows others are seeing it as well, though hits are few.
In looking at the select() code, I see that there are definitely cases where sys_select() or sys_pselect7() can return -ERESTARTNOHAND. However, I don't know if this is expected to be caught elsewhere, or if returning it here would send it back to user-space. Worse, I don't fully understand what the impact would be of trapping the ERESTARTNOHAND in the sys_select/sys_pselect7 functions would be.
Is this something that's intended to be retrned back to the user, in which case the message in include/linux/errno.h should be corrected and people using time.sleep() in python will just have to live with it sometimes raising an exception? Or is it something that definitely should never reach the user-space code, and there's some leak.
Just to be clear, this is happening only on one machine out of at least 4 where this has been tested. The machine where it's happening is a dual processor, dual core Xeon 2GHz 51xx series system. The other systems where it's not happening are single CPU Celeron or P4 class systems, though one is a 2-year-old quad CPU Xeon running something <2GHz, IIRC.
More details on my investigation are at:
http://www.tummy.com/journals/entries/jafo_20070110_154659
Thoughts?
Thanks,
Sean
(Not subscribed, I'll use the list archive to follow-up)
Signature · 4 lines
Electricity travels a foot in a nanosecond.
-- Commodore Grace Murray Hopper
Sean Reifschneider, Member of Technical Staff <[email protected]>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High AvailabilityFrom: Sean Reifschneider <[email protected]>
Date: Wed, 10 Jan 2007 16:42:38 -0700
Quoted 6 lines
In looking at the
select()code, I see that there are definitely cases wheresys_select()orsys_pselect7()can return-ERESTARTNOHAND. However, I don't know if this is expected to be caught elsewhere, or if returning it here would send it back to user-space. Worse, I don't fully understand what the impact would be of trapping the ERESTARTNOHAND in the sys_select/sys_pselect7 functions would be.
It gets caught by the return into userspace code.
Specifically the signal dispatch should repair that return value to a valid error return code when it tries to dispatch the signal that select() set in the task struct.
Note that select() only returns these values when signal_pending() is true.
Quoted 1 line · David Miller
It gets caught by the return into userspace code.
Ok, so somehow it is leaking. I have a system in the lab that is the same hardware as production, but it currently has no, you know, hard drives in it, so some assembly is required. I'll see if I can reproduce it in a test environment and then see if I can get more information on when/where it is leaking.
Quoted 2 lines
Note that
select()only returns these values whensignal_pending()is true.
Yes, I saw that. I didn't fully understand it, but I saw it.
Thanks,
Sean
Signature · 4 lines
CChheecckk yyoouurr dduupplleexx sswwiittcchh.. Sean Reifschneider, Member of Technical Staff <[email protected]> tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability Back off man. I'm a scientist. http://HackingSociety.org/
From: Sean Reifschneider <[email protected]>
Date: Wed, 10 Jan 2007 18:04:29 -0700
Quoted 8 lines · David Miller
Quoted 1 line
It gets caught by the return into userspace code.
Ok, so somehow it is leaking. I have a system in the lab that is the same hardware as production, but it currently has no, you know, hard drives in it, so some assembly is required. I'll see if I can reproduce it in a test environment and then see if I can get more information on when/where it is leaking.
If you're only seeing it in strace, that's expected due to some unfortunate things in the way that x86 and x86_64 handle signal return events via ptrace().
On sparc and sparc64 I fixed this long ago such that ptrace() will update the user registers before ptrace parents are notified, and therefore you'll never see those kernel internal error codes.
The upside of this is that you'll really need to see what value is making it to the application. What the kernel is probably doing is looping trying to restart the system call and sending the signal. If it's doing that the application is being rewound to call the system call again once the signal handler returns (if that is even being run at all).
Quoted 1 line · David Miller
If you're only seeing it in strace, that's expected due to some
Nope, I haven't looked in strace at all. It's definitely making it to user-space. The code in question is (abbreviated):
if (select(0, (fd_set *)0, (fd_set *)0, (fd_set *)0, &t) != 0) {
PyErr_SetFromErrno(PyExc_IOError);
return -1;
}which causes the Python interpreter to raise an IOError exception, including the value of errno, which is 514.
Thanks,
Sean
Signature · 4 lines
This mountain is PURE SNOW! Do you know what the street value of this mountain is!?! -- Better Off Dead Sean Reifschneider, Member of Technical Staff <[email protected]> tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability
Quoted 7 lines · Sean Reifschneider
Nope, I haven't looked in strace at all. It's definitely making it to user-space. The code in question is (abbreviated):
if (select(0, (fd_set *)0, (fd_set *)0, (fd_set *)0, &t) != 0) { PyErr_SetFromErrno(PyExc_IOError); return -1; }
Anything else relevant? Do you know which signal interrupted select? Is this a single or multithreaded application? And where did the signal come from?
I tried to reproduce your problem in various ways on 2.6.20-rc4, but it didn't appear.
Thanks.
Signature · 2 lines
http://www.PowerDNS.com Open source, database driven DNS Software http://netherlabs.nl Open and Closed source services
Quoted 2 lines · bert hubert
Anything else relevant? Do you know which signal interrupted select? Is this a single or multithreaded application? And where did the signal come from?
It is, AFAIK, a multi-threaded application. I don't have any information on which signal interrupted the process. I'll ask the person who reported it to me, Doug, to respond with additional information.
Quoted 2 lines
I tried to reproduce your problem in various ways on 2.6.20-rc4, but it didn't appear.
Thanks.
Sean
Signature · 4 lines
[...] Premature optimization is the root of all evil.
-- Donald Knuth
Sean Reifschneider, Member of Technical Staff <[email protected]>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High AvailabilityPATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).
In reply to Sean ReifschneiderQuoted 5 lines · On Wednesday January 10, [email protected] wrote:
In looking at the Linux code for ERESTARTNOHAND, I see that
include/linux/errno.hsays this errno should never make it to the user. However, in this instance we ARE seeing it. Looking around on google shows others are seeing it as well, though hits are few.
..
Quoted 2 lines
Thoughts?
Just a 'me too' at this point.
The X server on my shiny new notebook (Core 2 Duo) occasionally dies with 'select' repeatedly returning ERESTARTNOHAND. It is most annoying!
You don't mention in the Email which kernel version you use but I see from the web page you reference it is 2.6.19.1. I'm using 2.6.18.something.
I thought I'd have a quick look at the code, comparing i386 to x86-64 and guess what I found.....
On x86-64, regs->rax is "unsigned long", so the following is needed....
I haven't tried it yet.
NeilBrown
### Diffstat output
./arch/x86_64/kernel/signal.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86_64/kernel/signal.c +1 -1
@@ -331,7 +331,7 @@ handle_signal(unsigned long sig, siginfo/* Are we from a system call? */ if ((long )regs->orig_rax >=0 ) {/* If so, check system call restarting.. */ -switch (regs->rax) { +switch ((long )regs->rax) {case -ERESTART_RESTARTBLOCK :case -ERESTARTNOHAND : regs->rax = -EINTR ;
Re: PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).
Quoted 6 lines · Neil Brown
Quoted 5 lines · On Wednesday January 10, [email protected] wrote:
In looking at the Linux code for ERESTARTNOHAND, I see that
include/linux/errno.hsays this errno should never make it to the user. However, in this instance we ARE seeing it. Looking around on google shows others are seeing it as well, though hits are few.
Quoted 8 lines
..
Quoted 2 lines
Thoughts?
Just a 'me too' at this point.
The X server on my shiny new notebook (Core 2 Duo) occasionally dies with 'select' repeatedly returning ERESTARTNOHAND. It is most annoying!
Normally it should be only visible in strace. Did you see it without strace?
Quoted 10 lines
You don't mention in the Email which kernel version you use but I see from the web page you reference it is 2.6.19.1. I'm using 2.6.18.something.
I thought I'd have a quick look at the code, comparing i386 to x86-64 and guess what I found.....
On x86-64,
regs->raxis "unsigned long", so the following is needed....
regs->rax is unsigned long.
I don't think your patch will make any difference. What do you think it will change?
-Andi
Re: PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).
Quoted 7 lines · On Thursday January 11, [email protected] wrote:
Quoted 4 lines
Just a 'me too' at this point.
The X server on my shiny new notebook (Core 2 Duo) occasionally dies with 'select' repeatedly returning ERESTARTNOHAND. It is most annoying!Normally it should be only visible in strace. Did you see it without strace?
No, only in strace.
Quoted 9 lines
Quoted 9 lines
You don't mention in the Email which kernel version you use but I see from the web page you reference it is 2.6.19.1. I'm using 2.6.18.something.
I thought I'd have a quick look at the code, comparing i386 to x86-64 and guess what I found.....
Quoted 6 lines
Quoted 2 lines
On x86-64,
regs->raxis "unsigned long", so the following is needed....
regs->raxis unsigned long.
I don't think your patch will make any difference. What do you think it will change?
If regs->rax is unsigned long, then I would think the compiler would be allowed to convert
switch (regs->rax) {
case -514 : whatever;}
to a no-op, as regs->rax will never have a negative value.
However it appears that the current compiler doesn't make that optimisation so I guess I was too hasty.
Still, I think it would be safer to have the cast, in case the compiler decided to be clever.... or does the C standard ensure against that?
Sorry for the noise,
NeilBrown
Re: PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).
Quoted 10 lines · Neil Brown
Quoted 7 lines · On Thursday January 11, [email protected] wrote:
Quoted 4 lines
Just a 'me too' at this point.
The X server on my shiny new notebook (Core 2 Duo) occasionally dies with 'select' repeatedly returning ERESTARTNOHAND. It is most annoying!Normally it should be only visible in strace. Did you see it without strace?
No, only in strace.
strace leaks internal errors. At some point that should be fixed, but it's not really a serious problem.
There was one other report of internal errors leaking without strace, but it was vague and I never got confirmation.
Quoted 2 lines
Still, I think it would be safer to have the cast, in case the compiler decided to be clever.... or does the C standard ensure against that?
It does.
-Andi
Re: PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).
In reply to Neil BrownQuoted 5 lines · Neil Brown
Quoted 2 lines · On Thursday January 11, [email protected] wrote:
Normally it should be only visible in strace. Did you see it without strace?
No, only in strace.
I am absolutely seeing it outside of strace. It is showing up as an errno to the select call:
if (select(0, (fd_set *)0, (fd_set *)0, (fd_set *)0, &t) != 0) {
if (errno != EINTR) {
PyErr_SetFromErrno(PyExc_IOError);
return -1;
}This code is seeing errno=514.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
You don't mention in the Email which kernel version you use but I see from the web page you reference it is 2.6.19.1. I'm using
The production system is running CentOS 4.4, 2.6.9 kernel. However, it looks to be the same issue all the way up to 2.6.19.1, and google shows reports of it on 2.6.17.
Thanks,
Sean
Signature · 4 lines
George Washington was first in war, first in peace -- and first to have his birthday juggled to make a long weekend. Sean Reifschneider, Member of Technical Staff <[email protected]> tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability
Re: PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).
In reply to Neil BrownQuoted 8 lines · Neil Brown
If
regs->raxis unsigned long, then I would think the compiler would be allowed to convertswitch (regs->rax) { case -514 : whatever; }to a no-op, as
regs->raxwill never have a negative value.
In C, you never actually compare different types. They always promoted to some common type first.
both sides of (impicit) == here get promoted to "biggest" integer, in this case, to unsigned long. "-514" is an int, so it gets sign extended to the width of "long" and then converted to unsigned long.
--
vda
Re: PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).
In reply to Neil BrownFrom: Neil Brown <[email protected]>
Date: Thu, 11 Jan 2007 11:37:05 +1100
Quoted 4 lines
On x86-64,
regs->raxis "unsigned long", so the following is needed....I haven't tried it yet.
Doesn't type promotion take care of that? Did you verify that assember?
I checked the assembler on sparc64 for similar constructs and it does the right thing.