Skip to contentSkip to conversation

select() setting ERESTARTNOHAND (514).

DISCUSSION Sean Reifschneider 6 participants 14 messages Jan 11, 2007 – Jan 24, 2007

lore
Share & tools
lore
#0 lore raw

I've been looking at an issue in Python where a "time.sleep(1)" will sporadically raise an IOError exception with errno=514. time.sleep() is implemented with select(), to get sub-second resolution.

In looking at the Linux code for ERESTARTNOHAND, I see that include/linux/errno.h says this errno should never make it to the user. However, in this instance we ARE seeing it. Looking around on google shows others are seeing it as well, though hits are few.

In looking at the select() code, I see that there are definitely cases where sys_select() or sys_pselect7() can return -ERESTARTNOHAND. However, I don't know if this is expected to be caught elsewhere, or if returning it here would send it back to user-space. Worse, I don't fully understand what the impact would be of trapping the ERESTARTNOHAND in the sys_select/sys_pselect7 functions would be.

Is this something that's intended to be retrned back to the user, in which case the message in include/linux/errno.h should be corrected and people using time.sleep() in python will just have to live with it sometimes raising an exception? Or is it something that definitely should never reach the user-space code, and there's some leak.

Just to be clear, this is happening only on one machine out of at least 4 where this has been tested. The machine where it's happening is a dual processor, dual core Xeon 2GHz 51xx series system. The other systems where it's not happening are single CPU Celeron or P4 class systems, though one is a 2-year-old quad CPU Xeon running something <2GHz, IIRC.

More details on my investigation are at:

http://www.tummy.com/journals/entries/jafo_20070110_154659

Thoughts?

Thanks,
Sean
(Not subscribed, I'll use the list archive to follow-up)

Signature · 4 lines
 Electricity travels a foot in a nanosecond.
                 -- Commodore Grace Murray Hopper
Sean Reifschneider, Member of Technical Staff <[email protected]>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability
#1 lore raw

From: Sean Reifschneider <[email protected]>
Date: Wed, 10 Jan 2007 16:42:38 -0700

Quoted 6 lines

In looking at the select() code, I see that there are definitely cases where sys_select() or sys_pselect7() can return -ERESTARTNOHAND. However, I don't know if this is expected to be caught elsewhere, or if returning it here would send it back to user-space. Worse, I don't fully understand what the impact would be of trapping the ERESTARTNOHAND in the sys_select/sys_pselect7 functions would be.

It gets caught by the return into userspace code.

Specifically the signal dispatch should repair that return value to a valid error return code when it tries to dispatch the signal that select() set in the task struct.

Note that select() only returns these values when signal_pending() is true.

#2 lore raw
Quoted 1 line · David Miller

It gets caught by the return into userspace code.

Ok, so somehow it is leaking. I have a system in the lab that is the same hardware as production, but it currently has no, you know, hard drives in it, so some assembly is required. I'll see if I can reproduce it in a test environment and then see if I can get more information on when/where it is leaking.

Quoted 2 lines

Note that select() only returns these values when signal_pending() is true.

Yes, I saw that. I didn't fully understand it, but I saw it.

Thanks,
Sean

Signature · 4 lines
 CChheecckk yyoouurr dduupplleexx sswwiittcchh..
Sean Reifschneider, Member of Technical Staff <[email protected]>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability
      Back off man. I'm a scientist.   http://HackingSociety.org/
#3 lore raw

From: Sean Reifschneider <[email protected]>
Date: Wed, 10 Jan 2007 18:04:29 -0700

Quoted 8 lines · David Miller
Quoted 1 line

It gets caught by the return into userspace code.

Ok, so somehow it is leaking. I have a system in the lab that is the same hardware as production, but it currently has no, you know, hard drives in it, so some assembly is required. I'll see if I can reproduce it in a test environment and then see if I can get more information on when/where it is leaking.

If you're only seeing it in strace, that's expected due to some unfortunate things in the way that x86 and x86_64 handle signal return events via ptrace().

On sparc and sparc64 I fixed this long ago such that ptrace() will update the user registers before ptrace parents are notified, and therefore you'll never see those kernel internal error codes.

The upside of this is that you'll really need to see what value is making it to the application. What the kernel is probably doing is looping trying to restart the system call and sending the signal. If it's doing that the application is being rewound to call the system call again once the signal handler returns (if that is even being run at all).

#4 lore raw
Quoted 1 line · David Miller

If you're only seeing it in strace, that's expected due to some

Nope, I haven't looked in strace at all. It's definitely making it to user-space. The code in question is (abbreviated):

   if (select(0, (fd_set *)0, (fd_set *)0, (fd_set *)0, &t) != 0) {
      PyErr_SetFromErrno(PyExc_IOError);
      return -1;
      }

which causes the Python interpreter to raise an IOError exception, including the value of errno, which is 514.

Thanks,
Sean

Signature · 4 lines
 This mountain is PURE SNOW!  Do you know what the street value of this
 mountain is!?!                -- Better Off Dead
Sean Reifschneider, Member of Technical Staff <[email protected]>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability
#5 lore raw
Quoted 7 lines · Sean Reifschneider

Nope, I haven't looked in strace at all. It's definitely making it to user-space. The code in question is (abbreviated):

   if (select(0, (fd_set *)0, (fd_set *)0, (fd_set *)0, &t) != 0) {
      PyErr_SetFromErrno(PyExc_IOError);
      return -1;
      }

Anything else relevant? Do you know which signal interrupted select? Is this a single or multithreaded application? And where did the signal come from?

I tried to reproduce your problem in various ways on 2.6.20-rc4, but it didn't appear.

Thanks.

Signature · 2 lines
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services
#6 lore raw
Quoted 2 lines · bert hubert

Anything else relevant? Do you know which signal interrupted select? Is this a single or multithreaded application? And where did the signal come from?

It is, AFAIK, a multi-threaded application. I don't have any information on which signal interrupted the process. I'll ask the person who reported it to me, Doug, to respond with additional information.

Quoted 2 lines

I tried to reproduce your problem in various ways on 2.6.20-rc4, but it didn't appear.

Thanks.

Sean

Signature · 4 lines
 [...] Premature optimization is the root of all evil.
                 -- Donald Knuth
Sean Reifschneider, Member of Technical Staff <[email protected]>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability
#7 lore raw

PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).

In reply to Sean Reifschneider
Quoted 5 lines · On Wednesday January 10, [email protected] wrote:

In looking at the Linux code for ERESTARTNOHAND, I see that include/linux/errno.h says this errno should never make it to the user. However, in this instance we ARE seeing it. Looking around on google shows others are seeing it as well, though hits are few.

..

Quoted 2 lines

Thoughts?

Just a 'me too' at this point.
The X server on my shiny new notebook (Core 2 Duo) occasionally dies with 'select' repeatedly returning ERESTARTNOHAND. It is most annoying!

You don't mention in the Email which kernel version you use but I see from the web page you reference it is 2.6.19.1. I'm using 2.6.18.something.

I thought I'd have a quick look at the code, comparing i386 to x86-64 and guess what I found.....

On x86-64, regs->rax is "unsigned long", so the following is needed....

I haven't tried it yet.

NeilBrown

### Diffstat output

 ./arch/x86_64/kernel/signal.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86_64/kernel/signal.c +1 -1
diff .prev/arch/x86_64/kernel/signal.c ./arch/x86_64/kernel/signal.c
@@ -331,7 +331,7 @@ handle_signal(unsigned long sig, siginfo
 	/* Are we from a system call? */
 	if ((long)regs->orig_rax >= 0) {
 		/* If so, check system call restarting.. */
-		switch (regs->rax) {
+		switch ((long)regs->rax) {
 		        case -ERESTART_RESTARTBLOCK:
 			case -ERESTARTNOHAND:
 				regs->rax = -EINTR;
#8 lore raw

Re: PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).

Quoted 6 lines · Neil Brown
Quoted 5 lines · On Wednesday January 10, [email protected] wrote:

In looking at the Linux code for ERESTARTNOHAND, I see that include/linux/errno.h says this errno should never make it to the user. However, in this instance we ARE seeing it. Looking around on google shows others are seeing it as well, though hits are few.

Quoted 8 lines

..

Quoted 2 lines

Thoughts?

Just a 'me too' at this point.
The X server on my shiny new notebook (Core 2 Duo) occasionally dies with 'select' repeatedly returning ERESTARTNOHAND. It is most annoying!

Normally it should be only visible in strace. Did you see it without strace?

Quoted 10 lines

You don't mention in the Email which kernel version you use but I see from the web page you reference it is 2.6.19.1. I'm using 2.6.18.something.

I thought I'd have a quick look at the code, comparing i386 to x86-64 and guess what I found.....

On x86-64, regs->rax is "unsigned long", so the following is needed....

regs->rax is unsigned long.
I don't think your patch will make any difference. What do you think it will change?

-Andi

#9 lore raw

Re: PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).

Quoted 7 lines · On Thursday January 11, [email protected] wrote:
Quoted 4 lines

Just a 'me too' at this point.
The X server on my shiny new notebook (Core 2 Duo) occasionally dies with 'select' repeatedly returning ERESTARTNOHAND. It is most annoying!

Normally it should be only visible in strace. Did you see it without strace?

No, only in strace.

Quoted 9 lines
Quoted 9 lines

You don't mention in the Email which kernel version you use but I see from the web page you reference it is 2.6.19.1. I'm using 2.6.18.something.

I thought I'd have a quick look at the code, comparing i386 to x86-64 and guess what I found.....

Quoted 6 lines
Quoted 2 lines

On x86-64, regs->rax is "unsigned long", so the following is needed....

regs->rax is unsigned long.
I don't think your patch will make any difference. What do you think it will change?

If regs->rax is unsigned long, then I would think the compiler would be allowed to convert

   switch (regs->rax) {
	case -514 : whatever;

}

to a no-op, as regs->rax will never have a negative value.

However it appears that the current compiler doesn't make that optimisation so I guess I was too hasty.

Still, I think it would be safer to have the cast, in case the compiler decided to be clever.... or does the C standard ensure against that?

Sorry for the noise,

NeilBrown

#10 lore raw

Re: PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).

Quoted 10 lines · Neil Brown
Quoted 7 lines · On Thursday January 11, [email protected] wrote:
Quoted 4 lines

Just a 'me too' at this point.
The X server on my shiny new notebook (Core 2 Duo) occasionally dies with 'select' repeatedly returning ERESTARTNOHAND. It is most annoying!

Normally it should be only visible in strace. Did you see it without strace?

No, only in strace.

strace leaks internal errors. At some point that should be fixed, but it's not really a serious problem.

There was one other report of internal errors leaking without strace, but it was vague and I never got confirmation.

Quoted 2 lines

Still, I think it would be safer to have the cast, in case the compiler decided to be clever.... or does the C standard ensure against that?

It does.

-Andi

#11 lore raw

Re: PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).

In reply to Neil Brown
Quoted 5 lines · Neil Brown
Quoted 2 lines · On Thursday January 11, [email protected] wrote:

Normally it should be only visible in strace. Did you see it without strace?

No, only in strace.

I am absolutely seeing it outside of strace. It is showing up as an errno to the select call:

   if (select(0, (fd_set *)0, (fd_set *)0, (fd_set *)0, &t) != 0) {
      if (errno != EINTR) {
         PyErr_SetFromErrno(PyExc_IOError);
         return -1;
      }

This code is seeing errno=514.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

You don't mention in the Email which kernel version you use but I see from the web page you reference it is 2.6.19.1. I'm using

The production system is running CentOS 4.4, 2.6.9 kernel. However, it looks to be the same issue all the way up to 2.6.19.1, and google shows reports of it on 2.6.17.

Thanks,
Sean

Signature · 4 lines
 George Washington was first in war, first in peace -- and first to
 have his birthday juggled to make a long weekend.
Sean Reifschneider, Member of Technical Staff <[email protected]>
tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability
#12 lore raw

Re: PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).

In reply to Neil Brown
Quoted 8 lines · Neil Brown

If regs->rax is unsigned long, then I would think the compiler would be allowed to convert

   switch (regs->rax) {
	case -514 : whatever;
   }

to a no-op, as regs->rax will never have a negative value.

In C, you never actually compare different types. They always promoted to some common type first.

both sides of (impicit) == here get promoted to "biggest" integer, in this case, to unsigned long. "-514" is an int, so it gets sign extended to the width of "long" and then converted to unsigned long.
--
vda

#13 lore raw

Re: PATCH - x86-64 signed-compare bug, was Re: select() setting ERESTARTNOHAND (514).

In reply to Neil Brown

From: Neil Brown <[email protected]>
Date: Thu, 11 Jan 2007 11:37:05 +1100

Quoted 4 lines

On x86-64, regs->rax is "unsigned long", so the following is needed....

I haven't tried it yet.

Doesn't type promotion take care of that? Did you verify that assember?

I checked the assembler on sparc64 for similar constructs and it does the right thing.