Skip to contentSkip to conversation

Multithread select() bug

DISCUSSION Andre Ben Hamou 4 participants 10 messages May 10, 2004 – May 11, 2004

lore
Share & tools
lore
#0 lore raw

Here's the scenario...

- parent thread P creates a connected socket pair S[0, 1]
- P spawns a child thread C and passes it S
- C selects on S[0]
- P closes S[0]

As I understand the semantics of the select call, C should now return immediately in response to the closure (and it does on Mac OS X). However, the following test code behaves otherwise for the two test cases I've tried (2.4.21 and 2.6.5). Compilation command used: 'gcc foobar.c -lpthread'.

Cheers,

Andre Ben Hamou
Imperial College London

--- BEGIN TEST CODE (foobar.c)---

#include <assert.h>         // assert
#include <pthread.h>        // pthread_create
#include <sys/select.h>     // select
#include <sys/types.h>      // socketpair
#include <sys/socket.h>     // socketpair
#include <unistd.h>         // sleep
#include <stdio.h>          // printf
void *threadFuntion (void *sockets) {
    int socket = ((int *)sockets)[0];
    struct timeval timeout = {tv_sec: 5, tv_usec: 0};

    // Allocate a file descriptor set with the passed socket
    fd_set fds;
    FD_ZERO (&fds);
    FD_SET (socket, &fds);

    // Select to read / register exceptions on the FD set
    select (socket + 1, &fds, NULL, &fds, &timeout);

    return NULL;
}
int main (void) {
    int sockets[2];
    pthread_t thread;

    // Create a connected pair of sockets
    assert (socketpair (PF_UNIX, SOCK_STREAM, 0, sockets) != -1);
    printf ("sockets: {%i, %i}\n", sockets[0], sockets[1]);

    // Create a POSIX thread
    // - use the default configuration
    // - invoke 'threadFunction' as the root function of the thread
    // - pass the socket array to 'threadFunction'
    assert (pthread_create (&thread,
			    NULL,
                            threadFuntion,
                            sockets) == 0);

    // Wait for a second and then close the socket being selected on
    sleep (1);
    assert (close (sockets[0]) == 0);
    printf ("Socket closed\n");

    // Wait for the thread to exit - SHOULD BE ~ INSTANTANEOUS
    assert (pthread_join (thread, NULL) == 0);
    printf ("Thread joined\n");

    assert (close (sockets[1]) == 0);
    return 0;
}

--- END TEST CODE ---

Signature · 1 line
...and, on the seventh day, God switched off his Mac.
#1 lore raw
Quoted 11 lines · Andre Ben Hamou
void *threadFuntion (void *sockets) {
     int socket = ((int *)sockets)[0];
     struct timeval timeout = {tv_sec: 5, tv_usec: 0};

     // Allocate a file descriptor set with the passed socket
     fd_set fds;
     FD_ZERO (&fds);
     FD_SET (socket, &fds);

     // Select to read / register exceptions on the FD set
     select (socket + 1, &fds, NULL, &fds, &timeout);

Try:

	select (socket + 1, &fds, &fds, &fds, &timeout);
                                 ^^^^^

- Davide

#2 lore raw
Quoted 4 lines · Davide Libenzi

Try:

	select (socket + 1, &fds, &fds, &fds, &timeout);
                                 ^^^^^

That does work, but only as a workaround (and not a universally applicable one). Select *should* return upon the closure of either end of a socket connection that is in the read-FD-set only (unless I've completely misunderstood the various references).

Cheers,

Andre Ben Hamou
Imperial College London

Signature · 1 line
...and, on the seventh day, God switched off his Mac.
#3 lore raw
Quoted 10 lines · Andre Ben Hamou
Quoted 4 lines · Davide Libenzi

Try:

	select (socket + 1, &fds, &fds, &fds, &timeout);
                                 ^^^^^

That does work, but only as a workaround (and not a universally applicable one). Select *should* return upon the closure of either end of a socket connection that is in the read-FD-set only (unless I've completely misunderstood the various references).

The standard says that select should return when the operation against the fd would not block *or* would return an error different from would-have-blocked. The following operation on a closed fd would be an EBADF, that fits the standard description. OTOH, MT is messy. Consider a thread that closes the fd and immediately after open another one having the same fd#. Bottom line is, don't do it.

- Davide

#4 lore raw
In reply to Andre Ben Hamou
Quoted 7 lines · Andre Ben Hamou

Here's the scenario...

- parent thread P creates a connected socket pair S[0, 1]
- P spawns a child thread C and passes it S
- C selects on S[0]
- P closes S[0]

Your program is racy and have undefined behavior.

A thread should not close a handle 'used by another thread blocked in a sytemcall'

The race is : if a thread does a close(fd), then the fd value may be reused by another thread during an open()/socket()/dup()... syscall, and the first thread could issue the select() syscall (or read()/write()/...) on the bad file.

Some Unixes defines different semantics (Solaris comes to mind), but these semantics are not part of standards.

Eric

Quoted 65 lines

As I understand the semantics of the select call, C should now return immediately in response to the closure (and it does on Mac OS X). However, the following test code behaves otherwise for the two test cases I've tried (2.4.21 and 2.6.5). Compilation command used: 'gcc foobar.c -lpthread'.

Cheers,

Andre Ben Hamou
Imperial College London

--- BEGIN TEST CODE (foobar.c)---

#include <assert.h>         // assert
#include <pthread.h>        // pthread_create
#include <sys/select.h>     // select
#include <sys/types.h>      // socketpair
#include <sys/socket.h>     // socketpair
#include <unistd.h>         // sleep
#include <stdio.h>          // printf
void *threadFuntion (void *sockets) {
    int socket = ((int *)sockets)[0];
    struct timeval timeout = {tv_sec: 5, tv_usec: 0};

    // Allocate a file descriptor set with the passed socket
    fd_set fds;
    FD_ZERO (&fds);
    FD_SET (socket, &fds);

    // Select to read / register exceptions on the FD set
    select (socket + 1, &fds, NULL, &fds, &timeout);

    return NULL;
}
int main (void) {
    int sockets[2];
    pthread_t thread;

    // Create a connected pair of sockets
    assert (socketpair (PF_UNIX, SOCK_STREAM, 0, sockets) != -1);
    printf ("sockets: {%i, %i}\n", sockets[0], sockets[1]);

    // Create a POSIX thread
    // - use the default configuration
    // - invoke 'threadFunction' as the root function of the thread
    // - pass the socket array to 'threadFunction'
    assert (pthread_create (&thread,
                NULL,
                            threadFuntion,
                            sockets) == 0);

    // Wait for a second and then close the socket being selected on
    sleep (1);
    assert (close (sockets[0]) == 0);
    printf ("Socket closed\n");

    // Wait for the thread to exit - SHOULD BE ~ INSTANTANEOUS
    assert (pthread_join (thread, NULL) == 0);
    printf ("Thread joined\n");

    assert (close (sockets[1]) == 0);
    return 0;
}

--- END TEST CODE ---

#5 lore raw
Quoted 5 lines · Eric Dumazet

Your program is racy and have undefined behavior.

A thread should not close a handle 'used by another thread blocked in a sytemcall'

The race is : if a thread does a close(fd), then the fd value may be reused by another thread during an open()/socket()/dup()... syscall, and the first thread could issue the select() syscall (or read()/write()/...) on the bad file.

Apologies, but I don't follow this.

It was my understanding that the (potentially) many threads of a single process all share a canonical file descriptor table. Hence as long as the various calls you mention are issued in a guaranteed order, maintaining state as you go (which is what the 1 second sleep in the test code was a very quick and dirty way to almost do), I don't see how a race condition arises.

If I were to replace the sleep (1) with, say a global semaphore or something similar, would your explanation still hold?

Cheers,

Andre Ben Hamou
Imperial College London

Signature · 1 line
...and, on the seventh day, God switched off his Mac.
#6 lore raw
Quoted 6 lines · Andre Ben Hamou
Quoted 4 lines · Eric Dumazet

Your program is racy and have undefined behavior.

A thread should not close a handle 'used by another thread blocked in a sytemcall'

Quoted 9 lines
Quoted 1 line

The race is : if a thread does a close(fd), then the fd value may be reused by another thread during an open()/socket()/dup()... syscall, and the first thread could issue the select() syscall (or read()/write()/...) on the bad file.

Apologies, but I don't follow this.

It was my understanding that the (potentially) many threads of a single process all share a canonical file descriptor table. Hence as long as the various calls you mention are issued in a guaranteed order, maintaining state as you go (which is what the 1 second sleep in the test code was a very quick and dirty way to almost do), I don't see how a race condition arises.

If I were to replace the sleep (1) with, say a global semaphore or something similar, would your explanation still hold?

So please how do you guarantee that thread 1 runs *before* thread 2)

Thread 1)
select( fd)

Thread 2)
close(fd)

Thats not possible.

Only pthread synchronization are mutexes (or rwlocks, or semaphore). And you cannot release mutex/rwlock/semaphore *after* entering Thread1) select()

So yes, there is a race condition.

In you example, it can happens that thread 1 must sleep 10 seconds before calling select(), because of system scheduling. Your sleep(1) cannot garant that the close() is done after the select() call blocked into kernel. You can try whatever semaphore you want, you wont be able to have a 100% reliable program (even on Solaris)

Eric

Quoted 5 lines

Cheers,

Andre Ben Hamou
Imperial College London

#7 lore raw
Quoted 10 lines · Eric Dumazet

So please how do you guarantee that thread 1 runs *before* thread 2)

Thread 1)
select( fd)

Thread 2)
close(fd)

Thats not possible.

I see where you're coming from, in that there is a potential race condition as to the socket being connected as I reach the select call.

This is an important concern but it is, I think, orthogonal to the original problem as there are two possible socket states at the point at which select gets called (as far as I can see)...

1. The socket is in its connected state
2. The socket has already been closed by the parent thread

As I understand it, if 1 is true (which corresponds to my original post), then select should return the moment the socket gets closed and, if 2 is true (which I believe corresponds to your concern), then select should return immediately anyway as the socket would not block if read from.

Sorry to be a pest, but I'm trying to get this clear in my head. Is it possible I've over-estimated the thread-safety of the select and close calls?

Cheers,

Andre Ben Hamou
Imperial College London

Signature · 1 line
...and, on the seventh day, God switched off his Mac.
#8 lore raw
Quoted 11 lines · Andre Ben Hamou
Quoted 10 lines · Eric Dumazet

So please how do you guarantee that thread 1 runs *before* thread 2)

Thread 1)
select( fd)

Thread 2)
close(fd)

Thats not possible.

Some time ago I sent a patch for the select() systemcall to return if the last fd was closed. If select() specifies more than one fd, then staying in select is valid, but if there is no more fd left to wait on, select() should return I think. I don't know what standards say here or other Unix do in that case, so I did't have a good reason for the patch and it was ignored.
The patch was for 2.4, but 2.6 is the same and poll() as well.

Armin

#9 lore raw
In reply to Andre Ben Hamou
Quoted 12 lines · Andre Ben Hamou

I see where you're coming from, in that there is a potential race condition as to the socket being connected as I reach the select call.

This is an important concern but it is, I think, orthogonal to the original problem as there are two possible socket states at the point at which select gets called (as far as I can see)...

1. The socket is in its connected state
2. The socket has already been closed by the parent thread

As I understand it, if 1 is true (which corresponds to my original post), then select should return the moment the socket gets closed and, if 2 is true (which I believe corresponds to your concern), then select should return immediately anyway as the socket would not block if read from.

Sorry to be a pest, but I'm trying to get this clear in my head. Is it possible I've over-estimated the thread-safety of the select and close calls?

As the safety of what you are trying to do cannot be guaranted, you should not even try to do that.
Then, apart from this safety showstopper, you should consider that what you assumed is simply wrong :
- Some Operating Systems do something special to signal blocked threads (blocked on a read()/write()/select()/poll()) if the underlying file handle is closed.
- linux does NOT. And this is done for several reasons. These reasons are known and were discussed on this list at several occasions.

So the rule is : close() syscall should be serialized, according to a synchronization of your choice, so that no other thread could be using the file descriptor at the same time.
If you want to 'send some signal to the blocked thread' in a portable way, you could setup a pipe to be able to send a special message on it. The 'worker thread' could add the read side of the pipe (pipefd[0]) to its select fd set, and will be notified of the Thread 2) willing to dismantle your socket, using a standard write(pipefd[1]). This will be portable : When the 'worker thread' returns from select() call, it can then close the socket fd himself.

By the way, prefer poll() syscall to select() one. Poll is much more scalable/efficient if you have only one or two fd to monitor.

See you
Eric

Quoted 5 lines

Cheers,

Andre Ben Hamou
Imperial College London