PROBLEM: I/O system call never returns if file desc is closed in the meantime

[1.] I/O system call never returns if file desc is closed in the meantime

[2.] Full description of the problem/report:

This report describes a problem in the usage of file descriptors across multiple threads. When one thread closes a file descriptor, another thread which waits for an I/O on that file descriptor is not notified and blocks forever.

I don't know exactly what should be considered as "correct behavior" but all other operating systems we use (Sun Solaris, HP-UX, OpenVMS and Windows NT) have the exact same behavior. Linux is the only one to hang.

Context: We use multi-threaded applications. Some threads perform I/O on file descriptors (usually sockets, but not only sockets). Each thread manages one file descriptor (a "communication"). A manager thread detects application-defined conditions and may decide to interrupt a communication. To interrupt the communication, this manager thread closes the file descriptor. If the close operation occurs while the "communication" thread is waiting for data on this file descriptor (using poll(2) for instance), this thread never returns from the I/O system call and hangs forever.

On all other operating systems we use, the I/O system call in the "communication" thread returns and reports an error.

See example program below.

[3.] Keywords

file-descriptor thread async-close

[4.] Kernel version (from /proc/version):

Linux version 2.4.2-2 ([email protected]) (gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-79)) #1 Sun Apr 8 20:41:30 EDT 2001

[5.] Output of Oops.. message:

None.

[6.] A small shell script or example program which triggers the problem

Here is an example program in C.

using poll(2). Since no packet is received on this UDP port. poll blocks.

closed (while the main thread is suspended on poll).

On Solaris and HP-UX (as well as equivalent programs on OpenVMS and Windows NT), the poll returns in the main thread and reports an error on the file descriptor. On Linux, the poll never returns and the main thread is blocked forever.

This example uses a socket but it works exactly the same way using a pipe or any device which may block long enough.

--> Output on Solaris 5.8 and HP-UX B.11.00

main: starting poll
close_thread: starting thread
close_thread: closing socket
close_thread: close status = 0
main: poll status = 1
main: events = 00000001
main: revents = 00000020

... and the program exits.

--> Output on Linux RedHat 7.1 (kernel 2.4.2-2)

main: starting poll
close_thread: starting thread
close_thread: closing socket
close_thread: close status = 0

... and the program remains blocked.

--> Source code

/*
* Test the asynchronous close of an UDP socket.
*

 *  Linux   : gcc -o asclose_udp_c asclose_udp_c.c -lpthread
 *  Solaris : cc -o asclose_udp_c asclose_udp_c.c -lpthread -lsocket -lnsl
 *  HP-UX   : cc -o asclose_udp_c asclose_udp_c.c -lpthread

*/

#include <sys/types.h>
#include <sys/time.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <poll.h>
#include <pthread.h>

#define UDP_PORT 6666
#define CLOSE_DELAY 2 /* seconds */

/* UDP socket descriptor */

static int sock_fd = -1;

/* This procedure is a thread which closes the socket after a delay */

static void* close_thread (void *arg)

{
    int status;
    struct timeval now;
    struct timezone tz;
    struct timespec timeout;
    pthread_cond_t cond = PTHREAD_COND_INITIALIZER;
    pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;

    printf ("close_thread: starting thread\n");

    /* Wait for the specified delay */

    pthread_mutex_lock (&mutex);
    gettimeofday (&now, &tz);
    timeout.tv_sec = now.tv_sec + CLOSE_DELAY;
    timeout.tv_nsec = now.tv_usec * 1000;
    pthread_cond_timedwait (&cond, &mutex, &timeout);
    pthread_mutex_unlock (&mutex);
   
    /* Close the socket while the main task is blocked in a poll(2) */

    printf ("close_thread: closing socket\n");
    status = close (sock_fd);
    printf ("close_thread: close status = %d\n", status);
    
    return NULL;
}

/* Main program */

int main (int argc, char *argv[])

{
    int status;
    pthread_t th;
    struct pollfd fdset;
    struct sockaddr_in saddr;

    /* Create a UDP socket */

    if ((sock_fd = socket (AF_INET, SOCK_DGRAM, IPPROTO_UDP)) < 0) {
        perror ("socket");
        exit (EXIT_FAILURE);
    }

    saddr.sin_family = AF_INET;
    saddr.sin_port = htons (UDP_PORT);
    saddr.sin_addr.s_addr = 0;

    if (bind (sock_fd, (struct sockaddr*) &saddr, sizeof (saddr)) < 0) {
        perror ("bind");
        exit (EXIT_FAILURE);
    }

    /* Create a new thread to asynchronously close the socket */

    if ((status = pthread_create (&th, NULL, close_thread, NULL)) != 0) {
        fprintf (stderr, "pthread_create: %s\n", strerror (status));
        exit (EXIT_FAILURE);
    }

    /* Wait for input data using poll(2) and an infinite timeout */

    fdset.fd = sock_fd;
    fdset.events = POLLIN;
    fdset.revents = 0;

    printf ("main: starting poll\n");
    status = poll (&fdset, 1, -1);
    printf ("main: poll status = %d\n", status);
    printf ("main: events   = %08X\n", fdset.events);
    printf ("main: revents  = %08X\n", fdset.revents);
}

[7.] Environment

[7.1.] Software (add the output of the ver_linux script here)

Linux penguin.canal-plus.fr 2.4.2-2 #1 Sun Apr 8 20:41:30 EDT 2001 i686 unknown

Gnu C                  2.96
Gnu make               3.79.1
binutils               2.10.91.0.2
util-linux             2.11b
modutils               2.4.2
e2fsprogs              1.19
PPP                    2.4.0
isdn4k-utils           3.1pre1
Linux C Library        2.2.2
Dynamic linker (ldd)   2.2.2
Procps                 2.0.7
Net-tools              1.57
Console-tools          0.3.3
Sh-utils               2.0
Modules Loaded         ide-cd cdrom i810 agpgart autofs nfs lockd sunrpc e100 3c59x ipchains usb-uhci usbcore

[7.2.] Processor information (from /proc/cpuinfo):

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 8
model name : Pentium III (Coppermine)
stepping : 6
cpu MHz : 797.433
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1592.52

[7.3.] Module information (from /proc/modules):

ide-cd                 26848   0 (autoclean)
cdrom                  27232   0 (autoclean) [ide-cd]
i810                   80256   1
agpgart                23392   7 (autoclean)
autofs                 11264   1 (autoclean)
nfs                    79008   1 (autoclean)
lockd                  52464   1 (autoclean) [nfs]
sunrpc                 61328   1 (autoclean) [nfs lockd]
e100                   44240   1 (autoclean)
3c59x                  25344   1 (autoclean)
ipchains               38976   0 (unused)
usb-uhci               20720   0 (unused)
usbcore                49664   1 [usb-uhci]

[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)

0000-001f : dma1
0020-003f : pic1
0040-005f : timer
0060-006f : keyboard
0070-007f : rtc
0080-008f : dma page reg
00a0-00bf : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : ide1
01f0-01f7 : ide0
02f8-02ff : serial(auto)
0376-0376 : ide1
03c0-03df : vga+
03f6-03f6 : ide0
03f8-03ff : serial(auto)
0cf8-0cff : PCI conf1
1000-1fff : PCI Bus #02
1000-103f : Intel Corporation 82820 820 (Camino 2) Chipset Ethernet

    1000-101f : e100
  1040-107f : 3Com Corporation 3c900 Combo [Boomerang]
    1040-107f : eth0

2000-20ff : PCI device 8086:2445 (Intel Corporation) 2400-243f : PCI device 8086:2445 (Intel Corporation) 2440-245f : Intel Corporation 82820 820 (Camino 2) Chipset USB (Hub B)
2440-245f : usb-uhci
2460-246f : Intel Corporation 82820 820 (Camino 2) Chipset IDE U100
2460-2467 : ide0
2468-246f : ide1

00000000-0009fbff : System RAM
0009fc00-0009ffff : reserved
000a0000-000bffff : Video RAM area
000c0000-000c7fff : Video ROM
000e0000-000effff : Extension ROM
000f0000-000fffff : System ROM
00100000-07ecffff : System RAM
00100000-002557df : Kernel code
002557e0-0026c80b : Kernel data
07ed0000-07eeffff : ACPI Non-volatile Storage
07ef0000-07efffff : System RAM
40000000-402fffff : PCI Bus #02
40000000-40000fff : Intel Corporation 82820 820 (Camino 2) Chipset Ethernet
40300000-4037ffff : PCI device 8086:1132 (Intel Corporation) 44000000-47ffffff : PCI device 8086:1132 (Intel Corporation) feea0000-ffffffff : reserved

[7.5.] PCI information ('lspci -vvv' as root)

00:00.0 Host bridge: Intel Corporation 82815 815 Chipset Host Bridge and Memory Controller Hub (rev 02)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
Latency: 0
Capabilities: [88] #09 [f104]

00:02.0 VGA compatible controller: Intel Corporation 82815 CGC [Chipset Graphics Controller] (rev 02) (prog-if 00 [VGA])
Subsystem: Compaq Computer Corporation: Unknown device 0019
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin A routed to IRQ 11
Region 0: Memory at 44000000 (32-bit, prefetchable) [size=64M]
Region 1: Memory at 40300000 (32-bit, non-prefetchable) [size=512K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-

00:1e.0 PCI bridge: Intel Corporation 82820 820 (Camino 2) Chipset PCI (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Bus: primary=00, secondary=02, subordinate=02, sec-latency=64
I/O behind bridge: 00001000-00001fff
Memory behind bridge: 40000000-402fffff
Prefetchable memory behind bridge: fff00000-000fffff
BridgeCtl: Parity- SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-

00:1f.0 ISA bridge: Intel Corporation 82820 820 (Camino 2) Chipset ISA Bridge (ICH2) (rev 01)
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0

00:1f.1 IDE interface: Intel Corporation 82820 820 (Camino 2) Chipset IDE U100 (rev 01) (prog-if 80 [Master])
Subsystem: Intel Corporation: Unknown device 2411
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Region 4: I/O ports at 2460 [size=16]

00:1f.4 USB Controller: Intel Corporation 82820 820 (Camino 2) Chipset USB (Hub B) (rev 01) (prog-if 00 [UHCI])
Subsystem: Intel Corporation: Unknown device 2411
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin C routed to IRQ 11
Region 4: I/O ports at 2440 [size=32]

00:1f.5 Multimedia audio controller: Intel Corporation: Unknown device 2445 (rev 01)
Subsystem: Compaq Computer Corporation: Unknown device 000b
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin B routed to IRQ 11
Region 0: I/O ports at 2000 [size=256]
Region 1: I/O ports at 2400 [size=64]

02:08.0 Ethernet controller: Intel Corporation 82820 820 (Camino 2) Chipset Ethernet (rev 01)
Subsystem: Compaq Computer Corporation: Unknown device 0012
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 66 (2000ns min, 14000ns max), cache line size 08
Interrupt: pin A routed to IRQ 5
Region 0: Memory at 40000000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at 1000 [size=64]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=2 PME-

02:0b.0 Ethernet controller: 3Com Corporation 3c900 Combo [Boomerang]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (750ns min, 2000ns max)
Interrupt: pin A routed to IRQ 10
Region 0: I/O ports at 1040 [size=64]
Expansion ROM at <unassigned> [disabled] [size=64K]

[7.6.] SCSI information (from /proc/scsi/scsi)

No SCSI adapter.

[7.7.] Other information that might be relevant to the problem

None.

[X.] Other notes, patches, fixes, workarounds:

None.

THe I/O does not block forever, it blocks until completed. The actual final closure of the object occurs when the last operation on it exits

That's still "forever" if you don't specify a timeout in the select.

Select is defined as to return, with the appropriate bit set, if/when a nonblocking read/write on the file descriptor won't block. You'd get EBADF in this case, therefore causing the select to return would be a Good Thing.

A related problem is that the second thread my be inside a blocking read() instead of a select() call. It'd never continue. :-(

HOWEVER: IMHO it's bad design to distribute the responsibility for file descriptors between threads. Therefore I think that this behavior is a bug, but it's not one that needs to be fixed yesterday.

Matthias Urlichs

Bzzert. Wrong. It may easily block. open() from another thread might grab that descriptor just fine.

If you close descriptors being polled - don't try IO on them once select()/poll() returns. Regardless of aborting select().

Yes. So close() doesn't abort read(). Why would it?

Operations like read, select, etc. act of files. Not on descriptors.

Hi,

Alexander Viro:

Sorry, s/file descriptor/file-or-socket-or-whatever.

Actually trying to read from this descriptor is of course a Bad Thing, but that just means that the programmer forgot to record the file closing in the program's global state.

Well, if it triggers a return from select(), which it currently doesn't do under Linux, then it should also trigger a return from read() (with either zero or -EBADF).

I wonder whether any other Unixes do that..?

Right -- I sort-of implied that. My fault.

Matthias Urlichs     |     noris network AG     |     http://smurf.noris.de/

Then what EBADF are you talking about?

FreeBSD, Plan 9: same as Linux (get file by descriptor, forget about descriptor you've used for that)
OpenBSD, NetBSD: good chance of kernel panic (the whole handling of shared descriptor tables is broken)

the rest: entirely different threads model.

So why would closing a descriptor have any effect on them? struct file is alive and well, held by the operation we are doing.

If you don't want to block until an operation completes, then don't ask to!

That is not quite correct. That is a good approximate definition of
'select's behavior, but it is not exact. As for your assertion that a noblocking read/write wouldn't block, that's not necessarily true. Remember, the 'close' may not have taken affect yet, since the descriptor is still in use by virtue of being selected on.

Perfect. Doing this is absolutely, positively wrong and the more you are
punished for it, the better. It's as wrong as calling 'free' on a chunk of memory when another thread may be usign it. It is impossible to make this work safely, as another thread could open a socket or file and get the same descriptor write before the call to 'read' is entered. There's no possible way to do this because there is no 'unlock a mutex and read' operation.

Why? That's a great design and it's absolutely essential in many cases.
Suppose, for example, I have two descriptors I want to write to. If I assign one thread to each socket permanently, then I'm 100% guaranteed a context switch every time I change which socket I'm writing to, so if there's lots of small bits of data going out both of them, my performance will suck. But if I assign one thread to both socket descriptors, I'm guaranteed that one connection will stall if the the application-level send queue for the other has been swapped out to disk. Not distributing network I/O across threads dynamically is a recipe for either low performance or bursty performance.

DS

How do you avoid race conditions if more than one thread is creating file descriptors? I think you can only do that under very special circumstances, and it definitely requires some synchronization.

Florian Weimer 	                  [email protected]
University of Stuttgart           http://cert.uni-stuttgart.de/
RUS-CERT                          +49-711-685-5973/fax +49-711-685-5898

The same way as you do it for many threads doing any allocations.

There's a subtle difference: For malloc(), libc has a mutex (or whatever), but for open(), socket() etc., no locking is performed, and many libc functions create (and destroy) descriptors imlicitely.

I still don't see how you can write maintainable and reliable software with asynchronous close(). For example, if some select() call returns EBADF after an asynchronous close(), you would have to scan the descriptors to find the offending one, but in the meantime, it has been reused by another thread. What do you do in this case?

Florian Weimer 	                  [email protected]
University of Stuttgart           http://cert.uni-stuttgart.de/
RUS-CERT                          +49-711-685-5973/fax +49-711-685-5898

So? You don't have to close() descriptors you had not (to your code knowledge) opened. End of story.

You don't rely on EBADF. It's _your_ code that had closed the thing. Unless you pass descriptors of unknown origin into select() (hardly a good idea) you have all information you need to provide an exclusion.