Busy-wait delay in qmail 1.03 after upgrading to Linux 2.6

What Qmail did was basically to use a named pipe as a trigger, where one program select()s on the FIFO file descriptor, waiting for another program to write() the FIFO. Once select() returns, the listener close()s the FIFO (the data was not important, it was only used as a signal), does some work, then re-open()s the FIFO file, and ends up in the same select() waiting for the whole thing to happen again.

      for (;;) {
               while ((fd = open("test.fifo", O_WRONLY | O_NONBLOCK)) < 0)
                       ;
               gettimeofday(&tv1, NULL);
               if (write(fd, &fd, 1) == 1) {

                       gettimeofday(&tv2, NULL);
                       fprintf(stderr, "dt = %f ms\n",
                               (tv2.tv_sec - tv1.tv_sec) * 1000.0 +
                               (tv2.tv_usec - tv1.tv_usec) / 1000.0);
               }
               if (close(fd) < 0) {
                       perror("close");

What drains the fifo?
As far as I can see the fifo is filled by the write syscalls, and drained by chance if both the reader and the writer have closed their handles.

Quoted 6 lines

     for (;;) {
              while ((fd = open("test.fifo", O_WRONLY | O_NONBLOCK)) < 0)
                      ;
              gettimeofday(&tv1, NULL);
              if (write(fd, &fd, 1) == 1) {

xxx now a thread switch

Quoted 9 lines

                      gettimeofday(&tv2, NULL);
                      fprintf(stderr, "dt = %f ms\n",
                              (tv2.tv_sec - tv1.tv_sec) * 1000.0 +
                              (tv2.tv_usec - tv1.tv_usec) / 1000.0);
              }
              if (close(fd) < 0) {
                      perror("close");

If a thread switch happens in the indicated line, then the reader will loop, until it's timeslice expires - one full timeslice delay between the two gettimeofday() calls.

Running the reader with nice -20 resulted in delays of 200-1000 ms for each write call, nice 20 resulted in no slow calls. In both cases 100% cpu load.

[Manfred Spraul]

Quoted 3 lines

If a thread switch happens in the indicated line, then the reader will loop, until it's timeslice expires - one full timeslice delay between the two gettimeofday() calls.

Exactly. But on 2.6, the delay between the two gettimeofday() calls are sometimes up to 300 ms, which is 300 timeslices in 2.6, right? I have never observed more than _one_ timeslice delay in 2.4.

Quoted 3 lines

Running the reader with nice -20 resulted in delays of 200-1000 ms for each write call, nice 20 resulted in no slow calls. In both cases 100% cpu load.

But when the listener and the writer have the same nice value, how is it possible to have a delay of 300 ms? Both the writer and the listener are ready to run, so wouldn't a 300 ms delay mean that the listener was given the CPU 300 times in a row?