[Fwd: Weird ia64 problem]
It was suggested to me that this might be of interest outside the LSB, and that I should forward it more widely...
[It seems I've been using the wrong address for the list, and for some reason haven't been seeing bounces. This is a summary of my past attempts to send to the list. I apologize if this is a dup, or if my past messages make their way out of whatever limbo they are stuck in now.]
I've been seeing some weird test failures on ia64, namely:
/tset/ANSI.os/streamio/fwrite/T.fwrite 3 UNRESOLVED /tset/LSB.os/ioprim/writev_L/T.writev_L 13 FAIL /tset/POSIX.os/ioprim/write/T.write 13 FAIL
These all appear to be caused by the same problem: getting EAGAIN on a non-blocking pipe write when not expected. At first, I thought it might just be a test bug, since the behavior is legitimate according to the spec, but one of the tests (T.fwrite) actually looks specifically for this condition. What's weirder: I was seeing these faults on a DCC system, but not on Debian stable.
I was able to write a standalone test that reproduces the problem. (The C source is attached.) Basically, the tests seem to rely on the ability of a full pipe to receive more input (and thus unblock) once some of the data in the pipe is read. It seems that newer ia64 kernels don't do this.
On all Debian-based i386 and amd64 systems I've run it on, my test prints "the write appeared to succeed", which indicated the LSB tests above should pass. Running on any ia64 system with the stock Debian stable kernel (2.6.8) produces the same results. But running my test on either the 2.6.12 DCC kernel or the latest 2.6.12 kernel from Debian unstable prints "the write failed; EAGAIN seen", which corresponds to the weird results above.
Has anyone else seen this? My searches for a cause have come up fruitless so far. I'm reduced to building kernels 2.6.9 through 2.6.11 to try and find the exact place where the regression was introduced.
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
int main(int argc, char *argv[])
{
int pipes[2];
int flags;
long pipe_buf;
size_t bufsize1, bufsize2, nitems;
FILE *fp;
char *bp;
if (pipe(pipes) != 0) {
fprintf(stderr, "error: could not create pipe\n");
return 1;
}
flags = fcntl(pipes[1], F_GETFL, 0);
flags |= O_NONBLOCK;
fcntl(pipes[1], F_SETFL, flags);
pipe_buf = fpathconf(pipes[1], _PC_PIPE_BUF);
if (pipe_buf <= 0L) {
printf("cannot test: unlimited pipe buffer\n");
return 0;
}
bufsize1 = (size_t) pipe_buf + 16;
bufsize2 = bufsize1 + 8;
fp = fdopen(pipes[1], "w");
if (fp == NULL) {
fprintf(stderr, "error: could not reopen pipe\n");
return 1;
}
setbuf(fp, (char *) NULL);
bp = (char *) malloc(bufsize2);
if (bp == NULL) {
fprintf(stderr, "error: could not allocate memory\n");
return 1;
}
while (write(pipes[1], bp, (unsigned) pipe_buf/2) == pipe_buf/2) ;
read(pipes[0], bp, (unsigned) pipe_buf);
/* here we go */
nitems = fwrite((void *) bp, 1, (size_t) bufsize2, fp);
if (nitems > 0) {
printf("the write appeared to succeed\n");
} else if (ferror(fp)) {
if (errno == EAGAIN) {
printf("the write failed; EAGAIN seen\n");
} else {
printf("the write failed; error: %s\n", strerror(errno));
}
} else {
printf("something weird happened\n");
}
return 0;
}Quoted 3 lines
It was suggested to me that this might be of interest outside the LSB, and that I should forward it more widely...
Running under strace, I see you write 2048 bytes at a time until you see EAGAIN. Then you read 4096 bytes to empty the pipe a little.
Then you try to write 4120 bytes (more than you just emptied).
Is that what you are trying to do?
Do you expect a partial write, rather than the EAGAIN?
-Tony
Quoted 1 line
Do you expect a partial write, rather than the EAGAIN?
I suppose you should expect EAGAIN here: http://tinyurl.com/82wr9
describes this situation quite clearly.
So either:
1) ia64 thinks 4120 is less than PIPE_BUF, so that it believes that it should not do a partial write
2) Even though we removed some data from the pipe, it thinks that it is still all the way full.
[My tests on 2.6.14-rc2].
Which other architectures have you tried this on? Is ia64 all alone in failing this test?
-Tony
Quoted 9 lines · Tony
Quoted 2 lines
It was suggested to me that this might be of interest outside the LSB, and that I should forward it more widely...
Running under strace, I see you write 2048 bytes at a time until you see EAGAIN. Then you read 4096 bytes to empty the pipe a little.
Then you try to write 4120 bytes (more than you just emptied).
Is that what you are trying to do?
That sounds about right.
FWIW, if you'd like a look at the source of one of the LSB tests that's failing, see:
Look at function "test3", specifically the part after where it reports "testing assertion using non-blocking pipe".
Quoted 1 line
Do you expect a partial write, rather than the EAGAIN?
That appears to be what's happening in most other cases, and also appears to be what the LSB tests want.
Quoted 8 lines · Tony
Quoted 1 line
Do you expect a partial write, rather than the EAGAIN?
I suppose you should expect EAGAIN here: http://tinyurl.com/82wr9
describes this situation quite clearly.So either:
1) ia64 thinks 4120 is less thanPIPE_BUF, so that it believes that it should not do a partial write
Except that the test code asks the system for PIPE_BUF, and then adds to it to determine how much to write. (See lines 26, 32, and 33 of my test.) Specifically, we try to write PIPE_BUF + 24 bytes.
Quoted 2 lines
2) Even though we removed some data from the pipe, it thinks that it is still all the way full.
Hmm. The spec doesn't seem to say whether a read of PIPE_BUF bytes on a full pipe _must_ put the pipe in a state where it can accept more input. The test, certainly, seems to think this is mandatory.
Quoted 4 lines
[My tests on 2.6.14-rc2].
Which other architectures have you tried this on? Is ia64 all alone in failing this test?
Yup. In fact, only recent kernels fail; 2.6.8 succeeds, while 2.6.12 fails.
For other architectures, I've tested on i386 and amd64 for both 2.6.8 and 2.6.12 kernels.
Quoted 4 lines
Quoted 1 line
Do you expect a partial write, rather than the EAGAIN?
That appears to be what's happening in most other cases, and also appears to be what the LSB tests want.
Running with a kernel configured for 4K pagesize, this test passes (prints "the write appeared to succeed").
I expect this difference in behavior can be traced to when Linus improved pipe performance by allowing a list of many pages to implement the pipe buffer. See http://tinyurl.com/7jfen
The date on this is Jan 6th, 2005 ... which puts it between 2.6.10 and 2.6.11-rc1
-Tony
Quoted 9 lines · Tony
Running with a kernel configured for 4K pagesize, this test passes (prints "the write appeared to succeed").
I expect this difference in behavior can be traced to when Linus improved pipe performance by allowing a list of many pages to implement the pipe buffer. See http://tinyurl.com/7jfen
The date on this is Jan 6th, 2005 ... which puts it between 2.6.10 and 2.6.11-rc1
This implies that PIPE_BUF should be equal to the kernel's configured page size on ia64. I modified my test to report on PIPE_BUF's returned value, and it is 4096 on all my tested architectures, including ia64 on 2.6.8 and 2.6.12.
(Also, the page size on my 2.6.8 ia64 kernel is 16K, same as my 2.6.12 kernel.)
It strikes me that, if this is correct, forcing 4K page size isn't the right fix. Do you have any ideas on that front?
Quoted 4 lines
This implies that
PIPE_BUFshould be equal to the kernel's configured page size on ia64. I modified my test to report on PIPE_BUF's returned value, and it is 4096 on all my tested architectures, including ia64 on 2.6.8 and 2.6.12.
4096 is possibly for historical reasons.
Quoted 2 lines
(Also, the page size on my 2.6.8 ia64 kernel is 16K, same as my 2.6.12 kernel.)
Yes, the default pagesize on ia64 is 16k.
Quoted 2 lines
It strikes me that, if this is correct, forcing 4K page size isn't the right fix. Do you have any ideas on that front?
My experiment with a 4K page kernel was just that, an experiment. I didn't intend it to be a fix. But it is a good thing to try when some generic test fails on ia64 and works on everything else.
Looking at fs/pipe.c ... it appears that pipe_readv() only wants to let any writers know that we've read anything if we manage to empty one of the page buffers:
if (!buf->len) {
...
do_wakeup = 1;
}Now, the question becomes "is this a bug". As you pointed out in an earlier e-mail the standard makes no mention of what happens when you read some data from a full pipe. So we appear not to be in violation of the letter of the standard.
But this does fly in the face of common sense.
As you say above, reporting the PIPE_BUF value as PAGE_SIZE [probably max(4096, PAGE_SIZE) ... for any arch that has a page size smaller than 4k] would fix this. But then we get back to the historical properties of 4k as the PIPE_BUF size. Would such a change break existing applications that are not well enough written to use fpathconf(fd, _PC_PIPE_BUF)?
-Tony
Quoted 4 lines
This implies that
PIPE_BUFshould be equal to the kernel's configured page size on ia64. I modified my test to report on PIPE_BUF's returned value, and it is 4096 on all my tested architectures, including ia64 on 2.6.8 and 2.6.12.
Hmmph. According to strace, there are no system calls resulting from the fpathconf() call. How does it come up with the magic 4096 value?
-Tony
Quoted 2 lines · Tony
Hmmph. According to strace, there are no system calls resulting from the
fpathconf()call. How does it come up with the magic 4096 value?
Looks like it's built with the value of PIPE_BUF from include/linux/limits.h
Through a binary search, on my system (16K pages) with the original test program, setting pipe_buf to 10923 fails with EAGAIN, but add one more byte (10924) and it works. I can't immediately see how that number relates to anything.
-i
Quoted 5 lines · Tony
Now, the question becomes "is this a bug". As you pointed out in an earlier e-mail the standard makes no mention of what happens when you read some data from a full pipe. So we appear not to be in violation of the letter of the standard.
Quoted 8 lines
But this does fly in the face of common sense.
As you say above, reporting the
PIPE_BUFvalue asPAGE_SIZE[probablymax(4096, PAGE_SIZE)... for any arch that has a page size smaller than 4k] would fix this. But then we get back to the historical properties of 4k as thePIPE_BUFsize. Would such a change break existing applications that are not well enough written to usefpathconf(fd, _PC_PIPE_BUF)?
I think there's a problem with this. While PIPE_BUF is specified in a kernel header, it ends up becoming an embedded value in glibc, from what I can tell. Which also makes it an embedded value in statically linked apps. This looks like a dead end, even for apps that use fpathconf().
Also, from looking at some of the kernel comments (like where PIPE_SIZE is defined), it seems that the kernel powers-that-be also intend to keep PIPE_BUF and PAGE_SIZE decoupled.
So the way forward seems to be to add a test to pipe_readv() for this condition.
I'm thinking it should check if (PIPE_SIZE - buf->len) > PIPE_BUF, and set do_wakeup. The code could be #defined out if PIPE_SIZE = PIPE_BUF, so the change reduces to a no-op on other archs.
Does that sound right? I'm going to work on a patch here.
Quoted 5 lines
I'm thinking it should check if (
PIPE_SIZE-buf->len) >PIPE_BUF, and setdo_wakeup. The code could be #defined out ifPIPE_SIZE=PIPE_BUF, so the change reduces to a no-op on other archs.Does that sound right? I'm going to work on a patch here.
Roughly (without looking back as fs/pipe.c). When you have your patch, you will need to restart this discussion on the linux-kernel mailing list (since you'll be touching a generic file). Some random grepping for PAGE_SIZE (well PAGE_SHIFT) shows that some other artchitectures have some support for page size > PIPE_BUF. So there should be some interest in this fix.
-Tony