Spurious -EIO when reading a file being written with O_DIRECT?
Hello!
We were reported a problem where if a file being written in directio mode and being read at the same time (in "normal/buffered" mode), then reading process gets -EIO when near the end of file.
Initially I thought this is reiserfs-only problemm and digged in that direction, but then it turned out reiserfs does everything correctly
and the VFS itself seems to be racey (my current suspiction is directio process uses get_block() that extends the file <schedule> reading process gets the buffer and submits io, then waits for page to become uptodate <schedule> direct io process unmaps buffer's metadata
As a result - that page never becomes uptodate and we get -EIO from do_generic_file_read. ) If I take i_sem around call to do_generic_file_read in generic_file_read (in 2.4.21-pre10), that of course helps (this is of course not a correct fix, but just a demonstration that some VFS race is in place).
The same problem can be observed on ext2 in both 2.4.21-pre10 and in 2.6.0-test2 Attached is test_directio.c program, compile it and run with some filename as argument, immediately start "tail" with same filename and you'd get almost immediate I/O error from tail on 2.4 and you'd get same I/O error in 2.6 only after some more waiting.
Is this something known and expected (or may be somebody have a fix already? ;) )?
Bye,
Oleg
#include <stdio.h>
#include <fcntl.h>
#include <stdint.h>
#define BLOCK_SIZE (4096)
#warning ARCH DEPENDENT fixup for your arch
#define O_DIRECT 040000 // ARCH DEPENDENT fixup for your arch */
static char buf[128 * BLOCK_SIZE];
int main(int argc, char** argv)
{
int fd;
char* aligned_buf = (char*)(( (uintptr_t)&buf + (BLOCK_SIZE-1)) & ~(BLOCK_SIZE-1));
int aligned_size = sizeof(buf) - BLOCK_SIZE;
if (-1 == ( fd = open (argv[1], O_RDWR | O_CREAT | O_DIRECT))) {
perror("open: ");
return 1;
}
while (1) {
if( aligned_size!=write(fd, aligned_buf, aligned_size)) {
perror("write: ");
return 1;
}
}
}Quoted 20 lines · Oleg Drokin
We were reported a problem where if a file being written in directio mode
and being read at the same time (in "normal/buffered" mode), then reading process gets-EIOwhen near the end of file.Initially I thought this is reiserfs-only problemm and digged in that direction, but then it turned out reiserfs does everything correctly
and the VFS itself seems to be racey (my current suspiction is directio process usesget_block()that extends the file <schedule> reading process gets the buffer and submits io, then waits for page to become uptodate <schedule> direct io process unmaps buffer's metadata
As a result - that page never becomes uptodate and we get-EIOfromdo_generic_file_read. ) If I takei_semaround call todo_generic_file_readingeneric_file_read(in 2.4.21-pre10), that of course helps (this is of course not a correct fix, but just a demonstration that some VFS race is in place).
The same problem can be observed on ext2 in both 2.4.21-pre10 and in 2.6.0-test2 Attached istest_directio.cprogram, compile it and run with some filename as argument, immediately start "tail" with same filename and you'd get almost immediate I/O error from tail on 2.4 and you'd get same I/O error in 2.6 only after some more waiting.
Quoted 1 line
Is this something known and expected (or may be somebody have a fix already? ;) )?
Test a current 2.4 kernel - it has lots of redone O_DIRECT-vs-buffered locking.
A 2.6 forward-port of that was done by Badari but I lost it and need to find it again.
Hello!
Quoted 19 lines · Andrew Morton
Quoted 4 lines
We were reported a problem where if a file being written in directio mode
and being read at the same time (in "normal/buffered" mode), then reading process gets-EIOwhen near the end of file.Quoted 15 lines
Initially I thought this is reiserfs-only problemm and digged in that direction, but then it turned out reiserfs does everything correctly
and the VFS itself seems to be racey (my current suspiction is directio process usesget_block()that extends the file <schedule> reading process gets the buffer and submits io, then waits for page to become uptodate <schedule> direct io process unmaps buffer's metadata
As a result - that page never becomes uptodate and we get-EIOfromdo_generic_file_read. ) If I takei_semaround call todo_generic_file_readingeneric_file_read(in 2.4.21-pre10), that of course helps (this is of course not a correct fix, but just a demonstration that some VFS race is in place).
The same problem can be observed on ext2 in both 2.4.21-pre10 and in 2.6.0-test2 Attached istest_directio.cprogram, compile it and run with some filename as argument, immediately start "tail" with same filename and you'd get almost immediate I/O error from tail on 2.4 and you'd get same I/O error in 2.6 only after some more waiting. Is this something known and expected (or may be somebody have a fix already? ;) )?
Quoted 2 lines
Test a current 2.4 kernel - it has lots of redone O_DIRECT-vs-buffered locking.
Stupid me.
I mean I tested with 2.4.22-pre10 which is pretty current.
I mean, there were no changes to buffers code in between 2.4.22-pre10 .. 2.4.22-rc1
Quoted 2 lines
A 2.6 forward-port of that was done by Badari but I lost it and need to find it again.
Since it did not help 2.4.22 code, I think 2.6.0 won't benefit from it too. The testcase is really easy and everyone can reproduce the problem easily.
Bye,
Oleg