SMP read() stopping at memory page boundaries
Tested with various 2.6.x i386/x86-64 SMP kernels and CPUs, for example 2.6.21.3/x86-64.
Process 1:
- lock file
- write(4096 + 16 bytes)
- unlock file
Process 2:
- lock file
- read(8192 bytes)
- unlock file
Sometimes read() returns only 4096 bytes. I'm locking the file, so I don't think this should ever happen, right?
Attached a test program. Takes from a few seconds to half a minute with my computer to print "page size cut".
/*
gcc concurrency2.c -o concurrency -Wall
start two both a reader and a writer:
./concurrency
./concurrency 1
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <assert.h>
#include <sys/file.h>
#define MAX_PAGESIZE 8192
int main(int argc, char *argv[])
{
char buf[MAX_PAGESIZE*2];
int fd, ret, pagesize;
memset(buf, 0, sizeof(buf));
pagesize = getpagesize();
assert(pagesize <= MAX_PAGESIZE);
buf[pagesize] = 'h';
if (argc == 1) {
printf("writing, page size = %d\n", pagesize);
for (;;) {
fd = open("foo", O_RDWR | O_CREAT | O_TRUNC, 0600);
if (fd == -1) {
perror("open()");
return 1;
}
if (flock(fd, LOCK_EX) < 0)
perror("flock()");
write(fd, buf, pagesize+16);
if (flock(fd, LOCK_UN) < 0)
perror("flock()");
usleep(rand() % 1000);
close(fd);
}
} else {
printf("reading, page size = %d\n", pagesize);
fd = open("foo", O_RDWR, 0600);
if (fd == -1) {
perror("open()");
return 1;
}
for (;;) {
usleep(rand() % 1000);
if (flock(fd, LOCK_SH) < 0)
perror("flock()");
lseek(fd, 0, SEEK_SET);
ret = read(fd, buf, sizeof(buf));
if (flock(fd, LOCK_UN) < 0)
perror("flock()");
if (ret < pagesize) {
if (ret > 0)
printf("less than a page: %d\n", ret);
} else if (ret == pagesize) {
printf("page size cut\n");
} else if (buf[pagesize] != 'h') {
printf("broken data\n");
}
}
}
return 0;
}Quoted 2 lines · Timo Sirainen
Sometimes
read()returns only 4096 bytes. I'm locking the file, so I don't think this should ever happen, right?
Sorry, the problem was with file truncating so there's no bug with locking. I didn't notice it first because it happened to work with FreeBSD and Solaris.
Without locking there's a problem that I'd want to avoid, but I guess it's not necessarily a bug (although it doesn't seem to happen with FreeBSD or Solaris):
Process 1:
- create "foo2"
- rename() to "foo"
- lock
- write(4096 + 16 bytes)
- [fsync() here doesn't change anything]
- pwrite("1111" to offset=4096-4)
- unlock
Process 2:
- open("foo")
- read(8192 bytes)
read() sometimes returns 4096 bytes but with the "1111" already included in the data. Is there a way to avoid this without locking the file while reading? The "1111" tries to act as a kind of a lock.
Again attached a test program, which should really do what I intended. :)
/*
gcc concurrency.c -o concurrency -Wall
start two both a reader and a writer:
./concurrency
./concurrency 1
*/
#define _XOPEN_SOURCE 500
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <assert.h>
#include <sys/file.h>
#define MAX_PAGESIZE 8192
int main(int argc, char *argv[])
{
char buf[MAX_PAGESIZE*2], ones[4] = { 1, 1, 1, 1 };
int fd, ret, pagesize;
memset(buf, 0, sizeof(buf));
pagesize = getpagesize();
assert(pagesize <= MAX_PAGESIZE);
buf[pagesize] = 'h';
if (argc == 1) {
printf("writing, page size = %d\n", pagesize);
for (;;) {
fd = open("foo2", O_RDWR | O_CREAT | O_TRUNC, 0600);
if (fd == -1) {
perror("open()");
return 1;
}
if (rename("foo2", "foo") < 0)
perror("rename()");
usleep(rand() % 1000);
if (flock(fd, LOCK_EX) < 0)
perror("flock()");
pwrite(fd, buf, pagesize+16, 0);
//usleep(rand() % 1000);
//fdatasync(fd);
pwrite(fd, ones, 4, pagesize-4);
if (flock(fd, LOCK_UN) < 0)
perror("flock()");
usleep(rand() % 1000);
close(fd);
}
} else {
printf("reading, page size = %d\n", pagesize);
for (;; close(fd)) {
fd = open("foo", O_RDWR, 0600);
if (fd == -1) {
perror("open()");
return 1;
}
ret = pread(fd, buf, sizeof(buf), 0);
if (ret < pagesize) {
if (ret > 0)
printf("less than a page: %d\n", ret);
continue;
}
if (memcmp(buf + pagesize - 4, ones, 4) == 0) {
if (ret == pagesize) {
printf("page size cut\n");
} else if (buf[pagesize] != 'h') {
printf("broken data\n");
}
}
}
}
return 0;
}Quoted 3 lines · Timo Sirainen
Quoted 2 lines · Timo Sirainen
Sometimes
read()returns only 4096 bytes. I'm locking the file, so I don't think this should ever happen, right?
man 2 read
read() is always allowed to return less than you asked for. You need to go back and ask for the rest. That's why people wrap read() and write() in loops, and use those wrapped versions instead.
Quoted 2 lines
read()sometimes returns 4096 bytes but with the "1111" already included in the data.
Uhm, do you mean 'without the 1111'? 4096 with the 1111 sounds perfectly fine. (Though 4096 without the 1111 is also perfectly fine.)
Quoted 2 lines
Is there a way to avoid this without locking the file while reading? The "1111" tries to act as a kind of a lock.
I think you've misunderstood how read and write work. pread and pwrite are no different in this respect -- they just allow an atomic seek+read/write.
Ray
Quoted 10 lines · Ray Lee
Quoted 3 lines · Timo Sirainen
Quoted 2 lines · Timo Sirainen
Sometimes
read()returns only 4096 bytes. I'm locking the file, so I don't think this should ever happen, right?man 2 read
read()is always allowed to return less than you asked for. You need to go back and ask for the rest. That's why people wrapread()andwrite()in loops, and use those wrapped versions instead.
I thought this wouldn't happen when reading from local filesystems. Anyway, my real program was doing that, and it was still seeing partial data. But looks like I can't reproduce it in my test program with two pread()s, so I'll have to do some more debugging.
Quoted 4 lines
I thought this wouldn't happen when reading from local filesystems. Anyway, my real program was doing that, and it was still seeing partial data. But looks like I can't reproduce it in my test program with two pread()s, so I'll have to do some more debugging.
It will occur if you are reading as someone else changes the file size. Use file locking, it exists for a reason ;)
Alan
Quoted 4 lines · Alan Cox
Quoted 4 lines
I thought this wouldn't happen when reading from local filesystems. Anyway, my real program was doing that, and it was still seeing partial data. But looks like I can't reproduce it in my test program with two pread()s, so I'll have to do some more debugging.
Found the problem in my code. A buggy program, buggy test program and a buggy brain while debugging doesn't seem to give the best results. :)
Quoted 2 lines
It will occur if you are reading as someone else changes the file size. Use file locking, it exists for a reason ;)
Annoying extra overhead. Especially with NFS, when nowadays you can't even use flock() to create local locks..
Quoted 4 lines · Timo Sirainen
Quoted 2 lines
It will occur if you are reading as someone else changes the file size. Use file locking, it exists for a reason ;)
Annoying extra overhead. Especially with NFS, when nowadays you can't even use
flock()to create local locks..
It's not only "nowadays", it has been like that for quite a some time.
On the other hand you can use fcntl(F_SETLK) though, which works even through NFS (v3).
Signature · 1 line
Jiri Kosina