Skip to contentSkip to conversation

SMP read() stopping at memory page boundaries

DISCUSSION Timo Sirainen 4 participants 7 messages Jun 20, 2007

lore
Share & tools
lore
#0 lore raw

Tested with various 2.6.x i386/x86-64 SMP kernels and CPUs, for example 2.6.21.3/x86-64.

Process 1:

- lock file
- write(4096 + 16 bytes)
- unlock file

Process 2:

- lock file
- read(8192 bytes)
- unlock file

Sometimes read() returns only 4096 bytes. I'm locking the file, so I don't think this should ever happen, right?

Attached a test program. Takes from a few seconds to half a minute with my computer to print "page size cut".

/*
gcc concurrency2.c -o concurrency -Wall

start two both a reader and a writer:

./concurrency
./concurrency 1
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <assert.h>
#include <sys/file.h>

#define MAX_PAGESIZE 8192

int main(int argc, char *argv[])

{
	char buf[MAX_PAGESIZE*2];
	int fd, ret, pagesize;

	memset(buf, 0, sizeof(buf));

	pagesize = getpagesize();
	assert(pagesize <= MAX_PAGESIZE);

	buf[pagesize] = 'h';
	if (argc == 1) {
		printf("writing, page size = %d\n", pagesize);
		for (;;) {
			fd = open("foo", O_RDWR | O_CREAT | O_TRUNC, 0600);
			if (fd == -1) {
				perror("open()");
				return 1;
			}

			if (flock(fd, LOCK_EX) < 0)
				perror("flock()");
			write(fd, buf, pagesize+16);
			if (flock(fd, LOCK_UN) < 0)
				perror("flock()");
			usleep(rand() % 1000);
			close(fd);
		}
	} else {
		printf("reading, page size = %d\n", pagesize);
		fd = open("foo", O_RDWR, 0600);
		if (fd == -1) {
			perror("open()");
			return 1;
		}
		for (;;) {
			usleep(rand() % 1000);
			if (flock(fd, LOCK_SH) < 0)
				perror("flock()");
			lseek(fd, 0, SEEK_SET);
			ret = read(fd, buf, sizeof(buf));
			if (flock(fd, LOCK_UN) < 0)
				perror("flock()");

			if (ret < pagesize) {
				if (ret > 0)
					printf("less than a page: %d\n", ret);
			} else if (ret == pagesize) {
				printf("page size cut\n");
			} else if (buf[pagesize] != 'h') {
				printf("broken data\n");
			}
		}
	}
	return 0;
}
#1 lore raw
Quoted 2 lines · Timo Sirainen

Sometimes read() returns only 4096 bytes. I'm locking the file, so I don't think this should ever happen, right?

Sorry, the problem was with file truncating so there's no bug with locking. I didn't notice it first because it happened to work with FreeBSD and Solaris.

Without locking there's a problem that I'd want to avoid, but I guess it's not necessarily a bug (although it doesn't seem to happen with FreeBSD or Solaris):

Process 1:

- create "foo2"
- rename() to "foo"
- lock
- write(4096 + 16 bytes)
- [fsync() here doesn't change anything]
- pwrite("1111" to offset=4096-4)
- unlock

Process 2:

- open("foo")
- read(8192 bytes)

read() sometimes returns 4096 bytes but with the "1111" already included in the data. Is there a way to avoid this without locking the file while reading? The "1111" tries to act as a kind of a lock.

Again attached a test program, which should really do what I intended. :)

/*
gcc concurrency.c -o concurrency -Wall

start two both a reader and a writer:

./concurrency
./concurrency 1
*/
#define _XOPEN_SOURCE 500
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <assert.h>
#include <sys/file.h>

#define MAX_PAGESIZE 8192

int main(int argc, char *argv[])

{
	char buf[MAX_PAGESIZE*2], ones[4] = { 1, 1, 1, 1 };
	int fd, ret, pagesize;

	memset(buf, 0, sizeof(buf));

	pagesize = getpagesize();
	assert(pagesize <= MAX_PAGESIZE);

	buf[pagesize] = 'h';
	if (argc == 1) {
		printf("writing, page size = %d\n", pagesize);
		for (;;) {
			fd = open("foo2", O_RDWR | O_CREAT | O_TRUNC, 0600);
			if (fd == -1) {
				perror("open()");
				return 1;
			}
			if (rename("foo2", "foo") < 0)
				perror("rename()");
			usleep(rand() % 1000);

			if (flock(fd, LOCK_EX) < 0)
				perror("flock()");
			pwrite(fd, buf, pagesize+16, 0);
			//usleep(rand() % 1000);
			//fdatasync(fd);
			pwrite(fd, ones, 4, pagesize-4);
			if (flock(fd, LOCK_UN) < 0)
				perror("flock()");
			usleep(rand() % 1000);
			close(fd);
		}
	} else {
		printf("reading, page size = %d\n", pagesize);
		for (;; close(fd)) {
			fd = open("foo", O_RDWR, 0600);
			if (fd == -1) {
				perror("open()");
				return 1;
			}

			ret = pread(fd, buf, sizeof(buf), 0);
			if (ret < pagesize) {
				if (ret > 0)
					printf("less than a page: %d\n", ret);
				continue;
			}

			if (memcmp(buf + pagesize - 4, ones, 4) == 0) {
				if (ret == pagesize) {
					printf("page size cut\n");
				} else if (buf[pagesize] != 'h') {
					printf("broken data\n");
				}
			}
		}
	}
	return 0;
}
#2 lore raw
Quoted 3 lines · Timo Sirainen
Quoted 2 lines · Timo Sirainen

Sometimes read() returns only 4096 bytes. I'm locking the file, so I don't think this should ever happen, right?

man 2 read

read() is always allowed to return less than you asked for. You need to go back and ask for the rest. That's why people wrap read() and write() in loops, and use those wrapped versions instead.

Quoted 2 lines

read() sometimes returns 4096 bytes but with the "1111" already included in the data.

Uhm, do you mean 'without the 1111'? 4096 with the 1111 sounds perfectly fine. (Though 4096 without the 1111 is also perfectly fine.)

Quoted 2 lines

Is there a way to avoid this without locking the file while reading? The "1111" tries to act as a kind of a lock.

I think you've misunderstood how read and write work. pread and pwrite are no different in this respect -- they just allow an atomic seek+read/write.

Ray

#3 lore raw
Quoted 10 lines · Ray Lee
Quoted 3 lines · Timo Sirainen
Quoted 2 lines · Timo Sirainen

Sometimes read() returns only 4096 bytes. I'm locking the file, so I don't think this should ever happen, right?

man 2 read

read() is always allowed to return less than you asked for. You need to go back and ask for the rest. That's why people wrap read() and write() in loops, and use those wrapped versions instead.

I thought this wouldn't happen when reading from local filesystems. Anyway, my real program was doing that, and it was still seeing partial data. But looks like I can't reproduce it in my test program with two pread()s, so I'll have to do some more debugging.

#4 lore raw
Quoted 4 lines

I thought this wouldn't happen when reading from local filesystems. Anyway, my real program was doing that, and it was still seeing partial data. But looks like I can't reproduce it in my test program with two pread()s, so I'll have to do some more debugging.

It will occur if you are reading as someone else changes the file size. Use file locking, it exists for a reason ;)

Alan

#5 lore raw
Quoted 4 lines · Alan Cox
Quoted 4 lines

I thought this wouldn't happen when reading from local filesystems. Anyway, my real program was doing that, and it was still seeing partial data. But looks like I can't reproduce it in my test program with two pread()s, so I'll have to do some more debugging.

Found the problem in my code. A buggy program, buggy test program and a buggy brain while debugging doesn't seem to give the best results. :)

Quoted 2 lines

It will occur if you are reading as someone else changes the file size. Use file locking, it exists for a reason ;)

Annoying extra overhead. Especially with NFS, when nowadays you can't even use flock() to create local locks..

#6 lore raw
Quoted 4 lines · Timo Sirainen
Quoted 2 lines

It will occur if you are reading as someone else changes the file size. Use file locking, it exists for a reason ;)

Annoying extra overhead. Especially with NFS, when nowadays you can't even use flock() to create local locks..

It's not only "nowadays", it has been like that for quite a some time.

On the other hand you can use fcntl(F_SETLK) though, which works even through NFS (v3).

Signature · 1 line
Jiri Kosina