Skip to contentSkip to conversation

O_NONBLOCK on files

DISCUSSION Simon Kirby 5 participants 7 messages Sep 19, 2001 – Sep 24, 2001

lore
Share & tools
lore
#0 lore raw

I've always wondered why it's not possible to do this:

fd = open("an_actual_file",O_RDONLY);
fcntl(fd,F_SETFL,O_NONBLOCK);
r = read(fd,buf,4096);

And actually have read return -1 and errno == EWOULDBLOCK/EAGAIN if the block requested is not already cached.

Wouldn't this be the ideal interface for daemons of all types that want to stay single-threaded and still offer useful performance when the working set doesn't fit in cache? It works with sockets, so why not with files?

I see even TUX has to have I/O worker threads to work around this limitation, which seems a bit silly.

Simon-

[ Stormix Technologies Inc. ][ NetNation Communications Inc. ]
[ [email protected] ][ [email protected] ]
[ Opinions expressed are not necessarily those of my employers. ]

#1 lore raw
Quoted 6 lines · Simon Kirby

I've always wondered why it's not possible to do this:

fd = open("an_actual_file",O_RDONLY);
fcntl(fd,F_SETFL,O_NONBLOCK);
r = read(fd,buf,4096);

Quoted 7 lines

And actually have read return -1 and errno == EWOULDBLOCK/EAGAIN if the block requested is not already cached.

Wouldn't this be the ideal interface for daemons of all types that want to stay single-threaded and still offer useful performance when the working set doesn't fit in cache? It works with sockets, so why not with files?

Besides the SUS or the POSIX specs...
What would cause the data to be read in if read just checks the caches? With sockets the other side is clearing pushing or pulling the data. With files there is no other side...

This is resolveable just not currently implemented.

Eric

#2 lore raw
In reply to Simon Kirby
Quoted 1 line · Eric W. Biederman

Besides the SUS or the POSIX specs...

Yeah, well, blah.

Quoted 3 lines

What would cause the data to be read in if read just checks the caches? With sockets the other side is clearing pushing or pulling the data. With files there is no other side...

Hmm...Without even thinking about it, I assumed it would start a read and select() or poll() or some later call would return readable when my outstanding request was fulfilled. But yes, I guess you're right, this is different behavior because there is no other side.

Reading a file would need a receive queue to make this work, I guess. :)

Simon-

[ Stormix Technologies Inc. ][ NetNation Communications Inc. ]
[ [email protected] ][ [email protected] ]
[ Opinions expressed are not necessarily those of my employers. ]

#3 lore raw
In reply to Simon Kirby
Quoted 16 lines · Eric W. Biederman
Quoted 6 lines · Simon Kirby

I've always wondered why it's not possible to do this:

fd = open("an_actual_file",O_RDONLY);
fcntl(fd,F_SETFL,O_NONBLOCK);
r = read(fd,buf,4096);

Quoted 7 lines

And actually have read return -1 and errno == EWOULDBLOCK/EAGAIN if the block requested is not already cached.

Wouldn't this be the ideal interface for daemons of all types that want to stay single-threaded and still offer useful performance when the working set doesn't fit in cache? It works with sockets, so why not with files?

Quoted 6 lines

Besides the SUS or the POSIX specs...
What would cause the data to be read in if read just checks the caches? With sockets the other side is clearing pushing or pulling the data. With files there is no other side...

This is resolveable just not currently implemented.

I was actually playing around with this a few days back, since this would have the advantage of 1) maximizing utilization of the bottleneck device and 2) leave the app free to update a user interface etc while the kernel was busy reading/writing.

There are two problems with a dd type app. First the dd algorithm is fundamentally serial -- read from A, stop, write to B, stop, rinse, repeat. Second, it leaves the thread stuck in D state forever. For example, try something like
dd if=/dev/sda of=/dev/sdb bs=1M
(assuming these devices are not holding important stuff) and let it run for a bit and then hit ^C. Then wait for a minute or two while the kernel finishes flushing everything before dd finally exits... So to get sane stop behavior, you have to minimize the buffer size (and add many minutes to the copy time), while to maximize performance, you have to maximize the buffer size (goodbye user friendly).

So are there no single threaded semantics by which one can copy from device A to device B without blocking (thereby serializing the copy and leaving the app unresponsive)? I'd really hate to have to use threads to do this,

-Erik

--
Erik B. Andersen email: [email protected], formerly of Lineo
--This message was written using 73% post-consumer electrons--

/* vi: set sw=4 ts=4: */
#include <stdlib.h>
#include <stdio.h>
#include <stdarg.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>
#include <signal.h>
#include <dirent.h>
#include <unistd.h>
#include <utime.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/types.h>

static const char *const app_name = "scopy";
static int intr_flag = 0;

static void verror_msg(const char *s, va_list p)

{
	fflush(stdout);
	fprintf(stderr, "%s: ", app_name);
	vfprintf(stderr, s, p);
}
static void vperror_msg(const char *s, va_list p)
{
	int err = errno;

	if (s == 0)
		s = "";
	verror_msg(s, p);
	if (*s)
		s = ": ";
	fprintf(stderr, "%s%s\n", s, strerror(err));
}

static void perror_msg_and_die(const char *s, ...)

{
	va_list p;

	va_start(p, s);
	vperror_msg(s, p);
	va_end(p);
	exit(EXIT_FAILURE);
}

static FILE *xfopen(const char *path, const char *mode)

{
	FILE *fp;

	if ((fp = fopen(path, mode)) == NULL)
		perror_msg_and_die("%s", path);
	return fp;
}

static void show_usage(void)

{
	fprintf(stderr, "Usage: scopy source dest\n");
	exit(EXIT_FAILURE);
}

static void sig_int(int signo)

{
	intr_flag = 1;
}

int main(int argc, char **argv)

{
	int status = 0;
	int opt;
	char *sname, *dname;
	fd_set rfds, wfds;
	FILE *source, *dest;
	int source_fd, dest_fd, n;
	ssize_t rsize = 0, wsize = 0;
	char buf[1024 * 1024];
	ssize_t bufsiz = sizeof(buf);
	struct timeval tv = { 0, 250000 };	/* 0.25 seconds */

	while ((opt = getopt(argc, argv, "adfipR")) != -1)
		switch (opt) {
		default:
			show_usage();
		}

	if (optind + 2 > argc)
		show_usage();

	/* If there are only two arguments and...  */
	if (optind + 2 != argc)
		show_usage();

	/* Store filenames for later use */
	sname = argv[optind++];
	dname = argv[optind];

	source = xfopen(sname, "r");
	dest = xfopen(dname, "w");
	source_fd = fileno(source);
	dest_fd = fileno(dest);

	/* Watch to see if things happen */
	FD_ZERO(&rfds);
	FD_ZERO(&wfds);
	n = (source_fd > dest_fd) ? source_fd + 1 : dest_fd + 1;

	if (fcntl(source_fd, F_SETFL, O_NONBLOCK) == -1)
		perror_msg_and_die("Setting O_NONBLOCK on %s", sname);
	if (fcntl(dest_fd, F_SETFL, O_NONBLOCK) == -1)
		perror_msg_and_die("Setting O_NONBLOCK on %s", dname);
	signal(SIGINT, sig_int);

	/* Ok, do the deed */
	while (1) {
		if (intr_flag) {
			printf("interrupted...  Please wait...\n");
			exit(EXIT_FAILURE);
		}
		FD_SET(fileno(source), &rfds);
		FD_SET(fileno(dest), &wfds);
		status = select(n, &rfds, &wfds, NULL, &tv);
		if (status < 0) {
			if (errno == EINTR) {
				printf("interrupted...  Please wait...\n");
				exit(EXIT_FAILURE);
			} else if (errno != EBADF) {
				perror_msg_and_die("select");
			}
		}

		if (FD_ISSET(source_fd, &rfds)) {
			rsize = read(source_fd, buf, bufsiz);
			if (rsize < 0)
				perror_msg_and_die("reading %d", sname);
			if (rsize == 0)
				exit(EXIT_SUCCESS);
		}
		if (FD_ISSET(dest_fd, &wfds)) {
			wsize = write(dest_fd, buf, rsize);
			if (wsize < 0)
				perror_msg_and_die("writing %s", dname);
			if (wsize == 0)
				exit(EXIT_SUCCESS);
		}

	}

	return 0;
}
#4 lore raw
In reply to Simon Kirby
Quoted 9 lines · Simon Kirby
Quoted 3 lines · Eric W. Biederman

What would cause the data to be read in if read just checks the caches? With sockets the other side is clearing pushing or pulling the data. With files there is no other side...

Hmm...Without even thinking about it, I assumed it would start a read and select() or poll() or some later call would return readable when my outstanding request was fulfilled. But yes, I guess you're right, this is different behavior because there is no other side.

To push the idea into ultimate: AIO

You open file, start IO, and do other things while the machine
is doing IO. Doing open() asynchronously would be ultimate,
but alas, not particularly trivial.

There are problems also in the AIO status rendezvous mechanisms.

Your best choice could be (with moderation) to do synchronous operations at separate threads.

Quoted 3 lines

Reading a file would need a receive queue to make this work, I guess. :) Simon-
[ [email protected] ][ [email protected] ]

/Matti Aarnio

#5 lore raw
In reply to Simon Kirby
Quoted 9 lines · Matti Aarnio
Quoted 4 lines · Simon Kirby

Hmm...Without even thinking about it, I assumed it would start a read and select() or poll() or some later call would return readable when my outstanding request was fulfilled. But yes, I guess you're right, this is different behavior because there is no other side.

To push the idea into ultimate: AIO

Quoted 8 lines

You open file, start IO, and do other things while the machine
is doing IO. Doing open() asynchronously would be ultimate,
but alas, not particularly trivial.

There are problems also in the AIO status rendezvous mechanisms.

Your best choice could be (with moderation) to do synchronous operations at separate threads.

Yes, but this sucks. My whole intent was an interface design that would never need to context switch. I'm not sure if this is even possible, but if so it would be very nice from a userspace perspective.

Simon-

[ Stormix Technologies Inc. ][ NetNation Communications Inc. ]
[ [email protected] ][ [email protected] ]
[ Opinions expressed are not necessarily those of my employers. ]

#6 lore raw
In reply to Simon Kirby
Quoted 3 lines · Simon Kirby

Yes, but this sucks. My whole intent was an interface design that would never need to context switch. I'm not sure if this is even possible, but if so it would be very nice from a userspace perspective.

Actually, this was one of the design concerns that I had in coming up with the async io interfaces. The interface was designed to make use of the upcoming syscalls-in-userspace stubs that x86-64 implemented and will be passed on to x86 in 2.5. All completion events are passed through a ring buffer with minimal locking on the kernel side which only needs thread locking in userspace. This way the whole userspace to kernel transition can be avoided by the server under sufficiently high load.

-ben