Skip to contentSkip to conversation

writev() semantics with invalid iovec in the middle

RFC Al Viro 5 participants 7 messages Sep 14, 2016 – Sep 16, 2016 VFS

lore
Share & tools
lore

Al Viro

author maintainer
#0 lore raw

Right now writev() with 3-iovec array that has unmapped address in
the second element and total length less than PAGE_SIZE will write the first segment and stop at that. Among other things, it guarantees the short copy, and I would rather have it yeild 0-bytes write (and -EFAULT as return value).

All POSIX has to say about that is this (in 2.3 Error Numbers):

[EFAULT]
Bad address. The system detected an invalid address in attempting to use
an argument of a call. The reliable detection of this error cannot be guaranteed, and when not detected may result in the generation of a signal, indicating an address violation, which is sent to the process.

Note that unmapped page in the middle of a range covered already can lead to the same kind of short write - i.e. if we have

	p = mmap(0, 3*4096, PROT_READ, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
	munmap(p + 4096, 4096);
	fd = open("/tmp/foo", O_CREAT|O_TRUNC|O_RDWR, 0777);
	write(fd, p + 2048, 8192);

write() will yield -EFAULT, not a 2Kb stored. The same will happen with

	writev(fd, &(struct iovec){p + 2048, 8192}, 1);

BTW, adding lseek(fd, 2049, SEEK_SET); before that write (or writev) will result in 2047 bytes being written by the latter.

IOW, we do not try to squeeze every byte that can be squeezed out of the buffer; generally, an unmapped address anywhere in PAGE_SIZE worth of data that would go into the same page-aligned chunk of destination can result in short write cut at the beginning of that chunk. iovec boundaries act as barriers to short writes, mostly by accident.

Do we need to preserve that special treatment of iovec boundaries? I would really like to get rid of that - the current behaviour is an easy and reliable way to trigger a short copy case in ->write_end() and those are fairly brittle. Sure, we still need to cope with them, and I think I've got all instances in the current mainline fixed, but they are often suboptimal.

Objections?

#1 lore raw

If you squeeze out every byte won't you still have a short write? And the written data wouldn't be cut at the bad place, but it would have a weird hole or discontinuity there.

-Mike

Quoted 43 lines · Al Viro

Right now writev() with 3-iovec array that has unmapped address in
the second element and total length less than PAGE_SIZE will write the first segment and stop at that. Among other things, it guarantees the short copy, and I would rather have it yeild 0-bytes write (and -EFAULT as return value).

All POSIX has to say about that is this (in 2.3 Error Numbers):

[EFAULT]
Bad address. The system detected an invalid address in attempting to use
an argument of a call. The reliable detection of this error cannot be guaranteed, and when not detected may result in the generation of a signal, indicating an address violation, which is sent to the process.

Note that unmapped page in the middle of a range covered already can lead to the same kind of short write - i.e. if we have

        p = mmap(0, 3*4096, PROT_READ, MAP_ANONYMOUS|MAP_PRIVATE, -1, 0);
        munmap(p + 4096, 4096);
        fd = open("/tmp/foo", O_CREAT|O_TRUNC|O_RDWR, 0777);
        write(fd, p + 2048, 8192);

write() will yield -EFAULT, not a 2Kb stored. The same will happen with

        writev(fd, &(struct iovec){p + 2048, 8192}, 1);

BTW, adding lseek(fd, 2049, SEEK_SET); before that write (or writev) will result in 2047 bytes being written by the latter.

IOW, we do not try to squeeze every byte that can be squeezed out of the buffer; generally, an unmapped address anywhere in PAGE_SIZE worth of data that would go into the same page-aligned chunk of destination can result in short write cut at the beginning of that chunk. iovec boundaries act as barriers to short writes, mostly by accident.

Do we need to preserve that special treatment of iovec boundaries? I would really like to get rid of that - the current behaviour is an easy and reliable way to trigger a short copy case in ->write_end() and those are fairly brittle. Sure, we still need to cope with them, and I think I've got all instances in the current mainline fixed, but they are often suboptimal.

Objections?
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author maintainer
#2 lore raw
Quoted 3 lines · Mike Marshall

If you squeeze out every byte won't you still have a short write? And the written data wouldn't be cut at the bad place, but it would have a weird hole or discontinuity there.

???

What I mean is that if we have an invalid address in the middle of a buffer (unmapped, for example), we do not attempt to write every byte prior to that invalid address. Of course what we write is going to be contiguous.

Suppose we have a buffer spanning 10 pages (amd64, so these are 4K ones) - 7 valid, 3 invalid:
VVVVIIIVV
and it starts 100 bytes into the first page. And write goes into a regular file on e.g. tmpfs, starting at offset 31. We _can't_ write more than 4*4096-100 bytes, no matter what. It will be a short write. As the matter of fact, it will be even shorter than that - it will be 3*4096-31 bytes, up to the last pagecache boundary we can cover completely. That obviously depends upon the filesystem - not everything uses pagecache, for starters. However, the caller is *not* guaranteed that write() with an invalid page in the middle of a buffer would write everything up to the very beginning of the invalid page. A short write will happen, but the amount written might be up to page size less than the actual length of valid part in the beginning of the buffer.

Now, for writev() we could have invalid pages in any iovec; again, we obviously can't write anything past the first invalid page - we'll get either a short write or -EFAULT (if nothing got written). That's fine; the question is what the caller can count upon wrt shortening.

Again, we are *not* guaranteed writing up to exact boundary. However, the current implementation will end up shortening no more than to the iovec boundary. I.e. if the first iovec contains only valid pages and there's an invalid one in the second iovec, the current implementation will write at least everything in the first iovec. That's _not_ promised by POSIX or our manpages; moreover, I'm not sure if it's even true for each filesystem. And keeping that property is actually inconvenient - if we could discard it, we could make partial-copy ->write_end() calls a lot more infrequent.

Unfortunately, some of LTP writev tests end up checking that writev() does behave that way - they feed it a three-element iovec with shorter-than-page segments, the second of which is all invalid. And they check that the entire first segment had been written.

I would really like to drop that property, making it "if some addresses in the buffer(s) we are asked to write are invalid, the write will be shortened by up to a PAGE_SIZE from the first such invalid address", making writev() rules exactly the same as write() ones. Does anybody have objections to it?

#3 lore raw

PAGE_SIZE isn't accurate on architectures which do multiple page sizes, like 8k, 64k, 512k, 4M, 32M, 256M on SPARC64 and same on PPC64/Power.

Ced

Quoted 54 lines · Al Viro
Quoted 3 lines · Mike Marshall

If you squeeze out every byte won't you still have a short write? And the written data wouldn't be cut at the bad place, but it would have a weird hole or discontinuity there.

???

What I mean is that if we have an invalid address in the middle of a buffer (unmapped, for example), we do not attempt to write every byte prior to that invalid address. Of course what we write is going to be contiguous.

Suppose we have a buffer spanning 10 pages (amd64, so these are 4K ones) - 7 valid, 3 invalid:
VVVVIIIVV
and it starts 100 bytes into the first page. And write goes into a regular file on e.g. tmpfs, starting at offset 31. We _can't_ write more than 4*4096-100 bytes, no matter what. It will be a short write. As the matter of fact, it will be even shorter than that - it will be 3*4096-31 bytes, up to the last pagecache boundary we can cover completely. That obviously depends upon the filesystem - not everything uses pagecache, for starters. However, the caller is *not* guaranteed that write() with an invalid page in the middle of a buffer would write everything up to the very beginning of the invalid page. A short write will happen, but the amount written might be up to page size less than the actual length of valid part in the beginning of the buffer.

Now, for writev() we could have invalid pages in any iovec; again, we obviously can't write anything past the first invalid page - we'll get either a short write or -EFAULT (if nothing got written). That's fine; the question is what the caller can count upon wrt shortening.

Again, we are *not* guaranteed writing up to exact boundary. However, the current implementation will end up shortening no more than to the iovec boundary. I.e. if the first iovec contains only valid pages and there's an invalid one in the second iovec, the current implementation will write at least everything in the first iovec. That's _not_ promised by POSIX or our manpages; moreover, I'm not sure if it's even true for each filesystem. And keeping that property is actually inconvenient - if we could discard it, we could make partial-copy ->write_end() calls a lot more infrequent.

Unfortunately, some of LTP writev tests end up checking that writev() does behave that way - they feed it a three-element iovec with shorter-than-page segments, the second of which is all invalid. And they check that the entire first segment had been written.

I would really like to drop that property, making it "if some addresses in the buffer(s) we are asked to write are invalid, the write will be shortened by up to a PAGE_SIZE from the first such invalid address", making writev() rules exactly the same as write() ones. Does anybody have objections to it?
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 3 lines
Cedric Blancher <[email protected]>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur
#4 lore raw
In reply to Al Viro
Quoted 7 lines · Al Viro

Unfortunately, some of LTP writev tests end up checking that writev() does behave that way - they feed it a three-element iovec with shorter-than-page segments, the second of which is all invalid. And they check that the entire first segment had been written.

I would really like to drop that property,

I'm pretty sure you can and should do that.

The LTP test people have actually been pretty good about just fixing their tests when they cause problems and there is no reason for the particular behavior.

Linus

#5 lore raw
In reply to Al Viro
Quoted 4 lines

Unfortunately, some of LTP writev tests end up checking that writev() does behave that way - they feed it a three-element iovec with shorter-than-page segments, the second of which is all invalid. And they check that the entire first segment had been written.

1003.1 says

"Each iovec entry specifies the base address and length of an area in memory from which data should be written. The writev() function shall always write a complete area before proceeding to the next."

and I imagine that is what LTP is attempting to test.

The moment you pass an invalid address you are in the land of undefined behaviour, so I would read the standard as actually trying to deal with the behaviour in defined situations (eg out of disk space mid writev()).

Alan

#6 lore raw
Quoted 8 lines · One Thousand Gnomes

1003.1 says

"Each iovec entry specifies the base address and length of an area in memory from which data should be written. The writev() function shall always write a complete area before proceeding to the next."

and I imagine that is what LTP is attempting to test.

Ahh. Yes. But as you note, the EFAULT case is undefined behavior, so what that POSIX language is *really* about is presumably making sure that readers of a file cannot see the "later" writes without seeing the earlier ones.

So you cannot do some fancy threaded thing where you do different iovec parts concurrently, because that could be seen by a reader (or more likely mmap) as doing the writes out of order.

Or, as you mention, the disk-full case.

Linus