Skip to contentSkip to conversation

net: Limit socket I/O iovec total length to INT_MAX.

PATCH In review David Miller 5 participants 16 messages Oct 28, 2010 – Oct 29, 2010

Subsystems (4)
lore
Share & tools
lore

David Miller

author maintainer
#0 lore raw

PATCH 1/1 net: Limit socket I/O iovec total length to INT_MAX.

This helps protect us from overflow issues down in the individual protocol sendmsg/recvmsg handlers. Once we hit INT_MAX we truncate out the rest of the iovec by setting the iov_len members to zero.

This works because:

1) For SOCK_STREAM and SOCK_SEQPACKET sockets, partial writes are allowed and the application will just continue with another write to send the rest of the data.

2) For datagram oriented sockets, where there must be a one-to-one correspondance between write() calls and packets on the wire, INT_MAX is going to be far larger than the packet size limit the protocol is going to check for and signal with -EMSGSIZE.

Based upon a patch by Linus Torvalds.

---

Ok, this is the patch I am testing right now. It ought to plug the TIPC holes wrt. handling iovecs given by the user.

I'll look at the recently discovered RDS crap next :-/

 include/linux/socket.h |    2 +-
 net/compat.c           |   12 +++++++-----
 net/core/iovec.c       |   19 +++++++++----------
 3 files changed, 17 insertions(+), 16 deletions(-)
include/linux/socket.h +1 -1
index 5146b50..86b652f 100644
@@ -322,7 +322,7 @@ extern int csum_partial_copy_fromiovecend(unsigned char *kdata,
 					  int offset, 
 					  unsigned int len, __wsum *csump);
 
-extern long verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode);
+extern int verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode);
 extern int memcpy_toiovec(struct iovec *v, unsigned char *kdata, int len);
 extern int memcpy_toiovecend(const struct iovec *v, unsigned char *kdata,
 			     int offset, int len);
net/compat.c +7 -5
index 63d260e..71bfd8e 100644
@@ -34,17 +34,19 @@ static inline int iov_from_user_compat_to_kern(struct iovec *kiov,
 					  struct compat_iovec __user *uiov32,
 					  int niov)
 {
-	int tot_len = 0;
+	size_t tot_len = 0;
 
 	while (niov > 0) {
 		compat_uptr_t buf;
 		compat_size_t len;
 
 		if (get_user(len, &uiov32->iov_len) ||
-		   get_user(buf, &uiov32->iov_base)) {
-			tot_len = -EFAULT;
-			break;
-		}
+		    get_user(buf, &uiov32->iov_base))
+			return -EFAULT;
+
+		if (len > INT_MAX - tot_len)
+			len = INT_MAX - tot_len;
+
 		tot_len += len;
 		kiov->iov_base = compat_ptr(buf);
 		kiov->iov_len = (__kernel_size_t) len;
net/core/iovec.c +9 -10
index 72aceb1..e7f5b29 100644
@@ -35,10 +35,10 @@
  *	in any case.
  */
 
-long verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode)
+int verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode)
 {
 	int size, ct;
-	long err;
+	size_t err;
 
 	if (m->msg_namelen) {
 		if (mode == VERIFY_READ) {
@@ -62,14 +62,13 @@ long verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address,
 	err = 0;
 
 	for (ct = 0; ct < m->msg_iovlen; ct++) {
-		err += iov[ct].iov_len;
-		/*
-		 * Goal is not to verify user data, but to prevent returning
-		 * negative value, which is interpreted as errno.
-		 * Overflow is still possible, but it is harmless.
-		 */
-		if (err < 0)
-			return -EMSGSIZE;
+		size_t len = iov[ct].iov_len;
+
+		if (len > INT_MAX - err) {
+			len = INT_MAX - err;
+			iov[ct].iov_len = len;
+		}
+		err += len;
 	}
 
 	return err;
Signature · 1 line
1.7.3.2
#1 lore raw
Quoted 3 lines · David Miller

-       int tot_len = 0;
+       size_t tot_len = 0;

I would actually keep "tot_len" as an "int".

The whole point of this:

Quoted 4 lines

+               if (len > INT_MAX - tot_len)
+                       len = INT_MAX - tot_len;
+
               tot_len += len;

Is that "tot_len" can _never_ become larger than INT_MAX, because we never add a "len" that would make it bigger than that.

So "len" itself should be the correct unsigned size_t (so that the "len > INT_MAX - tot_len" thing is done as an unsigned comparison), but "tot_len" itself is very much designed to fit in "int".

Quoted 5 lines

+int verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode) {
       int size, ct;
-       long err;
+       size_t err;

Same thing here. Making "err" be an "int" is actually the right thing to do, because then it matches the return type (iow, if it was any other type, there would be an implicit cast, and if it didn't fit in "int", that would be a bug anyway).

Linus

author maintainer
#2 lore raw

From: Linus Torvalds <[email protected]>
Date: Thu, 28 Oct 2010 11:33:56 -0700

Quoted 6 lines · David Miller
Quoted 3 lines

-       int tot_len = 0;
+       size_t tot_len = 0;

I would actually keep "tot_len" as an "int".

...

Quoted 10 lines
Quoted 5 lines

+int verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode) {
       int size, ct;
-       long err;
+       size_t err;

Same thing here. Making "err" be an "int" is actually the right thing to do, because then it matches the return type (iow, if it was any other type, there would be an implicit cast, and if it didn't fit in "int", that would be a bug anyway).

Yep, agreed on all counts, I'll make those changes.

#3 lore raw
In reply to David Miller

Oh, btw, noticed another small detail..

I don't know if this matters, but the regular read/write routines don't actually use INT_MAX as the limit, but instead a "maximally page-aligned value that fits in an int":

   #define MAX_RW_COUNT (INT_MAX & PAGE_CACHE_MASK)

because the code does _not_ want to turn a nice set of huge page-aligned big writes into a write of an odd number (2GB-1).

This may not make much of a difference to networking - you guys are already used to working with odd sizes like 1500 bytes of data payload per packet etc. Most regular filesystems are much more sensitive to things like block (and particularly page-cache sized) boundaries because of the vagaries of disk and cache granularities. But MAX_INT is a _really_ odd size, and things like csum_and_copy still tends to want to get things at least word-aligned, no? And if nothing else, the memory copies tend to be better with cacheline boundaries.

It would be sad if a 4GB aligned write turns into
- one 2GB-1 aligned write
- one pessimally unaligned 2G-1 write where every read from user
space is unaligned
- finally a single 2-byte write.

I suspect it would be better off using the same kind of (MAX_INT & PAGE_CACHE_MASK) logic - that 4GB write would still get split into three partial writes (and _lots_ of packets ;), but at least they'd all be word-aligned.

Does it matter? I dunno. Once you do 2GB+ writes, these kinds of small details may be totally hidden in all the noise.

Linus

Quoted 119 lines · David Miller

This helps protect us from overflow issues down in the individual protocol sendmsg/recvmsg handlers.  Once we hit INT_MAX we truncate out the rest of the iovec by setting the iov_len members to zero.

This works because:

1) For SOCK_STREAM and SOCK_SEQPACKET sockets, partial writes are allowed and the application will just continue with another write to send the rest of the data.

2) For datagram oriented sockets, where there must be a one-to-one correspondance between write() calls and packets on the wire, INT_MAX is going to be far larger than the packet size limit the protocol is going to check for and signal with -EMSGSIZE.

Based upon a patch by Linus Torvalds.

---

Ok, this is the patch I am testing right now.  It ought to plug the TIPC holes wrt. handling iovecs given by the user.

I'll look at the recently discovered RDS crap next :-/

 include/linux/socket.h |    2 +- net/compat.c           |   12 +++++++----- net/core/iovec.c       |   19 +++++++++----------

 3 files changed, 17 insertions(+), 16 deletions(-)
include/linux/socket.h +1 -1
index 5146b50..86b652f 100644
@@ -322,7 +322,7 @@ extern int csum_partial_copy_fromiovecend(unsigned char *kdata,
                                          int offset,
                                          unsigned int len, __wsum *csump);
 
-extern long verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode);
+extern int verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode);
  extern int memcpy_toiovec(struct iovec *v, unsigned char *kdata, int len);
  extern int memcpy_toiovecend(const struct iovec *v, unsigned char *kdata,
                             int offset, int len);
net/compat.c +1 -1
index 63d260e..71bfd8e 100644
@@ -34,17 +34,19 @@ static inline int iov_from_user_compat_to_kern(struct iovec *kiov,
                                          struct compat_iovec __user *uiov32,
                                          int niov)
  {
-       int tot_len = 0;
+       size_t tot_len = 0;
 
        while (niov > 0) {
                compat_uptr_t buf;
                compat_size_t len;
 

               if (get_user(len, &uiov32->iov_len) ||
-                  get_user(buf, &uiov32->iov_base)) {
-                       tot_len = -EFAULT;
-                       break;
-               }
+                   get_user(buf, &uiov32->iov_base))
+                       return -EFAULT;

net/compat.c +4 -0
+
+               if (len > INT_MAX - tot_len)
+                       len = INT_MAX - tot_len;
+

               tot_len += len;
               kiov->iov_base = compat_ptr(buf); kiov->iov_len = (__kernel_size_t) len;

net/core/iovec.c +9 -11
index 72aceb1..e7f5b29 100644
@@ -35,10 +35,10 @@
  *     in any case.
  */
 
-long verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode)
+int verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode)
  {
        int size, ct;
-       long err;
+       size_t err;
 
        if (m->msg_namelen) {
                if (mode == VERIFY_READ) {
@@ -62,14 +62,13 @@ long verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address,
        err = 0;
 
        for (ct = 0; ct < m->msg_iovlen; ct++) {
-               err += iov[ct].iov_len;
-               /*
-                * Goal is not to verify user data, but to prevent returning
-                * negative value, which is interpreted as errno.
-                * Overflow is still possible, but it is harmless.
-                */
-               if (err < 0)
-                       return -EMSGSIZE;
+               size_t len = iov[ct].iov_len;
+
+               if (len > INT_MAX - err) {
+                       len = INT_MAX - err;
+                       iov[ct].iov_len = len;
+               }
+               err += len;
        }
 
        return err;
--

1.7.3.2

#4 lore raw

While you guys are at it, you might consider preventing sendto(), etc. calls from requesting >= 2GB data in one go. Several families have no restrictions on total size (or even worse, assign the size to a signed int type and then do a signed comparison as a check). This can result in all kinds of ugliness when allocating sk_buffs based on that size, some of which result in kernel panics (due to bad sk_buff tail position) or heap corruption.

If you'd rather I dig up specific examples, I can do that as well, but I think making changes to core code to protect individual modules from their own inevitable stupid decisions is the best choice.

-Dan

Quoted 156 lines · Linus Torvalds

Oh, btw, noticed another small detail..

I don't know if this matters, but the regular read/write routines don't actually use INT_MAX as the limit, but instead a "maximally page-aligned value that fits in an int":

   #define MAX_RW_COUNT (INT_MAX & PAGE_CACHE_MASK)

because the code does _not_ want to turn a nice set of huge page-aligned big writes into a write of an odd number (2GB-1).

This may not make much of a difference to networking - you guys are already used to working with odd sizes like 1500 bytes of data payload per packet etc. Most regular filesystems are much more sensitive to things like block (and particularly page-cache sized) boundaries because of the vagaries of disk and cache granularities. But MAX_INT is a _really_ odd size, and things like csum_and_copy still tends to want to get things at least word-aligned, no? And if nothing else, the memory copies tend to be better with cacheline boundaries.

It would be sad if a 4GB aligned write turns into
- one 2GB-1 aligned write
- one pessimally unaligned 2G-1 write where every read from user
space is unaligned
- finally a single 2-byte write.

I suspect it would be better off using the same kind of (MAX_INT & PAGE_CACHE_MASK) logic - that 4GB write would still get split into three partial writes (and _lots_ of packets ;), but at least they'd all be word-aligned.

Does it matter? I dunno. Once you do 2GB+ writes, these kinds of small details may be totally hidden in all the noise.

Linus

Quoted 119 lines · David Miller

This helps protect us from overflow issues down in the individual protocol sendmsg/recvmsg handlers. Once we hit INT_MAX we truncate out the rest of the iovec by setting the iov_len members to zero.

This works because:

1) For SOCK_STREAM and SOCK_SEQPACKET sockets, partial writes are allowed and the application will just continue with another write to send the rest of the data.

2) For datagram oriented sockets, where there must be a one-to-one correspondance between write() calls and packets on the wire, INT_MAX is going to be far larger than the packet size limit the protocol is going to check for and signal with -EMSGSIZE.

Based upon a patch by Linus Torvalds.

---

Ok, this is the patch I am testing right now. It ought to plug the TIPC holes wrt. handling iovecs given by the user.

I'll look at the recently discovered RDS crap next :-/

 include/linux/socket.h |    2 +-
 net/compat.c           |   12 +++++++-----
 net/core/iovec.c       |   19 +++++++++----------
 3 files changed, 17 insertions(+), 16 deletions(-)
include/linux/socket.h +1 -1
index 5146b50..86b652f 100644
@@ -322,7 +322,7 @@ extern int csum_partial_copy_fromiovecend(unsigned char *kdata,
                                         int offset,
                                         unsigned int len, __wsum *csump);
 
-extern long verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode);
+extern int verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode);
 extern int memcpy_toiovec(struct iovec *v, unsigned char *kdata, int len);
 extern int memcpy_toiovecend(const struct iovec *v, unsigned char *kdata,
                            int offset, int len);
net/compat.c +7 -5
index 63d260e..71bfd8e 100644
@@ -34,17 +34,19 @@ static inline int iov_from_user_compat_to_kern(struct iovec *kiov,
                                         struct compat_iovec __user *uiov32,
                                         int niov)
 {
-       int tot_len = 0;
+       size_t tot_len = 0;
 
       while (niov > 0) {
               compat_uptr_t buf;
               compat_size_t len;
 
               if (get_user(len, &uiov32->iov_len) ||
-                  get_user(buf, &uiov32->iov_base)) {
-                       tot_len = -EFAULT;
-                       break;
-               }
+                   get_user(buf, &uiov32->iov_base))
+                       return -EFAULT;
+
+               if (len > INT_MAX - tot_len)
+                       len = INT_MAX - tot_len;
+
               tot_len += len;
               kiov->iov_base = compat_ptr(buf);
               kiov->iov_len = (__kernel_size_t) len;
net/core/iovec.c +9 -11
index 72aceb1..e7f5b29 100644
@@ -35,10 +35,10 @@
 *     in any case.
 */
 
-long verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode)
+int verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address, int mode)
 {
       int size, ct;
-       long err;
+       size_t err;
 
       if (m->msg_namelen) {
               if (mode == VERIFY_READ) {
@@ -62,14 +62,13 @@ long verify_iovec(struct msghdr *m, struct iovec *iov, struct sockaddr *address,
       err = 0;
 
       for (ct = 0; ct < m->msg_iovlen; ct++) {
-               err += iov[ct].iov_len;
-               /*
-                * Goal is not to verify user data, but to prevent returning
-                * negative value, which is interpreted as errno.
-                * Overflow is still possible, but it is harmless.
-                */
-               if (err < 0)
-                       return -EMSGSIZE;
+               size_t len = iov[ct].iov_len;
+
+               if (len > INT_MAX - err) {
+                       len = INT_MAX - err;
+                       iov[ct].iov_len = len;
+               }
+               err += len;
       }
 
       return err;
--

1.7.3.2

#5 lore raw
Quoted 3 lines · Dan Rosenberg

While you guys are at it, you might consider preventing sendto(), etc. calls from requesting >= 2GB data in one go.

Indeed. David - I think we have to, because that thing converts its arguments to an iovec and then does a sendmsg, but since it's already in kernel space it doesn't go through the verify_iovec() path.

So sendto/recvfrom (and possibly others that build their own msg struct in kernel space) should be limited to MAX_INT too, so that there's no back way to create a big iovec..

In fs/read_write.c, do_sync_read/write() do that iovec thing too, but at least for the regular vfs_read()/vfs_write cases they will have gone through rw_verify_area() first, which does the size limiting for them.

We do need to fix the readv/writev path, though. It does the rw_verify_area(), but it doesn't seem to limit the size to the returned length, but still uses the original one. Hmm.

I think I'll take care of the readv/writev thing, and send it by Al to verify.

Linus

#6 lore raw
Quoted 6 lines · Linus Torvalds

We do need to fix the readv/writev path, though. It does the rw_verify_area(), but it doesn't seem to limit the size to the returned length, but still uses the original one. Hmm.

I think I'll take care of the readv/writev thing, and send it by Al to verify.

Ok, something like the attached should do the same as the suggested networking iovec verification (except the VFS code considers the iov 'len' to be a ssize_t, and a negative value to be an error, and claims that that is what SuS says. I don't much care, since we ignore SuS wrt the overflow anyway, and now limit IO to MAX_RW_COUNT the same way we do for regular reads and writes, but it's possibly worth thinking about unifying the networking verify_iovec and the VFS layer one)

Al? What do you think? You've not seen some of the background, but it all boils down to simple "some protocols overflow in 'int'", the exact same way we had filesystems that had int counters for access lengths etc. So for the same reason we did the MAX_RW_COUNT thing for regular read/write calls, the networking code is now going to do it for all the sendmsg etc iovec things.

Even outside of any networking issues, this unifies the handling of read/write vs readv/writev in the "cap IO size" area, so I think we should have done this originally (the readv/writev cases actually already did the capping by calling rw_verify_area(), but then the code threw away the capped value, and only looked at the error return, and used the uncapped size)

NOTE! UNTESTED! It compiles, but I didn't actually boot to check. Maybe it has some stupid thinko in it. Also, the patch ended up being a bit bigger than it needed to be, because I found some annoying whitespace issues in rw_copy_check_uvector (spaces at the beginning of lines that had tabs in them) that I couldn't keep myself from not fixing.

Comments?

Linus

 fs/compat.c        |   12 +++++-----
 fs/read_write.c    |   62 +++++++++++++++++++++++++++------------------------
 include/linux/fs.h |    1 +
 3 files changed, 40 insertions(+), 35 deletions(-)
fs/compat.c +6 -6
index 52cfeb6..ff66c0d 100644
@@ -606,14 +606,14 @@ ssize_t compat_rw_copy_check_uvector(int type,
 	/*
 	 * Single unix specification:
 	 * We should -EINVAL if an element length is not >= 0 and fitting an
-	 * ssize_t.  The total length is fitting an ssize_t
+	 * ssize_t.
 	 *
-	 * Be careful here because iov_len is a size_t not an ssize_t
+	 * In Linux, the total length is limited to MAX_RW_COUNT, there is
+	 * no overflow possibility.
 	 */
 	tot_len = 0;
 	ret = -EINVAL;
 	for (seg = 0; seg < nr_segs; seg++) {
-		compat_ssize_t tmp = tot_len;
 		compat_uptr_t buf;
 		compat_ssize_t len;
 
@@ -624,13 +624,13 @@ ssize_t compat_rw_copy_check_uvector(int type,
 		}
 		if (len < 0)	/* size_t not fitting in compat_ssize_t .. */
 			goto out;
-		tot_len += len;
-		if (tot_len < tmp) /* maths overflow on the compat_ssize_t */
-			goto out;
 		if (!access_ok(vrfy_dir(type), compat_ptr(buf), len)) {
 			ret = -EFAULT;
 			goto out;
 		}
+		if (len > MAX_RW_COUNT - tot_len)
+			len = MAX_RW_COUNT - tot_len;
+		tot_len += len;
 		iov->iov_base = compat_ptr(buf);
 		iov->iov_len = (compat_size_t) len;
 		uvector++;
fs/read_write.c +33 -29

106 lines not on this page · raw message

include/linux/fs.h +1 -0
index 4d07902..7b7b507 100644
@@ -1867,6 +1867,7 @@ extern int current_umask(void);
 /* /sys/fs */
 extern struct kobject *fs_kobj;
 
+#define MAX_RW_COUNT (INT_MAX & PAGE_CACHE_MASK)
 extern int rw_verify_area(int, struct file *, loff_t *, size_t);
 
 #define FLOCK_VERIFY_READ  1
maintainer
#7 lore raw
Quoted 13 lines · Linus Torvalds
fs/compat.c +0 -4
 	ret = -EINVAL;
 	for (seg = 0; seg < nr_segs; seg++) {
-		compat_ssize_t tmp = tot_len;
 		compat_uptr_t buf;
 		compat_ssize_t len;
@@ -624,13 +624,13 @@ ssize_t compat_rw_copy_check_uvector(int type,
 		}
 		if (len < 0)	/* size_t not fitting in compat_ssize_t .. */
 			goto out;
-		tot_len += len;
-		if (tot_len < tmp) /* maths overflow on the compat_ssize_t */
-			goto out;

fs/compat.c ssize_t compat_rw_copy_check_uvector(int type, · line 636

Quoted 10 lines
fs/compat.c +3 -0
 		if (!access_ok(vrfy_dir(type), compat_ptr(buf), len)) {
 			ret = -EFAULT;
 			goto out;
 		}
+		if (len > MAX_RW_COUNT - tot_len)
+			len = MAX_RW_COUNT - tot_len;
+		tot_len += len;
 		iov->iov_base = compat_ptr(buf);
 		iov->iov_len = (compat_size_t) len;
 		uvector++;

Interesting... Had anybody tested vectors with 0 iov_len in the end and/or middle? Looks like something rarely hit in practice...

I don't see anything obviously broken (and we obviously have allowed iov_len == 0 cases all along, so if anything, breakage won't be new). However, I wonder if things like sendmsg() for datagrams have warranties against silent truncation. Davem?

#8 lore raw
Quoted 5 lines · Al Viro

I don't see anything obviously broken (and we obviously have allowed iov_len == 0 cases all along, so if anything, breakage won't be new). However, I wonder if things like sendmsg() for datagrams have warranties against silent truncation.  Davem?

You missed that discussion - my argument is that anybody who thinks that they can send a single packet that is 2GB+ in size are already screwed. And the packet protocol will have some inherent upper limit anyway (possibly introduced by just allocation issues, but quite likely inherent to the protocol itself)

And yes, the iov_len = 0 case has always been possible and accepted so my patch doesn't really change anything. In fact, I think it even happens (simple example: the easiest way for user space to resume a partial writev() is to basically subtract out the return value from the iovec and then re-submit it - so getting zero iovec entries at the beginning in particular would not necessarily even be odd)

Linus

maintainer
#9 lore raw
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Al Viro

I don't see anything obviously broken (and we obviously have allowed iov_len == 0 cases all along, so if anything, breakage won't be new). However, I wonder if things like sendmsg() for datagrams have warranties against silent truncation. ?Davem?

Quoted 5 lines

You missed that discussion - my argument is that anybody who thinks that they can send a single packet that is 2GB+ in size are already screwed. And the packet protocol will have some inherent upper limit anyway (possibly introduced by just allocation issues, but quite likely inherent to the protocol itself)

Sure, but... do we want to send something truncated in that case or should we just fail? Note that with your change previously deliberately b0rken iovecs (anything with sum of lengths equal to 1<<31 on 32bit) will get a chance to be accepted *OR* (much more likely) get rejected with unexpected error value. It may well be OK, but I'd like to hear from network folks...

author maintainer
#10 lore raw
In reply to Linus Torvalds

From: Linus Torvalds <[email protected]>
Date: Fri, 29 Oct 2010 09:21:16 -0700

Quoted 8 lines

NOTE! UNTESTED! It compiles, but I didn't actually boot to check. Maybe it has some stupid thinko in it. Also, the patch ended up being a bit bigger than it needed to be, because I found some annoying whitespace issues in rw_copy_check_uvector (spaces at the beginning of lines that had tabs in them) that I couldn't keep myself from not fixing.

Comments?

I just got out of a long dentist appointment, will look at this right now, thanks!

#11 lore raw
Quoted 3 lines · David Miller

I just got out of a long dentist appointment, will look at this right now, thanks!

I booted with it and committed it as "obvious". Let's see if there is any fallout. I doubt it, but I also doubt we'll find any until we have lots of testers, unless I made some subtly totally buggy change that just didn't happen to show up during a normal boot.

Linus

author maintainer
#12 lore raw

From: Linus Torvalds <[email protected]>
Date: Fri, 29 Oct 2010 12:37:29 -0700

Quoted 9 lines · David Miller
Quoted 3 lines

I just got out of a long dentist appointment, will look at this right now, thanks!

I booted with it and committed it as "obvious". Let's see if there is any fallout. I doubt it, but I also doubt we'll find any until we have lots of testers, unless I made some subtly totally buggy change that just didn't happen to show up during a normal boot.

It ought to be ok.

Let me send you a pull request so you can get the verify_iovec() change.

#13 lore raw

Thanks for your work on this. Just a friendly reminder not to forget the compat code. :)

-Dan

Quoted 16 lines · David Miller

From: Linus Torvalds <[email protected]>
Date: Fri, 29 Oct 2010 12:37:29 -0700

Quoted 9 lines · David Miller
Quoted 3 lines

I just got out of a long dentist appointment, will look at this right now, thanks!

I booted with it and committed it as "obvious". Let's see if there is any fallout. I doubt it, but I also doubt we'll find any until we have lots of testers, unless I made some subtly totally buggy change that just didn't happen to show up during a normal boot.

It ought to be ok.

Let me send you a pull request so you can get the verify_iovec() change.

#14 lore raw
In reply to Linus Torvalds

It may be but a paint splatter on the bikeshed, or considered a case of "Doctor! Doctor! It hurts when I do this" "Then don't do that!" but elsewhere (not in the context of Linux) I've seen mention made of ISV software posting some particularly large receives and such - one case I saw was over 1GB, where that was tied to the size of a log buffer the creation of which I believe was not limited to ~INT_MAX by the application software.

rick jones

#15 lore raw
Quoted 6 lines · Rick Jones

It may be but a paint splatter on the bikeshed, or considered a case of "Doctor! Doctor! It hurts when I do this" "Then don't do that!" but elsewhere (not in the context of Linux) I've seen mention made of ISV software posting some particularly large receives and such - one case I saw was over 1GB, where that was tied to the size of a log buffer the creation of which I believe was not limited to ~INT_MAX by the application software.

Sure. And that is why I very much don't think it's a good idea to disallow large reads or writes. Returning an error would be bad, because it's not entirely unreasonable for some user to just have a really big object (presumably for unrelated reasons), and do IO on it in one go.

But expecting anybody to _fill_ that really big object in one go is unreasonable. Even for regular files, anybody who has ever worked with NFS and interruptible mounts knows that they have to be able to handle partial IO. And with non-files you obviously have that all the time.

So I think it's perfectly fine to do a terabyte read() or write(). The fact that the kernel will then internally limit it, and won't ever actually then write more than 2GB-1 at a time ends up being just an "implementation issue" that happens to protect the kernel against subsystems that happen to have issues.

And an application that cannot handle partial reads or writes, yet works with gigabyte+ data sets is _not_ an application that I consider reasonable. That's a user space bug, pure and simple, and no amount of "but but ..." makes any difference what-so-ever.

Linus