Skip to contentSkip to conversation

dup2: Fix return value with oldfd == newfd and invalid fd

PATCH In review Jeff Mahoney 3 participants 6 messages May 11, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 dup2: Fix return value with oldfd == newfd and invalid fd

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The return value of dup2 when oldfd == newfd and the fd isn't valid is not getting properly sign extended. We end up with 4294967287 instead of -EBADF.

I've reproduced this on SLE11 (2.6.27.21), openSUSE Factory (2.6.29-rc5), and Ubuntu 9.04 (2.6.28).

This patch uses a signed int for the error value so it is properly extended.

Commit 6c5d0512a091480c9f981162227fdb1c9d70e555 introduced this regression.

- ---

 fs/fcntl.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

- --- a/fs/fcntl.c
+++ b/fs/fcntl.c
@@ -115,13 +115,14 @@ out_unlock:

SYSCALL_DEFINE2(dup2, unsigned int, oldfd, unsigned int, newfd) {
+ int ret = oldfd;
if (unlikely(newfd == oldfd)) { /* corner case */
struct files_struct *files = current->files; rcu_read_lock();
if (!fcheck_files(files, oldfd))
- - oldfd = -EBADF;
+ ret = -EBADF;
rcu_read_unlock();
- - return oldfd;
+ return ret;
}
return sys_dup3(oldfd, newfd, 0);
}

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkoIbZcACgkQLPWxlyuTD7JBVACgnNFiWRb4lhW9JgqR36BnT6SD 4uQAoJDcfqV2jsjCV340HlQLkk585Yw6
=IBqW
-----END PGP SIGNATURE-----

#1 lore raw
Quoted 3 lines · Jeff Mahoney

The return value of dup2 when oldfd == newfd and the fd isn't valid is not getting properly sign extended. We end up with 4294967287 instead of -EBADF.

Indeed.

Quoted 1 line

This patch uses a signed int for the error value so it is properly extended.

However, I'd rather move the new variable into the block where it is used, and keep the whole corner-case thing self-contained.

So can you verify that this trivial variation on the patch is ok by you, and I'll commit it as yours with your message? (I realize it's really trivial, and I could just do this myself, but good to get the change ack'ed anyway).

Linus

---
 fs/fcntl.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)
fs/fcntl.c +4 -2
index cc8e4de..1ad7031 100644
@@ -117,11 +117,13 @@ SYSCALL_DEFINE2(dup2, unsigned int, oldfd, unsigned int, newfd)
 {
 	if (unlikely(newfd == oldfd)) { /* corner case */
 		struct files_struct *files = current->files;
+		int retval = oldfd;
+
 		rcu_read_lock();
 		if (!fcheck_files(files, oldfd))
-			oldfd = -EBADF;
+			retval = -EBADF;
 		rcu_read_unlock();
-		return oldfd;
+		return retval;
 	}
 	return sys_dup3(oldfd, newfd, 0);
 }
#2 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 9 lines · Linus Torvalds
Quoted 2 lines · Jeff Mahoney

The return value of dup2 when oldfd == newfd and the fd isn't valid is not getting properly sign extended. We end up with 4294967287 instead of -EBADF.

Indeed.

Quoted 1 line

This patch uses a signed int for the error value so it is properly extended.

Quoted 7 lines

However, I'd rather move the new variable into the block where it is used, and keep the whole corner-case thing self-contained.

So can you verify that this trivial variation on the patch is ok by you, and I'll commit it as yours with your message? (I realize it's really trivial, and I could just do this myself, but good to get the change ack'ed anyway).

Oh, of course. That's fine with me.

- -Jeff

Quoted 17 lines

Linus

---
 fs/fcntl.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)
fs/fcntl.c +2 -0
index cc8e4de..1ad7031 100644
@@ -117,11 +117,13 @@ SYSCALL_DEFINE2(dup2, unsigned int, oldfd, unsigned int, newfd)
 {
 	if (unlikely(newfd == oldfd)) { /* corner case */
 		struct files_struct *files = current->files;
+		int retval = oldfd;
+

fs/fcntl.c SYSCALL_DEFINE2(dup2, unsigned int, oldfd, unsigned int, newfd) · line 129

Quoted 10 lines
fs/fcntl.c +2 -2
 		rcu_read_lock();
 		if (!fcheck_files(files, oldfd))
-			oldfd = -EBADF;
+			retval = -EBADF;
 		rcu_read_unlock();
-		return oldfd;
+		return retval;
 	}
 	return sys_dup3(oldfd, newfd, 0);
 }

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkoIdeUACgkQLPWxlyuTD7J8AACgmhqZUznCzrN25qBX6CzexXBu NkcAn0qOA99rP6c72k+xs7ZRCbhLKKnR
=/rzV
-----END PGP SIGNATURE-----

maintainer
#3 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Jeff Mahoney

The return value of dup2 when oldfd == newfd and the fd isn't valid is not getting properly sign extended. We end up with 4294967287 instead of -EBADF.

Indeed.

Quoted 1 line

This patch uses a signed int for the error value so it is properly extended.

Quoted 7 lines

However, I'd rather move the new variable into the block where it is used, and keep the whole corner-case thing self-contained.

So can you verify that this trivial variation on the patch is ok by you, and I'll commit it as yours with your message? (I realize it's really trivial, and I could just do this myself, but good to get the change ack'ed anyway).

I'm not sure that it's a right fix, actually. Note that userland declaration of that sucker is int dup2(int, int); so should we really take unsigned int as arguments?

#4 lore raw
Quoted 4 lines · Al Viro

I'm not sure that it's a right fix, actually. Note that userland declaration of that sucker is int dup2(int, int); so should we really take unsigned int as arguments?

Hmm. They've been "unsigned int" for as long as our history goes back (including BK), but yes, making them "int" would have hidden this issue as well.

That said, I think we had reasons to do our fd's as unsigned, ie the whole "compare against MAX" thing that doesn't take negative values into account.

In fact, I think we should do more of those. Right now we literally depend on things like "max_fds" being "unsigned int", and that the compiler then turns all the

	if (fd < fdt->max_fds)

tests silently into unsigned tests even when 'fd' is 'int'.

So I suspect we should probably make fs/file.c use _more_ "unsigned int" rather than having less of them.

Linus

maintainer
#5 lore raw
Quoted 12 lines · Linus Torvalds

Hmm. They've been "unsigned int" for as long as our history goes back (including BK), but yes, making them "int" would have hidden this issue as well.

That said, I think we had reasons to do our fd's as unsigned, ie the whole "compare against MAX" thing that doesn't take negative values into account.

In fact, I think we should do more of those. Right now we literally depend on things like "max_fds" being "unsigned int", and that the compiler then turns all the

Quoted 6 lines
	if (fd < fdt->max_fds)

tests silently into unsigned tests even when 'fd' is 'int'.

So I suspect we should probably make fs/file.c use _more_ "unsigned int" rather than having less of them.

What we should do is a careful review of the propagation paths of file descriptors ;-/ As it is, we have an interesting mix of int/unsigned/long used to carry those around, and quite a few of those are used for -E... as well. Note, BTW, that for userland code this bug mostly isn't - libc will convert that value to int before returning to caller, so sign expansion or not, we won't notice. The things like strace will, though...