Skip to contentSkip to conversation

bug in passing file descriptors

DISCUSSION Steve Rago 5 participants 16 messages Oct 7, 2013 – Oct 9, 2013

Subsystems (2)
lore
Share & tools
lore
#1 lore raw
In reply to Steve Rago
Quoted 17 lines · Steve Rago

Sending this to a larger group at mtk's suggestion.

I recently found a bug with passing file descriptors over Unix domain sockets. The attached program illustrates the problem. I believe the source of the problem is in net/core/scm.c. In put_cmsg(), cmlen is calculated as CMSG_SPACE(len) for the purposes of setting msg_controllen, but it probably should be CMSG_LEN(len), at least in this particular case (I didn't investigate other use cases). On a 32-bit platform, a long is a 4-byte quantity and the file descriptor is already aligned to a 4-byte quantity by placing it after the cmsghdr structure. On a 64-byte platform, however, a long is an 8-byte quantity, and CMSG_SPACE() assumes that len is aligned on an 8-byte boundary, which isn't a requirement for passing file descriptors (which are 4-byte quantities; see scm_fp_copy() to verify). Anyway, the end result is that recvmsg(2) returns with msg_controllen 4 bytes larger than it should be on a 64-bit platform. The attached program prints out a warning message when this happens.

Quoted 5 lines

I've tried this on kernels as old as 2.6.18, so it appears this bug has been around for a while. I originally found it on a 3.2.0 kernel. Michael verified it still exists on a 3.11 kernel.

Let me know if you need any more information

ISTM that, in order for further cmsgs to be correctly decoded, all of the relevant things need to match.

put_cmsg uses this layout: cmsghdr, padding, payload, padding. CMSG_SPACE matches that calculation.

scm_detach_fds is the actual code path for SCM_RIGHTS. It does the same thing.

CMSG_DATA also things that there's possible padding after cmsghdr.

So I think everything's okay.

--Andy

author
#2 lore raw
Quoted 7 lines · Andy Lutomirski

ISTM that, in order for further cmsgs to be correctly decoded, all of the relevant things need to match.

put_cmsg uses this layout: cmsghdr, padding, payload, padding. CMSG_SPACE matches that calculation.

Quoted 8 lines

scm_detach_fds is the actual code path for SCM_RIGHTS. It does the same thing.

CMSG_DATA also things that there's possible padding after cmsghdr.

So I think everything's okay.

--Andy

Maybe. So a client expecting to receive x bytes of control information should make sure their buffer is at least CMSG_SPACE(x) bytes long instead of CMSG_LEN(x) bytes long, because you feel compelled to copy the final padding from kernel space to user space? Seems wrong to me. IMHO, the final padding should only come into play when calculating where the next header should begin.

Steve

Steve Rago

author
#0 lore raw

Sending this to a larger group at mtk's suggestion.

I recently found a bug with passing file descriptors over Unix domain sockets. The attached program illustrates the problem. I believe the source of the problem is in net/core/scm.c. In put_cmsg(), cmlen is calculated as CMSG_SPACE(len) for the purposes of setting msg_controllen, but it probably should be CMSG_LEN(len), at least in this particular case (I didn't investigate other use cases). On a 32-bit platform, a long is a 4-byte quantity and the file descriptor is already aligned to a 4-byte quantity by placing it after the cmsghdr structure. On a 64-byte platform, however, a long is an 8-byte quantity, and CMSG_SPACE() assumes that len is aligned on an 8-byte boundary, which isn't a requirement for passing file descriptors (which are 4-byte quantities; see scm_fp_copy() to verify). Anyway, the end result is that recvmsg(2) returns with msg_controllen 4 bytes larger than it should be on a 64-bit platform. The attached program prints out a warning message when this happens.

I've tried this on kernels as old as 2.6.18, so it appears this bug has been around for a while. I originally found it on a 3.2.0 kernel. Michael verified it still exists on a 3.11 kernel.

Let me know if you need any more information

Steve Rago
[email protected]

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>
#include <sys/socket.h>
#include <sys/stat.h>

void
sendfd(int sockfd, int fd)

{
	char dummybuf;
	struct iovec iov;
	struct msghdr mh;
	struct cmsghdr *cmp = malloc(CMSG_LEN(sizeof(int)));

	if (cmp == NULL) {
		fprintf(stderr, "can't malloc: %s\n", strerror(errno));
		exit(1);
	}
	iov.iov_base = &dummybuf;
	iov.iov_len = 1;
	mh.msg_iov = &iov;
	mh.msg_iovlen = 1;
	mh.msg_name = NULL;
	mh.msg_namelen = 0;
	mh.msg_control = cmp;
	mh.msg_controllen = CMSG_LEN(sizeof(int));
	cmp->cmsg_level = SOL_SOCKET;
	cmp->cmsg_type = SCM_RIGHTS;
	cmp->cmsg_len = CMSG_LEN(sizeof(int));
	*(int *)CMSG_DATA(cmp) = fd;
	if (sendmsg(sockfd, &mh, 0) < 0) {
		fprintf(stderr, "sendmsg failed: %s\n", strerror(errno));
		exit(1);
	}
	free(cmp);
}

int
recvfd(int sockfd)

{
	struct msghdr mh;
	int nfd;
	char dummybuf;
	struct iovec iov;
	long csz = CMSG_LEN(sizeof(int));
	struct cmsghdr *cmp = malloc(csz);

	if (cmp == NULL) {
		fprintf(stderr, "can't malloc: %s\n", strerror(errno));
		exit(1);
	}
	iov.iov_base = &dummybuf;
	iov.iov_len = 1;
	mh.msg_iov = &iov;
	mh.msg_iovlen = 1;
	mh.msg_name = NULL;
	mh.msg_namelen = 0;
	mh.msg_control = cmp;
	mh.msg_controllen = csz;
	if (recvmsg(sockfd, &mh, 0) < 0) {
		fprintf(stderr, "recvmsg failed: %s\n", strerror(errno));
		exit(1);
	}
	if (mh.msg_controllen == 0) {
		fprintf(stderr, "no control data in message\n");
		exit(1);
	}
	if (mh.msg_controllen != csz)
		fprintf(stderr, "WARNING: controllen %ld, should be %ld\n", mh.msg_controllen, csz);
	nfd = *(int *)CMSG_DATA(cmp);
	free(cmp);
	return(nfd);
}

int
main()

{
	int nfd, nfd2;
	int fd[2];
	struct stat sbuf[2];

	printf("sizeof(int) = %d\n", (int)sizeof(int));
	printf("sizeof(long) = %d\n", (int)sizeof(long));
	printf("sizeof(size_t) = %d\n", (int)sizeof(size_t));
	printf("CMSG_LEN(sizeof(int)) = %d\n", (int)CMSG_LEN(sizeof(int)));
	printf("CMSG_SPACE(sizeof(int)) = %d\n", (int)CMSG_SPACE(sizeof(int)));
	if (socketpair(AF_UNIX, SOCK_STREAM, 0, fd) < 0) {
		fprintf(stderr, "can't create socket pair: %s\n", strerror(errno));
		exit(1);
	}
	if ((nfd = open("/etc/services", O_RDONLY)) < 0) {
		fprintf(stderr, "can't open /etc/services: %s", strerror(errno));
		exit(1);
	}
	sendfd(fd[0], nfd);
	nfd2 = recvfd(fd[1]);
	if (fstat(nfd, &sbuf[0]) < 0 || fstat(nfd2, &sbuf[1]) < 0) {
		fprintf(stderr, "fstat failed: %s\n", strerror(errno));
		exit(1);
	}
	if (sbuf[0].st_dev == sbuf[1].st_dev && sbuf[0].st_ino == sbuf[1].st_ino)
		printf("the files are the same\n");
	else
		printf("the files are different\n");
	exit(0);
}
maintainer
#3 lore raw
In reply to Steve Rago

From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 15:06:02 -0400

Quoted 6 lines

Maybe. So a client expecting to receive x bytes of control information should make sure their buffer is at least CMSG_SPACE(x) bytes long instead of CMSG_LEN(x) bytes long, because you feel compelled to copy the final padding from kernel space to user space? Seems wrong to me. IMHO, the final padding should only come into play when calculating where the next header should begin.

Yes, all control messages must be aligned to, and be of a length of a multiple of, "sizeof(long)".

This is the only correct way to program control messages.

author
#4 lore raw
Quoted 10 lines · David Miller

From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 15:06:02 -0400

Quoted 6 lines

Maybe. So a client expecting to receive x bytes of control information should make sure their buffer is at least CMSG_SPACE(x) bytes long instead of CMSG_LEN(x) bytes long, because you feel compelled to copy the final padding from kernel space to user space? Seems wrong to me. IMHO, the final padding should only come into play when calculating where the next header should begin.

Quoted 5 lines

Yes, all control messages must be aligned to, and be of a length of a multiple of, "sizeof(long)".

This is the only correct way to program control messages.

Except when sizeof(long) can change and you need to maintain binary compatibility with older applications. x86 comes to mind as a relevant example: used to be 32 bits, but is 64 now.

Steve

maintainer
#5 lore raw

From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 15:17:24 -0400

Quoted 3 lines

Except when sizeof(long) can change and you need to maintain binary compatibility with older applications. x86 comes to mind as a relevant example: used to be 32 bits, but is 64 now.

There is no compatability issue.

32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.

author
#6 lore raw
Quoted 5 lines · David Miller

There is no compatability issue.

32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.

Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not going to overrun my buffer when CMSG_SPACE led me to allocate an insufficient amount of memory needed to account for padding on the 64-bit platform?

By the way, FreeBSD, Mac OS X, and Solaris all behave as I described, so if you're happy with Linux behaving differently, then I'll stop wasting bandwidth.

Steve

maintainer
#7 lore raw

From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 16:29:15 -0400

Quoted 11 lines · David Miller
Quoted 5 lines

There is no compatability issue.

32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.

Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not going to overrun my buffer when CMSG_SPACE led me to allocate an insufficient amount of memory needed to account for padding on the 64-bit platform?

We have a compatability layer that gives 32-bit applications the same behavior as if they had run on a 32-bit machine.

Search around for the MSG_MSG_COMPAT flag and how that is used in net/socket.c

#8 lore raw
Quoted 15 lines · David Miller

From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 16:29:15 -0400

Quoted 11 lines · David Miller
Quoted 5 lines

There is no compatability issue.

32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.

Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not going to overrun my buffer when CMSG_SPACE led me to allocate an insufficient amount of memory needed to account for padding on the 64-bit platform?

Quoted 5 lines

We have a compatability layer that gives 32-bit applications the same behavior as if they had run on a 32-bit machine.

Search around for the MSG_MSG_COMPAT flag and how that is used in net/socket.c

But it seems the compat layer doesn't handle this correctly, otherwise Steve's original test case would work.

Must be a bug somewhere in the compat layer.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
#15 lore raw
In reply to Steve Rago
Quoted 11 lines
Quoted 5 lines

There is no compatability issue.

32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.

Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not
going to overrun my buffer when CMSG_SPACE led me to allocate an insufficient amount of memory needed to account for
padding on the 64-bit platform?

Worth checking what happens if you try to send an fd between a 32 bit application and a 64bit one.
Also is the fd actually closed if the receiving applications buffer isn't big enough.

David

author
#9 lore raw
In reply to Andi Kleen
Quoted 23 lines · Andi Kleen
Quoted 15 lines · David Miller

From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 16:29:15 -0400

Quoted 11 lines · David Miller
Quoted 5 lines

There is no compatability issue.

32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.

Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not going to overrun my buffer when CMSG_SPACE led me to allocate an insufficient amount of memory needed to account for padding on the 64-bit platform?

Quoted 5 lines

We have a compatability layer that gives 32-bit applications the same behavior as if they had run on a 32-bit machine.

Search around for the MSG_MSG_COMPAT flag and how that is used in net/socket.c

Quoted 7 lines

But it seems the compat layer doesn't handle this correctly, otherwise Steve's original test case would work.

Must be a bug somewhere in the compat layer.

-Andi

I did some research last night and I think the problem stems from an underspecified standard. CMSG_LEN and CMSG_SPACE seem to have originated with RFC 2292, which has since been obsoleted by RFC 3542. The difference is that CMSG_SPACE accounts for padding at the end, which is needed when you stuff multiple cmsghdr objects in the same buffer. CMSG_LEN is required to be used to initialize the cmsg_len member of the structure. When you only have one cmsghdr object in your call to recvmsg, it is unclear whether you need to have a buffer as large as CMSG_SPACE or CMSG_LEN. Historically, BSD-based platforms never had these macros and didn't return the ancillary data if the space provided by the application wasn't big enough. Linux, *which has a bug*, won't copy more bytes than cmsg_len specifies when the application uses CMSG_LEN instead of CMSG_SPACE, but then lies to the application by overwriting msg_controllen. Look in put_cmsg() in net/core/scm.c: "cmlen" is calculated using CM_LEN, and then msg_controllen is checked against it to make sure you don't overwrite the user's buffer. However, in recvmsg(), msg_controllen is overwritten by "msg_sys.msg_control - cmsg_ptr".
msg_control was incremented by CMSG_SPACE at the end of scm_detach_fds().

The Linux kernel code seems to copy the right amount of data, even in the compat case, as far as I can tell. The only bug I see is that msg_controllen returns from recvmsg() with the wrong value.

Steve

#10 lore raw
Quoted 55 lines · Steve Rago
Quoted 27 lines · Andi Kleen
Quoted 17 lines · David Miller

From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 16:29:15 -0400

Quoted 12 lines · David Miller
Quoted 6 lines

There is no compatability issue.

32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.

Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not going to overrun my buffer when CMSG_SPACE led me to allocate an insufficient amount of memory needed to account for padding on the 64-bit platform?

Quoted 5 lines

We have a compatability layer that gives 32-bit applications the same behavior as if they had run on a 32-bit machine.

Search around for the MSG_MSG_COMPAT flag and how that is used in net/socket.c

Quoted 7 lines

But it seems the compat layer doesn't handle this correctly, otherwise Steve's original test case would work.

Must be a bug somewhere in the compat layer.

-Andi

I did some research last night and I think the problem stems from an underspecified standard. CMSG_LEN and CMSG_SPACE seem to have originated with RFC 2292, which has since been obsoleted by RFC 3542. The difference is that CMSG_SPACE accounts for padding at the end, which is needed when you stuff multiple cmsghdr objects in the same buffer. CMSG_LEN is required to be used to initialize the cmsg_len member of the structure. When you only have one cmsghdr object in your call to recvmsg, it is unclear whether you need to have a buffer as large as CMSG_SPACE or CMSG_LEN. Historically, BSD-based platforms never had these macros and didn't return the ancillary data if the space provided by the application wasn't big enough. Linux, *which has a bug*, won't copy more bytes than cmsg_len specifies when the application uses CMSG_LEN instead of CMSG_SPACE, but then lies to the application by overwriting msg_controllen. Look in put_cmsg() in net/core/scm.c: "cmlen" is calculated using CM_LEN, and then msg_controllen is checked against it to make sure you don't overwrite the user's buffer. However, in recvmsg(), msg_controllen is overwritten by "msg_sys.msg_control
- cmsg_ptr". msg_control was incremented by CMSG_SPACE at the end of scm_detach_fds().

Quoted 3 lines

The Linux kernel code seems to copy the right amount of data, even in the compat case, as far as I can tell. The only bug I see is that msg_controllen returns from recvmsg() with the wrong value.

I don't see how the other behavior would be any less surprising. Suppose you had a control message with CMSG_LEN=12 and CMSG_SPACE=16. Then, with the semantics you want, one of them takes 12 bytes, two of them take 28 bytes, three take 44 bytes, etc.

That seems considerably more surprising than having then take 16, 32, and 48 bytes respectively.

--Andy

author
#11 lore raw
Quoted 65 lines · Andy Lutomirski
Quoted 55 lines · Steve Rago
Quoted 27 lines · Andi Kleen
Quoted 17 lines · David Miller

From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 16:29:15 -0400

Quoted 12 lines · David Miller
Quoted 6 lines

There is no compatability issue.

32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.

Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not going to overrun my buffer when CMSG_SPACE led me to allocate an insufficient amount of memory needed to account for padding on the 64-bit platform?

Quoted 5 lines

We have a compatability layer that gives 32-bit applications the same behavior as if they had run on a 32-bit machine.

Search around for the MSG_MSG_COMPAT flag and how that is used in net/socket.c

Quoted 7 lines

But it seems the compat layer doesn't handle this correctly, otherwise Steve's original test case would work.

Must be a bug somewhere in the compat layer.

-Andi

I did some research last night and I think the problem stems from an underspecified standard. CMSG_LEN and CMSG_SPACE seem to have originated with RFC 2292, which has since been obsoleted by RFC 3542. The difference is that CMSG_SPACE accounts for padding at the end, which is needed when you stuff multiple cmsghdr objects in the same buffer. CMSG_LEN is required to be used to initialize the cmsg_len member of the structure. When you only have one cmsghdr object in your call to recvmsg, it is unclear whether you need to have a buffer as large as CMSG_SPACE or CMSG_LEN. Historically, BSD-based platforms never had these macros and didn't return the ancillary data if the space provided by the application wasn't big enough. Linux, *which has a bug*, won't copy more bytes than cmsg_len specifies when the application uses CMSG_LEN instead of CMSG_SPACE, but then lies to the application by overwriting msg_controllen. Look in put_cmsg() in net/core/scm.c: "cmlen" is calculated using CM_LEN, and then msg_controllen is checked against it to make sure you don't overwrite the user's buffer. However, in recvmsg(), msg_controllen is overwritten by "msg_sys.msg_control
- cmsg_ptr". msg_control was incremented by CMSG_SPACE at the end of scm_detach_fds().

Quoted 3 lines

The Linux kernel code seems to copy the right amount of data, even in the compat case, as far as I can tell. The only bug I see is that msg_controllen returns from recvmsg() with the wrong value.

I don't see how the other behavior would be any less surprising. Suppose you had a control message with CMSG_LEN=12 and CMSG_SPACE=16. Then, with the semantics you want, one of them takes 12 bytes, two of them take 28 bytes, three take 44 bytes, etc.

Quoted 5 lines

That seems considerably more surprising than having then take 16, 32, and 48 bytes respectively.

--Andy

I just want the semantics to be consistent. If you want Linux to always require applications that call recvmsg to provide a buffer size of CMSG_SPACE bytes long to retrieve control information, then fail the system call when the buffer is smaller. But if you do this, you risk breaking applications that work with FreeBSD, Mac OS X, Solaris, and probably a few others.

No wonder the Single UNIX Specification didn't standardize these two macros.

Regardless, copying 20 bytes and telling me you copied 24 is misleading and wrong.

Steve

#12 lore raw
Quoted 6 lines

I just want the semantics to be consistent. If you want Linux to always require applications that call recvmsg to provide a buffer size of CMSG_SPACE bytes long to retrieve control information, then fail the system call when the buffer is smaller. But if you do this, you risk breaking applications that work with FreeBSD, Mac OS X, Solaris, and probably a few others.

The primary concern is to be binary compatible with Linux.

But not being compatible between 32bit and 64bit Linux processes on the same host would seem like a serious problem to me.

Quoted 1 line

Regardless, copying 20 bytes and telling me you copied 24 is misleading and wrong.

The question is could it break existing Linux applications to change it? And would it help with the 32/64bit compatibility?

If not some other way to fix the compat layer would need to be found.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
author
#13 lore raw
Quoted 12 lines · Andi Kleen
Quoted 6 lines

I just want the semantics to be consistent. If you want Linux to always require applications that call recvmsg to provide a buffer size of CMSG_SPACE bytes long to retrieve control information, then fail the system call when the buffer is smaller. But if you do this, you risk breaking applications that work with FreeBSD, Mac OS X, Solaris, and probably a few others.

The primary concern is to be binary compatible with Linux.

But not being compatible between 32bit and 64bit Linux processes on the same host would seem like a serious problem to me.

Quoted 9 lines
Quoted 1 line

Regardless, copying 20 bytes and telling me you copied 24 is misleading and wrong.

The question is could it break existing Linux applications to change it? And would it help with the 32/64bit compatibility?

If not some other way to fix the compat layer would need to be found.

-Andi

I'm not sure if a 64-bit process and a 32-bit process exchange file descriptors on the same system has a problem. It certainly looks like the compat code does the right thing. I can test this tonight if you want. The discrepancy arises because file descriptors are 4-byte quantities and treated differently (for example, when more than one is specified, each one isn't padded to an 8-byte boundary).

The way I discovered the problem is that I had an example program in APUE that was validating that msg_controllen had the correct value after calling recvmsg(). It worked on my 32-bit platform, but when I recompiled it and ran it on my 64-bit platform, the test failed, because msg_controllen was larger than the size that was sent via sendmsg().

Steve

author
#14 lore raw
Quoted 9 lines · Steve Rago
Quoted 8 lines · Andi Kleen
Quoted 6 lines

I just want the semantics to be consistent. If you want Linux to always require applications that call recvmsg to provide a buffer size of CMSG_SPACE bytes long to retrieve control information, then fail the system call when the buffer is smaller. But if you do this, you risk breaking applications that work with FreeBSD, Mac OS X, Solaris, and probably a few others.

The primary concern is to be binary compatible with Linux.

Not to application developers, by the way.

Quoted 9 lines
Quoted 9 lines

But not being compatible between 32bit and 64bit Linux processes on the same host would seem like a serious problem to me.

Quoted 1 line

Regardless, copying 20 bytes and telling me you copied 24 is misleading and wrong.

The question is could it break existing Linux applications to change it? And would it help with the 32/64bit compatibility?

Quoted 7 lines
Quoted 4 lines

If not some other way to fix the compat layer would need to be found.

-Andi

I'm not sure if a 64-bit process and a 32-bit process exchange file descriptors on the same system has a problem. It certainly looks like the compat code does the right thing. I can test this tonight if you want.

I tested all combinations and they work fine as far as Linux binary compatibility is concerned.

Steve