bug in passing file descriptors
Subsystems (2)
Sending this to a larger group at mtk's suggestion.
I recently found a bug with passing file descriptors over Unix domain sockets. The attached program illustrates the problem. I believe the source of the problem is in net/core/scm.c. In put_cmsg(), cmlen is calculated as CMSG_SPACE(len) for the purposes of setting msg_controllen, but it probably should be CMSG_LEN(len), at least in this particular case (I didn't investigate other use cases). On a 32-bit platform, a long is a 4-byte quantity and the file descriptor is already aligned to a 4-byte quantity by placing it after the cmsghdr structure. On a 64-byte platform, however, a long is an 8-byte quantity, and CMSG_SPACE() assumes that len is aligned on an 8-byte boundary, which isn't a requirement for passing file descriptors (which are 4-byte quantities; see scm_fp_copy() to verify). Anyway, the end result is that recvmsg(2) returns with msg_controllen 4 bytes larger than it should be on a 64-bit platform. The attached program prints out a warning message when this happens.
I've tried this on kernels as old as 2.6.18, so it appears this bug has been around for a while. I originally found it on a 3.2.0 kernel. Michael verified it still exists on a 3.11 kernel.
Let me know if you need any more information
Steve Rago
[email protected]
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>
#include <sys/socket.h>
#include <sys/stat.h>
voidsendfd(int sockfd, int fd)
{
char dummybuf;
struct iovec iov;
struct msghdr mh;
struct cmsghdr *cmp = malloc(CMSG_LEN(sizeof(int)));
if (cmp == NULL) {
fprintf(stderr, "can't malloc: %s\n", strerror(errno));
exit(1);
}
iov.iov_base = &dummybuf;
iov.iov_len = 1;
mh.msg_iov = &iov;
mh.msg_iovlen = 1;
mh.msg_name = NULL;
mh.msg_namelen = 0;
mh.msg_control = cmp;
mh.msg_controllen = CMSG_LEN(sizeof(int));
cmp->cmsg_level = SOL_SOCKET;
cmp->cmsg_type = SCM_RIGHTS;
cmp->cmsg_len = CMSG_LEN(sizeof(int));
*(int *)CMSG_DATA(cmp) = fd;
if (sendmsg(sockfd, &mh, 0) < 0) {
fprintf(stderr, "sendmsg failed: %s\n", strerror(errno));
exit(1);
}
free(cmp);
}int
recvfd(int sockfd)
{
struct msghdr mh;
int nfd;
char dummybuf;
struct iovec iov;
long csz = CMSG_LEN(sizeof(int));
struct cmsghdr *cmp = malloc(csz);
if (cmp == NULL) {
fprintf(stderr, "can't malloc: %s\n", strerror(errno));
exit(1);
}
iov.iov_base = &dummybuf;
iov.iov_len = 1;
mh.msg_iov = &iov;
mh.msg_iovlen = 1;
mh.msg_name = NULL;
mh.msg_namelen = 0;
mh.msg_control = cmp;
mh.msg_controllen = csz;
if (recvmsg(sockfd, &mh, 0) < 0) {
fprintf(stderr, "recvmsg failed: %s\n", strerror(errno));
exit(1);
}
if (mh.msg_controllen == 0) {
fprintf(stderr, "no control data in message\n");
exit(1);
}
if (mh.msg_controllen != csz)
fprintf(stderr, "WARNING: controllen %ld, should be %ld\n", mh.msg_controllen, csz);
nfd = *(int *)CMSG_DATA(cmp);
free(cmp);
return(nfd);
}intmain()
{
int nfd, nfd2;
int fd[2];
struct stat sbuf[2];
printf("sizeof(int) = %d\n", (int)sizeof(int));
printf("sizeof(long) = %d\n", (int)sizeof(long));
printf("sizeof(size_t) = %d\n", (int)sizeof(size_t));
printf("CMSG_LEN(sizeof(int)) = %d\n", (int)CMSG_LEN(sizeof(int)));
printf("CMSG_SPACE(sizeof(int)) = %d\n", (int)CMSG_SPACE(sizeof(int)));
if (socketpair(AF_UNIX, SOCK_STREAM, 0, fd) < 0) {
fprintf(stderr, "can't create socket pair: %s\n", strerror(errno));
exit(1);
}
if ((nfd = open("/etc/services", O_RDONLY)) < 0) {
fprintf(stderr, "can't open /etc/services: %s", strerror(errno));
exit(1);
}
sendfd(fd[0], nfd);
nfd2 = recvfd(fd[1]);
if (fstat(nfd, &sbuf[0]) < 0 || fstat(nfd2, &sbuf[1]) < 0) {
fprintf(stderr, "fstat failed: %s\n", strerror(errno));
exit(1);
}
if (sbuf[0].st_dev == sbuf[1].st_dev && sbuf[0].st_ino == sbuf[1].st_ino)
printf("the files are the same\n");
else
printf("the files are different\n");
exit(0);
}Quoted 17 lines · Steve Rago
Sending this to a larger group at mtk's suggestion.
I recently found a bug with passing file descriptors over Unix domain sockets. The attached program illustrates the problem. I believe the source of the problem is in
net/core/scm.c. Input_cmsg(), cmlen is calculated asCMSG_SPACE(len)for the purposes of settingmsg_controllen, but it probably should beCMSG_LEN(len), at least in this particular case (I didn't investigate other use cases). On a 32-bit platform, a long is a 4-byte quantity and the file descriptor is already aligned to a 4-byte quantity by placing it after the cmsghdr structure. On a 64-byte platform, however, a long is an 8-byte quantity, andCMSG_SPACE()assumes that len is aligned on an 8-byte boundary, which isn't a requirement for passing file descriptors (which are 4-byte quantities; seescm_fp_copy()to verify). Anyway, the end result is that recvmsg(2) returns withmsg_controllen4 bytes larger than it should be on a 64-bit platform. The attached program prints out a warning message when this happens.
Quoted 5 lines
I've tried this on kernels as old as 2.6.18, so it appears this bug has been around for a while. I originally found it on a 3.2.0 kernel. Michael verified it still exists on a 3.11 kernel.
Let me know if you need any more information
ISTM that, in order for further cmsgs to be correctly decoded, all of the relevant things need to match.
put_cmsg uses this layout: cmsghdr, padding, payload, padding. CMSG_SPACE matches that calculation.
scm_detach_fds is the actual code path for SCM_RIGHTS. It does the same thing.
CMSG_DATA also things that there's possible padding after cmsghdr.
So I think everything's okay.
--Andy
Quoted 7 lines · Andy Lutomirski
ISTM that, in order for further cmsgs to be correctly decoded, all of the relevant things need to match.
put_cmsguses this layout: cmsghdr, padding, payload, padding.CMSG_SPACEmatches that calculation.
Quoted 8 lines
scm_detach_fdsis the actual code path forSCM_RIGHTS. It does the same thing.
CMSG_DATAalso things that there's possible padding after cmsghdr.So I think everything's okay.
--Andy
Maybe. So a client expecting to receive x bytes of control information should make sure their buffer is at least CMSG_SPACE(x) bytes long instead of CMSG_LEN(x) bytes long, because you feel compelled to copy the final padding from kernel space to user space? Seems wrong to me. IMHO, the final padding should only come into play when calculating where the next header should begin.
Steve
David Miller
maintainerFrom: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 15:06:02 -0400
Quoted 6 lines
Maybe. So a client expecting to receive x bytes of control information should make sure their buffer is at least
CMSG_SPACE(x)bytes long instead ofCMSG_LEN(x)bytes long, because you feel compelled to copy the final padding from kernel space to user space? Seems wrong to me. IMHO, the final padding should only come into play when calculating where the next header should begin.
Yes, all control messages must be aligned to, and be of a length of a multiple of, "sizeof(long)".
This is the only correct way to program control messages.
Quoted 10 lines · David Miller
From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 15:06:02 -0400Quoted 6 lines
Maybe. So a client expecting to receive x bytes of control information should make sure their buffer is at least
CMSG_SPACE(x)bytes long instead ofCMSG_LEN(x)bytes long, because you feel compelled to copy the final padding from kernel space to user space? Seems wrong to me. IMHO, the final padding should only come into play when calculating where the next header should begin.
Quoted 5 lines
Yes, all control messages must be aligned to, and be of a length of a multiple of, "sizeof(long)".
This is the only correct way to program control messages.
Except when sizeof(long) can change and you need to maintain binary compatibility with older applications. x86 comes to mind as a relevant example: used to be 32 bits, but is 64 now.
Steve
David Miller
maintainerFrom: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 15:17:24 -0400
Quoted 3 lines
Except when sizeof(long) can change and you need to maintain binary compatibility with older applications. x86 comes to mind as a relevant example: used to be 32 bits, but is 64 now.
There is no compatability issue.
32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.
Quoted 5 lines · David Miller
There is no compatability issue.
32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.
Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not going to overrun my buffer when CMSG_SPACE led me to allocate an insufficient amount of memory needed to account for padding on the 64-bit platform?
By the way, FreeBSD, Mac OS X, and Solaris all behave as I described, so if you're happy with Linux behaving differently, then I'll stop wasting bandwidth.
Steve
David Miller
maintainerFrom: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 16:29:15 -0400
Quoted 11 lines · David Miller
Quoted 5 lines
There is no compatability issue.
32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.
Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not going to overrun my buffer when
CMSG_SPACEled me to allocate an insufficient amount of memory needed to account for padding on the 64-bit platform?
We have a compatability layer that gives 32-bit applications the same behavior as if they had run on a 32-bit machine.
Search around for the MSG_MSG_COMPAT flag and how that is used in net/socket.c
Quoted 15 lines · David Miller
From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 16:29:15 -0400Quoted 11 lines · David Miller
Quoted 5 lines
There is no compatability issue.
32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.
Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not going to overrun my buffer when
CMSG_SPACEled me to allocate an insufficient amount of memory needed to account for padding on the 64-bit platform?
Quoted 5 lines
We have a compatability layer that gives 32-bit applications the same behavior as if they had run on a 32-bit machine.
Search around for the
MSG_MSG_COMPATflag and how that is used innet/socket.c
But it seems the compat layer doesn't handle this correctly, otherwise Steve's original test case would work.
Must be a bug somewhere in the compat layer.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
Quoted 23 lines · Andi Kleen
Quoted 15 lines · David Miller
From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 16:29:15 -0400Quoted 11 lines · David Miller
Quoted 5 lines
There is no compatability issue.
32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.
Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not going to overrun my buffer when
CMSG_SPACEled me to allocate an insufficient amount of memory needed to account for padding on the 64-bit platform?Quoted 5 lines
We have a compatability layer that gives 32-bit applications the same behavior as if they had run on a 32-bit machine.
Search around for the
MSG_MSG_COMPATflag and how that is used innet/socket.c
Quoted 7 lines
But it seems the compat layer doesn't handle this correctly, otherwise Steve's original test case would work.
Must be a bug somewhere in the compat layer.
-Andi
I did some research last night and I think the problem stems from an underspecified standard. CMSG_LEN and CMSG_SPACE seem to have originated with RFC 2292, which has since been obsoleted by RFC 3542. The difference is that CMSG_SPACE accounts for padding at the end, which is needed when you stuff multiple cmsghdr objects in the same buffer. CMSG_LEN is required to be used to initialize the cmsg_len member of the structure. When you only have one cmsghdr object in your call to recvmsg, it is unclear whether you need to have a buffer as large as CMSG_SPACE or CMSG_LEN. Historically, BSD-based platforms never had these macros and didn't return the ancillary data if the space provided by the application wasn't big enough. Linux, *which has a bug*, won't copy more bytes than cmsg_len specifies when the application uses CMSG_LEN instead of CMSG_SPACE, but then lies to the application by overwriting msg_controllen. Look in put_cmsg() in net/core/scm.c: "cmlen" is calculated using CM_LEN, and then msg_controllen is checked against it to make sure you don't overwrite the user's buffer. However, in recvmsg(), msg_controllen is overwritten by "msg_sys.msg_control - cmsg_ptr".
msg_control was incremented by CMSG_SPACE at the end of scm_detach_fds().
The Linux kernel code seems to copy the right amount of data, even in the compat case, as far as I can tell. The only bug I see is that msg_controllen returns from recvmsg() with the wrong value.
Steve
Quoted 55 lines · Steve Rago
Quoted 27 lines · Andi Kleen
Quoted 17 lines · David Miller
From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 16:29:15 -0400Quoted 12 lines · David Miller
Quoted 6 lines
There is no compatability issue.
32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.
Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not going to overrun my buffer when
CMSG_SPACEled me to allocate an insufficient amount of memory needed to account for padding on the 64-bit platform?Quoted 5 lines
We have a compatability layer that gives 32-bit applications the same behavior as if they had run on a 32-bit machine.
Search around for the
MSG_MSG_COMPATflag and how that is used innet/socket.cQuoted 7 lines
But it seems the compat layer doesn't handle this correctly, otherwise Steve's original test case would work.
Must be a bug somewhere in the compat layer.
-Andi
I did some research last night and I think the problem stems from an underspecified standard.
CMSG_LENandCMSG_SPACEseem to have originated with RFC 2292, which has since been obsoleted by RFC 3542. The difference is thatCMSG_SPACEaccounts for padding at the end, which is needed when you stuff multiple cmsghdr objects in the same buffer.CMSG_LENis required to be used to initialize thecmsg_lenmember of the structure. When you only have one cmsghdr object in your call to recvmsg, it is unclear whether you need to have a buffer as large asCMSG_SPACEorCMSG_LEN. Historically, BSD-based platforms never had these macros and didn't return the ancillary data if the space provided by the application wasn't big enough. Linux, *which has a bug*, won't copy more bytes thancmsg_lenspecifies when the application usesCMSG_LENinstead ofCMSG_SPACE, but then lies to the application by overwritingmsg_controllen. Look input_cmsg()innet/core/scm.c: "cmlen" is calculated usingCM_LEN, and thenmsg_controllenis checked against it to make sure you don't overwrite the user's buffer. However, inrecvmsg(),msg_controllenis overwritten by "msg_sys.msg_control
-cmsg_ptr".msg_controlwas incremented byCMSG_SPACEat the end ofscm_detach_fds().
Quoted 3 lines
The Linux kernel code seems to copy the right amount of data, even in the compat case, as far as I can tell. The only bug I see is that
msg_controllenreturns fromrecvmsg()with the wrong value.
I don't see how the other behavior would be any less surprising. Suppose you had a control message with CMSG_LEN=12 and CMSG_SPACE=16. Then, with the semantics you want, one of them takes 12 bytes, two of them take 28 bytes, three take 44 bytes, etc.
That seems considerably more surprising than having then take 16, 32, and 48 bytes respectively.
--Andy
Quoted 65 lines · Andy Lutomirski
Quoted 55 lines · Steve Rago
Quoted 27 lines · Andi Kleen
Quoted 17 lines · David Miller
From: Steve Rago <[email protected]>
Date: Mon, 7 Oct 2013 16:29:15 -0400Quoted 12 lines · David Miller
Quoted 6 lines
There is no compatability issue.
32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.
Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not going to overrun my buffer when
CMSG_SPACEled me to allocate an insufficient amount of memory needed to account for padding on the 64-bit platform?Quoted 5 lines
We have a compatability layer that gives 32-bit applications the same behavior as if they had run on a 32-bit machine.
Search around for the
MSG_MSG_COMPATflag and how that is used innet/socket.cQuoted 7 lines
But it seems the compat layer doesn't handle this correctly, otherwise Steve's original test case would work.
Must be a bug somewhere in the compat layer.
-Andi
I did some research last night and I think the problem stems from an underspecified standard.
CMSG_LENandCMSG_SPACEseem to have originated with RFC 2292, which has since been obsoleted by RFC 3542. The difference is thatCMSG_SPACEaccounts for padding at the end, which is needed when you stuff multiple cmsghdr objects in the same buffer.CMSG_LENis required to be used to initialize thecmsg_lenmember of the structure. When you only have one cmsghdr object in your call to recvmsg, it is unclear whether you need to have a buffer as large asCMSG_SPACEorCMSG_LEN. Historically, BSD-based platforms never had these macros and didn't return the ancillary data if the space provided by the application wasn't big enough. Linux, *which has a bug*, won't copy more bytes thancmsg_lenspecifies when the application usesCMSG_LENinstead ofCMSG_SPACE, but then lies to the application by overwritingmsg_controllen. Look input_cmsg()innet/core/scm.c: "cmlen" is calculated usingCM_LEN, and thenmsg_controllenis checked against it to make sure you don't overwrite the user's buffer. However, inrecvmsg(),msg_controllenis overwritten by "msg_sys.msg_control
-cmsg_ptr".msg_controlwas incremented byCMSG_SPACEat the end ofscm_detach_fds().Quoted 3 lines
The Linux kernel code seems to copy the right amount of data, even in the compat case, as far as I can tell. The only bug I see is that
msg_controllenreturns fromrecvmsg()with the wrong value.I don't see how the other behavior would be any less surprising. Suppose you had a control message with CMSG_LEN=12 and CMSG_SPACE=16. Then, with the semantics you want, one of them takes 12 bytes, two of them take 28 bytes, three take 44 bytes, etc.
Quoted 5 lines
That seems considerably more surprising than having then take 16, 32, and 48 bytes respectively.
--Andy
I just want the semantics to be consistent. If you want Linux to always require applications that call recvmsg to provide a buffer size of CMSG_SPACE bytes long to retrieve control information, then fail the system call when the buffer is smaller. But if you do this, you risk breaking applications that work with FreeBSD, Mac OS X, Solaris, and probably a few others.
No wonder the Single UNIX Specification didn't standardize these two macros.
Regardless, copying 20 bytes and telling me you copied 24 is misleading and wrong.
Steve
Quoted 6 lines
I just want the semantics to be consistent. If you want Linux to always require applications that call recvmsg to provide a buffer size of
CMSG_SPACEbytes long to retrieve control information, then fail the system call when the buffer is smaller. But if you do this, you risk breaking applications that work with FreeBSD, Mac OS X, Solaris, and probably a few others.
The primary concern is to be binary compatible with Linux.
But not being compatible between 32bit and 64bit Linux processes on the same host would seem like a serious problem to me.
Quoted 1 line
Regardless, copying 20 bytes and telling me you copied 24 is misleading and wrong.
The question is could it break existing Linux applications to change it? And would it help with the 32/64bit compatibility?
If not some other way to fix the compat layer would need to be found.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 12 lines · Andi Kleen
Quoted 6 lines
I just want the semantics to be consistent. If you want Linux to always require applications that call recvmsg to provide a buffer size of
CMSG_SPACEbytes long to retrieve control information, then fail the system call when the buffer is smaller. But if you do this, you risk breaking applications that work with FreeBSD, Mac OS X, Solaris, and probably a few others.The primary concern is to be binary compatible with Linux.
But not being compatible between 32bit and 64bit Linux processes on the same host would seem like a serious problem to me.
Quoted 9 lines
Quoted 1 line
Regardless, copying 20 bytes and telling me you copied 24 is misleading and wrong.
The question is could it break existing Linux applications to change it? And would it help with the 32/64bit compatibility?
If not some other way to fix the compat layer would need to be found.
-Andi
I'm not sure if a 64-bit process and a 32-bit process exchange file descriptors on the same system has a problem. It certainly looks like the compat code does the right thing. I can test this tonight if you want. The discrepancy arises because file descriptors are 4-byte quantities and treated differently (for example, when more than one is specified, each one isn't padded to an 8-byte boundary).
The way I discovered the problem is that I had an example program in APUE that was validating that msg_controllen had the correct value after calling recvmsg(). It worked on my 32-bit platform, but when I recompiled it and ran it on my 64-bit platform, the test failed, because msg_controllen was larger than the size that was sent via sendmsg().
Steve
Quoted 9 lines · Steve Rago
Quoted 8 lines · Andi Kleen
Quoted 6 lines
I just want the semantics to be consistent. If you want Linux to always require applications that call recvmsg to provide a buffer size of
CMSG_SPACEbytes long to retrieve control information, then fail the system call when the buffer is smaller. But if you do this, you risk breaking applications that work with FreeBSD, Mac OS X, Solaris, and probably a few others.The primary concern is to be binary compatible with Linux.
Not to application developers, by the way.
Quoted 9 lines
Quoted 9 lines
But not being compatible between 32bit and 64bit Linux processes on the same host would seem like a serious problem to me.
Quoted 1 line
Regardless, copying 20 bytes and telling me you copied 24 is misleading and wrong.
The question is could it break existing Linux applications to change it? And would it help with the 32/64bit compatibility?
Quoted 7 lines
Quoted 4 lines
If not some other way to fix the compat layer would need to be found.
-Andi
I'm not sure if a 64-bit process and a 32-bit process exchange file descriptors on the same system has a problem. It certainly looks like the compat code does the right thing. I can test this tonight if you want.
I tested all combinations and they work fine as far as Linux binary compatibility is concerned.
Steve
Quoted 11 lines
Quoted 5 lines
There is no compatability issue.
32-bit tasks will always see the 4-byte align/length. 64-bit tasks will always see the 8-byte align/length.
Really? So when I compile my application on a 32-bit Linux box and then try to run it on a 64-bit Linux box, you're not
going to overrun my buffer whenCMSG_SPACEled me to allocate an insufficient amount of memory needed to account for
padding on the 64-bit platform?
Worth checking what happens if you try to send an fd between a 32 bit application and a 64bit one.
Also is the fd actually closed if the receiving applications buffer isn't big enough.
David