recv.2: Document MSG_CMSG_CLOEXEC as returned in msg_flags
Subsystems (4)
PATCH 1/1 recv.2: Document MSG_CMSG_CLOEXEC as returned in msg_flags
--- man2/recv.2 | 9 +++++++++ 1 file changed, 9 insertions(+)
man2/recv.2 +9 -0
@@ -412,6 +412,15 @@ is returned to indicate that expedited or out-of-band data was received. .B MSG_ERRQUEUE indicates that no data was received but an extended error from the socket error queue. +.TP +.BR MSG_CMSG_CLOEXEC " (since Linux 2.6.23)" +.\" commit 4a19542e5f694cd408a32c3d9dc593ba9366e2d7 +indicates that +.B MSG_CMSG_CLOEXEC +was specified in +the +.I flags +argument. .SH RETURN VALUE These calls return the number of bytes received, or \-1 if an error occurred.
Signature · 1 line
2.41.0
Alejandro Colomar
maintainerHi Matthew,
Quoted 12 lines · Matthew House
Signed-off-by Matthew House <[email protected]>--- man2/recv.2 | 9 +++++++++ 1 file changed, 9 insertions(+)man2/recv.2 +0
-0@@ -412,6 +412,15 @@ is returned to indicate that expedited or out-of-band data was received. .B MSG_ERRQUEUE indicates that no data was received but an extended error from the socket
man2/recv.2 is returned to indicate that expedited or out-of-band data was received. · line 423
Quoted 10 lines
man2/recv.2 +9
-0error queue. +.TP +.BR MSG_CMSG_CLOEXEC " (since Linux 2.6.23)" +.\" commit 4a19542e5f694cd408a32c3d9dc593ba9366e2d7 +indicates that +.B MSG_CMSG_CLOEXEC +was specified in +the +.I flags +argument.
You mean in the flags argument of a sendmsg(2) call? Please specify.
Thanks,
Alex
Quoted 3 lines
man2/recv.2 +0
-0.SH RETURN VALUE These calls return the number of bytes received, or \-1 if an error occurred.
Signature · 2 lines
<http://www.alejandro-colomar.es/> GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5
PATCH v2 1/1 recv.2: Document MSG_CMSG_CLOEXEC as returned in msg_flags
---
Clarified that the argument comes from the recvmsg() call. It feels a bit redundant to name recvmsg() again here, given that the list of flags is immediately preceded by, "The msg_flags field in the msghdr is set on return of recvmsg(). It can contain several flags: [...]" But I'll let you be the judge of that.
man2/recv.2 | 9 +++++++++ 1 file changed, 9 insertions(+)
man2/recv.2 +9 -0
@@ -412,6 +412,15 @@ is returned to indicate that expedited or out-of-band data was received. .B MSG_ERRQUEUE indicates that no data was received but an extended error from the socket error queue. +.TP +.BR MSG_CMSG_CLOEXEC " (since Linux 2.6.23)" +.\" commit 4a19542e5f694cd408a32c3d9dc593ba9366e2d7 +indicates that +.B MSG_CMSG_CLOEXEC +was specified in the +.I flags +argument of +.BR recvmsg (). .SH RETURN VALUE These calls return the number of bytes received, or \-1 if an error occurred.
Signature · 1 line
2.41.0
Alejandro Colomar
maintainerHi Matthew,
Quoted 18 lines · Matthew House
Signed-off-by Matthew House <[email protected]>---
Clarified that the argument comes from therecvmsg()call. It feels a bit redundant to namerecvmsg()again here, given that the list of flags is immediately preceded by, "Themsg_flagsfield in the msghdr is set on return ofrecvmsg(). It can contain several flags: [...]" But I'll let you be the judge of that.man2/recv.2 | 9 +++++++++ 1 file changed, 9 insertions(+)man2/recv.2 +0
-0@@ -412,6 +412,15 @@ is returned to indicate that expedited or out-of-band data was received. .B MSG_ERRQUEUE indicates that no data was received but an extended error from the socket
man2/recv.2 is returned to indicate that expedited or out-of-band data was received. · line 423
Quoted 10 lines
man2/recv.2 +9
-0error queue. +.TP +.BR MSG_CMSG_CLOEXEC " (since Linux 2.6.23)" +.\" commit 4a19542e5f694cd408a32c3d9dc593ba9366e2d7 +indicates that +.B MSG_CMSG_CLOEXEC +was specified in the +.I flags +argument of +.BR recvmsg ().
I don't understand what's the purpose of this. The kernel sets a bit just to report to the caller that it set a bit? No other purpose? It feels very weird. Of course, the caller already has that info, doesn't it?
Thanks,
Alex
Quoted 3 lines
man2/recv.2 +0
-0.SH RETURN VALUE These calls return the number of bytes received, or \-1 if an error occurred.
Signature · 2 lines
<http://www.alejandro-colomar.es/> GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5
Quoted 6 lines · Alejandro Colomar
Hi Matthew,
I don't understand what's the purpose of this. The kernel sets a bit just to report to the caller that it set a bit? No other purpose? It feels very weird. Of course, the caller already has that info, doesn't it?
The main reason I posted this patch was because I was confused by the flag's presence in the msg_flags when I was looking at some strace logs, so I figured that it would be a good idea to document it. As for the original purpose of the behavior, it's not really clear, and it may well have been an implementation artifact that got enshrined in the user space ABI. (Even io_uring is careful to replicate this behavior!)
This behavior began when the MSG_CMSG_CLOEXEC flag was first added in Linux 2.6.23, with Ulrich Drepper's commit 4a19542e5f69 ("O_CLOEXEC for SCM_RIGHTS"). Per the commit message, the flag was designed to be "passe[d]... just like the existing MSG_CMSG_COMPAT flag". Since it was added to the msg_flags at the start of sys_recvmsg(), the scm_detach_fds[_compat]() functions in net/core/scm.c and net/compat.c could read the flag off of msg->msg_flags without having to thread the recvmsg() flags through.
This was indeed similar to the behavior of MSG_CMSG_COMPAT. That flag was added in Linux 2.5.65, with commit 3225fc8a85f4 ("[NET]: Simplify scm handling and sendmsg/recvmsg invocation, consolidate net compat syscalls."), in which put_cmsg() and scm_detach_fds() in net/core/scm.c read it off of msg->msg_flags. (It wouldn't actually be set in msg_flags until Linux 2.5.67, with commit 7e8d06bc1d90, "[COMPAT]: Fix MSG_CMSG_COMPAT flag passing, kill cmsg_compat_recvmsg_fixup." Both of these commits are from history/history.git.)
However, the MSG_CMSG_COMPAT flag has been scrubbed from the output msg_flags since Linux 2.6.14, with commit 37f7f421cce1 ("[NET]: Do not leak MSG_CMSG_COMPAT into userspace."). That's what I find so unclear:MSG_CMSG_CLOEXEC was added after the kernel started scrubbing MSG_CMSG_COMPAT from the output, but the new flag was never written to be similarly scrubbed.
Later, in Linux 3.10, with commits 1be374a0518a ("net: Block MSG_CMSG_COMPAT in send(m)msg and recv(m)msg") and a7526eb5d06b ("net:
Unbreak compat_sys_{send,recv}msg"), MSG_CMSG_COMPAT was banned from being passed to the *msg() syscalls' flags from user space, with the rationale that they were "not intended to be part of the API". Then, in Linux 4.0, we reached the current status quo with commit d720d8cec563 ("net: compat:
Ignore MSG_CMSG_COMPAT in compat_sys_{send, recv}msg"), where MSG_CMSG_COMPAT is allowed (and a no-op) in compat syscalls, but banned from non-compat syscalls.
So I agree that it's very weird that this flag gets returned to user space, even while the internal flag that it's modeled after doesn't. I suppose I could spin up a nice story, where the user-space function calling recvmsg() is totally separate from the function processing the returned struct msghdr, and the latter function would really like to know whether the fds in that message are close-on-exec without having to call fcntl(F_GETFD). But that's all just a total guess. If you want to know for sure, perhaps cc'ing Drepper may be worthwhile?
A cursory look hasn't shown me any existing user-space code that depends on this behavior. Though one library appears to be aware of this behavior, actively filtering MSG_CMSG_CLOEXEC out of the result flags:
<https://github.com/dutchanddutch/node-socket-calls/blob/ca759a0da87cb112875d158f4a81b45b31f4a871/src/socket_calls.cc#L417>
Also, only somewhat relatedly, some libraries incorrectly attempt to request MSG_CMSG_CLOEXEC by passing it into the msg_flags field instead of the flags argument:
<https://git.samba.org/samba.git/?p=samba.git;a=blob;f=lib/messaging/messages_dgm.c;hb=refs/tags/samba-4.17.9#l1272>
<https://github.com/genodelabs/genode/blob/23.05/repos/base-linux/src/lib/base/ipc.cc#L132>
<https://github.com/proxmox/pve-lxc-syscalld/blob/a14430f3e75c2b695332ad712164e599464177fc/src/io/seq_packet.rs#L123>
Thank you,
Matthew House
Alejandro Colomar
maintainerHi Matthew,
Quoted 11 lines · Matthew House
Quoted 6 lines · Alejandro Colomar
Hi Matthew,
I don't understand what's the purpose of this. The kernel sets a bit just to report to the caller that it set a bit? No other purpose? It feels very weird. Of course, the caller already has that info, doesn't it?
The main reason I posted this patch was because I was confused by the flag's presence in the
msg_flagswhen I was looking at some strace logs, so I figured that it would be a good idea to document it.
Makes sense.
Quoted 4 lines
As for the original
purpose of the behavior, it's not really clear, and it may well have been an implementation artifact that got enshrined in the user space ABI. (Evenio_uringis careful to replicate this behavior!)
This is what worries me. I've CCd a bunch of people to see if they can bring some light.
Quoted 50 lines
This behavior began when the
MSG_CMSG_CLOEXECflag was first added in Linux 2.6.23, with Ulrich Drepper's commit4a19542e5f69("O_CLOEXECforSCM_RIGHTS"). Per the commit message, the flag was designed to be "passe[d]... just like the existingMSG_CMSG_COMPATflag". Since it was added to themsg_flagsat the start ofsys_recvmsg(), the scm_detach_fds[_compat]() functions innet/core/scm.candnet/compat.ccould read the flag off ofmsg->msg_flagswithout having to thread therecvmsg()flags through.This was indeed similar to the behavior of
MSG_CMSG_COMPAT. That flag was added in Linux 2.5.65, with commit3225fc8a85f4("[NET]: Simplify scm handling and sendmsg/recvmsg invocation, consolidate net compat syscalls."), in whichput_cmsg()andscm_detach_fds()innet/core/scm.cread it off ofmsg->msg_flags. (It wouldn't actually be set inmsg_flagsuntil Linux 2.5.67, with commit7e8d06bc1d90, "[COMPAT]: FixMSG_CMSG_COMPATflag passing, killcmsg_compat_recvmsg_fixup." Both of these commits are from history/history.git.)However, the
MSG_CMSG_COMPATflag has been scrubbed from the outputmsg_flagssince Linux 2.6.14, with commit37f7f421cce1("[NET]: Do not leakMSG_CMSG_COMPATinto userspace."). That's what I find so unclear:MSG_CMSG_CLOEXECwas added after the kernel started scrubbingMSG_CMSG_COMPATfrom the output, but the new flag was never written to be similarly scrubbed.Later, in Linux 3.10, with commits
1be374a0518a("net: BlockMSG_CMSG_COMPATin send(m)msg and recv(m)msg") anda7526eb5d06b("net:
Unbreak compat_sys_{send,recv}msg"),MSG_CMSG_COMPATwas banned from being passed to the*msg()syscalls' flags from user space, with the rationale that they were "not intended to be part of the API". Then, in Linux 4.0, we reached the current status quo with commitd720d8cec563("net: compat:
IgnoreMSG_CMSG_COMPATin compat_sys_{send, recv}msg"), whereMSG_CMSG_COMPATis allowed (and a no-op) in compat syscalls, but banned from non-compat syscalls.So I agree that it's very weird that this flag gets returned to user space, even while the internal flag that it's modeled after doesn't. I suppose I could spin up a nice story, where the user-space function calling
recvmsg()is totally separate from the function processing the returned struct msghdr, and the latter function would really like to know whether the fds in that message are close-on-exec without having to callfcntl(F_GETFD). But that's all just a total guess. If you want to know for sure, perhaps cc'ing Drepper may be worthwhile?A cursory look hasn't shown me any existing user-space code that depends on this behavior. Though one library appears to be aware of this behavior, actively filtering
MSG_CMSG_CLOEXECout of the result flags:
<https://github.com/dutchanddutch/node-socket-calls/blob/ca759a0da87cb112875d158f4a81b45b31f4a871/src/socket_calls.cc#L417>
Quoted 6 lines
Also, only somewhat relatedly, some libraries incorrectly attempt to request
MSG_CMSG_CLOEXECby passing it into themsg_flagsfield instead of the flags argument:
<https://git.samba.org/samba.git/?p=samba.git;a=blob;f=lib/messaging/messages_dgm.c;hb=refs/tags/samba-4.17.9#l1272>
<https://github.com/genodelabs/genode/blob/23.05/repos/base-linux/src/lib/base/ipc.cc#L132>
<https://github.com/proxmox/pve-lxc-syscalld/blob/a14430f3e75c2b695332ad712164e599464177fc/src/io/seq_packet.rs#L123>
In any case, all of this mail has been very interesting, and it would be useful to have it in the commit message of the patch. Please send an v2 with it, and add 'Cc:' tags for all these people:
(I don't know if <[email protected]> still works. Does anyone know?)
Thanks!
Alex
Quoted 3 lines
Thank you,
Matthew House
Signature · 2 lines
<http://www.alejandro-colomar.es/> GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5
PATCH v3 1/1 recv.2: Document MSG_CMSG_CLOEXEC as returned in msg_flags
Ever since commit 4a19542e5f69 ("O_CLOEXEC for SCM_RIGHTS") added the MSG_CMSG_CLOEXEC flag to recvmsg(2), the flag has also been copied into the returned msg->msg_flags when specified, regardless of whether any file descriptors were actually received, or whether the protocol supports receiving file descriptors at all. This behavior was primarily an implementation artifact: by copying MSG_CMSG_CLOEXEC into the msg_flags, scm_detach_fds() in net/core/scm.c (and its _compat() counterpart in net/compat.c) could determine whether it was set without having to receive a copy of the recvmsg(2) flags.
This mechanism was closely modeled after the internal MSG_CMSG_COMPAT flag, which is passed by the compat versions of the send[m]msg(2) and recv[m]msg(2) syscalls to inform various functions that user space expects a compat layout. When the flag was first implemented by commits 3225fc8a85f4 ("[NET]: Simplify scm handling and sendmsg/recvmsg invocation, consolidate net compat syscalls.") and 7e8d06bc1d90 ("[COMPAT]: Fix MSG_CMSG_COMPAT flag passing, kill cmsg_compat_recvmsg_fixup.") (in history/history.git), the behavior was very similar: recvmsg(2) would add MSG_CMSG_COMPAT to the msg_flags, and put_cmsg() and scm_detach_fds() in net/core/scm.c would read the flag to determine whether to delegate to their _compat() counterparts.
However, after the initial implementation, more work was done to hide MSG_CMSG_COMPAT from user space. First, commit 37f7f421cce1 ("[NET]: Do not leak MSG_CMSG_COMPAT into userspace.") started scrubbing the bit from msg_flags right before copying it back into user space. Then, since passing the MSG_CMSG_COMPAT flag into the syscalls from non-compat code could confuse the kernel, commits 1be374a0518a ("net: Block MSG_CMSG_COMPAT in send(m)msg and recv(m)msg") and a7526eb5d06b ("net: Unbreak compat_sys_{send,recv}msg") made them return -EINVAL if user space attempted to pass the flag. But to reduce breakage, commit d720d8cec563 ("net: compat: Ignore MSG_CMSG_COMPAT in compat_sys_{send, recv}msg") rolled that back somewhat, making MSG_CMSG_COMPAT an error for the non-compat syscalls and a no-op for the compat syscalls, which is the current status quo.
Even though MSG_CMSG_CLOEXEC was implemented after the kernel started scrubbing MSG_CMSG_COMPAT from the returned msg_flags, the newer flag never received the same treatment. At this point, this behavior has effectively become part of the user-space API, to the extent that io_uring has been careful in commit 9bb66906f23e ("io_uring: support multishot in recvmsg") to replicate the behavior in its multishot IORING_OP_RECVMSG operation.
Therefore, document this behavior to avoid confusion when user space sees MSG_CMSG_CLOEXEC returned in msg->msg_flags.
---
Alright, I've summarized the history in the commit message, and I've added the CCs you requested.
Also, for future reference, Drepper gave a reply to the last email, which did not make it onto the list:
Quoted 8 lines · Ulrich Drepper
Quoted 6 lines · Alejandro Colomar
Quoted 3 lines
As for the original
purpose of the behavior, it's not really clear, and it may well have been an implementation artifact that got enshrined in the user space ABI.(Even
Quoted 1 line
io_uringis careful to replicate this behavior!)
Quoted 8 lines
Quoted 3 lines
This is what worries me. I've CCd a bunch of people to see if they can bring some light.
It definitely was an artifact of the implementation. I haven't tested getting the close-on-exec flag information for all interfaces. The assumption was that the information about the close-on-exec flag is received with the universal
fcntl()call.
Thank you,
Matthew House
man2/recv.2 | 9 +++++++++ 1 file changed, 9 insertions(+)
man2/recv.2 +9 -0
@@ -412,6 +412,15 @@ is returned to indicate that expedited or out-of-band data was received. .B MSG_ERRQUEUE indicates that no data was received but an extended error from the socket error queue. +.TP +.BR MSG_CMSG_CLOEXEC " (since Linux 2.6.23)" +.\" commit 4a19542e5f694cd408a32c3d9dc593ba9366e2d7 +indicates that +.B MSG_CMSG_CLOEXEC +was specified in the +.I flags +argument of +.BR recvmsg (). .SH RETURN VALUE These calls return the number of bytes received, or \-1 if an error occurred.
Signature · 1 line
2.41.0
Alejandro Colomar
maintainerHi Matthew, Ulrich,
Quoted 52 lines · Matthew House
Ever since commit
4a19542e5f69("O_CLOEXECforSCM_RIGHTS") added theMSG_CMSG_CLOEXECflag to recvmsg(2), the flag has also been copied into the returnedmsg->msg_flagswhen specified, regardless of whether any file descriptors were actually received, or whether the protocol supports receiving file descriptors at all. This behavior was primarily an implementation artifact: by copyingMSG_CMSG_CLOEXECinto themsg_flags,scm_detach_fds()innet/core/scm.c(and its_compat()counterpart innet/compat.c) could determine whether it was set without having to receive a copy of the recvmsg(2) flags.This mechanism was closely modeled after the internal
MSG_CMSG_COMPATflag, which is passed by the compat versions of the send[m]msg(2) and recv[m]msg(2) syscalls to inform various functions that user space expects a compat layout. When the flag was first implemented by commits3225fc8a85f4("[NET]: Simplify scm handling and sendmsg/recvmsg invocation, consolidate net compat syscalls.") and7e8d06bc1d90("[COMPAT]: FixMSG_CMSG_COMPATflag passing, killcmsg_compat_recvmsg_fixup.") (in history/history.git), the behavior was very similar: recvmsg(2) would addMSG_CMSG_COMPATto themsg_flags, andput_cmsg()andscm_detach_fds()innet/core/scm.cwould read the flag to determine whether to delegate to their_compat()counterparts.However, after the initial implementation, more work was done to hide
MSG_CMSG_COMPATfrom user space. First, commit37f7f421cce1("[NET]: Do not leakMSG_CMSG_COMPATinto userspace.") started scrubbing the bit frommsg_flagsright before copying it back into user space. Then, since passing theMSG_CMSG_COMPATflag into the syscalls from non-compat code could confuse the kernel, commits1be374a0518a("net: BlockMSG_CMSG_COMPATin send(m)msg and recv(m)msg") anda7526eb5d06b("net: Unbreak compat_sys_{send,recv}msg") made them return-EINVALif user space attempted to pass the flag. But to reduce breakage, commitd720d8cec563("net: compat: IgnoreMSG_CMSG_COMPATin compat_sys_{send, recv}msg") rolled that back somewhat, makingMSG_CMSG_COMPATan error for the non-compat syscalls and a no-op for the compat syscalls, which is the current status quo.Even though
MSG_CMSG_CLOEXECwas implemented after the kernel started scrubbingMSG_CMSG_COMPATfrom the returnedmsg_flags, the newer flag never received the same treatment. At this point, this behavior has effectively become part of the user-space API, to the extent thatio_uringhas been careful in commit9bb66906f23e("io_uring: support multishot in recvmsg") to replicate the behavior in its multishotIORING_OP_RECVMSGoperation.Therefore, document this behavior to avoid confusion when user space sees
MSG_CMSG_CLOEXECreturned inmsg->msg_flags.Cc Ulrich Drepper <[email protected]>Cc "David S. Miller" <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Matthew House <[email protected]>
Quoted 24 lines
---
Alright, I've summarized the history in the commit message, and I've added the CCs you requested.Also, for future reference, Drepper gave a reply to the last email, which did not make it onto the list:
Quoted 16 lines · Ulrich Drepper
Quoted 9 lines · Alejandro Colomar
Quoted 3 lines
As for the original
purpose of the behavior, it's not really clear, and it may well have been an implementation artifact that got enshrined in the user space ABI.(Even
Quoted 1 line
io_uringis careful to replicate this behavior!)This is what worries me. I've CCd a bunch of people to see if they can bring some light.
It definitely was an artifact of the implementation. I haven't tested getting the close-on-exec flag information for all interfaces. The assumption was that the information about the close-on-exec flag is received with the universal
fcntl()call.
Patch applied. I've included Drepper's quote in the commit message too. Thank you both!
Cheers,
Alex
Quoted 27 lines
Thank you,
Matthew Houseman2/recv.2 | 9 +++++++++ 1 file changed, 9 insertions(+)man2/recv.2 +9
-0@@ -412,6 +412,15 @@ is returned to indicate that expedited or out-of-band data was received. .B MSG_ERRQUEUE indicates that no data was received but an extended error from the socket error queue. +.TP +.BR MSG_CMSG_CLOEXEC " (since Linux 2.6.23)" +.\" commit 4a19542e5f694cd408a32c3d9dc593ba9366e2d7 +indicates that +.B MSG_CMSG_CLOEXEC +was specified in the +.I flags +argument of +.BR recvmsg (). .SH RETURN VALUE These calls return the number of bytes received, or \-1 if an error occurred.
Signature · 2 lines
<http://www.alejandro-colomar.es/> GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5