SOCKET_URING_OP_GETSOCKOPT SOL_SOCKET restriction
Hi Jens,
while playing with the kernel QUIC driver [1],
I noticed it does a lot of getsockopt() and setsockopt() calls to sync the required state into and out of the kernel.
My long term plan is to let the userspace quic handshake logic work with SOCKET_URING_OP_GETSOCKOPT and SOCKET_URING_OP_SETSOCKOPT.
The used level is SOL_QUIC and that won't work
as io_uring_cmd_getsockopt() has a restriction toSOL_SOCKET, while there's no restriction inio_uring_cmd_setsockopt().
What's the reason to have that restriction?
And why is it only for the get path and not
the set path?
metze
Jens Axboe
maintainerQuoted 9 lines · Stefan Metzmacher
Hi Jens,
while playing with the kernel QUIC driver [1],
I noticed it does a lot ofgetsockopt()andsetsockopt()calls to sync the required state into and out of the kernel.My long term plan is to let the userspace quic handshake logic work with
SOCKET_URING_OP_GETSOCKOPTandSOCKET_URING_OP_SETSOCKOPT.
Quoted 8 lines
The used level is
SOL_QUICand that won't work
asio_uring_cmd_getsockopt()has a restriction toSOL_SOCKET, while there's no restriction inio_uring_cmd_setsockopt().What's the reason to have that restriction?
And why is it only for the get path and not
the set path?
There's absolutely no reason for that, looks like a pure oversight?!
Signature · 1 line
Jens Axboe
Quoted 15 lines · Jens Axboe
Quoted 14 lines · Stefan Metzmacher
Hi Jens,
while playing with the kernel QUIC driver [1],
I noticed it does a lot ofgetsockopt()andsetsockopt()calls to sync the required state into and out of the kernel.My long term plan is to let the userspace quic handshake logic work with
SOCKET_URING_OP_GETSOCKOPTandSOCKET_URING_OP_SETSOCKOPT.The used level is
SOL_QUICand that won't work
asio_uring_cmd_getsockopt()has a restriction toSOL_SOCKET, while there's no restriction inio_uring_cmd_setsockopt().
Quoted 5 lines
Quoted 3 lines
What's the reason to have that restriction?
And why is it only for the get path and not
the set path?There's absolutely no reason for that, looks like a pure oversight?!
Cc Breno, he can explain better, but IIRC that's because most of set/get sockopt options expect user pointers to be passed in, and io_uring wants to use kernel memory. It's plumbed for SOL_SOCKET with sockptr_t, but there was a push back against converting the rest.
The implications are not the uapi side. For example, io_uring get/setsockopt returns err / len in a cqe->res. We can't do that for SOL_SOCKET without the kernel pointer support, otherwise io-uring uapi would need to get hacky. E.g. you'd need to pass another user pointer is an SQE for socklen and read it after completion.
Signature · 1 line
Pavel Begunkov
Quoted 22 lines · Pavel Begunkov
Quoted 15 lines · Jens Axboe
Quoted 14 lines · Stefan Metzmacher
Hi Jens,
while playing with the kernel QUIC driver [1],
I noticed it does a lot ofgetsockopt()andsetsockopt()calls to sync the required state into and out of the kernel.My long term plan is to let the userspace quic handshake logic work with
SOCKET_URING_OP_GETSOCKOPTandSOCKET_URING_OP_SETSOCKOPT.The used level is
SOL_QUICand that won't work
asio_uring_cmd_getsockopt()has a restriction toSOL_SOCKET, while there's no restriction inio_uring_cmd_setsockopt().Quoted 5 lines
Quoted 3 lines
What's the reason to have that restriction?
And why is it only for the get path and not
the set path?There's absolutely no reason for that, looks like a pure oversight?!
Quoted 7 lines
Cc Breno, he can explain better, but IIRC that's because most of set/get sockopt options expect user pointers to be passed in, and
io_uringwants to use kernel memory. It's plumbed forSOL_SOCKETwithsockptr_t, but there was a push back against converting the rest.The implications are not the uapi side. For example,
io_uring
s/not/on/
Quoted 6 lines
get/setsockopt returns err / len in a
cqe->res. We can't do that forSOL_SOCKETwithout the kernel pointer support, otherwise io-uring uapi would need to get hacky. E.g. you'd need to pass another user pointer is an SQE for socklen and read it after completion.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 20 lines · Pavel Begunkov
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Stefan Metzmacher
Hi Jens,
while playing with the kernel QUIC driver [1],
I noticed it does a lot ofgetsockopt()andsetsockopt()calls to sync the required state into and out of the kernel.My long term plan is to let the userspace quic handshake logic work with
SOCKET_URING_OP_GETSOCKOPTandSOCKET_URING_OP_SETSOCKOPT.The used level is
SOL_QUICand that won't work
asio_uring_cmd_getsockopt()has a restriction toSOL_SOCKET, while there's no restriction inio_uring_cmd_setsockopt().What's the reason to have that restriction?
And why is it only for the get path and not
the set path?
Quoted 7 lines
Quoted 1 line
There's absolutely no reason for that, looks like a pure oversight?!
Cc Breno, he can explain better, but IIRC that's because most of set/get sockopt options expect user pointers to be passed in, and
io_uringwants to use kernel memory. It's plumbed forSOL_SOCKETwithsockptr_t, but there was a push back against converting the rest.
Gah yes, now I remember. What's pretty annoying though, as it leaves the get/setsockopt parts less useful than they should be, compared to the regular syscalls.
Did we ever ponder ways of getting this sorted out on the net side?
Signature · 1 line
Jens Axboe
Quoted 29 lines · Jens Axboe
Quoted 20 lines · Pavel Begunkov
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Stefan Metzmacher
Hi Jens,
while playing with the kernel QUIC driver [1],
I noticed it does a lot ofgetsockopt()andsetsockopt()calls to sync the required state into and out of the kernel.My long term plan is to let the userspace quic handshake logic work with
SOCKET_URING_OP_GETSOCKOPTandSOCKET_URING_OP_SETSOCKOPT.The used level is
SOL_QUICand that won't work
asio_uring_cmd_getsockopt()has a restriction toSOL_SOCKET, while there's no restriction inio_uring_cmd_setsockopt().What's the reason to have that restriction?
And why is it only for the get path and not
the set path?Quoted 7 lines
Quoted 1 line
There's absolutely no reason for that, looks like a pure oversight?!
Cc Breno, he can explain better, but IIRC that's because most of set/get sockopt options expect user pointers to be passed in, and
io_uringwants to use kernel memory. It's plumbed forSOL_SOCKETwithsockptr_t, but there was a push back against converting the rest.
Quoted 5 lines
Gah yes, now I remember. What's pretty annoying though, as it leaves the get/setsockopt parts less useful than they should be, compared to the regular syscalls.
Did we ever ponder ways of getting this sorted out on the net side?
I remember Breno looking at several different options.
Breno, can you remind me, why can't we convert ->getsockopt to take a normal kernel ptr for length while passing a user ptr for value as before?
Signature · 1 line
Pavel Begunkov
Quoted 34 lines · Pavel Begunkov
Quoted 33 lines · Jens Axboe
Quoted 20 lines · Pavel Begunkov
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Stefan Metzmacher
Hi Jens,
while playing with the kernel QUIC driver [1],
I noticed it does a lot ofgetsockopt()andsetsockopt()calls to sync the required state into and out of the kernel.My long term plan is to let the userspace quic handshake logic work with
SOCKET_URING_OP_GETSOCKOPTandSOCKET_URING_OP_SETSOCKOPT.The used level is
SOL_QUICand that won't work
asio_uring_cmd_getsockopt()has a restriction toSOL_SOCKET, while there's no restriction inio_uring_cmd_setsockopt().What's the reason to have that restriction?
And why is it only for the get path and not
the set path?Quoted 7 lines
Quoted 1 line
There's absolutely no reason for that, looks like a pure oversight?!
Cc Breno, he can explain better, but IIRC that's because most of set/get sockopt options expect user pointers to be passed in, and
io_uringwants to use kernel memory. It's plumbed forSOL_SOCKETwithsockptr_t, but there was a push back against converting the rest.Gah yes, now I remember. What's pretty annoying though, as it leaves the get/setsockopt parts less useful than they should be, compared to the regular syscalls.
Quoted 7 lines
Quoted 1 line
Did we ever ponder ways of getting this sorted out on the net side?
I remember Breno looking at several different options.
Breno, can you remind me, why can't we convert ->getsockopt to take a normal kernel ptr for length while passing a user ptr for value as before?
Similar to this:
getsockopt_syscall(void __user *len_uptr) {
int klen;
copy_from_user(&klen, len_uptr);
->getsockopt(&klen);
copy_to_user(len_uptr, &klen);
}Signature · 1 line
Pavel Begunkov
Hello Pavel,
Quoted 13 lines · Pavel Begunkov
Quoted 9 lines · Pavel Begunkov
Quoted 3 lines · Jens Axboe
Quoted 2 lines · Pavel Begunkov
Quoted 1 line · Jens Axboe
On 3/28/25 8:27 AM, Stefan Metzmacher wrote:
I remember Breno looking at several different options.
Breno, can you remind me, why can't we convert ->getsockopt to take a normal kernel ptr for length while passing a user ptr for value as before?
Similar to this:
Quoted 7 lines
getsockopt_syscall(void __user *len_uptr) {
int klen;
copy_from_user(&klen, len_uptr);
->getsockopt(&klen);
copy_to_user(len_uptr, &klen);
}We have a few limitations if I remember correct:
getsockopt() callback expects __user pointers:
int (*getsockopt)(struct socket *sock, int level, int optname, char __user *optval, int __user *optlen);
So, you cannot copy the memory content and call ->getsockopt() with kernel memory.
A solution was to use sockptr, as done by setsockopt(), but, that was discouraged.
Another important thing, some getsockopt() callback changes the pointer, so, doing copy_to_user() directly in the getsocktopt callback, which would break your approach above.
I understand that the next steps here are:
1) Make getsockopt() operate with either userspace or kernel buffer.
a) This buffer needs will be written and read on both side. I.e, you pass data in the buffer from userspace to kernel space, and kernel will overwrite that buffer in kernelspace.
In other words, this is a read-write buffer (which is not something we have in iovec IIRC).
2) Call the same callbacks from io_uring subsystem using kernel memory
3) Regular syscalls will continue to user userspace memory.
Quoted 26 lines · Breno Leitao
Hello Pavel,
Quoted 13 lines · Pavel Begunkov
Quoted 9 lines · Pavel Begunkov
Quoted 3 lines · Jens Axboe
Quoted 2 lines · Pavel Begunkov
Quoted 1 line · Jens Axboe
On 3/28/25 8:27 AM, Stefan Metzmacher wrote:
I remember Breno looking at several different options.
Breno, can you remind me, why can't we convert ->getsockopt to take a normal kernel ptr for length while passing a user ptr for value as before?
Similar to this:
Quoted 7 lines
getsockopt_syscall(void __user *len_uptr) { int klen; copy_from_user(&klen, len_uptr); ->getsockopt(&klen); copy_to_user(len_uptr, &klen); }We have a few limitations if I remember correct:
Quoted 8 lines
getsockopt()callback expects__userpointers:int (*getsockopt)(struct socket *sock, int level, int optname, char __user *optval, int __user *optlen);So, you cannot copy the memory content and call ->getsockopt() with kernel memory.
Right, I'm rather asking about changing the callback to pass a kernel pointer and make the caller to do the copy_to_user if needed.
Quoted 6 lines
A solution was to use sockptr, as done by
setsockopt(), but, that was discouraged.Another important thing, some
getsockopt()callback changes the pointer, so, doingcopy_to_user()directly in the getsocktopt callback, which would break your approach above.
Do you mean writing to it? That's why the snippet passes a kernel _pointer_ to length. Did I misunderstand you?
Quoted 14 lines
I understand that the next steps here are:
1) Make
getsockopt()operate with either userspace or kernel buffer.
a) This buffer needs will be written and read on both side. I.e, you pass data in the buffer from userspace to kernel space, and kernel will overwrite that buffer in kernelspace.In other words, this is a read-write buffer (which is not something we have in iovec IIRC).
2) Call the same callbacks from
io_uringsubsystem using kernel memory3) Regular syscalls will continue to user userspace memory.
Signature · 1 line
Pavel Begunkov
Am 28.03.25 um 18:21 schrieb Pavel Begunkov:
Quoted 45 lines · Pavel Begunkov
Quoted 34 lines · Jens Axboe
Quoted 33 lines · Pavel Begunkov
Quoted 20 lines · Jens Axboe
Quoted 19 lines · Stefan Metzmacher
Quoted 17 lines
Hi Jens,
while playing with the kernel QUIC driver [1],
I noticed it does a lot ofgetsockopt()andsetsockopt()calls to sync the required state into and out of the kernel.My long term plan is to let the userspace quic handshake logic work with
SOCKET_URING_OP_GETSOCKOPTandSOCKET_URING_OP_SETSOCKOPT.The used level is
SOL_QUICand that won't work
asio_uring_cmd_getsockopt()has a restriction toSOL_SOCKET, while there's no restriction inio_uring_cmd_setsockopt().What's the reason to have that restriction?
And why is it only for the get path and not
the set path?Quoted 7 lines
Quoted 1 line
There's absolutely no reason for that, looks like a pure oversight?!
Cc Breno, he can explain better, but IIRC that's because most of set/get sockopt options expect user pointers to be passed in, and
io_uringwants to use kernel memory. It's plumbed forSOL_SOCKETwithsockptr_t, but there was a push back against converting the rest.Gah yes, now I remember. What's pretty annoying though, as it leaves the get/setsockopt parts less useful than they should be, compared to the regular syscalls.
Quoted 7 lines
Quoted 1 line
Did we ever ponder ways of getting this sorted out on the net side?
I remember Breno looking at several different options.
Breno, can you remind me, why can't we convert ->getsockopt to take a normal kernel ptr for length while passing a user ptr for value as before?
Similar to this:
Quoted 7 lines
getsockopt_syscall(void __user *len_uptr){
int klen;copy_from_user(&klen,
len_uptr);
->getsockopt(&klen);
copy_to_user(len_uptr, &klen);
}
Yes, I was thinking about something similar.
I'll give it a go next week.
Thanks!
metze
Am 28.03.25 um 15:30 schrieb Jens Axboe:
Quoted 15 lines · Stefan Metzmacher
Quoted 14 lines
Hi Jens,
while playing with the kernel QUIC driver [1],
I noticed it does a lot ofgetsockopt()andsetsockopt()calls to sync the required state into and out of the kernel.My long term plan is to let the userspace quic handshake logic work with
SOCKET_URING_OP_GETSOCKOPTandSOCKET_URING_OP_SETSOCKOPT.The used level is
SOL_QUICand that won't work
asio_uring_cmd_getsockopt()has a restriction toSOL_SOCKET, while there's no restriction inio_uring_cmd_setsockopt().
Quoted 5 lines
Quoted 3 lines
What's the reason to have that restriction?
And why is it only for the get path and not
the set path?There's absolutely no reason for that, looks like a pure oversight?!
It seems RFC had the limitation on both:
https://lore.kernel.org/io-uring/[email protected]/
v0 had it only for get because of some userpointer restrictions:
https://lore.kernel.org/io-uring/[email protected]/
The merged v7 also talks about the restriction:
https://lore.kernel.org/all/[email protected]/
Adding Breno ...
It seems proto_ops->getsockopt is the problem as it's not changed to sockptr_t yet.
metze
Am 28.03.25 um 16:02 schrieb Stefan Metzmacher:
Quoted 28 lines
Am 28.03.25 um 15:30 schrieb Jens Axboe:
Quoted 15 lines · Stefan Metzmacher
Quoted 14 lines
Hi Jens,
while playing with the kernel QUIC driver [1],
I noticed it does a lot ofgetsockopt()andsetsockopt()calls to sync the required state into and out of the kernel.My long term plan is to let the userspace quic handshake logic work with
SOCKET_URING_OP_GETSOCKOPTandSOCKET_URING_OP_SETSOCKOPT.The used level is
SOL_QUICand that won't work
asio_uring_cmd_getsockopt()has a restriction toSOL_SOCKET, while there's no restriction inio_uring_cmd_setsockopt().Quoted 5 lines
Quoted 3 lines
What's the reason to have that restriction?
And why is it only for the get path and not
the set path?There's absolutely no reason for that, looks like a pure oversight?!
It seems RFC had the limitation on both:
https://lore.kernel.org/io-uring/[email protected]/v0 had it only for get because of some userpointer restrictions:
https://lore.kernel.org/io-uring/[email protected]/
Quoted 7 lines
The merged v7 also talks about the restriction:
https://lore.kernel.org/all/[email protected]/Adding Breno ...
It seems
proto_ops->getsockoptis the problem as it's not changed tosockptr_tyet.
commit a7b75c5a8c41445f33efb663887ff5f5c3b4454b
Author: Christoph Hellwig <[email protected]> Date: Thu Jul 23 08:09:07 2020 +0200
net: pass a sockptr_t into ->setsockopt
Rework the remaining setsockopt code to pass a sockptr_t instead of a plain user pointer. This removes the last remaining set_fs(KERNEL_DS) outside of architecture specific code.
only converted setsockopt function pointer...
Is there any reason not to change getsockopt too?
Except for someone needs to do the work?
metze
Hello Stefan,
Quoted 28 lines · Stefan Metzmacher
Am 28.03.25 um 15:30 schrieb Jens Axboe:
Quoted 15 lines · Stefan Metzmacher
Quoted 14 lines
Hi Jens,
while playing with the kernel QUIC driver [1],
I noticed it does a lot ofgetsockopt()andsetsockopt()calls to sync the required state into and out of the kernel.My long term plan is to let the userspace quic handshake logic work with
SOCKET_URING_OP_GETSOCKOPTandSOCKET_URING_OP_SETSOCKOPT.The used level is
SOL_QUICand that won't work
asio_uring_cmd_getsockopt()has a restriction toSOL_SOCKET, while there's no restriction inio_uring_cmd_setsockopt().Quoted 5 lines
Quoted 3 lines
What's the reason to have that restriction?
And why is it only for the get path and not
the set path?There's absolutely no reason for that, looks like a pure oversight?!
It seems RFC had the limitation on both:
https://lore.kernel.org/io-uring/[email protected]/v0 had it only for get because of some userpointer restrictions:
https://lore.kernel.org/io-uring/[email protected]/
Quoted 7 lines
The merged v7 also talks about the restriction:
https://lore.kernel.org/all/[email protected]/Adding Breno ...
It seems
proto_ops->getsockoptis the problem as it's not changed tosockptr_tyet.
Correct. That is because Linus detests sockptr and didn't recommend adding new code to use it.
Quoted 1 line
New code does *not* have that excuse.
https://lore.kernel.org/all/CAHk-=wgGV61xrG=gO0=dXH64o2TDWWrXn1mx-CX885JZ7h84Og@mail.gmail.com/
This was raised by Jakub in:
https://lore.kernel.org/all/[email protected]/
So, in order to implement the missing part, we need to move this to something else. The initial suggestion was to use iovec, but, I found it very hard to move that code to iovec.