Skip to contentSkip to conversation

SOCKET_URING_OP_GETSOCKOPT SOL_SOCKET restriction

DISCUSSION Stefan Metzmacher 4 participants 13 messages Mar 28, 2025 – Mar 29, 2025 io_uring

lore
Share & tools
lore
#0 lore raw

Hi Jens,

while playing with the kernel QUIC driver [1],
I noticed it does a lot of getsockopt() and setsockopt() calls to sync the required state into and out of the kernel.

My long term plan is to let the userspace quic handshake logic work with SOCKET_URING_OP_GETSOCKOPT and SOCKET_URING_OP_SETSOCKOPT.

The used level is SOL_QUIC and that won't work
as io_uring_cmd_getsockopt() has a restriction to
SOL_SOCKET, while there's no restriction in
io_uring_cmd_setsockopt().

What's the reason to have that restriction?
And why is it only for the get path and not
the set path?

metze

[1] https://github.com/lxin/quic

maintainer
#1 lore raw
Quoted 9 lines · Stefan Metzmacher

Hi Jens,

while playing with the kernel QUIC driver [1],
I noticed it does a lot of getsockopt() and setsockopt() calls to sync the required state into and out of the kernel.

My long term plan is to let the userspace quic handshake logic work with SOCKET_URING_OP_GETSOCKOPT and SOCKET_URING_OP_SETSOCKOPT.

Quoted 8 lines

The used level is SOL_QUIC and that won't work
as io_uring_cmd_getsockopt() has a restriction to
SOL_SOCKET, while there's no restriction in
io_uring_cmd_setsockopt().

What's the reason to have that restriction?
And why is it only for the get path and not
the set path?

There's absolutely no reason for that, looks like a pure oversight?!

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 15 lines · Jens Axboe
Quoted 14 lines · Stefan Metzmacher

Hi Jens,

while playing with the kernel QUIC driver [1],
I noticed it does a lot of getsockopt() and setsockopt() calls to sync the required state into and out of the kernel.

My long term plan is to let the userspace quic handshake logic work with SOCKET_URING_OP_GETSOCKOPT and SOCKET_URING_OP_SETSOCKOPT.

The used level is SOL_QUIC and that won't work
as io_uring_cmd_getsockopt() has a restriction to
SOL_SOCKET, while there's no restriction in
io_uring_cmd_setsockopt().

Quoted 5 lines
Quoted 3 lines

What's the reason to have that restriction?
And why is it only for the get path and not
the set path?

There's absolutely no reason for that, looks like a pure oversight?!

Cc Breno, he can explain better, but IIRC that's because most of set/get sockopt options expect user pointers to be passed in, and io_uring wants to use kernel memory. It's plumbed for SOL_SOCKET with sockptr_t, but there was a push back against converting the rest.

The implications are not the uapi side. For example, io_uring get/setsockopt returns err / len in a cqe->res. We can't do that for SOL_SOCKET without the kernel pointer support, otherwise io-uring uapi would need to get hacky. E.g. you'd need to pass another user pointer is an SQE for socklen and read it after completion.

Signature · 1 line
Pavel Begunkov
#3 lore raw
Quoted 22 lines · Pavel Begunkov
Quoted 15 lines · Jens Axboe
Quoted 14 lines · Stefan Metzmacher

Hi Jens,

while playing with the kernel QUIC driver [1],
I noticed it does a lot of getsockopt() and setsockopt() calls to sync the required state into and out of the kernel.

My long term plan is to let the userspace quic handshake logic work with SOCKET_URING_OP_GETSOCKOPT and SOCKET_URING_OP_SETSOCKOPT.

The used level is SOL_QUIC and that won't work
as io_uring_cmd_getsockopt() has a restriction to
SOL_SOCKET, while there's no restriction in
io_uring_cmd_setsockopt().

Quoted 5 lines
Quoted 3 lines

What's the reason to have that restriction?
And why is it only for the get path and not
the set path?

There's absolutely no reason for that, looks like a pure oversight?!

Quoted 7 lines

Cc Breno, he can explain better, but IIRC that's because most of set/get sockopt options expect user pointers to be passed in, and io_uring wants to use kernel memory. It's plumbed for SOL_SOCKET with sockptr_t, but there was a push back against converting the rest.

The implications are not the uapi side. For example, io_uring

s/not/on/

Quoted 6 lines

get/setsockopt returns err / len in a cqe->res. We can't do that for SOL_SOCKET without the kernel pointer support, otherwise io-uring uapi would need to get hacky. E.g. you'd need to pass another user pointer is an SQE for socklen and read it after completion.

Signature · 1 line
Pavel Begunkov
maintainer
#4 lore raw
In reply to Pavel Begunkov
Quoted 20 lines · Pavel Begunkov
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Stefan Metzmacher

Hi Jens,

while playing with the kernel QUIC driver [1],
I noticed it does a lot of getsockopt() and setsockopt() calls to sync the required state into and out of the kernel.

My long term plan is to let the userspace quic handshake logic work with SOCKET_URING_OP_GETSOCKOPT and SOCKET_URING_OP_SETSOCKOPT.

The used level is SOL_QUIC and that won't work
as io_uring_cmd_getsockopt() has a restriction to
SOL_SOCKET, while there's no restriction in
io_uring_cmd_setsockopt().

What's the reason to have that restriction?
And why is it only for the get path and not
the set path?

Quoted 7 lines
Quoted 1 line

There's absolutely no reason for that, looks like a pure oversight?!

Cc Breno, he can explain better, but IIRC that's because most of set/get sockopt options expect user pointers to be passed in, and io_uring wants to use kernel memory. It's plumbed for SOL_SOCKET with sockptr_t, but there was a push back against converting the rest.

Gah yes, now I remember. What's pretty annoying though, as it leaves the get/setsockopt parts less useful than they should be, compared to the regular syscalls.

Did we ever ponder ways of getting this sorted out on the net side?

Signature · 1 line
Jens Axboe
#5 lore raw
Quoted 29 lines · Jens Axboe
Quoted 20 lines · Pavel Begunkov
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Stefan Metzmacher

Hi Jens,

while playing with the kernel QUIC driver [1],
I noticed it does a lot of getsockopt() and setsockopt() calls to sync the required state into and out of the kernel.

My long term plan is to let the userspace quic handshake logic work with SOCKET_URING_OP_GETSOCKOPT and SOCKET_URING_OP_SETSOCKOPT.

The used level is SOL_QUIC and that won't work
as io_uring_cmd_getsockopt() has a restriction to
SOL_SOCKET, while there's no restriction in
io_uring_cmd_setsockopt().

What's the reason to have that restriction?
And why is it only for the get path and not
the set path?

Quoted 7 lines
Quoted 1 line

There's absolutely no reason for that, looks like a pure oversight?!

Cc Breno, he can explain better, but IIRC that's because most of set/get sockopt options expect user pointers to be passed in, and io_uring wants to use kernel memory. It's plumbed for SOL_SOCKET with sockptr_t, but there was a push back against converting the rest.

Quoted 5 lines

Gah yes, now I remember. What's pretty annoying though, as it leaves the get/setsockopt parts less useful than they should be, compared to the regular syscalls.

Did we ever ponder ways of getting this sorted out on the net side?

I remember Breno looking at several different options.

Breno, can you remind me, why can't we convert ->getsockopt to take a normal kernel ptr for length while passing a user ptr for value as before?

Signature · 1 line
Pavel Begunkov
#6 lore raw
Quoted 34 lines · Pavel Begunkov
Quoted 33 lines · Jens Axboe
Quoted 20 lines · Pavel Begunkov
Quoted 19 lines · Jens Axboe
Quoted 17 lines · Stefan Metzmacher

Hi Jens,

while playing with the kernel QUIC driver [1],
I noticed it does a lot of getsockopt() and setsockopt() calls to sync the required state into and out of the kernel.

My long term plan is to let the userspace quic handshake logic work with SOCKET_URING_OP_GETSOCKOPT and SOCKET_URING_OP_SETSOCKOPT.

The used level is SOL_QUIC and that won't work
as io_uring_cmd_getsockopt() has a restriction to
SOL_SOCKET, while there's no restriction in
io_uring_cmd_setsockopt().

What's the reason to have that restriction?
And why is it only for the get path and not
the set path?

Quoted 7 lines
Quoted 1 line

There's absolutely no reason for that, looks like a pure oversight?!

Cc Breno, he can explain better, but IIRC that's because most of set/get sockopt options expect user pointers to be passed in, and io_uring wants to use kernel memory. It's plumbed for SOL_SOCKET with sockptr_t, but there was a push back against converting the rest.

Gah yes, now I remember. What's pretty annoying though, as it leaves the get/setsockopt parts less useful than they should be, compared to the regular syscalls.

Quoted 7 lines
Quoted 1 line

Did we ever ponder ways of getting this sorted out on the net side?

I remember Breno looking at several different options.

Breno, can you remind me, why can't we convert ->getsockopt to take a normal kernel ptr for length while passing a user ptr for value as before?

Similar to this:

getsockopt_syscall(void __user *len_uptr) {
	int klen;

	copy_from_user(&klen, len_uptr);
	->getsockopt(&klen);
	copy_to_user(len_uptr, &klen);
}
Signature · 1 line
Pavel Begunkov
#7 lore raw

Hello Pavel,

Quoted 13 lines · Pavel Begunkov
Quoted 9 lines · Pavel Begunkov
Quoted 3 lines · Jens Axboe
Quoted 2 lines · Pavel Begunkov
Quoted 1 line · Jens Axboe

On 3/28/25 8:27 AM, Stefan Metzmacher wrote:

I remember Breno looking at several different options.

Breno, can you remind me, why can't we convert ->getsockopt to take a normal kernel ptr for length while passing a user ptr for value as before?

Similar to this:

Quoted 7 lines
getsockopt_syscall(void __user *len_uptr) {
	int klen;

	copy_from_user(&klen, len_uptr);
	->getsockopt(&klen);
	copy_to_user(len_uptr, &klen);
}

We have a few limitations if I remember correct:

getsockopt() callback expects __user pointers:

	int             (*getsockopt)(struct socket *sock, int level,
			int optname, char __user *optval, int __user *optlen);

So, you cannot copy the memory content and call ->getsockopt() with kernel memory.

A solution was to use sockptr, as done by setsockopt(), but, that was discouraged.

Another important thing, some getsockopt() callback changes the pointer, so, doing copy_to_user() directly in the getsocktopt callback, which would break your approach above.

I understand that the next steps here are:

1) Make getsockopt() operate with either userspace or kernel buffer.
a) This buffer needs will be written and read on both side. I.e, you pass data in the buffer from userspace to kernel space, and kernel will overwrite that buffer in kernelspace.

In other words, this is a read-write buffer (which is not something we have in iovec IIRC).

2) Call the same callbacks from io_uring subsystem using kernel memory

3) Regular syscalls will continue to user userspace memory.

#8 lore raw
Quoted 26 lines · Breno Leitao

Hello Pavel,

Quoted 13 lines · Pavel Begunkov
Quoted 9 lines · Pavel Begunkov
Quoted 3 lines · Jens Axboe
Quoted 2 lines · Pavel Begunkov
Quoted 1 line · Jens Axboe

On 3/28/25 8:27 AM, Stefan Metzmacher wrote:

I remember Breno looking at several different options.

Breno, can you remind me, why can't we convert ->getsockopt to take a normal kernel ptr for length while passing a user ptr for value as before?

Similar to this:

Quoted 7 lines
getsockopt_syscall(void __user *len_uptr) {
	int klen;

	copy_from_user(&klen, len_uptr);
	->getsockopt(&klen);
	copy_to_user(len_uptr, &klen);
}

We have a few limitations if I remember correct:

Quoted 8 lines

getsockopt() callback expects __user pointers:

	int             (*getsockopt)(struct socket *sock, int level,
			int optname, char __user *optval, int __user *optlen);

So, you cannot copy the memory content and call ->getsockopt() with kernel memory.

Right, I'm rather asking about changing the callback to pass a kernel pointer and make the caller to do the copy_to_user if needed.

Quoted 6 lines

A solution was to use sockptr, as done by setsockopt(), but, that was discouraged.

Another important thing, some getsockopt() callback changes the pointer, so, doing copy_to_user() directly in the getsocktopt callback, which would break your approach above.

Do you mean writing to it? That's why the snippet passes a kernel _pointer_ to length. Did I misunderstand you?

Quoted 14 lines

I understand that the next steps here are:

1) Make getsockopt() operate with either userspace or kernel buffer.
a) This buffer needs will be written and read on both side. I.e, you pass data in the buffer from userspace to kernel space, and kernel will overwrite that buffer in kernelspace.

In other words, this is a read-write buffer (which is not something we have in iovec IIRC).

2) Call the same callbacks from io_uring subsystem using kernel memory

3) Regular syscalls will continue to user userspace memory.

Signature · 1 line
Pavel Begunkov
#9 lore raw
In reply to Pavel Begunkov

Am 28.03.25 um 18:21 schrieb Pavel Begunkov:

Quoted 45 lines · Pavel Begunkov
Quoted 34 lines · Jens Axboe
Quoted 33 lines · Pavel Begunkov
Quoted 20 lines · Jens Axboe
Quoted 19 lines · Stefan Metzmacher
Quoted 17 lines

Hi Jens,

while playing with the kernel QUIC driver [1],
I noticed it does a lot of getsockopt() and setsockopt() calls to sync the required state into and out of the kernel.

My long term plan is to let the userspace quic handshake logic work with SOCKET_URING_OP_GETSOCKOPT and SOCKET_URING_OP_SETSOCKOPT.

The used level is SOL_QUIC and that won't work
as io_uring_cmd_getsockopt() has a restriction to
SOL_SOCKET, while there's no restriction in
io_uring_cmd_setsockopt().

What's the reason to have that restriction?
And why is it only for the get path and not
the set path?

Quoted 7 lines
Quoted 1 line

There's absolutely no reason for that, looks like a pure oversight?!

Cc Breno, he can explain better, but IIRC that's because most of set/get sockopt options expect user pointers to be passed in, and io_uring wants to use kernel memory. It's plumbed for SOL_SOCKET with sockptr_t, but there was a push back against converting the rest.

Gah yes, now I remember. What's pretty annoying though, as it leaves the get/setsockopt parts less useful than they should be, compared to the regular syscalls.

Quoted 7 lines
Quoted 1 line

Did we ever ponder ways of getting this sorted out on the net side?

I remember Breno looking at several different options.

Breno, can you remind me, why can't we convert ->getsockopt to take a normal kernel ptr for length while passing a user ptr for value as before?

Similar to this:

Quoted 7 lines

getsockopt_syscall(void __user *len_uptr) {
    int klen;

    copy_from_user(&klen, len_uptr);
    ->getsockopt(&klen);
    copy_to_user(len_uptr, &klen);
}

Yes, I was thinking about something similar.

I'll give it a go next week.

Thanks!
metze

#10 lore raw
In reply to Jens Axboe

Am 28.03.25 um 15:30 schrieb Jens Axboe:

Quoted 15 lines · Stefan Metzmacher
Quoted 14 lines

Hi Jens,

while playing with the kernel QUIC driver [1],
I noticed it does a lot of getsockopt() and setsockopt() calls to sync the required state into and out of the kernel.

My long term plan is to let the userspace quic handshake logic work with SOCKET_URING_OP_GETSOCKOPT and SOCKET_URING_OP_SETSOCKOPT.

The used level is SOL_QUIC and that won't work
as io_uring_cmd_getsockopt() has a restriction to
SOL_SOCKET, while there's no restriction in
io_uring_cmd_setsockopt().

Quoted 5 lines
Quoted 3 lines

What's the reason to have that restriction?
And why is it only for the get path and not
the set path?

There's absolutely no reason for that, looks like a pure oversight?!

It seems RFC had the limitation on both:
https://lore.kernel.org/io-uring/[email protected]/

v0 had it only for get because of some userpointer restrictions:
https://lore.kernel.org/io-uring/[email protected]/

The merged v7 also talks about the restriction:
https://lore.kernel.org/all/[email protected]/

Adding Breno ...

It seems proto_ops->getsockopt is the problem as it's not changed to sockptr_t yet.

metze

author Acked-by Stefan Schmidt Acked-by Matthieu Baerts
#11 lore raw

Am 28.03.25 um 16:02 schrieb Stefan Metzmacher:

Quoted 28 lines

Am 28.03.25 um 15:30 schrieb Jens Axboe:

Quoted 15 lines · Stefan Metzmacher
Quoted 14 lines

Hi Jens,

while playing with the kernel QUIC driver [1],
I noticed it does a lot of getsockopt() and setsockopt() calls to sync the required state into and out of the kernel.

My long term plan is to let the userspace quic handshake logic work with SOCKET_URING_OP_GETSOCKOPT and SOCKET_URING_OP_SETSOCKOPT.

The used level is SOL_QUIC and that won't work
as io_uring_cmd_getsockopt() has a restriction to
SOL_SOCKET, while there's no restriction in
io_uring_cmd_setsockopt().

Quoted 5 lines
Quoted 3 lines

What's the reason to have that restriction?
And why is it only for the get path and not
the set path?

There's absolutely no reason for that, looks like a pure oversight?!

It seems RFC had the limitation on both:
https://lore.kernel.org/io-uring/[email protected]/

v0 had it only for get because of some userpointer restrictions:
https://lore.kernel.org/io-uring/[email protected]/

Quoted 7 lines

The merged v7 also talks about the restriction:
https://lore.kernel.org/all/[email protected]/

Adding Breno ...

It seems proto_ops->getsockopt is the problem as it's not changed to sockptr_t yet.

commit a7b75c5a8c41445f33efb663887ff5f5c3b4454b

Author: Christoph Hellwig <[email protected]>
Date:   Thu Jul 23 08:09:07 2020 +0200

net: pass a sockptr_t into ->setsockopt

Rework the remaining setsockopt code to pass a sockptr_t instead of a plain user pointer. This removes the last remaining set_fs(KERNEL_DS) outside of architecture specific code.

only converted setsockopt function pointer...

Is there any reason not to change getsockopt too?
Except for someone needs to do the work?

metze

#12 lore raw
In reply to Stefan Metzmacher

Hello Stefan,

Quoted 28 lines · Stefan Metzmacher

Am 28.03.25 um 15:30 schrieb Jens Axboe:

Quoted 15 lines · Stefan Metzmacher
Quoted 14 lines

Hi Jens,

while playing with the kernel QUIC driver [1],
I noticed it does a lot of getsockopt() and setsockopt() calls to sync the required state into and out of the kernel.

My long term plan is to let the userspace quic handshake logic work with SOCKET_URING_OP_GETSOCKOPT and SOCKET_URING_OP_SETSOCKOPT.

The used level is SOL_QUIC and that won't work
as io_uring_cmd_getsockopt() has a restriction to
SOL_SOCKET, while there's no restriction in
io_uring_cmd_setsockopt().

Quoted 5 lines
Quoted 3 lines

What's the reason to have that restriction?
And why is it only for the get path and not
the set path?

There's absolutely no reason for that, looks like a pure oversight?!

It seems RFC had the limitation on both:
https://lore.kernel.org/io-uring/[email protected]/

v0 had it only for get because of some userpointer restrictions:
https://lore.kernel.org/io-uring/[email protected]/

Quoted 7 lines

The merged v7 also talks about the restriction:
https://lore.kernel.org/all/[email protected]/

Adding Breno ...

It seems proto_ops->getsockopt is the problem as it's not changed to sockptr_t yet.

Correct. That is because Linus detests sockptr and didn't recommend adding new code to use it.

Quoted 1 line

New code does *not* have that excuse.

https://lore.kernel.org/all/CAHk-=wgGV61xrG=gO0=dXH64o2TDWWrXn1mx-CX885JZ7h84Og@mail.gmail.com/

This was raised by Jakub in:

https://lore.kernel.org/all/[email protected]/

So, in order to implement the missing part, we need to move this to something else. The initial suggestion was to use iovec, but, I found it very hard to move that code to iovec.