io_uring vs in_compat_syscall()
Subsystems (3)
Hi Jens,
I just found a (so far theoretical) issue with the io_uring submission offloading to workqueues or threads. We have lots of places using in_compat_syscall() to check if a syscall needs compat treatmenet. While the biggest users is iocttl(), we also have a fair amount of places using in_compat_task() in read and write methods, and these will not do the wrong thing when used with io_uring under certain conditions. I'm not sure how to best fix this, except for making sure in_compat_syscall() returns true one way or another for these cases.
Jens Axboe
maintainerQuoted 10 lines · Christoph Hellwig
Hi Jens,
I just found a (so far theoretical) issue with the
io_uringsubmission offloading to workqueues or threads. We have lots of places usingin_compat_syscall()to check if a syscall needs compat treatmenet. While the biggest users isiocttl(), we also have a fair amount of places usingin_compat_task()in read and write methods, and these will not do the wrong thing when used withio_uringunder certain conditions. I'm not sure how to best fix this, except for making surein_compat_syscall()returns true one way or another for these cases.
We can probably propagate this information in the io_kiocb via a flag, and have the io-wq worker set TS_COMPAT if that's the case.
Signature · 1 line
Jens Axboe
Quoted 15 lines · Jens Axboe
Quoted 10 lines · Christoph Hellwig
Hi Jens,
I just found a (so far theoretical) issue with the
io_uringsubmission offloading to workqueues or threads. We have lots of places usingin_compat_syscall()to check if a syscall needs compat treatmenet. While the biggest users isiocttl(), we also have a fair amount of places usingin_compat_task()in read and write methods, and these will not do the wrong thing when used withio_uringunder certain conditions. I'm not sure how to best fix this, except for making surein_compat_syscall()returns true one way or another for these cases.
Quoted 3 lines
We can probably propagate this information in the
io_kiocbvia a flag, and have the io-wq worker setTS_COMPATif that's the case.
Is TS_COMPAT actually a cross-arch concept for which this is safe? Having a real arch helper for “set the current syscall arch for the current kernel thread” seems more sensible to me.
Jens Axboe
maintainerQuoted 15 lines · Andy Lutomirski
Quoted 15 lines · Jens Axboe
Quoted 10 lines · Christoph Hellwig
Hi Jens,
I just found a (so far theoretical) issue with the
io_uringsubmission offloading to workqueues or threads. We have lots of places usingin_compat_syscall()to check if a syscall needs compat treatmenet. While the biggest users isiocttl(), we also have a fair amount of places usingin_compat_task()in read and write methods, and these will not do the wrong thing when used withio_uringunder certain conditions. I'm not sure how to best fix this, except for making surein_compat_syscall()returns true one way or another for these cases.
Quoted 7 lines
Quoted 3 lines
We can probably propagate this information in the
io_kiocbvia a flag, and have the io-wq worker setTS_COMPATif that's the case.Is
TS_COMPATactually a cross-arch concept for which this is safe? Having a real arch helper for “set the current syscall arch for the current kernel thread” seems more sensible to me.
Sure, I'd consider that implementation detail for the actual patch(es) for this issue.
Signature · 1 line
Jens Axboe
Quoted 24 lines · Jens Axboe
Quoted 19 lines · Andy Lutomirski
Quoted 18 lines
Quoted 2 lines · Jens Axboe
Quoted 10 lines · Christoph Hellwig
Hi Jens,
I just found a (so far theoretical) issue with the
io_uringsubmission offloading to workqueues or threads. We have lots of places usingin_compat_syscall()to check if a syscall needs compat treatmenet. While the biggest users isiocttl(), we also have a fair amount of places usingin_compat_task()in read and write methods, and these will not do the wrong thing when used withio_uringunder certain conditions. I'm not sure how to best fix this, except for making surein_compat_syscall()returns true one way or another for these cases.We can probably propagate this information in the
io_kiocbvia a flag, and have the io-wq worker setTS_COMPATif that's the case.
Quoted 6 lines
Quoted 3 lines
Is
TS_COMPATactually a cross-arch concept for which this is safe? Having a real arch helper for “set the current syscall arch for the current kernel thread” seems more sensible to me.Sure, I'd consider that implementation detail for the actual patch(es) for this issue.
There’s a corner case, though: doesn’t io_uring submission frequently do the work synchronously in the context of the calling thread? If so, can a thread do a 64-bit submit with 32-bit work or vice versa?
Sometimes I think that in_compat_syscall() should have a mode in which calling it warns (e.g. not actually in a syscall when doing things in io_uring). And the relevant operations should be properly wired up to avoid global state like this.
Quoted 4 lines
Signature · 2 lines
-- Jens Axboe
Quoted 4 lines · Andy Lutomirski
Quoted 2 lines
Sure, I'd consider that implementation detail for the actual patch(es) for this issue.
There’s a corner case, though: doesn’t
io_uringsubmission frequently do the work synchronously in the context of the calling thread?
Yes.
Quoted 1 line
If so, can a thread do a 64-bit submit with 32-bit work or vice versa?
In theory you could share an fd created in a 32-bit thread to a 64-bit thread or vice versa, but I think at that point you absolutely are in "you get to keep the pieces" land.
Quoted 1 line
Sometimes I think that
in_compat_syscall()should have a mode in which calling it warns (e.g. not actually in a syscall when doing things inio_uring). And the relevant operations should be properly wired up to avoid global state like this.
What do you mean with "properly wired up". Do you really want to spread ->compat_foo methods everywhere, including read and write? I found in_compat_syscall() a lot small and easier to maintain than all the separate compat cruft.
Quoted 7 lines · Christoph Hellwig
Quoted 4 lines · Andy Lutomirski
Quoted 2 lines
Sure, I'd consider that implementation detail for the actual patch(es) for this issue.
There’s a corner case, though: doesn’t
io_uringsubmission frequently do the work synchronously in the context of the calling thread?
Quoted 7 lines
Yes.
Quoted 1 line
If so, can a thread do a 64-bit submit with 32-bit work or vice versa?
In theory you could share an fd created in a 32-bit thread to a 64-bit thread or vice versa, but I think at that point you absolutely are in "you get to keep the pieces" land.
That seems potentially okay as long as these are pieces of userspace and not pieces of the kernel. If the kernel freaks out, we have a problem.
Quoted 7 lines
Quoted 2 lines
Sometimes I think that
in_compat_syscall()should have a mode in which calling it warns (e.g. not actually in a syscall when doing things inio_uring). And the relevant operations should be properly wired up to avoid global state like this.What do you mean with "properly wired up". Do you really want to spread ->compat_foo methods everywhere, including read and write? I found
in_compat_syscall()a lot small and easier to maintain than all the separate compat cruft.
I was imagining using a flag. Some of the net code uses MSG_CMSG_COMPAT for this purpose.
--Andy
Quoted 7 lines · Andy Lutomirski
Quoted 4 lines
What do you mean with "properly wired up". Do you really want to spread ->compat_foo methods everywhere, including read and write? I found
in_compat_syscall()a lot small and easier to maintain than all the separate compat cruft.I was imagining using a flag. Some of the net code uses
MSG_CMSG_COMPATfor this purpose.
Killing that nightmarish monster is what actually got me into looking io_uring and starting this thread.
Quoted 7 lines · Christoph Hellwig
Quoted 5 lines · Andy Lutomirski
Quoted 4 lines
What do you mean with "properly wired up". Do you really want to spread ->compat_foo methods everywhere, including read and write? I found
in_compat_syscall()a lot small and easier to maintain than all the separate compat cruft.
Quoted 5 lines
Quoted 2 lines
I was imagining using a flag. Some of the net code uses
MSG_CMSG_COMPATfor this purpose.Killing that nightmarish monster is what actually got me into looking
io_uringand starting this thread.
I agree that MSG_CMSG_COMPAT is nasty, but I think the concept is sound -- rather than tracking whether we're compat by using a different function or a per-thread variable, actually explicitly tracking the mode seems sensible.
If we're going to play in_compat_syscall() games, let's please make io_uring_enter() return -EINVAL if in_compat_syscall() != ctx->compat.
--Andy
Quoted 11 lines · Andy Lutomirski
Quoted 10 lines · Christoph Hellwig
Quoted 7 lines · Andy Lutomirski
Quoted 4 lines
What do you mean with "properly wired up". Do you really want to spread ->compat_foo methods everywhere, including read and write? I found
in_compat_syscall()a lot small and easier to maintain than all the separate compat cruft.I was imagining using a flag. Some of the net code uses
MSG_CMSG_COMPATfor this purpose.
Quoted 7 lines
Quoted 2 lines
Killing that nightmarish monster is what actually got me into looking
io_uringand starting this thread.I agree that
MSG_CMSG_COMPATis nasty, but I think the concept is sound -- rather than tracking whether we're compat by using a different function or a per-thread variable, actually explicitly tracking the mode seems sensible.
I very strongly disagree. Two recent projects I did was to remove the compat_exec mess, and the compat get/setsockopt mess, and each time it removed hundreds of lines of code duplicating native functionality, often in slightly broken ways. We need a generic out of band way to transfer the information down and just check in in a few strategic places, and in_compat_syscall() does the right thing for that.
Quoted 2 lines
If we're going to play
in_compat_syscall()games, let's please makeio_uring_enter()return-EINVALifin_compat_syscall()!=ctx->compat.
That sounds like a plan, but still doesn't help with submissions from the offload WQ or the sqpoll thread.
From: Christoph Hellwig
Quoted 1 line
Sent: 22 July 2020 07:31
...
Quoted 5 lines
Quoted 4 lines
I agree that
MSG_CMSG_COMPATis nasty, but I think the concept is sound -- rather than tracking whether we're compat by using a different function or a per-thread variable, actually explicitly tracking the mode seems sensible.
Quoted 7 lines
I very strongly disagree. Two recent projects I did was to remove the
compat_execmess, and the compat get/setsockopt mess, and each time it removed hundreds of lines of code duplicating native functionality, often in slightly broken ways. We need a generic out of band way to transfer the information down and just check in in a few strategic places, andin_compat_syscall()does the right thing for that.
Hmmm... set_fs(KERNEL_DS) is a per-thread variable that indicates that the current system call is being done by the kernel.
So you are pulling two different bits of code in opposite directions.
It has to be safer to track the flag through with the request. Then once any conversion has been done the flag can be corrected.
Imagine something like a bpf hook on a compat syscall. Having read the user buffer into kernel space it may decide to reformat it to the native layout to process it. After which the code has a native format buffer even though in_compat_syscall() returns true.
To the native/compat flag is actually a property of the buffer much the same as whether it is user/kernel.
The other property of the buffer is whether embedded addresses are user or kernel.
If the buffer has been read from userspace then they are user. If the request originated in the kernel they are kernel. This difference may matter in the future.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)