Skip to contentSkip to conversation

UML for arm64

DISCUSSION Rob Herring 6 participants 11 messages Jun 22, 2023 – Jun 24, 2023 User-mode Linux (UML)

lore
Share & tools
lore
maintainer
#1 lore raw
Quoted 4 lines · Rob Herring

Hi all,

I'm interested in getting UML working on Arm. Is anyone aware of any efforts to do this already?

Not me ... But yeah, agree that it seems likely to require some refactoring, even if _some_ degree of attempts were made to keep arch- specific things there ...

There might be a ton of arch-specific things in the ptrace code too, so perhaps it'd be easier to start with the seccomp-based model:

https://patchwork.ozlabs.org/project/linux-um/list/?series=329466

But that's not really done yet either.

johannes

_______________________________________________
linux-um mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-um

#2 lore raw

Hi,

Quoted 9 lines · Johannes Berg
Quoted 2 lines · Rob Herring

I'm interested in getting UML working on Arm. Is anyone aware of any efforts to do this already?

Not me ... But yeah, agree that it seems likely to require some refactoring, even if _some_ degree of attempts were made to keep arch- specific things there ...

There might be a ton of arch-specific things in the ptrace code too

At least a lot of the register handling, but that will always be somewhat architecture specific.

Quoted 5 lines

... , so perhaps it'd be easier to start with the seccomp-based model:

https://patchwork.ozlabs.org/project/linux-um/list/?series=329466

But that's not really done yet either.

Yes, though I have quite some updates locally that solve the security issues (i.e. prevent host syscalls from userspace). So, it does need rebasing and I am sure bugfixes, but I think it is getting into a pretty good shape overall.

Either way, the old patchset will give you a good idea about how it all works, the changes are mostly in the details. I am happy to push out a new version sooner rather than later if it might help with any efforts on your side.

Benjamin

_______________________________________________
linux-um mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-um

#3 lore raw

[CC'ing Al]

----- Ursprüngliche Mail -----

Quoted 7 lines

Von: "Benjamin Berg" <[email protected]>
An: "Johannes Berg" <[email protected]>, "robh" <[email protected]>, "richard" <[email protected]>, "anton ivanov" <[email protected]>
CC: "linux-um" <[email protected]>
Gesendet: Donnerstag, 22. Juni 2023 22:13:56
Betreff: Re: UML for arm64

Quoted 12 lines

Hi,

Quoted 9 lines · Johannes Berg
Quoted 2 lines · Rob Herring

I'm interested in getting UML working on Arm. Is anyone aware of any efforts to do this already?

Not me ... But yeah, agree that it seems likely to require some refactoring, even if _some_ degree of attempts were made to keep arch- specific things there ...

There might be a ton of arch-specific things in the ptrace code too

I think Al did some work in the past to support archs other than x86. It was not arm64, though. ppc or sparc IIRC. Maybe he has some input.

Quoted 14 lines

At least a lot of the register handling, but that will always be somewhat architecture specific.

Quoted 5 lines

... , so perhaps it'd be easier to start with the seccomp-based model:

https://patchwork.ozlabs.org/project/linux-um/list/?series=329466

But that's not really done yet either.

Yes, though I have quite some updates locally that solve the security issues (i.e. prevent host syscalls from userspace). So, it does need rebasing and I am sure bugfixes, but I think it is getting into a pretty good shape overall.

Quoted 4 lines

Either way, the old patchset will give you a good idea about how it all works, the changes are mostly in the details. I am happy to push out a new version sooner rather than later if it might help with any efforts on your side.

I think reviving the seccomp patch set is a good idea in any case. Maybe it is even a way to allow SMP support.

Thanks,
//richard

_______________________________________________
linux-um mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-um

#4 lore raw

HI,

Quoted 16 lines · Richard Weinberger

[SNIP]

Quoted 14 lines

At least a lot of the register handling, but that will always be somewhat architecture specific.

Quoted 5 lines

... , so perhaps it'd be easier to start with the seccomp-based model:

https://patchwork.ozlabs.org/project/linux-um/list/?series=329466

But that's not really done yet either.

Yes, though I have quite some updates locally that solve the security issues (i.e. prevent host syscalls from userspace). So, it does need rebasing and I am sure bugfixes, but I think it is getting into a pretty good shape overall.

Quoted 7 lines
Quoted 4 lines

Either way, the old patchset will give you a good idea about how it all works, the changes are mostly in the details. I am happy to push out a new version sooner rather than later if it might help with any efforts on your side.

I think reviving the seccomp patch set is a good idea in any case. Maybe it is even a way to allow SMP support.

OK, I'll see about resubmitting the SECCOMP patchset.

About SMP support, it is tricky if one wants to do it safely. Doing it in an unsafe way is probably reasonably straight forward as you can simply map multiple stub-data areas into the same process. However, the other threads can read/write memory at the same time, which breaks the entire memory security model that I had in mind for SECCOMP mode. My approach there heavily relies on killing the process *after* it mapped memory without permission, but before this can be exploited.

So, I fear no simple and secure SMP support. Having to do host syscalls from the userspace process means that you just cannot have another thread running. And then, you are stuck with multiple processes and the headache and overhead of keeping their memory in sync.

I think secure SMP we would need some other new and exciting approach for handling process memory. An example could be a special (memory) file system on the host that explicitly allows setting up page ranges to be transparently shared between files.

Benjamin

_______________________________________________
linux-um mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-um

maintainer
#5 lore raw
In reply to Benjamin Berg
Quoted 7 lines · Benjamin Berg

Hi,

Quoted 4 lines · Johannes Berg
Quoted 2 lines · Rob Herring

I'm interested in getting UML working on Arm. Is anyone aware of any efforts to do this already?

Quoted 8 lines
Quoted 5 lines

Not me ... But yeah, agree that it seems likely to require some refactoring, even if _some_ degree of attempts were made to keep arch- specific things there ...

There might be a ton of arch-specific things in the ptrace code too

At least a lot of the register handling, but that will always be somewhat architecture specific.

In the arm case there may be some paging issues.

Arm pages are not necessarily 4096. I tried to implement non-4096 page support for UML a couple of years ago and gave up.

It will be nice if we manage to fix this. End of the day - UML uses mmap(), so it should be able to use page sizing which differs from the host one.

Quoted 24 lines
Quoted 6 lines

... , so perhaps it'd be easier to start with the seccomp-based model:

https://patchwork.ozlabs.org/project/linux-um/list/?series=329466

But that's not really done yet either.

Yes, though I have quite some updates locally that solve the security issues (i.e. prevent host syscalls from userspace). So, it does need rebasing and I am sure bugfixes, but I think it is getting into a pretty good shape overall.

Either way, the old patchset will give you a good idea about how it all works, the changes are mostly in the details. I am happy to push out a new version sooner rather than later if it might help with any efforts on your side.

Benjamin

_______________________________________________
linux-um mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-um

Signature · 8 lines
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

_______________________________________________
linux-um mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-um
#10 lore raw
In reply to Rob Herring

Hi,

Quoted 6 lines

Hi all,

I'm interested in getting UML working on Arm. Is anyone aware of any efforts to do this already? I've not found anything searching around. Or perhaps a port to any other arch as I'd imagine any 2nd arch will require some refactorin

I am also quite interested in such a port. We may help with some testing or even more, if it fits my schedule.

From my longer digging in the code, I can only confirm, that several parts are primarily thought from x86-ISA standpoint and already the switch to amd64 is flimsy at some places.

Quoted 3 lines

Thanks,
Rob

Regards

Benjamin

_______________________________________________
linux-um mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-um

#6 lore raw
In reply to Benjamin Berg
Quoted 28 lines · Benjamin Berg

Hi,

Quoted 9 lines · Johannes Berg
Quoted 2 lines · Rob Herring

I'm interested in getting UML working on Arm. Is anyone aware of any efforts to do this already?

Not me ... But yeah, agree that it seems likely to require some refactoring, even if _some_ degree of attempts were made to keep arch- specific things there ...

There might be a ton of arch-specific things in the ptrace code too

At least a lot of the register handling, but that will always be somewhat architecture specific.

Quoted 5 lines

... , so perhaps it'd be easier to start with the seccomp-based model:

https://patchwork.ozlabs.org/project/linux-um/list/?series=329466

But that's not really done yet either.

Yes, though I have quite some updates locally that solve the security issues (i.e. prevent host syscalls from userspace). So, it does need rebasing and I am sure bugfixes, but I think it is getting into a pretty good shape overall.

Quoted 4 lines

Either way, the old patchset will give you a good idea about how it all works, the changes are mostly in the details. I am happy to push out a new version sooner rather than later if it might help with any efforts on your side.

From a quick scan, it looks like there's some cleanups in the series which would be helpful without seccomp parts. One of the initial issues I've found is UML using older ptrace interfaces which arm64 doesn't implement. PTRACE_GETREGS for example.

Rob

_______________________________________________
linux-um mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-um

maintainer
#7 lore raw
Quoted 7 lines · Rob Herring
Quoted 6 lines

Either way, the old patchset will give you a good idea about how it all works, the changes are mostly in the details. I am happy to push out a new version sooner rather than later if it might help with any efforts on your side.

Quoted 5 lines

From a quick scan, it looks like there's some cleanups in the series which would be helpful without seccomp parts. One of the initial issues I've found is UML using older ptrace interfaces which arm64 doesn't implement. PTRACE_GETREGS for example.

I don't think that completely gets rid of PTRACE_GETREGS though, and if I remember correctly, we really kind of need that there?

Though then again it's all been a while, and I only faulted the seccomp mode back in in discussions with Benjamin. Looks like we've found a potentially nicer way to make it secure than his secret-based approach, and in fact in a way that should even make it SMP-safe, at least in theory, obviously a lot of infrastructure is missing to make it SMP in the first place.

Currently, UML has a host process per VMA. Obviously, you need multiple host processes for SMP (to get SMP), i.e. one per (used) CPU per VMA, with CLONE_VM.

The problem with the secrets-based approach here for SMP is that the secret will be readable to the other running in the VMA (1) and then can be used for circumventing the protection by jumping into the stub area and calling host syscalls, see
https://patchwork.ozlabs.org/project/linux-um/patch/[email protected]/
and
https://patchwork.ozlabs.org/project/linux-um/patch/[email protected]/

Now the new idea we came up with is this: We can make the per-CPU VMA with CLONE_VM but *not* CLONE_FILES. Then, in the stub, when we need to execute some real host syscalls on behalf of the child, we

* send the FD over in a message
* use the FD for mmap (and also always use mmap instead of mprotect)
* close the FD

Without CLONE_FILES, another thread cannot "steal" the (real, host) FD, it's useless in the other thread. Note that I'm talking about host FDs here, in-UML FDs are just numbers and it works all differently, I'm just talking about executing host syscalls inside the VMA, to set up the VMA correctly etc.

The BPF program now allows any the relevant syscalls (2) inside the stub area, but since you don't have the FD unless you actually executed the recvmsg() call at the beginning of the stub you can't do anything with that by jumping into the stub.

There are some other details involved such as having to split the stub data into a read-only "what to execute" (3) and a writeable "results" page, but those are reasonably easy to deal with.

At the end of the stub, of course the FD must be closed. This is fine though since mappings persist after the FD was closed. On the next page fault we have to do it all again, but yeah, page faults were always super expensive in UML ...

johannes

(1) unless you extract it directly out of the BPF program into registers via some magic syscalls that get a BPF return, but that's kind of icky too

(2) mmap & munmap are the most relevant ones, some others but they're not that critical for security; notably mprotect is not allowed but must be done with mmap instead

(3) so another thread can't actually overwrite the instructions of what the kernel wants to run inside the VMA process from another thread while it's happening

_______________________________________________
linux-um mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-um

#8 lore raw
Quoted 8 lines · Johannes Berg
Quoted 7 lines · Rob Herring
Quoted 6 lines

Either way, the old patchset will give you a good idea about how it all works, the changes are mostly in the details. I am happy to push out a new version sooner rather than later if it might help with any efforts on your side.

Quoted 8 lines
Quoted 5 lines

From a quick scan, it looks like there's some cleanups in the series which would be helpful without seccomp parts. One of the initial issues I've found is UML using older ptrace interfaces which arm64 doesn't implement. PTRACE_GETREGS for example.

I don't think that completely gets rid of PTRACE_GETREGS though, and if I remember correctly, we really kind of need that there?

The SECCOMP code should not need any ptrace at all. All it does is read/write the mcontext that is generated by the host. I think there was just some mangling there to map the basic registers into the format that UML expects internally (floating point, SSE, etc. are just copied directly though).

Quoted 6 lines

Though then again it's all been a while, and I only faulted the seccomp mode back in in discussions with Benjamin. Looks like we've found a potentially nicer way to make it secure than his secret-based approach, and in fact in a way that should even make it SMP-safe, at least in theory, obviously a lot of infrastructure is missing to make it SMP in the first place.

Yeah, the idea with FD passing and CLONE_VM without CLONE_FILES does indeed seem very promising both for a secure SECCOMP model and SMP support specifically. It is should be much easier to implement than my previous secret based syscall authentication idea.

Benjamin

Quoted 49 lines

Currently, UML has a host process per VMA. Obviously, you need multiple host processes for SMP (to get SMP), i.e. one per (used) CPU per VMA, with CLONE_VM.

The problem with the secrets-based approach here for SMP is that the secret will be readable to the other running in the VMA (1) and then can be used for circumventing the protection by jumping into the stub area and calling host syscalls, see
https://patchwork.ozlabs.org/project/linux-um/patch/[email protected]/
and
https://patchwork.ozlabs.org/project/linux-um/patch/[email protected]/

Now the new idea we came up with is this: We can make the per-CPU VMA with CLONE_VM but *not* CLONE_FILES. Then, in the stub, when we need to execute some real host syscalls on behalf of the child, we

 * send the FD over in a message
 * use the FD for mmap (and also always use mmap instead of mprotect)
 * close the FD

Without CLONE_FILES, another thread cannot "steal" the (real, host) FD, it's useless in the other thread. Note that I'm talking about host FDs here, in-UML FDs are just numbers and it works all differently, I'm just talking about executing host syscalls inside the VMA, to set up the VMA correctly etc.

The BPF program now allows any the relevant syscalls (2) inside the stub area, but since you don't have the FD unless you actually executed the recvmsg() call at the beginning of the stub you can't do anything with that by jumping into the stub.

There are some other details involved such as having to split the stub data into a read-only "what to execute" (3) and a writeable "results" page, but those are reasonably easy to deal with.

At the end of the stub, of course the FD must be closed. This is fine though since mappings persist after the FD was closed. On the next page fault we have to do it all again, but yeah, page faults were always super expensive in UML ...

johannes

(1) unless you extract it directly out of the BPF program into registers via some magic syscalls that get a BPF return, but that's kind of icky too

Quoted 8 lines

(2) mmap & munmap are the most relevant ones, some others but they're not that critical for security; notably mprotect is not allowed but must be done with mmap instead

(3) so another thread can't actually overwrite the instructions of what the kernel wants to run inside the VMA process from another thread while it's happening

_______________________________________________
linux-um mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-um

maintainer
#9 lore raw
Quoted 15 lines · Benjamin Berg
Quoted 14 lines · Johannes Berg
Quoted 12 lines · Rob Herring
Quoted 6 lines

Either way, the old patchset will give you a good idea about how it all works, the changes are mostly in the details. I am happy to push out a new version sooner rather than later if it might help with any efforts on your side.

From a quick scan, it looks like there's some cleanups in the series which would be helpful without seccomp parts. One of the initial issues I've found is UML using older ptrace interfaces which arm64 doesn't implement. PTRACE_GETREGS for example.

Quoted 5 lines
Quoted 2 lines

I don't think that completely gets rid of PTRACE_GETREGS though, and if I remember correctly, we really kind of need that there?

The SECCOMP code should not need any ptrace at all.

Indeed. For the record, I was referring to "some cleanups in the series" not getting rid of PTRACE_GETREGS entirely :-)

There are indeed some instances of PTRACE_GETREGS removed in your patches, however, I believe that they're still needed (in ptrace mode) after that. If that's true, then it seems like the ptrace mode can't really be ported to arm64.

Then again, a bunch of the stuff you removed there is for detecting sysemu support, which is documented to only exist on x86, which I'm not sure is true because arm64 defines PTRACE_SYSEMU[_SINGLESTEP] and TIF_SYSCALL_EMU and then the generic ptrace code should actually implement it?

However, PTRACE_GETREGS is still invoked unconditionally in the userspace() loop afaict, and thus it can't actually work?

So seems that the best bet for other architectures anyway would be seccomp mode anyway, since that's pretty well supported across architectures.

You have some endianness bugs in the BPF code for 64-bit code though, it only works on little endian ;-)

But there are good reasons to use it anyway, vs. ptrace.

Quoted 5 lines

All it does is
read/write the mcontext that is generated by the host. I think there was just some mangling there to map the basic registers into the format that UML expects internally (floating point, SSE, etc. are just copied directly though).

Yeah that sounds about right.

Quoted 10 lines
Quoted 6 lines

Though then again it's all been a while, and I only faulted the seccomp mode back in in discussions with Benjamin. Looks like we've found a potentially nicer way to make it secure than his secret-based approach, and in fact in a way that should even make it SMP-safe, at least in theory, obviously a lot of infrastructure is missing to make it SMP in the first place.

Yeah, the idea with FD passing and CLONE_VM without CLONE_FILES does indeed seem very promising both for a secure SECCOMP model and SMP support specifically.

I don't believe we'll get SMP support easily one way or the other, but it's nice to know that seccomp mode won't actively break it :-)

johannes

_______________________________________________
linux-um mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-um