x86 vdso updates

x86-64: Add time to vDSO

Quoted 1 line · Thomas Gleixner

x86-64: Add time to vDSO

And here's the glibc bug:

http://sourceware.org/bugzilla/show_bug.cgi?id=12813

Let the deprecation of the vsyscall page begin :)

Quoted 8 lines · Andy Lutomirski

Quoted 1 line · Thomas Gleixner

      x86-64: Add time to vDSO

And here's the glibc bug:

http://sourceware.org/bugzilla/show_bug.cgi?id=12813

Let the deprecation of the vsyscall page begin :)

Yeah :-)

On a related note, now that these bits are upstream, what are your rough plans for doing the int81 vsyscall emulation patch?

The int81 patch would actually be (much!) more important to the average Linux user than getting rid of the vsyscall from static
binaries: the vsyscall is the last predictable executable address in PIE daemons with a dangerous SYSENTER instruction in it ...

I'd actually consider accelerating it into v2.6.40.

Thanks,

       Ingo

Quoted 17 lines · Ingo Molnar

Quoted 8 lines · Andy Lutomirski

Quoted 1 line · Thomas Gleixner

      x86-64: Add time to vDSO

And here's the glibc bug:

http://sourceware.org/bugzilla/show_bug.cgi?id=12813

Let the deprecation of the vsyscall page begin :)

Yeah :-)

On a related note, now that these bits are upstream, what are your rough plans for doing the int81 vsyscall emulation patch?

Quoted 6 lines

The int81 patch would actually be (much!) more important to the average Linux user than getting rid of the vsyscall from static
binaries: the vsyscall is the last predictable executable address in PIE daemons with a dangerous SYSENTER instruction in it ...

I'd actually consider accelerating it into v2.6.40.

Working on it slowly.  In principle I'm writing my thesis right now on things that have very little to do with software :)

My rough plans for the near-term stuff are:

1. Move vvars out of the vsyscall page.  Otherwise we have at least one predictable syscall instruction every 1<<16 seconds because the time is executable.  It's prettier that way, too.

2. Remove the vsyscall64 sysctl.  That will reduce the number of vsyscalls that require the kernel's help to one.  (This is a bit unfortunate for UML users, but I'm not sure what to do about that. UML vgetcpu is already terminally broken.)

Quoted 4 lines · Andrew Lutomirski

2. Remove the vsyscall64 sysctl.  That will reduce the number of vsyscalls that require the kernel's help to one.  (This is a bit unfortunate for UML users, but I'm not sure what to do about that. UML vgetcpu is already terminally broken.)

Can you please be a bit more precise on that?
When there is anything I can do, let me know...

--
Thanks,
//richard

Quoted 8 lines · richard -rw- weinberger

Quoted 4 lines · Andrew Lutomirski

2. Remove the vsyscall64 sysctl.  That will reduce the number of vsyscalls that require the kernel's help to one.  (This is a bit unfortunate for UML users, but I'm not sure what to do about that. UML vgetcpu is already terminally broken.)

Can you please be a bit more precise on that?
When there is anything I can do, let me know...

glibc's implementation of time, sched_getcpu, and (for static binaries) gettimeofday calls a fixed address in kernel space which returns the answer.  On UML, that address is presumably in *host* kernel space and the host will return what the host thinks the answer is.

That's a bit unfortunate if the guest has a different idea of cpu numbering or what time it is.  Right now, setting kernel.vsyscall64=0 will turn gettimeofday and time (but not sched_getcpu) into a real syscall.  The problem is that I want to remove the vsyscall64 sysctl.

If this is considered enough of a regression, then I guess we can leave vsyscall64 around for awhile, but it will require extra work in the soon-to-be syscall emulation hack to make sure that UML can still trap the syscall.

The real solution is to fix glibc to use the vDSO which should avoid this problem entirely.

Quoted 5 lines · Andrew Lutomirski

On Fri, May 27, 2011 at 7:59 AM, richard -rw- weinberger If this is considered enough of a regression, then I guess we can leave vsyscall64 around for awhile, but it will require extra work in the soon-to-be syscall emulation hack to make sure that UML can still trap the syscall.

As long the time within UML is synchronous with the host everything is fine, right?

So, as _last_ choice we could disable the ability to change the time within UML.

IMHO it's not a big deal when getcpu() returns a wrong CPU layout on UML.

Quoted 2 lines

The real solution is to fix glibc to use the vDSO which should avoid this problem entirely.

Is this "bug" known? And the biggest question, will Ulrich fix it some day?

Quoted 7 lines · richard -rw- weinberger

Quoted 5 lines · Andrew Lutomirski

On Fri, May 27, 2011 at 7:59 AM, richard -rw- weinberger If this is considered enough of a regression, then I guess we can leave vsyscall64 around for awhile, but it will require extra work in the soon-to-be syscall emulation hack to make sure that UML can still trap the syscall.

Quoted 2 lines

As long the time within UML is synchronous with the host everything is fine, right?

I think so.  I haven't used UML in a long time.

Quoted 7 lines

So, as _last_ choice we could disable the ability to change the time within UML.

IMHO it's not a big deal when getcpu() returns a wrong CPU layout on UML.

Quoted 2 lines

The real solution is to fix glibc to use the vDSO which should avoid this problem entirely.

Quoted 9 lines · Andrew Lutomirski

Quoted 7 lines · richard -rw- weinberger

Quoted 5 lines · Andrew Lutomirski

On Fri, May 27, 2011 at 7:59 AM, richard -rw- weinberger If this is considered enough of a regression, then I guess we can leave vsyscall64 around for awhile, but it will require extra work in the soon-to-be syscall emulation hack to make sure that UML can still trap the syscall.

Quoted 12 lines

Quoted 2 lines

As long the time within UML is synchronous with the host everything is fine, right?

I think so.  I haven't used UML in a long time.

Quoted 7 lines

So, as _last_ choice we could disable the ability to change the time within UML.

IMHO it's not a big deal when getcpu() returns a wrong CPU layout on UML.

Quoted 2 lines

The real solution is to fix glibc to use the vDSO which should avoid this problem entirely.

Yesterday I had a closer look at 64bit UML.
Glibc is always using vsyscalls because 64bit UML does not support the vDSO.

On 32bit UML simply scans the ELF auxiliary vector provided by the host to get the address of the vDSO.
How can I get this address on a 64bit host?

Quoted 23 lines · richard -rw- weinberger

Quoted 21 lines · Andrew Lutomirski

Quoted 9 lines · richard -rw- weinberger

Quoted 5 lines · Andrew Lutomirski

On Fri, May 27, 2011 at 7:59 AM, richard -rw- weinberger If this is considered enough of a regression, then I guess we can leave vsyscall64 around for awhile, but it will require extra work in the soon-to-be syscall emulation hack to make sure that UML can still trap the syscall.

As long the time within UML is synchronous with the host everything is fine, right?

I think so.  I haven't used UML in a long time.

Quoted 7 lines

So, as _last_ choice we could disable the ability to change the time within UML.

IMHO it's not a big deal when getcpu() returns a wrong CPU layout on UML.

Quoted 2 lines

The real solution is to fix glibc to use the vDSO which should avoid this problem entirely.

Quoted 6 lines

Yesterday I had a closer look at 64bit UML.
Glibc is always using vsyscalls because 64bit UML does not support the vDSO.

On 32bit UML simply scans the ELF auxiliary vector provided by the host to get the address of the vDSO.
How can I get this address on a 64bit host?

I believe it's exactly the same.  There's an auxv entry that points to the vDSO.

Quoted 10 lines · Andrew Lutomirski

Quoted 6 lines · richard -rw- weinberger

Yesterday I had a closer look at 64bit UML.
Glibc is always using vsyscalls because 64bit UML does not support the vDSO.

On 32bit UML simply scans the ELF auxiliary vector provided by the host to get the address of the vDSO.
How can I get this address on a 64bit host?

I believe it's exactly the same.  There's an auxv entry that points to the vDSO.

I don't think so.
See:
http://www.win.tue.nl/~aeb/linux/lk/lk-4.html
Section "Address space randomization".
The demo program finds the vDSO only on x86.

UML uses quite the same method to find it.
arch/um/os-Linux/elf_aux.c

Quoted 18 lines · richard -rw- weinberger

Quoted 10 lines · Andrew Lutomirski

Quoted 6 lines · richard -rw- weinberger

Yesterday I had a closer look at 64bit UML.
Glibc is always using vsyscalls because 64bit UML does not support the vDSO.

On 32bit UML simply scans the ELF auxiliary vector provided by the host to get the address of the vDSO.
How can I get this address on a 64bit host?

I believe it's exactly the same.  There's an auxv entry that points to the vDSO.

I don't think so.
See:
http://www.win.tue.nl/~aeb/linux/lk/lk-4.html
Section "Address space randomization".
The demo program finds the vDSO only on x86.

Quoted 2 lines

UML uses quite the same method to find it.
arch/um/os-Linux/elf_aux.c

The attached program works for me.

I don't know what this is supposed to mean, though:

                               /* See if the page is under TASK_SIZE */ if (vsyscall_ehdr < (unsigned long) envp) vsyscall_ehdr = 0;

First, envp != TASK_SIZE.

Second, the vDSO can be wherever it wants.  On current kernels at least it is *always* mapped below TASK_SIZE (in the unsigned sense) because it's mapped into user address space.

3. Add int 0xcc and use it from vgettimeofday.  It will SIGSEGV if called from a user address (so it has no risk of ever becoming ABI) and it will do gettimeofday if called from the right address.  (I like 0xcc better than 0x81 because then I don't have to wonder whether any syscall-like instructions start with 0x81.)  I'm not convinced that the existing syscall entries are usable, because syscall itself has a different calling convention and int 0x80 is a compat syscall.

Quoted 8 lines · Andrew Lutomirski

3. Add int 0xcc and use it from vgettimeofday.  It will SIGSEGV if called from a user address (so it has no risk of ever becoming ABI) and it will do gettimeofday if called from the right address.  (I like 0xcc better than 0x81 because then I don't have to wonder whether any syscall-like instructions start with 0x81.)  I'm not convinced that the existing syscall entries are usable, because syscall itself has a different calling convention and int 0x80 is a compat syscall.

I started looking at what needs to be done and I wanted to get your opinion before I wrote a bunch of code that you'd reject. Here are three ideas for how the int 0xcc / int 0x81 entry could work:

*** Idea 1 ***

Make it a real syscall but with extra constraints. It would have the same calling convention as the syscall instruction, but it would turn into SIGKILL if the calling address isn't in the VSYSCALL page or if the syscall number isn't __NR_clock_gettimeofday. It would BUG() if called from kernel mode. There are two ways to implement this:

1. Have the interrupt entry check constraints, twiddle its stack frame to look like a syscall instruction, and jump to the syscall entry. This way there's little code duplication. (Is it safe to sysret back to userspace from an interrupt gate? I don't see why not, but it seems to violate the spirit of the thing.)

Quoted 22 lines · Andrew Lutomirski

Quoted 8 lines · Andrew Lutomirski

3. Add int 0xcc and use it from vgettimeofday.  It will SIGSEGV if called from a user address (so it has no risk of ever becoming ABI) and it will do gettimeofday if called from the right address.  (I like 0xcc better than 0x81 because then I don't have to wonder whether any syscall-like instructions start with 0x81.)  I'm not convinced that the existing syscall entries are usable, because syscall itself has a different calling convention and int 0x80 is a compat syscall.

I started looking at what needs to be done and I wanted to get your opinion before I wrote a bunch of code that you'd reject.  Here are three ideas for how the int 0xcc / int 0x81 entry could work:

*** Idea 1 ***

Make it a real syscall but with extra constraints.  It would have the same calling convention as the syscall instruction, but it would turn into SIGKILL if the calling address isn't in the VSYSCALL page or if the syscall number isn't __NR_clock_gettimeofday.  It would BUG() if called from kernel mode.  There are two ways to implement this:

Quoted 5 lines

1. Have the interrupt entry check constraints, twiddle its stack frame to look like a syscall instruction, and jump to the syscall entry. This way there's little code duplication.  (Is it safe to sysret back to userspace from an interrupt gate?  I don't see why not, but it seems to violate the spirit of the thing.)

Yeah, i think it should be safe. Lets try this? It looks like the simplest variant.

Quoted 4 lines · Andrew Lutomirski

Quoted 3 lines · Andrew Lutomirski

3. Add int 0xcc and use it from vgettimeofday.  It will SIGSEGV if called from a user address (so it has no risk of ever becoming ABI) and it will do gettimeofday if called from the right address.  (I like

Quoted 3 lines

Make it a real syscall but with extra constraints. It would have the same calling convention as the syscall instruction, but it would turn into SIGKILL if the calling address isn't in the VSYSCALL page

Ingo Molnar writes:
 >
 > * Andrew Lutomirski <[email protected]> wrote:
 >
 > > On Fri, May 27, 2011 at 7:36 AM, Andrew Lutomirski <[email protected]> wrote:
 > > > 3. Add int 0xcc and use it from vgettimeofday.  It will SIGSEGV if > > > called from a user address (so it has no risk of ever becoming ABI) > > > and it will do gettimeofday if called from the right address.  (I like
...
 > > Make it a real syscall but with extra constraints.  It would have the > > same calling convention as the syscall instruction, but it would turn > > into SIGKILL if the calling address isn't in the VSYSCALL page

This will make things difficult for user-space dynamic binary instrumentation applications, since these normally execute generated code at different addresses than the original code.

Is there a safe fallback for this particular vsyscall?

Quoted 13 lines · Mikael Pettersson

Ingo Molnar writes:
 >
 > * Andrew Lutomirski <[email protected]> wrote:
 >
 > > On Fri, May 27, 2011 at 7:36 AM, Andrew Lutomirski <[email protected]> wrote:
 > > > 3. Add int 0xcc and use it from vgettimeofday.  It will SIGSEGV if > > > called from a user address (so it has no risk of ever becoming ABI) > > > and it will do gettimeofday if called from the right address.  (I like
...
 > > Make it a real syscall but with extra constraints.  It would have the > > same calling convention as the syscall instruction, but it would turn > > into SIGKILL if the calling address isn't in the VSYSCALL page

Quoted 5 lines

This will make things difficult for user-space dynamic binary instrumentation applications, since these normally execute generated code at different addresses than the original code.

Is there a safe fallback for this particular vsyscall?

All of the vsyscalls have vDSO versions that work like any other code.

Alternatively, if the dynamic instrumentation code knew about vsyscalls, it could just not instrument addresses in the vsyscall page.

What existing applications would get broken?

Andrew Lutomirski writes:
 > On Sun, May 29, 2011 at 10:39 AM, Mikael Pettersson <[email protected]> wrote:
 > > Ingo Molnar writes:
 > >  >
 > >  > * Andrew Lutomirski <[email protected]> wrote:
 > >  >
 > >  > > On Fri, May 27, 2011 at 7:36 AM, Andrew Lutomirski <[email protected]> wrote:
 > >  > > > 3. Add int 0xcc and use it from vgettimeofday.  It will SIGSEGV if > >  > > > called from a user address (so it has no risk of ever becoming ABI) > >  > > > and it will do gettimeofday if called from the right address.  (I like > > ...
 > >  > > Make it a real syscall but with extra constraints.  It would have the > >  > > same calling convention as the syscall instruction, but it would turn > >  > > into SIGKILL if the calling address isn't in the VSYSCALL page > >
 > > This will make things difficult for user-space dynamic binary instrumentation > > applications, since these normally execute generated code at different > > addresses than the original code.
 > >
 > > Is there a safe fallback for this particular vsyscall?
 >
 > All of the vsyscalls have vDSO versions that work like any other code.

Easiest would be if we can simply map int $0xcc with rAX==FOO to syscall or int 0x80 with rAX==BAR.

We currently don't even know about the vDSO, it's all just user-space code to us.

 > Alternatively, if the dynamic instrumentation code knew about > vsyscalls, it could just not instrument addresses in the vsyscall > page.

Not instrumenting code is not an option, unless we can prove that the code in question has no relevant side-effects or unexpected control-flow. (Where "side-effects" relate both to the integrity of the instrumentation engine and the application-specific payload it's attaching to the code.)

 > What existing applications would get broken?

My concern is ThreadSpotter, but any user-space dynamic binary instrumentation engine that instruments down to the raw kernel interface (syscall/sysenter/int instructions) would have a problem with syscalls that only work at specific addresses.

Anyway, if I can map that vsyscall to a plain proper syscall, then I'm OK.

Quoted 23 lines · Mikael Pettersson

Andrew Lutomirski writes:
 > On Sun, May 29, 2011 at 10:39 AM, Mikael Pettersson <[email protected]> wrote:
 > > Ingo Molnar writes:
 > >  >
 > >  > * Andrew Lutomirski <[email protected]> wrote:
 > >  >
 > >  > > On Fri, May 27, 2011 at 7:36 AM, Andrew Lutomirski <[email protected]> wrote:
 > >  > > > 3. Add int 0xcc and use it from vgettimeofday.  It will SIGSEGV if > >  > > > called from a user address (so it has no risk of ever becoming ABI) > >  > > > and it will do gettimeofday if called from the right address.  (I like > > ...
 > >  > > Make it a real syscall but with extra constraints.  It would have the > >  > > same calling convention as the syscall instruction, but it would turn > >  > > into SIGKILL if the calling address isn't in the VSYSCALL page > >
 > > This will make things difficult for user-space dynamic binary instrumentation > > applications, since these normally execute generated code at different > > addresses than the original code.
 > >
 > > Is there a safe fallback for this particular vsyscall?
 >
 > All of the vsyscalls have vDSO versions that work like any other code.

Quoted 2 lines

Easiest would be if we can simply map int $0xcc with rAX==FOO to syscall or int 0x80 with rAX==BAR.

Yes and no.

With the code I just posted (and am fixing up now) that will work. But if we want to replace the entire vsyscall page with three int 0xcc and 4090 int3 instructions, then we can't look at eax because it won't contain anything meaningful.

--Andy

Quoted 4 lines

We currently don't even know about the vDSO, it's all just user-space code to us.

Quoted 8 lines

 > Alternatively, if the dynamic instrumentation code knew about > vsyscalls, it could just not instrument addresses in the vsyscall > page.

Not instrumenting code is not an option, unless we can prove that the code in question has no relevant side-effects or unexpected control-flow. (Where "side-effects" relate both to the integrity of the instrumentation engine and the application-specific payload it's attaching to the code.)

Calls to 0xffffffffff600000, 0xffffffffff600400, and 0xffffffffff600800 are syscalls, as an (unfortunate) part of the ABI.

Quoted 7 lines

 > What existing applications would get broken?

My concern is ThreadSpotter, but any user-space dynamic binary instrumentation engine that instruments down to the raw kernel interface (syscall/sysenter/int instructions) would have a problem with syscalls that only work at specific addresses.

I'll look.

Quoted 2 lines

Anyway, if I can map that vsyscall to a plain proper syscall, then I'm OK.

All three vsyscalls can be replaced with real syscalls without side effects. Would it be possible to teach the instrumentation code to deal with that?

Andrew Lutomirski writes:
 > On Sun, May 29, 2011 at 12:01 PM, Mikael Pettersson <[email protected]> wrote:
 > > Andrew Lutomirski writes:
 > >  >
 > >  > All of the vsyscalls have vDSO versions that work like any other code. > >
 > > Easiest would be if we can simply map int $0xcc with rAX==FOO to syscall or > > int 0x80 with rAX==BAR.
 >
 > Yes and no.
 >
 > With the code I just posted (and am fixing up now) that will work. > But if we want to replace the entire vsyscall page with three int 0xcc > and 4090 int3 instructions, then we can't look at eax because it won't > contain anything meaningful.

I can relatively easily also consider the original application rIP when decoding and translating these instructions.

 >
 > --Andy
 >
 > >
 > > We currently don't even know about the vDSO, it's all just user-space code > > to us.
 > >
 > >  > Alternatively, if the dynamic instrumentation code knew about > >  > vsyscalls, it could just not instrument addresses in the vsyscall > >  > page.
 > >
 > > Not instrumenting code is not an option, unless we can prove that the > > code in question has no relevant side-effects or unexpected control-flow. > > (Where "side-effects" relate both to the integrity of the instrumentation > > engine and the application-specific payload it's attaching to the code.) >
 > Calls to 0xffffffffff600000, 0xffffffffff600400, and
 > 0xffffffffff600800 are syscalls, as an (unfortunate) part of the ABI. >
 > >
 > >  > What existing applications would get broken?
 > >
 > > My concern is ThreadSpotter, but any user-space dynamic binary instrumentation > > engine that instruments down to the raw kernel interface (syscall/sysenter/int > > instructions) would have a problem with syscalls that only work at specific > > addresses.
 >
 > I'll look.
 >
 > >
 > > Anyway, if I can map that vsyscall to a plain proper syscall, then I'm OK. >
 > All three vsyscalls can be replaced with real syscalls without side > effects.  Would it be possible to teach the instrumentation code to > deal with that?

Yes, I just need to know how to identify them and what their equivalents are. E.g., an int3 at <known address> becomes syscall rAX=<some constant>.

Sounds like this change will be manageable after all.  Thanks.

/Mikael

Quoted 55 lines · Mikael Pettersson

Andrew Lutomirski writes:
 > On Sun, May 29, 2011 at 12:01 PM, Mikael Pettersson <[email protected]> wrote:
 > > Andrew Lutomirski writes:
 > >  >
 > >  > All of the vsyscalls have vDSO versions that work like any other code. > >
 > > Easiest would be if we can simply map int $0xcc with rAX==FOO to syscall or > > int 0x80 with rAX==BAR.
 >
 > Yes and no.
 >
 > With the code I just posted (and am fixing up now) that will work. > But if we want to replace the entire vsyscall page with three int 0xcc > and 4090 int3 instructions, then we can't look at eax because it won't > contain anything meaningful.

I can relatively easily also consider the original application rIP when decoding and translating these instructions.

 >
 > --Andy
 >
 > >
 > > We currently don't even know about the vDSO, it's all just user-space code > > to us.
 > >
 > >  > Alternatively, if the dynamic instrumentation code knew about > >  > vsyscalls, it could just not instrument addresses in the vsyscall > >  > page.
 > >
 > > Not instrumenting code is not an option, unless we can prove that the > > code in question has no relevant side-effects or unexpected control-flow. > > (Where "side-effects" relate both to the integrity of the instrumentation > > engine and the application-specific payload it's attaching to the code.) >
 > Calls to 0xffffffffff600000, 0xffffffffff600400, and
 > 0xffffffffff600800 are syscalls, as an (unfortunate) part of the ABI. >
 > >
 > >  > What existing applications would get broken?
 > >
 > > My concern is ThreadSpotter, but any user-space dynamic binary instrumentation > > engine that instruments down to the raw kernel interface (syscall/sysenter/int > > instructions) would have a problem with syscalls that only work at specific > > addresses.
 >
 > I'll look.
 >
 > >
 > > Anyway, if I can map that vsyscall to a plain proper syscall, then I'm OK. >
 > All three vsyscalls can be replaced with real syscalls without side > effects.  Would it be possible to teach the instrumentation code to > deal with that?

Quoted 4 lines

Yes, I just need to know how to identify them and what their equivalents are. E.g., an int3 at <known address> becomes syscall rAX=<some constant>.

Sounds like this change will be manageable after all.  Thanks.

I'm not entirely sure I like that -- that way if we ever change it again we break your stuff again.

Here are two proposals.

1. Teach your code that call 0xffffffffff600000 means gettimeofday(rdi, rsi).  That's guaranteed to never change and will keep working even if we start to emulate vsyscalls by marking the page not present and trapping the instruction fetch fault.