arm64: Signal context expansion

This new record can be added in sigframe.__reserved[] if there is a need to allocate extra space beyond the standard signal frame. The extra block of memory referenced by extra_context can then be parsed in the same way as sigcontext.__reserved[]. Old code should just ignore the whole thing as an unrecognised record. To maintain backward compatibility, signal context records defined today are always placed directly in __reserved[], never in the block referenced by extra_context.

Quoted 8 lines · Dave Martin

This new record can be added in sigframe.__reserved[] if there is a need to allocate extra space beyond the standard signal frame. The extra block of memory referenced by extra_context can then be parsed in the same way as sigcontext.__reserved[]. Old code should just ignore the whole thing as an unrecognised record. To maintain backward compatibility, signal context records defined today are always placed directly in __reserved[], never in the block referenced by extra_context.

Do you add this extra information only if the stack is sufficiently large?

x86_64 adds the new information even for small stacks set up with sigaltstack, leading to memory corruption on bleeding-edge hardware:

<https://bugzilla.kernel.org/show_bug.cgi?id=153531>

Quoted 5 lines

Do you add this extra information only if the stack is sufficiently large?

x86_64 adds the new information even for small stacks set up with sigaltstack, leading to memory corruption on bleeding-edge hardware:

Quoted 1 line

<https://bugzilla.kernel.org/show_bug.cgi?id=153531>

Hmmm, not yet. We already check that the whole frame is writable user memory, but this isn't sufficient to avoid user corruption in the case of alternate signal stacks. I'll fix this -- thanks for flagging it.

If the stack isn't large enough, we'll still have to SEGV the task though.

We can (and should) bump up the SIG{,MIN}STKSZ constants when adding the SVE support proper to the kernel,

I wonder whether we should make the signal stack size runtime discoverable through sysconf() instead...

Quoted 11 lines · Dave Martin

Quoted 6 lines

Do you add this extra information only if the stack is sufficiently large?

x86_64 adds the new information even for small stacks set up with sigaltstack, leading to memory corruption on bleeding-edge hardware:

<https://bugzilla.kernel.org/show_bug.cgi?id=153531>

Hmmm, not yet. We already check that the whole frame is writable user memory, but this isn't sufficient to avoid user corruption in the case of alternate signal stacks. I'll fix this -- thanks for flagging it.

Quoted 2 lines

If the stack isn't large enough, we'll still have to SEGV the task though.

You could skip copying the data and not install a pointer to it in the existing signal context.

Quoted 2 lines

We can (and should) bump up the SIG{,MIN}STKSZ constants when adding the SVE support proper to the kernel,

That's a userspace ABI change (libraries use these constants to size struct members), and not a good idea. You might get away with at this stage, but you can't do this every time you add some new process state you want to add to signal handlers.

Quoted 2 lines

I wonder whether we should make the signal stack size runtime discoverable through sysconf() instead...

Quoted 5 lines

Quoted 2 lines

If the stack isn't large enough, we'll still have to SEGV the task though.

You could skip copying the data and not install a pointer to it in the existing signal context.

We could, but then we'd corrupt the task state in sigreturn, since we wouldn't have been able to save/restore part of the state.

The least-wrong thing I can think of to do is:

* deprecate but continue to support the existing sigaltstack API/ABI with today's {,MIN}SIGSTKSZ definitions

* guarantee (as much as possible) that software using this ABI continues to work (by saving/restoring only data that _must_ be saved/restored at each signal, which may be small enough to fit)

* providing a clean failure mode (fatal signal) when this proves impossible at signal delivery/return time;

* define a new interface for runtime-querying the required signal stack size;

* define a new syscall or new stack_t.ss_flags flags (say, SS_STRICT) that permits the kernel to enforce a runtime-determined minimum greater than MINSIGSTKSZ when calling sigaltstack().

Another option would be:

* define a new interface for runtime-querying the required signal stack size, and

* support the current API/ABI, but make a call to sigaltstack() SEGV or SIGILL the caller if it specifies ss_stack >= MINSIGSTKSZ but smaller than the actual runtime minimum.

(this would cause old software to break immediately in an obvious way on new systems, forcing people to fix their software -- which they might or might not actually bother to do).

Quoted 3 lines · Dave Martin

Quoted 3 lines

Quoted 2 lines

If the stack isn't large enough, we'll still have to SEGV the task though.

Quoted 5 lines

Quoted 2 lines

You could skip copying the data and not install a pointer to it in the existing signal context.

We could, but then we'd corrupt the task state in sigreturn, since we wouldn't have been able to save/restore part of the state.

Ah, I wasn't aware that the kernel doesn't have a copy of the state. Could the kernel reserve space for it when sigaltstack is called?

Quoted 4 lines

The least-wrong thing I can think of to do is:

* deprecate but continue to support the existing sigaltstack API/ABI with today's {,MIN}SIGSTKSZ definitions

There is also PTHREAD_STACK_MIN. The glibc default for that (which is overriden by some architectures) is 16 KiB. It is also quite small; see the sequence of events mentioned here:

<https://sourceware.org/bugzilla/show_bug.cgi?id=20249>

Quoted 24 lines

* guarantee (as much as possible) that software using this ABI continues to work (by saving/restoring only data that _must_ be saved/restored at each signal, which may be small enough to fit)

* providing a clean failure mode (fatal signal) when this proves impossible at signal delivery/return time;

* define a new interface for runtime-querying the required signal stack size;

* define a new syscall or new stack_t.ss_flags flags (say, SS_STRICT) that permits the kernel to enforce a runtime-determined minimum greater than MINSIGSTKSZ when calling sigaltstack().

Another option would be:

* define a new interface for runtime-querying the required signal stack size, and

* support the current API/ABI, but make a call to sigaltstack() SEGV or SIGILL the caller if it specifies ss_stack >= MINSIGSTKSZ but smaller than the actual runtime minimum.

Quoted 3 lines

(this would cause old software to break immediately in an obvious way on new systems, forcing people to fix their software -- which they might or might not actually bother to do).

The second option looks a bit problematic from a support perspective.

Do you think it would be possible to block access to hardware features that cause signal stack bloat on a per-process basis? Then we could bump the kernel requirement enforced by sigaltstack directly for the default, and define a compatibility personality that minimizes the signal stack size for old applications. (A way to query the required alternative signal stack size would still come in handy, though.)

Quoted 13 lines · Florian Weimer

Quoted 11 lines · Dave Martin

Quoted 6 lines

Do you add this extra information only if the stack is sufficiently large?

x86_64 adds the new information even for small stacks set up with sigaltstack, leading to memory corruption on bleeding-edge hardware:

<https://bugzilla.kernel.org/show_bug.cgi?id=153531>

Hmmm, not yet. We already check that the whole frame is writable user memory, but this isn't sufficient to avoid user corruption in the case of alternate signal stacks. I'll fix this -- thanks for flagging it.

Quoted 5 lines

Quoted 2 lines

If the stack isn't large enough, we'll still have to SEGV the task though.

You could skip copying the data and not install a pointer to it in the existing signal context.

We could, but then we'd corrupt the task state in sigreturn, since we wouldn't have been able to save/restore part of the state.

Quoted 7 lines

Quoted 2 lines

We can (and should) bump up the SIG{,MIN}STKSZ constants when adding the SVE support proper to the kernel,

That's a userspace ABI change (libraries use these constants to size struct members), and not a good idea. You might get away with at this stage, but you can't do this every time you add some new process state you want to add to signal handlers.

For internal interfaces within a single component that's tolerable, since a single value would be used for each of these constants throughout the build of that component.

A quick search on sources.debian.org suggests that the total number of packages that expose {,MIN}SIGSTKSZ dependent definitions in their public interfaces is small (I couldn't find any after paging through dozens of pages of results -- so the total is maybe in the range 0-10) -- hopefully few enough to eyeball.

Quoted 2 lines

Quoted 2 lines

I wonder whether we should make the signal stack size runtime discoverable through sysconf() instead...

I will likely suggest this for the future, but of course it doesn't help for current binaries.

Note that MINSIGSTKSZ stared life wrong for arm64, and has since gone through a few ABI breaking changes. I don't condone this, but we have form in this area :/

sigaltstack() already fails with ENOMEM for software that passes ss_size = MINSIGSTKSZ, and is built against glibc<2.22 [1], [2], running on linux>=4.3 [3], which is an ABI break in case where sigaltstack() is otherwise guaranteed to succeed.

The bottom line here is that the sigaltstack() API is broken with regard to extensibility, so we cannot extend the amount of signal state without breaking something.

The least-wrong thing I can think of to do is:

* deprecate but continue to support the existing sigaltstack API/ABI with today's {,MIN}SIGSTKSZ definitions

* guarantee (as much as possible) that software using this ABI continues to work (by saving/restoring only data that _must_ be saved/restored at each signal, which may be small enough to fit)

* providing a clean failure mode (fatal signal) when this proves impossible at signal delivery/return time;

* define a new interface for runtime-querying the required signal stack size;

* define a new syscall or new stack_t.ss_flags flags (say, SS_STRICT) that permits the kernel to enforce a runtime-determined minimum greater than MINSIGSTKSZ when calling sigaltstack().

Another option would be:

* define a new interface for runtime-querying the required signal stack size, and

* support the current API/ABI, but make a call to sigaltstack() SEGV or SIGILL the caller if it specifies ss_stack >= MINSIGSTKSZ but smaller than the actual runtime minimum.

(this would cause old software to break immediately in an obvious way on new systems, forcing people to fix their software -- which they might or might not actually bother to do).

Thoughts?

Cheers
---Dave

[1] https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=b763f6ae859ecea70a5dacb8ad45c71d5f667e2e

[2] https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/bits/sigstack.h;h=e143034ce24dc383016b4882b89cfebc7a6a62d7;hb=b8079dd0d360648e4e8de48656c5c38972621072

[3] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c9692657c0321fec7bcb3ca8c6db56c08c640ace

On 12/09/16 12:17, Dave Martin wrote:

Quoted 1 line

Quoted 1 line · Dave Martin

Quoted 7 lines

Quoted 2 lines

Quoted 2 lines

I wonder whether we should make the signal stack size runtime discoverable through sysconf() instead...

I will likely suggest this for the future, but of course it doesn't help for current binaries.

Quoted 9 lines

Note that MINSIGSTKSZ stared life wrong for arm64, and has since gone through a few ABI breaking changes. I don't condone this, but we have form in this area :/

sigaltstack() already fails with ENOMEM for software that passes ss_size = MINSIGSTKSZ, and is built against glibc<2.22 [1], [2], running on linux>=4.3 [3], which is an ABI break in case where sigaltstack() is otherwise guaranteed to succeed.

yes, this was abi breaking change.

if glibc does not care about existing binaries that use sigaltstack with MINSIGSTKSZ then it can increase the size, but i think the kernel should not change the abi (there are other libcs and libc independent runtime systems on linux for aarch64 with their own sigaltstack setup, not all of them may care about SVE).

i assume the kernel can avoid saving SVE regs when they are not used by the process.

Quoted 6 lines

The bottom line here is that the sigaltstack() API is broken with regard to extensibility, so we cannot extend the amount of signal state without breaking something.

extending signal state can break things independently of sigaltstack.

binaries with strict guarantees about worst case stack usage can change behaviour.

fortunately glibc PTHREAD_STACK_MIN is huge on aarch64 so applications using it are unlikely to break because of the increased signal state.
(this also means it's impossible to have threads with tiny stacks on glibc, so large amount of threads means large amount of commit charge.)

Quoted 1 line · Szabolcs Nagy

On 12/09/16 12:17, Dave Martin wrote:

[...]

Quoted 1 line

Quoted 1 line

Quoted 1 line · Dave Martin

[...]

Quoted 27 lines

Quoted 7 lines

Quoted 2 lines

Quoted 2 lines

I wonder whether we should make the signal stack size runtime discoverable through sysconf() instead...

I will likely suggest this for the future, but of course it doesn't help for current binaries.

Quoted 9 lines

Note that MINSIGSTKSZ stared life wrong for arm64, and has since gone through a few ABI breaking changes. I don't condone this, but we have form in this area :/

sigaltstack() already fails with ENOMEM for software that passes ss_size = MINSIGSTKSZ, and is built against glibc<2.22 [1], [2], running on linux>=4.3 [3], which is an ABI break in case where sigaltstack() is otherwise guaranteed to succeed.

yes, this was abi breaking change.

if glibc does not care about existing binaries that use sigaltstack with MINSIGSTKSZ then it can increase the size, but i think the kernel should not change the abi (there are other libcs and libc independent runtime systems on linux for aarch64 with their own sigaltstack setup, not all of them may care about SVE).

Quoted 2 lines

i assume the kernel can avoid saving SVE regs when they are not used by the process.

I can (and do), in my patches (not posted yet).

The real issue here is that a recently updated shared library might be optimised to use SVE, where the program using it is an older, SVE- unaware binary.

(think of an optimised math library using some new fancy SVE-based number crunching internally).

Quoted 10 lines

Quoted 6 lines

The bottom line here is that the sigaltstack() API is broken with regard to extensibility, so we cannot extend the amount of signal state without breaking something.

extending signal state can break things independently of sigaltstack.

Quoted 2 lines

binaries with strict guarantees about worst case stack usage can change behaviour.

Indeed, but this is not a new issue. Software must run with enough stack in order to be portable, but there is no portable way to determine how much stack is needed.

Quoted 6 lines

fortunately glibc PTHREAD_STACK_MIN is huge on aarch64 so applications using it are unlikely to break because of the increased signal state.
(this also means it's impossible to have threads with tiny stacks on glibc, so large amount of threads means large amount of commit charge.)

Again, not a new problem.

[...]

Cheers
---Dave

Quoted 45 lines · Dave Martin

Quoted 1 line · Szabolcs Nagy

On 12/09/16 12:17, Dave Martin wrote:

[...]

Quoted 1 line

Quoted 1 line

Quoted 1 line · Dave Martin

[...]

Quoted 27 lines

Quoted 7 lines

Quoted 2 lines

Quoted 2 lines

I wonder whether we should make the signal stack size runtime discoverable through sysconf() instead...

I will likely suggest this for the future, but of course it doesn't help for current binaries.

Quoted 9 lines

Note that MINSIGSTKSZ stared life wrong for arm64, and has since gone through a few ABI breaking changes. I don't condone this, but we have form in this area :/

sigaltstack() already fails with ENOMEM for software that passes ss_size = MINSIGSTKSZ, and is built against glibc<2.22 [1], [2], running on linux>=4.3 [3], which is an ABI break in case where sigaltstack() is otherwise guaranteed to succeed.

yes, this was abi breaking change.

if glibc does not care about existing binaries that use sigaltstack with MINSIGSTKSZ then it can increase the size, but i think the kernel should not change the abi (there are other libcs and libc independent runtime systems on linux for aarch64 with their own sigaltstack setup, not all of them may care about SVE).

Quoted 2 lines

i assume the kernel can avoid saving SVE regs when they are not used by the process.

I can (and do), in my patches (not posted yet).

The real issue here is that a recently updated shared library might be optimised to use SVE, where the program using it is an older, SVE- unaware binary.

Quoted 3 lines

(think of an optimised math library using some new fancy SVE-based number crunching internally).

this is why sve would be a new abi in an ideal world, instead of subtly changing behaviour behind existing binaries.

Quoted 10 lines

Quoted 10 lines

Quoted 6 lines

The bottom line here is that the sigaltstack() API is broken with regard to extensibility, so we cannot extend the amount of signal state without breaking something.

extending signal state can break things independently of sigaltstack.

Quoted 7 lines

Quoted 2 lines

binaries with strict guarantees about worst case stack usage can change behaviour.

Indeed, but this is not a new issue. Software must run with enough stack in order to be portable, but there is no portable way to determine how much stack is needed.

why not?

musl libc has stack usage guarantee that can be checked by the compiler that built the libc on all supported targets. (with some manual checks in a few cases.)

however as soon as targets change the signal frame size randomly, these worst case guarantees get broken. (musl's MINSIGSTKSZ was always 6k on aarch64, even before the kernel changed it.)

Quoted 9 lines

Quoted 6 lines

fortunately glibc PTHREAD_STACK_MIN is huge on aarch64 so applications using it are unlikely to break because of the increased signal state.
(this also means it's impossible to have threads with tiny stacks on glibc, so large amount of threads means large amount of commit charge.)

Again, not a new problem.

on musl PTHREAD_STACK_MIN is small, so the sve signal state can break existing binaries that use tiny thread stack (there is at least one linux distro that supports musl+aarch64, although they don't distribute binary packages yet).

so even if the glibc abi changes, i think the kernel should not do the change, at least not immediately.

(and if the kernel decides to increase MINSIGSTKSZ i think it should wait for distros picking up new enough glibc with increased MINSIGSTKSZ to avoid spuriously failing sigaltstack calls.)

Quoted 2 lines · Szabolcs Nagy

i assume the kernel can avoid saving SVE regs when they are not used by the process.

I can (and do), in my patches (not posted yet).

The real issue here is that a recently updated shared library might be optimised to use SVE, where the program using it is an older, SVE- unaware binary.

(think of an optimised math library using some new fancy SVE-based number crunching internally).

Quoted 4 lines · Dave Martin

Quoted 2 lines · Szabolcs Nagy

i assume the kernel can avoid saving SVE regs when they are not used by the process.

Quoted 8 lines

I can (and do), in my patches (not posted yet).

The real issue here is that a recently updated shared library might be optimised to use SVE, where the program using it is an older, SVE- unaware binary.

(think of an optimised math library using some new fancy SVE-based number crunching internally).

In my little world, binaries are expected to do run-time CPU feature discovery and fall back to the baseline ABI if the system does not support ABI extensions. This means that a personality flag can disable the extension, the optimized math libraries run with the slower baseline code instead, and the rest of the application sees the expected kernel interface.

From this perspective, it is quite important that it is possible to get legacy applications running on newer hardware, preferably with out wrapping them in a dedicated VM.

I understand that there are other environments were this is less of a concern, but I also believe that the age of an architecture factors less into this than one might want.

Agreed. I'll need to think some more about how this should work in general.

Quoted 2 lines · Dave Martin

Agreed. I'll need to think some more about how this should work in general.

Thanks.

Depending on some SVE implementations details (which I know nothing about, I only saw some public overview slides), we may also need additional storage space to preserve SVE registers in the dynamic linker. Due to lazy binding, this code cn be called from a signal handler, so this needs to be factored into stack size requirements as well.

Problematic are register width extensions used for argument passing and callee-saved registers whose width has been extended. Both are particularly challenging to deal with if existing vector instructions clear the extension part (which may be desirable for other reasons).

The size of the jmp_buf type is a concern as well.

Quoted 7 lines · Florian Weimer

Quoted 2 lines · Dave Martin

Agreed. I'll need to think some more about how this should work in general.

Thanks.

Quoted 5 lines

Depending on some SVE implementations details (which I know nothing about, I only saw some public overview slides), we may also need additional storage space to preserve SVE registers in the dynamic linker. Due to lazy binding, this code cn be called from a signal handler, so this needs to be factored into stack size requirements as well.

Yes and no. The kernel SIGSTKSZ constants don't care about ld.so -- that's userspace overhead, not kernel overhead.

Quoted 6 lines

Problematic are register width extensions used for argument passing and callee-saved registers whose width has been extended. Both are particularly challenging to deal with if existing vector instructions clear the extension part (which may be desirable for other reasons).

The size of the jmp_buf type is a concern as well.

The default PCS for SVE will not be introducing any extra save/restore requirements for SVE -- i.e., everything is caller-save at public interfaces, except for the FPSIMD register bits that are already callee- save under the existing PCS.

Quoted 14 lines · Dave Martin

Quoted 12 lines · Florian Weimer

Quoted 2 lines · Dave Martin

Agreed. I'll need to think some more about how this should work in general.

Thanks.

Depending on some SVE implementations details (which I know nothing about, I only saw some public overview slides), we may also need additional storage space to preserve SVE registers in the dynamic linker. Due to lazy binding, this code cn be called from a signal handler, so this needs to be factored into stack size requirements as well.

Quoted 2 lines

Yes and no. The kernel SIGSTKSZ constants don't care about ld.so -- that's userspace overhead, not kernel overhead.

Well yes, so is jmp_buf. But I think you still should consider the full picture. :)

At least the lazy binding stack overhead can be avoided with LD_BIND_NOW=1.

Quoted 6 lines

Quoted 6 lines

Problematic are register width extensions used for argument passing and callee-saved registers whose width has been extended. Both are particularly challenging to deal with if existing vector instructions clear the extension part (which may be desirable for other reasons).

Quoted 6 lines

Quoted 1 line

The size of the jmp_buf type is a concern as well.

The default PCS for SVE will not be introducing any extra save/restore requirements for SVE -- i.e., everything is caller-save at public interfaces, except for the FPSIMD register bits that are already callee- save under the existing PCS.

Is it possible to pass arguments in the register extension parts? If existing non-SVE code can clobber these bits, we need some adjustments in libc to save and restore SVE state in some places (which may need further stack space).