rseq/x86: choosing rseq code signature
Subsystems (2)
Mathieu Desnoyers
maintainerHi,
We are about to include the code signature required prior to restartable sequences abort handlers into glibc, which will make this ABI choice final. We need architecture maintainer input on that signature value.
That code signature is placed before each abort handler, so the kernel can validate that it is indeed jumping to an abort handler (and not some arbitrary attacker-chosen code). The signature is never executed.
Currently, tools/testing/selftests/rseq/rseq-x86.h defines RSEQ_SIG as 0x53053053, and uses it as an immediate operand to the following instruction opcodes (as suggested by Andy Lutomirski):
x86-32:
- .byte 0x0f, 0x1f, 0x05: nopl <sig>
x86-64:
- .byte 0x0f, 0x1f, 0x05: nopl <sig>(%rip)
The current discussion thread on the glibc mailing list leads us towards using a trap with uncommon immediate operand, which simplifies integration with disassemblers, emulators, makes it easier to debug if the control flow gets redirected there by mistake, and is nicer for some architecture's speculative execution.
The main advantage of choosing a trap instruction over a no-op is to ensure the program traps if the execution flow gets redirected to the signature by mistake (makes it easier to debug). It's not a hard requirement, but it would be a bonus.
Are there trap instructions that take an uncommon 4-byte immediate operand you would recommend on x86 32/64 ? Or is the current choice of nopl confirmed to be right one ?
Here is an example of rseq signature definition template:
/*
* TODO: document trap instruction objdump output on each sub-architecture
* instruction sets, as well as instruction set extensions.
*/
#define RSEQ_SIG 0x########
Thanks!
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerQuoted 35 lines · ----- On Apr 9, 2019, at 3:32 PM, Mathieu Desnoyers…
Hi,
We are about to include the code signature required prior to restartable sequences abort handlers into glibc, which will make this ABI choice final. We need architecture maintainer input on that signature value.
That code signature is placed before each abort handler, so the kernel can validate that it is indeed jumping to an abort handler (and not some arbitrary attacker-chosen code). The signature is never executed.
Currently,
tools/testing/selftests/rseq/rseq-x86.hdefinesRSEQ_SIGas0x53053053, and uses it as an immediate operand to the following instruction opcodes (as suggested by Andy Lutomirski):x86-32:
- .byte0x0f,0x1f,0x05: nopl <sig>x86-64:
- .byte0x0f,0x1f,0x05: nopl <sig>(%rip)The current discussion thread on the glibc mailing list leads us towards using a trap with uncommon immediate operand, which simplifies integration with disassemblers, emulators, makes it easier to debug if the control flow gets redirected there by mistake, and is nicer for some architecture's speculative execution.
The main advantage of choosing a trap instruction over a no-op is to ensure the program traps if the execution flow gets redirected to the signature by mistake (makes it easier to debug). It's not a hard requirement, but it would be a bonus.
Are there trap instructions that take an uncommon 4-byte immediate operand you would recommend on x86 32/64 ? Or is the current choice of nopl confirmed to be right one ?
Quoted 7 lines
Here is an example of rseq signature definition template:
/*
* TODO: document trap instruction objdump output on each sub-architecture
* instruction sets, as well as instruction set extensions.
*/
#defineRSEQ_SIG0x########
Peter Zijlstra suggested to use "invlpg" in user-space, which should generate a trap. The only concern would be emulators, but ideally they would not try to decode an instruction that is never executed. This would lead to the following patch. Any objections/ack ?
tools/testing/selftests/rseq/rseq-x86.h +9 -4
@@ -7,6 +7,11 @@#include <stdint.h> +/* +* RSEQ_SIG is used with the following privileged instructions, which trap in user-space: +* x86-32: 0f 01 3d 53 30 05 53 invlpg 0x53053053 +* x86-64: 0f 01 3d 53 30 05 53 invlpg 0x53053053(%rip) +*/ #define RSEQ_SIG 0x53053053#ifdef __x86_64__
@@ -78,8 +83,8 @@ do { \#define RSEQ_ASM_DEFINE_ABORT (label, teardown, abort_label) \ ".pushsection __rseq_failure, \"ax\"\n\t" \ -/* Disassembler-friendly signature: nopl <sig>(%rip). */ \ -".byte 0x0f, 0x1f, 0x05\n\t" \ +/* Disassembler-friendly signature: invlpg <sig>(%rip). */ \ +".byte 0x0f, 0x01, 0x3d\n\t" \ ".long " __rseq_str(RSEQ_SIG) "\n\t" \ __rseq_str(label) ":\n\t" \ teardown \
@@ -605,8 +610,8 @@ do { \#define RSEQ_ASM_DEFINE_ABORT (label, teardown, abort_label) \ ".pushsection __rseq_failure, \"ax\"\n\t" \ -/* Disassembler-friendly signature: nopl <sig>. */ \ -".byte 0x0f, 0x1f, 0x05\n\t" \ +/* Disassembler-friendly signature: invlpg <sig>. */ \ +".byte 0x0f, 0x1f, 0x3d\n\t" \ ".long " __rseq_str(RSEQ_SIG) "\n\t" \ __rseq_str(label) ":\n\t" \ teardown \
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 25 lines · Mathieu Desnoyers
Quoted 24 lines · ----- On Apr 9, 2019, at 3:32 PM, Mathieu Desnoyers…
We are about to include the code signature required prior to restartable sequences abort handlers into glibc, which will make this ABI choice final. We need architecture maintainer input on that signature value.
That code signature is placed before each abort handler, so the kernel can validate that it is indeed jumping to an abort handler (and not some arbitrary attacker-chosen code). The signature is never executed.
Currently,
tools/testing/selftests/rseq/rseq-x86.hdefinesRSEQ_SIGas0x53053053, and uses it as an immediate operand to the following instruction opcodes (as suggested by Andy Lutomirski):x86-32:
- .byte0x0f,0x1f,0x05: nopl <sig>x86-64:
- .byte0x0f,0x1f,0x05: nopl <sig>(%rip)The current discussion thread on the glibc mailing list leads us towards using a trap with uncommon immediate operand, which simplifies integration with disassemblers, emulators, makes it easier to debug if the control flow gets redirected there by mistake, and is nicer for some architecture's speculative execution.
...
Quoted 4 lines
Peter Zijlstra suggested to use "invlpg" in user-space, which should generate a trap. The only concern would be emulators, but ideally they would not try to decode an instruction that is never executed. This would lead to the following patch. Any objections/ack ?
...
Quoted 6 lines
+/*
+ *RSEQ_SIGis used with the following privileged instructions, which trap in user-space:+ * x86-32: 0f 01 3d 53 30 05 53 invlpg 0x53053053 + * x86-64: 0f 01 3d 53 30 05 53 invlpg 0x53053053(%rip)+ */
#defineRSEQ_SIG0x53053053
On x86, you have to worry about what happens if control flow gets redirected to an arbitrary byte address. The proposed sequence 0f 01 3d 53 30 05 53 is a trap instruction if control lands seven bytes before the beginning of the abort handler, but if it lands anywhere _else_ within the marker sequence, you get one of these instruction sequences, none of which trap, all but one of which will corrupt the process state, and three of which will consume three bytes from the beginning of the abort handler's code, continuing execution with a misaligned PC:
01 3d 53 30 05 53 add %edi,0x53053053(%rip)
3d 53 30 05 53 cmp $0x53053053,%eax
53 30 05 53 XX XX XX push %rbx; xor %al,0xXXXXXX78(%rip)
30 05 53 XX XX XX xor %al,0xXXXXXX78(%rip)
05 53 XX XX XX add $0xXXXXXX53,%eax
53 push %rbxSo I'm going to suggest instead the four-byte sequence CD CF CD CF. That's INT $0xCF if control lands either two or four bytes before the beginning of the abort handler, and IRET if it lands one or three bytes before. I believe both of these possibilities are currently also forbidden in user mode. It doesn't need to be longer, does it?
zw
Quoted 58 lines · Zack Weinberg
Quoted 25 lines · Mathieu Desnoyers
Quoted 24 lines · ----- On Apr 9, 2019, at 3:32 PM, Mathieu Desnoyers…
We are about to include the code signature required prior to restartable sequences abort handlers into glibc, which will make this ABI choice final. We need architecture maintainer input on that signature value.
That code signature is placed before each abort handler, so the kernel can validate that it is indeed jumping to an abort handler (and not some arbitrary attacker-chosen code). The signature is never executed.
Currently,
tools/testing/selftests/rseq/rseq-x86.hdefinesRSEQ_SIGas0x53053053, and uses it as an immediate operand to the following instruction opcodes (as suggested by Andy Lutomirski):x86-32:
- .byte0x0f,0x1f,0x05: nopl <sig>x86-64:
- .byte0x0f,0x1f,0x05: nopl <sig>(%rip)The current discussion thread on the glibc mailing list leads us towards using a trap with uncommon immediate operand, which simplifies integration with disassemblers, emulators, makes it easier to debug if the control flow gets redirected there by mistake, and is nicer for some architecture's speculative execution.
...
Quoted 4 lines
Peter Zijlstra suggested to use "invlpg" in user-space, which should generate a trap. The only concern would be emulators, but ideally they would not try to decode an instruction that is never executed. This would lead to the following patch. Any objections/ack ?
...
Quoted 6 lines
+/*
+ *RSEQ_SIGis used with the following privileged instructions, which trap in user-space:+ * x86-32: 0f 01 3d 53 30 05 53 invlpg 0x53053053 + * x86-64: 0f 01 3d 53 30 05 53 invlpg 0x53053053(%rip)+ */
#defineRSEQ_SIG0x53053053On x86, you have to worry about what happens if control flow gets redirected to an arbitrary byte address. The proposed sequence
0f 01 3d 53 30 05 53is a trap instruction if control lands seven bytes before the beginning of the abort handler, but if it lands anywhere _else_ within the marker sequence, you get one of these instruction sequences, none of which trap, all but one of which will corrupt the process state, and three of which will consume three bytes from the beginning of the abort handler's code, continuing execution with a misaligned PC:01 3d 53 30 05 53 add %edi,0x53053053(%rip) 3d 53 30 05 53 cmp $0x53053053,%eax 53 30 05 53 XX XX XX push %rbx; xor %al,0xXXXXXX78(%rip) 30 05 53 XX XX XX xor %al,0xXXXXXX78(%rip) 05 53 XX XX XX add $0xXXXXXX53,%eax 53 push %rbx
Quoted 6 lines
So I'm going to suggest instead the four-byte sequence CD CF CD CF. That's INT $0xCF if control lands either two or four bytes before the beginning of the abort handler, and IRET if it lands one or three bytes before. I believe both of these possibilities are currently also forbidden in user mode. It doesn't need to be longer, does it?
IRET works in user mode just fine. Why are you concerned about landing in the middle of the signature? A misaligned jump into code is screwy pretty much no matter what. It does seem genuinely useful to trap if you accidentally fall through to the beginning of the signature, but I don't see the point of worrying about jumping to the middle.
There's some argument that, for consistency with CET, the last couple bytes of the signature should match ENDBR.
Mathieu, how many bytes do we have for the signature?
Mathieu Desnoyers
maintainerQuoted 75 lines · ----- On Apr 9, 2019, at 9:57 PM, Andy Lutomirski [email protected] wrote:
Quoted 60 lines · Zack Weinberg
Quoted 26 lines · Mathieu Desnoyers
----- On Apr 9, 2019, at 3:32 PM, Mathieu Desnoyers
Quoted 24 lines · [email protected] wrote:
We are about to include the code signature required prior to restartable sequences abort handlers into glibc, which will make this ABI choice final. We need architecture maintainer input on that signature value.
That code signature is placed before each abort handler, so the kernel can validate that it is indeed jumping to an abort handler (and not some arbitrary attacker-chosen code). The signature is never executed.
Currently,
tools/testing/selftests/rseq/rseq-x86.hdefinesRSEQ_SIGas0x53053053, and uses it as an immediate operand to the following instruction opcodes (as suggested by Andy Lutomirski):x86-32:
- .byte0x0f,0x1f,0x05: nopl <sig>x86-64:
- .byte0x0f,0x1f,0x05: nopl <sig>(%rip)The current discussion thread on the glibc mailing list leads us towards using a trap with uncommon immediate operand, which simplifies integration with disassemblers, emulators, makes it easier to debug if the control flow gets redirected there by mistake, and is nicer for some architecture's speculative execution.
...
Quoted 4 lines
Peter Zijlstra suggested to use "invlpg" in user-space, which should generate a trap. The only concern would be emulators, but ideally they would not try to decode an instruction that is never executed. This would lead to the following patch. Any objections/ack ?
...
Quoted 7 lines
+/*
+ *RSEQ_SIGis used with the following privileged instructions, which trap in user-space:+ * x86-32: 0f 01 3d 53 30 05 53 invlpg 0x53053053 + * x86-64: 0f 01 3d 53 30 05 53 invlpg 0x53053053(%rip)+ */
#defineRSEQ_SIG0x53053053On x86, you have to worry about what happens if control flow gets redirected to an arbitrary byte address. The proposed sequence
0f 01 3d 53 30 05 53is a trap instruction if control lands seven bytes before the beginning of the abort handler, but if it lands anywhere _else_ within the marker sequence, you get one of these instruction sequences, none of which trap, all but one of which will corrupt the process state, and three of which will consume three bytes from the beginning of the abort handler's code, continuing execution with a misaligned PC:01 3d 53 30 05 53 add %edi,0x53053053(%rip) 3d 53 30 05 53 cmp $0x53053053,%eax 53 30 05 53 XX XX XX push %rbx; xor %al,0xXXXXXX78(%rip) 30 05 53 XX XX XX xor %al,0xXXXXXX78(%rip) 05 53 XX XX XX add $0xXXXXXX53,%eax 53 push %rbxQuoted 6 lines
So I'm going to suggest instead the four-byte sequence CD CF CD CF. That's INT $0xCF if control lands either two or four bytes before the beginning of the abort handler, and IRET if it lands one or three bytes before. I believe both of these possibilities are currently also forbidden in user mode. It doesn't need to be longer, does it?
IRET works in user mode just fine. Why are you concerned about landing in the middle of the signature? A misaligned jump into code is screwy pretty much no matter what. It does seem genuinely useful to trap if you accidentally fall through to the beginning of the signature, but I don't see the point of worrying about jumping to the middle.
Quoted 4 lines
There's some argument that, for consistency with CET, the last couple bytes of the signature should match ENDBR.
Mathieu, how many bytes do we have for the signature?
The signature is 4 bytes. Those 4 bytes need to be uncommon. You can have a longer instruction than that, but then the additional bytes at the beginning of the instruction will not be part of the signature per se.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Peter Zijlstra
maintainerQuoted 5 lines · Mathieu Desnoyers
+/*
+ *RSEQ_SIGis used with the following privileged instructions, which trap in user-space:+ * x86-32: 0f 01 3d 53 30 05 53 invlpg 0x53053053 + * x86-64: 0f 01 3d 53 30 05 53 invlpg 0x53053053(%rip)+ */
Right, and the alternative is: 0f b9 3d $SIG, which decodes to:
UD1 $SIG(%rip),%edi
which will trap unconditionally. The only problem is that gas will not actually assemble it, but since we're .byte coding it, it doesn't matter.
UD1 is specified by both AMD and Intel to take a ModR/M, unlike UD0 where they disagree on the ModR/M.
Mathieu Desnoyers
maintainerQuoted 10 lines · ----- On Apr 10, 2019, at 2:54 AM, Peter Zijlstra [email protected] wrote:
Quoted 6 lines · Mathieu Desnoyers
+/*
+ *RSEQ_SIGis used with the following privileged instructions, which trap in user-space:+ * x86-32: 0f 01 3d 53 30 05 53 invlpg 0x53053053 + * x86-64: 0f 01 3d 53 30 05 53 invlpg 0x53053053(%rip)+ */
Right, and the alternative is: 0f b9 3d $SIG, which decodes to:
Quoted 8 lines
UD1 $SIG(%rip),%edi
which will trap unconditionally. The only problem is that gas will not actually assemble it, but since we're .byte coding it, it doesn't matter.
UD1 is specified by both AMD and Intel to take a ModR/M, unlike UD0 where they disagree on the ModR/M.
UD1 is even better from a code emulator perspective. It won't have to try to emulate invlpg if it sees it.
Byte coding UD1 as your example above gives the following objdump output, is it expected ?
objdump --version
GNU objdump (GNU Binutils for Debian) 2.28
x86-32:
14: 0f b9 ud1
16: 3d 53 30 05 53 cmp $0x53053053,%eax
x86-64:
b: 0f b9 ud1
d: 3d 53 30 05 53 cmp $0x53053053,%eax
Thanks!
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Peter Zijlstra
maintainerQuoted 18 lines · Mathieu Desnoyers
Quoted 16 lines · ----- On Apr 10, 2019, at 2:54 AM, Peter Zijlstra [email protected] wrote:
Quoted 6 lines · Mathieu Desnoyers
+/*
+ *RSEQ_SIGis used with the following privileged instructions, which trap in user-space:+ * x86-32: 0f 01 3d 53 30 05 53 invlpg 0x53053053 + * x86-64: 0f 01 3d 53 30 05 53 invlpg 0x53053053(%rip)+ */
Right, and the alternative is: 0f b9 3d $SIG, which decodes to:
UD1 $SIG(%rip),%edi
which will trap unconditionally. The only problem is that gas will not actually assemble it, but since we're .byte coding it, it doesn't matter.
Quoted 5 lines
Quoted 2 lines
UD1 is specified by both AMD and Intel to take a ModR/M, unlike UD0 where they disagree on the ModR/M.
UD1 is even better from a code emulator perspective. It won't have to try to emulate invlpg if it sees it.
Some emulators terminate on UD2, not aware of any special UD1 behaviour.
Quoted 8 lines
Byte coding UD1 as your example above gives the following objdump output, is it expected ?
objdump --version
GNU objdump (GNU Binutils for Debian) 2.28x86-32:
Quoted 7 lines
14: 0f b9 ud1
16: 3d 53 30 05 53 cmp $0x53053053,%eaxx86-64:
b: 0f b9 ud1
d: 3d 53 30 05 53 cmp $0x53053053,%eax
GNU objdump (GNU Binutils for Debian) 2.31.1
0f b9 3d 78 56 34 12 ud1 0x12345678(%rip),%edi
So I suppose your objdump is too old :/
Mathieu Desnoyers
maintainerQuoted 40 lines · ----- On Apr 10, 2019, at 1:57 PM, Peter Zijlstra [email protected] wrote:
Quoted 18 lines · Mathieu Desnoyers
Quoted 16 lines · ----- On Apr 10, 2019, at 2:54 AM, Peter Zijlstra [email protected] wrote:
Quoted 6 lines · Mathieu Desnoyers
+/*
+ *RSEQ_SIGis used with the following privileged instructions, which trap in user-space:+ * x86-32: 0f 01 3d 53 30 05 53 invlpg 0x53053053 + * x86-64: 0f 01 3d 53 30 05 53 invlpg 0x53053053(%rip)+ */
Right, and the alternative is: 0f b9 3d $SIG, which decodes to:
UD1 $SIG(%rip),%edi
which will trap unconditionally. The only problem is that gas will not actually assemble it, but since we're .byte coding it, it doesn't matter.
Quoted 5 lines
Quoted 2 lines
UD1 is specified by both AMD and Intel to take a ModR/M, unlike UD0 where they disagree on the ModR/M.
UD1 is even better from a code emulator perspective. It won't have to try to emulate invlpg if it sees it.
Some emulators terminate on UD2, not aware of any special UD1 behaviour.
Quoted 13 lines
Byte coding UD1 as your example above gives the following objdump output, is it expected ?
objdump --version
GNU objdump (GNU Binutils for Debian) 2.28x86-32:
14: 0f b9 ud1
16: 3d 53 30 05 53 cmp $0x53053053,%eaxx86-64:
Quoted 8 lines
Quoted 2 lines
b: 0f b9 ud1
d: 3d 53 30 05 53 cmp $0x53053053,%eaxGNU objdump (GNU Binutils for Debian) 2.31.1
0f b9 3d 78 56 34 12 ud1 0x12345678(%rip),%ediSo I suppose your objdump is too old :/
Well at least it decodes _something_ which matches the overall instruction length of 7 bytes, which I think should be OK. So let's use ud1 unless anyone objects.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com