Skip to contentSkip to conversation

x86 : Ensure X86_FLAGS_NT is cleared on syscall entry

PATCH In review Anish Bhatt 7 participants 24 messages Sep 25, 2014 – Sep 30, 2014 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86 : Ensure X86_FLAGS_NT is cleared on syscall entry

The MSR_SYSCALL_MASK, which is responsible for clearing specific EFLAGS on
syscall entry, should also clear the nested task (NT) flag to be safe from userspace injection. Without this fix the application segmentation faults on syscall return because of the changed meaning of the IRET instruction.

Further details can be seen here https://bugs.winehq.org/show_bug.cgi?id=33275

---
 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/common.c +1 -1
index e4ab2b4..3126558 100644
@@ -1184,7 +1184,7 @@ void syscall_init(void)
 	/* Flags to clear on syscall */
 	wrmsrl(MSR_SYSCALL_MASK,
 	       X86_EFLAGS_TF|X86_EFLAGS_DF|X86_EFLAGS_IF|
-	       X86_EFLAGS_IOPL|X86_EFLAGS_AC);
+	       X86_EFLAGS_IOPL|X86_EFLAGS_AC|X86_EFLAGS_NT);
 }
 
 /*
Signature · 1 line
2.1.0
#1 lore raw
Quoted 19 lines · On Thu, 25 Sep 2014 12:42:51 -0700 Anish Bhatt <[email protected]> wrote:

The MSR_SYSCALL_MASK, which is responsible for clearing specific EFLAGS on
syscall entry, should also clear the nested task (NT) flag to be safe from userspace injection. Without this fix the application segmentation faults on syscall return because of the changed meaning of the IRET instruction.

Further details can be seen here https://bugs.winehq.org/show_bug.cgi?id=33275

---
 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/common.c +0 -0
index e4ab2b4..3126558 100644

arch/x86/kernel/cpu/common.c void syscall_init(void) · line 1190

Quoted 8 lines
arch/x86/kernel/cpu/common.c +1 -1
@@ -1184,7 +1184,7 @@ void syscall_init(void)
 	/* Flags to clear on syscall */
 	wrmsrl(MSR_SYSCALL_MASK,
 	       X86_EFLAGS_TF|X86_EFLAGS_DF|X86_EFLAGS_IF|
-	       X86_EFLAGS_IOPL|X86_EFLAGS_AC);
+	       X86_EFLAGS_IOPL|X86_EFLAGS_AC|X86_EFLAGS_NT);
 }
 
 /*

To demonstrate the bug this fixes, just strace this program, adapted from the linked bug report:

#include <stdio.h>
#include <unistd.h>

int main()

{
	asm volatile("pushfq           \n\t" \
		     "pop %rax         \n\t" \
		     "or $0x4000,%rax  \n\t" \
		     "push %rax        \n\t" \
		     "popfq            \n\t");
	sleep(1);

	return 0;
}

I ran strace under gdb and found it dies here with SIGSEGV:

0x00007ffff7a505c2 <+50>: mov $0xea,%eax
0x00007ffff7a505c7 <+55>: syscall
=> 0x00007ffff7a505c9 <+57>: cmp $0xfffffffffffff000,%rax

#2 lore raw
Quoted 10 lines · Chuck Ebbert

#include <stdio.h>
#include <unistd.h>

int main()

{
	asm volatile("pushfq           \n\t" \
		     "pop %rax         \n\t" \
		     "or $0x4000,%rax  \n\t" \
		     "push %rax        \n\t" \
		     "popfq            \n\t");

You're missing clobbers here. But fixing that has no effect.

Quoted 4 lines
	sleep(1);

	return 0;
}
#3 lore raw
In reply to Anish Bhatt
Quoted 19 lines · On Thu, 25 Sep 2014 12:42:51 -0700 Anish Bhatt <[email protected]> wrote:

The MSR_SYSCALL_MASK, which is responsible for clearing specific EFLAGS on
syscall entry, should also clear the nested task (NT) flag to be safe from userspace injection. Without this fix the application segmentation faults on syscall return because of the changed meaning of the IRET instruction.

Further details can be seen here https://bugs.winehq.org/show_bug.cgi?id=33275

---
 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/common.c +0 -0
index e4ab2b4..3126558 100644

arch/x86/kernel/cpu/common.c void syscall_init(void) · line 1190

Quoted 8 lines
arch/x86/kernel/cpu/common.c +1 -1
@@ -1184,7 +1184,7 @@ void syscall_init(void)
 	/* Flags to clear on syscall */
 	wrmsrl(MSR_SYSCALL_MASK,
 	       X86_EFLAGS_TF|X86_EFLAGS_DF|X86_EFLAGS_IF|
-	       X86_EFLAGS_IOPL|X86_EFLAGS_AC);
+	       X86_EFLAGS_IOPL|X86_EFLAGS_AC|X86_EFLAGS_NT);
 }
 
 /*

I don't get it. Why isn't this patch acceptable, at least on x86-64 where NT is never valid?

Bueller?

#4 lore raw
Quoted 39 lines

-----Original Message-----
From: Chuck Ebbert [mailto:[email protected]]
Sent: Friday, September 26, 2014 3:01 PM
To: Anish Bhatt
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Linus Torvalds
Subject: Re: [PATCH] x86 : Ensure X86_FLAGS_NT is cleared on syscall entry

Quoted 27 lines · On Thu, 25 Sep 2014 12:42:51 -0700 Anish Bhatt <[email protected]> wrote:

The MSR_SYSCALL_MASK, which is responsible for clearing specific EFLAGS on syscall entry, should also clear the nested task (NT) flag to be safe from userspace injection. Without this fix the application segmentation faults on syscall return because of the changed meaning of the IRET instruction.

Further details can be seen here
https://bugs.winehq.org/show_bug.cgi?id=33275

---
 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/arch/x86/kernel/cpu/common.c index e4ab2b4..3126558 100644

arch/x86/kernel/cpu/common.c +1 -1
@@ -1184,7 +1184,7 @@ void syscall_init(void)
 	/* Flags to clear on syscall */
 	wrmsrl(MSR_SYSCALL_MASK,
 	       X86_EFLAGS_TF|X86_EFLAGS_DF|X86_EFLAGS_IF|
-	       X86_EFLAGS_IOPL|X86_EFLAGS_AC);
+	       X86_EFLAGS_IOPL|X86_EFLAGS_AC|X86_EFLAGS_NT);
 }
 
Quoted 6 lines

arch/x86/kernel/cpu/common.c void syscall_init(void) · line 1190

Quoted 1 line
arch/x86/kernel/cpu/common.c +0 -0
 /*

I don't get it. Why isn't this patch acceptable, at least on x86-64 where NT is never valid?

Bueller?

Chuck, I don't think anyone has gotten around to reviewing, accepting or
rejecting this patch yet.
-Anish

#5 lore raw
In reply to Chuck Ebbert
Quoted 3 lines · Chuck Ebbert

I don't get it. Why isn't this patch acceptable, at least on x86-64 where NT is never valid?

Why do you think it's not acceptable? Why do you raise a stink *one* day after the patch - that seems to not be very important - is sent out?

Why would SIGSEGV even be that bad? The program does crap things, why accept it silently?

I don't think the patch is necessarily wrong, but I don't see why it would be critical, and I *definitely* don't see why the f*ck you are making a big deal of it.

Go away, stop bothering people.

Linus

#6 lore raw
In reply to Anish Bhatt
Quoted 19 lines · Anish Bhatt

The MSR_SYSCALL_MASK, which is responsible for clearing specific EFLAGS on
syscall entry, should also clear the nested task (NT) flag to be safe from userspace injection. Without this fix the application segmentation faults on syscall return because of the changed meaning of the IRET instruction.

Further details can be seen here https://bugs.winehq.org/show_bug.cgi?id=33275

---
 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/common.c +0 -0
index e4ab2b4..3126558 100644

arch/x86/kernel/cpu/common.c void syscall_init(void) · line 1187

Quoted 5 lines
arch/x86/kernel/cpu/common.c +1 -1
@@ -1184,7 +1184,7 @@ void syscall_init(void)
 	/* Flags to clear on syscall */
 	wrmsrl(MSR_SYSCALL_MASK,
 	       X86_EFLAGS_TF|X86_EFLAGS_DF|X86_EFLAGS_IF|
-	       X86_EFLAGS_IOPL|X86_EFLAGS_AC);
+	       X86_EFLAGS_IOPL|X86_EFLAGS_AC|X86_EFLAGS_NT);

Something's weird here, and at the very least the changelog is insufficiently informative.

The Intel SDM says:

If the NT flag is set and the processor is in IA-32e mode, the IRET instruction causes a general protection exception.

Presumably interrupt delivery clears NT. I haven't spotted where that's documented yet.

sysret doesn't appear to care about NT at all.

So: the test code doesn't appear to do anything interesting *unless* it goes through syscall followed by the iret exit path. Then it receives #GP on return, which turns into a signal.

On the premise that the slow and fast return paths ought to be indistinguishable from userspace, I think we should fix this. But I want to understand it better first.

Also, 32-bit may need more care here.

--Andy

#7 lore raw
Quoted 29 lines · Andy Lutomirski
Quoted 19 lines · Anish Bhatt

The MSR_SYSCALL_MASK, which is responsible for clearing specific EFLAGS on
syscall entry, should also clear the nested task (NT) flag to be safe from userspace injection. Without this fix the application segmentation faults on syscall return because of the changed meaning of the IRET instruction.

Further details can be seen here https://bugs.winehq.org/show_bug.cgi?id=33275

---
 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/common.c +0 -0
index e4ab2b4..3126558 100644

arch/x86/kernel/cpu/common.c void syscall_init(void) · line 1187

Quoted 5 lines
arch/x86/kernel/cpu/common.c +1 -1
@@ -1184,7 +1184,7 @@ void syscall_init(void)
 	/* Flags to clear on syscall */
 	wrmsrl(MSR_SYSCALL_MASK,
 	       X86_EFLAGS_TF|X86_EFLAGS_DF|X86_EFLAGS_IF|
-	       X86_EFLAGS_IOPL|X86_EFLAGS_AC);
+	       X86_EFLAGS_IOPL|X86_EFLAGS_AC|X86_EFLAGS_NT);

Something's weird here, and at the very least the changelog is insufficiently informative.

Quoted 7 lines

The Intel SDM says:

If the NT flag is set and the processor is in IA-32e mode, the IRET instruction causes a general protection exception.

Presumably interrupt delivery clears NT. I haven't spotted where that's documented yet.

Well, the best documentation I've found is something like
http://www.fermimn.gov.it/linux/quarta/x86/int.htm

which states:

--- snip ---
INTERRUPT-TO-INNER-PRIVILEGE:
[...]
TF := 0;
NT := 0;
--- snip ---
(Doesn't say anything about HW interrupts though)

This also makes sense at my opinion, since the interrupt handler has to know if it should return to the previous task (when NT=1) or to the same task (when NT=0).

Quoted 6 lines

sysret doesn't appear to care about NT at all.

So: the test code doesn't appear to do anything interesting *unless* it goes through syscall followed by the iret exit path. Then it receives #GP on return, which turns into a signal.

Yep, thats also my interpretation of this issue. If the processor would be in 32-bit/protected-mode the NT flag would be interpreted as a task return, and it would probably cause a different exception, because the kernel never uses the task link property of the TSS.

Quoted 4 lines

On the premise that the slow and fast return paths ought to be indistinguishable from userspace, I think we should fix this. But I want to understand it better first.

A reliable way to force the slow return path is to use ptrace, see:
http://lxr.free-electrons.com/source/arch/x86/kernel/entry_64.S#L544

This also matches the experience: The test application only crashes with a small probability, except you use strace, then it will always crash (because the kernel forces the slow return path).

Two additional remarks:

* A reliable way to let it crash without strace, is to run the fork()/clone() syscall afterwards and compile as 32-bit.

* When you run exec*() afterwards, the crash will happen at the entry of the new executable. Doesn't matter if the target process is SUID or not. I don't see a way to exploit this issue, though, but probably some more people should take a look at it...

Quoted 2 lines

Also, 32-bit may need more care here.

That might be possible. It probably makes sense to review other parts of the code, for similar issues.

Quoted 3 lines

--Andy

Regards,
Sebastian

#8 lore raw
Quoted 81 lines · Sebastian Lackner
Quoted 29 lines · Andy Lutomirski
Quoted 19 lines · Anish Bhatt

The MSR_SYSCALL_MASK, which is responsible for clearing specific EFLAGS on
syscall entry, should also clear the nested task (NT) flag to be safe from userspace injection. Without this fix the application segmentation faults on syscall return because of the changed meaning of the IRET instruction.

Further details can be seen here https://bugs.winehq.org/show_bug.cgi?id=33275

---
 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/common.c +0 -0
index e4ab2b4..3126558 100644

arch/x86/kernel/cpu/common.c void syscall_init(void) · line 1187

Quoted 5 lines
arch/x86/kernel/cpu/common.c +1 -1
@@ -1184,7 +1184,7 @@ void syscall_init(void)
     /* Flags to clear on syscall */
     wrmsrl(MSR_SYSCALL_MASK,
            X86_EFLAGS_TF|X86_EFLAGS_DF|X86_EFLAGS_IF|
-           X86_EFLAGS_IOPL|X86_EFLAGS_AC);
+           X86_EFLAGS_IOPL|X86_EFLAGS_AC|X86_EFLAGS_NT);

Something's weird here, and at the very least the changelog is insufficiently informative.

Quoted 7 lines

The Intel SDM says:

If the NT flag is set and the processor is in IA-32e mode, the IRET instruction causes a general protection exception.

Presumably interrupt delivery clears NT. I haven't spotted where that's documented yet.

Well, the best documentation I've found is something like
http://www.fermimn.gov.it/linux/quarta/x86/int.htm

which states:

--- snip ---
INTERRUPT-TO-INNER-PRIVILEGE:

   [...]
   TF := 0;
   NT := 0;

--- snip ---
(Doesn't say anything about HW interrupts though)

This also makes sense at my opinion, since the interrupt handler has to know if it should return to the previous task (when NT=1) or to the same task (when NT=0).

Quoted 6 lines

sysret doesn't appear to care about NT at all.

So: the test code doesn't appear to do anything interesting *unless* it goes through syscall followed by the iret exit path. Then it receives #GP on return, which turns into a signal.

Yep, thats also my interpretation of this issue. If the processor would be in 32-bit/protected-mode the NT flag would be interpreted as a task return, and it would probably cause a different exception, because the kernel never uses the task link property of the TSS.

Quoted 4 lines

On the premise that the slow and fast return paths ought to be indistinguishable from userspace, I think we should fix this. But I want to understand it better first.

A reliable way to force the slow return path is to use ptrace, see:
http://lxr.free-electrons.com/source/arch/x86/kernel/entry_64.S#L544

This also matches the experience: The test application only crashes with a small probability, except you use strace, then it will always crash (because the kernel forces the slow return path).

Two additional remarks:

* A reliable way to let it crash without strace, is to run the fork()/clone() syscall afterwards and compile as 32-bit.

Quoted 8 lines

* When you run exec*() afterwards, the crash will happen at the entry of the new executable. Doesn't matter if the target process is SUID or not. I don't see a way to exploit this issue, though, but probably some more people should take a look at it...

Quoted 2 lines

Also, 32-bit may need more care here.

That might be possible. It probably makes sense to review other parts of the code, for similar issues.

sysenter probably has the same problem.

--Andy

#9 lore raw
In reply to Sebastian Lackner
Quoted 11 lines · Sebastian Lackner

On 29.09.2014 19:40, Andy Lutomirski wrote:
Well, the best documentation I've found is something like
http://www.fermimn.gov.it/linux/quarta/x86/int.htm

which states:

--- snip ---
INTERRUPT-TO-INNER-PRIVILEGE:

   [...]
   TF := 0;
   NT := 0;
Quoted 6 lines

--- snip ---
(Doesn't say anything about HW interrupts though)

This also makes sense at my opinion, since the interrupt handler has to know if it should return to the previous task (when NT=1) or to the same task (when NT=0).

No, it does not. Simply because Linux does not support nested tasks at all, because the TSS is not accessible and the TSS.back_link is sturdily NULL. So even if it would not explode with a #GP in IA-32e mode it would explode while trying to execute the instruction at NULL.

Quoted 2 lines

That might be possible. It probably makes sense to review other parts of the code, for similar issues.

What's the issue? Stupid user space programs segfaulting?

Thanks,

tglx

#10 lore raw
Quoted 15 lines · Thomas Gleixner
Quoted 14 lines · Sebastian Lackner

On 29.09.2014 19:40, Andy Lutomirski wrote:
Well, the best documentation I've found is something like
http://www.fermimn.gov.it/linux/quarta/x86/int.htm

which states:

--- snip ---
INTERRUPT-TO-INNER-PRIVILEGE:

   [...]
   TF := 0;
   NT := 0;

--- snip ---
(Doesn't say anything about HW interrupts though)

Quoted 8 lines
Quoted 3 lines

This also makes sense at my opinion, since the interrupt handler has to know if it should return to the previous task (when NT=1) or to the same task (when NT=0).

No, it does not. Simply because Linux does not support nested tasks at all, because the TSS is not accessible and the TSS.back_link is sturdily NULL. So even if it would not explode with a #GP in IA-32e mode it would explode while trying to execute the instruction at NULL.

Sure, I also mentioned this in my last mail:

Quoted 8 lines · Sebastian Lackner

NT flag would be interpreted as a task return, and it would probably cause a different exception, because the kernel never uses the task link property of the TSS.

Quoted 2 lines

That might be possible. It probably makes sense to review other parts of the code, for similar issues.

What's the issue? Stupid user space programs segfaulting?

I see several issues here:

* At first the behaviour is not consistent between several system call and return instructions. For example calling syscalls by using 'int' doesn't have this issue, as it clears the NT flag before entering kernel code. Return instructions also don't show this issue all the time, just when it hits one of the problematic pieces of code.

* The kernel might execute all kind of other code (for example inside of drivers) and start additional threads. I didn't find any good example yet, but its not that unlikely, that the exception can also happen in a completely unrelated thread, where the kernel can not just kill the corresponding usermode app...

I'm fine with all kind of solutions, either it should be allowed to set NT, or the kernel should at least throw a proper exception, so that usermode has a chance to catch and handle it. At the moment the segfault is deadly, as the segfault handler immediately segfaults again - no chance to recover from such an error.

Regards,
Sebastian

#11 lore raw
Quoted 8 lines · Sebastian Lackner

I see several issues here:

* At first the behaviour is not consistent between several system call and return instructions. For example calling syscalls by using 'int' doesn't have this issue, as it clears the NT flag before entering kernel code. Return instructions also don't show this issue all the time, just when it hits one of the problematic pieces of code.

And why should we care? The use of NT is not supported. Period. So it does not matter whether A explodes and B does not.

Quoted 5 lines

* The kernel might execute all kind of other code (for example inside of drivers) and start additional threads. I didn't find any good example yet, but its not that unlikely, that the exception can also happen in a completely unrelated thread, where the kernel can not just kill the corresponding usermode app...

The thread can execute whatever it wants in kernel context. The exception is going to hit on return to user space and not in some random kernel context. The threads it creates are going to die as well.

Quoted 5 lines

I'm fine with all kind of solutions, either it should be allowed to set NT, or the kernel should at least throw a proper exception, so that usermode has a chance to catch and handle it. At the moment the segfault is deadly, as the segfault handler immediately segfaults again - no chance to recover from such an error.

# gdb ./crap core

(gdb) info registers

eflags 0x4296 [ PF AF SF IF NT ]

Tells you very much that there is state which is not supported.

Thanks,

tglx

#12 lore raw
In reply to Andy Lutomirski
Quoted 29 lines · Andy Lutomirski
Quoted 19 lines · Anish Bhatt

The MSR_SYSCALL_MASK, which is responsible for clearing specific EFLAGS on
syscall entry, should also clear the nested task (NT) flag to be safe from userspace injection. Without this fix the application segmentation faults on syscall return because of the changed meaning of the IRET instruction.

Further details can be seen here https://bugs.winehq.org/show_bug.cgi?id=33275

---
 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/common.c +0 -0
index e4ab2b4..3126558 100644

arch/x86/kernel/cpu/common.c void syscall_init(void) · line 1187

Quoted 5 lines
arch/x86/kernel/cpu/common.c +1 -1
@@ -1184,7 +1184,7 @@ void syscall_init(void)
 	/* Flags to clear on syscall */
 	wrmsrl(MSR_SYSCALL_MASK,
 	       X86_EFLAGS_TF|X86_EFLAGS_DF|X86_EFLAGS_IF|
-	       X86_EFLAGS_IOPL|X86_EFLAGS_AC);
+	       X86_EFLAGS_IOPL|X86_EFLAGS_AC|X86_EFLAGS_NT);

Something's weird here, and at the very least the changelog is insufficiently informative.

Quoted 7 lines

The Intel SDM says:

If the NT flag is set and the processor is in IA-32e mode, the IRET instruction causes a general protection exception.

Presumably interrupt delivery clears NT. I haven't spotted where that's documented yet.

Nope, that's unrelated.

See Volume 3, chapter 7.4 "Task linking":

"The previous task link field of the TSS (sometimes called the
“backlink”) and the NT flag in the EFLAGS register are used to return execution to the previous task. EFLAGS.NT = 1 indicates that the currently executing task is nested within the execution of another task.

When a CALL instruction, an interrupt, or an exception causes a task
switch: the processor copies the segment selector for the current TSS to the previous task link field of the TSS for the new task; it then sets EFLAGS.NT = 1. If software uses an IRET instruction to suspend the new task, the processor checks for EFLAGS.NT = 1; it then uses the value in the previous task link field to return to the previous task. See Figures 7-8."

Now, Linux does not care about that. Thread management is done purely in software. So nothing uses and nothing can use the TSS backlink and NT mode.

In IA-32e mode a IRET seing EFLAGS.NT=1 will cause #GP. In non IA-32e mode it would simply explode by returning to TSS.back_link, which is reliably NULL.

So there is nothing to see here other than the stupid user space task fiddling with the NT flag being killed rightfully.

Thanks,

tglx

#13 lore raw
Quoted 60 lines · Thomas Gleixner
Quoted 29 lines · Andy Lutomirski
Quoted 19 lines · Anish Bhatt

The MSR_SYSCALL_MASK, which is responsible for clearing specific EFLAGS on
syscall entry, should also clear the nested task (NT) flag to be safe from userspace injection. Without this fix the application segmentation faults on syscall return because of the changed meaning of the IRET instruction.

Further details can be seen here https://bugs.winehq.org/show_bug.cgi?id=33275

---
 arch/x86/kernel/cpu/common.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/common.c +0 -0
index e4ab2b4..3126558 100644

arch/x86/kernel/cpu/common.c void syscall_init(void) · line 1187

Quoted 5 lines
arch/x86/kernel/cpu/common.c +1 -1
@@ -1184,7 +1184,7 @@ void syscall_init(void)
    /* Flags to clear on syscall */
    wrmsrl(MSR_SYSCALL_MASK,
           X86_EFLAGS_TF|X86_EFLAGS_DF|X86_EFLAGS_IF|
-          X86_EFLAGS_IOPL|X86_EFLAGS_AC);
+          X86_EFLAGS_IOPL|X86_EFLAGS_AC|X86_EFLAGS_NT);

Something's weird here, and at the very least the changelog is insufficiently informative.

Quoted 7 lines

The Intel SDM says:

If the NT flag is set and the processor is in IA-32e mode, the IRET instruction causes a general protection exception.

Presumably interrupt delivery clears NT. I haven't spotted where that's documented yet.

Nope, that's unrelated.

See Volume 3, chapter 7.4 "Task linking":

"The previous task link field of the TSS (sometimes called the
“backlink”) and the NT flag in the EFLAGS register are used to return execution to the previous task. EFLAGS.NT = 1 indicates that the currently executing task is nested within the execution of another task.

When a CALL instruction, an interrupt, or an exception causes a task
switch: the processor copies the segment selector for the current TSS to the previous task link field of the TSS for the new task; it then sets EFLAGS.NT = 1. If software uses an IRET instruction to suspend the new task, the processor checks for EFLAGS.NT = 1; it then uses the value in the previous task link field to return to the previous task. See Figures 7-8."

Now, Linux does not care about that. Thread management is done purely in software. So nothing uses and nothing can use the TSS backlink and NT mode.

Quoted 6 lines

In IA-32e mode a IRET seing EFLAGS.NT=1 will cause #GP. In non IA-32e mode it would simply explode by returning to TSS.back_link, which is reliably NULL.

So there is nothing to see here other than the stupid user space task fiddling with the NT flag being killed rightfully.

Except that we're exposing ourselves to security issues. I don't see any off the top of my head, but what if an unprivileged or semi-privileged process sets NT, does syscall or sysenter, and causes the kernel to jump to EFI code? Or, hell, what if there's something in the kernel that fakes interrupt delivery and blows up on return? Or what if we're running in kernel mode with NT set and we take an NMI or even a nested NMI? What happens if we have NT set in kernel mode and enter a VM?

I think it's absurd that you can set NT at all from CPL3 in long mode, but we should at least try to be graceful about it.

I don't know of any actual bugs here, but fixing this (at least for syscall) will have absolutely no performance impact.

--Andy

#14 lore raw
In reply to Thomas Gleixner
Quoted 5 lines · Thomas Gleixner
Quoted 2 lines · Andy Lutomirski

Presumably interrupt delivery clears NT. I haven't spotted where that's documented yet.

Nope, that's unrelated.

If it weren't the case, then we'd be totally screwed. Fortunately, it is. I found it: SDM Volume 3 6.12.1.2 says:

(On calls to exception and interrupt
handlers, the processor also clears the VM, RF, and NT flags in the EFLAGS register,
after they are saved on the stack.)

--Andy

#15 lore raw
Quoted 7 lines · Andy Lutomirski
Quoted 5 lines · Thomas Gleixner
Quoted 2 lines · Andy Lutomirski

Presumably interrupt delivery clears NT. I haven't spotted where that's documented yet.

Nope, that's unrelated.

Quoted 7 lines

If it weren't the case, then we'd be totally screwed. Fortunately, it is. I found it: SDM Volume 3 6.12.1.2 says:

(On calls to exception and interrupt
handlers, the processor also clears the VM, RF, and NT flags in the EFLAGS register,
after they are saved on the stack.)

Sorry, I misunderstood your question.

And yes on exception and interrupt entry it is cleared. Otherwise the whole feature would not work at all ...

But that's why I'm really not worried about it. While we can mask out the stupid bit easily, it does not provide any value except protecting silly userspace from rightfully raised exceptions.

When I first saw that patch, I was worried about the security impact, but after staring long enough at the SDM and the code, the only way it can explode is when returning to user space. It cannot explode in the kernel.

So in IA-32e it creates a #GP otherwise it falls over the return to NULL (TSS.back_link). So what?

Thanks,

tglx

#16 lore raw
Quoted 18 lines · Thomas Gleixner
Quoted 8 lines

If it weren't the case, then we'd be totally screwed. Fortunately, it is. I found it: SDM Volume 3 6.12.1.2 says:

(On calls to exception and interrupt
handlers, the processor also clears the VM, RF, and NT flags in the EFLAGS register,
after they are saved on the stack.)

Sorry, I misunderstood your question.

And yes on exception and interrupt entry it is cleared. Otherwise the whole feature would not work at all ...

But that's why I'm really not worried about it. While we can mask out the stupid bit easily, it does not provide any value except protecting silly userspace from rightfully raised exceptions.

Quoted 8 lines

When I first saw that patch, I was worried about the security impact, but after staring long enough at the SDM and the code, the only way it can explode is when returning to user space. It cannot explode in the kernel.

So in IA-32e it creates a #GP otherwise it falls over the return to NULL (TSS.back_link). So what?

How about "it's a bug, but it's not (necessarily) a security issue?"

I think we should mask the bit anyway.

-hpa

#17 lore raw
Quoted 24 lines · H. Peter Anvin
Quoted 23 lines · Thomas Gleixner
Quoted 8 lines

If it weren't the case, then we'd be totally screwed. Fortunately, it is. I found it: SDM Volume 3 6.12.1.2 says:

(On calls to exception and interrupt
handlers, the processor also clears the VM, RF, and NT flags in the EFLAGS register,
after they are saved on the stack.)

Sorry, I misunderstood your question.

And yes on exception and interrupt entry it is cleared. Otherwise the whole feature would not work at all ...

But that's why I'm really not worried about it. While we can mask out the stupid bit easily, it does not provide any value except protecting silly userspace from rightfully raised exceptions.

When I first saw that patch, I was worried about the security impact, but after staring long enough at the SDM and the code, the only way it can explode is when returning to user space. It cannot explode in the kernel.

Quoted 5 lines
Quoted 3 lines

So in IA-32e it creates a #GP otherwise it falls over the return to NULL (TSS.back_link). So what?

How about "it's a bug, but it's not (necessarily) a security issue?"

It's a bug in user space, but as I explained it is NOT a security issue because the only place it can explode is the return to the buggy user space code via IRET.

If we start to use IRET in the kernel for random crap where we should not, then its better we trap over the stupid NT bit as well.

Quoted 1 line

I think we should mask the bit anyway.

I tend to disagree. If we clear it there we need to consequentely audit ALL other possibilites and if there are any we need to clear the bit there as well. Just to make buggy user space happy?

Thanks,

tglx

#18 lore raw
Quoted 7 lines · Thomas Gleixner
Quoted 2 lines

I think we should mask the bit anyway.

I tend to disagree. If we clear it there we need to consequentely audit ALL other possibilites and if there are any we need to clear the bit there as well. Just to make buggy user space happy?

The entry options into the kernel are: interrupt/exception (already known to be OK), SYSENTER32, SYSCALL32, and SYSCALL64. It is not too much to work through those issues, I don't think.

-hpa

#19 lore raw
Quoted 4 lines · H. Peter Anvin
Quoted 3 lines · Thomas Gleixner
Quoted 2 lines

I think we should mask the bit anyway.

Quoted 8 lines
Quoted 4 lines

I tend to disagree. If we clear it there we need to consequentely audit ALL other possibilites and if there are any we need to clear the bit there as well. Just to make buggy user space happy?

The entry options into the kernel are: interrupt/exception (already known to be OK), SYSENTER32, SYSCALL32, and SYSCALL64. It is not too much to work through those issues, I don't think.

Fair enough.

I still don't see why we should care. Supporting buggy user space was never high on my list.

Thanks,

tglx

#20 lore raw
In reply to H. Peter Anvin
Quoted 9 lines · H. Peter Anvin
Quoted 7 lines · Thomas Gleixner
Quoted 2 lines

I think we should mask the bit anyway.

I tend to disagree. If we clear it there we need to consequentely audit ALL other possibilites and if there are any we need to clear the bit there as well. Just to make buggy user space happy?

Quoted 4 lines

The entry options into the kernel are: interrupt/exception (already known to be OK),

SYSENTER32,

I don't immediately see how to fix that without adding overhead. Maybe do it in CLEAR_RREGS? This won't do any good if EFI ever starts using IRET, though. We could suck it up and fix it on entry, adding maybe ten cycles (wild guess).

Quoted 1 line

SYSCALL32

This should happen for free if we fix SYSCALL64, I think.

Also, I thought entry_64.S was a mess. Eww, ia32entry.S.

I can try to write the patch later today. But I don't want to touch the actual 32-bit kernel code -- I'll stick to the 64-bit native and compat stuff, thanks.

--Andy

and SYSCALL64. It is not too

Quoted 4 lines

much to work through those issues, I don't think.

-hpa

Signature · 2 lines
Andy Lutomirski
AMA Capital Management, LLC
#21 lore raw
In reply to Thomas Gleixner
Quoted 21 lines · Thomas Gleixner
Quoted 7 lines · Andy Lutomirski
Quoted 5 lines · Thomas Gleixner
Quoted 2 lines · Andy Lutomirski

Presumably interrupt delivery clears NT. I haven't spotted where that's documented yet.

Nope, that's unrelated.

Quoted 7 lines

If it weren't the case, then we'd be totally screwed. Fortunately, it is. I found it: SDM Volume 3 6.12.1.2 says:

(On calls to exception and interrupt
handlers, the processor also clears the VM, RF, and NT flags in the EFLAGS register,
after they are saved on the stack.)

Sorry, I misunderstood your question.

And yes on exception and interrupt entry it is cleared. Otherwise the whole feature would not work at all ...

Quoted 8 lines

But that's why I'm really not worried about it. While we can mask out the stupid bit easily, it does not provide any value except protecting silly userspace from rightfully raised exceptions.

When I first saw that patch, I was worried about the security impact, but after staring long enough at the SDM and the code, the only way it can explode is when returning to user space. It cannot explode in the kernel.

This is only true as long as the only use of lret from a system call (or kernel thread started from a system call) is to return to userspace.

For example, __efi64_thunk uses lretq, so mixed-mode EFI doesn't violate this assumption, but __efi64_thunk could just as easily have used iret.

IOW, I don't think there's any vulnerability here, but this makes me nervous.

--Andy

#22 lore raw
Quoted 31 lines · Andy Lutomirski
Quoted 21 lines · Thomas Gleixner
Quoted 7 lines · Andy Lutomirski
Quoted 5 lines · Thomas Gleixner
Quoted 2 lines · Andy Lutomirski

Presumably interrupt delivery clears NT. I haven't spotted where that's documented yet.

Nope, that's unrelated.

Quoted 7 lines

If it weren't the case, then we'd be totally screwed. Fortunately, it is. I found it: SDM Volume 3 6.12.1.2 says:

(On calls to exception and interrupt
handlers, the processor also clears the VM, RF, and NT flags in the EFLAGS register,
after they are saved on the stack.)

Sorry, I misunderstood your question.

And yes on exception and interrupt entry it is cleared. Otherwise the whole feature would not work at all ...

Quoted 8 lines

But that's why I'm really not worried about it. While we can mask out the stupid bit easily, it does not provide any value except protecting silly userspace from rightfully raised exceptions.

When I first saw that patch, I was worried about the security impact, but after staring long enough at the SDM and the code, the only way it can explode is when returning to user space. It cannot explode in the kernel.

Quoted 7 lines

This is only true as long as the only use of lret from a system call (or kernel thread started from a system call) is to return to userspace.

For example, __efi64_thunk uses lretq, so mixed-mode EFI doesn't violate this assumption, but __efi64_thunk could just as easily have used iret.

And if __efi64_thunk would use iret, it would be wrong to begin with, really. I'd rather see it die right there.

Quoted 2 lines

IOW, I don't think there's any vulnerability here, but this makes me nervous.

I was pretty relaxed until you mentioned EFI ....

Thanks,

tglx

#23 lore raw
Quoted 40 lines · Thomas Gleixner
Quoted 31 lines · Andy Lutomirski
Quoted 21 lines · Thomas Gleixner
Quoted 7 lines · Andy Lutomirski
Quoted 5 lines · Thomas Gleixner
Quoted 2 lines · Andy Lutomirski

Presumably interrupt delivery clears NT. I haven't spotted where that's documented yet.

Nope, that's unrelated.

Quoted 7 lines

If it weren't the case, then we'd be totally screwed. Fortunately, it is. I found it: SDM Volume 3 6.12.1.2 says:

(On calls to exception and interrupt
handlers, the processor also clears the VM, RF, and NT flags in the EFLAGS register,
after they are saved on the stack.)

Sorry, I misunderstood your question.

And yes on exception and interrupt entry it is cleared. Otherwise the whole feature would not work at all ...

Quoted 8 lines

But that's why I'm really not worried about it. While we can mask out the stupid bit easily, it does not provide any value except protecting silly userspace from rightfully raised exceptions.

When I first saw that patch, I was worried about the security impact, but after staring long enough at the SDM and the code, the only way it can explode is when returning to user space. It cannot explode in the kernel.

Quoted 7 lines

This is only true as long as the only use of lret from a system call (or kernel thread started from a system call) is to return to userspace.

For example, __efi64_thunk uses lretq, so mixed-mode EFI doesn't violate this assumption, but __efi64_thunk could just as easily have used iret.

Quoted 7 lines

And if __efi64_thunk would use iret, it would be wrong to begin with, really. I'd rather see it die right there.

Quoted 2 lines

IOW, I don't think there's any vulnerability here, but this makes me nervous.

I was pretty relaxed until you mentioned EFI ....

:)

I have a patch that seems to work. It won't have any effect at all on syscall performance (32- or 64-bit), but it slows down sysenter by 15 cycles or so. On the other hand, if we did this, then we could possible stop saving and restoring RFLAGS in switch_to, which might be worthwhile, since 64-bit code ought to be more common than 32-bit these days.

See also:
https://lkml.org/lkml/2006/9/18/161

--Andy

Quoted 4 lines

Thanks,

tglx

Signature · 2 lines
Andy Lutomirski
AMA Capital Management, LLC