Skip to contentSkip to conversation

perf test fails on AMD CPUs

BUG/RFC Jiri Olsa 4 participants 14 messages Aug 16, 2015 – Dec 10, 2015

Subsystems (3)
lore
Share & tools
lore

Jiri Olsa

author
#0 lore raw

hi,
'perf test 18' is failing on systems with AMD processor.

The only reason I could find is that AMD does not set 'resume flag' in RFLAGS register the way the Intel CPU does.

(simplified) test scenario:

- create breakpoint (on test_function) perf event with SIGIO signal to be delivered any time the breakpoint is hit
- run test_function

expected course of actions is:
1) CPU hits 'test_function'
2) DB exception is triggered, with RFLAGS.RF=0
3) DB exception handler sets regs->RFLAGS.RF=1 and perf handler triggers irq_work pending work
4) DB exception executes iretd
5) irq_work interrupt is triggered, with RFLAGS.RF=1
6) irq_work interrupt calls kill_fasync with SIGIO signal
7) irq_work interrupt on return to userspace calls prepare_exit_to_usermode which actually delivers the SIGIO signal
8) sigreturn syscall prepare registers to return to the instruction from step 1) and sets RFLAGS.RF to the its original value from step 5) (RFLAGS.RF=1)
9) CPU hits 'test_function' and DB exception is NOT triggered due to RFLAGS.RF=1

this is how I see it works on Intel

But AMD gives me RFLAGS.RF=0 on step 5, which makes the step 9 to trigger the DB exception once again and makes the test fail.

I'm not sure this test ever worked on AMD CPUs, anyway is there anything I'm missing or is this some AMD/Intel quirk?

thanks,
jirka

AMD description of RF flag (SDM 3.1.6):
=======================================
Resume Flag (RF) Bit. Bit 16. The RF bit allows an instruction to be restarted following an instruction breakpoint resulting in a debug exception (#DB). This bit prevents multiple debug exceptions from occurring on the same instruction.
The processor clears the RF bit after every instruction is successfully executed, except when the instruction is:
•
•
An IRET that sets the RF bit.
JMP, CALL, or INTn through a task gate.
In both of the above cases, RF is not cleared to 0 until the next instruction successfully executes. When an exception occurs (or when a string instruction is interrupted), the processor normally sets RF=1 in the RFLAGS image saved on the interrupt stack. However, when a #DB exception occurs as a result of an instruction breakpoint, the processor clears the RF bit to 0 in the interrupt-stack RFLAGS image.
For instruction restart to work properly following an instruction breakpoint, the #DB exception handler must set RF to 1 in the interrupt-stack RFLAGS image. When an IRET is later executed to return to the instruction that caused the instruction-breakpoint #DB exception, the set RF bit (RF=1) is loaded from the interrupt-stack RFLAGS image. RF is not cleared by the processor until the instruction causing the #DB exception successfully executes.

Intel description of RF flag (SDM 17.3.1.1):
============================================
Because the debug exception for an instruction breakpoint is generated before the instruction is executed, if the instruction breakpoint is not removed by the exception handler; the processor will detect the instruction breakpoint again when the instruction is restarted and generate another debug exception. To prevent looping on an instruction breakpoint, the Intel 64 and IA-32 architectures provide the RF flag (resume flag) in the EFLAGS register (see Section 2.3, “System Flags and Fields in the EFLAGS Register,” in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A). When the RF flag is set, the processor ignores instruction breakpoints. All Intel 64 and IA-32 processors manage the RF flag as follows. The RF Flag is cleared at the start of the instruction after the check for code breakpoint, CS limit violation and FP exceptions. Task Switches and IRETD/IRETQ instruc- tions transfer the RF image from the TSS/stack to the EFLAGS register.
When calling an event handler, Intel 64 and IA-32 processors establish the value of the RF flag in the EFLAGS image pushed on the stack:
• For any fault-class exception except a debug exception generated in response to an instruction breakpoint, the value pushed for RF is 1.
• For any interrupt arriving after any iteration of a repeated string instruction but the last iteration, the value pushed for RF is 1.
• For any trap-class exception generated by any iteration of a repeated string instruction but the last iteration, the value pushed for RF is 1.
• For other cases, the value pushed for RF is the value that was in EFLAG.RF at the time the event handler was called. This includes:
— Debug exceptions generated in response to instruction breakpoints
— Hardware-generated interrupts arriving between instructions (including those arriving after the last iteration of a repeated string instruction)
— Trap-class exceptions generated after an instruction completes (including those generated after the last iteration of a repeated string instruction)
— Software-generated interrupts (RF is pushed as 0, since it was cleared at the start of the software interrupt) As noted above, the processor does not set the RF flag prior to calling the debug exception handler for debug exceptions resulting from instruction breakpoints. The debug exception handler can prevent recurrence of the instruction breakpoint by setting the RF flag in the EFLAGS image on the stack. If the RF flag in the EFLAGS image 17-8 Vol. 3BDEBUG, BRANCH PROFILE, TSC, AND RESOURCE MONITORING FEATURES
is set when the processor returns from the exception handler, it is copied into the RF flag in the EFLAGS register by IRETD/IRETQ or a task switch that causes the return. The processor then ignores instruction breakpoints for the duration of the next instruction. (Note that the POPF, POPFD, and IRET instructions do not transfer the RF image into the EFLAGS register.) Setting the RF flag does not prevent other types of debug-exception conditions (such as, I/O or data breakpoints) from being detected, nor does it prevent non-debug exceptions from being generated. For the Pentium processor, when an instruction breakpoint coincides with another fault-type exception (such as a page fault), the processor may generate one spurious debug exception after the second exception has been handled, even though the debug exception handler set the RF flag in the EFLAGS image. To prevent a spurious exception with Pentium processors, all fault-class exception handlers should set the RF flag in the EFLAGS image.

#1 lore raw
Quoted 2 lines · Jiri Olsa

hi,
'perf test 18' is failing on systems with AMD processor.

Hmm, still using that b0rked test box? :-)

Also, which kernel?

There have been substantial changes to the entry code recently. Although I don't see anything being done differently on AMD there except X86_BUG_SYSRET_SS_ATTRS but that should be unrelated.

Quoted 26 lines

The only reason I could find is that AMD does not set 'resume flag' in RFLAGS register the way the Intel CPU does.

(simplified) test scenario:

- create breakpoint (on test_function) perf event with SIGIO signal to be delivered any time the breakpoint is hit
- run test_function

expected course of actions is:
1) CPU hits 'test_function'
2) DB exception is triggered, with RFLAGS.RF=0
3) DB exception handler sets regs->RFLAGS.RF=1 and perf handler triggers irq_work pending work
4) DB exception executes iretd
5) irq_work interrupt is triggered, with RFLAGS.RF=1
6) irq_work interrupt calls kill_fasync with SIGIO signal
7) irq_work interrupt on return to userspace calls prepare_exit_to_usermode which actually delivers the SIGIO signal
8) sigreturn syscall prepare registers to return to the instruction from step 1) and sets RFLAGS.RF to the its original value from step 5) (RFLAGS.RF=1)
9) CPU hits 'test_function' and DB exception is NOT triggered due to RFLAGS.RF=1

Quoted 4 lines

this is how I see it works on Intel

But AMD gives me RFLAGS.RF=0 on step 5, which makes the step 9 to trigger the DB exception once again and makes the test fail.

Adding Andy, he might have an idea. Leaving in the rest for reference.

Quoted 70 lines

I'm not sure this test ever worked on AMD CPUs, anyway is there anything I'm missing or is this some AMD/Intel quirk?

thanks,
jirka

AMD description of RF flag (SDM 3.1.6):
=======================================
Resume Flag (RF) Bit. Bit 16. The RF bit allows an instruction to be restarted following an instruction breakpoint resulting in a debug exception (#DB). This bit prevents multiple debug exceptions from occurring on the same instruction.
The processor clears the RF bit after every instruction is successfully executed, except when the instruction is:
•
•
An IRET that sets the RF bit.
JMP, CALL, or INTn through a task gate.
In both of the above cases, RF is not cleared to 0 until the next instruction successfully executes. When an exception occurs (or when a string instruction is interrupted), the processor normally sets RF=1 in the RFLAGS image saved on the interrupt stack. However, when a #DB exception occurs as a result of an instruction breakpoint, the processor clears the RF bit to 0 in the interrupt-stack RFLAGS image.
For instruction restart to work properly following an instruction breakpoint, the #DB exception handler must set RF to 1 in the interrupt-stack RFLAGS image. When an IRET is later executed to return to the instruction that caused the instruction-breakpoint #DB exception, the set RF bit (RF=1) is loaded from the interrupt-stack RFLAGS image. RF is not cleared by the processor until the instruction causing the #DB exception successfully executes.

Intel description of RF flag (SDM 17.3.1.1):
============================================
Because the debug exception for an instruction breakpoint is generated before the instruction is executed, if the instruction breakpoint is not removed by the exception handler; the processor will detect the instruction breakpoint again when the instruction is restarted and generate another debug exception. To prevent looping on an instruction breakpoint, the Intel 64 and IA-32 architectures provide the RF flag (resume flag) in the EFLAGS register (see Section 2.3, “System Flags and Fields in the EFLAGS Register,” in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A). When the RF flag is set, the processor ignores instruction breakpoints. All Intel 64 and IA-32 processors manage the RF flag as follows. The RF Flag is cleared at the start of the instruction after the check for code breakpoint, CS limit violation and FP exceptions. Task Switches and IRETD/IRETQ instruc- tions transfer the RF image from the TSS/stack to the EFLAGS register.
When calling an event handler, Intel 64 and IA-32 processors establish the value of the RF flag in the EFLAGS image pushed on the stack:
• For any fault-class exception except a debug exception generated in response to an instruction breakpoint, the value pushed for RF is 1.
• For any interrupt arriving after any iteration of a repeated string instruction but the last iteration, the value pushed for RF is 1.
• For any trap-class exception generated by any iteration of a repeated string instruction but the last iteration, the value pushed for RF is 1.
• For other cases, the value pushed for RF is the value that was in EFLAG.RF at the time the event handler was called. This includes:
— Debug exceptions generated in response to instruction breakpoints
— Hardware-generated interrupts arriving between instructions (including those arriving after the last iteration of a repeated string instruction)
— Trap-class exceptions generated after an instruction completes (including those generated after the last iteration of a repeated string instruction)
— Software-generated interrupts (RF is pushed as 0, since it was cleared at the start of the software interrupt) As noted above, the processor does not set the RF flag prior to calling the debug exception handler for debug exceptions resulting from instruction breakpoints. The debug exception handler can prevent recurrence of the instruction breakpoint by setting the RF flag in the EFLAGS image on the stack. If the RF flag in the EFLAGS image 17-8 Vol. 3BDEBUG, BRANCH PROFILE, TSC, AND RESOURCE MONITORING FEATURES
is set when the processor returns from the exception handler, it is copied into the RF flag in the EFLAGS register by IRETD/IRETQ or a task switch that causes the return. The processor then ignores instruction breakpoints for the duration of the next instruction. (Note that the POPF, POPFD, and IRET instructions do not transfer the RF image into the EFLAGS register.) Setting the RF flag does not prevent other types of debug-exception conditions (such as, I/O or data breakpoints) from being detected, nor does it prevent non-debug exceptions from being generated. For the Pentium processor, when an instruction breakpoint coincides with another fault-type exception (such as a page fault), the processor may generate one spurious debug exception after the second exception has been handled, even though the debug exception handler set the RF flag in the EFLAGS image. To prevent a spurious exception with Pentium processors, all fault-class exception handlers should set the RF flag in the EFLAGS image.

Signature · 7 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
author
#2 lore raw
Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Jiri Olsa

hi,
'perf test 18' is failing on systems with AMD processor.

Hmm, still using that b0rked test box? :-)

heh, nope.. have seen this on at least 2 boxes so far

Quoted 2 lines

Also, which kernel?

oops, it's latest tip master

4fa1239db610 Merge branch 'ras/core'

jirka

Quoted 117 lines

There have been substantial changes to the entry code recently. Although I don't see anything being done differently on AMD there except X86_BUG_SYSRET_SS_ATTRS but that should be unrelated.

Quoted 26 lines

The only reason I could find is that AMD does not set 'resume flag' in RFLAGS register the way the Intel CPU does.

(simplified) test scenario:

- create breakpoint (on test_function) perf event with SIGIO signal to be delivered any time the breakpoint is hit
- run test_function

expected course of actions is:
1) CPU hits 'test_function'
2) DB exception is triggered, with RFLAGS.RF=0
3) DB exception handler sets regs->RFLAGS.RF=1 and perf handler triggers irq_work pending work
4) DB exception executes iretd
5) irq_work interrupt is triggered, with RFLAGS.RF=1
6) irq_work interrupt calls kill_fasync with SIGIO signal
7) irq_work interrupt on return to userspace calls prepare_exit_to_usermode which actually delivers the SIGIO signal
8) sigreturn syscall prepare registers to return to the instruction from step 1) and sets RFLAGS.RF to the its original value from step 5) (RFLAGS.RF=1)
9) CPU hits 'test_function' and DB exception is NOT triggered due to RFLAGS.RF=1

Quoted 4 lines

this is how I see it works on Intel

But AMD gives me RFLAGS.RF=0 on step 5, which makes the step 9 to trigger the DB exception once again and makes the test fail.

Adding Andy, he might have an idea. Leaving in the rest for reference.

Quoted 70 lines

I'm not sure this test ever worked on AMD CPUs, anyway is there anything I'm missing or is this some AMD/Intel quirk?

thanks,
jirka

AMD description of RF flag (SDM 3.1.6):
=======================================
Resume Flag (RF) Bit. Bit 16. The RF bit allows an instruction to be restarted following an instruction breakpoint resulting in a debug exception (#DB). This bit prevents multiple debug exceptions from occurring on the same instruction.
The processor clears the RF bit after every instruction is successfully executed, except when the instruction is:
•
•
An IRET that sets the RF bit.
JMP, CALL, or INTn through a task gate.
In both of the above cases, RF is not cleared to 0 until the next instruction successfully executes. When an exception occurs (or when a string instruction is interrupted), the processor normally sets RF=1 in the RFLAGS image saved on the interrupt stack. However, when a #DB exception occurs as a result of an instruction breakpoint, the processor clears the RF bit to 0 in the interrupt-stack RFLAGS image.
For instruction restart to work properly following an instruction breakpoint, the #DB exception handler must set RF to 1 in the interrupt-stack RFLAGS image. When an IRET is later executed to return to the instruction that caused the instruction-breakpoint #DB exception, the set RF bit (RF=1) is loaded from the interrupt-stack RFLAGS image. RF is not cleared by the processor until the instruction causing the #DB exception successfully executes.

Intel description of RF flag (SDM 17.3.1.1):
============================================
Because the debug exception for an instruction breakpoint is generated before the instruction is executed, if the instruction breakpoint is not removed by the exception handler; the processor will detect the instruction breakpoint again when the instruction is restarted and generate another debug exception. To prevent looping on an instruction breakpoint, the Intel 64 and IA-32 architectures provide the RF flag (resume flag) in the EFLAGS register (see Section 2.3, “System Flags and Fields in the EFLAGS Register,” in the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A). When the RF flag is set, the processor ignores instruction breakpoints. All Intel 64 and IA-32 processors manage the RF flag as follows. The RF Flag is cleared at the start of the instruction after the check for code breakpoint, CS limit violation and FP exceptions. Task Switches and IRETD/IRETQ instruc- tions transfer the RF image from the TSS/stack to the EFLAGS register.
When calling an event handler, Intel 64 and IA-32 processors establish the value of the RF flag in the EFLAGS image pushed on the stack:
• For any fault-class exception except a debug exception generated in response to an instruction breakpoint, the value pushed for RF is 1.
• For any interrupt arriving after any iteration of a repeated string instruction but the last iteration, the value pushed for RF is 1.
• For any trap-class exception generated by any iteration of a repeated string instruction but the last iteration, the value pushed for RF is 1.
• For other cases, the value pushed for RF is the value that was in EFLAG.RF at the time the event handler was called. This includes:
— Debug exceptions generated in response to instruction breakpoints
— Hardware-generated interrupts arriving between instructions (including those arriving after the last iteration of a repeated string instruction)
— Trap-class exceptions generated after an instruction completes (including those generated after the last iteration of a repeated string instruction)
— Software-generated interrupts (RF is pushed as 0, since it was cleared at the start of the software interrupt) As noted above, the processor does not set the RF flag prior to calling the debug exception handler for debug exceptions resulting from instruction breakpoints. The debug exception handler can prevent recurrence of the instruction breakpoint by setting the RF flag in the EFLAGS image on the stack. If the RF flag in the EFLAGS image 17-8 Vol. 3BDEBUG, BRANCH PROFILE, TSC, AND RESOURCE MONITORING FEATURES
is set when the processor returns from the exception handler, it is copied into the RF flag in the EFLAGS register by IRETD/IRETQ or a task switch that causes the return. The processor then ignores instruction breakpoints for the duration of the next instruction. (Note that the POPF, POPFD, and IRET instructions do not transfer the RF image into the EFLAGS register.) Setting the RF flag does not prevent other types of debug-exception conditions (such as, I/O or data breakpoints) from being detected, nor does it prevent non-debug exceptions from being generated. For the Pentium processor, when an instruction breakpoint coincides with another fault-type exception (such as a page fault), the processor may generate one spurious debug exception after the second exception has been handled, even though the debug exception handler set the RF flag in the EFLAGS image. To prevent a spurious exception with Pentium processors, all fault-class exception handlers should set the RF flag in the EFLAGS image.

Signature · 7 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
#3 lore raw
In reply to Borislav Petkov
Quoted 38 lines · Borislav Petkov
Quoted 2 lines · Jiri Olsa

hi,
'perf test 18' is failing on systems with AMD processor.

Hmm, still using that b0rked test box? :-)

Also, which kernel?

There have been substantial changes to the entry code recently. Although I don't see anything being done differently on AMD there except X86_BUG_SYSRET_SS_ATTRS but that should be unrelated.

Quoted 26 lines

The only reason I could find is that AMD does not set 'resume flag' in RFLAGS register the way the Intel CPU does.

(simplified) test scenario:

- create breakpoint (on test_function) perf event with SIGIO signal to be delivered any time the breakpoint is hit
- run test_function

expected course of actions is:
1) CPU hits 'test_function'
2) DB exception is triggered, with RFLAGS.RF=0
3) DB exception handler sets regs->RFLAGS.RF=1 and perf handler triggers irq_work pending work
4) DB exception executes iretd
5) irq_work interrupt is triggered, with RFLAGS.RF=1
6) irq_work interrupt calls kill_fasync with SIGIO signal
7) irq_work interrupt on return to userspace calls prepare_exit_to_usermode which actually delivers the SIGIO signal
8) sigreturn syscall prepare registers to return to the instruction from step 1) and sets RFLAGS.RF to the its original value from step 5) (RFLAGS.RF=1)
9) CPU hits 'test_function' and DB exception is NOT triggered due to RFLAGS.RF=1

Quoted 6 lines
Quoted 4 lines

this is how I see it works on Intel

But AMD gives me RFLAGS.RF=0 on step 5, which makes the step 9 to trigger the DB exception once again and makes the test fail.

Adding Andy, he might have an idea. Leaving in the rest for reference.

Gee thanks :-p

Jiri, did you instrument the code and observe do_IRQ sees RF clear in its pt_regs? Also, it might be worth checking that regs->ip in the irq_work matches regs->ip.

It's *possible* that I messed up and broke RF restore with opportunistic sysret, but the code looks correct:

        testq   $(X86_EFLAGS_RF|X86_EFLAGS_TF), %r11
        jnz     opportunistic_sysret_failed
Quoted 9 lines
Quoted 9 lines

I'm not sure this test ever worked on AMD CPUs, anyway is there anything I'm missing or is this some AMD/Intel quirk?

thanks,
jirka

Quoted 16 lines
Quoted 8 lines

AMD description of RF flag (SDM 3.1.6):
=======================================
Resume Flag (RF) Bit. Bit 16. The RF bit allows an instruction to be restarted following an instruction breakpoint resulting in a debug exception (#DB). This bit prevents multiple debug exceptions from occurring on the same instruction.
The processor clears the RF bit after every instruction is successfully executed, except when the instruction is:
•

Quoted 8 lines

•
An IRET that sets the RF bit.
JMP, CALL, or INTn through a task gate.
In both of the above cases, RF is not cleared to 0 until the next instruction successfully executes. When an exception occurs (or when a string instruction is interrupted), the processor normally sets RF=1 in the RFLAGS image saved on the interrupt stack. However, when a #DB exception occurs as a result of an instruction breakpoint, the processor clears the RF bit to 0 in the interrupt-stack RFLAGS image.

That's a little weird, I think. Shouldn't RF be zero on #DB due to a *watchpoint* so that a watchpoint followed immediately by a breakpoint works?

Quoted 5 lines
Quoted 5 lines

• For other cases, the value pushed for RF is the value that was in EFLAG.RF at the time the event handler was called. This includes:
— Debug exceptions generated in response to instruction breakpoints
— Hardware-generated interrupts arriving between instructions (including those arriving after the last iteration of a repeated string instruction)

This appears to be why it works on Intel. Does AMD not do that? We could probably work around this in software (by not using irq work for this), but yuck.

--Andy

#4 lore raw
Quoted 16 lines · Andy Lutomirski
Quoted 16 lines
Quoted 16 lines

expected course of actions is:
1) CPU hits 'test_function'
2) DB exception is triggered, with RFLAGS.RF=0
3) DB exception handler sets regs->RFLAGS.RF=1 and perf handler triggers irq_work pending work
4) DB exception executes iretd
5) irq_work interrupt is triggered, with RFLAGS.RF=1
6) irq_work interrupt calls kill_fasync with SIGIO signal
7) irq_work interrupt on return to userspace calls prepare_exit_to_usermode which actually delivers the SIGIO signal
8) sigreturn syscall prepare registers to return to the instruction from step 1) and sets RFLAGS.RF to the its original value from step 5) (RFLAGS.RF=1)
9) CPU hits 'test_function' and DB exception is NOT triggered due to RFLAGS.RF=1

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

this is how I see it works on Intel

But AMD gives me RFLAGS.RF=0 on step 5, which makes the step 9 to trigger the DB exception once again and makes the test fail.

Waaaiit a minute!

APM says #DB exception handler must set RF in the EFLAGS image on the exception stack (or whereever it is running) so that the breakpoint doesn't trigger again.

Now: do_debug() *doesn't* do that but hw_breakpoint_handler() does. So do we call hw_breakpoint_handler() in those steps above?

Because if we don't, that could explain the issue...

Quoted 3 lines
Quoted 1 line

Adding Andy, he might have an idea. Leaving in the rest for reference.

Gee thanks :-p

For what, adding you to CC or leaving in the rest? :-P

Quoted 3 lines

Jiri, did you instrument the code and observe do_IRQ sees RF clear in its pt_regs? Also, it might be worth checking that regs->ip in the irq_work matches regs->ip.

Hohumm.

Quoted 5 lines

It's *possible* that I messed up and broke RF restore with opportunistic sysret, but the code looks correct:

        testq   $(X86_EFLAGS_RF|X86_EFLAGS_TF), %r11
        jnz     opportunistic_sysret_failed

Yeah, I was looking at that too.

Quoted 11 lines
Quoted 7 lines
Quoted 7 lines

An IRET that sets the RF bit.
JMP, CALL, or INTn through a task gate.
In both of the above cases, RF is not cleared to 0 until the next instruction successfully executes. When an exception occurs (or when a string instruction is interrupted), the processor normally sets RF=1 in the RFLAGS image saved on the interrupt stack. However, when a #DB exception occurs as a result of an instruction breakpoint, the processor clears the RF bit to 0 in the interrupt-stack RFLAGS image.

That's a little weird, I think. Shouldn't RF be zero on #DB due to a *watchpoint* so that a watchpoint followed immediately by a breakpoint works?

What is a watchpoint? R/Wn bit = 1?

Btw, that sounds weird - why would the #DB exception clear RF just so that the #DB handler to set it right after... I'm probably missing something obvious.

Quoted 9 lines
Quoted 5 lines
Quoted 5 lines

• For other cases, the value pushed for RF is the value that was in EFLAG.RF at the time the event handler was called. This includes:
— Debug exceptions generated in response to instruction breakpoints
— Hardware-generated interrupts arriving between instructions (including those arriving after the last iteration of a repeated string instruction)

This appears to be why it works on Intel. Does AMD not do that? We could probably work around this in software (by not using irq work for this), but yuck.

See above.

Signature · 7 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
author
#5 lore raw
In reply to Andy Lutomirski
Quoted 44 lines · Andy Lutomirski
Quoted 43 lines · Borislav Petkov
Quoted 2 lines · Jiri Olsa

hi,
'perf test 18' is failing on systems with AMD processor.

Hmm, still using that b0rked test box? :-)

Also, which kernel?

There have been substantial changes to the entry code recently. Although I don't see anything being done differently on AMD there except X86_BUG_SYSRET_SS_ATTRS but that should be unrelated.

Quoted 30 lines

The only reason I could find is that AMD does not set 'resume flag' in RFLAGS register the way the Intel CPU does.

(simplified) test scenario:

- create breakpoint (on test_function) perf event with SIGIO signal to be delivered any time the breakpoint is hit
- run test_function

expected course of actions is:
1) CPU hits 'test_function'
2) DB exception is triggered, with RFLAGS.RF=0
3) DB exception handler sets regs->RFLAGS.RF=1 and perf handler triggers irq_work pending work
4) DB exception executes iretd
5) irq_work interrupt is triggered, with RFLAGS.RF=1
6) irq_work interrupt calls kill_fasync with SIGIO signal
7) irq_work interrupt on return to userspace calls prepare_exit_to_usermode which actually delivers the SIGIO signal
8) sigreturn syscall prepare registers to return to the instruction from step 1) and sets RFLAGS.RF to the its original value from step 5) (RFLAGS.RF=1)
9) CPU hits 'test_function' and DB exception is NOT triggered due to RFLAGS.RF=1

this is how I see it works on Intel

But AMD gives me RFLAGS.RF=0 on step 5, which makes the step 9 to trigger the DB exception once again and makes the test fail.

Quoted 7 lines
Quoted 1 line

Adding Andy, he might have an idea. Leaving in the rest for reference.

Gee thanks :-p

Jiri, did you instrument the code and observe do_IRQ sees RF clear in its pt_regs? Also, it might be worth checking that regs->ip in the irq_work matches regs->ip.

yep, thats what I saw.. once irq_work interrupt was triggered the regs->ip was same as for the previous debug exception but the RFLAGS.RF was 0

Quoted 6 lines

It's *possible* that I messed up and broke RF restore with opportunistic sysret, but the code looks correct:

        testq   $(X86_EFLAGS_RF|X86_EFLAGS_TF), %r11
        jnz     opportunistic_sysret_failed

AFAICS the problematic paths did not hit syscalls

buuuuuut anyway, it looks like latest AMD firmware issue:

[root@amd-pike-07 ~]# cat /sys/devices/system/cpu/cpu0/microcode/version 0x6000822
[root@amd-pike-07 perf]# ./perf test 18
18: Test breakpoint overflow signal handler : Ok

[root@amd-pike-07 perf]# cat /sys/devices/system/cpu/cpu0/microcode/version 0x6000832
[root@amd-pike-07 perf]# ./perf test 18
18: Test breakpoint overflow signal handler : FAILED!

[root@amd-pike-07 ~]# cat /proc/cpuinfo
processor : 7
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD Opteron(tm) Processor 3380
stepping : 0
microcode : 0x6000832

SNIP

Quoted 17 lines
Quoted 8 lines
Quoted 8 lines

AMD description of RF flag (SDM 3.1.6):
=======================================
Resume Flag (RF) Bit. Bit 16. The RF bit allows an instruction to be restarted following an instruction breakpoint resulting in a debug exception (#DB). This bit prevents multiple debug exceptions from occurring on the same instruction.
The processor clears the RF bit after every instruction is successfully executed, except when the instruction is:
•

Quoted 8 lines
Quoted 8 lines

•
An IRET that sets the RF bit.
JMP, CALL, or INTn through a task gate.
In both of the above cases, RF is not cleared to 0 until the next instruction successfully executes. When an exception occurs (or when a string instruction is interrupted), the processor normally sets RF=1 in the RFLAGS image saved on the interrupt stack. However, when a #DB exception occurs as a result of an instruction breakpoint, the processor clears the RF bit to 0 in the interrupt-stack RFLAGS image.

Quoted 3 lines

That's a little weird, I think. Shouldn't RF be zero on #DB due to a *watchpoint* so that a watchpoint followed immediately by a breakpoint works?

the AMD description looked to be more vague (compared to Intels)

Quoted 10 lines
Quoted 6 lines
Quoted 6 lines

• For other cases, the value pushed for RF is the value that was in EFLAG.RF at the time the event handler was called. This includes:
— Debug exceptions generated in response to instruction breakpoints
— Hardware-generated interrupts arriving between instructions (including those arriving after the last iteration of a repeated string instruction)

This appears to be why it works on Intel. Does AMD not do that? We could probably work around this in software (by not using irq work for this), but yuck.

yep, but hopefuly it's the issue microcode ;-) Cc-ing guys from linux-firmware git

Sherry, Suravee, any idea?

thanks,
jirka

#6 lore raw
Quoted 22 lines · Jiri Olsa

buuuuuut anyway, it looks like latest AMD firmware issue:

[root@amd-pike-07 ~]# cat /sys/devices/system/cpu/cpu0/microcode/version 0x6000822
[root@amd-pike-07 perf]# ./perf test 18
18: Test breakpoint overflow signal handler : Ok

[root@amd-pike-07 perf]# cat /sys/devices/system/cpu/cpu0/microcode/version 0x6000832
[root@amd-pike-07 perf]# ./perf test 18
18: Test breakpoint overflow signal handler : FAILED!

[root@amd-pike-07 ~]# cat /proc/cpuinfo
processor : 7
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD Opteron(tm) Processor 3380
stepping : 0
microcode : 0x6000832

Quoted 1 line

SNIP

Whoops.

Can you please confirm with your debugging code that with version 0x6000822 EFLAGS.RF is set and with 0x6000832 it isn't when running the aforementioned test?

Thanks.

Signature · 7 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
author
#7 lore raw
Quoted 23 lines · Borislav Petkov
Quoted 22 lines · Jiri Olsa

buuuuuut anyway, it looks like latest AMD firmware issue:

[root@amd-pike-07 ~]# cat /sys/devices/system/cpu/cpu0/microcode/version 0x6000822
[root@amd-pike-07 perf]# ./perf test 18
18: Test breakpoint overflow signal handler : Ok

[root@amd-pike-07 perf]# cat /sys/devices/system/cpu/cpu0/microcode/version 0x6000832
[root@amd-pike-07 perf]# ./perf test 18
18: Test breakpoint overflow signal handler : FAILED!

[root@amd-pike-07 ~]# cat /proc/cpuinfo
processor : 7
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD Opteron(tm) Processor 3380
stepping : 0
microcode : 0x6000832

Quoted 8 lines
Quoted 1 line

SNIP

Whoops.

Can you please confirm with your debugging code that with version 0x6000822 EFLAGS.RF is set and with 0x6000832 it isn't when running the aforementioned test?

please check the attached patch (over current tip/master a987577)

this is the perf breakpoint address:
000000000045b260 <test_function>:

this is trace_printk output for NEW microcode 0x6000832:

DEBUG EX

            perf-893   [003] d...  1358.053633: sync_regs: sync_regs eregs ffff88012ecc7f58, regs ffff8800c9f1bf58
            perf-893   [003] d...  1358.053635: do_debug: do_debug-1 regs ffff8800c9f1bf58, eflags 217, rip 45b260
            perf-893   [003] d.h.  1358.053641: do_debug: do_debug-2 eflags 10217, rip 45b260
            perf-893   [003] d...  1358.053642: prepare_exit_to_usermode: prepare_exit_to_usermode1 regs ffff8800c9f1bf58, eflags 10217, rip 45b260
            perf-893   [003] d...  1358.053643: prepare_exit_to_usermode: prepare_exit_to_usermode3 regs ffff8800c9f1bf58, eflags 10217, rip 45b260

WORK_IRQ

 --->       perf-893   [003] d...  1358.053645: smp_irq_work_interrupt: smp_irq_work_interrupt1 regs ffff8800c9f1bf58, eflags 217, rip 45b260
            perf-893   [003] d.h.  1358.053650: perf_event_wakeup: irq_work SIGIO
            perf-893   [003] d...  1358.053651: smp_irq_work_interrupt: smp_irq_work_interrupt2 regs ffff8800c9f1bf58, eflags 217, rip 45b260
            perf-893   [003] d...  1358.053652: prepare_exit_to_usermode: prepare_exit_to_usermode1 regs ffff8800c9f1bf58, eflags 217, rip 45b260

this is trace_printk output for OLD microcode 0x6000822:

DEBUG EX

            perf-898   [005] d...    87.098816: sync_regs: sync_regs eregs ffff88012ed47f58, regs ffff8800c9c8ff58
            perf-898   [005] d...    87.098817: do_debug: do_debug-1 regs ffff8800c9c8ff58, eflags 217, rip 45b260
            perf-898   [005] d.h.    87.098823: do_debug: do_debug-2 eflags 10217, rip 45b260
            perf-898   [005] d...    87.098824: prepare_exit_to_usermode: prepare_exit_to_usermode1 regs ffff8800c9c8ff58, eflags 10217, rip 45b260
            perf-898   [005] d...    87.098825: prepare_exit_to_usermode: prepare_exit_to_usermode3 regs ffff8800c9c8ff58, eflags 10217, rip 45b260

WORK_IRQ

 --->       perf-898   [005] d...    87.098827: smp_irq_work_interrupt: smp_irq_work_interrupt1 regs ffff8800c9c8ff58, eflags 10217, rip 45b260
            perf-898   [005] d.h.    87.098832: perf_event_wakeup: irq_work SIGIO
            perf-898   [005] d...    87.098833: smp_irq_work_interrupt: smp_irq_work_interrupt2 regs ffff8800c9c8ff58, eflags 10217, rip 45b260
            perf-898   [005] d...    87.098833: prepare_exit_to_usermode: prepare_exit_to_usermode1 regs ffff8800c9c8ff58, eflags 10217, rip 45b260

thanks,
jirka

---

arch/x86/entry/common.c +3 -0
index 80dcc92..d52d598 100644
@@ -217,6 +217,8 @@ static struct thread_info *pt_regs_to_thread_info(struct pt_regs *regs)
 /* Called with IRQs disabled. */
 __visible void prepare_exit_to_usermode(struct pt_regs *regs)
 {
+	trace_printk("prepare_exit_to_usermode1 regs %p, eflags %lx, rip %lx\n", regs, regs->flags, regs->ip);
+
 	if (WARN_ON(!irqs_disabled()))
 		local_irq_disable();
 
@@ -263,6 +265,7 @@ __visible void prepare_exit_to_usermode(struct pt_regs *regs)
 	}
 
 	user_enter();
+	trace_printk("prepare_exit_to_usermode3 regs %p, eflags %lx, rip %lx\n", regs, regs->flags, regs->ip);
 }
 
 /*
arch/x86/kernel/irq_work.c +4 -0
index dc5fa6a..52fe376 100644
@@ -18,9 +18,13 @@ static inline void __smp_irq_work_interrupt(void)
 
 __visible void smp_irq_work_interrupt(struct pt_regs *regs)
 {
+	trace_printk("smp_irq_work_interrupt1 regs %p, eflags %lx, rip %lx\n", regs, regs->flags, regs->ip);
+
 	ipi_entering_ack_irq();
 	__smp_irq_work_interrupt();
 	exiting_irq();
+
+	trace_printk("smp_irq_work_interrupt2 regs %p, eflags %lx, rip %lx\n", regs, regs->flags, regs->ip);
 }
 
 __visible void smp_trace_irq_work_interrupt(struct pt_regs *regs)
arch/x86/kernel/signal.c +2 -0
index da52e6b..cb199dc 100644
@@ -398,6 +398,7 @@ static int __setup_rt_frame(int sig, struct ksignal *ksig,
 	regs->ss = __USER_DS;
 	regs->cs = __USER_CS;
 
+trace_printk("__setup_rt_frame regs %p, eflags %lx, rip %lx\n", regs, regs->flags, regs->ip);
 	return 0;
 }
 #else /* !CONFIG_X86_32 */
@@ -583,6 +584,7 @@ asmlinkage long sys_rt_sigreturn(void)
 	if (restore_altstack(&frame->uc.uc_stack))
 		goto badframe;
 
+trace_printk("sys_rt_sigreturn regs %p, eflags %lx, rip %lx\n", regs, regs->flags, regs->ip);
 	return regs->ax;
 
 badframe:
arch/x86/kernel/traps.c +4 -0
index bfc4f90..cee75d8 100644
@@ -536,6 +536,7 @@ asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs)
 {
 	struct pt_regs *regs = task_pt_regs(current);
 	*regs = *eregs;
+trace_printk("sync_regs eregs %p, regs %p\n", eregs, regs);
 	return regs;
 }
 NOKPROBE_SYMBOL(sync_regs);
@@ -602,6 +603,8 @@ dotraplinkage void do_debug(struct pt_regs *regs, long error_code)
 	unsigned long dr6;
 	int si_code;
 
+trace_printk("do_debug-1 regs %p, eflags %lx, rip %lx\n", regs, regs->flags, regs->ip);
+
 	ist_enter(regs);
 
 	get_debugreg(dr6, 6);
@@ -677,6 +680,7 @@ dotraplinkage void do_debug(struct pt_regs *regs, long error_code)
 	debug_stack_usage_dec();
 
 exit:
+	trace_printk("do_debug-2 eflags %lx, rip %lx\n", regs->flags, regs->ip);
 	ist_exit(regs);
 }
 NOKPROBE_SYMBOL(do_debug);
kernel/events/core.c +1 -0
index ae16867..6977f20 100644
@@ -4803,6 +4803,7 @@ void perf_event_wakeup(struct perf_event *event)
 
 	if (event->pending_kill) {
 		kill_fasync(perf_event_fasync(event), SIGIO, event->pending_kill);
+trace_printk("irq_work SIGIO\n");
 		event->pending_kill = 0;
 	}
 }
#8 lore raw
Quoted 11 lines · Jiri Olsa

this is the perf breakpoint address:
000000000045b260 <test_function>:

this is trace_printk output for NEW microcode 0x6000832:

DEBUG EX

            perf-893   [003] d...  1358.053633: sync_regs: sync_regs eregs ffff88012ecc7f58, regs ffff8800c9f1bf58
            perf-893   [003] d...  1358.053635: do_debug: do_debug-1 regs ffff8800c9f1bf58, eflags 217, rip 45b260
            perf-893   [003] d.h.  1358.053641: do_debug: do_debug-2 eflags 10217, rip 45b260
            perf-893   [003] d...  1358.053642: prepare_exit_to_usermode: prepare_exit_to_usermode1 regs ffff8800c9f1bf58, eflags 10217, rip 45b260
            perf-893   [003] d...  1358.053643: prepare_exit_to_usermode: prepare_exit_to_usermode3 regs ffff8800c9f1bf58, eflags 10217, rip 45b260

Yikes!

Something cleared EFLAGS.RF here. And the same thing doesn't happen with the older ucode version. Are you sure nothing happens in-between prepare_exit_to_usermode() and smp_irq_work_interrupt()?

It looks like nothing does because the timestamps are really close.

Quoted 17 lines

WORK_IRQ

 --->       perf-893   [003] d...  1358.053645: smp_irq_work_interrupt: smp_irq_work_interrupt1 regs ffff8800c9f1bf58, eflags 217, rip 45b260
            perf-893   [003] d.h.  1358.053650: perf_event_wakeup: irq_work SIGIO
            perf-893   [003] d...  1358.053651: smp_irq_work_interrupt: smp_irq_work_interrupt2 regs ffff8800c9f1bf58, eflags 217, rip 45b260
            perf-893   [003] d...  1358.053652: prepare_exit_to_usermode: prepare_exit_to_usermode1 regs ffff8800c9f1bf58, eflags 217, rip 45b260

this is trace_printk output for OLD microcode 0x6000822:

DEBUG EX

            perf-898   [005] d...    87.098816: sync_regs: sync_regs eregs ffff88012ed47f58, regs ffff8800c9c8ff58
            perf-898   [005] d...    87.098817: do_debug: do_debug-1 regs ffff8800c9c8ff58, eflags 217, rip 45b260
            perf-898   [005] d.h.    87.098823: do_debug: do_debug-2 eflags 10217, rip 45b260
            perf-898   [005] d...    87.098824: prepare_exit_to_usermode: prepare_exit_to_usermode1 regs ffff8800c9c8ff58, eflags 10217, rip 45b260
            perf-898   [005] d...    87.098825: prepare_exit_to_usermode: prepare_exit_to_usermode3 regs ffff8800c9c8ff58, eflags 10217, rip 45b260
Quoted 6 lines

WORK_IRQ

 --->       perf-898   [005] d...    87.098827: smp_irq_work_interrupt: smp_irq_work_interrupt1 regs ffff8800c9c8ff58, eflags 10217, rip 45b260
            perf-898   [005] d.h.    87.098832: perf_event_wakeup: irq_work SIGIO
            perf-898   [005] d...    87.098833: smp_irq_work_interrupt: smp_irq_work_interrupt2 regs ffff8800c9c8ff58, eflags 10217, rip 45b260
            perf-898   [005] d...    87.098833: prepare_exit_to_usermode: prepare_exit_to_usermode1 regs ffff8800c9c8ff58, eflags 10217, rip 45b260

Thanks.

Signature · 7 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
author
#9 lore raw
Quoted 13 lines · Borislav Petkov
Quoted 11 lines · Jiri Olsa

this is the perf breakpoint address:
000000000045b260 <test_function>:

this is trace_printk output for NEW microcode 0x6000832:

DEBUG EX

            perf-893   [003] d...  1358.053633: sync_regs: sync_regs eregs ffff88012ecc7f58, regs ffff8800c9f1bf58
            perf-893   [003] d...  1358.053635: do_debug: do_debug-1 regs ffff8800c9f1bf58, eflags 217, rip 45b260
            perf-893   [003] d.h.  1358.053641: do_debug: do_debug-2 eflags 10217, rip 45b260
            perf-893   [003] d...  1358.053642: prepare_exit_to_usermode: prepare_exit_to_usermode1 regs ffff8800c9f1bf58, eflags 10217, rip 45b260
            perf-893   [003] d...  1358.053643: prepare_exit_to_usermode: prepare_exit_to_usermode3 regs ffff8800c9f1bf58, eflags 10217, rip 45b260
Quoted 5 lines

Yikes!

Something cleared EFLAGS.RF here. And the same thing doesn't happen with the older ucode version. Are you sure nothing happens in-between prepare_exit_to_usermode() and smp_irq_work_interrupt()?

if anything happens there, I dont see it ;-)

I was told RHEL7 went off and on several times with AMD microcode, now it's the period we actually install it.. I'll poke around to see who made that decissions, he/she might know more ;-)

jirka

#10 lore raw
Quoted 4 lines · Jiri Olsa

if anything happens there, I dont see it ;-)

I was told RHEL7 went off and on several times with AMD microcode, now it's the period we actually install it..

"went off an on"? What do you mean exactly?

Quoted 2 lines

I'll poke around to see who made that decissions, he/she might know more ;-)

Yeah, especially if they have the info which microcode patch level fixes which erratum...

Signature · 7 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--
author
#11 lore raw
Quoted 7 lines · Borislav Petkov
Quoted 4 lines · Jiri Olsa

if anything happens there, I dont see it ;-)

I was told RHEL7 went off and on several times with AMD microcode, now it's the period we actually install it..

"went off an on"? What do you mean exactly?

it was included and excluded from linux-firmware package several times

jirka

#12 lore raw
In reply to Jiri Olsa
Quoted 114 lines · Jiri Olsa
Quoted 40 lines · Andy Lutomirski
Quoted 39 lines · Borislav Petkov
Quoted 2 lines · Jiri Olsa

hi,
'perf test 18' is failing on systems with AMD processor.

Hmm, still using that b0rked test box? :-)

Also, which kernel?

There have been substantial changes to the entry code recently. Although I don't see anything being done differently on AMD there except X86_BUG_SYSRET_SS_ATTRS but that should be unrelated.

Quoted 28 lines

The only reason I could find is that AMD does not set 'resume flag' in RFLAGS register the way the Intel CPU does.

(simplified) test scenario:

- create breakpoint (on test_function) perf event with SIGIO signal to be delivered any time the breakpoint is hit
- run test_function

expected course of actions is:
1) CPU hits 'test_function'
2) DB exception is triggered, with RFLAGS.RF=0
3) DB exception handler sets regs->RFLAGS.RF=1 and perf handler triggers irq_work pending work
4) DB exception executes iretd
5) irq_work interrupt is triggered, with RFLAGS.RF=1
6) irq_work interrupt calls kill_fasync with SIGIO signal
7) irq_work interrupt on return to userspace calls prepare_exit_to_usermode which actually delivers the SIGIO signal
8) sigreturn syscall prepare registers to return to the instruction from step 1) and sets RFLAGS.RF to the its original value from step 5) (RFLAGS.RF=1)
9) CPU hits 'test_function' and DB exception is NOT triggered due to RFLAGS.RF=1

this is how I see it works on Intel

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

But AMD gives me RFLAGS.RF=0 on step 5, which makes the step 9 to trigger the DB exception once again and makes the test fail.

Adding Andy, he might have an idea. Leaving in the rest for reference.

Gee thanks :-p

Jiri, did you instrument the code and observe do_IRQ sees RF clear in its pt_regs? Also, it might be worth checking that regs->ip in the irq_work matches regs->ip.

yep, thats what I saw.. once irq_work interrupt was triggered the regs->ip was same as for the previous debug exception but the RFLAGS.RF was 0

Quoted 5 lines

It's *possible* that I messed up and broke RF restore with opportunistic sysret, but the code looks correct:

         testq   $(X86_EFLAGS_RF|X86_EFLAGS_TF), %r11
         jnz     opportunistic_sysret_failed

AFAICS the problematic paths did not hit syscalls

buuuuuut anyway, it looks like latest AMD firmware issue:

[root@amd-pike-07 ~]# cat /sys/devices/system/cpu/cpu0/microcode/version 0x6000822
[root@amd-pike-07 perf]# ./perf test 18
18: Test breakpoint overflow signal handler : Ok

[root@amd-pike-07 perf]# cat /sys/devices/system/cpu/cpu0/microcode/version 0x6000832
[root@amd-pike-07 perf]# ./perf test 18
18: Test breakpoint overflow signal handler : FAILED!

[root@amd-pike-07 ~]# cat /proc/cpuinfo
processor : 7
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD Opteron(tm) Processor 3380
stepping : 0
microcode : 0x6000832

SNIP

Quoted 16 lines
Quoted 8 lines
Quoted 8 lines

AMD description of RF flag (SDM 3.1.6):
=======================================
Resume Flag (RF) Bit. Bit 16. The RF bit allows an instruction to be restarted following an instruction breakpoint resulting in a debug exception (#DB). This bit prevents multiple debug exceptions from occurring on the same instruction.
The processor clears the RF bit after every instruction is successfully executed, except when the instruction is:
•

Quoted 8 lines
Quoted 8 lines

•
An IRET that sets the RF bit.
JMP, CALL, or INTn through a task gate.
In both of the above cases, RF is not cleared to 0 until the next instruction successfully executes. When an exception occurs (or when a string instruction is interrupted), the processor normally sets RF=1 in the RFLAGS image saved on the interrupt stack. However, when a #DB exception occurs as a result of an instruction breakpoint, the processor clears the RF bit to 0 in the interrupt-stack RFLAGS image.

Quoted 3 lines

That's a little weird, I think. Shouldn't RF be zero on #DB due to a *watchpoint* so that a watchpoint followed immediately by a breakpoint works?

the AMD description looked to be more vague (compared to Intels)

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

• For other cases, the value pushed for RF is the value that was in EFLAG.RF at the time the event handler was called. This includes:
— Debug exceptions generated in response to instruction breakpoints
— Hardware-generated interrupts arriving between instructions (including those arriving after the last iteration of a repeated string instruction)

This appears to be why it works on Intel. Does AMD not do that? We could probably work around this in software (by not using irq work for this), but yuck.

Quoted 6 lines

yep, but hopefuly it's the issue microcode ;-) Cc-ing guys from linux-firmware git

Sherry, Suravee, any idea?

thanks,
jirka

Jiri,
I have duplicated your problem and asked the HW architect that wrote 832 to review the diff between the 822 and 832 microcode patch.

Thanks,
Sherry

#13 lore raw

On Mon, Aug 24, 2015 at 05:37:17PM -0500, sherry hurwitz wrote:
Hey Sherry,

Quoted 2 lines

I have duplicated your problem and asked the HW architect that wrote 832 to review the diff between the 822 and 832 microcode patch.

were there any updates to this issue in the meantime?

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
--