Skip to contentSkip to conversation

x86/segment: Introduce storesegment() helper to write segment selectors to memory

PATCH Superseded Uros Bizjak 2 participants 8 messages Mar 31 – Apr 1

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 x86/segment: Introduce storesegment() helper to write segment selectors to memory

Introduce a new helper, storesegment(), that stores a segment selector directly into a u16 (or compatible) memory location without using an intermediate general-purpose register.

To support this, split the existing SAVE_SEGMENT macro into two parts:

SAVE_SEGMENT_VAR(): retains the current behavior of reading a segment register into an unsigned long via a register. SAVE_SEGMENT_PTR(): adds a new variant that writes the 16-bit selector directly to memory.

The combined SAVE_SEGMENT() macro now generates both helpers for each segment register.

The new storesegment() interface is preferred over savesegment() when the value only needs to be stored (e.g. into a struct field), avoiding an unnecessary register move and making the intent clearer.

No functional change for existing users of savesegment().

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Dave Hansen <[email protected]>
---
 arch/x86/include/asm/segment.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
arch/x86/include/asm/segment.h +29 -1
index dbd90fede5e7..b9728c4acd46 100644
@@ -344,7 +344,7 @@ static inline void __loadsegment_fs(u16 value)
 /*
  * Save a segment register away:
  */
-#define SAVE_SEGMENT(seg)				\
+#define SAVE_SEGMENT_VAR(seg)				\
 static inline unsigned long __savesegment_##seg(void)	\
 {							\
 	unsigned long v;				\
@@ -352,6 +352,16 @@ static inline unsigned long __savesegment_##seg(void)	\
 	return v;					\
 }
 
+#define SAVE_SEGMENT_PTR(seg)				\
+static inline void __savesegment_##seg##_ptr(u16 *p)	\
+{							\
+	asm volatile("movw %%" #seg ",%0" : "=m" (*p));	\
+}
+
+#define SAVE_SEGMENT(seg)				\
+	SAVE_SEGMENT_VAR(seg)				\
+	SAVE_SEGMENT_PTR(seg)
+
 SAVE_SEGMENT(cs)
 SAVE_SEGMENT(ss)
 SAVE_SEGMENT(ds)
@@ -359,10 +369,28 @@ SAVE_SEGMENT(es)
 SAVE_SEGMENT(fs)
 SAVE_SEGMENT(gs)
 
+#undef SAVE_SEGMENT_VAR
+#undef SAVE_SEGMENT_PTR
 #undef SAVE_SEGMENT
 
+/*
+ * savesegment(seg, var) - Read a segment register into an unsigned long.
+ *
+ * Reads the segment selector via a general-purpose register into an
+ * unsigned long. Preferred when the value is needed in a register for
+ * subsequent arithmetic or comparison.
+ */
 #define savesegment(seg, var) ((var) = __savesegment_##seg())
 
+/*
+ * storesegment(seg, loc) - Store a segment register directly to a u16 location.
+ *
+ * Writes the 16-bit segment selector directly to memory, bypassing any
+ * intermediate general-purpose register. Preferred over savesegment()
+ * for simply saving a segment register to a u16 (or compatible) field.
+ */
+#define storesegment(seg, loc) __savesegment_##seg##_ptr(&(loc))
+
 #endif /* !__ASSEMBLER__ */
 #endif /* __KERNEL__ */
 
Signature · 1 line
2.53.0
#1 lore raw

PATCH 2/2 x86/process: Use storesegment() when saving segment selectors

Switch segment save sites in copy_thread() and save_fsgs() from savesegment() to the newly introduced storesegment() helper.

These call sites only store the segment selector into a u16 field and do not require the value in a general-purpose register. Using storesegment() avoids the unnecessary register intermediate and better matches the intended use.

The generated code is improved from, e.g.:

34f: 8c c0 mov %es,%eax
351: 66 89 83 08 0c 00 00 mov %ax,0xc08(%rbx)

to a single direct store:

     34f:	8c 83 08 0c 00 00    	mov    %es,0xc08(%rbx)

This is a mechanical follow-up to the introduction of storesegment(), with no functional change.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Dave Hansen <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
---
 arch/x86/kernel/process.c    | 4 ++--
 arch/x86/kernel/process_64.c | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/process.c +2 -2
index 4c718f8adc59..10e2ba668a14 100644
@@ -197,8 +197,8 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 	p->thread.gsindex = current->thread.gsindex;
 	p->thread.gsbase = current->thread.gsbase;
 
-	savesegment(es, p->thread.es);
-	savesegment(ds, p->thread.ds);
+	storesegment(es, p->thread.es);
+	storesegment(ds, p->thread.ds);
 
 	if (p->mm && (clone_flags & (CLONE_VM | CLONE_VFORK)) == CLONE_VM)
 		set_bit(MM_CONTEXT_LOCK_LAM, &p->mm->context.flags);
arch/x86/kernel/process_64.c +2 -2
index b85e715ebb30..fe391961ed86 100644
@@ -275,8 +275,8 @@ static __always_inline void save_base_legacy(struct task_struct *prev_p,
 
 static __always_inline void save_fsgs(struct task_struct *task)
 {
-	savesegment(fs, task->thread.fsindex);
-	savesegment(gs, task->thread.gsindex);
+	storesegment(fs, task->thread.fsindex);
+	storesegment(gs, task->thread.gsindex);
 	if (static_cpu_has(X86_FEATURE_FSGSBASE)) {
 		/*
 		 * If FSGSBASE is enabled, we can't make any useful guesses
Signature · 1 line
2.53.0
#2 lore raw
In reply to Uros Bizjak
Quoted 11 lines · Uros Bizjak

Introduce a new helper, storesegment(), that stores a segment selector directly into a u16 (or compatible) memory location without using an intermediate general-purpose register.

To support this, split the existing SAVE_SEGMENT macro into two parts:

SAVE_SEGMENT_VAR(): retains the current behavior of reading a segment register into an unsigned long via a register. SAVE_SEGMENT_PTR(): adds a new variant that writes the 16-bit selector directly to memory.

Quoted 8 lines

The combined SAVE_SEGMENT() macro now generates both helpers for each segment register.

The new storesegment() interface is preferred over savesegment() when the value only needs to be stored (e.g. into a struct field), avoiding an unnecessary register move and making the intent clearer.

No functional change for existing users of savesegment().

Why does the API have to be split into =r and =m variants?

Coulnd't we use a more generic constraint and let the compiler decide what the target is? Would that negatively impact other aspects of code generation?

Thanks,

Ingo

#3 lore raw
Quoted 22 lines · Ingo Molnar
Quoted 18 lines · Uros Bizjak

Introduce a new helper, storesegment(), that stores a segment selector directly into a u16 (or compatible) memory location without using an intermediate general-purpose register.

To support this, split the existing SAVE_SEGMENT macro into two parts:

SAVE_SEGMENT_VAR(): retains the current behavior of reading a segment register into an unsigned long via a register. SAVE_SEGMENT_PTR(): adds a new variant that writes the 16-bit selector directly to memory.

The combined SAVE_SEGMENT() macro now generates both helpers for each segment register.

The new storesegment() interface is preferred over savesegment() when the value only needs to be stored (e.g. into a struct field), avoiding an unnecessary register move and making the intent clearer.

Quoted 7 lines
Quoted 1 line

No functional change for existing users of savesegment().

Why does the API have to be split into =r and =m variants?

Coulnd't we use a more generic constraint and let the compiler decide what the target is? Would that negatively impact other aspects of code generation?

The "=r" variant actually outputs zero-extended value to the whole register width. So, the "=r" variant is used to eliminate zero-extensions when the value is used in the follow-up calculations, comparisons, or when the value is stored to a location that is more than 16-bits wide. Additionally, "r" variant always uses MOVL, where operand size prefix byte (0x66) is not needed.

The "=m" variant only outputs to a 16-bit location. Having "=rm" here would always emit a 0x66 operand size prefix when register is used as an output, and there would be many zero-extensions emitted, because the compiler needs to zero-extend the value from 'unsigned short' to anything wider.

Other than that, GCC (and Clang, too) has serious problems with "=rm" output constraints. Forward propagation (AKA combine pass) does not work reliably with assembly outputs (due to always present clobbers for assembly clauses), so there will be many cases of moves to a temporary register and even to a temporary stack location with this constraint. Having two separate functions (with clear and informational function comment) leaves the decision to the programmer, which function is the most optimal.

Uros.

#4 lore raw
Quoted 45 lines · Uros Bizjak
Quoted 22 lines · Ingo Molnar
Quoted 18 lines · Uros Bizjak

Introduce a new helper, storesegment(), that stores a segment selector directly into a u16 (or compatible) memory location without using an intermediate general-purpose register.

To support this, split the existing SAVE_SEGMENT macro into two parts:

SAVE_SEGMENT_VAR(): retains the current behavior of reading a segment register into an unsigned long via a register. SAVE_SEGMENT_PTR(): adds a new variant that writes the 16-bit selector directly to memory.

The combined SAVE_SEGMENT() macro now generates both helpers for each segment register.

The new storesegment() interface is preferred over savesegment() when the value only needs to be stored (e.g. into a struct field), avoiding an unnecessary register move and making the intent clearer.

Quoted 7 lines
Quoted 1 line

No functional change for existing users of savesegment().

Why does the API have to be split into =r and =m variants?

Coulnd't we use a more generic constraint and let the compiler decide what the target is? Would that negatively impact other aspects of code generation?

The "=r" variant actually outputs zero-extended value to the whole register width. So, the "=r" variant is used to eliminate zero-extensions when the value is used in the follow-up calculations, comparisons, or when the value is stored to a location that is more than 16-bits wide. Additionally, "r" variant always uses MOVL, where operand size prefix byte (0x66) is not needed.

The "=m" variant only outputs to a 16-bit location. Having "=rm" here would always emit a 0x66 operand size prefix when register is used as an output, and there would be many zero-extensions emitted, because the compiler needs to zero-extend the value from 'unsigned short' to anything wider.

Quoted 8 lines

Other than that, GCC (and Clang, too) has serious problems with "=rm" output constraints. Forward propagation (AKA combine pass) does not work reliably with assembly outputs (due to always present clobbers for assembly clauses), so there will be many cases of moves to a temporary register and even to a temporary stack location with this constraint. Having two separate functions (with clear and informational function comment) leaves the decision to the programmer, which function is the most optimal.

I forgot to say that there are more opportunities for storesegment() in other subsystems, please see the attached patch.

Uros.

arch/x86/hyperv/hv_crash.c +6 -6
index 5ffcc23255de..1d29ada3d9f1 100644
@@ -207,12 +207,12 @@ static void hv_hvcrash_ctxt_save(void)
 	asm volatile("movq %%cr2, %0" : "=r"(ctxt->cr2));
 	asm volatile("movq %%cr8, %0" : "=r"(ctxt->cr8));
 
-	asm volatile("movw %%cs, %0" : "=m"(ctxt->cs));
-	asm volatile("movw %%ss, %0" : "=m"(ctxt->ss));
-	asm volatile("movw %%ds, %0" : "=m"(ctxt->ds));
-	asm volatile("movw %%es, %0" : "=m"(ctxt->es));
-	asm volatile("movw %%fs, %0" : "=m"(ctxt->fs));
-	asm volatile("movw %%gs, %0" : "=m"(ctxt->gs));
+	storesegment(cs, ctxt->cs);
+	storesegment(ss, ctxt->ss);
+	storesegment(ds, ctxt->ds);
+	storesegment(es, ctxt->es);
+	storesegment(fs, ctxt->fs);
+	storesegment(gs, ctxt->gs);
 
 	native_store_gdt(&ctxt->gdtr);
 	store_idt(&ctxt->idtr);
arch/x86/include/asm/kexec.h +4 -4
index 5cfb27f26583..9b20eb699c54 100644
@@ -106,11 +106,11 @@ static inline void crash_setup_regs(struct pt_regs *newregs,
 		asm volatile("mov %%r14,%0" : "=m"(newregs->r14));
 		asm volatile("mov %%r15,%0" : "=m"(newregs->r15));
 #endif
-		asm volatile("mov %%ss,%k0" : "=a"(newregs->ss));
-		asm volatile("mov %%cs,%k0" : "=a"(newregs->cs));
+		storesegment(ss, newregs->ss);
+		storesegment(cs, newregs->cs);
 #ifdef CONFIG_X86_32
-		asm volatile("mov %%ds,%k0" : "=a"(newregs->ds));
-		asm volatile("mov %%es,%k0" : "=a"(newregs->es));
+		storesegment(ds, newregs->ds);
+		storesegment(es, newregs->es);
 #endif
 		asm volatile("pushf\n\t"
 			     "pop %0" : "=m"(newregs->flags));
arch/x86/kvm/vmx/vmx.c +2 -2
index 8b24e682535b..4e9939a8ff30 100644
@@ -1390,8 +1390,8 @@ void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu)
 	host_state->ldt_sel = kvm_read_ldt();
 
 #ifdef CONFIG_X86_64
-	savesegment(ds, host_state->ds_sel);
-	savesegment(es, host_state->es_sel);
+	storesegment(ds, host_state->ds_sel);
+	storesegment(es, host_state->es_sel);
 
 	gs_base = cpu_kernelmode_gs_base(cpu);
 	if (likely(is_64bit_mm(current->mm))) {
arch/x86/power/cpu.c +4 -4
index 702f30eaf9c4..24253f40fa52 100644
@@ -105,11 +105,11 @@ static void __save_processor_state(struct saved_context *ctxt)
 	/*
 	 * segment registers
 	 */
-	savesegment(gs, ctxt->gs);
+	storesegment(gs, ctxt->gs);
 #ifdef CONFIG_X86_64
-	savesegment(fs, ctxt->fs);
-	savesegment(ds, ctxt->ds);
-	savesegment(es, ctxt->es);
+	storesegment(fs, ctxt->fs);
+	storesegment(ds, ctxt->ds);
+	storesegment(es, ctxt->es);
 
 	rdmsrq(MSR_FS_BASE, ctxt->fs_base);
 	rdmsrq(MSR_GS_BASE, ctxt->kernelmode_gs_base);
#5 lore raw
In reply to Uros Bizjak
Quoted 44 lines · Uros Bizjak
Quoted 22 lines · Ingo Molnar
Quoted 18 lines · Uros Bizjak

Introduce a new helper, storesegment(), that stores a segment selector directly into a u16 (or compatible) memory location without using an intermediate general-purpose register.

To support this, split the existing SAVE_SEGMENT macro into two parts:

SAVE_SEGMENT_VAR(): retains the current behavior of reading a segment register into an unsigned long via a register. SAVE_SEGMENT_PTR(): adds a new variant that writes the 16-bit selector directly to memory.

The combined SAVE_SEGMENT() macro now generates both helpers for each segment register.

The new storesegment() interface is preferred over savesegment() when the value only needs to be stored (e.g. into a struct field), avoiding an unnecessary register move and making the intent clearer.

Quoted 7 lines
Quoted 1 line

No functional change for existing users of savesegment().

Why does the API have to be split into =r and =m variants?

Coulnd't we use a more generic constraint and let the compiler decide what the target is? Would that negatively impact other aspects of code generation?

The "=r" variant actually outputs zero-extended value to the whole register width. So, the "=r" variant is used to eliminate zero-extensions when the value is used in the follow-up calculations, comparisons, or when the value is stored to a location that is more than 16-bits wide. Additionally, "r" variant always uses MOVL, where operand size prefix byte (0x66) is not needed.

The "=m" variant only outputs to a 16-bit location. Having "=rm" here would always emit a 0x66 operand size prefix when register is used as an output, and there would be many zero-extensions emitted, because the compiler needs to zero-extend the value from 'unsigned short' to anything wider.

Quoted 8 lines

Other than that, GCC (and Clang, too) has serious problems with "=rm" output constraints. Forward propagation (AKA combine pass) does not work reliably with assembly outputs (due to always present clobbers for assembly clauses), so there will be many cases of moves to a temporary register and even to a temporary stack location with this constraint. Having two separate functions (with clear and informational function comment) leaves the decision to the programmer, which function is the most optimal.

Yeah, so I still have a problem with the split API, 'savesegment()' is very similar to 'storesegment()' and there's no real way to tell them apart in practice.

Since the main difference is that the =m variant outputs to an u16, I'd suggest naming the two APIs according to the type they handle, not some random word that nobody remembers:

	savesegment()
	savesegment_u16()

... or so?

Thanks,

Ingo

#6 lore raw
Quoted 32 lines · Ingo Molnar
Quoted 19 lines
Quoted 5 lines

Why does the API have to be split into =r and =m variants?

Coulnd't we use a more generic constraint and let the compiler decide what the target is? Would that negatively impact other aspects of code generation?

The "=r" variant actually outputs zero-extended value to the whole register width. So, the "=r" variant is used to eliminate zero-extensions when the value is used in the follow-up calculations, comparisons, or when the value is stored to a location that is more than 16-bits wide. Additionally, "r" variant always uses MOVL, where operand size prefix byte (0x66) is not needed.

The "=m" variant only outputs to a 16-bit location. Having "=rm" here would always emit a 0x66 operand size prefix when register is used as an output, and there would be many zero-extensions emitted, because the compiler needs to zero-extend the value from 'unsigned short' to anything wider.

Quoted 8 lines

Other than that, GCC (and Clang, too) has serious problems with "=rm" output constraints. Forward propagation (AKA combine pass) does not work reliably with assembly outputs (due to always present clobbers for assembly clauses), so there will be many cases of moves to a temporary register and even to a temporary stack location with this constraint. Having two separate functions (with clear and informational function comment) leaves the decision to the programmer, which function is the most optimal.

Yeah, so I still have a problem with the split API, 'savesegment()' is very similar to 'storesegment()' and there's no real way to tell them apart in practice.

Quoted 8 lines

Since the main difference is that the =m variant outputs to an u16, I'd suggest naming the two APIs according to the type they handle, not some random word that nobody remembers:

        savesegment()
        savesegment_u16()

... or so?

Yes, the above is a good proposal. I was undecided how to name the new function, but the above is definitely more informative. Maybe also emphasize that the new function operates on memory, so:

    savesegment_mem16()

(the location does not need to be unsigned)?

Thanks,
Uros.

#7 lore raw
Quoted 43 lines · Uros Bizjak
Quoted 32 lines · Ingo Molnar
Quoted 19 lines
Quoted 5 lines

Why does the API have to be split into =r and =m variants?

Coulnd't we use a more generic constraint and let the compiler decide what the target is? Would that negatively impact other aspects of code generation?

The "=r" variant actually outputs zero-extended value to the whole register width. So, the "=r" variant is used to eliminate zero-extensions when the value is used in the follow-up calculations, comparisons, or when the value is stored to a location that is more than 16-bits wide. Additionally, "r" variant always uses MOVL, where operand size prefix byte (0x66) is not needed.

The "=m" variant only outputs to a 16-bit location. Having "=rm" here would always emit a 0x66 operand size prefix when register is used as an output, and there would be many zero-extensions emitted, because the compiler needs to zero-extend the value from 'unsigned short' to anything wider.

Quoted 8 lines

Other than that, GCC (and Clang, too) has serious problems with "=rm" output constraints. Forward propagation (AKA combine pass) does not work reliably with assembly outputs (due to always present clobbers for assembly clauses), so there will be many cases of moves to a temporary register and even to a temporary stack location with this constraint. Having two separate functions (with clear and informational function comment) leaves the decision to the programmer, which function is the most optimal.

Yeah, so I still have a problem with the split API, 'savesegment()' is very similar to 'storesegment()' and there's no real way to tell them apart in practice.

Quoted 8 lines

Since the main difference is that the =m variant outputs to an u16, I'd suggest naming the two APIs according to the type they handle, not some random word that nobody remembers:

        savesegment()
        savesegment_u16()

... or so?

Quoted 7 lines

Yes, the above is a good proposal. I was undecided how to name the new function, but the above is definitely more informative. Maybe also emphasize that the new function operates on memory, so:

    savesegment_mem16()

(the location does not need to be unsigned)?

Sounds good to me.

Thanks,

Ingo