Skip to contentSkip to conversation

add support for new persistent memory instructions

PATCH Superseded Ross Zwisler 6 participants 20 messages Jan 27, 2015 – Apr 3, 2015 x86

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover add support for new persistent memory instructions

This patch set adds support for two new persistent memory instructions, pcommit and clwb. These instructions were announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

These patches apply cleanly to v3.19-rc6.

Changes from v2:

- Added instruction descriptions and flows to the patch descriptions.
- Added needed sfence to pcommit alternatives assembly. The inline function is now called pcommit_sfence(). If pcommit is not supported on the platform both the pcommit and the sfence will be nops.

Cc H Peter Anvin <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Borislav Petkov <[email protected]>

Ross Zwisler (2):
x86: Add support for the pcommit instruction
x86: Add support for the clwb instruction

 arch/x86/include/asm/cpufeature.h    |  2 ++
 arch/x86/include/asm/special_insns.h | 22 ++++++++++++++++++++++
 2 files changed, 24 insertions(+)
Signature · 1 line
1.9.3
#1 lore raw

PATCH v3 1/2 x86: Add support for the pcommit instruction

Add support for the new pcommit (persistent commit) instruction. This instruction was announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

The pcommit instruction ensures that data that has been flushed from the processor's cache hierarchy with clwb, clflushopt or clflush is accepted to memory and is durable on the DIMM. The primary use case for this is persistent memory.

This function shows how to properly use clwb/clflushopt/clflush and pcommit with appropriate fencing:

void flush_and_commit_buffer(void *vaddr, unsigned int size)

{
	void *vend = vaddr + size - 1;

	for (; vaddr < vend; vaddr += boot_cpu_data.x86_clflush_size)
		clwb(vaddr);

	/* Flush any possible final partial cacheline */
	clwb(vend);

	/*
	 * sfence to order clwb/clflushopt/clflush cache flushes
	 * mfence via mb() also works
	 */
	wmb();

	/* pcommit and the required sfence for ordering */
	pcommit_sfence();
}

After this function completes the data pointed to by vaddr is has been accepted to memory and will be durable if the vaddr points to persistent memory.

Pcommit must always be ordered by an mfence or sfence, so to help simplify things we include both the pcommit and the required sfence in the alternatives generated by pcommit_sfence(). The other option is to keep them separated, but on platforms that don't support pcommit this would then turn into:

void flush_and_commit_buffer(void *vaddr, unsigned int size)

{
        void *vend = vaddr + size - 1;

        for (; vaddr < vend; vaddr += boot_cpu_data.x86_clflush_size)
                clwb(vaddr);

        /* Flush any possible final partial cacheline */
        clwb(vend);

        /*
         * sfence to order clwb/clflushopt/clflush cache flushes
         * mfence via mb() also works
         */
        wmb();

        nop(); /* from pcommit(), via alternatives */

        /*
         * sfence to order pcommit
         * mfence via mb() also works
         */
        wmb();
}

This is still correct, but now you've got two fences separated by only a nop. With the commit and the fence together in pcommit_sfence() you avoid the final unneeded fence.

Cc H Peter Anvin <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Borislav Petkov <[email protected]>
---
 arch/x86/include/asm/cpufeature.h    | 1 +
 arch/x86/include/asm/special_insns.h | 8 ++++++++
 2 files changed, 9 insertions(+)
arch/x86/include/asm/cpufeature.h +1 -0
index bb9b258..dfdd689 100644
@@ -220,6 +220,7 @@
 #define X86_FEATURE_RDSEED	( 9*32+18) /* The RDSEED instruction */
 #define X86_FEATURE_ADX		( 9*32+19) /* The ADCX and ADOX instructions */
 #define X86_FEATURE_SMAP	( 9*32+20) /* Supervisor Mode Access Prevention */
+#define X86_FEATURE_PCOMMIT	( 9*32+22) /* PCOMMIT instruction */
 #define X86_FEATURE_CLFLUSHOPT	( 9*32+23) /* CLFLUSHOPT instruction */
 #define X86_FEATURE_AVX512PF	( 9*32+26) /* AVX-512 Prefetch */
 #define X86_FEATURE_AVX512ER	( 9*32+27) /* AVX-512 Exponential and Reciprocal */
arch/x86/include/asm/special_insns.h +8 -0
index e820c08..d686f9b 100644
@@ -199,6 +199,14 @@ static inline void clflushopt(volatile void *__p)
 		       "+m" (*(volatile char __force *)__p));
 }
 
+static inline void pcommit_sfence(void)
+{
+	alternative(ASM_NOP7,
+		    ".byte 0x66, 0x0f, 0xae, 0xf8\n\t" /* pcommit */
+		    "sfence",
+		    X86_FEATURE_PCOMMIT);
+}
+
 #define nop() asm volatile ("nop")
 
 
Signature · 1 line
1.9.3
maintainer Acked-by on v3 1/2
#2 lore raw

Re: [PATCH v3 1/2] x86: Add support for the pcommit instruction

Quoted 14 lines · Ross Zwisler

Add support for the new pcommit (persistent commit) instruction. This instruction was announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

The pcommit instruction ensures that data that has been flushed from the processor's cache hierarchy with clwb, clflushopt or clflush is accepted to memory and is durable on the DIMM. The primary use case for this is persistent memory.

This function shows how to properly use clwb/clflushopt/clflush and pcommit with appropriate fencing:

...

Quoted 9 lines

This is still correct, but now you've got two fences separated by only a nop. With the commit and the fence together in pcommit_sfence() you avoid the final unneeded fence.

Cc H Peter Anvin <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Borislav Petkov <[email protected]>
Signature · 5 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
#3 lore raw

RE: [PATCH v3 1/2] x86: Add support for the pcommit instruction

In reply to Ross Zwisler
Quoted 19 lines

-----Original Message-----
From: [email protected] [mailto:linux-kernel- [email protected]] On Behalf Of Ross Zwisler
Sent: Tuesday, 27 January, 2015 10:54 AM
To: [email protected]
Cc: Ross Zwisler; H Peter Anvin; Ingo Molnar; Thomas Gleixner; Borislav Petkov
Subject: [PATCH v3 1/2] x86: Add support for the pcommit instruction

Add support for the new pcommit (persistent commit) instruction. This instruction was announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-
022.pdf

...

Quoted 4 lines
---
 arch/x86/include/asm/cpufeature.h    | 1 +
 arch/x86/include/asm/special_insns.h | 8 ++++++++
 2 files changed, 9 insertions(+)

Should this patch series also add defines for the virtual machine control data structure changes?

1. Add the new VM-Execution Controls bit 21 as

    SECONDARY_EXEC_PCOMMIT_EXITING 0x00200000

to arch/x86/include/asm/vmx.h.

2. Add the new exit reason of 64 (0x41) as EXIT_REASON_PCOMMIT 64
to arch/x86/include/uapi/asm/vmx.h and (with a VMX_EXIT_REASONS string) to usr/include/asm/vmx.h.

3. Add a kvm_vmx_exit_handler to arch/x86/kvm/vmx.c.

---
Rob Elliott HP Server Storage

maintainer
#4 lore raw

Re: [PATCH v3 1/2] x86: Add support for the pcommit instruction

Quoted 5 lines · Robert (Server Storage)

Should this patch series also add defines for the virtual machine control data structure changes?

1. Add the new VM-Execution Controls bit 21 as

    SECONDARY_EXEC_PCOMMIT_EXITING 0x00200000
Quoted 8 lines

to arch/x86/include/asm/vmx.h.

2. Add the new exit reason of 64 (0x41) as EXIT_REASON_PCOMMIT 64
to arch/x86/include/uapi/asm/vmx.h and (with a VMX_EXIT_REASONS string) to usr/include/asm/vmx.h.

3. Add a kvm_vmx_exit_handler to arch/x86/kvm/vmx.c.

These look like a separate patchset for kvm enablement to me.

Signature · 5 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
#5 lore raw

Re: [PATCH v3 1/2] x86: Add support for the pcommit instruction

Quoted 8 lines · Borislav Petkov
Quoted 7 lines · Robert (Server Storage)

Should this patch series also add defines for the virtual machine control data structure changes?

1. Add the new VM-Execution Controls bit 21 as

    SECONDARY_EXEC_PCOMMIT_EXITING 0x00200000

to arch/x86/include/asm/vmx.h.

Quoted 8 lines
Quoted 6 lines

2. Add the new exit reason of 64 (0x41) as EXIT_REASON_PCOMMIT 64
to arch/x86/include/uapi/asm/vmx.h and (with a VMX_EXIT_REASONS string) to usr/include/asm/vmx.h.

3. Add a kvm_vmx_exit_handler to arch/x86/kvm/vmx.c.

These look like a separate patchset for kvm enablement to me.

Agreed, I think they are a separate patch set.

Acked-by on v3 1/2
#6 lore raw

Re: [PATCH v3 1/2] x86: Add support for the pcommit instruction

In reply to Ross Zwisler
Quoted 70 lines · Ross Zwisler

Add support for the new pcommit (persistent commit) instruction. This instruction was announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

The pcommit instruction ensures that data that has been flushed from the processor's cache hierarchy with clwb, clflushopt or clflush is accepted to memory and is durable on the DIMM. The primary use case for this is persistent memory.

This function shows how to properly use clwb/clflushopt/clflush and pcommit with appropriate fencing:

void flush_and_commit_buffer(void *vaddr, unsigned int size)

{
	void *vend = vaddr + size - 1;

	for (; vaddr < vend; vaddr += boot_cpu_data.x86_clflush_size)
		clwb(vaddr);

	/* Flush any possible final partial cacheline */
	clwb(vend);

	/*
	 * sfence to order clwb/clflushopt/clflush cache flushes
	 * mfence via mb() also works
	 */
	wmb();

	/* pcommit and the required sfence for ordering */
	pcommit_sfence();
}

After this function completes the data pointed to by vaddr is has been accepted to memory and will be durable if the vaddr points to persistent memory.

Pcommit must always be ordered by an mfence or sfence, so to help simplify things we include both the pcommit and the required sfence in the alternatives generated by pcommit_sfence(). The other option is to keep them separated, but on platforms that don't support pcommit this would then turn into:

void flush_and_commit_buffer(void *vaddr, unsigned int size)

{
        void *vend = vaddr + size - 1;

        for (; vaddr < vend; vaddr += boot_cpu_data.x86_clflush_size)
                clwb(vaddr);

        /* Flush any possible final partial cacheline */
        clwb(vend);

        /*
         * sfence to order clwb/clflushopt/clflush cache flushes
         * mfence via mb() also works
         */
        wmb();

        nop(); /* from pcommit(), via alternatives */

        /*
         * sfence to order pcommit
         * mfence via mb() also works
         */
        wmb();
}
Quoted 3 lines

This is still correct, but now you've got two fences separated by only a nop. With the commit and the fence together in pcommit_sfence() you avoid the final unneeded fence.

#7 lore raw

[tip:x86/asm] x86: Add support for the pcommit instruction

In reply to Ross Zwisler
x86: Add support for the pcommit instruction
Commit-ID:  a71ef01336f2228dc9d47320492360d6848e591e
Gitweb:     http://git.kernel.org/tip/a71ef01336f2228dc9d47320492360d6848e591e
Author:     Ross Zwisler <[email protected]>
AuthorDate: Tue, 27 Jan 2015 09:53:50 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Thu, 19 Feb 2015 00:06:37 +0100

x86: Add support for the pcommit instruction

Add support for the new pcommit (persistent commit) instruction.
This instruction was announced in the document "Intel
Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

The pcommit instruction ensures that data that has been flushed from the processor's cache hierarchy with clwb, clflushopt or clflush is accepted to memory and is durable on the DIMM. The primary use case for this is persistent memory.

This function shows how to properly use clwb/clflushopt/clflush and pcommit with appropriate fencing:

void flush_and_commit_buffer(void *vaddr, unsigned int size)

{
	void *vend = vaddr + size - 1;

	for (; vaddr < vend; vaddr += boot_cpu_data.x86_clflush_size)
		clwb(vaddr);

	/* Flush any possible final partial cacheline */
	clwb(vend);

	/*
	 * sfence to order clwb/clflushopt/clflush cache flushes
	 * mfence via mb() also works
	 */
	wmb();

	/* pcommit and the required sfence for ordering */
	pcommit_sfence();
}

After this function completes the data pointed to by vaddr is has been accepted to memory and will be durable if the vaddr points to persistent memory.

Pcommit must always be ordered by an mfence or sfence, so to help simplify things we include both the pcommit and the required sfence in the alternatives generated by pcommit_sfence(). The other option is to keep them separated, but on platforms that don't support pcommit this would then turn into:

void flush_and_commit_buffer(void *vaddr, unsigned int size)

{
        void *vend = vaddr + size - 1;

        for (; vaddr < vend; vaddr += boot_cpu_data.x86_clflush_size)
                clwb(vaddr);

        /* Flush any possible final partial cacheline */
        clwb(vend);

        /*
         * sfence to order clwb/clflushopt/clflush cache flushes
         * mfence via mb() also works
         */
        wmb();

        nop(); /* from pcommit(), via alternatives */

        /*
         * sfence to order pcommit
         * mfence via mb() also works
         */
        wmb();
}

This is still correct, but now you've got two fences separated by only a nop. With the commit and the fence together in pcommit_sfence() you avoid the final unneeded fence.

---
 arch/x86/include/asm/cpufeature.h    | 1 +
 arch/x86/include/asm/special_insns.h | 8 ++++++++
 2 files changed, 9 insertions(+)
arch/x86/include/asm/cpufeature.h +1 -0
index 90a5485..d6428ea 100644
@@ -231,6 +231,7 @@
 #define X86_FEATURE_RDSEED	( 9*32+18) /* The RDSEED instruction */
 #define X86_FEATURE_ADX		( 9*32+19) /* The ADCX and ADOX instructions */
 #define X86_FEATURE_SMAP	( 9*32+20) /* Supervisor Mode Access Prevention */
+#define X86_FEATURE_PCOMMIT	( 9*32+22) /* PCOMMIT instruction */
 #define X86_FEATURE_CLFLUSHOPT	( 9*32+23) /* CLFLUSHOPT instruction */
 #define X86_FEATURE_AVX512PF	( 9*32+26) /* AVX-512 Prefetch */
 #define X86_FEATURE_AVX512ER	( 9*32+27) /* AVX-512 Exponential and Reciprocal */
arch/x86/include/asm/special_insns.h +8 -0
index e820c08..d686f9b 100644
@@ -199,6 +199,14 @@ static inline void clflushopt(volatile void *__p)
 		       "+m" (*(volatile char __force *)__p));
 }
 
+static inline void pcommit_sfence(void)
+{
+	alternative(ASM_NOP7,
+		    ".byte 0x66, 0x0f, 0xae, 0xf8\n\t" /* pcommit */
+		    "sfence",
+		    X86_FEATURE_PCOMMIT);
+}
+
 #define nop() asm volatile ("nop")
 
 
#8 lore raw

Re: [tip:x86/asm] x86: Add support for the pcommit instruction

Quoted 8 lines · tip-bot for Ross Zwisler
Commit-ID:  a71ef01336f2228dc9d47320492360d6848e591e
Gitweb:     http://git.kernel.org/tip/a71ef01336f2228dc9d47320492360d6848e591e
Author:     Ross Zwisler <[email protected]>
AuthorDate: Tue, 27 Jan 2015 09:53:50 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Thu, 19 Feb 2015 00:06:37 +0100

x86: Add support for the pcommit instruction

So this breaks the UML build:

/home/mingo/tip/arch/x86/include/asm/special_insns.h: In function ‘pcommit_sfence’:
/home/mingo/tip/arch/x86/include/asm/special_insns.h:218:14: error: expected ‘:’ or ‘)’ before ‘ASM_NOP7’

Thanks,

Ingo

#9 lore raw

Re: [tip:x86/asm] x86: Add support for the pcommit instruction

Quoted 11 lines · Ingo Molnar
Quoted 8 lines · tip-bot for Ross Zwisler
Commit-ID:  a71ef01336f2228dc9d47320492360d6848e591e
Gitweb:     http://git.kernel.org/tip/a71ef01336f2228dc9d47320492360d6848e591e
Author:     Ross Zwisler <[email protected]>
AuthorDate: Tue, 27 Jan 2015 09:53:50 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Thu, 19 Feb 2015 00:06:37 +0100

x86: Add support for the pcommit instruction

Quoted 8 lines

So this breaks the UML build:

/home/mingo/tip/arch/x86/include/asm/special_insns.h: In function ‘pcommit_sfence’:
/home/mingo/tip/arch/x86/include/asm/special_insns.h:218:14: error: expected ‘:’ or ‘)’ before ‘ASM_NOP7’

Thanks,

Ingo

Interesting, it looks like I need to include <asm/nops.h> explicitly for UML. New patch on the way.

Thanks,
- Ross

maintainer
#10 lore raw

Re: [tip:x86/asm] x86: Add support for the pcommit instruction

Quoted 2 lines · Ross Zwisler

Interesting, it looks like I need to include <asm/nops.h> explicitly for UML. New patch on the way.

You'd need to do an incremental fix ontop, though.

Signature · 5 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
#11 lore raw

Re: [tip:x86/asm] x86: Add support for the pcommit instruction

Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Ross Zwisler

Interesting, it looks like I need to include <asm/nops.h> explicitly for UML. New patch on the way.

You'd need to do an incremental fix ontop, though.

Oh, instead of just sending out a new patch that does the include?

Sorry, didn't see this before I sent out v4 of the patch that added the include - Ingo, if you'd rather have a separate patch that adds the include to fix the compile error, please let me know & I can send one out.

#12 lore raw

PATCH v3 2/2 x86: Add support for the clwb instruction

In reply to Ross Zwisler

Add support for the new clwb (cache line write back) instruction. This instruction was announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

The clwb instruction is used to write back the contents of dirtied cache lines to memory without evicting the cache lines from the processor's cache hierarchy. This should be used in favor of clflushopt or clflush in cases where you require the cache line to be written to memory but plan to access the data again in the near future.

One of the main use cases for this is with persistent memory where clwb can be used with pcommit to ensure that data has been accepted to memory and is durable on the DIMM.

This function shows how to properly use clwb/clflushopt/clflush and pcommit with appropriate fencing:

void flush_and_commit_buffer(void *vaddr, unsigned int size)

{
	void *vend = vaddr + size - 1;

	for (; vaddr < vend; vaddr += boot_cpu_data.x86_clflush_size)
		clwb(vaddr);

	/* Flush any possible final partial cacheline */
	clwb(vend);

	/*
	 * sfence to order clwb/clflushopt/clflush cache flushes
	 * mfence via mb() also works
	 */
	wmb();

	/* pcommit and the required sfence for ordering */
	pcommit_sfence();
}

After this function completes the data pointed to by vaddr is has been accepted to memory and will be durable if the vaddr points to persistent memory.

Regarding the details of how the alternatives assembly is set up, we need one additional byte at the beginning of the clflush so that we can flip it into a clflushopt by changing that byte into a 0x66 prefix. Two options are to either insert a 1 byte ASM_NOP1, or to add a 1 byte NOP_DS_PREFIX. Both have no functional effect with the plain clflush, but I've been told that executing a clflush + prefix should be faster than executing a clflush + NOP.

We had to hard code the assembly for clwb because, lacking the ability to assemble the clwb instruction itself, the next closest thing is to have an xsaveopt instruction with a 0x66 prefix. Unfortunately xsaveopt itself is also relatively new, and isn't included by all the GCC versions that the kernel needs to support.

Cc H Peter Anvin <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Borislav Petkov <[email protected]>
---
 arch/x86/include/asm/cpufeature.h    |  1 +
 arch/x86/include/asm/special_insns.h | 14 ++++++++++++++
 2 files changed, 15 insertions(+)
arch/x86/include/asm/cpufeature.h +1 -0
index dfdd689..dc91747 100644
@@ -222,6 +222,7 @@
 #define X86_FEATURE_SMAP	( 9*32+20) /* Supervisor Mode Access Prevention */
 #define X86_FEATURE_PCOMMIT	( 9*32+22) /* PCOMMIT instruction */
 #define X86_FEATURE_CLFLUSHOPT	( 9*32+23) /* CLFLUSHOPT instruction */
+#define X86_FEATURE_CLWB	( 9*32+24) /* CLWB instruction */
 #define X86_FEATURE_AVX512PF	( 9*32+26) /* AVX-512 Prefetch */
 #define X86_FEATURE_AVX512ER	( 9*32+27) /* AVX-512 Exponential and Reciprocal */
 #define X86_FEATURE_AVX512CD	( 9*32+28) /* AVX-512 Conflict Detection */
arch/x86/include/asm/special_insns.h +14 -0
index d686f9b..0772365 100644
@@ -199,6 +199,20 @@ static inline void clflushopt(volatile void *__p)
 		       "+m" (*(volatile char __force *)__p));
 }
 
+static inline void clwb(volatile void *__p)
+{
+	volatile struct { char x[64]; } *p = __p;
+
+	asm volatile(ALTERNATIVE_2(
+		".byte " __stringify(NOP_DS_PREFIX) "; clflush (%[pax])",
+		".byte 0x66; clflush (%[pax])", /* clflushopt (%%rax) */
+		X86_FEATURE_CLFLUSHOPT,
+		".byte 0x66, 0x0f, 0xae, 0x30",  /* clwb (%%rax) */
+		X86_FEATURE_CLWB)
+		: [p] "+m" (*p)
+		: [pax] "a" (p));
+}
+
 static inline void pcommit_sfence(void)
 {
 	alternative(ASM_NOP7,
Signature · 1 line
1.9.3
maintainer Acked-by on v3 2/2
#13 lore raw

Re: [PATCH v3 2/2] x86: Add support for the clwb instruction

Quoted 4 lines · Ross Zwisler

Add support for the new clwb (cache line write back) instruction. This instruction was announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

...

Quoted 18 lines

After this function completes the data pointed to by vaddr is has been accepted to memory and will be durable if the vaddr points to persistent memory.

Regarding the details of how the alternatives assembly is set up, we need one additional byte at the beginning of the clflush so that we can flip it into a clflushopt by changing that byte into a 0x66 prefix. Two options are to either insert a 1 byte ASM_NOP1, or to add a 1 byte NOP_DS_PREFIX. Both have no functional effect with the plain clflush, but I've been told that executing a clflush + prefix should be faster than executing a clflush + NOP.

We had to hard code the assembly for clwb because, lacking the ability to assemble the clwb instruction itself, the next closest thing is to have an xsaveopt instruction with a 0x66 prefix. Unfortunately xsaveopt itself is also relatively new, and isn't included by all the GCC versions that the kernel needs to support.

Quoted 5 lines
Cc H Peter Anvin <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Borislav Petkov <[email protected]>
Signature · 5 lines
Regards/Gruss,
    Boris.

ECO tip #101: Trim your mails when you reply.
--
Acked-by on v3 2/2
#14 lore raw

Re: [PATCH v3 2/2] x86: Add support for the clwb instruction

In reply to Ross Zwisler
Quoted 13 lines · Ross Zwisler

Add support for the new clwb (cache line write back) instruction. This instruction was announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

The clwb instruction is used to write back the contents of dirtied cache lines to memory without evicting the cache lines from the processor's cache hierarchy. This should be used in favor of clflushopt or clflush in cases where you require the cache line to be written to memory but plan to access the data again in the near future.

Quoted 3 lines

One of the main use cases for this is with persistent memory where clwb can be used with pcommit to ensure that data has been accepted to memory and is durable on the DIMM.

#15 lore raw

[tip:x86/asm] x86: Add support for the clwb instruction

In reply to Ross Zwisler
x86: Add support for the clwb instruction
Commit-ID:  3b68983dc66c61da3ab4191b891084a7ab09e3e1
Gitweb:     http://git.kernel.org/tip/3b68983dc66c61da3ab4191b891084a7ab09e3e1
Author:     Ross Zwisler <[email protected]>
AuthorDate: Tue, 27 Jan 2015 09:53:51 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Thu, 19 Feb 2015 00:06:38 +0100

x86: Add support for the clwb instruction

Add support for the new clwb (cache line write back) instruction. This instruction was announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

The clwb instruction is used to write back the contents of dirtied cache lines to memory without evicting the cache lines from the processor's cache hierarchy. This should be used in favor of clflushopt or clflush in cases where you require the cache line to be written to memory but plan to access the data again in the near future.

One of the main use cases for this is with persistent memory where clwb can be used with pcommit to ensure that data has been accepted to memory and is durable on the DIMM.

This function shows how to properly use clwb/clflushopt/clflush and pcommit with appropriate fencing:

void flush_and_commit_buffer(void *vaddr, unsigned int size)

{
	void *vend = vaddr + size - 1;

	for (; vaddr < vend; vaddr += boot_cpu_data.x86_clflush_size)
		clwb(vaddr);

	/* Flush any possible final partial cacheline */
	clwb(vend);

	/*
	 * sfence to order clwb/clflushopt/clflush cache flushes
	 * mfence via mb() also works
	 */
	wmb();

	/* pcommit and the required sfence for ordering */
	pcommit_sfence();
}

After this function completes the data pointed to by vaddr is has been accepted to memory and will be durable if the vaddr points to persistent memory.

Regarding the details of how the alternatives assembly is set up, we need one additional byte at the beginning of the clflush so that we can flip it into a clflushopt by changing that byte into a 0x66 prefix. Two options are to either insert a 1 byte ASM_NOP1, or to add a 1 byte NOP_DS_PREFIX. Both have no functional effect with the plain clflush, but I've been told that executing a clflush + prefix should be faster than executing a clflush + NOP.

We had to hard code the assembly for clwb because, lacking the ability to assemble the clwb instruction itself, the next closest thing is to have an xsaveopt instruction with a 0x66 prefix. Unfortunately xsaveopt itself is also relatively new, and isn't included by all the GCC versions that the kernel needs to support.

---
 arch/x86/include/asm/cpufeature.h    |  1 +
 arch/x86/include/asm/special_insns.h | 14 ++++++++++++++
 2 files changed, 15 insertions(+)
arch/x86/include/asm/cpufeature.h +1 -0
index d6428ea..bc96e78 100644
@@ -233,6 +233,7 @@
 #define X86_FEATURE_SMAP	( 9*32+20) /* Supervisor Mode Access Prevention */
 #define X86_FEATURE_PCOMMIT	( 9*32+22) /* PCOMMIT instruction */
 #define X86_FEATURE_CLFLUSHOPT	( 9*32+23) /* CLFLUSHOPT instruction */
+#define X86_FEATURE_CLWB	( 9*32+24) /* CLWB instruction */
 #define X86_FEATURE_AVX512PF	( 9*32+26) /* AVX-512 Prefetch */
 #define X86_FEATURE_AVX512ER	( 9*32+27) /* AVX-512 Exponential and Reciprocal */
 #define X86_FEATURE_AVX512CD	( 9*32+28) /* AVX-512 Conflict Detection */
arch/x86/include/asm/special_insns.h +14 -0
index d686f9b..0772365 100644
@@ -199,6 +199,20 @@ static inline void clflushopt(volatile void *__p)
 		       "+m" (*(volatile char __force *)__p));
 }
 
+static inline void clwb(volatile void *__p)
+{
+	volatile struct { char x[64]; } *p = __p;
+
+	asm volatile(ALTERNATIVE_2(
+		".byte " __stringify(NOP_DS_PREFIX) "; clflush (%[pax])",
+		".byte 0x66; clflush (%[pax])", /* clflushopt (%%rax) */
+		X86_FEATURE_CLFLUSHOPT,
+		".byte 0x66, 0x0f, 0xae, 0x30",  /* clwb (%%rax) */
+		X86_FEATURE_CLWB)
+		: [p] "+m" (*p)
+		: [pax] "a" (p));
+}
+
 static inline void pcommit_sfence(void)
 {
 	alternative(ASM_NOP7,
#16 lore raw

Re: [tip:x86/asm] x86: Add support for the clwb instruction

Quoted 70 lines · tip-bot for Ross Zwisler
Commit-ID:  3b68983dc66c61da3ab4191b891084a7ab09e3e1
Gitweb:     http://git.kernel.org/tip/3b68983dc66c61da3ab4191b891084a7ab09e3e1
Author:     Ross Zwisler <[email protected]>
AuthorDate: Tue, 27 Jan 2015 09:53:51 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Thu, 19 Feb 2015 00:06:38 +0100

x86: Add support for the clwb instruction

Add support for the new clwb (cache line write back) instruction. This instruction was announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

The clwb instruction is used to write back the contents of dirtied cache lines to memory without evicting the cache lines from the processor's cache hierarchy. This should be used in favor of clflushopt or clflush in cases where you require the cache line to be written to memory but plan to access the data again in the near future.

One of the main use cases for this is with persistent memory where clwb can be used with pcommit to ensure that data has been accepted to memory and is durable on the DIMM.

This function shows how to properly use clwb/clflushopt/clflush and pcommit with appropriate fencing:

void flush_and_commit_buffer(void *vaddr, unsigned int size)

{
	void *vend = vaddr + size - 1;

	for (; vaddr < vend; vaddr += boot_cpu_data.x86_clflush_size)
		clwb(vaddr);

	/* Flush any possible final partial cacheline */
	clwb(vend);

	/*
	 * sfence to order clwb/clflushopt/clflush cache flushes
	 * mfence via mb() also works
	 */
	wmb();

	/* pcommit and the required sfence for ordering */
	pcommit_sfence();
}

After this function completes the data pointed to by vaddr is has been accepted to memory and will be durable if the vaddr points to persistent memory.

Regarding the details of how the alternatives assembly is set up, we need one additional byte at the beginning of the clflush so that we can flip it into a clflushopt by changing that byte into a 0x66 prefix. Two options are to either insert a 1 byte ASM_NOP1, or to add a 1 byte NOP_DS_PREFIX. Both have no functional effect with the plain clflush, but I've been told that executing a clflush + prefix should be faster than executing a clflush + NOP.

We had to hard code the assembly for clwb because, lacking the ability to assemble the clwb instruction itself, the next closest thing is to have an xsaveopt instruction with a 0x66 prefix. Unfortunately xsaveopt itself is also relatively new, and isn't included by all the GCC versions that the kernel needs to support.

Quoted 7 lines

Ping on this patch - it looks like the pcommit patch is in the tip tree, but this one is missing?

I'm looking at the tree as of:
9a760fbbdc7 "Merge branch 'tools/kvm'"

#17 lore raw

Re: [tip:x86/asm] x86: Add support for the clwb instruction

Quoted 79 lines · Ross Zwisler
Quoted 70 lines · tip-bot for Ross Zwisler
Commit-ID:  3b68983dc66c61da3ab4191b891084a7ab09e3e1
Gitweb:     http://git.kernel.org/tip/3b68983dc66c61da3ab4191b891084a7ab09e3e1
Author:     Ross Zwisler <[email protected]>
AuthorDate: Tue, 27 Jan 2015 09:53:51 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Thu, 19 Feb 2015 00:06:38 +0100

x86: Add support for the clwb instruction

Add support for the new clwb (cache line write back) instruction. This instruction was announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

The clwb instruction is used to write back the contents of dirtied cache lines to memory without evicting the cache lines from the processor's cache hierarchy. This should be used in favor of clflushopt or clflush in cases where you require the cache line to be written to memory but plan to access the data again in the near future.

One of the main use cases for this is with persistent memory where clwb can be used with pcommit to ensure that data has been accepted to memory and is durable on the DIMM.

This function shows how to properly use clwb/clflushopt/clflush and pcommit with appropriate fencing:

void flush_and_commit_buffer(void *vaddr, unsigned int size)

{
	void *vend = vaddr + size - 1;

	for (; vaddr < vend; vaddr += boot_cpu_data.x86_clflush_size)
		clwb(vaddr);

	/* Flush any possible final partial cacheline */
	clwb(vend);

	/*
	 * sfence to order clwb/clflushopt/clflush cache flushes
	 * mfence via mb() also works
	 */
	wmb();

	/* pcommit and the required sfence for ordering */
	pcommit_sfence();
}

After this function completes the data pointed to by vaddr is has been accepted to memory and will be durable if the vaddr points to persistent memory.

Regarding the details of how the alternatives assembly is set up, we need one additional byte at the beginning of the clflush so that we can flip it into a clflushopt by changing that byte into a 0x66 prefix. Two options are to either insert a 1 byte ASM_NOP1, or to add a 1 byte NOP_DS_PREFIX. Both have no functional effect with the plain clflush, but I've been told that executing a clflush + prefix should be faster than executing a clflush + NOP.

We had to hard code the assembly for clwb because, lacking the ability to assemble the clwb instruction itself, the next closest thing is to have an xsaveopt instruction with a 0x66 prefix. Unfortunately xsaveopt itself is also relatively new, and isn't included by all the GCC versions that the kernel needs to support.

Quoted 7 lines
Quoted 2 lines

Ping on this patch - it looks like the pcommit patch is in the tip tree, but this one is missing?

Yeah, I applied it initially, then had some reservations about it - but those are now resolved so I've applied it to tip:x86/asm again.

Thanks,

Ingo

#18 lore raw

[tip:x86/asm] x86/asm: Add support for the CLWB instruction

In reply to Ross Zwisler
x86/asm: Add support for the CLWB instruction
Commit-ID:  d9dc64f30abe42f71bc7e9eb9d38c41006cf39f9
Gitweb:     http://git.kernel.org/tip/d9dc64f30abe42f71bc7e9eb9d38c41006cf39f9
Author:     Ross Zwisler <[email protected]>
AuthorDate: Tue, 27 Jan 2015 09:53:51 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 3 Apr 2015 06:56:38 +0200

x86/asm: Add support for the CLWB instruction

Add support for the new CLWB (cache line write back) instruction. This instruction was announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

The CLWB instruction is used to write back the contents of dirtied cache lines to memory without evicting the cache lines from the processor's cache hierarchy. This should be used in favor of clflushopt or clflush in cases where you require the cache line to be written to memory but plan to access the data again in the near future.

One of the main use cases for this is with persistent memory where CLWB can be used with PCOMMIT to ensure that data has been accepted to memory and is durable on the DIMM.

This function shows how to properly use CLWB/CLFLUSHOPT/CLFLUSH and PCOMMIT with appropriate fencing:

void flush_and_commit_buffer(void *vaddr, unsigned int size)

{
	void *vend = vaddr + size - 1;

	for (; vaddr < vend; vaddr += boot_cpu_data.x86_clflush_size)
		clwb(vaddr);

	/* Flush any possible final partial cacheline */
	clwb(vend);

	/*
	 * Use SFENCE to order CLWB/CLFLUSHOPT/CLFLUSH cache flushes.
	 * (MFENCE via mb() also works)
	 */
	wmb();

	/* PCOMMIT and the required SFENCE for ordering */
	pcommit_sfence();
}

After this function completes the data pointed to by vaddr is has been accepted to memory and will be durable if the vaddr points to persistent memory.

Regarding the details of how the alternatives assembly is set up, we need one additional byte at the beginning of the CLFLUSH so that we can flip it into a CLFLUSHOPT by changing that byte into a 0x66 prefix. Two options are to either insert a 1 byte ASM_NOP1, or to add a 1 byte NOP_DS_PREFIX. Both have no functional effect with the plain CLFLUSH, but I've been told that executing a CLFLUSH + prefix should be faster than executing a CLFLUSH + NOP.

We had to hard code the assembly for CLWB because, lacking the ability to assemble the CLWB instruction itself, the next closest thing is to have an xsaveopt instruction with a 0x66 prefix. Unfortunately XSAVEOPT itself is also relatively new, and isn't included by all the GCC versions that the kernel needs to support.

---
 arch/x86/include/asm/cpufeature.h    |  1 +
 arch/x86/include/asm/special_insns.h | 14 ++++++++++++++
 2 files changed, 15 insertions(+)
arch/x86/include/asm/cpufeature.h +1 -0
index 0f7a5a1..854c04b 100644
@@ -233,6 +233,7 @@
 #define X86_FEATURE_SMAP	( 9*32+20) /* Supervisor Mode Access Prevention */
 #define X86_FEATURE_PCOMMIT	( 9*32+22) /* PCOMMIT instruction */
 #define X86_FEATURE_CLFLUSHOPT	( 9*32+23) /* CLFLUSHOPT instruction */
+#define X86_FEATURE_CLWB	( 9*32+24) /* CLWB instruction */
 #define X86_FEATURE_AVX512PF	( 9*32+26) /* AVX-512 Prefetch */
 #define X86_FEATURE_AVX512ER	( 9*32+27) /* AVX-512 Exponential and Reciprocal */
 #define X86_FEATURE_AVX512CD	( 9*32+28) /* AVX-512 Conflict Detection */
arch/x86/include/asm/special_insns.h +14 -0
index 2ec1a53..aeb4666e 100644
@@ -201,6 +201,20 @@ static inline void clflushopt(volatile void *__p)
 		       "+m" (*(volatile char __force *)__p));
 }
 
+static inline void clwb(volatile void *__p)
+{
+	volatile struct { char x[64]; } *p = __p;
+
+	asm volatile(ALTERNATIVE_2(
+		".byte " __stringify(NOP_DS_PREFIX) "; clflush (%[pax])",
+		".byte 0x66; clflush (%[pax])", /* clflushopt (%%rax) */
+		X86_FEATURE_CLFLUSHOPT,
+		".byte 0x66, 0x0f, 0xae, 0x30",  /* clwb (%%rax) */
+		X86_FEATURE_CLWB)
+		: [p] "+m" (*p)
+		: [pax] "a" (p));
+}
+
 static inline void pcommit_sfence(void)
 {
 	alternative(ASM_NOP7,
#19 lore raw
In reply to Ross Zwisler
Quoted 21 lines · Ross Zwisler

This patch set adds support for two new persistent memory instructions, pcommit and clwb. These instructions were announced in the document "Intel Architecture Instruction Set Extensions Programming Reference" with reference number 319433-022.

https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

These patches apply cleanly to v3.19-rc6.

Changes from v2:

- Added instruction descriptions and flows to the patch descriptions.
- Added needed sfence to pcommit alternatives assembly. The inline function is now called pcommit_sfence(). If pcommit is not supported on the platform both the pcommit and the sfence will be nops.

Cc H Peter Anvin <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Borislav Petkov <[email protected]>
Quoted 7 lines

Ross Zwisler (2):
x86: Add support for the pcommit instruction
x86: Add support for the clwb instruction

 arch/x86/include/asm/cpufeature.h    |  2 ++
 arch/x86/include/asm/special_insns.h | 22 ++++++++++++++++++++++
 2 files changed, 24 insertions(+)

Ping? :)