membarrier: provide core serialization

Add a new MEMBARRIER_FLAG_SYNC_CORE flag to the membarrier system call. It allows membarrier to issue core serializing barriers in addition to memory barriers on target threads whenever a membarrier command is performed.

It is relevant for reclaim of JIT code, which requires to issue core serializing barriers on all threads running on behalf of a process after ensuring the old code is not visible anymore, before re-using memory for new code.

The new MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED used with MEMBARRIER_FLAG_SYNC_CORE flag registers the current process as requiring core serialization. It may block. It can be used to ensure MEMBARRIER_CMD_PRIVATE_EXPEDITED never blocks, even the first time it is invoked by a process with the MEMBARRIER_FLAG_SYNC_CORE flag.

* Scheduler Overhead Benchmarks

Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Linux v4.13-rc6

Inter-thread scheduling
taskset 01 ./perf bench sched pipe -T

Avg. usecs/op Std.Dev. usecs/op

Before this change:         2.55                   0.10
With this change:           2.49                   0.08
SYNC_CORE processes:        2.70                   0.10

Inter-process scheduling
taskset 01 ./perf bench sched pipe

Before this change:         2.93                   0.13
With this change:           2.93                   0.13
SYNC_CORE processes:        3.20                   0.06

Changes since v2:
- Rename MEMBARRIER_CMD_REGISTER_SYNC_CORE to
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,

- Introduce the "MEMBARRIER_FLAG_SYNC_CORE" flag.
- Introduce CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE, only implemented by x86 32/64 initially.
- Introduce arch_membarrier_user_icache_flush, a no-op on x86 32/64, which can be implemented on architectures with incoherent data and instruction caches. It is associated with CONFIG_ARCH_HAS_MEMBARRIER_USER_ICACHE_FLUSH.

Quoted 36 lines · Mathieu Desnoyers

Add a new MEMBARRIER_FLAG_SYNC_CORE flag to the membarrier system call. It allows membarrier to issue core serializing barriers in addition to memory barriers on target threads whenever a membarrier command is performed.

It is relevant for reclaim of JIT code, which requires to issue core serializing barriers on all threads running on behalf of a process after ensuring the old code is not visible anymore, before re-using memory for new code.

The new MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED used with MEMBARRIER_FLAG_SYNC_CORE flag registers the current process as requiring core serialization. It may block. It can be used to ensure MEMBARRIER_CMD_PRIVATE_EXPEDITED never blocks, even the first time it is invoked by a process with the MEMBARRIER_FLAG_SYNC_CORE flag.

* Scheduler Overhead Benchmarks

Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Linux v4.13-rc6

Inter-thread scheduling
taskset 01 ./perf bench sched pipe -T

Avg. usecs/op Std.Dev. usecs/op

Before this change:         2.55                   0.10
With this change:           2.49                   0.08
SYNC_CORE processes:        2.70                   0.10

Inter-process scheduling
taskset 01 ./perf bench sched pipe

Before this change:         2.93                   0.13
With this change:           2.93                   0.13
SYNC_CORE processes:        3.20                   0.06

Quoted 3 lines

Changes since v2:
- Rename MEMBARRIER_CMD_REGISTER_SYNC_CORE to
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,

I'm still not convinced that this registration step is needed (at least for arm, power and x86), but my previous comments were ignored.

Quoted 8 lines

- Introduce the "MEMBARRIER_FLAG_SYNC_CORE" flag.
- Introduce CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE, only implemented by x86 32/64 initially.
- Introduce arch_membarrier_user_icache_flush, a no-op on x86 32/64, which can be implemented on architectures with incoherent data and instruction caches. It is associated with CONFIG_ARCH_HAS_MEMBARRIER_USER_ICACHE_FLUSH.

Given that MEMBARRIER_FLAG_SYNC_CORE is about flushing the internal CPU pipeline (iiuc), could we rename this so that it doesn't mention the I-cache, please? I-cache flushing is a very different operation on most architectures I'm aware of, and on arm64 it's even available to userspace (and broadcast in hardware to other cores).

Quoted 41 lines · ----- On Sep 1, 2017, at 12:25 PM, Will Deacon [email protected] wrote:

Quoted 39 lines · Mathieu Desnoyers

Add a new MEMBARRIER_FLAG_SYNC_CORE flag to the membarrier system call. It allows membarrier to issue core serializing barriers in addition to memory barriers on target threads whenever a membarrier command is performed.

It is relevant for reclaim of JIT code, which requires to issue core serializing barriers on all threads running on behalf of a process after ensuring the old code is not visible anymore, before re-using memory for new code.

The new MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED used with MEMBARRIER_FLAG_SYNC_CORE flag registers the current process as requiring core serialization. It may block. It can be used to ensure MEMBARRIER_CMD_PRIVATE_EXPEDITED never blocks, even the first time it is invoked by a process with the MEMBARRIER_FLAG_SYNC_CORE flag.

* Scheduler Overhead Benchmarks

Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Linux v4.13-rc6

Inter-thread scheduling
taskset 01 ./perf bench sched pipe -T

Avg. usecs/op Std.Dev. usecs/op

Before this change:         2.55                   0.10
With this change:           2.49                   0.08
SYNC_CORE processes:        2.70                   0.10

Inter-process scheduling
taskset 01 ./perf bench sched pipe

Before this change:         2.93                   0.13
With this change:           2.93                   0.13
SYNC_CORE processes:        3.20                   0.06

Changes since v2:
- Rename MEMBARRIER_CMD_REGISTER_SYNC_CORE to
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,

Quoted 2 lines

I'm still not convinced that this registration step is needed (at least for arm, power and x86), but my previous comments were ignored.

I mistakenly thought that your previous comments were addressed in other legs of the previous thread, sorry about that.

Let's take x86 as an example. The private expedited membarrier command iterates on all cpu runqueues, checking if rq->curr->mm match current->mm, and only IPI if it matches.

We can very well have a CPU for which the scheduler goes back and forth between user-space thread and a kernel thread, in which case the mm state is kept as is, and rq->curr->mm is temporarily saved into rq->curr->active_mm.

This means that while that CPU is executing a kthread, we won't send any IPI that that CPU, but it could then schedule back a thread belonging to the original process, and then we go back executing user-space code without having issued any kind of core serializing barrier (assuming we return to userspace with sysexit).

Now about arm64, given that as you say it issues a core serializing barrier when returning to user-space, and has a strong barrier in switch_to, this means that the explicit sync_core() in sched_in is not needed.

However, AFAIU, arm64 does not guarantee consistent data and instruction caches.

I'm actually trying to wrap my head around what would be the sequence of operations of a JIT trying to reclaim memory. Can we combine core serialization and instruction cache flushing into a single system call invocation, or we need to split this into two separate operations ?

The JIT reclaim usage scheme I envision is:

- userspace unpublish all reference to old code,
- userspace ensure no thread use the old code anymore,
- sys_membarrier
- for each executing threads
- issue core serializing barrier
- userspace use a separate system call to issue data cache flush for the modified range
- sys_membarrier
- for each executing threads
- issue instruction cache flush

So my current thinking is that we may need to change the membarrier system call so one command serializes the core, and a separate command issues cache flush.

Hi Mathieu,

Quoted 43 lines · Mathieu Desnoyers

Quoted 41 lines · ----- On Sep 1, 2017, at 12:25 PM, Will Deacon [email protected] wrote:

Quoted 39 lines · Mathieu Desnoyers

Add a new MEMBARRIER_FLAG_SYNC_CORE flag to the membarrier system call. It allows membarrier to issue core serializing barriers in addition to memory barriers on target threads whenever a membarrier command is performed.

It is relevant for reclaim of JIT code, which requires to issue core serializing barriers on all threads running on behalf of a process after ensuring the old code is not visible anymore, before re-using memory for new code.

The new MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED used with MEMBARRIER_FLAG_SYNC_CORE flag registers the current process as requiring core serialization. It may block. It can be used to ensure MEMBARRIER_CMD_PRIVATE_EXPEDITED never blocks, even the first time it is invoked by a process with the MEMBARRIER_FLAG_SYNC_CORE flag.

* Scheduler Overhead Benchmarks

Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
Linux v4.13-rc6

Inter-thread scheduling
taskset 01 ./perf bench sched pipe -T

Avg. usecs/op Std.Dev. usecs/op

Before this change:         2.55                   0.10
With this change:           2.49                   0.08
SYNC_CORE processes:        2.70                   0.10

Inter-process scheduling
taskset 01 ./perf bench sched pipe

Before this change:         2.93                   0.13
With this change:           2.93                   0.13
SYNC_CORE processes:        3.20                   0.06

Changes since v2:
- Rename MEMBARRIER_CMD_REGISTER_SYNC_CORE to
MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED,

Quoted 5 lines

Quoted 2 lines

I'm still not convinced that this registration step is needed (at least for arm, power and x86), but my previous comments were ignored.

I mistakenly thought that your previous comments were addressed in other legs of the previous thread, sorry about that.

No problem, thanks for replying this time!

Quoted 10 lines

Let's take x86 as an example. The private expedited membarrier command iterates on all cpu runqueues, checking if rq->curr->mm match current->mm, and only IPI if it matches.

We can very well have a CPU for which the scheduler goes back and forth between user-space thread and a kernel thread, in which case the mm state is kept as is, and rq->curr->mm is temporarily saved into rq->curr->active_mm.

Quoted 6 lines

This means that while that CPU is executing a kthread, we won't send any IPI that that CPU, but it could then schedule back a thread belonging to the original process, and then we go back executing user-space code without having issued any kind of core serializing barrier (assuming we return to userspace with sysexit).

Right, ok. I forgot about Andy's sysexit optimisation on x86.

Quoted 4 lines

Now about arm64, given that as you say it issues a core serializing barrier when returning to user-space, and has a strong barrier in switch_to, this means that the explicit sync_core() in sched_in is not needed.

Good, that's what I thought.

Quoted 3 lines

However, AFAIU, arm64 does not guarantee consistent data and instruction caches.

Correct, but:

* On 32-bit arm, we have a syscall to do that (and this is already used by JITs and things like __builtin_clear_cache)

* On arm64, cache maintenance instructions are directly available to userspace

In both cases, the maintenance is broadcast by the hardware to all CPUs. The only part that cannot be broadcast is the pipeline flush, which is the part we need to do above and is implicit on exception return.

Quoted 5 lines

I'm actually trying to wrap my head around what would be the sequence of operations of a JIT trying to reclaim memory. Can we combine core serialization and instruction cache flushing into a single system call invocation, or we need to split this into two separate operations ?

I think that cache-flushing and pipeline-flushing should be separated, as they tend to be in the CPU architectures I'm familiar with.

Quoted 14 lines

The JIT reclaim usage scheme I envision is:

- userspace unpublish all reference to old code,
- userspace ensure no thread use the old code anymore,
- sys_membarrier
- for each executing threads
- issue core serializing barrier
- userspace use a separate system call to issue data cache flush for the modified range
- sys_membarrier
- for each executing threads
- issue instruction cache flush

Quoted 3 lines

So my current thinking is that we may need to change the membarrier system call so one command serializes the core, and a separate command issues cache flush.

Yeah, and the sequence is slightly different I think, as we need the pipeline flush to come *after* the I-cache invalidation (otherwise the stale instructions can just be refetched).

If you're at LPC in a week's time, this might be a good thing to sit down and bash our heads against (espec. if we can grab PPC and x86 folks too).

Will

Quoted 7 lines

[Mathieu: ]

Assuming we don't need a sync core before updating the old code, an aggressive approach would be:

reclaim and re-use (aggressive):

Quoted 11 lines

1- userspace unpublish all reference to old code,
2- userspace ensure no thread use the old code anymore (e.g. URCU), 3- userspace updates old code -> new code
4- issue data cache flush for the modified range (if needed) 5- sys_membarrier
- for each executing threads
- issue core serializing barrier
6- issue instruction cache flush for the modified range (if needed)
(may be required on all active threads on some architectures)
7- userspace publish reference to new code

My assumption was that right sequence here, at least on Aarch64, is to do 5 and 6 in the opposite order; flush the icache,which I believe can be done from the thread that wrote the code, and then issue a sys_membarrier for the core serializing barrier.

It would be useful to get that clarified.

What's the status of MEMBARRIER_FLAG_SYNC_CORE? The discussion I saw left it unclear whether this would be a separate flag, or included by default. Did I miss something? I think we're fine with either, but we do have s strong interest in getting this in in some form...
I also believe we're fine with MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED. And that seems to me like a reasonable way to deal with the added overhead.

Thanks!

Quoted 31 lines · Will Deacon

Quoted 18 lines · Hans Boehm

Quoted 7 lines

[Mathieu: ]

Assuming we don't need a sync core before updating the old code, an aggressive approach would be:

reclaim and re-use (aggressive):

Quoted 10 lines

1- userspace unpublish all reference to old code,
2- userspace ensure no thread use the old code anymore (e.g. URCU), 3- userspace updates old code -> new code
4- issue data cache flush for the modified range (if needed) 5- sys_membarrier
- for each executing threads
- issue core serializing barrier
6- issue instruction cache flush for the modified range (if needed) (may be required on all active threads on some architectures) 7- userspace publish reference to new code

Quoted 6 lines

My assumption was that right sequence here, at least on Aarch64, is to do 5 and 6 in the opposite order; flush the icache,which I believe can be done from the thread that wrote the code, and then issue a sys_membarrier for the core serializing barrier.

It would be useful to get that clarified.

FWIW, Mathieu and I spent a while talking about this during LPC last week and ended up agreeing that the ISB (core serialisation) is required *after* the cache-maintenance to publish the new code has completed.

Will

Hi Hans,

I'm currently making sure the MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED command makes its way into the 4.14 kernel before the end of the release candidates. Once that is done, I plan to post a patch adding a new MEMBARRIER_FLAG_SYNC_CORE flag for the 4.15 merge window.

I have done a bit of research on the various architecture requirements for core serialization. Here are my findings so far about instructions providing core serialization on the main architectures supported by Linux.

There are two places where we need it: in the interrupt handler for the membarrier IPI, and between scheduler execution (which can change the current "mm") and return to user-space.

Please let me know if I missed anything.

x86: iret, cpuid, wbinvd -> iret currently provides core serialization when going back to userspace and at the end of the IPI. There are plans to implement a return path without iret in the future, in which case I would need to issue an explicit "cpuid" instruction (sync_core()) in switch_mm() if the process is registered with MEMBARRIER_FLAG_SYNC_CORE.

arm32: returning to user-space provides core serialization. Same at the end of membarrier IPI (to be confirmed). aarch64: ERET instruction used when returning to user-space provides core sync. Same at the end of membarrier IPI (to be confirmed).

parisc: core serialization is ensured by issuing at least 7 instructions. We should have at least that when going back to user-space (to be confirmed). Similar for IPI.
[ https://parisc.wiki.kernel.org/images-parisc/6/68/Pa11_acd.pdf |
https://parisc.wiki.kernel.org/images-parisc/6/68/Pa11_acd.pdf ] 5-152

mips: eret instruction used when going back to user-space provides core sync on all SMP architectures. Probably same for IPI (to be confirmed).
[ https://www.cs.cornell.edu/courses/cs3410/2008fa/MIPS_Vol2.pdf |
https://www.cs.cornell.edu/courses/cs3410/2008fa/MIPS_Vol2.pdf ] p. 121
on R3k and TX39XX, rfe is used instead, but those are uniprocessor, so they do not matter.
[ http://os161.eecs.harvard.edu/documentation/sys161/mips.html |
http://os161.eecs.harvard.edu/documentation/sys161/mips.html ]

sparc: seems to require an explicit "flush" instruction followed by at most 5 instructions to perform core serialization. Not sure if implied by return to user-space in any way.

Quoted 17 lines · Mathieu Desnoyers

Hi Hans,

I'm currently making sure the MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED command makes its way into the 4.14 kernel before the end of the release candidates. Once that is done, I plan to post a patch adding a new MEMBARRIER_FLAG_SYNC_CORE flag for the 4.15 merge window.

I have done a bit of research on the various architecture requirements for core serialization. Here are my findings so far about instructions providing core serialization on the main architectures supported by Linux.

There are two places where we need it: in the interrupt handler for the membarrier IPI, and between scheduler execution (which can change the current "mm") and return to user-space.

Quoted 8 lines

Please let me know if I missed anything.

x86: iret, cpuid, wbinvd -> iret currently provides core serialization when going back to userspace and at the end of the IPI. There are plans to implement a return path without iret in the future, in which case I would need to issue an explicit "cpuid" instruction (sync_core()) in switch_mm() if the process is registered with MEMBARRIER_FLAG_SYNC_CORE.

I would much prefer setting a TIF flag that forces the IRET path instead of doing additional work in switch_mm().

Quoted 4 lines

arm32: returning to user-space provides core serialization. Same at the end of membarrier IPI (to be confirmed). aarch64: ERET instruction used when returning to user-space provides core sync. Same at the end of membarrier IPI (to be confirmed).

I thought Will already confirmed ERET did what we need, no?