Memory synchronization improvements
cover Memory synchronization improvements
Hi Jens,
The two patches in this series reduce the number of memory barrier calls from liburing and also fix a few memory synchronization bugs. Please consider these patches for the official liburing git repository.
Thanks,
Bart.
Bart Van Assche (2):
__io_uring_get_cqe(): Use io_uring_for_each_cqe() Fix the use of memory barriers
man/io_uring_setup.2 | 6 ++- src/barrier.h | 87 +++++++++++++++++++++++++++++++++++++++++-- src/liburing.h | 15 +++----- src/queue.c | 48 ++++++------------------ test/io_uring_enter.c | 8 ++-- 5 files changed, 110 insertions(+), 54 deletions(-)
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
PATCH 1/2 __io_uring_get_cqe(): Use io_uring_for_each_cqe()
Use io_uring_for_each_cqe() inside __io_uring_get_cqe() such that it becomes possible to test the io_uring_for_each_cqe() implementation from inside the liburing project.
--- src/queue.c | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-)
src/queue.c +3 -15
@@ -14,26 +14,14 @@ static int __io_uring_get_cqe(struct io_uring *ring, struct io_uring_cqe **cqe_ptr, int wait) { - struct io_uring_cq *cq = &ring->cq; - const unsigned mask = *cq->kring_mask; unsigned head; int ret; - *cqe_ptr = NULL; - head = *cq->khead; do { - /* - * It's necessary to use a read_barrier() before reading - * the CQ tail, since the kernel updates it locklessly. The - * kernel has the matching store barrier for the update. The - * kernel also ensures that previous stores to CQEs are ordered - * with the tail update. - */ - read_barrier(); - if (head != *cq->ktail) { - *cqe_ptr = &cq->cqes[head & mask]; + io_uring_for_each_cqe(ring, head, *cqe_ptr) + break; + if (*cqe_ptr) break; - } if (!wait) break; ret = io_uring_enter(ring->ring_fd, 0, 1,
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
PATCH 2/2 Fix the use of memory barriers
In reply to Bart Van AsscheIntroduce the smp_load_acquire() and smp_store_release() macros. Fix synchronization in io_uring_cq_advance() and __io_uring_get_cqe(). Remove a superfluous local variable, if-test and write barrier from __io_uring_submit(). Remove a superfluous barrier from test/io_uring_enter.c.
--- man/io_uring_setup.2 | 6 ++- src/barrier.h | 87 +++++++++++++++++++++++++++++++++++++++++-- src/liburing.h | 15 +++----- src/queue.c | 30 ++++----------- test/io_uring_enter.c | 8 ++-- 5 files changed, 107 insertions(+), 39 deletions(-)
man/io_uring_setup.2 +5 -1
@@ -97,7 +97,11 @@ call with the following code sequence: .in +4n .EX -read_barrier(); +/* + * Ensure that the wakeup flag is read after the tail pointer has been + * written. + */ +smp_mb(); if (*sq_ring->flags & IORING_SQ_NEED_WAKEUP) io_uring_enter(fd, 0, 0, IORING_ENTER_SQ_WAKEUP); .EE
src/barrier.h +83 -4
99 lines not on this page · raw message
src/liburing.h +6 -9
@@ -88,11 +88,10 @@ extern int io_uring_register_eventfd(struct io_uring *ring, int fd);extern int io_uring_unregister_eventfd (struct io_uring *ring);#define io_uring_for_each_cqe (ring, head, cqe) \ +/* smp_load_acquire() enforces the order of tail and CQE reads. */ \ for (head = *(ring)->cq.khead; \ -/* See read_barrier() explanation in __io_uring_get_cqe() */ \ - ({read_barrier (); \ - cqe = (head != *(ring)->cq.ktail ? \ - &(ring)->cq.cqes[head & (*(ring)->cq.kring_mask)] :NULL );}); \ + (cqe = (head != smp_load_acquire((ring)->cq.ktail) ? \ + &(ring)->cq.cqes[head & (*(ring)->cq.kring_mask)] : NULL)); \ head++) \
@@ -105,13 +104,11 @@ static inline void io_uring_cq_advance(struct io_uring *ring,if (nr) {struct io_uring_cq *cq = &ring->cq; - (*cq->khead) += nr; -/* -* Ensure that the kernel sees our new head, the kernel has -* the matching read barrier. +* Ensure that the kernel only sees the new value of the head +* index after the CQEs have been read. */ -write_barrier (); +smp_store_release (cq->khead, *cq->khead + nr); } }
src/queue.c +8 -22
@@ -77,7 +77,7 @@ static int __io_uring_submit(struct io_uring *ring, unsigned wait_nr) {struct io_uring_sq *sq = &ring->sq;const unsigned mask = *sq->kring_mask; -unsigned ktail, ktail_next, submitted, to_submit; +unsigned ktail, submitted, to_submit;unsigned flags;int ret;
@@ -88,15 +88,11 @@ static int __io_uring_submit(struct io_uring *ring, unsigned wait_nr)* Fill in sqes that we have queued up, adding them to the kernel ring */ submitted =0 ; - ktail = ktail_next = *sq->ktail; + ktail = *sq->ktail; to_submit = sq->sqe_tail - sq->sqe_head;while (to_submit--) { - ktail_next++; -read_barrier (); - sq->array[ktail & mask] = sq->sqe_head & mask; - ktail = ktail_next; - + ktail++; sq->sqe_head++; submitted++; }
@@ -104,21 +100,11 @@ static int __io_uring_submit(struct io_uring *ring, unsigned wait_nr)if (!submitted)return 0 ; -if (*sq->ktail != ktail) { -/* -* First write barrier ensures that the SQE stores are updated -* with the tail update. This is needed so that the kernel -* will never see a tail update without the preceeding sQE -* stores being done. -*/ -write_barrier (); - *sq->ktail = ktail; -/* -* The kernel has the matching read barrier for reading the -* SQ tail. -*/ -write_barrier (); - } +/* +* Ensure that the kernel sees the SQE updates before it sees the tail +* update. +*/ +smp_store_release (sq->ktail, ktail); flags =0 ;if (wait_nr ||sq_ring_needs_enter (ring, &flags)) {
test/io_uring_enter.c +5 -3
@@ -262,9 +262,11 @@ main(int argc, char **argv) ktail = *sq->ktail; sq->array[ktail & mask] = index; ++ktail; -write_barrier (); - *sq->ktail = ktail; -write_barrier (); +/* +* Ensure that the kernel sees the SQE update before it sees the tail +* update. +*/ +smp_store_release (sq->ktail, ktail); ret =io_uring_enter (ring.ring_fd,1 ,0 ,0 ,NULL );/* now check to see if our sqe was dropped */
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
Re: [PATCH liburing 2/2] Fix the use of memory barriers
Hi Bart,
On 2019-07-01 23:42, Bart Van Assche wrote:
...
src/barrier.h line 65
Quoted 13 lines
src/barrier.h +13
-0+#if defined(__x86_64__) +#define smp_store_release (p, v) \ +do { \ + barrier(); \ + WRITE_ONCE(*(p), (v)); \ +} while (0) + +#define smp_load_acquire (p) \ +({ \ + typeof(*p) ___p1 = READ_ONCE(*(p)); \ + barrier(); \ + ___p1; \ +})
Can we have these two macros for x86_32 as well? For i386 it will take another branch with full mb, which is not needed.
Besides that both patches looks good to me.
--
Roman
Re: [PATCH liburing 2/2] Fix the use of memory barriers
Quoted 24 lines · Roman Penyaev
Hi Bart,
On 2019-07-01 23:42, Bart Van Assche wrote:
...
src/barrier.h line 65
Quoted 13 lines
src/barrier.h +13
-0+#if defined(__x86_64__) +#define smp_store_release (p, v) \ +do { \ + barrier(); \ + WRITE_ONCE(*(p), (v)); \ +} while (0) + +#define smp_load_acquire (p) \ +({ \ + typeof(*p) ___p1 = READ_ONCE(*(p)); \ + barrier(); \ + ___p1; \ +})Can we have these two macros for
x86_32as well? For i386 it will take another branch with full mb, which is not needed.
Quoted 1 line
Besides that both patches looks good to me.
Hi Roman,
Thanks for having taken a look. From Linux kernel source file arch/x86/include/asm/barrier.h:
#ifdef CONFIG_X86_32
#define mb() asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)",\
"mfence", X86_FEATURE_XMM2) ::: "memory", "cc")
#define rmb() asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)",\
"lfence", X86_FEATURE_XMM2) ::: "memory", "cc")
#define wmb() asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)",\
"sfence", X86_FEATURE_XMM2) ::: "memory", "cc")#else
#define mb() asm volatile("mfence":::"memory")
#define rmb() asm volatile("lfence":::"memory")
#define wmb() asm volatile("sfence" ::: "memory")
#endif
In other words, I think that 32-bit and 64-bit systems really have to be treated in a different way.
Bart.
Re: [PATCH liburing 2/2] Fix the use of memory barriers
Quoted 39 lines · Bart Van Assche
Quoted 24 lines · Roman Penyaev
Hi Bart,
On 2019-07-01 23:42, Bart Van Assche wrote:
...
src/barrier.h line 65
Quoted 13 lines
src/barrier.h +13
-0+#if defined(__x86_64__) +#define smp_store_release (p, v) \ +do { \ + barrier(); \ + WRITE_ONCE(*(p), (v)); \ +} while (0) + +#define smp_load_acquire (p) \ +({ \ + typeof(*p) ___p1 = READ_ONCE(*(p)); \ + barrier(); \ + ___p1; \ +})Can we have these two macros for
x86_32as well? For i386 it will take another branch with full mb, which is not needed.Quoted 1 line
Besides that both patches looks good to me.
Hi Roman,
Thanks for having taken a look. From Linux kernel source file
arch/x86/include/asm/barrier.h:#ifdef
CONFIG_X86_32#define mb() asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)",\ "mfence", X86_FEATURE_XMM2) ::: "memory", "cc") #define rmb() asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)",\ "lfence", X86_FEATURE_XMM2) ::: "memory", "cc") #define wmb() asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)",\ "sfence", X86_FEATURE_XMM2) ::: "memory", "cc")
Quoted 8 lines
#else
#definemb()asmvolatile("mfence":::"memory")
#definermb()asmvolatile("lfence":::"memory")
#definewmb()asmvolatile("sfence" ::: "memory")
#endifIn other words, I think that 32-bit and 64-bit systems really have to be treated in a different way.
I meant smp_load_acquire / smp_store_release
--
Roman
Re: [PATCH liburing 2/2] Fix the use of memory barriers
Quoted 46 lines · Roman Penyaev
Quoted 45 lines · Bart Van Assche
Quoted 24 lines · Roman Penyaev
Hi Bart,
On 2019-07-01 23:42, Bart Van Assche wrote:
...
src/barrier.h line 65
Quoted 13 lines
src/barrier.h +13
-0+#if defined(__x86_64__) +#define smp_store_release (p, v) \ +do { \ + barrier(); \ + WRITE_ONCE(*(p), (v)); \ +} while (0) + +#define smp_load_acquire (p) \ +({ \ + typeof(*p) ___p1 = READ_ONCE(*(p)); \ + barrier(); \ + ___p1; \ +})Can we have these two macros for
x86_32as well? For i386 it will take another branch with full mb, which is not needed.Quoted 1 line
Besides that both patches looks good to me.
Hi Roman,
Thanks for having taken a look. From Linux kernel source file
arch/x86/include/asm/barrier.h:#ifdef
CONFIG_X86_32
#definemb()asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)",\
"mfence",X86_FEATURE_XMM2) ::: "memory", "cc") #definermb()asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)",\
"lfence",X86_FEATURE_XMM2) ::: "memory", "cc") #definewmb()asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)",\
"sfence",X86_FEATURE_XMM2) ::: "memory", "cc") #else
#define mb() asmvolatile("mfence":::"memory")#define rmb() asmvolatile("lfence":::"memory")#define wmb() asmvolatile("sfence" ::: "memory")#endif
Quoted 4 lines
Quoted 2 lines
In other words, I think that 32-bit and 64-bit systems really have to be treated in a different way.
I meant
smp_load_acquire/smp_store_release
Hi Roman,
Since there are 32-bit x86 CPUs that can reorder loads and stores I'm not sure it is safe to use the 64-bit smp_load_acquire() / smp_store_release() implementations for 32-bit CPUs. In case you would not yet have encountered this paper, a thorough discussion of the x86 memory model is available in Sewell, Peter, Susmit Sarkar, Scott Owens, Francesco Zappa Nardelli, and Magnus O. Myreen. "x86-TSO: a rigorous and usable programmer's model for x86 multiprocessors." Communications of the ACM 53, no. 7 (2010): 89-97
(https://dl.acm.org/citation.cfm?id=1785443 / http://www.spinroot.com/spin/Doc/course/x86_tso.pdf).
Bart.
Re: [PATCH liburing 2/2] Fix the use of memory barriers
Quoted 47 lines · Bart Van Assche
Quoted 46 lines · Roman Penyaev
Quoted 45 lines · Bart Van Assche
Quoted 24 lines · Roman Penyaev
Hi Bart,
On 2019-07-01 23:42, Bart Van Assche wrote:
...
src/barrier.h line 65
Quoted 13 lines
src/barrier.h +13
-0+#if defined(__x86_64__) +#define smp_store_release (p, v) \ +do { \ + barrier(); \ + WRITE_ONCE(*(p), (v)); \ +} while (0) + +#define smp_load_acquire (p) \ +({ \ + typeof(*p) ___p1 = READ_ONCE(*(p)); \ + barrier(); \ + ___p1; \ +})Can we have these two macros for
x86_32as well? For i386 it will take another branch with full mb, which is not needed.Quoted 1 line
Besides that both patches looks good to me.
Hi Roman,
Thanks for having taken a look. From Linux kernel source file
arch/x86/include/asm/barrier.h:#ifdef
CONFIG_X86_32
#definemb()asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)",\
"mfence",X86_FEATURE_XMM2) ::: "memory", "cc") #definermb()asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)",\
"lfence",X86_FEATURE_XMM2) ::: "memory", "cc") #definewmb()asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)",\
"sfence",X86_FEATURE_XMM2) ::: "memory", "cc") #else
#define mb() asmvolatile("mfence":::"memory")#define rmb() asmvolatile("lfence":::"memory")#define wmb() asmvolatile("sfence" ::: "memory")#endif
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
In other words, I think that 32-bit and 64-bit systems really have to be treated in a different way.
I meant
smp_load_acquire/smp_store_releaseHi Roman,
Hi Bart,
It seems you took the chunk of the code with macros from tools/arch/x86/include/asm/barrier.h, where indeed smp_store_release and smp_load_acquire are defined only for x86-64. If I am not mistaken kernel defines both __smp_store_release / __smp_load_acquire equally for x86 memory model in arch/x86/include/asm/barrier.h.
Quoted 1 line
Since there are 32-bit x86 CPUs that can reorder loads and stores
Only "loads may be reordered with earlier stores"
(Intel® 64 and IA-32 Architectures, section 8.2.3.4), but "stores are not reordered with earlier loads" (section 8.2.3.3), so smp_store_release and and smp_load_acquire can be relaxed.
--
Roman
Re: [PATCH liburing 2/2] Fix the use of memory barriers
Quoted 5 lines · Roman Penyaev
It seems you took the chunk of the code with macros from
tools/arch/x86/include/asm/barrier.h, where indeedsmp_store_releaseandsmp_load_acquireare defined only for x86-64. If I am not mistaken kernel defines both__smp_store_release/__smp_load_acquireequally for x86 memory model inarch/x86/include/asm/barrier.h.
Agreed.
Quoted 6 lines
Quoted 1 line
Since there are 32-bit x86 CPUs that can reorder loads and stores
Only "loads may be reordered with earlier stores"
(Intel® 64 and IA-32 Architectures, section 8.2.3.4), but "stores are not reordered with earlier loads" (section 8.2.3.3), sosmp_store_releaseand andsmp_load_acquirecan be relaxed.
Certain IA-32 CPUs do not follow that model. From Linux kernel v4.15:
config X86_PPRO_FENCE
bool "PentiumPro memory ordering errata workaround" depends on M686 || M586MMX || M586TSC || M586 || M486 || MGEODEGX1 ---help---
Old PentiumPro multiprocessor systems had errata that could cause memory operations to violate the x86 ordering standard in rare cases. Enabling this option will attempt to work around some (but not all) occurrences of this problem, at the cost of much heavier spinlock and memory barrier operations.
If unsure, say n here. Even distro kernels should think twice before enabling this: there are few systems, and an unlikely bug.
Since support for these CPUs has been removed from the Linux kernel it's probably fine not to worry about these CPUs in liburing either. See also commit 5927145efd5d ("x86/cpu: Remove the CONFIG_X86_PPRO_FENCE=y quirk") # v4.16.
Bart.
Quoted 5 lines · Bart Van Assche
Hi Jens,
The two patches in this series reduce the number of memory barrier calls from liburing and also fix a few memory synchronization bugs. Please consider these patches for the official liburing git repository.
Applied, thanks.
Signature · 1 line
Jens Axboe