Skip to contentSkip to conversation

two atomic_cmpxchg() related fixes

PATCH Applied Halil Pasic 5 participants 22 messages Jun 16, 2020 – Jul 6, 2020

lore
Share & tools
lore
#0 lore raw

cover two atomic_cmpxchg() related fixes

The story short: compiler can generate code that does two distinct fetches of *ind_addr for old and _old. If that happens we can not figure out if we had the desired xchg or not.

Halil Pasic (2):
virtio-ccw: fix virtio_set_ind_atomic
s390x/pci: fix set_ind_atomic

 hw/s390x/s390-pci-bus.c | 16 +++++++++-------
 hw/s390x/virtio-ccw.c   | 18 ++++++++++--------
 2 files changed, 19 insertions(+), 15 deletions(-)

base-commit: 7d3660e79830a069f1848bb4fa1cdf8f666424fb

Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/2 virtio-ccw: fix virtio_set_ind_atomic

The atomic_cmpxchg() loop is broken because we occasionally end up with old and _old having different values (a legit compiler can generate code that accessed *ind_addr again to pick up a value for _old instead of using the value of old that was already fetched according to the rules of the abstract machine). This means the underlying CS instruction may use a different old (_old) than the one we intended to use if atomic_cmpxchg() performed the xchg part.

Let us use volatile to force the rules of the abstract machine for accesses to *ind_addr. Let us also rewrite the loop so, we that the new old is used to compute the new desired value if the xchg part is not performed.

Fixes 7e7494627f ("s390x/virtio-ccw: Adapter interrupt support.")
---
 hw/s390x/virtio-ccw.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
hw/s390x/virtio-ccw.c +10 -8
index c1f4bb1d33..3c988a000b 100644
@@ -786,9 +786,10 @@ static inline VirtioCcwDevice *to_virtio_ccw_dev_fast(DeviceState *d)
 static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                                      uint8_t to_be_set)
 {
-    uint8_t ind_old, ind_new;
+    uint8_t expected, actual;
     hwaddr len = 1;
-    uint8_t *ind_addr;
+    /* avoid  multiple fetches */
+    uint8_t volatile *ind_addr;
 
     ind_addr = cpu_physical_memory_map(ind_loc, &len, true);
     if (!ind_addr) {
@@ -796,14 +797,15 @@ static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                      __func__, sch->cssid, sch->ssid, sch->schid);
         return -1;
     }
+    actual = *ind_addr;
     do {
-        ind_old = *ind_addr;
-        ind_new = ind_old | to_be_set;
-    } while (atomic_cmpxchg(ind_addr, ind_old, ind_new) != ind_old);
-    trace_virtio_ccw_set_ind(ind_loc, ind_old, ind_new);
-    cpu_physical_memory_unmap(ind_addr, len, 1, len);
+        expected = actual;
+        actual = atomic_cmpxchg(ind_addr, expected, expected | to_be_set);
+    } while (actual != expected);
+    trace_virtio_ccw_set_ind(ind_loc, actual, actual | to_be_set);
+    cpu_physical_memory_unmap((void *)ind_addr, len, 1, len);
 
-    return ind_old;
+    return actual;
 }
 
 static void virtio_ccw_notify(DeviceState *d, uint16_t vector)
Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH 1/2] virtio-ccw: fix virtio_set_ind_atomic

Quoted 38 lines · Halil Pasic

The atomic_cmpxchg() loop is broken because we occasionally end up with old and _old having different values (a legit compiler can generate code that accessed *ind_addr again to pick up a value for _old instead of using the value of old that was already fetched according to the rules of the abstract machine). This means the underlying CS instruction may use a different old (_old) than the one we intended to use if atomic_cmpxchg() performed the xchg part.

Let us use volatile to force the rules of the abstract machine for accesses to *ind_addr. Let us also rewrite the loop so, we that the new old is used to compute the new desired value if the xchg part is not performed.

Fixes 7e7494627f ("s390x/virtio-ccw: Adapter interrupt support.")
---
 hw/s390x/virtio-ccw.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
hw/s390x/virtio-ccw.c +3 -2
index c1f4bb1d33..3c988a000b 100644
@@ -786,9 +786,10 @@ static inline VirtioCcwDevice *to_virtio_ccw_dev_fast(DeviceState *d)
 static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                                      uint8_t to_be_set)
 {
-    uint8_t ind_old, ind_new;
+    uint8_t expected, actual;
     hwaddr len = 1;
-    uint8_t *ind_addr;
+    /* avoid  multiple fetches */
+    uint8_t volatile *ind_addr;
 
     ind_addr = cpu_physical_memory_map(ind_loc, &len, true);
     if (!ind_addr) {

hw/s390x/virtio-ccw.c static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc, · line 800

Quoted 6 lines
hw/s390x/virtio-ccw.c +1 -1
@@ -796,14 +797,15 @@ static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                      __func__, sch->cssid, sch->ssid, sch->schid);
         return -1;
     }
+    actual = *ind_addr;
     do {
-        ind_old = *ind_addr;

to make things easier to understand. Adding a barrier in here also fixes the issue. Reasoning follows below:

hw/s390x/virtio-ccw.c static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc, · line 801

Quoted 1 line
hw/s390x/virtio-ccw.c +0 -1
-        ind_new = ind_old | to_be_set;

with an analysis from Andreas (cc)

#define atomic_cmpxchg__nocheck(ptr, old, new) ({ \

     typeof_strip_qual(*ptr) _old = (old);                               \   
 
     (void)__atomic_compare_exchange_n(ptr, &_old, new, false,           \   
 
                               __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);      \   
 
     _old;                                                               \   

})

ind_old is copied into _old in the macro. Instead of doing the copy from the register the compiler reloads the value from memory. The result is that _old and ind_old end up having different values. _old in r1 with the bits set already and ind_old in r10 with the bits cleared. _old gets updated by CS and matches ind_old afterwards - both with the bits being 0. So the != compare is false and the loop is left without having set any bits.

Paolo (to),
I am asking myself if it would be safer to add a barrier or something like this in the macros in include/qemu/atomic.h.

Quoted 15 lines
hw/s390x/virtio-ccw.c +6 -4
-    } while (atomic_cmpxchg(ind_addr, ind_old, ind_new) != ind_old);
-    trace_virtio_ccw_set_ind(ind_loc, ind_old, ind_new);
-    cpu_physical_memory_unmap(ind_addr, len, 1, len);
+        expected = actual;
+        actual = atomic_cmpxchg(ind_addr, expected, expected | to_be_set);
+    } while (actual != expected);
+    trace_virtio_ccw_set_ind(ind_loc, actual, actual | to_be_set);
+    cpu_physical_memory_unmap((void *)ind_addr, len, 1, len);
 
-    return ind_old;
+    return actual;
 }
 
 static void virtio_ccw_notify(DeviceState *d, uint16_t vector)
#3 lore raw

Re: [PATCH 1/2] virtio-ccw: fix virtio_set_ind_atomic

Quoted 73 lines · On Tue, 16 Jun 2020 07:58:53 +0200 Christian Borntraeger…
Quoted 38 lines · Halil Pasic

The atomic_cmpxchg() loop is broken because we occasionally end up with old and _old having different values (a legit compiler can generate code that accessed *ind_addr again to pick up a value for _old instead of using the value of old that was already fetched according to the rules of the abstract machine). This means the underlying CS instruction may use a different old (_old) than the one we intended to use if atomic_cmpxchg() performed the xchg part.

Let us use volatile to force the rules of the abstract machine for accesses to *ind_addr. Let us also rewrite the loop so, we that the new old is used to compute the new desired value if the xchg part is not performed.

Fixes 7e7494627f ("s390x/virtio-ccw: Adapter interrupt support.")
---
 hw/s390x/virtio-ccw.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
hw/s390x/virtio-ccw.c +3 -2
index c1f4bb1d33..3c988a000b 100644
@@ -786,9 +786,10 @@ static inline VirtioCcwDevice *to_virtio_ccw_dev_fast(DeviceState *d)
 static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                                      uint8_t to_be_set)
 {
-    uint8_t ind_old, ind_new;
+    uint8_t expected, actual;
     hwaddr len = 1;
-    uint8_t *ind_addr;
+    /* avoid  multiple fetches */
+    uint8_t volatile *ind_addr;
 
     ind_addr = cpu_physical_memory_map(ind_loc, &len, true);
     if (!ind_addr) {

hw/s390x/virtio-ccw.c static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc, · line 800

Quoted 6 lines
hw/s390x/virtio-ccw.c +1 -1
@@ -796,14 +797,15 @@ static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                      __func__, sch->cssid, sch->ssid, sch->schid);
         return -1;
     }
+    actual = *ind_addr;
     do {
-        ind_old = *ind_addr;  

to make things easier to understand. Adding a barrier in here also fixes the issue. Reasoning follows below:

hw/s390x/virtio-ccw.c static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc, · line 801

Quoted 1 line
hw/s390x/virtio-ccw.c +0 -1
-        ind_new = ind_old | to_be_set;  

with an analysis from Andreas (cc)

#define atomic_cmpxchg__nocheck(ptr, old, new) ({ \

     typeof_strip_qual(*ptr) _old = (old);                               \   
 
     (void)__atomic_compare_exchange_n(ptr, &_old, new, false,           \   
 
                               __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);      \   
 
     _old;                                                               \   

})

ind_old is copied into _old in the macro. Instead of doing the copy from the register the compiler reloads the value from memory. The result is that _old and ind_old end up having different values. _old in r1 with the bits set already and ind_old in r10 with the bits cleared. _old gets updated by CS and matches ind_old afterwards - both with the bits being 0. So the != compare is false and the loop is left without having set any bits.

Quoted 3 lines

Paolo (to),
I am asking myself if it would be safer to add a barrier or something like this in the macros in include/qemu/atomic.h.

I'm also wondering whether this has been seen on other architectures as well? There are also some callers in non-s390x code, and dealing with this in common code would catch them as well.

Quoted 21 lines
Quoted 19 lines
hw/s390x/virtio-ccw.c +6 -4
-    } while (atomic_cmpxchg(ind_addr, ind_old, ind_new) != ind_old);
-    trace_virtio_ccw_set_ind(ind_loc, ind_old, ind_new);
-    cpu_physical_memory_unmap(ind_addr, len, 1, len);
+        expected = actual;
+        actual = atomic_cmpxchg(ind_addr, expected, expected | to_be_set);
+    } while (actual != expected);
+    trace_virtio_ccw_set_ind(ind_loc, actual, actual | to_be_set);
+    cpu_physical_memory_unmap((void *)ind_addr, len, 1, len);
 
-    return ind_old;
+    return actual;
 }
 
 static void virtio_ccw_notify(DeviceState *d, uint16_t vector)
#4 lore raw

Re: [PATCH 1/2] virtio-ccw: fix virtio_set_ind_atomic

Quoted 76 lines · Cornelia Huck
Quoted 73 lines · On Tue, 16 Jun 2020 07:58:53 +0200 Christian Borntraeger…
Quoted 38 lines · Halil Pasic

The atomic_cmpxchg() loop is broken because we occasionally end up with old and _old having different values (a legit compiler can generate code that accessed *ind_addr again to pick up a value for _old instead of using the value of old that was already fetched according to the rules of the abstract machine). This means the underlying CS instruction may use a different old (_old) than the one we intended to use if atomic_cmpxchg() performed the xchg part.

Let us use volatile to force the rules of the abstract machine for accesses to *ind_addr. Let us also rewrite the loop so, we that the new old is used to compute the new desired value if the xchg part is not performed.

Fixes 7e7494627f ("s390x/virtio-ccw: Adapter interrupt support.")
---
 hw/s390x/virtio-ccw.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
hw/s390x/virtio-ccw.c +3 -2
index c1f4bb1d33..3c988a000b 100644
@@ -786,9 +786,10 @@ static inline VirtioCcwDevice *to_virtio_ccw_dev_fast(DeviceState *d)
 static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                                      uint8_t to_be_set)
 {
-    uint8_t ind_old, ind_new;
+    uint8_t expected, actual;
     hwaddr len = 1;
-    uint8_t *ind_addr;
+    /* avoid  multiple fetches */
+    uint8_t volatile *ind_addr;
 
     ind_addr = cpu_physical_memory_map(ind_loc, &len, true);
     if (!ind_addr) {

hw/s390x/virtio-ccw.c static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc, · line 800

Quoted 6 lines
hw/s390x/virtio-ccw.c +1 -1
@@ -796,14 +797,15 @@ static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                      __func__, sch->cssid, sch->ssid, sch->schid);
         return -1;
     }
+    actual = *ind_addr;
     do {
-        ind_old = *ind_addr;  

to make things easier to understand. Adding a barrier in here also fixes the issue. Reasoning follows below:

hw/s390x/virtio-ccw.c static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc, · line 801

Quoted 1 line
hw/s390x/virtio-ccw.c +0 -1
-        ind_new = ind_old | to_be_set;  

with an analysis from Andreas (cc)

#define atomic_cmpxchg__nocheck(ptr, old, new) ({ \

     typeof_strip_qual(*ptr) _old = (old);                               \   
 
     (void)__atomic_compare_exchange_n(ptr, &_old, new, false,           \   
 
                               __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);      \   
 
     _old;                                                               \   

})

ind_old is copied into _old in the macro. Instead of doing the copy from the register the compiler reloads the value from memory. The result is that _old and ind_old end up having different values. _old in r1 with the bits set already and ind_old in r10 with the bits cleared. _old gets updated by CS and matches ind_old afterwards - both with the bits being 0. So the != compare is false and the loop is left without having set any bits.

Quoted 3 lines
Quoted 3 lines

Paolo (to),
I am asking myself if it would be safer to add a barrier or something like this in the macros in include/qemu/atomic.h.

Having said this, I think that the refactoring from Halil (to re-use actual) also makes sense independent of the fix.

Quoted 27 lines

I'm also wondering whether this has been seen on other architectures as well? There are also some callers in non-s390x code, and dealing with this in common code would catch them as well.

Quoted 21 lines
Quoted 19 lines
hw/s390x/virtio-ccw.c +6 -4
-    } while (atomic_cmpxchg(ind_addr, ind_old, ind_new) != ind_old);
-    trace_virtio_ccw_set_ind(ind_loc, ind_old, ind_new);
-    cpu_physical_memory_unmap(ind_addr, len, 1, len);
+        expected = actual;
+        actual = atomic_cmpxchg(ind_addr, expected, expected | to_be_set);
+    } while (actual != expected);
+    trace_virtio_ccw_set_ind(ind_loc, actual, actual | to_be_set);
+    cpu_physical_memory_unmap((void *)ind_addr, len, 1, len);
 
-    return ind_old;
+    return actual;
 }
 
 static void virtio_ccw_notify(DeviceState *d, uint16_t vector)
#5 lore raw

Re: [PATCH 1/2] virtio-ccw: fix virtio_set_ind_atomic

Quoted 77 lines · On Tue, 16 Jun 2020 08:45:14 +0200 Christian Borntraeger…
Quoted 76 lines · Cornelia Huck
Quoted 73 lines · On Tue, 16 Jun 2020 07:58:53 +0200 Christian Borntraeger…
Quoted 38 lines · Halil Pasic

The atomic_cmpxchg() loop is broken because we occasionally end up with old and _old having different values (a legit compiler can generate code that accessed *ind_addr again to pick up a value for _old instead of using the value of old that was already fetched according to the rules of the abstract machine). This means the underlying CS instruction may use a different old (_old) than the one we intended to use if atomic_cmpxchg() performed the xchg part.

Let us use volatile to force the rules of the abstract machine for accesses to *ind_addr. Let us also rewrite the loop so, we that the new old is used to compute the new desired value if the xchg part is not performed.

Fixes 7e7494627f ("s390x/virtio-ccw: Adapter interrupt support.")
---
 hw/s390x/virtio-ccw.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
hw/s390x/virtio-ccw.c +3 -2
index c1f4bb1d33..3c988a000b 100644
@@ -786,9 +786,10 @@ static inline VirtioCcwDevice *to_virtio_ccw_dev_fast(DeviceState *d)
 static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                                      uint8_t to_be_set)
 {
-    uint8_t ind_old, ind_new;
+    uint8_t expected, actual;
     hwaddr len = 1;
-    uint8_t *ind_addr;
+    /* avoid  multiple fetches */
+    uint8_t volatile *ind_addr;
 
     ind_addr = cpu_physical_memory_map(ind_loc, &len, true);
     if (!ind_addr) {

hw/s390x/virtio-ccw.c static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc, · line 800

Quoted 6 lines
hw/s390x/virtio-ccw.c +1 -1
@@ -796,14 +797,15 @@ static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                      __func__, sch->cssid, sch->ssid, sch->schid);
         return -1;
     }
+    actual = *ind_addr;
     do {
-        ind_old = *ind_addr;    

to make things easier to understand. Adding a barrier in here also fixes the issue. Reasoning follows below:

hw/s390x/virtio-ccw.c static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc, · line 801

Quoted 1 line
hw/s390x/virtio-ccw.c +0 -1
-        ind_new = ind_old | to_be_set;    

with an analysis from Andreas (cc)

#define atomic_cmpxchg__nocheck(ptr, old, new) ({ \

     typeof_strip_qual(*ptr) _old = (old);                               \   
 
     (void)__atomic_compare_exchange_n(ptr, &_old, new, false,           \   
 
                               __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);      \   
 
     _old;                                                               \   

})

ind_old is copied into _old in the macro. Instead of doing the copy from the register the compiler reloads the value from memory. The result is that _old and ind_old end up having different values. _old in r1 with the bits set already and ind_old in r10 with the bits cleared. _old gets updated by CS and matches ind_old afterwards - both with the bits being 0. So the != compare is false and the loop is left without having set any bits.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

Paolo (to),
I am asking myself if it would be safer to add a barrier or something like this in the macros in include/qemu/atomic.h.

Having said this, I think that the refactoring from Halil (to re-use actual) also makes sense independent of the fix.

What about adding a barrier instead, as you suggested?

(Still wondering about other users of atomic_cmpxchg(), though.)

#6 lore raw

Re: [PATCH 1/2] virtio-ccw: fix virtio_set_ind_atomic

In reply to Cornelia Huck
Quoted 20 lines · On Tue, 16 Jun 2020 08:33:33 +0200 Cornelia Huck <[email protected]> wrote:
Quoted 20 lines

#define atomic_cmpxchg__nocheck(ptr, old, new) ({ \

     typeof_strip_qual(*ptr) _old = (old);                               \   
 
     (void)__atomic_compare_exchange_n(ptr, &_old, new, false,           \   
 
                               __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);      \   
 
     _old;                                                               \   

})

ind_old is copied into _old in the macro. Instead of doing the copy from the register the compiler reloads the value from memory. The result is that _old and ind_old end up having different values. _old in r1 with the bits set already and ind_old in r10 with the bits cleared. _old gets updated by CS and matches ind_old afterwards - both with the bits being 0. So the != compare is false and the loop is left without having set any bits.

Quoted 7 lines
Quoted 3 lines

Paolo (to),
I am asking myself if it would be safer to add a barrier or something like this in the macros in include/qemu/atomic.h.

I'm also wondering whether this has been seen on other architectures as well? There are also some callers in non-s390x code, and dealing with this in common code would catch them as well.

Quite a bunch of users use something like old = atomic_read(..), where atomic_read is documented as in docs/devel/atomics.rst:
- ``atomic_read()`` and ``atomic_set()``; these prevent the compiler from optimizing accesses out of existence and creating unsolicited accesses, but do not otherwise impose any ordering on loads and
stores: both the compiler and the processor are free to reorder them.

Maybe I should have used that instead of volatile, but my problem was that I didn't fully understand what atomic_read() does, and if it does more than we need. I found the documentation just now.

Another bunch uses constants as old, which is also fine. And there is a third bunch where I don't know whats up, partly because I did not dig deep enough.

Regards,
Halil

#7 lore raw

Re: [PATCH 1/2] virtio-ccw: fix virtio_set_ind_atomic

Quoted 39 lines · Halil Pasic
Quoted 20 lines · On Tue, 16 Jun 2020 08:33:33 +0200 Cornelia Huck <[email protected]> wrote:
Quoted 20 lines

#define atomic_cmpxchg__nocheck(ptr, old, new) ({ \

     typeof_strip_qual(*ptr) _old = (old);                               \   
 
     (void)__atomic_compare_exchange_n(ptr, &_old, new, false,           \   
 
                               __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);      \   
 
     _old;                                                               \   

})

ind_old is copied into _old in the macro. Instead of doing the copy from the register the compiler reloads the value from memory. The result is that _old and ind_old end up having different values. _old in r1 with the bits set already and ind_old in r10 with the bits cleared. _old gets updated by CS and matches ind_old afterwards - both with the bits being 0. So the != compare is false and the loop is left without having set any bits.

Quoted 7 lines
Quoted 3 lines

Paolo (to),
I am asking myself if it would be safer to add a barrier or something like this in the macros in include/qemu/atomic.h.

I'm also wondering whether this has been seen on other architectures as well? There are also some callers in non-s390x code, and dealing with this in common code would catch them as well.

Quite a bunch of users use something like old = atomic_read(..), where atomic_read is documented as in docs/devel/atomics.rst:
- ``atomic_read()`` and ``atomic_set()``; these prevent the compiler from optimizing accesses out of existence and creating unsolicited accesses, but do not otherwise impose any ordering on loads and
stores: both the compiler and the processor are free to reorder them.

Quoted 3 lines

Maybe I should have used that instead of volatile, but my problem was that I didn't fully understand what atomic_read() does, and if it does more than we need. I found the documentation just now.

IIRC, atomic_read() is the right way of doing it, at least in the kernel. I use such a loop in QEMU in

https://lkml.kernel.org/r/[email protected]

But reading docs/devel/atomics.rst:"Comparison with Linux kernel primitives" I do wonder if that is sufficient.

Any experts around?

Signature · 3 lines
Thanks,

David / dhildenb
#8 lore raw

Re: [PATCH 1/2] virtio-ccw: fix virtio_set_ind_atomic

Quoted 40 lines · On Fri, 19 Jun 2020 09:33:44 +0200 David Hildenbrand <[email protected]> wrote:
Quoted 39 lines · Halil Pasic
Quoted 20 lines · On Tue, 16 Jun 2020 08:33:33 +0200 Cornelia Huck <[email protected]> wrote:
Quoted 20 lines

#define atomic_cmpxchg__nocheck(ptr, old, new) ({ \

     typeof_strip_qual(*ptr) _old = (old);                               \   
 
     (void)__atomic_compare_exchange_n(ptr, &_old, new, false,           \   
 
                               __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);      \   
 
     _old;                                                               \   

})

ind_old is copied into _old in the macro. Instead of doing the copy from the register the compiler reloads the value from memory. The result is that _old and ind_old end up having different values. _old in r1 with the bits set already and ind_old in r10 with the bits cleared. _old gets updated by CS and matches ind_old afterwards - both with the bits being 0. So the != compare is false and the loop is left without having set any bits.

Quoted 7 lines
Quoted 3 lines

Paolo (to),
I am asking myself if it would be safer to add a barrier or something like this in the macros in include/qemu/atomic.h.

I'm also wondering whether this has been seen on other architectures as well? There are also some callers in non-s390x code, and dealing with this in common code would catch them as well.

Quite a bunch of users use something like old = atomic_read(..), where atomic_read is documented as in docs/devel/atomics.rst:
- ``atomic_read()`` and ``atomic_set()``; these prevent the compiler from optimizing accesses out of existence and creating unsolicited accesses, but do not otherwise impose any ordering on loads and
stores: both the compiler and the processor are free to reorder them.

Quoted 6 lines
Quoted 3 lines

Maybe I should have used that instead of volatile, but my problem was that I didn't fully understand what atomic_read() does, and if it does more than we need. I found the documentation just now.

IIRC, atomic_read() is the right way of doing it, at least in the kernel.

In kernel I would use READ_ONCE.

https://elixir.bootlin.com/linux/v5.8-rc1/source/include/asm-generic/atomic.h#L171

In this case we are not manipulating an atomic variable. For uint8_t that boils down to an access through a volatile pointer. And that is what I did :).

Quoted 9 lines

I use such a loop in QEMU in

https://lkml.kernel.org/r/[email protected]

But reading docs/devel/atomics.rst:"Comparison with Linux kernel primitives" I do wonder if that is sufficient.

Any experts around?

IMHO what we want here is READ_ONCE, i.e. volatile access, and not necessarily atomic access. But I suppose atomic access implies volatile access (the C11 standard refers to atomic_load as
C atomic_load(volatile A *object)). Because QEMU seems to use atomic_read() in such situations, and does not have READ_ONCE, for me atomic_read would also do.

Regards,
Halil

#9 lore raw

Re: [PATCH 1/2] virtio-ccw: fix virtio_set_ind_atomic

In reply to Christian Borntraeger
Quoted 65 lines · On Tue, 16 Jun 2020 07:58:53 +0200 Christian Borntraeger…
Quoted 38 lines · Halil Pasic

The atomic_cmpxchg() loop is broken because we occasionally end up with old and _old having different values (a legit compiler can generate code that accessed *ind_addr again to pick up a value for _old instead of using the value of old that was already fetched according to the rules of the abstract machine). This means the underlying CS instruction may use a different old (_old) than the one we intended to use if atomic_cmpxchg() performed the xchg part.

Let us use volatile to force the rules of the abstract machine for accesses to *ind_addr. Let us also rewrite the loop so, we that the new old is used to compute the new desired value if the xchg part is not performed.

Fixes 7e7494627f ("s390x/virtio-ccw: Adapter interrupt support.")
---
 hw/s390x/virtio-ccw.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
hw/s390x/virtio-ccw.c +3 -2
index c1f4bb1d33..3c988a000b 100644
@@ -786,9 +786,10 @@ static inline VirtioCcwDevice *to_virtio_ccw_dev_fast(DeviceState *d)
 static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                                      uint8_t to_be_set)
 {
-    uint8_t ind_old, ind_new;
+    uint8_t expected, actual;
     hwaddr len = 1;
-    uint8_t *ind_addr;
+    /* avoid  multiple fetches */
+    uint8_t volatile *ind_addr;
 
     ind_addr = cpu_physical_memory_map(ind_loc, &len, true);
     if (!ind_addr) {

hw/s390x/virtio-ccw.c static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc, · line 800

Quoted 6 lines
hw/s390x/virtio-ccw.c +1 -1
@@ -796,14 +797,15 @@ static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                      __func__, sch->cssid, sch->ssid, sch->schid);
         return -1;
     }
+    actual = *ind_addr;
     do {
-        ind_old = *ind_addr;

to make things easier to understand. Adding a barrier in here also fixes the issue. Reasoning follows below:

hw/s390x/virtio-ccw.c static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc, · line 801

Quoted 1 line
hw/s390x/virtio-ccw.c +0 -1
-        ind_new = ind_old | to_be_set;

with an analysis from Andreas (cc)

#define atomic_cmpxchg__nocheck(ptr, old, new) ({ \

     typeof_strip_qual(*ptr) _old = (old);                               \   
 
     (void)__atomic_compare_exchange_n(ptr, &_old, new, false,           \   
 
                               __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);      \   
 
     _old;                                                               \   
Quoted 2 lines

})

There is also the

#define atomic_cmpxchg(ptr, old, new) __sync_val_compare_and_swap(ptr, old, new)

variant, I guess, when the C11 stuff is not available. I don't know if that variant is guaranteed to not have problems with multiple loads.

Quoted 8 lines

ind_old is copied into _old in the macro. Instead of doing the copy from the register the compiler reloads the value from memory. The result is that _old and ind_old end up having different values. _old in r1 with the bits set already and ind_old in r10 with the bits cleared. _old gets updated by CS and matches ind_old afterwards - both with the bits being 0. So the != compare is false and the loop is left without having set any bits.

Quoted 4 lines

Paolo (to),
I am asking myself if it would be safer to add a barrier or something like this in the macros in include/qemu/atomic.h.

I think accessing the initial value via a volatile pointer initially and using the value loaded by cmpxchg for subsequent iterations is cleaner.

Regards,
Halil

Quoted 19 lines
Quoted 16 lines
hw/s390x/virtio-ccw.c +6 -4
-    } while (atomic_cmpxchg(ind_addr, ind_old, ind_new) != ind_old);
-    trace_virtio_ccw_set_ind(ind_loc, ind_old, ind_new);
-    cpu_physical_memory_unmap(ind_addr, len, 1, len);
+        expected = actual;
+        actual = atomic_cmpxchg(ind_addr, expected, expected | to_be_set);
+    } while (actual != expected);
+    trace_virtio_ccw_set_ind(ind_loc, actual, actual | to_be_set);
+    cpu_physical_memory_unmap((void *)ind_addr, len, 1, len);
 
-    return ind_old;
+    return actual;
 }
 
 static void virtio_ccw_notify(DeviceState *d, uint16_t vector)
Reviewed-by on v1 1/2
#10 lore raw

Re: [PATCH 1/2] virtio-ccw: fix virtio_set_ind_atomic

In reply to Halil Pasic
Quoted 8 lines · Halil Pasic

The atomic_cmpxchg() loop is broken because we occasionally end up with old and _old having different values (a legit compiler can generate code that accessed *ind_addr again to pick up a value for _old instead of using the value of old that was already fetched according to the rules of the abstract machine). This means the underlying CS instruction may use a different old (_old) than the one we intended to use if atomic_cmpxchg() performed the xchg part.

Quoted 8 lines

Let us use volatile to force the rules of the abstract machine for accesses to *ind_addr. Let us also rewrite the loop so, we that the new old is used to compute the new desired value if the xchg part is not performed.

Fixes 7e7494627f ("s390x/virtio-ccw: Adapter interrupt support.")
Quoted 44 lines
---
 hw/s390x/virtio-ccw.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
hw/s390x/virtio-ccw.c +10 -8
index c1f4bb1d33..3c988a000b 100644
@@ -786,9 +786,10 @@ static inline VirtioCcwDevice *to_virtio_ccw_dev_fast(DeviceState *d)
 static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                                      uint8_t to_be_set)
 {
-    uint8_t ind_old, ind_new;
+    uint8_t expected, actual;
     hwaddr len = 1;
-    uint8_t *ind_addr;
+    /* avoid  multiple fetches */
+    uint8_t volatile *ind_addr;
 
     ind_addr = cpu_physical_memory_map(ind_loc, &len, true);
     if (!ind_addr) {
@@ -796,14 +797,15 @@ static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                      __func__, sch->cssid, sch->ssid, sch->schid);
         return -1;
     }
+    actual = *ind_addr;
     do {
-        ind_old = *ind_addr;
-        ind_new = ind_old | to_be_set;
-    } while (atomic_cmpxchg(ind_addr, ind_old, ind_new) != ind_old);
-    trace_virtio_ccw_set_ind(ind_loc, ind_old, ind_new);
-    cpu_physical_memory_unmap(ind_addr, len, 1, len);
+        expected = actual;
+        actual = atomic_cmpxchg(ind_addr, expected, expected | to_be_set);
+    } while (actual != expected);
+    trace_virtio_ccw_set_ind(ind_loc, actual, actual | to_be_set);
+    cpu_physical_memory_unmap((void *)ind_addr, len, 1, len);
 
-    return ind_old;
+    return actual;
 }
 
 static void virtio_ccw_notify(DeviceState *d, uint16_t vector)
#11 lore raw

Re: [PATCH 1/2] virtio-ccw: fix virtio_set_ind_atomic

In reply to Halil Pasic
Quoted 7 lines · Halil Pasic

The atomic_cmpxchg() loop is broken because we occasionally end up with old and _old having different values (a legit compiler can generate code that accessed *ind_addr again to pick up a value for _old instead of using the value of old that was already fetched according to the rules of the abstract machine). This means the underlying CS instruction may use a different old (_old) than the one we intended to use if atomic_cmpxchg() performed the xchg part.

And was this ever observed in the field? Or is this a theoretical issue? commit log should probably say ...

Quoted 3 lines

Let us use volatile to force the rules of the abstract machine for accesses to *ind_addr. Let us also rewrite the loop so, we that the

we that -> we know that?

Quoted 37 lines

new old is used to compute the new desired value if the xchg part is not performed.

Fixes 7e7494627f ("s390x/virtio-ccw: Adapter interrupt support.")
---
 hw/s390x/virtio-ccw.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
hw/s390x/virtio-ccw.c +4 -6
index c1f4bb1d33..3c988a000b 100644
@@ -786,9 +786,10 @@ static inline VirtioCcwDevice *to_virtio_ccw_dev_fast(DeviceState *d)
 static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                                      uint8_t to_be_set)
 {
-    uint8_t ind_old, ind_new;
+    uint8_t expected, actual;
     hwaddr len = 1;
-    uint8_t *ind_addr;
+    /* avoid  multiple fetches */
+    uint8_t volatile *ind_addr;
 
     ind_addr = cpu_physical_memory_map(ind_loc, &len, true);
     if (!ind_addr) {
@@ -796,14 +797,15 @@ static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                      __func__, sch->cssid, sch->ssid, sch->schid);
         return -1;
     }
+    actual = *ind_addr;
     do {
-        ind_old = *ind_addr;
-        ind_new = ind_old | to_be_set;
-    } while (atomic_cmpxchg(ind_addr, ind_old, ind_new) != ind_old);
-    trace_virtio_ccw_set_ind(ind_loc, ind_old, ind_new);

hw/s390x/virtio-ccw.c static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc, · line 809

Quoted 10 lines
hw/s390x/virtio-ccw.c +6 -2
-    cpu_physical_memory_unmap(ind_addr, len, 1, len);
+        expected = actual;
+        actual = atomic_cmpxchg(ind_addr, expected, expected | to_be_set);
+    } while (actual != expected);
+    trace_virtio_ccw_set_ind(ind_loc, actual, actual | to_be_set);
+    cpu_physical_memory_unmap((void *)ind_addr, len, 1, len);
 
-    return ind_old;
+    return actual;
 }

I wonder whether cpuXX APIs should accept volatile pointers, too:
casting away volatile is always suspicious.
But that is a separate issue ...

Quoted 3 lines
hw/s390x/virtio-ccw.c +0 -0
 static void virtio_ccw_notify(DeviceState *d, uint16_t vector)
Signature · 1 line
2.17.1
#12 lore raw

Re: [PATCH 1/2] virtio-ccw: fix virtio_set_ind_atomic

Quoted 11 lines · Michael S. Tsirkin
Quoted 7 lines · Halil Pasic

The atomic_cmpxchg() loop is broken because we occasionally end up with old and _old having different values (a legit compiler can generate code that accessed *ind_addr again to pick up a value for _old instead of using the value of old that was already fetched according to the rules of the abstract machine). This means the underlying CS instruction may use a different old (_old) than the one we intended to use if atomic_cmpxchg() performed the xchg part.

And was this ever observed in the field? Or is this a theoretical issue? commit log should probably say ...

It was observed in the field when the xml specified qemu instead of vhost.

#13 lore raw

Re: [PATCH 1/2] virtio-ccw: fix virtio_set_ind_atomic

In reply to Michael S. Tsirkin
Quoted 9 lines · On Sat, 4 Jul 2020 14:34:04 -0400 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 7 lines · Halil Pasic

The atomic_cmpxchg() loop is broken because we occasionally end up with old and _old having different values (a legit compiler can generate code that accessed *ind_addr again to pick up a value for _old instead of using the value of old that was already fetched according to the rules of the abstract machine). This means the underlying CS instruction may use a different old (_old) than the one we intended to use if atomic_cmpxchg() performed the xchg part.

Quoted 3 lines

And was this ever observed in the field? Or is this a theoretical issue? commit log should probably say ...

It was observed in the field (Christian already answered). I think the message already implies this, because the only conjunctive is about the compiler behavior.

Quoted 5 lines
Quoted 3 lines

Let us use volatile to force the rules of the abstract machine for accesses to *ind_addr. Let us also rewrite the loop so, we that the

we that -> we know that?

s/we//

It would be nice to fix this before the patch gets merged.

Quoted 45 lines
Quoted 45 lines

new old is used to compute the new desired value if the xchg part is not performed.

Fixes 7e7494627f ("s390x/virtio-ccw: Adapter interrupt support.")
---
 hw/s390x/virtio-ccw.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
hw/s390x/virtio-ccw.c +9 -7
index c1f4bb1d33..3c988a000b 100644
@@ -786,9 +786,10 @@ static inline VirtioCcwDevice *to_virtio_ccw_dev_fast(DeviceState *d)
 static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                                      uint8_t to_be_set)
 {
-    uint8_t ind_old, ind_new;
+    uint8_t expected, actual;
     hwaddr len = 1;
-    uint8_t *ind_addr;
+    /* avoid  multiple fetches */
+    uint8_t volatile *ind_addr;
 
     ind_addr = cpu_physical_memory_map(ind_loc, &len, true);
     if (!ind_addr) {
@@ -796,14 +797,15 @@ static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc,
                      __func__, sch->cssid, sch->ssid, sch->schid);
         return -1;
     }
+    actual = *ind_addr;
     do {
-        ind_old = *ind_addr;
-        ind_new = ind_old | to_be_set;
-    } while (atomic_cmpxchg(ind_addr, ind_old, ind_new) != ind_old);
-    trace_virtio_ccw_set_ind(ind_loc, ind_old, ind_new);
-    cpu_physical_memory_unmap(ind_addr, len, 1, len);
+        expected = actual;
+        actual = atomic_cmpxchg(ind_addr, expected, expected | to_be_set);
+    } while (actual != expected);
+    trace_virtio_ccw_set_ind(ind_loc, actual, actual | to_be_set);
+    cpu_physical_memory_unmap((void *)ind_addr, len, 1, len);
 
Quoted 8 lines

hw/s390x/virtio-ccw.c static uint8_t virtio_set_ind_atomic(SubchDev *sch, uint64_t ind_loc, · line 809

Quoted 3 lines
hw/s390x/virtio-ccw.c +1 -1
-    return ind_old;
+    return actual;
 }

I wonder whether cpuXX APIs should accept volatile pointers, too:
casting away volatile is always suspicious.
But that is a separate issue ...

Nod.

Thanks for having a look!

Quoted 4 lines
Quoted 3 lines
hw/s390x/virtio-ccw.c +0 -0
 static void virtio_ccw_notify(DeviceState *d, uint16_t vector)
Signature · 1 line
2.17.1
#14 lore raw

PATCH 2/2 s390x/pci: fix set_ind_atomic

In reply to Halil Pasic

The atomic_cmpxchg() loop is broken because we occasionally end up with old and _old having different values (a legit compiler can generate code that accessed *ind_addr again to pick up a value for _old instead of using the value of old that was already fetched according to the rules of the abstract machine). This means the underlying CS instruction may use a different old (_old) than the one we intended to use if atomic_cmpxchg() performed the xchg part.

Let us use volatile to force the rules of the abstract machine for accesses to *ind_addr. Let us also rewrite the loop so, we that the new old is used to compute the new desired value if the xchg part is not performed.

Fixes 8cba80c3a0 ("s390: Add PCI bus support")
---
 hw/s390x/s390-pci-bus.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
hw/s390x/s390-pci-bus.c +9 -7
index c4a4259f0c..80eb957a91 100644
@@ -637,22 +637,24 @@ static AddressSpace *s390_pci_dma_iommu(PCIBus *bus, void *opaque, int devfn)
 
 static uint8_t set_ind_atomic(uint64_t ind_loc, uint8_t to_be_set)
 {
-    uint8_t ind_old, ind_new;
+    uint8_t expected, actual;
     hwaddr len = 1;
-    uint8_t *ind_addr;
+    /* avoid  multiple fetches */
+    uint8_t volatile *ind_addr;
 
     ind_addr = cpu_physical_memory_map(ind_loc, &len, true);
     if (!ind_addr) {
         s390_pci_generate_error_event(ERR_EVENT_AIRERR, 0, 0, 0, 0);
         return -1;
     }
+    actual = *ind_addr;
     do {
-        ind_old = *ind_addr;
-        ind_new = ind_old | to_be_set;
-    } while (atomic_cmpxchg(ind_addr, ind_old, ind_new) != ind_old);
-    cpu_physical_memory_unmap(ind_addr, len, 1, len);
+        expected = actual;
+        actual = atomic_cmpxchg(ind_addr, expected, expected | to_be_set);
+    } while (actual != expected);
+    cpu_physical_memory_unmap((void *)ind_addr, len, 1, len);
 
-    return ind_old;
+    return actual;
 }
 
 static void s390_msi_ctrl_write(void *opaque, hwaddr addr, uint64_t data,
Signature · 1 line
2.17.1
Reviewed-by on v1 2/2
#15 lore raw

Re: [PATCH 2/2] s390x/pci: fix set_ind_atomic

Quoted 8 lines · Halil Pasic

The atomic_cmpxchg() loop is broken because we occasionally end up with old and _old having different values (a legit compiler can generate code that accessed *ind_addr again to pick up a value for _old instead of using the value of old that was already fetched according to the rules of the abstract machine). This means the underlying CS instruction may use a different old (_old) than the one we intended to use if atomic_cmpxchg() performed the xchg part.

Quoted 8 lines

Let us use volatile to force the rules of the abstract machine for accesses to *ind_addr. Let us also rewrite the loop so, we that the new old is used to compute the new desired value if the xchg part is not performed.

Fixes 8cba80c3a0 ("s390: Add PCI bus support")
Quoted 41 lines
---
 hw/s390x/s390-pci-bus.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
hw/s390x/s390-pci-bus.c +9 -7
index c4a4259f0c..80eb957a91 100644
@@ -637,22 +637,24 @@ static AddressSpace *s390_pci_dma_iommu(PCIBus *bus, void *opaque, int devfn)
 
 static uint8_t set_ind_atomic(uint64_t ind_loc, uint8_t to_be_set)
 {
-    uint8_t ind_old, ind_new;
+    uint8_t expected, actual;
     hwaddr len = 1;
-    uint8_t *ind_addr;
+    /* avoid  multiple fetches */
+    uint8_t volatile *ind_addr;
 
     ind_addr = cpu_physical_memory_map(ind_loc, &len, true);
     if (!ind_addr) {
         s390_pci_generate_error_event(ERR_EVENT_AIRERR, 0, 0, 0, 0);
         return -1;
     }
+    actual = *ind_addr;
     do {
-        ind_old = *ind_addr;
-        ind_new = ind_old | to_be_set;
-    } while (atomic_cmpxchg(ind_addr, ind_old, ind_new) != ind_old);
-    cpu_physical_memory_unmap(ind_addr, len, 1, len);
+        expected = actual;
+        actual = atomic_cmpxchg(ind_addr, expected, expected | to_be_set);
+    } while (actual != expected);
+    cpu_physical_memory_unmap((void *)ind_addr, len, 1, len);
 
-    return ind_old;
+    return actual;
 }
 
 static void s390_msi_ctrl_write(void *opaque, hwaddr addr, uint64_t data,
#16 lore raw
In reply to Halil Pasic
Quoted 8 lines · On Tue, 16 Jun 2020 06:50:33 +0200 Halil Pasic <[email protected]> wrote:

The story short: compiler can generate code that does two distinct fetches of *ind_addr for old and _old. If that happens we can not figure out if we had the desired xchg or not.

Halil Pasic (2):
virtio-ccw: fix virtio_set_ind_atomic
s390x/pci: fix set_ind_atomic

Quoted 6 lines
 hw/s390x/s390-pci-bus.c | 16 +++++++++-------
 hw/s390x/virtio-ccw.c   | 18 ++++++++++--------
 2 files changed, 19 insertions(+), 15 deletions(-)

base-commit: 7d3660e79830a069f1848bb4fa1cdf8f666424fb

Have we managed to reach any kind of agreement on this? (A v2?)

We can still get in fixes post-softfreeze, of course.

#17 lore raw
Quoted 16 lines · Cornelia Huck
Quoted 13 lines · On Tue, 16 Jun 2020 06:50:33 +0200 Halil Pasic <[email protected]> wrote:

The story short: compiler can generate code that does two distinct fetches of *ind_addr for old and _old. If that happens we can not figure out if we had the desired xchg or not.

Halil Pasic (2):
virtio-ccw: fix virtio_set_ind_atomic
s390x/pci: fix set_ind_atomic

 hw/s390x/s390-pci-bus.c | 16 +++++++++-------
 hw/s390x/virtio-ccw.c   | 18 ++++++++++--------
 2 files changed, 19 insertions(+), 15 deletions(-)
Quoted 5 lines
Quoted 1 line

base-commit: 7d3660e79830a069f1848bb4fa1cdf8f666424fb

Have we managed to reach any kind of agreement on this? (A v2?)

We can still get in fixes post-softfreeze, of course.

Unless Halil has a v2 ready,
I think the current patch is fine as is as a fix. I would suggest to go with that and we can then do more beautification later when necessary.

#18 lore raw
Quoted 19 lines · On Wed, 1 Jul 2020 14:06:11 +0200 Christian Borntraeger…
Quoted 18 lines · Cornelia Huck
Quoted 14 lines · On Tue, 16 Jun 2020 06:50:33 +0200 Halil Pasic <[email protected]> wrote:

The story short: compiler can generate code that does two distinct fetches of *ind_addr for old and _old. If that happens we can not figure out if we had the desired xchg or not.

Halil Pasic (2):
virtio-ccw: fix virtio_set_ind_atomic
s390x/pci: fix set_ind_atomic

 hw/s390x/s390-pci-bus.c | 16 +++++++++-------
 hw/s390x/virtio-ccw.c   | 18 ++++++++++--------
 2 files changed, 19 insertions(+), 15 deletions(-)

base-commit: 7d3660e79830a069f1848bb4fa1cdf8f666424fb

Quoted 8 lines
Quoted 3 lines

Have we managed to reach any kind of agreement on this? (A v2?)

We can still get in fixes post-softfreeze, of course.

Unless Halil has a v2 ready,
I think the current patch is fine as is as a fix. I would suggest to go with that and we can then do more beautification later when necessary.

Sure, no objection to the patches as they are now.

I would like to see some R-bs/A-bs, though :)

#19 lore raw
In reply to Christian Borntraeger
Quoted 21 lines · On Wed, 1 Jul 2020 14:06:11 +0200 Christian Borntraeger…
Quoted 18 lines · Cornelia Huck
Quoted 14 lines · On Tue, 16 Jun 2020 06:50:33 +0200 Halil Pasic <[email protected]> wrote:

The story short: compiler can generate code that does two distinct fetches of *ind_addr for old and _old. If that happens we can not figure out if we had the desired xchg or not.

Halil Pasic (2):
virtio-ccw: fix virtio_set_ind_atomic
s390x/pci: fix set_ind_atomic

 hw/s390x/s390-pci-bus.c | 16 +++++++++-------
 hw/s390x/virtio-ccw.c   | 18 ++++++++++--------
 2 files changed, 19 insertions(+), 15 deletions(-)

base-commit: 7d3660e79830a069f1848bb4fa1cdf8f666424fb

Quoted 10 lines
Quoted 3 lines

Have we managed to reach any kind of agreement on this? (A v2?)

We can still get in fixes post-softfreeze, of course.

Unless Halil has a v2 ready,
I think the current patch is fine as is as a fix. I would suggest to go with that and we can then do more beautification later when necessary.

I think we were waiting for Paolo to chime in regarding the barrier(). The thing I could beautify is using atomic_read(), but frankly I'm not sure about it, especially considering multi-proccess. My understanding of volatile is better than my understanding of atomic_read(). On the other hand, same multi-process question can be asked about atomic_cmpxchg() and __atomic_compare_exchange_n()...

Regards,
Halil

#20 lore raw
Quoted 35 lines · On Fri, 3 Jul 2020 15:37:11 +0200 Halil Pasic <[email protected]> wrote:
Quoted 21 lines · On Wed, 1 Jul 2020 14:06:11 +0200 Christian Borntraeger…
Quoted 18 lines · Cornelia Huck
Quoted 14 lines · On Tue, 16 Jun 2020 06:50:33 +0200 Halil Pasic <[email protected]> wrote:

The story short: compiler can generate code that does two distinct fetches of *ind_addr for old and _old. If that happens we can not figure out if we had the desired xchg or not.

Halil Pasic (2):
virtio-ccw: fix virtio_set_ind_atomic
s390x/pci: fix set_ind_atomic

 hw/s390x/s390-pci-bus.c | 16 +++++++++-------
 hw/s390x/virtio-ccw.c   | 18 ++++++++++--------
 2 files changed, 19 insertions(+), 15 deletions(-)

base-commit: 7d3660e79830a069f1848bb4fa1cdf8f666424fb

Quoted 10 lines
Quoted 3 lines

Have we managed to reach any kind of agreement on this? (A v2?)

We can still get in fixes post-softfreeze, of course.

Unless Halil has a v2 ready,
I think the current patch is fine as is as a fix. I would suggest to go with that and we can then do more beautification later when necessary.

Quoted 6 lines

I think we were waiting for Paolo to chime in regarding the barrier(). The thing I could beautify is using atomic_read(), but frankly I'm not sure about it, especially considering multi-proccess. My understanding of volatile is better than my understanding of atomic_read(). On the other hand, same multi-process question can be asked about atomic_cmpxchg() and __atomic_compare_exchange_n()...

We can just improve the code on top later. Having something that works now beats something not yet developed that might be nicer :)

(But yes, it also might be good to look at non-s390 examples of this pattern and check if something can be done in a more central place.)

#21 lore raw
In reply to Halil Pasic
Quoted 8 lines · On Tue, 16 Jun 2020 06:50:33 +0200 Halil Pasic <[email protected]> wrote:

The story short: compiler can generate code that does two distinct fetches of *ind_addr for old and _old. If that happens we can not figure out if we had the desired xchg or not.

Halil Pasic (2):
virtio-ccw: fix virtio_set_ind_atomic
s390x/pci: fix set_ind_atomic

Quoted 6 lines
 hw/s390x/s390-pci-bus.c | 16 +++++++++-------
 hw/s390x/virtio-ccw.c   | 18 ++++++++++--------
 2 files changed, 19 insertions(+), 15 deletions(-)

base-commit: 7d3660e79830a069f1848bb4fa1cdf8f666424fb

Thanks, applied.