Skip to contentSkip to conversation

Question about potential missed IPI events

DISCUSSION Bo Gan 4 participants 13 messages Nov 11, 2023 – Nov 21, 2023

lore
Share & tools
lore

Bo Gan

author
#0 lore raw

Hi OpenSBI community,

Current IPI send/recv implementation uses ipi_data->ipi_type to indicate an IPI event pending. In sbi_ipi_send, I see have the following:

Quoted 9 lines

/*
* Set IPI type on remote hart's scratch area and
* trigger the interrupt
*/
atomic_raw_set_bit(event, &ipi_data->ipi_type);
smp_wmb();

if (ipi_dev && ipi_dev->ipi_send)

	ipi_dev->ipi_send(remote_hartindex)

And in sbi_ipi_process, the following:

Quoted 6 lines

if (ipi_dev && ipi_dev->ipi_clear)

	ipi_dev->ipi_clear(hartindex);

ipi_type = atomic_raw_xchg_ulong(&ipi_data->ipi_type, 0); ipi_event = 0;
while (ipi_type) {

I'm think of if there's any possibility of missed IPIs. Assuming we are talking about sifive/clint (aclint_mswi.c), I know it means that there're both smp_wmb (fence w,w) and __io_bw (fence w,o) between atomic_raw_set_bit and the write to remote hart's msip register in clint. Consider the following

Right before Hart A ipi_dev->ipi_clear() for the msip write-1 issued by hart B, Hart C did an ipi_dev->ipi_send to hart A. Hart A then cleared the msip bit, resulting in both IPI (from B and C) coalesced to a single one. In this case, A will observe the ipi_type set by B, due to the smp_wmb + __io_bw, and the IPI was indeed generated by B's write to msip. However will the barrier ensure A also observe the write to ipi_type from hart C? For the coalescing to happen, clint will observe C's write-1 before A's write-0, does it imply A'll observe C's write to ipi_type from xchg?

I doubt it mainly due to 2 reasons:

1. ipi_dev->ipi_clear is a writel(msip), which has __io_bw (fence w,o) I don't think it helps preventing the writel to happen before xchg. If that happens, it's possible that A missed C's write to ipi_type.

2. Suppose the ipi_clear() happened before xchg, from A's standpoint, it's a write, not a read. A wasn't reading from msip, but only writing to it. I'm not sure if A's write to msip after C's write (observed by clint) implies A's observing the same order. A is not reading from msip, so what helps to establish this order? Will the RVWMO global memory order help? Can we think of clint as another hart that follows RVWMO?

Bo

author
#1 lore raw
Quoted 44 lines · Bo Gan

Hi OpenSBI community,

Current IPI send/recv implementation uses ipi_data->ipi_type to indicate an IPI event pending. In sbi_ipi_send, I see have the following:

Quoted 9 lines

/*
?* Set IPI type on remote hart's scratch area and
?* trigger the interrupt
?*/
atomic_raw_set_bit(event, &ipi_data->ipi_type);
smp_wmb();

if (ipi_dev && ipi_dev->ipi_send)
????ipi_dev->ipi_send(remote_hartindex)

And in sbi_ipi_process, the following:

Quoted 6 lines

if (ipi_dev && ipi_dev->ipi_clear)
????ipi_dev->ipi_clear(hartindex);

ipi_type = atomic_raw_xchg_ulong(&ipi_data->ipi_type, 0); ipi_event = 0;
while (ipi_type) {

I'm think of if there's any possibility of missed IPIs. Assuming we are talking about sifive/clint (aclint_mswi.c), I know it means that there're both smp_wmb (fence w,w) and __io_bw (fence w,o) between atomic_raw_set_bit and the write to remote hart's msip register in clint. Consider the following

Right before Hart A ipi_dev->ipi_clear() for the msip write-1 issued by hart B, Hart C did an ipi_dev->ipi_send to hart A. Hart A then cleared the msip bit, resulting in both IPI (from B and C) coalesced to a single one. In this case, A will observe the ipi_type set by B, due to the smp_wmb + __io_bw, and the IPI was indeed generated by B's write to msip. However will the barrier ensure A also observe the write to ipi_type from hart C? For the coalescing to happen, clint will observe C's write-1 before A's write-0, does it imply A'll observe C's write to ipi_type from xchg?

I doubt it mainly due to 2 reasons:

1. ipi_dev->ipi_clear is a writel(msip), which has __io_bw (fence w,o) I don't ?? think it helps preventing the writel to happen before xchg. If that happens, ?? it's possible that A missed C's write to ipi_type.

Quoted 9 lines

2. Suppose the ipi_clear() happened before xchg, from A's standpoint, it's a ?? write, not a read. A wasn't reading from msip, but only writing to it. I'm ?? not sure if A's write to msip after C's write (observed by clint) implies ?? A's observing the same order. A is not reading from msip, so what helps to ?? establish this order? Will the RVWMO global memory order help? Can we think ?? of clint as another hart that follows RVWMO?

Bo

Hi @Atish @Anup and other folks, any thoughts on this?

#2 lore raw

? 2023-11-15???? 16:33 -0800?Bo Gan???

Quoted 30 lines · Bo Gan
Quoted 29 lines

Hi OpenSBI community,

Current IPI send/recv implementation uses ipi_data->ipi_type to indicate an IPI event pending. In sbi_ipi_send, I see have the following:

Quoted 9 lines

/*
?* Set IPI type on remote hart's scratch area and
?* trigger the interrupt
?*/
atomic_raw_set_bit(event, &ipi_data->ipi_type);
smp_wmb();

if (ipi_dev && ipi_dev->ipi_send)
????ipi_dev->ipi_send(remote_hartindex)

And in sbi_ipi_process, the following:

Quoted 6 lines

if (ipi_dev && ipi_dev->ipi_clear)
????ipi_dev->ipi_clear(hartindex);

ipi_type = atomic_raw_xchg_ulong(&ipi_data->ipi_type, 0); ipi_event = 0;
while (ipi_type) {

I'm think of if there's any possibility of missed IPIs. Assuming we are talking about sifive/clint (aclint_mswi.c), I know it means that there're both smp_wmb (fence w,w) and __io_bw (fence w,o) between atomic_raw_set_bit and the write to remote hart's msip register in clint. Consider the following

Quoted 8 lines
Quoted 8 lines

Right before Hart A ipi_dev->ipi_clear() for the msip write-1 issued by hart B, Hart C did an ipi_dev->ipi_send to hart A. Hart A then cleared the msip bit, resulting in both IPI (from B and C) coalesced to a single one. In this case, A will observe the ipi_type set by B, due to the smp_wmb + __io_bw, and the IPI was indeed generated by B's write to msip. However will the barrier ensure A also observe the write to ipi_type from hart C? For the coalescing to happen, clint will observe C's write-1 before A's write-0, does it imply A'll observe C's write to ipi_type from xchg?

Does the picture below represent what you mean?

 B             A             C
 |             |             |
 | xchg        |             |
 | send------->|             |   
 |             | xchg        |
 |             |             | xchg
 |             |<------------| send
 |             | clear       |
Quoted 1 line

In fact, A will not lose C's ipi message and will be deleted when it receives a new ipi interrupt, as shown below

 B             A             C             D
 |             |             |             |
 | xchg        |             |             |
 | send------->|             |             |
 |             | xchg        |             |
 |             |             | xchg        |
 |             |<------------| send        |
 |             | clear       |             |
               |                           |
               |<--------------------------| send

There is another situation that will cause the efficiency to drop. Please refer to the figure below.

 B             A             C
 |             |             |
 | xchg        |             |
 | send------->|             | xchg  
 |             | clear       |
 |             | xchg        |
 |             |<------------| send
 |             |             |

A may enter exception handling one more time.

I think we need to protect xchg/send and clear/xchg with spinlock.

Regards,
Xiang W

Quoted 18 lines
Quoted 6 lines

I doubt it mainly due to 2 reasons:

1. ipi_dev->ipi_clear is a writel(msip), which has __io_bw (fence w,o) I don't ??? think it helps preventing the writel to happen before xchg. If that happens, ??? it's possible that A missed C's write to ipi_type.

Quoted 9 lines

2. Suppose the ipi_clear() happened before xchg, from A's standpoint, it's a ??? write, not a read. A wasn't reading from msip, but only writing to it. I'm ??? not sure if A's write to msip after C's write (observed by clint) implies ??? A's observing the same order. A is not reading from msip, so what helps to ??? establish this order? Will the RVWMO global memory order help? Can we think ??? of clint as another hart that follows RVWMO?

Bo

Hi @Atish @Anup and other folks, any thoughts on this?

author
#3 lore raw
Quoted 12 lines · Xiang W

? 2023-11-15???? 16:33 -0800?Bo Gan???

Quoted 9 lines · Bo Gan
Quoted 8 lines

Right before Hart A ipi_dev->ipi_clear() for the msip write-1 issued by hart B, Hart C did an ipi_dev->ipi_send to hart A. Hart A then cleared the msip bit, resulting in both IPI (from B and C) coalesced to a single one. In this case, A will observe the ipi_type set by B, due to the smp_wmb + __io_bw, and the IPI was indeed generated by B's write to msip. However will the barrier ensure A also observe the write to ipi_type from hart C? For the coalescing to happen, clint will observe C's write-1 before A's write-0, does it imply A'll observe C's write to ipi_type from xchg?

Does the picture below represent what you mean?

Quoted 8 lines
  B             A             C
  |             |             |
  | xchg        |             |
  | send------->|             |
  |             | xchg        |
  |             |             | xchg
  |             |<------------| send
  |             | clear       |

Hi Xiang, Thank you so much for replying. I'd like to revise your chart a little bit:

Case 1, xchg_ulong in ipi_process gets reordered before ipi_clear:

   B                 A                C                 CLINT
   |                 |                |                  |
   | bit_set         |                |                  |
   | send -----------|----------------|----------------> |
   |                 | xchg           |                  |
   |                 |                | bit_set          |
   |                 |                | send ----------> |
   |                 | clear----------|----------------> |

In this case, A would observe the ipi_data->ipi_type *eventually*, but when? A won't process C's IPI request until another IPI comes in the future, which might cause the IPI request to wait indefinitely. This is not an efficiency problem, but a correctness problem. I think the fix should be adding specific fence (need to reason about the correct one) between ipi_dev->ipi_clear and atomic_raw_xchg_ulong in ipi_process

Case 2, xchg_ulong doesn't get reordered, however:

   B                 A                C                 CLINT
   |                 |                |                  |
   | bit_set         |                |                  |
   | send -----------|----------------|----------------> |
   |                 | xchg           |            /---> |
   |                 | clear----------|-----------/----> |
   |                 |                | bit_set  /       |
   |                 |                | send ----        |

In this case, the same thing can happen. The request from C can be delayed indefinitely. The cause is due to the clint observes memory order differently than harts. I'm wondering if the clint can be viewed as another hart that follows RVWMO? If yes, then this case is likely invalid.

Bo

#4 lore raw

? 2023-11-16???? 00:41 -0800?Bo Gan???

Quoted 35 lines · Xiang W
Quoted 12 lines

? 2023-11-15???? 16:33 -0800?Bo Gan???

Quoted 9 lines · Bo Gan
Quoted 8 lines

Right before Hart A ipi_dev->ipi_clear() for the msip write-1 issued by hart B, Hart C did an ipi_dev->ipi_send to hart A. Hart A then cleared the msip bit, resulting in both IPI (from B and C) coalesced to a single one. In this case, A will observe the ipi_type set by B, due to the smp_wmb + __io_bw, and the IPI was indeed generated by B's write to msip. However will the barrier ensure A also observe the write to ipi_type from hart C? For the coalescing to happen, clint will observe C's write-1 before A's write-0, does it imply A'll observe C's write to ipi_type from xchg?

Does the picture below represent what you mean?

Quoted 8 lines

? B???????????? A???????????? C

? |???????????? |???????????? |
? | xchg??????? |???????????? |
? | send------->|???????????? |
? |???????????? | xchg??????? |
? |???????????? |???????????? | xchg
? |???????????? |<------------| send
? |???????????? | clear?????? |

Hi Xiang, Thank you so much for replying. I'd like to revise your chart a little bit:

Case 1, xchg_ulong in ipi_process gets reordered before ipi_clear:

?? B???????????????? A??????????????? C???????????????? CLINT

?? |???????????????? |??????????????? |????????????????? |
?? | bit_set???????? |??????????????? |????????????????? |
?? | send -----------|----------------|----------------> |
?? |???????????????? | xchg?????????? |????????????????? |
?? |???????????????? |??????????????? | bit_set????????? |
?? |???????????????? |??????????????? | send ----------> |
?? |???????????????? | clear----------|----------------> |
Quoted 6 lines

In this case, A would observe the ipi_data->ipi_type *eventually*, but when? A won't process C's IPI request until another IPI comes in the future, which might cause the IPI request to wait indefinitely. This is not an efficiency problem, but a correctness problem. I think the fix should be adding specific fence (need to reason about the correct one) between ipi_dev->ipi_clear and atomic_raw_xchg_ulong in ipi_process

Your statement is more correct.

Quoted 4 lines

Case 2, xchg_ulong doesn't get reordered, however:

Quoted 8 lines

?? B???????????????? A??????????????? C???????????????? CLINT

?? |???????????????? |??????????????? |????????????????? |
?? | bit_set???????? |??????????????? |????????????????? |
?? | send -----------|----------------|----------------> |
?? |???????????????? | xchg?????????? |??????????? /---> |
?? |???????????????? | clear----------|-----------/----> |
?? |???????????????? |??????????????? | bit_set? /?????? |
?? |???????????????? |??????????????? | send ----??????? |
 B             A             C
 |             |             |
 | bit_set     |             |
 | send------->|             | bit_set  
 |             | clear       |
 |             | xchg        |
 |             |<------------| send
 |             |             |

What I want to say about this picture is that due to different running speeds, A may get C's bit_set befor C send ipi. This will cause A to enter exception handling one more time.

Why is there an upward-sloping arrow in your picture? Multiple cores observing atomic operations@the same address should not be inconsistent.

Regards,
Xiang W

Quoted 7 lines

In this case, the same thing can happen. The request from C can be delayed indefinitely. The cause is due to the clint observes memory order differently than harts. I'm wondering if the clint can be viewed as another hart that follows RVWMO? If yes, then this case is likely invalid.

Bo

author
#5 lore raw
Quoted 15 lines · Xiang W

? 2023-11-16???? 00:41 -0800?Bo Gan???

Quoted 14 lines

Hi Xiang, Thank you so much for replying. I'd like to revise your chart a little bit:

Case 1, xchg_ulong in ipi_process gets reordered before ipi_clear:

?? B???????????????? A??????????????? C???????????????? CLINT

 ?? |???????????????? |??????????????? |????????????????? |
 ?? | bit_set???????? |??????????????? |????????????????? |
 ?? | send -----------|----------------|----------------> |
 ?? |???????????????? | xchg?????????? |????????????????? |
 ?? |???????????????? |??????????????? | bit_set????????? |
 ?? |???????????????? |??????????????? | send ----------> |
 ?? |???????????????? | clear----------|----------------> |
Quoted 8 lines
Quoted 6 lines

In this case, A would observe the ipi_data->ipi_type *eventually*, but when? A won't process C's IPI request until another IPI comes in the future, which might cause the IPI request to wait indefinitely. This is not an efficiency problem, but a correctness problem. I think the fix should be adding specific fence (need to reason about the correct one) between ipi_dev->ipi_clear and atomic_raw_xchg_ulong in ipi_process

Your statement is more correct.

For case 1, can we agree there's an issue in the current code, and it needs to be fixed?

Quoted 12 lines
Quoted 4 lines

Case 2, xchg_ulong doesn't get reordered, however:

Quoted 8 lines

?? B???????????????? A??????????????? C???????????????? CLINT

 ?? |???????????????? |??????????????? |????????????????? |
 ?? | bit_set???????? |??????????????? |????????????????? |
 ?? | send -----------|----------------|----------------> |
 ?? |???????????????? | xchg?????????? |??????????? /---> |
 ?? |???????????????? | clear----------|-----------/----> |
 ?? |???????????????? |??????????????? | bit_set? /?????? |
 ?? |???????????????? |??????????????? | send ----??????? |
Quoted 4 lines

Why is there an upward-sloping arrow in your picture? Multiple cores observing atomic operations at the same address should not be inconsistent.

Let me clarify case 2 a little bit more. Let me use a downward-sloping arrow. In this case the clear sent from A took so long that it arrived at CLINT after C's send.

     B                 A                C                 CLINT
     |                 |                |                  |
     | bit_set ------> |                |                  |
     | send -----------|----------------|----------------> |
     |                 | xchg           |                  |
     |                 | clear----------|----------------> |
     |                 |          \     |                  |
     |                 | <---------\----|-bit_set          |
     |                 |            \   | send ----------> |
     |                 |             \  |                  |
     |                 |              --|----------------> |

Back to your question. Here we have two memory/IO locations, not one. First is the ipi_data->ipi_type, second is the msip MMIO for hart A. Essentially, can the following happen?

i. Assume there's proper fence to ensure xchg orders before clear in A.

ii. From A's point of view, the xchg observes bit_set from B,
but not bit_set from C.

iii.From CLINT's point of view, it observes send from B, then send from C,
then clear from A.

I don't know how to reason about this. The only thing I can think of is the global memory order defined by RVWMO. Can we apply the global memory order to the interaction between hart and clint?

Bo

#6 lore raw

? 2023-11-16???? 01:44 -0800?Bo Gan???

Quoted 22 lines · Xiang W
Quoted 21 lines

? 2023-11-16???? 00:41 -0800?Bo Gan???

Quoted 20 lines

Hi Xiang, Thank you so much for replying. I'd like to revise your chart a little bit:

Case 1, xchg_ulong in ipi_process gets reordered before ipi_clear:

??? B???????????????? A??????????????? C???????????????? CLINT

??? |???????????????? |??????????????? |????????????????? |
??? | bit_set???????? |??????????????? |????????????????? |
??? | send -----------|----------------|----------------> |
??? |???????????????? | xchg?????????? |????????????????? |
??? |???????????????? |??????????????? | bit_set????????? |
??? |???????????????? |??????????????? | send ----------> |
??? |???????????????? | clear----------|----------------> |

In this case, A would observe the ipi_data->ipi_type *eventually*, but when? A won't process C's IPI request until another IPI comes in the future, which might cause the IPI request to wait indefinitely. This is not an efficiency problem, but a correctness problem. I think the fix should be adding specific fence (need to reason about the correct one) between ipi_dev->ipi_clear and atomic_raw_xchg_ulong in ipi_process

Quoted 5 lines
Quoted 2 lines

Your statement is more correct.

For case 1, can we agree there's an issue in the current code, and it needs to be fixed?

Yes!

Quoted 43 lines
Quoted 13 lines
Quoted 5 lines

Case 2, xchg_ulong doesn't get reordered, however:

Quoted 8 lines

??? B???????????????? A??????????????? C???????????????? CLINT

??? |???????????????? |??????????????? |????????????????? |
??? | bit_set???????? |??????????????? |????????????????? |
??? | send -----------|----------------|----------------> |
??? |???????????????? | xchg?????????? |??????????? /---> |
??? |???????????????? | clear----------|-----------/----> |
??? |???????????????? |??????????????? | bit_set? /?????? |
??? |???????????????? |??????????????? | send ----??????? |
Quoted 4 lines

Why is there an upward-sloping arrow in your picture? Multiple cores observing atomic operations at the same address should not be inconsistent.

Let me clarify case 2 a little bit more. Let me use a downward-sloping arrow. In this case the clear sent from A took so long that it arrived at CLINT after C's send.

???? B???????????????? A??????????????? C???????????????? CLINT

???? |???????????????? |??????????????? |????????????????? |
???? | bit_set ------> |??????????????? |????????????????? |
???? | send -----------|----------------|----------------> |
???? |???????????????? | xchg?????????? |????????????????? |
???? |???????????????? | clear----------|----------------> |
???? |???????????????? |????????? \???? |????????????????? |
???? |???????????????? | <---------\----|-bit_set????????? |
???? |???????????????? |??????????? \?? | send ----------> |
???? |???????????????? |???????????? \? |????????????????? |
???? |???????????????? |????????????? --|----------------> |

????????????????????????????????
Back to your question. Here we have two memory/IO locations, not one. First is the ipi_data->ipi_type, second is the msip MMIO for hart A. Essentially, can the following happen?

? i.? Assume there's proper fence to ensure xchg orders before clear in A.

? ii. From A's point of view, the xchg observes bit_set from B, ????? but not bit_set from C.

Quoted 6 lines

? iii.From CLINT's point of view, it observes send from B, then send from C, ????? then clear from A.

I don't know how to reason about this. The only thing I can think of is the global memory order defined by RVWMO. Can we apply the global memory order to the interaction between hart and clint?

I understand what you mean. The one I proposed is caused by inconsistent core speeds and has nothing to do with memory order. Do you think the memory order observed by hart and clint is inconsistent?

How to solve this? fence should only restrict memory ordering on the same core.

Regards,
Xiang W

Quoted 2 lines

Bo

author
#7 lore raw
Quoted 23 lines · Xiang W

? 2023-11-16???? 01:44 -0800?Bo Gan???

Quoted 22 lines · Xiang W
Quoted 21 lines

? 2023-11-16???? 00:41 -0800?Bo Gan???

Quoted 20 lines

Hi Xiang, Thank you so much for replying. I'd like to revise your chart a little bit:

Case 1, xchg_ulong in ipi_process gets reordered before ipi_clear:

??? B???????????????? A??????????????? C???????????????? CLINT

 ??? |???????????????? |??????????????? |????????????????? |
 ??? | bit_set???????? |??????????????? |????????????????? |
 ??? | send -----------|----------------|----------------> |
 ??? |???????????????? | xchg?????????? |????????????????? |
 ??? |???????????????? |??????????????? | bit_set????????? |
 ??? |???????????????? |??????????????? | send ----------> |
 ??? |???????????????? | clear----------|----------------> |

In this case, A would observe the ipi_data->ipi_type *eventually*, but when? A won't process C's IPI request until another IPI comes in the future, which might cause the IPI request to wait indefinitely. This is not an efficiency problem, but a correctness problem. I think the fix should be adding specific fence (need to reason about the correct one) between ipi_dev->ipi_clear and atomic_raw_xchg_ulong in ipi_process

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

Your statement is more correct.

For case 1, can we agree there's an issue in the current code, and it needs to be fixed?

Yes!

Great, and let's fix it!

Quoted 50 lines
Quoted 43 lines
Quoted 13 lines
Quoted 5 lines

Case 2, xchg_ulong doesn't get reordered, however:

Quoted 8 lines

??? B???????????????? A??????????????? C???????????????? CLINT

 ??? |???????????????? |??????????????? |????????????????? |
 ??? | bit_set???????? |??????????????? |????????????????? |
 ??? | send -----------|----------------|----------------> |
 ??? |???????????????? | xchg?????????? |??????????? /---> |
 ??? |???????????????? | clear----------|-----------/----> |
 ??? |???????????????? |??????????????? | bit_set? /?????? |
 ??? |???????????????? |??????????????? | send ----??????? |
Quoted 4 lines

Why is there an upward-sloping arrow in your picture? Multiple cores observing atomic operations at the same address should not be inconsistent.

Let me clarify case 2 a little bit more. Let me use a downward-sloping arrow. In this case the clear sent from A took so long that it arrived at CLINT after C's send.

???? B???????????????? A??????????????? C???????????????? CLINT

 ???? |???????????????? |??????????????? |????????????????? |
 ???? | bit_set ------> |??????????????? |????????????????? |
 ???? | send -----------|----------------|----------------> |
 ???? |???????????????? | xchg?????????? |????????????????? |
 ???? |???????????????? | clear----------|----------------> |
 ???? |???????????????? |????????? \???? |????????????????? |
 ???? |???????????????? | <---------\----|-bit_set????????? |
 ???? |???????????????? |??????????? \?? | send ----------> |
 ???? |???????????????? |???????????? \? |????????????????? |
 ???? |???????????????? |????????????? --|----------------> |

Back to your question. Here we have two memory/IO locations, not one. First is the ipi_data->ipi_type, second is the msip MMIO for hart A. Essentially, can the following happen?

? i.? Assume there's proper fence to ensure xchg orders before clear in A.

? ii. From A's point of view, the xchg observes bit_set from B, ????? but not bit_set from C.

Quoted 6 lines

? iii.From CLINT's point of view, it observes send from B, then send from C, ????? then clear from A.

I don't know how to reason about this. The only thing I can think of is the global memory order defined by RVWMO. Can we apply the global memory order to the interaction between hart and clint?

Quoted 8 lines

I understand what you mean. The one I proposed is caused by inconsistent core speeds and has nothing to do with memory order. Do you think the memory order observed by hart and clint is inconsistent?

How to solve this? fence should only restrict memory ordering on the same core.

Regards,
Xiang W

Yes, my issue is about memory ordering, and yes, I'm questioning the global memory order observed by hart and clint. What kind of rules do they follow? Where are the rules defined? In RISCV ISA? Or Sifive manual? If this is not possible, I'm happy to be lectured on the reasoning.

Bo

#8 lore raw

? 2023-11-16???? 03:15 -0800?Bo Gan???

Quoted 24 lines · Xiang W
Quoted 23 lines

? 2023-11-16???? 01:44 -0800?Bo Gan???

Quoted 22 lines · Xiang W
Quoted 21 lines

? 2023-11-16???? 00:41 -0800?Bo Gan???

Quoted 20 lines

Hi Xiang, Thank you so much for replying. I'd like to revise your chart a little bit:

Case 1, xchg_ulong in ipi_process gets reordered before ipi_clear:

???? B???????????????? A??????????????? C???????????????? CLINT

???? |???????????????? |??????????????? |????????????????? |
???? | bit_set???????? |??????????????? |????????????????? |
???? | send -----------|----------------|----------------> |
???? |???????????????? | xchg?????????? |????????????????? |
???? |???????????????? |??????????????? | bit_set????????? |
???? |???????????????? |??????????????? | send ----------> |
???? |???????????????? | clear----------|----------------> |

In this case, A would observe the ipi_data->ipi_type *eventually*, but when? A won't process C's IPI request until another IPI comes in the future, which might cause the IPI request to wait indefinitely. This is not an efficiency problem, but a correctness problem. I think the fix should be adding specific fence (need to reason about the correct one) between ipi_dev->ipi_clear and atomic_raw_xchg_ulong in ipi_process

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

Your statement is more correct.

For case 1, can we agree there's an issue in the current code, and it needs to be fixed?

Yes!

Great, and let's fix it!

I do not think fence can be help. We need to use spinlock to enter the critical section to protect bit_set/send and clear/xchg.

Regards,
Xiang W

Quoted 68 lines
Quoted 51 lines
Quoted 44 lines
Quoted 14 lines
Quoted 6 lines

Case 2, xchg_ulong doesn't get reordered, however:

Quoted 8 lines

???? B???????????????? A??????????????? C???????????????? CLINT

???? |???????????????? |??????????????? |????????????????? |
???? | bit_set???????? |??????????????? |????????????????? |
???? | send -----------|----------------|----------------> |
???? |???????????????? | xchg?????????? |??????????? /---> |
???? |???????????????? | clear----------|-----------/----> |
???? |???????????????? |??????????????? | bit_set? /?????? |
???? |???????????????? |??????????????? | send ----??????? |
Quoted 4 lines

Why is there an upward-sloping arrow in your picture? Multiple cores observing atomic operations at the same address should not be inconsistent.

Let me clarify case 2 a little bit more. Let me use a downward-sloping arrow. In this case the clear sent from A took so long that it arrived at CLINT after C's send.

????? B???????????????? A??????????????? C???????????????? CLINT

????? |???????????????? |??????????????? |????????????????? |
????? | bit_set ------> |??????????????? |????????????????? |
????? | send -----------|----------------|----------------> |
????? |???????????????? | xchg?????????? |????????????????? |
????? |???????????????? | clear----------|----------------> |
????? |???????????????? |????????? \???? |????????????????? |
????? |???????????????? | <---------\----|-bit_set????????? |
????? |???????????????? |??????????? \?? | send ----------> |
????? |???????????????? |???????????? \? |????????????????? |
????? |???????????????? |????????????? --|----------------> |

?????????????????????????????????
Back to your question. Here we have two memory/IO locations, not one. First is the ipi_data->ipi_type, second is the msip MMIO for hart A. Essentially, can the following happen?

?? i.? Assume there's proper fence to ensure xchg orders before clear in A.

?? ii. From A's point of view, the xchg observes bit_set from B, ?????? but not bit_set from C.

Quoted 6 lines

?? iii.From CLINT's point of view, it observes send from B, then send from C, ?????? then clear from A.

I don't know how to reason about this. The only thing I can think of is the global memory order defined by RVWMO. Can we apply the global memory order to the interaction between hart and clint?

Quoted 8 lines

I understand what you mean. The one I proposed is caused by inconsistent core speeds and has nothing to do with memory order. Do you think the memory order observed by hart and clint is inconsistent?

How to solve this? fence should only restrict memory ordering on the same core.

Regards,
Xiang W

Yes, my issue is about memory ordering, and yes, I'm questioning the global memory order observed by hart and clint. What kind of rules do they follow? Where are the rules defined? In RISCV ISA? Or Sifive manual? If this is not possible, I'm happy to be lectured on the reasoning.

Bo

#9 lore raw
In reply to Bo Gan

? 2023-11-16???? 03:15 -0800?Bo Gan???

Quoted 24 lines · Xiang W
Quoted 23 lines

? 2023-11-16???? 01:44 -0800?Bo Gan???

Quoted 22 lines · Xiang W
Quoted 21 lines

? 2023-11-16???? 00:41 -0800?Bo Gan???

Quoted 20 lines

Hi Xiang, Thank you so much for replying. I'd like to revise your chart a little bit:

Case 1, xchg_ulong in ipi_process gets reordered before ipi_clear:

???? B???????????????? A??????????????? C???????????????? CLINT

???? |???????????????? |??????????????? |????????????????? |
???? | bit_set???????? |??????????????? |????????????????? |
???? | send -----------|----------------|----------------> |
???? |???????????????? | xchg?????????? |????????????????? |
???? |???????????????? |??????????????? | bit_set????????? |
???? |???????????????? |??????????????? | send ----------> |
???? |???????????????? | clear----------|----------------> |

In this case, A would observe the ipi_data->ipi_type *eventually*, but when? A won't process C's IPI request until another IPI comes in the future, which might cause the IPI request to wait indefinitely. This is not an efficiency problem, but a correctness problem. I think the fix should be adding specific fence (need to reason about the correct one) between ipi_dev->ipi_clear and atomic_raw_xchg_ulong in ipi_process

Quoted 58 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

Your statement is more correct.

For case 1, can we agree there's an issue in the current code, and it needs to be fixed?

Yes!

Great, and let's fix it!

Quoted 49 lines
Quoted 49 lines
Quoted 13 lines
Quoted 5 lines

Case 2, xchg_ulong doesn't get reordered, however:

Quoted 8 lines

???? B???????????????? A??????????????? C???????????????? CLINT

???? |???????????????? |??????????????? |????????????????? |
???? | bit_set???????? |??????????????? |????????????????? |
???? | send -----------|----------------|----------------> |
???? |???????????????? | xchg?????????? |??????????? /---> |
???? |???????????????? | clear----------|-----------/----> |
???? |???????????????? |??????????????? | bit_set? /?????? |
???? |???????????????? |??????????????? | send ----??????? |
Quoted 4 lines

Why is there an upward-sloping arrow in your picture? Multiple cores observing atomic operations at the same address should not be inconsistent.

Let me clarify case 2 a little bit more. Let me use a downward-sloping arrow. In this case the clear sent from A took so long that it arrived at CLINT after C's send.

????? B???????????????? A??????????????? C???????????????? CLINT

????? |???????????????? |??????????????? |????????????????? |
????? | bit_set ------> |??????????????? |????????????????? |
????? | send -----------|----------------|----------------> |
????? |???????????????? | xchg?????????? |????????????????? |
????? |???????????????? | clear----------|----------------> |
????? |???????????????? |????????? \???? |????????????????? |
????? |???????????????? | <---------\----|-bit_set????????? |
????? |???????????????? |??????????? \?? | send ----------> |
????? |???????????????? |???????????? \? |????????????????? |
????? |???????????????? |????????????? --|----------------> |

?????????????????????????????????
Back to your question. Here we have two memory/IO locations, not one. First is the ipi_data->ipi_type, second is the msip MMIO for hart A. Essentially, can the following happen?

?? i.? Assume there's proper fence to ensure xchg orders before clear in A.

?? ii. From A's point of view, the xchg observes bit_set from B, ?????? but not bit_set from C.

?? iii.From CLINT's point of view, it observes send from B, then send from C, ?????? then clear from A.

I don't know how to reason about this. The only thing I can think of is the global memory order defined by RVWMO. Can we apply the global memory order to the interaction between hart and clint?

It is not normal for the peripheral to observe a reorder. This is obvious on a single hart because many operations have timing requirements. This should also be true on multiple harts, otherwise an error will occur when a device is shared by multiple harts.

Regards,
Xiang W

Quoted 18 lines
Quoted 9 lines

I understand what you mean. The one I proposed is caused by inconsistent core speeds and has nothing to do with memory order. Do you think the memory order observed by hart and clint is inconsistent?

How to solve this? fence should only restrict memory ordering on the same core.

Regards,
Xiang W

Yes, my issue is about memory ordering, and yes, I'm questioning the global memory order observed by hart and clint. What kind of rules do they follow? Where are the rules defined? In RISCV ISA? Or Sifive manual? If this is not possible, I'm happy to be lectured on the reasoning.

Bo

#10 lore raw
In reply to Bo Gan
Quoted 25 lines · On Sat, Nov 11, 2023 at 11:36?AM Bo Gan <[email protected]> wrote:

Hi OpenSBI community,

Current IPI send/recv implementation uses ipi_data->ipi_type to indicate an IPI event pending. In sbi_ipi_send, I see have the following:

Quoted 9 lines

/*
* Set IPI type on remote hart's scratch area and
* trigger the interrupt
*/
atomic_raw_set_bit(event, &ipi_data->ipi_type);
smp_wmb();

if (ipi_dev && ipi_dev->ipi_send)

      ipi_dev->ipi_send(remote_hartindex)

And in sbi_ipi_process, the following:

Quoted 6 lines

if (ipi_dev && ipi_dev->ipi_clear)

      ipi_dev->ipi_clear(hartindex);

ipi_type = atomic_raw_xchg_ulong(&ipi_data->ipi_type, 0); ipi_event = 0;
while (ipi_type) {

Quoted 6 lines

I'm think of if there's any possibility of missed IPIs. Assuming we are talking about sifive/clint (aclint_mswi.c), I know it means that there're both smp_wmb (fence w,w) and __io_bw (fence w,o) between atomic_raw_set_bit and the write to remote hart's msip register in clint. Consider the following

Right before Hart A ipi_dev->ipi_clear() for the msip write-1 issued by hart B,

This means hart A is already in sbi_ipi_process() and has not yet read its own ipi_type using atomic_raw_xchg_ulong().

This also means hart B has issued smp_wmb() which is before ipi_send() hence the changes in hart A ipi_type done by hart B must be visible to hart A.

Quoted 1 line

Hart C did an ipi_dev->ipi_send to hart A. Hart A then cleared the msip bit,

This also means hart C has issued smp_wmb() which is before ipi_send() hence the changes in hart A ipi_type done by hart C must be visible to hart A.

Quoted 1 line

resulting in both IPI (from B and C) coalesced to a single one. In this case,

This is perfectly fine.

We can have N harts (N > 1) do ipi_send() to hart A just before hart A does ipi_clear(). The fact that hart A is about to do ipi_clear() it means hart A is already in sbi_ipi_process() and it is going to atomically read-n-clear its own ipi_type immediately after ipi_clear() so hart A will not miss IPI from any of the N harts.

Quoted 3 lines

A will observe the ipi_type set by B, due to the smp_wmb + __io_bw, and the IPI was indeed generated by B's write to msip. However will the barrier ensure A also observe the write to ipi_type from hart C? For the coalescing to happen,

This is an incorrect understanding of ipi_type. The ipi_type is in-fact a bitmask
of different IPIs muxed over a single physical IPI. Further, we are using atomic operations to guarantee multiple harts can safely update ipi_type.

Quoted 2 lines

clint will observe C's write-1 before A's write-0, does it imply A'll observe C's write to ipi_type from xchg?

Yes, xchg() called by hart A will observe hart C's changes to ipi_type because we have smp_wmb() just before ipi_send().

Quoted 7 lines

I doubt it mainly due to 2 reasons:

1. ipi_dev->ipi_clear is a writel(msip), which has __io_bw (fence w,o) I don't think it helps preventing the writel to happen before xchg. If that happens, it's possible that A missed C's write to ipi_type.

Quoted 6 lines

2. Suppose the ipi_clear() happened before xchg, from A's standpoint, it's a write, not a read. A wasn't reading from msip, but only writing to it. I'm not sure if A's write to msip after C's write (observed by clint) implies A's observing the same order. A is not reading from msip, so what helps to establish this order? Will the RVWMO global memory order help? Can we think of clint as another hart that follows RVWMO?

The clint HW is supposed to handle writes coming from multiple harts so we can safely assume that multiple writes reaching to clint will be completed in the order in which clint observes them.

Things can only break if hart A does xchg() out-of-order before writel() issued by ipi_clear() completes.

This can be understood by an example which is as follows:
1) hart A orders xchg() before writel() issued by ipi_clear()
2) hart B is about to do set_bit() for hart A ipi_type but hart A does xchg() before hart B does set_bit() so hart A will not observe anything in ipi_type
3) hart B does set_bit() and ipi_send() before hart A does writel() from ipi_clear()

The above result is hart A totally missing IPI from hart B but this is only possible if hart A orders xchg() before writel() from ipi_clear(). This is a very very rare scenario not observed on any RISC-V implementations so far but to prevent this we need a matching smp_wmb() between ipi_clear() and xchg() in sbi_ipi_process().

Quoted 7 lines

Bo

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Regards,
Anup

author
#11 lore raw
Quoted 29 lines · Anup Patel
Quoted 7 lines

I doubt it mainly due to 2 reasons:

1. ipi_dev->ipi_clear is a writel(msip), which has __io_bw (fence w,o) I don't think it helps preventing the writel to happen before xchg. If that happens, it's possible that A missed C's write to ipi_type.

Quoted 6 lines

2. Suppose the ipi_clear() happened before xchg, from A's standpoint, it's a write, not a read. A wasn't reading from msip, but only writing to it. I'm not sure if A's write to msip after C's write (observed by clint) implies A's observing the same order. A is not reading from msip, so what helps to establish this order? Will the RVWMO global memory order help? Can we think of clint as another hart that follows RVWMO?

The clint HW is supposed to handle writes coming from multiple harts so we can safely assume that multiple writes reaching to clint will be completed in the order in which clint observes them.

Things can only break if hart A does xchg() out-of-order before writel() issued by ipi_clear() completes.

This can be understood by an example which is as follows:
1) hart A orders xchg() before writel() issued by ipi_clear()
2) hart B is about to do set_bit() for hart A ipi_type but hart A does xchg() before hart B does set_bit() so hart A will not observe anything in ipi_type
3) hart B does set_bit() and ipi_send() before hart A does writel() from ipi_clear()

Quoted 7 lines

The above result is hart A totally missing IPI from hart B but this is only possible if hart A orders xchg() before writel() from ipi_clear(). This is a very very rare scenario not observed on any RISC-V implementations so far but to prevent this we need a matching smp_wmb() between ipi_clear() and xchg() in sbi_ipi_process().

Hi Anup,

Thank you so much for the detailed explanation. I totally get your point. I just have one doubt about smp_wmb. In OpenSBI, smp_wmb is RISCV_FENCE(w,w). If I understand correctly, it won't enforce MMIO order. Given your analysis, I assume you meant wmb(), which is RISCV_FENCE(ow,ow). Am I right? Or do we want more precise fences, such as:

RISCV_FENCE(w,o) between set_bit(ipi_type) and ipi_dev->ipi_send in Hart B RISCV_FENCE(o,w) between ipi_dev->ipi_clear and xchg(ipi_type) in Hart A

Let me know your thoughts, and I can make a patch and send to mailing list.

Bo

#12 lore raw
Quoted 47 lines · On Tue, Nov 21, 2023 at 10:51?AM Bo Gan <[email protected]> wrote:
Quoted 29 lines · Anup Patel
Quoted 7 lines

I doubt it mainly due to 2 reasons:

1. ipi_dev->ipi_clear is a writel(msip), which has __io_bw (fence w,o) I don't think it helps preventing the writel to happen before xchg. If that happens, it's possible that A missed C's write to ipi_type.

Quoted 6 lines

2. Suppose the ipi_clear() happened before xchg, from A's standpoint, it's a write, not a read. A wasn't reading from msip, but only writing to it. I'm not sure if A's write to msip after C's write (observed by clint) implies A's observing the same order. A is not reading from msip, so what helps to establish this order? Will the RVWMO global memory order help? Can we think of clint as another hart that follows RVWMO?

The clint HW is supposed to handle writes coming from multiple harts so we can safely assume that multiple writes reaching to clint will be completed in the order in which clint observes them.

Things can only break if hart A does xchg() out-of-order before writel() issued by ipi_clear() completes.

This can be understood by an example which is as follows:
1) hart A orders xchg() before writel() issued by ipi_clear()
2) hart B is about to do set_bit() for hart A ipi_type but hart A does xchg() before hart B does set_bit() so hart A will not observe anything in ipi_type
3) hart B does set_bit() and ipi_send() before hart A does writel() from ipi_clear()

Quoted 7 lines

The above result is hart A totally missing IPI from hart B but this is only possible if hart A orders xchg() before writel() from ipi_clear(). This is a very very rare scenario not observed on any RISC-V implementations so far but to prevent this we need a matching smp_wmb() between ipi_clear() and xchg() in sbi_ipi_process().

Hi Anup,

Thank you so much for the detailed explanation. I totally get your point. I just have one doubt about smp_wmb. In OpenSBI, smp_wmb is RISCV_FENCE(w,w). If I understand correctly, it won't enforce MMIO order. Given your analysis, I assume you meant wmb(), which is RISCV_FENCE(ow,ow). Am I right? Or do we want more precise fences, such as:

Quoted 4 lines

RISCV_FENCE(w,o) between set_bit(ipi_type) and ipi_dev->ipi_send in Hart B RISCV_FENCE(o,w) between ipi_dev->ipi_clear and xchg(ipi_type) in Hart A

Let me know your thoughts, and I can make a patch and send to mailing list.

The smp_wmb() is fine between set_bit(ipi_type) and ipi_dev->ipi_send but we should use wmb() between ipi_clear() and xchg() in sbi_ipi_process().

Quoted 8 lines

Bo

--
opensbi mailing list
opensbi at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Regards,
Anup