From: Bo Gan Date: Fri, 10 Nov 2023 22:05:58 -0800 Subject: Question about potential missed IPI events Message-ID: <35016005-5bf6-d80b-2a5f-6803afea729e@gmail.com> List-Id: To: opensbi@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi OpenSBI community, Current IPI send/recv implementation uses ipi_data->ipi_type to indicate an IPI event pending. In sbi_ipi_send, I see have the following: > /* > * Set IPI type on remote hart's scratch area and > * trigger the interrupt > */ > atomic_raw_set_bit(event, &ipi_data->ipi_type); > smp_wmb(); > > if (ipi_dev && ipi_dev->ipi_send) > ipi_dev->ipi_send(remote_hartindex) And in sbi_ipi_process, the following: > if (ipi_dev && ipi_dev->ipi_clear) > ipi_dev->ipi_clear(hartindex); > > ipi_type = atomic_raw_xchg_ulong(&ipi_data->ipi_type, 0); > ipi_event = 0; > while (ipi_type) { I'm think of if there's any possibility of missed IPIs. Assuming we are talking about sifive/clint (aclint_mswi.c), I know it means that there're both smp_wmb (fence w,w) and __io_bw (fence w,o) between atomic_raw_set_bit and the write to remote hart's msip register in clint. Consider the following Right before Hart A ipi_dev->ipi_clear() for the msip write-1 issued by hart B, Hart C did an ipi_dev->ipi_send to hart A. Hart A then cleared the msip bit, resulting in both IPI (from B and C) coalesced to a single one. In this case, A will observe the ipi_type set by B, due to the smp_wmb + __io_bw, and the IPI was indeed generated by B's write to msip. However will the barrier ensure A also observe the write to ipi_type from hart C? For the coalescing to happen, clint will observe C's write-1 before A's write-0, does it imply A'll observe C's write to ipi_type from xchg? I doubt it mainly due to 2 reasons: 1. ipi_dev->ipi_clear is a writel(msip), which has __io_bw (fence w,o) I don't think it helps preventing the writel to happen before xchg. If that happens, it's possible that A missed C's write to ipi_type. 2. Suppose the ipi_clear() happened before xchg, from A's standpoint, it's a write, not a read. A wasn't reading from msip, but only writing to it. I'm not sure if A's write to msip after C's write (observed by clint) implies A's observing the same order. A is not reading from msip, so what helps to establish this order? Will the RVWMO global memory order help? Can we think of clint as another hart that follows RVWMO? Bo