From: Bo Gan Date: Wed, 15 Nov 2023 16:33:29 -0800 Subject: Question about potential missed IPI events In-Reply-To: <35016005-5bf6-d80b-2a5f-6803afea729e@gmail.com> References: <35016005-5bf6-d80b-2a5f-6803afea729e@gmail.com> Message-ID: List-Id: To: opensbi@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On 11/10/23 10:05 PM, Bo Gan wrote: > Hi OpenSBI community, > > Current IPI send/recv implementation uses ipi_data->ipi_type to indicate > an IPI event pending. In sbi_ipi_send, I see have the following: > >> /* >> ?* Set IPI type on remote hart's scratch area and >> ?* trigger the interrupt >> ?*/ >> atomic_raw_set_bit(event, &ipi_data->ipi_type); >> smp_wmb(); >> >> if (ipi_dev && ipi_dev->ipi_send) >> ????ipi_dev->ipi_send(remote_hartindex) > > And in sbi_ipi_process, the following: > >> if (ipi_dev && ipi_dev->ipi_clear) >> ????ipi_dev->ipi_clear(hartindex); >> >> ipi_type = atomic_raw_xchg_ulong(&ipi_data->ipi_type, 0); >> ipi_event = 0; >> while (ipi_type) { > > I'm think of if there's any possibility of missed IPIs. Assuming we are > talking about sifive/clint (aclint_mswi.c), I know it means that there're > both smp_wmb (fence w,w) and __io_bw (fence w,o) between atomic_raw_set_bit > and the write to remote hart's msip register in clint. Consider the following > > Right before Hart A ipi_dev->ipi_clear() for the msip write-1 issued by hart B, > Hart C did an ipi_dev->ipi_send to hart A. Hart A then cleared the msip bit, > resulting in both IPI (from B and C) coalesced to a single one. In this case, > A will observe the ipi_type set by B, due to the smp_wmb + __io_bw, and the IPI > was indeed generated by B's write to msip. However will the barrier ensure A > also observe the write to ipi_type from hart C? For the coalescing to happen, > clint will observe C's write-1 before A's write-0, does it imply A'll observe > C's write to ipi_type from xchg? > > I doubt it mainly due to 2 reasons: > > 1. ipi_dev->ipi_clear is a writel(msip), which has __io_bw (fence w,o) I don't > ?? think it helps preventing the writel to happen before xchg. If that happens, > ?? it's possible that A missed C's write to ipi_type. > > 2. Suppose the ipi_clear() happened before xchg, from A's standpoint, it's a > ?? write, not a read. A wasn't reading from msip, but only writing to it. I'm > ?? not sure if A's write to msip after C's write (observed by clint) implies > ?? A's observing the same order. A is not reading from msip, so what helps to > ?? establish this order? Will the RVWMO global memory order help? Can we think > ?? of clint as another hart that follows RVWMO? > > Bo > Hi @Atish @Anup and other folks, any thoughts on this?