is it possible to avoid syncing after an rdma write?
Right now, RDS follows each RDMA write op with a Send op, which 1) causes an interrupt and 2) includes the info we need to call ib_dma_sync_sg_for_cpu() for the target of the rdma write.
We want to omit the Send. If we don't do the sync on the machine that is the target of the RDMA write, the result is... what exactly? I assume the write to memory is snooped by CPUs, so their cachelines will be properly invalidated. However, Linux DMA-API docs seem pretty clear in insisting on the sync.
Is the issue IOMMUs? Or for compatibility with bounce buffering?
Thanks in advance -- Regards -- Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Andy Grover
Right now, RDS follows each RDMA write op with a Send op, which 1) causes an interrupt and 2) includes the info we need to call
ib_dma_sync_sg_for_cpu()for the target of the rdma write.We want to omit the Send. If we don't do the sync on the machine that is the target of the RDMA write, the result is... what exactly? I assume the write to memory is snooped by CPUs, so their cachelines will be properly invalidated. However, Linux DMA-API docs seem pretty clear in insisting on the sync.
I'm curious about this too, but I will point out that at least the user RDMA interface has no match for the kernel DMA calls, so in practice RDMA does not work on systems that require them. That means bounce buffering is not used and IO/CPU caches are coherent.
Though, I guess, the kernel could use weaker memory ordering types in kernel mode that do require the DMA api calls.
Quoted 1 line
Is the issue IOMMUs? Or for compatibility with bounce buffering?
As long as the memory is registered the IOMMU should remain configured.
What do you intend to replace the SEND with? spin on last byte? There are other issues to consider like ordering within the PCI-E fabric..
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20100217005827.GF16490-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Why not use an RDMA write w/ immed? That forces the consumption of a receive WQE and can be used to create a completion event. Since the immediate data is carried in the last packet of a multi-packet RDMA write, you are guaranteed that all data has been placed in the receive buffer, in order.
I'm a hardware guy, so this may be completely off-the-wall w.r.t. this particular discussion.
-Paul
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jason Gunthorpe
Sent: Tuesday, February 16, 2010 4:58 PM
To: Andy Grover
Cc: [email protected]
Subject: Re: is it possible to avoid syncing after an rdma write?
Quoted 9 lines · Andy Grover
Right now, RDS follows each RDMA write op with a Send op, which 1) causes an interrupt and 2) includes the info we need to call
ib_dma_sync_sg_for_cpu()for the target of the rdma write.We want to omit the Send. If we don't do the sync on the machine that is the target of the RDMA write, the result is... what exactly? I assume the write to memory is snooped by CPUs, so their cachelines will be properly invalidated. However, Linux DMA-API docs seem pretty clear in insisting on the sync.
I'm curious about this too, but I will point out that at least the user RDMA interface has no match for the kernel DMA calls, so in practice RDMA does not work on systems that require them. That means bounce buffering is not used and IO/CPU caches are coherent.
Though, I guess, the kernel could use weaker memory ordering types in kernel mode that do require the DMA api calls.
Quoted 1 line
Is the issue IOMMUs? Or for compatibility with bounce buffering?
As long as the memory is registered the IOMMU should remain configured.
What do you intend to replace the SEND with? spin on last byte? There are other issues to consider like ordering within the PCI-E fabric..
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Paul Grun
Why not use an RDMA write w/ immed? That forces the consumption of a receive WQE and can be used to create a completion event. Since the immediate data is carried in the last packet of a multi-packet RDMA write, you are guaranteed that all data has been placed in the receive buffer, in order.
Yes, RDMA WRITE w/ immediate data is perfectly fine. I've even implemented some protocols that use it to good effect.
Not sure what the performance trade off is like though. The immediate data pretty much behaves exactly like a SEND WC on the receive side, but there may be some performance and latency advantages, particularly on the send side.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20100217011224.GH16490-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Two advantages come to mind vs an RDMA Write followed by a SEND:
Using a SEND will consume a second WQE on the send side, and the synchronizing SEND will cause an entire new transaction, which will consume a(n infinitesimally) small amount of additional wire bandwidth, as well as incurring a(infinitesimally) small likelihood of a dropped or lost packets.
Nits? Yes, probably infinitesimally small ones. (hardware guys tend to
worry about the small ones.)
To answer Andy's original question, the behavior on the receive side is not guaranteed until control of the receive buffer has been formally returned to the receiver. I expect that most HCAs are pretty well behaved here, as are most CPU/memory/root complexes...but you never know. Can anybody guarantee that the inbound packet gets written to the memory in order?
If something odd did happen, it seems like one of those places that would require an incredible stroke of luck to debug.
OTOH, I know that many applications simply poll the receive buffer looking for a flag everyday and get away with it.
-----Original Message-----
From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
Sent: Tuesday, February 16, 2010 5:12 PM
To: Paul Grun
Cc: 'Andy Grover'; [email protected]
Subject: Re: is it possible to avoid syncing after an rdma write?
Quoted 5 lines · Paul Grun
Why not use an RDMA write w/ immed? That forces the consumption of a receive WQE and can be used to create a completion event. Since the immediate data is carried in the last packet of a multi-packet RDMA write, you are guaranteed that all data has been placed in the receive buffer, in order.
Yes, RDMA WRITE w/ immediate data is perfectly fine. I've even implemented some protocols that use it to good effect.
Not sure what the performance trade off is like though. The immediate data pretty much behaves exactly like a SEND WC on the receive side, but there may be some performance and latency advantages, particularly on the send side.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Paul Grun
Two advantages come to mind vs an RDMA Write followed by a SEND:
Using a SEND will consume a second WQE on the send side, and the synchronizing SEND will cause an entire new transaction, which will consume a(n infinitesimally) small amount of additional wire bandwidth, as well as incurring a(infinitesimally) small likelihood of a dropped or lost packets.
The practical reason I've used it is because it doesn't consume an unack'd message slot. My apps can exceed the max unacked messages in flight ...
Quoted 6 lines
To answer Andy's original question, the behavior on the receive side is not guaranteed until control of the receive buffer has been formally returned to the receiver. I expect that most HCAs are pretty well behaved here, as are most CPU/memory/root complexes...but you never know. Can anybody guarantee that the inbound packet gets written to the memory in order?
The HCA could guarentee this. It needs to issue the PCI-E transaction that contains the last byte with relaxed ordering cleared, after all other transactions have been issued. In this case it would have identical synchronizing properties to the WCE write. PCI-E requires that the host observe the writes in a transaction in-order and that a host observe multiple transactions according to PCI-E rules.
Presumably common HCAs do all this to support MPI.
Andy, I think if want to do poll on completion in the kernel then maybe introducing a new ib_dma_poll call for that purpose is a good idea? First crack could just fail if there is bounce buffering, non-coherent caching, etc.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Jason Gunthorpe
Quoted 9 lines · Andy Grover
Right now, RDS follows each RDMA write op with a Send op, which 1) causes an interrupt and 2) includes the info we need to call
ib_dma_sync_sg_for_cpu()for the target of the rdma write.We want to omit the Send. If we don't do the sync on the machine that is the target of the RDMA write, the result is... what exactly? I assume the write to memory is snooped by CPUs, so their cachelines will be properly invalidated. However, Linux DMA-API docs seem pretty clear in insisting on the sync.
<snip>
Quoted 2 lines
What do you intend to replace the SEND with? spin on last byte? There are other issues to consider like ordering within the PCI-E fabric..
Well, hopefully nothing. What I'm looking for is to write to a target region multiple times, as efficiently as possible, but be able to occasionally read it on the target machine and get consistent results. I definitely don't want to take an event, and avoiding the CQE would be nice.
What I'm hearing is that I don't have to worry about what the Linux DMA-API docs say about noncoherent mappings, but I need to be mindful of IB spec 9.5 section o9-20:
---
o9-20: An application shall not depend of the contents of an RDMA WRITE buffer at the responder until one of the following has occurred:
* Arrival and Completion of the last RDMA WRITE request packet when used with Immediate data.
* Arrival and completion of a subsequent SEND message.
* Update of a memory element by a subsequent ATOMIC operation.
---
So if I do an RDMA write and follow it up with an atomic op, it sounds like I can achieve the behavior I want, and without an event or CQE. Although for my particular use case with ongoing writes, the CPU couldn't fetch more than one value (64bit?) without potentially reading data from a later write, I would think.
Regards -- Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines · Andy Grover
Quoted 2 lines
What do you intend to replace the SEND with? spin on last byte? There are other issues to consider like ordering within the PCI-E fabric..
Well, hopefully nothing. What I'm looking for is to write to a target region multiple times, as efficiently as possible, but be able to occasionally read it on the target machine and get consistent results. I definitely don't want to take an event, and avoiding the CQE would be nice.
Ahhh, interesting, I've thought about doing something like that as well. Sounds to me like you want to often RDMA WRITE some state information and have the CPU read that state from time to time, ie some kind of pointer values or whatever.
I didn't come to a satisfactory method and gave up on the idea..
IMHO, the critical problem to solve is that you cannot re-write over the same region again and again. Guaranteeing CPU and RDMA consistency is hard. For instance if the CPU reads two 64 bit values from your WRITE region there is no way to guarentee anything about them, other than all of the bytes were written at some point by the far side.
For instance, a 32 bit CPU might read a 64 bit value with two memory transactions and there is no chance of guaranteed coherence.
Basically, it depends on your requirements for the data. If you have an array of 32 bit values that have no inter-relationships then I think it can work OK. Anything else becomes alot harder.
Quoted 3 lines
What I'm hearing is that I don't have to worry about what the Linux DMA-API docs say about noncoherent mappings, but I need to be mindful of IB spec 9.5 section o9-20:
You cannot ignore it completely, but to support userspace there is a way to ensure you get the right kind of mapping for this to work.
Quoted 5 lines
So if I do an RDMA write and follow it up with an atomic op, it sounds like I can achieve the behavior I want, and without an event or CQE. Although for my particular use case with ongoing writes, the CPU couldn't fetch more than one value (64bit?) without potentially reading data from a later write, I would think.
You don't need the atomic at all, it doesn't do anything if you intend to start another RDMA WRITE to the same memory soon. The problem you face is not knowing when the last write finished but knowing when the next write is going to start.
sizeof(atomic_t) is probably all you get, which will be 32 bits on 32 bit Linux.
For instance, a strategy that can work OK would to have an array of your states and the far side RDMA WRITEs into consecutive positions and uses an unsignaled immediate data to indicate the tail. The recv side runs through the CQEs and determines the latest write region. If you run out of slots or out of CQEs then the sender waits for more..
Or replace the immediate data with a last-byte-written poll (like MPI).
Either way, the key is that you are never writing twice without synchronizing both sides.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Andy Grover
RDS follows each RDMA write op with a Send op [...] we want to omit the Send
Andy,
This way or another the side which isn't initiating the rdma write has to be notified that the local buffer && rkey (stag) they advertised can now invalidated from the HCA/RNIC IOMMU, its mapping from the node IOMMU, returned to the pool it was allocated from, reclaimed by higher layers etc.
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html