strong ordering for data registered memory
Some time ago there was an email sent to this group with the subject "weak ordering for data registered memory". I don't recall any action resulting from this thread. So, I have a question. If a bit were defined to specify "strong ordering", perhaps as a "access" flag (see ibv_access_flags) and used with ibv_reg_mr(), would that be sufficient for (1) client applications that need a HW "guarantee" of writing the last byte of an RDMA last and (2) platform implementations that need to deliver that feature?
-David
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines
Some time ago there was an email sent to this group with the subject "weak ordering for data registered memory". I don't recall any action resulting from this thread. So, I have a question. If a bit were defined to specify "strong ordering", perhaps as a "access" flag (see
ibv_access_flags) and used withibv_reg_mr(), would that be sufficient for (1) client applications that need a HW "guarantee" of writing the last byte of an RDMA last and (2) platform implementations that need to deliver that feature?
What would happen if an application asked for strong ordering and the adapter and/or platform is not capable of that?
Weak ordering is a bit easier to handle -- the app is saying "if you can make things go faster, don't worry about ordering here" and a platform where it doesn't matter can just ignore it.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Today apps are forced to assume that all transports can not provide strong ordering.. and hence must implement solutions to work around this.
There are specific optimizations an app might make if it knows the underpinning transport can make these guarantees..
It would be useful if "strong ordering" were exposed as attribute from a transport.. and as well - have the ability to provide a hint to enable "strong ordering" on either registration or per operation or at the qp level .
Are there any HCAs that provide "strong ordering" today ?
Quoted 16 lines · Roland Dreier
Quoted 8 lines
Some time ago there was an email sent to this group with the subject "weak ordering for data registered memory". I don't recall any action resulting from this thread. So, I have a question. If a bit were defined to specify "strong ordering", perhaps as a "access" flag (see
ibv_access_flags) and used withibv_reg_mr(), would that be sufficient for (1) client applications that need a HW "guarantee" of writing the last byte of an RDMA last and (2) platform implementations that need to deliver that feature?What would happen if an application asked for strong ordering and the adapter and/or platform is not capable of that?
Weak ordering is a bit easier to handle -- the app is saying "if you can make things go faster, don't worry about ordering here" and a platform where it doesn't matter can just ignore it.
Quoted 6 lines
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
I decided to minimize the impact of an API change on the class of applications that use the current verbs interface because those applications can safely run on platforms that deliver optimal performance using weak ordering for data buffers. New binaries aren't required for this class of application.
I thought it would be more appropriate to put the burden of added complexity on the class of applications that bypass the verbs to access special features in the hardware. In fact, those applications are selective about memory regions that need this special handling and would register lots of memory without the "strong ordering' bit. How applications determine that the platform is capable of performing the request would be beyond the scope of the verbs, however, I suppose that the verbs framework could check and return an error.
If there are applications that expect the hardware to support "strong ordering" and don't check the hardware, then these might be a problem. Do any of these exists?
By the way, if I had proposed this bit several years ago, then I would have chosen a "weak ordering" flag. Instead, I decided to try protecting the existing base of verbs-based software.
-David
Quoted 16 lines · Roland Dreier
Quoted 8 lines
Some time ago there was an email sent to this group with the subject "weak ordering for data registered memory". I don't recall any action resulting from this thread. So, I have a question. If a bit were defined to specify "strong ordering", perhaps as a "access" flag (see
ibv_access_flags) and used withibv_reg_mr(), would that be sufficient for (1) client applications that need a HW "guarantee" of writing the last byte of an RDMA last and (2) platform implementations that need to deliver that feature?What would happen if an application asked for strong ordering and the adapter and/or platform is not capable of that?
Weak ordering is a bit easier to handle -- the app is saying "if you can make things go faster, don't worry about ordering here" and a platform where it doesn't matter can just ignore it.
Quoted 6 lines
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Yes, there are HCAs that provide strong ordering. And an application such as OpenMPI checks the HCA model and if appropriate enables a mechanism called "eager RDMA" that depends on it.
-David
Quoted 31 lines · Richard Frank
Today apps are forced to assume that all transports can not provide strong ordering.. and hence must implement solutions to work around this.
There are specific optimizations an app might make if it knows the underpinning transport can make these guarantees..
It would be useful if "strong ordering" were exposed as attribute from a transport.. and as well - have the ability to provide a hint to enable "strong ordering" on either registration or per operation or at the qp level .
Are there any HCAs that provide "strong ordering" today ?
Quoted 18 lines · Roland Dreier
Quoted 2 lines
Some time ago there was an email sent to this group with the subject "weak ordering for data registered memory". I don't recall any
action
Quoted 3 lines
resulting from this thread. So, I have a question. If a bit were defined to specify "strong ordering", perhaps as a "access" flag (see
ibv_access_flags) and used withibv_reg_mr(), would that besufficient
Quoted 3 lines
for (1) client applications that need a HW "guarantee" of writing the last byte of an RDMA last and (2) platform implementations that need to deliver that feature?
What would happen if an application asked for strong ordering and the adapter and/or platform is not capable of that?
Weak ordering is a bit easier to handle -- the app is saying "if you can make things go faster, don't worry about ordering here" and a platform where it doesn't matter can just ignore it.
Quoted 10 lines
Quoted 6 lines
- R.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Would anyone like to through out the list of HCAs that do this... I can guess at a few... and can ask the vendors directly.. if not.. .
It would be much nicer to not hardcode names of adapters.. but that won't stop us.. :)
Quoted 44 lines · David Brean
Yes, there are HCAs that provide strong ordering. And an application such as OpenMPI checks the HCA model and if appropriate enables a mechanism called "eager RDMA" that depends on it.
-David
Quoted 39 lines · Richard Frank
Today apps are forced to assume that all transports can not provide strong ordering.. and hence must implement solutions to work around this.
There are specific optimizations an app might make if it knows the underpinning transport can make these guarantees..
It would be useful if "strong ordering" were exposed as attribute from a transport.. and as well - have the ability to provide a hint to enable "strong ordering" on either registration or per operation or at the qp level .
Are there any HCAs that provide "strong ordering" today ?
Quoted 26 lines · Roland Dreier
Quoted 2 lines
Some time ago there was an email sent to this group with the subject "weak ordering for data registered memory". I don't recall any
action
Quoted 2 lines
resulting from this thread. So, I have a question. If a bit were defined to specify "strong ordering", perhaps as a "access" flag
(see
Quoted 1 line
ibv_access_flags) and used withibv_reg_mr(), would that besufficient
Quoted 1 line
for (1) client applications that need a HW "guarantee" of writing
the
Quoted 2 lines
last byte of an RDMA last and (2) platform implementations that need to deliver that feature?
What would happen if an application asked for strong ordering and the adapter and/or platform is not capable of that?
Weak ordering is a bit easier to handle -- the app is saying "if you can make things go faster, don't worry about ordering here" and a platform where it doesn't matter can just ignore it.
- R.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines
Quoted 4 lines
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 16 lines
I decided to minimize the impact of an API change on the class of applications that use the current verbs interface because those applications can safely run on platforms that deliver optimal performance using weak ordering for data buffers. New binaries aren't required for this class of application.
I thought it would be more appropriate to put the burden of added complexity on the class of applications that bypass the verbs to access special features in the hardware. In fact, those applications are selective about memory regions that need this special handling and would register lots of memory without the "strong ordering' bit. How applications determine that the platform is capable of performing the request would be beyond the scope of the verbs, however, I suppose that the verbs framework could check and return an error.
Quoted 7 lines
If there are applications that expect the hardware to support "strong ordering" and don't check the hardware, then these might be a problem. Do any of these exists?
By the way, if I had proposed this bit several years ago, then I would have chosen a "weak ordering" flag. Instead, I decided to try protecting the existing base of verbs-based software.
I can't really follow this. Right now Open MPI et al assume that if they see a Mellanox adapter, they get the "last byte of RDMA becomes visible last" behavior. And there is not a way that I know of to turn this off at all, let alone get any performance difference. The exception being the Cell processor system that started the previous discussion, where weak ordering at the platform level helped things.
But given that current software does seem to rely on ordering, it seems that opting into weak ordering would break fewer applications.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Richard Frank
Would anyone like to through out the list of HCAs that do this... I can guess at a few... and can ask the vendors directly.. if not.. .
It would be much nicer to not hardcode names of adapters.. but that won't stop us.. :)
Isn't it more complex than this? AFAIK the PCI-E standard does not specify the order which data inside a single transfer becomes visible, only how different transfers relate. To work on the most agressive PCI-E system the HCA would have to transfer the last XX bytes as a seperate PCI-E transaction without relaxed ordering.
This is the sort of thing that might start to matter on QPI and HT memory-interleaved configurations. A multi-cache line transfer will be split up and completed on different chips - it may not be fully coherent 100% of the time.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 11 Nov 2009, Jason Gunthorpe wrote:
| On Wed, Nov 11, 2009 at 05:44:59PM -0500, Richard Frank wrote: | | > Would anyone like to through out the list of HCAs that do this... I | > can guess at a few... and can ask the vendors directly.. if not.. . | > | > It would be much nicer to not hardcode names of adapters.. but that won't | > stop us.. :) | | Isn't it more complex than this? AFAIK the PCI-E standard does not | specify the order which data inside a single transfer becomes visible, | only how different transfers relate. To work on the most agressive | PCI-E system the HCA would have to transfer the last XX bytes as a | seperate PCI-E transaction without relaxed ordering.
I can't speak to the specifics of this on PCIe, but yes, by default the pcie transfers within a single tag can be unordered.
| This is the sort of thing that might start to matter on QPI and HT | memory-interleaved configurations. A multi-cache line transfer will be | split up and completed on different chips - it may not be fully | coherent 100% of the time.
HT is fine, by design, at least on AMD processors (probably don't care too much about the older sibyte cpus, since they weren't fully HT-compliant).
I don't know about QPI.
Dave Olson
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Some of the SPARC servers from Sun support an IO memory mapping mechanism for marking pages for weak or strong ordering. It has no dependency on the HCA. Marking memory pages for data buffers as weak ordered delivers significantly better throughput on IB.
I assumed that there are many more applications using the user verbs that don't care about ordering for data buffers, but would need to be modified to set a "weak ordering" flag.
Nonetheless, I'm somewhat flexible and willing to go with bit definition that that verbs consumers prefer.
-David
Quoted 30 lines · Roland Dreier
Quoted 15 lines
I decided to minimize the impact of an API change on the class of applications that use the current verbs interface because those applications can safely run on platforms that deliver optimal performance using weak ordering for data buffers. New binaries aren't required for this class of application.
I thought it would be more appropriate to put the burden of added complexity on the class of applications that bypass the verbs to access special features in the hardware. In fact, those applications are selective about memory regions that need this special handling and would register lots of memory without the "strong ordering' bit. How applications determine that the platform is capable of performing the request would be beyond the scope of the verbs, however, I suppose that the verbs framework could check and return an error.
Quoted 7 lines
If there are applications that expect the hardware to support "strong ordering" and don't check the hardware, then these might be a problem. Do any of these exists?
By the way, if I had proposed this bit several years ago, then I would have chosen a "weak ordering" flag. Instead, I decided to try protecting the existing base of verbs-based software.
I can't really follow this. Right now Open MPI et al assume that if they see a Mellanox adapter, they get the "last byte of RDMA becomes visible last" behavior. And there is not a way that I know of to turn this off at all, let alone get any performance difference. The exception being the Cell processor system that started the previous discussion, where weak ordering at the platform level helped things.
Quoted 9 lines
But given that current software does seem to rely on ordering, it seems that opting into weak ordering would break fewer applications.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
See section 4 in the paper called "High Performance RDMA Based MPI Implementation over InfiniBand" on the MVAPICH web page for description of one implementation that polls on data buffers. Specifically, look at text around the statement "Although the approach uses the in-order implementation of hardware for RDMA write which is not specified in the InfiniBand standard, this feature is very likely to be kept by different hardware designers." Although this paper is describing a PCI-X implementation, the feature is also exists on PCIe.
It's assumed that the host memory interconnect complies with statements described in the "Update Ordering and Granularity Provided by a Write Transaction" of the PCI spec. This particular application depends on PCI WRITE behavior, not READ.
Does this help?
Quoted 14 lines · Jason Gunthorpe
Quoted 5 lines · Richard Frank
Would anyone like to through out the list of HCAs that do this... I can guess at a few... and can ask the vendors directly.. if not.. .
It would be much nicer to not hardcode names of adapters.. but that won't stop us.. :)
Isn't it more complex than this? AFAIK the PCI-E standard does not specify the order which data inside a single transfer becomes visible, only how different transfers relate. To work on the most agressive PCI-E system the HCA would have to transfer the last XX bytes as a seperate PCI-E transaction without relaxed ordering.
Quoted 6 lines
This is the sort of thing that might start to matter on QPI and HT memory-interleaved configurations. A multi-cache line transfer will be split up and completed on different chips - it may not be fully coherent 100% of the time.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html