rdma-core 12 breaks libibverbs ABI
Hi,
the rmda-core commit 683e55aa5d291c8fda338dae565fbdae0c0000b5 "ibverbs:
Allow vendor data response in create_ah command" breaks the ABI of the libibverbs library: The symbol ibv_cmd_create_ah@IBVERBS_1.0 is replaced by ibv_cmd_create_ah@IBVERBS_1.4. Please bring back ibv_cmd_create_ah@IB VERBS_1.0 to restore the ABI. Otherwise the version of the soname needs to be bumped from 1 to 2.
Signature · 18 lines
Benjamin Drung System Developer Debian & Ubuntu Developer ProfitBricks GmbH Greifswalder Str. 207 D - 10405 Berlin Email: [email protected] URL: http://www.profitbricks.com Sitz der Gesellschaft: Berlin. Registergericht: Amtsgericht Charlottenburg, HRB 125506B. Geschäftsführer: Andreas Gauger, Achim Weiss. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · Benjamin Drung
the rmda-core commit
683e55aa5d291c8fda338dae565fbdae0c0000b5"ibverbs:
Allow vendor data response increate_ahcommand" breaks the ABI of the libibverbs library: The symbol ibv_cmd_create_ah@IBVERBS_1.0 is replaced by ibv_cmd_create_ah@IBVERBS_1.4. Please bring back ibv_cmd_create_ah@IB VERBS_1.0 to restore the ABI. Otherwise the version of the soname needs to be bumped from 1 to 2.
The ibv_cmd_* family of symbols are now considered private internal symbols only for use between libibverbs and the included plugin libraries.
The decision was made to stop supporting providers outside of rdma-core, so the library SONAME only reflects the public facing ABI.
I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 patch was merged, but before we release 13 we need to drop all the ibv_cmd_* entry points from the public headers so that nothing outside rdma-core can link to these symbols.
AFAIK this is an accepted way to build a library that works with private loadable plugins.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170126171325.GA16562-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 4 lines · Jason Gunthorpe
I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 patch was merged, but before we release 13 we need to drop all the ibv_cmd_* entry points from the public headers so that nothing outside rdma-core can link to these symbols.
A good thing would be to move all these symbols into a @IBVERBS_PRIVATE namespace, similar to what glibc does for it's internal library only exports.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines · Jason Gunthorpe
I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 patch was merged, but before we release 13 we need to drop all the ibv_cmd_* entry points from the public headers so that nothing outside rdma-core can link to these symbols.
A good thing would be to move all these symbols into a @IBVERBS_PRIVATE namespace, similar to what glibc does for it's internal library only exports.
Makes sense for clarity.
I guess we should remove the public headers and change the symbol version all at once in '13'? This would instantly make all out of tree providers totally unworkable.
Or go back to providing strong compat to the providers as Ben suggested.. In this case someone will have to vote for this with a patch to put the missing symbol back...
I know mlx voted for the first path - Ira?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170126180926.GB20330-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 3 lines · Jason Gunthorpe
Or go back to providing strong compat to the providers as Ben suggested.. In this case someone will have to vote for this with a patch to put the missing symbol back...
There should not be any compat - it's just going to drag down development and everyone is free to submit their provider to rdma-core, same as kernel drivers.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Jason Gunthorpe
Or go back to providing strong compat to the providers as Ben suggested.. In this case someone will have to vote for this with a patch to put the missing symbol back...
There should not be any compat - it's just going to drag down development and everyone is free to submit their provider to rdma-core, same as kernel drivers.
Yep, that was the argument that won the day so far.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170126182753.GA5056-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 6 lines
Quoted 2 lines
There should not be any compat - it's just going to drag down development and everyone is free to submit their provider to rdma-
core,
Quoted 1 line
same as kernel drivers.
Yep, that was the argument that won the day so far.
I don't disagree with the direction, but I agree with Benjamin. The intent behind a public symbol/structure and their actual usage by applications may differ. Why wasn't the soname updated?
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A82373AB0E132B-P5GAC/[email protected]
Quoted 10 lines · Sean
Quoted 6 lines
Quoted 2 lines
There should not be any compat - it's just going to drag down development and everyone is free to submit their provider to rdma-
core,
Quoted 1 line
same as kernel drivers.
Yep, that was the argument that won the day so far.
I don't disagree with the direction, but I agree with Benjamin. The intent behind a public symbol/structure and their actual usage by applications may differ. Why wasn't the soname updated?
Well, no it cannot differ. If apps use private symbols they get burned. Glibc and gcc have established that precedent decades ago.
The soname is not bumped because libibverbs is a symver based library, which means we can manage our private ABI safely without ever requiring soname changes. If something does manage to link to it then it gets a runtime dynamic linking failure, not a random crash.
Remember, libibverbs fundamentally does not support two copies of the library in the same address space, so one of the major motivations to use a soname to manage version is invalid. Even worse, because of this, if we did bump the soname, it would be very painful and dangerous.
So that leaves us with only one point to argue on: Should libibverbs have a private ABI or not? If yes, then we are doing it properly.
The issue is muddied a bit because the private ABI has not been as absolutely private as I'd like, so we could add compat for the two symbols and defer the incompat transition till a later release.. This argument would be best supported if someone knows of something that actually breaks out in the wild...
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Jason Gunthorpe
Or go back to providing strong compat to the providers as Ben suggested.. In this case someone will have to vote for this with a patch to put the missing symbol back...
There should not be any compat - it's just going to drag down development and everyone is free to submit their provider to rdma-core, same as kernel drivers.
Agree, I have no desire to support out-of-tree providers.
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170126202330.GX6005-U/[email protected]
Quoted 6 lines
Quoted 2 lines
There should not be any compat - it's just going to drag down development and everyone is free to submit their provider to rdma-
core,
Quoted 1 line
same as kernel drivers.
Agree, I have no desire to support out-of-tree providers.
What is the plan for handling live updates to a single vendor library?
Is the intent that vendors must now (potentially) provide different versions of their libraries, based on which libibverbs release is deployed by the user? --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A82373AB0E15AD-P5GAC/[email protected]
Quoted 5 lines · Sean
Quoted 5 lines
Quoted 2 lines
There should not be any compat - it's just going to drag down development and everyone is free to submit their provider to rdma-
core,
Quoted 1 line
same as kernel drivers.
Quoted 7 lines
Quoted 1 line
Agree, I have no desire to support out-of-tree providers.
What is the plan for handling live updates to a single vendor library?
Is the intent that vendors must now (potentially) provide different versions of their libraries, based on which libibverbs release is deployed by the user?
Well, this has been the case for awhile, at least for certain providers (eg mlx5 will not link to older libibverbs).
The patch to add IBVERBS_PRIVATE_xx certainly makes it every provider's problem..
As I see it, anyone doing an update driver distribution will have to QA it with a certain libibverbs, so I doubt there is much point in making it too easy for a user to mix and match too widely??
So vendors have a few basic choices:
- Ship libibverbs.rpm with the updated driver and matched libibverbs This is pretty easy, I probably would recommend this.
- QA the driver with the libibverbs in the various targetted distro releases, as you suggested
- Ship their fix to the distro and have the distro push the update
It isn't really any different from the problem a vendor would face getting the kernel driver updated..
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170127010452.GA7826-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 2 lines
- Ship libibverbs.rpm with the updated driver and matched libibverbs This is pretty easy, I probably would recommend this.
This has issues if there are multiple providers present. Maybe that doesn't happen in non-testing environments. I wonder if the distros would agree with this approach.
Quoted 6 lines
- QA the driver with the libibverbs in the various targetted distro releases, as you suggested
- Ship their fix to the distro and have the distro push the updateIt isn't really any different from the problem a vendor would face getting the kernel driver updated..
I agree.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A82373AB0E1609-P5GAC/[email protected]
Quoted 4 lines · Sean
Quoted 2 lines
- Ship libibverbs.rpm with the updated driver and matched libibverbs This is pretty easy, I probably would recommend this.
This has issues if there are multiple providers present. Maybe that doesn't happen in non-testing environments. I wonder if the distros would agree with this approach.
I don't know how much I would rely on the belief that mixed environments don't happen outside of testing. In particular, the structure of Lustre filesystems or the need to gateway from a fabric that one brand of storage server doesn't support to another one would both seem to be times when mixed hardware might exist. For that matter, the typical stock trading RDMA environment mixes IB and Ethernet, maybe in the future it will have OPA in there too? I think you are right that it is the most common, but I don't think we can rely on it being that way all the time.
Signature · 3 lines
Doug Ledford <[email protected]> GPG Key ID: B826A3330E572FDD Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD
A message that is not in the archive (sent off-list or not yet received): 41e74ed7-fd8d-89cf-2c79-b99a8d786fac-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Quoted 10 lines · Doug Ledford
Quoted 6 lines · Sean
Quoted 2 lines
- Ship libibverbs.rpm with the updated driver and matched libibverbs This is pretty easy, I probably would recommend this.
This has issues if there are multiple providers present. Maybe that doesn't happen in non-testing environments. I wonder if the distros would agree with this approach.
I don't know how much I would rely on the belief that mixed environments don't happen outside of testing.
Yes, I've heard many complaints in production in this area.
As it stands today the changes to libibverbs in vendor trees are too extensive, the only way to resolve them is to merge the vendors patches into one source tree and build a custom release. I think rdma-core will make that much simpler.
Making the ABI private or public doesn't really help things either way, as far as I can tell. Remember the stable ABI is upstream, not in a vendor tree, so don't expect a random vendor libibverbs to work with any other providers, even today! They might link, but based on what I've seen they will be subtly broken.
Ideally vendors will get their stuff upstream and a site could choose to just use upstream rdma-core instead of a vendor tree.
I would still suggest vendors shipping libibverbs.rpm as their update is and reasonable and easy approach for their simple users.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Doug Ledford
Quoted 4 lines · Sean
Quoted 2 lines
- Ship libibverbs.rpm with the updated driver and matched libibverbs This is pretty easy, I probably would recommend this.
This has issues if there are multiple providers present. Maybe that doesn't happen in non-testing environments. I wonder if the distros would agree with this approach.
Quoted 7 lines
I don't know how much I would rely on the belief that mixed environments don't happen outside of testing. In particular, the structure of Lustre filesystems or the need to gateway from a fabric that one brand of storage server doesn't support to another one would both seem to be times when mixed hardware might exist. For that matter, the typical stock trading RDMA environment mixes IB and Ethernet, maybe in the future it will have OPA in there too?
This is happening now - not in the future.
LANL and other DOE labs already have OPA and IB/mlnx on the same host.
Quoted 9 lines
I think you are right that it is
the most common, but I don't think we can rely on it being that way all the time.Signature · 3 lines
Doug Ledford <[email protected]> GPG Key ID: B826A3330E572FDD Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD
==================================================
Susan Coulter / HPC-DES
Network Capability Lead
(505) 667-8425
“Once in a while you get shown the light
In the strangest of places if you look at it right” Robert Hunter
==================================================
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): A4803088-726E-4B88-B421-CE1969B6E25B-K3XgeB3KlJxL3NSqSJqx7PIbXMQ5te18@public.gmane.org
Quoted 2 lines · Susan K
This is happening now - not in the future.
LANL and other DOE labs already have OPA and IB/mlnx on the same host.
And have setups with up to three different providers :)
But the whole point is that the new rdma-core setup gives us a sane way to handle this: patch your rdma-core tree with whatever vendor updates you need, same as the kernel. The current model where vendors supply a new libibverbs that has all kinds of incompatible hacks is simply not sustainable.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines
It isn't really any different from the problem a vendor would face getting the kernel driver updated..
My concern is that OFA (and the vendors) have not exactly had a good track record of kernel updates in the past. So I'm not sure that this statement give me a lot of comfort.
That said, I know that most vendors seem to be making a good effort at integrating upstream. And there is still a mechanism which allows for an external provider.
So I'm not opposed to this.
Ira
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Am Donnerstag, den 26.01.2017, 09:35 -0800 schrieb Christoph Hellwig:
Quoted 9 lines · Jason Gunthorpe
Quoted 7 lines
I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 patch
was merged, but before we release 13 we need to drop all the ibv_cmd_*
entry points from the public headers so that nothing outside rdma- core
can link to these symbols.
Quoted 4 lines
A good thing would be to move all these symbols into a @IBVERBS_PRIVATE
namespace, similar to what glibc does for it's internal library only exports.
I am fine with moving the private symbols into a IBVERBS_PRIVATE_XX namespace as long as all consumers of the private symbols are build from the rdma-core source package. I need to check if there is a package in Debian that uses the private symbols.
So either go forward (extra namespace for private symbols) or revert the ABI breakage (for the time being). The discussion on this list seem to prefer the first option.
Signature · 18 lines
Benjamin Drung System Developer Debian & Ubuntu Developer ProfitBricks GmbH Greifswalder Str. 207 D - 10405 Berlin Email: [email protected] URL: http://www.profitbricks.com Sitz der Gesellschaft: Berlin. Registergericht: Amtsgericht Charlottenburg, HRB 125506B. Geschäftsführer: Andreas Gauger, Achim Weiss. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Am Dienstag, den 31.01.2017, 13:23 +0100 schrieb Benjamin Drung:
Quoted 17 lines
Am Donnerstag, den 26.01.2017, 09:35 -0800 schrieb Christoph Hellwig:
Quoted 10 lines · Jason Gunthorpe
Quoted 8 lines
I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 patch
was merged, but before we release 13 we need to drop all the ibv_cmd_*
entry points from the public headers so that nothing outside rdma-
core
can link to these symbols.Quoted 5 lines
A good thing would be to move all these symbols into a @IBVERBS_PRIVATE
namespace, similar to what glibc does for it's internal library only
exports.
Quoted 4 lines
I am fine with moving the private symbols into a
IBVERBS_PRIVATE_XXnamespace as long as all consumers of the private symbols are build from the rdma-core source package. I need to check if there is a package in Debian that uses the private symbols.
I brought this question to the debian-devel mailing list:
https://lists.debian.org/debian-devel/2017/01/msg00791.html
One private feedback was that removing symbols without bumping the soname is a really bad idea (even if they are private symbols). So one option would be to bump the soname once and remove all private symbols in that run.
I searched for "ibv_cmd_" in Debian's source code:
https://codesearch.debian.net/search?q=ibv_cmd_
It boils down to these source packages:
shipped by rdma-core
====================
* librdmacm
* libibverbs
provided by ibverbs-providers
=============================
* libcxgb3
* libmlx5
* libmthca
* libnes
* libipathverbs
* libmlx4
remaining
=========
* openmpi
openmpi uses ibv_cmd_open_xrcd in config/opal_check_openfabrics.m4:
# We have to check functions both exits *and* are declared
# since some distro ship broken ibverbs devel headers # IBV_DEVICE_XRC is common to all OFED versions # ibv_create_xrc_rcv_qp was added in OFED 1.3 # ibv_cmd_open_xrcd (aka XRC Domains) was added in OFED 3.12
So I recommend to contact openmpi to let them change their configure check to not check for ibv_cmd_open_xrcd any more.
Signature · 18 lines
Benjamin Drung System Developer Debian & Ubuntu Developer ProfitBricks GmbH Greifswalder Str. 207 D - 10405 Berlin Email: [email protected] URL: http://www.profitbricks.com Sitz der Gesellschaft: Berlin. Registergericht: Amtsgericht Charlottenburg, HRB 125506B. Geschäftsführer: Andreas Gauger, Achim Weiss. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · Benjamin Drung
One private feedback was that removing symbols without bumping the soname is a really bad idea (even if they are private symbols). So one option would be to bump the soname once and remove all private symbols in that run.
Bumping the soname is also a really bad idea - we need to pick which one is less bad :)
Quoted 3 lines
I searched for "
ibv_cmd_" in Debian's source code:
https://codesearch.debian.net/search?q=ibv_cmd_
It boils down to these source packages:
Neat. I also looked at the removed headerfiles which will cause FTBS.
dmtcp seems to include an old wrong copy of libibverbs/examples/devinfo.c - I have no idea why.
usr/include/infiniband/arch.h
dpdk
dmtcp
fio
qlvnictools
These seem to be using htonll/etc (replace with glibc's htobe64)
usr/include/infiniband/driver.h
openmpi
dmtcp
libfabric
Most of these are probably unnecessary..
infiniband/marshall.h has none
Quoted 10 lines
openmpi uses
ibv_cmd_open_xrcdin config/opal_check_openfabrics.m4:# We have to check functions both exits *and* are declared
# since some distro ship broken ibverbs devel headers # IBV_DEVICE_XRC is common to all OFED versions # ibv_create_xrc_rcv_qp was added in OFED 1.3 # ibv_cmd_open_xrcd (aka XRC Domains) was added in OFED 3.12So I recommend to contact openmpi to let them change their configure check to not check for
ibv_cmd_open_xrcdany more.
Yes, they will need to make a few small changes
This is a very small amount of usage and other than the htonll stuff, none of it looks at all right to me.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 15 lines · Benjamin Drung
Am Donnerstag, den 26.01.2017, 09:35 -0800 schrieb Christoph Hellwig:
Quoted 9 lines · Jason Gunthorpe
Quoted 7 lines
I missed asking for this when the ibv_cmd_create_ah@IBVERBS_1.4 patch
was merged, but before we release 13 we need to drop all the ibv_cmd_*
entry points from the public headers so that nothing outside rdma- core
can link to these symbols.Quoted 4 lines
A good thing would be to move all these symbols into a @IBVERBS_PRIVATE
namespace, similar to what glibc does for it's internal library only exports.
Quoted 8 lines
I am fine with moving the private symbols into a
IBVERBS_PRIVATE_XXnamespace as long as all consumers of the private symbols are build from the rdma-core source package. I need to check if there is a package in Debian that uses the private symbols.So either go forward (extra namespace for private symbols) or revert the ABI breakage (for the time being). The discussion on this list seem to prefer the first option.
Yes.
We must implement one of the two options before releasing '13'
I have sent patches on github to do the first..
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html