What's the current status with bnxt_re-abi.h
Hello,
In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in all libibverbs to determine the min AND max supported ABI.
bnxt_re isn't currently mainlined in the kernel, and those eager to use the driver need to rely on the one provided by broadcom on their website.
The thing is, they bumped their ABI version multiple times (current is 6). In the current context, one can't use the manually compiled bnxt_re driver with libibverbs as any call will error due to the bnxt_re abi version being outside of min/max supported abi version.
What's the current situation regarding bnxt_re, should we consider libibverb support of bnxt_re as deprecated?
Of course I could have missed something, sorry for that if that's the case.
Bests,
Signature · 1 line
PEB
Pierre-Elliott Bécue <[email protected]> wrote on 12/06/2024 at 18:47:36+0200:
Quoted 6 lines
Hello,
In
bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in all libibverbs to determine the min AND max supported ABI.
bnxt_reisn't currently mainlined in the kernel,
Sorry, a word is missing: "Recent bnxt_re isn't currently mainlined"
Quoted 16 lines
and those eager to use
the driver need to rely on the one provided by broadcom on their website.The thing is, they bumped their ABI version multiple times (current is 6). In the current context, one can't use the manually compiled
bnxt_redriver with libibverbs as any call will error due to thebnxt_reabi version being outside of min/max supported abi version.What's the current situation regarding
bnxt_re, should we consider libibverb support ofbnxt_reas deprecated?Of course I could have missed something, sorry for that if that's the case.
Bests,
Hi,
bnxt_re-abi.h in linux kernel and rdma-core uses abi version 1. We
dont bump up the version in upstream and backward compatibility is maintained using the comp_mask field of the interface structures.
If you are using the latest drivers maintained in the Broadcom site (which uses ABI version 6), you need to use the libbnxt_re hosted in the Broacom site itself. We maintain compatibility between the Out of tree driver and Out of tree library.
Thanks,
Selvin
Quoted 28 lines · Pierre-Elliott Bécue
Pierre-Elliott Bécue <[email protected]> wrote on 12/06/2024 at 18:47:36+0200:
Quoted 6 lines
Hello,
In
bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in all libibverbs to determine the min AND max supported ABI.
bnxt_reisn't currently mainlined in the kernel,Sorry, a word is missing: "Recent
bnxt_reisn't currently mainlined"Quoted 16 lines
and those eager to use
the driver need to rely on the one provided by broadcom on their website.The thing is, they bumped their ABI version multiple times (current is 6). In the current context, one can't use the manually compiled
bnxt_redriver with libibverbs as any call will error due to thebnxt_reabi version being outside of min/max supported abi version.What's the current situation regarding
bnxt_re, should we consider libibverb support ofbnxt_reas deprecated?Of course I could have missed something, sorry for that if that's the case.
Bests,
De : Selvin Xavier <[email protected]> À : Pierre-Elliott Bécue <[email protected]>
Cc : [email protected]
Date : 12 juin 2024 19:18:34
Objet : Re: What's the current status with bnxt_re-abi.h
Quoted 10 lines
Hi,
bnxt_re-abi.hin linux kernel and rdma-core uses abi version 1. We dont bump up the version in upstream and backward compatibility is maintained using thecomp_maskfield of the interface structures.If you are using the latest drivers maintained in the Broadcom site (which uses ABI version 6), you need to use the
libbnxt_rehosted in the Broacom site itself. We maintain compatibility between the Out of tree driver and Out of tree library.
Quoted 32 lines
Thanks,
SelvinQuoted 28 lines · Pierre-Elliott Bécue
Pierre-Elliott Bécue <[email protected]> wrote on 12/06/2024 at 18:47:36+0200:
Quoted 6 lines
Hello,
In
bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in all libibverbs to determine the min AND max supported ABI.
bnxt_reisn't currently mainlined in the kernel,Sorry, a word is missing: "Recent
bnxt_reisn't currently mainlined"Quoted 16 lines
and those eager to use
the driver need to rely on the one provided by broadcom on their website.The thing is, they bumped their ABI version multiple times (current is 6). In the current context, one can't use the manually compiled
bnxt_redriver with libibverbs as any call will error due to thebnxt_reabi version being outside of min/max supported abi version.What's the current situation regarding
bnxt_re, should we consider libibverb support ofbnxt_reas deprecated?Of course I could have missed something, sorry for that if that's the case.
Bests,
Hey Selvin,
Thanks a lot for clarifying this.
I built libbnxt_re, but the thing is, I can't use the usual infiniband tooling (ib_write_bw) et al without using libibverbs which is still the linux-rdma one.
Do you have an alternative to suggest that I should consider?
Also, would a RDMA+SRIOV-capable broadcom card with the latest firmware (229 something) work fine with the drivers shipped in rdma-core.
If so maybe I should use what's provided by the kernel and rdma-core.
Bests,
Signature · 1 line
Pierre-Elliott Bécue
Quoted 52 lines · Pierre-Elliott Bécue
De : Selvin Xavier <[email protected]> À : Pierre-Elliott Bécue <[email protected]>
Cc : [email protected]
Date : 12 juin 2024 19:18:34
Objet : Re: What's the current status withbnxt_re-abi.hQuoted 14 lines
Hi,
bnxt_re-abi.hin linux kernel and rdma-core uses abi version 1. We dont bump up the version in upstream and backward compatibility is maintained using thecomp_maskfield of the interface structures.If you are using the latest drivers maintained in the Broadcom site (which uses ABI version 6), you need to use the
libbnxt_rehosted in the Broacom site itself. We maintain compatibility between the Out of tree driver and Out of tree library.Thanks,
SelvinOn Wed, Jun 12, 2024 at 10:21 PM Pierre-Elliott Bécue <[email protected]>
Quoted 2 lines · wrote:
Quoted 2 lines
Pierre-Elliott Bécue <[email protected]> wrote on 12/06/2024 at
18:47:36+0200:
Quoted 19 lines
Quoted 19 lines
Quoted 7 lines
Hello,
In
bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in all libibverbs to determine the min AND max supported ABI.
bnxt_reisn't currently mainlined in the kernel,Sorry, a word is missing: "Recent
bnxt_reisn't currently mainlined"Quoted 9 lines
and those eager to use
the driver need to rely on the one provided by broadcom on their website.The thing is, they bumped their ABI version multiple times (current is 6). In the current context, one can't use the manually compiled
bnxt_redriver with libibverbs as any call will error due to thebnxt_reabi version being outside of min/max supported abi version.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
What's the current situation regarding
bnxt_re, should we consider libibverb support ofbnxt_reas deprecated?Of course I could have missed something, sorry for that if that's the case.
Bests,
Hey Selvin,
Quoted 8 lines
Thanks a lot for clarifying this.
I built
libbnxt_re, but the thing is, I can't use the usual infiniband tooling (ib_write_bw) et al without using libibverbs which is still the linux-rdma one.Do you have an alternative to suggest that I should consider?
It should work fine with the libibverbs installed by linux-rdma-core. The Broadcom support team can help you with the necessary instructions. I will forward your mail to them and they will help.
Quoted 6 lines
Also, would a RDMA+SRIOV-capable broadcom card with the latest firmware (229 something) work fine with the drivers shipped in rdma-core.
If so maybe I should use what's provided by the kernel and rdma-core.
The drivers shipped with rdma-core and kernel should work for most of your use cases. I suggest you try this.
Quoted 6 lines
Bests,
--
Pierre-Elliott Bécue
Selvin Xavier <[email protected]> wrote on 13/06/2024 at 10:49:00+0200:
Quoted 54 lines · Pierre-Elliott Bécue
De : Selvin Xavier <[email protected]> À : Pierre-Elliott Bécue <[email protected]>
Cc : [email protected]
Date : 12 juin 2024 19:18:34
Objet : Re: What's the current status withbnxt_re-abi.hQuoted 10 lines
Hi,
bnxt_re-abi.hin linux kernel and rdma-core uses abi version 1. We dont bump up the version in upstream and backward compatibility is maintained using thecomp_maskfield of the interface structures.If you are using the latest drivers maintained in the Broadcom site (which uses ABI version 6), you need to use the
libbnxt_rehosted in the Broacom site itself. We maintain compatibility between the Out of tree driver and Out of tree library.Quoted 32 lines
Thanks,
SelvinQuoted 28 lines · Pierre-Elliott Bécue
Pierre-Elliott Bécue <[email protected]> wrote on 12/06/2024 at 18:47:36+0200:
Quoted 6 lines
Hello,
In
bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in all libibverbs to determine the min AND max supported ABI.
bnxt_reisn't currently mainlined in the kernel,Sorry, a word is missing: "Recent
bnxt_reisn't currently mainlined"Quoted 16 lines
and those eager to use
the driver need to rely on the one provided by broadcom on their website.The thing is, they bumped their ABI version multiple times (current is 6). In the current context, one can't use the manually compiled
bnxt_redriver with libibverbs as any call will error due to thebnxt_reabi version being outside of min/max supported abi version.What's the current situation regarding
bnxt_re, should we consider libibverb support ofbnxt_reas deprecated?Of course I could have missed something, sorry for that if that's the case.
Bests,
Hey Selvin,
Thanks a lot for clarifying this.
Quoted 7 lines
I built
libbnxt_re, but the thing is, I can't use the usual infiniband tooling (ib_write_bw) et al without using libibverbs which is still the linux-rdma one.Do you have an alternative to suggest that I should consider?
It should work fine with the libibverbs installed by linux-rdma-core.
Due to the ABI version being 1 in rdma-core, no. libibverbs verifies that the current driver ABI version (6 for bnxt_re from broadcom's website) is between ops->match_min_abi_version and ops->match_max_abi_version, both values for bnxt_re are set to the ABI version (1).
https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/init.c#L363-L372
So when trying to use tooling from rdma-core (like ib_write_bw), the match_device function returns false and the card can't be used.
Quoted 3 lines
The Broadcom support team can help you with the
necessary instructions. I will forward your mail to them and they will help.
I already interacted with the support team, and I'm not confident they can help. But thanks. :)
Quoted 7 lines
Also, would a RDMA+SRIOV-capable broadcom card with the latest firmware (229 something) work fine with the drivers shipped in rdma-core.
If so maybe I should use what's provided by the kernel and rdma-core.
The drivers shipped with rdma-core and kernel should work for most of your use cases. I suggest you try this.
Ack, I'll try and compare perfs. Thanks!
Signature · 1 line
PEB