Skip to contentSkip to conversation

What's the current status with bnxt_re-abi.h

DISCUSSION Pierre-Elliott Bécue 2 participants 6 messages Jun 12, 2024 – Jun 13, 2024 InfiniBand

lore
Share & tools
lore
#0 lore raw

Hello,

In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in all libibverbs to determine the min AND max supported ABI.

bnxt_re isn't currently mainlined in the kernel, and those eager to use the driver need to rely on the one provided by broadcom on their website.

The thing is, they bumped their ABI version multiple times (current is 6). In the current context, one can't use the manually compiled bnxt_re driver with libibverbs as any call will error due to the bnxt_re abi version being outside of min/max supported abi version.

What's the current situation regarding bnxt_re, should we consider libibverb support of bnxt_re as deprecated?

Of course I could have missed something, sorry for that if that's the case.

Bests,

Signature · 1 line
PEB
#1 lore raw

Pierre-Elliott Bécue <[email protected]> wrote on 12/06/2024 at 18:47:36+0200:

Quoted 6 lines

Hello,

In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in all libibverbs to determine the min AND max supported ABI.

bnxt_re isn't currently mainlined in the kernel,

Sorry, a word is missing: "Recent bnxt_re isn't currently mainlined"

Quoted 16 lines

and those eager to use
the driver need to rely on the one provided by broadcom on their website.

The thing is, they bumped their ABI version multiple times (current is 6). In the current context, one can't use the manually compiled bnxt_re driver with libibverbs as any call will error due to the bnxt_re abi version being outside of min/max supported abi version.

What's the current situation regarding bnxt_re, should we consider libibverb support of bnxt_re as deprecated?

Of course I could have missed something, sorry for that if that's the case.

Bests,

#2 lore raw

Hi,
bnxt_re-abi.h in linux kernel and rdma-core uses abi version 1. We
dont bump up the version in upstream and backward compatibility is maintained using the comp_mask field of the interface structures.

If you are using the latest drivers maintained in the Broadcom site (which uses ABI version 6), you need to use the libbnxt_re hosted in the Broacom site itself. We maintain compatibility between the Out of tree driver and Out of tree library.

Thanks,
Selvin

Quoted 28 lines · Pierre-Elliott Bécue

Pierre-Elliott Bécue <[email protected]> wrote on 12/06/2024 at 18:47:36+0200:

Quoted 6 lines

Hello,

In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in all libibverbs to determine the min AND max supported ABI.

bnxt_re isn't currently mainlined in the kernel,

Sorry, a word is missing: "Recent bnxt_re isn't currently mainlined"

Quoted 16 lines

and those eager to use
the driver need to rely on the one provided by broadcom on their website.

The thing is, they bumped their ABI version multiple times (current is 6). In the current context, one can't use the manually compiled bnxt_re driver with libibverbs as any call will error due to the bnxt_re abi version being outside of min/max supported abi version.

What's the current situation regarding bnxt_re, should we consider libibverb support of bnxt_re as deprecated?

Of course I could have missed something, sorry for that if that's the case.

Bests,

#3 lore raw

De : Selvin Xavier <[email protected]> À : Pierre-Elliott Bécue <[email protected]>
Cc : [email protected]
Date : 12 juin 2024 19:18:34
Objet : Re: What's the current status with bnxt_re-abi.h

Quoted 10 lines

Hi,
bnxt_re-abi.h in linux kernel and rdma-core uses abi version 1. We dont bump up the version in upstream and backward compatibility is maintained using the comp_mask field of the interface structures.

If you are using the latest drivers maintained in the Broadcom site (which uses ABI version 6), you need to use the libbnxt_re hosted in the Broacom site itself. We maintain compatibility between the Out of tree driver and Out of tree library.

Quoted 32 lines

Thanks,
Selvin

Quoted 28 lines · Pierre-Elliott Bécue

Pierre-Elliott Bécue <[email protected]> wrote on 12/06/2024 at 18:47:36+0200:

Quoted 6 lines

Hello,

In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in all libibverbs to determine the min AND max supported ABI.

bnxt_re isn't currently mainlined in the kernel,

Sorry, a word is missing: "Recent bnxt_re isn't currently mainlined"

Quoted 16 lines

and those eager to use
the driver need to rely on the one provided by broadcom on their website.

The thing is, they bumped their ABI version multiple times (current is 6). In the current context, one can't use the manually compiled bnxt_re driver with libibverbs as any call will error due to the bnxt_re abi version being outside of min/max supported abi version.

What's the current situation regarding bnxt_re, should we consider libibverb support of bnxt_re as deprecated?

Of course I could have missed something, sorry for that if that's the case.

Bests,

Hey Selvin,

Thanks a lot for clarifying this.

I built libbnxt_re, but the thing is, I can't use the usual infiniband tooling (ib_write_bw) et al without using libibverbs which is still the linux-rdma one.

Do you have an alternative to suggest that I should consider?

Also, would a RDMA+SRIOV-capable broadcom card with the latest firmware (229 something) work fine with the drivers shipped in rdma-core.

If so maybe I should use what's provided by the kernel and rdma-core.

Bests,

Signature · 1 line
Pierre-Elliott Bécue
#4 lore raw
Quoted 52 lines · Pierre-Elliott Bécue

De : Selvin Xavier <[email protected]> À : Pierre-Elliott Bécue <[email protected]>
Cc : [email protected]
Date : 12 juin 2024 19:18:34
Objet : Re: What's the current status with bnxt_re-abi.h

Quoted 14 lines

Hi,
bnxt_re-abi.h in linux kernel and rdma-core uses abi version 1. We dont bump up the version in upstream and backward compatibility is maintained using the comp_mask field of the interface structures.

If you are using the latest drivers maintained in the Broadcom site (which uses ABI version 6), you need to use the libbnxt_re hosted in the Broacom site itself. We maintain compatibility between the Out of tree driver and Out of tree library.

Thanks,
Selvin

On Wed, Jun 12, 2024 at 10:21 PM Pierre-Elliott Bécue <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines

Pierre-Elliott Bécue <[email protected]> wrote on 12/06/2024 at

18:47:36+0200:

Quoted 19 lines
Quoted 19 lines
Quoted 7 lines

Hello,

In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in all libibverbs to determine the min AND max supported ABI.

bnxt_re isn't currently mainlined in the kernel,

Sorry, a word is missing: "Recent bnxt_re isn't currently mainlined"

Quoted 9 lines

and those eager to use
the driver need to rely on the one provided by broadcom on their website.

The thing is, they bumped their ABI version multiple times (current is 6). In the current context, one can't use the manually compiled bnxt_re driver with libibverbs as any call will error due to the bnxt_re abi version being outside of min/max supported abi version.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

What's the current situation regarding bnxt_re, should we consider libibverb support of bnxt_re as deprecated?

Of course I could have missed something, sorry for that if that's the case.

Bests,

Hey Selvin,

Quoted 8 lines

Thanks a lot for clarifying this.

I built libbnxt_re, but the thing is, I can't use the usual infiniband tooling (ib_write_bw) et al without using libibverbs which is still the linux-rdma one.

Do you have an alternative to suggest that I should consider?

It should work fine with the libibverbs installed by linux-rdma-core. The Broadcom support team can help you with the necessary instructions. I will forward your mail to them and they will help.

Quoted 6 lines

Also, would a RDMA+SRIOV-capable broadcom card with the latest firmware (229 something) work fine with the drivers shipped in rdma-core.

If so maybe I should use what's provided by the kernel and rdma-core.

The drivers shipped with rdma-core and kernel should work for most of your use cases. I suggest you try this.

Quoted 6 lines

Bests,

--
Pierre-Elliott Bécue

#5 lore raw

Selvin Xavier <[email protected]> wrote on 13/06/2024 at 10:49:00+0200:

Quoted 54 lines · Pierre-Elliott Bécue

De : Selvin Xavier <[email protected]> À : Pierre-Elliott Bécue <[email protected]>
Cc : [email protected]
Date : 12 juin 2024 19:18:34
Objet : Re: What's the current status with bnxt_re-abi.h

Quoted 10 lines

Hi,
bnxt_re-abi.h in linux kernel and rdma-core uses abi version 1. We dont bump up the version in upstream and backward compatibility is maintained using the comp_mask field of the interface structures.

If you are using the latest drivers maintained in the Broadcom site (which uses ABI version 6), you need to use the libbnxt_re hosted in the Broacom site itself. We maintain compatibility between the Out of tree driver and Out of tree library.

Quoted 32 lines

Thanks,
Selvin

Quoted 28 lines · Pierre-Elliott Bécue

Pierre-Elliott Bécue <[email protected]> wrote on 12/06/2024 at 18:47:36+0200:

Quoted 6 lines

Hello,

In bnxt_re-abi.h, the abi version mentioned is 1. It's used as it's in all libibverbs to determine the min AND max supported ABI.

bnxt_re isn't currently mainlined in the kernel,

Sorry, a word is missing: "Recent bnxt_re isn't currently mainlined"

Quoted 16 lines

and those eager to use
the driver need to rely on the one provided by broadcom on their website.

The thing is, they bumped their ABI version multiple times (current is 6). In the current context, one can't use the manually compiled bnxt_re driver with libibverbs as any call will error due to the bnxt_re abi version being outside of min/max supported abi version.

What's the current situation regarding bnxt_re, should we consider libibverb support of bnxt_re as deprecated?

Of course I could have missed something, sorry for that if that's the case.

Bests,

Hey Selvin,

Thanks a lot for clarifying this.

Quoted 7 lines

I built libbnxt_re, but the thing is, I can't use the usual infiniband tooling (ib_write_bw) et al without using libibverbs which is still the linux-rdma one.

Do you have an alternative to suggest that I should consider?

It should work fine with the libibverbs installed by linux-rdma-core.

Due to the ABI version being 1 in rdma-core, no. libibverbs verifies that the current driver ABI version (6 for bnxt_re from broadcom's website) is between ops->match_min_abi_version and ops->match_max_abi_version, both values for bnxt_re are set to the ABI version (1).

https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/init.c#L363-L372

So when trying to use tooling from rdma-core (like ib_write_bw), the match_device function returns false and the card can't be used.

Quoted 3 lines

The Broadcom support team can help you with the
necessary instructions. I will forward your mail to them and they will help.

I already interacted with the support team, and I'm not confident they can help. But thanks. :)

Quoted 7 lines

Also, would a RDMA+SRIOV-capable broadcom card with the latest firmware (229 something) work fine with the drivers shipped in rdma-core.

If so maybe I should use what's provided by the kernel and rdma-core.

The drivers shipped with rdma-core and kernel should work for most of your use cases. I suggest you try this.

Ack, I'll try and compare perfs. Thanks!

Signature · 1 line
PEB