Skip to contentSkip to conversation

question about in-tree vs out-of-tree Broadcom ROCE drivers.

DISCUSSION Chris Friesen 3 participants 5 messages Feb 29, 2024 – Mar 1, 2024 InfiniBand

lore
Share & tools
lore
#0 lore raw

Hi,

I got your address from the Linux kernel MAINTAINERS file, I was wondering if you could clear something up?

As far as I can tell, the in-tree driver at drivers/infiniband/hw/bnxt_re uses a BNXT_RE_ABI_VERSION value of 1, as defined in include/uapi/rdma/bnxt_re-abi.h.

On the other hand, the libbnxt_re-228.0.133.0 package and the bnxt_re-228.0.133.0 driver embedded within https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0 are using a BNXT_RE_ABI_VERSION of 6.

This seems to indicate that the in-tree kernel driver cannot be used with the official version of libbnxt_re as distributed by Broadcom.   Is this correct?   If so, is there a separate version of libbnxt_re intended to be used with the in-kernel driver?

Thanks,

Chris Friesen

#1 lore raw

Hi, Chris

Added the comments inline.

Thanks,
Jiping

-----Original Message-----
From: Friesen, Chris <[email protected]>
Sent: Friday, March 1, 2024 4:31 AM
To: [email protected]; [email protected]
Cc: Asselstine, Mark <[email protected]>; Ma, Jiping <[email protected]>
Subject: question about in-tree vs out-of-tree Broadcom ROCE drivers.

Hi,

I got your address from the Linux kernel MAINTAINERS file, I was wondering if you could clear something up?

As far as I can tell, the in-tree driver at drivers/infiniband/hw/bnxt_re uses a BNXT_RE_ABI_VERSION value of 1, as defined in include/uapi/rdma/bnxt_re-abi.h.

On the other hand, the libbnxt_re-228.0.133.0 package and the
bnxt_re-228.0.133.0 driver embedded within
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0 are using a BNXT_RE_ABI_VERSION of 6.

[Jiping] The abi_version of OOT bnxt_re driver is 6. BNXT_RE_ABI_VERSION of 6 of libbnxt_re is to compatible with OOT driver. Perhaps libbnxt_re does not consider the in-tree driver.

This seems to indicate that the in-tree kernel driver cannot be used with the official version of libbnxt_re as distributed by Broadcom.   Is this correct?   If so, is there a separate version of libbnxt_re intended to be used with the in-kernel driver?

[Jiping] Yes, it looks like. But I did RDMA test, the test passed. So I suggest we can do more full tests for this part, such as wrcp regression test etc. In addition, I did not do more search if there is other libbnxt_re can be used for in-tree driver. Could we also confirm with Broadcom for this question?

Thanks,

Chris Friesen

#2 lore raw

Hi Chris/Jiping,
please see my comments below.

Thanks,
Selvin Xavier

Quoted 25 lines · Jiping

Hi, Chris

Added the comments inline.

Thanks,
Jiping

-----Original Message-----
From: Friesen, Chris <[email protected]>
Sent: Friday, March 1, 2024 4:31 AM
To: [email protected]; [email protected]
Cc: Asselstine, Mark <[email protected]>; Ma, Jiping <[email protected]>
Subject: question about in-tree vs out-of-tree Broadcom ROCE drivers.

Hi,

I got your address from the Linux kernel MAINTAINERS file, I was wondering if you could clear something up?

As far as I can tell, the in-tree driver at drivers/infiniband/hw/bnxt_re uses a BNXT_RE_ABI_VERSION value of 1, as defined in include/uapi/rdma/bnxt_re-abi.h.

On the other hand, the libbnxt_re-228.0.133.0 package and the
bnxt_re-228.0.133.0 driver embedded within
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0 are using a BNXT_RE_ABI_VERSION of 6.

Quoted 5 lines

[Jiping] The abi_version of OOT bnxt_re driver is 6. BNXT_RE_ABI_VERSION of 6 of libbnxt_re is to compatible with OOT driver. Perhaps libbnxt_re does not consider the in-tree driver.

This seems to indicate that the in-tree kernel driver cannot be used with the official version of libbnxt_re as distributed by Broadcom. Is this correct? If so, is there a separate version of libbnxt_re intended to be used with the in-kernel driver?

[Jiping] Yes, it looks like. But I did RDMA test, the test passed. So I suggest we can do more full tests for this part, such as wrcp regression test etc. In addition, I did not do more search if there is other libbnxt_re can be used for in-tree driver. Could we also confirm with Broadcom for this question?

If you are using an in-tree kernel driver, then you can use the libbnxt_re coming with the rdma-core/libibverbs packages in the OS distro. libbnxt_re is part of rdma-core
https://github.com/linux-rdma/rdma-core/tree/master/providers/bnxt_re.
So if you are using the in tree bnxt_re driver, no need to install any libbnxt_re separately. Please install the rdma-core/libibverbs package.

If you are using the OOT driver, please use the OOT libbnxt_re and bnxt_re/bnxt_en driver from the Broadcom download section.
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0

Quoted 5 lines

Thanks,

Chris Friesen

#3 lore raw

Hi, Xavier

Could you help to confirm if the in-tree driver bnxt_re/bnxt_en in kernel 6.6.7 can replace OOT drivers? We need support two kinds of drivers OOT and in-tree at the same time if in-tree driver can not replace OOT drivers. We need two libbnxt_re and rdma-core packages in the system to support in-tree drivers and OOT drivers.

Thanks,
Jiping

-----Original Message-----
From: Selvin Xavier <[email protected]>
Sent: Friday, March 1, 2024 2:07 PM
To: Ma, Jiping <[email protected]>
Cc: Friesen, Chris <[email protected]>; [email protected]; Asselstine, Mark <[email protected]>; Tao, Yue <[email protected]>; Wang, Linda <[email protected]>; Bicakci, Vefa <[email protected]>
Subject: Re: question about in-tree vs out-of-tree Broadcom ROCE drivers.

Hi Chris/Jiping,
please see my comments below.

Thanks,
Selvin Xavier

Quoted 25 lines · Jiping

Hi, Chris

Added the comments inline.

Thanks,
Jiping

-----Original Message-----
From: Friesen, Chris <[email protected]>
Sent: Friday, March 1, 2024 4:31 AM
To: [email protected]; [email protected]
Cc: Asselstine, Mark <[email protected]>; Ma, Jiping <[email protected]>
Subject: question about in-tree vs out-of-tree Broadcom ROCE drivers.

Hi,

I got your address from the Linux kernel MAINTAINERS file, I was wondering if you could clear something up?

As far as I can tell, the in-tree driver at drivers/infiniband/hw/bnxt_re uses a BNXT_RE_ABI_VERSION value of 1, as defined in include/uapi/rdma/bnxt_re-abi.h.

On the other hand, the libbnxt_re-228.0.133.0 package and the
bnxt_re-228.0.133.0 driver embedded within
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0 are using a BNXT_RE_ABI_VERSION of 6.

Quoted 5 lines

[Jiping] The abi_version of OOT bnxt_re driver is 6. BNXT_RE_ABI_VERSION of 6 of libbnxt_re is to compatible with OOT driver. Perhaps libbnxt_re does not consider the in-tree driver.

This seems to indicate that the in-tree kernel driver cannot be used with the official version of libbnxt_re as distributed by Broadcom. Is this correct? If so, is there a separate version of libbnxt_re intended to be used with the in-kernel driver?

[Jiping] Yes, it looks like. But I did RDMA test, the test passed. So I suggest we can do more full tests for this part, such as wrcp regression test etc. In addition, I did not do more search if there is other libbnxt_re can be used for in-tree driver. Could we also confirm with Broadcom for this question?

If you are using an in-tree kernel driver, then you can use the libbnxt_re coming with the rdma-core/libibverbs packages in the OS distro. libbnxt_re is part of rdma-core
https://github.com/linux-rdma/rdma-core/tree/master/providers/bnxt_re.
So if you are using the in tree bnxt_re driver, no need to install any libbnxt_re separately. Please install the rdma-core/libibverbs package.

If you are using the OOT driver, please use the OOT libbnxt_re and bnxt_re/bnxt_en driver from the Broadcom download section.
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0

Quoted 5 lines

Thanks,

Chris Friesen

#4 lore raw

can you confirm the adapter version that you use?
OOT and In-tree drivers should support all released adapters. You dont have to (or can not) install both drivers in the same system.

Quoted 71 lines · Jiping

Hi, Xavier

Could you help to confirm if the in-tree driver bnxt_re/bnxt_en in kernel 6.6.7 can replace OOT drivers? We need support two kinds of drivers OOT and in-tree at the same time if in-tree driver can not replace OOT drivers. We need two libbnxt_re and rdma-core packages in the system to support in-tree drivers and OOT drivers.

Thanks,
Jiping

-----Original Message-----
From: Selvin Xavier <[email protected]>
Sent: Friday, March 1, 2024 2:07 PM
To: Ma, Jiping <[email protected]>
Cc: Friesen, Chris <[email protected]>; [email protected]; Asselstine, Mark <[email protected]>; Tao, Yue <[email protected]>; Wang, Linda <[email protected]>; Bicakci, Vefa <[email protected]>
Subject: Re: question about in-tree vs out-of-tree Broadcom ROCE drivers.

Hi Chris/Jiping,
please see my comments below.

Thanks,
Selvin Xavier

Quoted 25 lines · Jiping

Hi, Chris

Added the comments inline.

Thanks,
Jiping

-----Original Message-----
From: Friesen, Chris <[email protected]>
Sent: Friday, March 1, 2024 4:31 AM
To: [email protected]; [email protected]
Cc: Asselstine, Mark <[email protected]>; Ma, Jiping <[email protected]>
Subject: question about in-tree vs out-of-tree Broadcom ROCE drivers.

Hi,

I got your address from the Linux kernel MAINTAINERS file, I was wondering if you could clear something up?

As far as I can tell, the in-tree driver at drivers/infiniband/hw/bnxt_re uses a BNXT_RE_ABI_VERSION value of 1, as defined in include/uapi/rdma/bnxt_re-abi.h.

On the other hand, the libbnxt_re-228.0.133.0 package and the
bnxt_re-228.0.133.0 driver embedded within
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0 are using a BNXT_RE_ABI_VERSION of 6.

Quoted 5 lines

[Jiping] The abi_version of OOT bnxt_re driver is 6. BNXT_RE_ABI_VERSION of 6 of libbnxt_re is to compatible with OOT driver. Perhaps libbnxt_re does not consider the in-tree driver.

This seems to indicate that the in-tree kernel driver cannot be used with the official version of libbnxt_re as distributed by Broadcom. Is this correct? If so, is there a separate version of libbnxt_re intended to be used with the in-kernel driver?

[Jiping] Yes, it looks like. But I did RDMA test, the test passed. So I suggest we can do more full tests for this part, such as wrcp regression test etc. In addition, I did not do more search if there is other libbnxt_re can be used for in-tree driver. Could we also confirm with Broadcom for this question?

If you are using an in-tree kernel driver, then you can use the libbnxt_re coming with the rdma-core/libibverbs packages in the OS distro. libbnxt_re is part of rdma-core
https://github.com/linux-rdma/rdma-core/tree/master/providers/bnxt_re.
So if you are using the in tree bnxt_re driver, no need to install any libbnxt_re separately. Please install the rdma-core/libibverbs package.

If you are using the OOT driver, please use the OOT libbnxt_re and bnxt_re/bnxt_en driver from the Broadcom download section.
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0

Quoted 5 lines

Thanks,

Chris Friesen