question about in-tree vs out-of-tree Broadcom ROCE drivers.
Hi,
I got your address from the Linux kernel MAINTAINERS file, I was wondering if you could clear something up?
As far as I can tell, the in-tree driver at drivers/infiniband/hw/bnxt_re uses a BNXT_RE_ABI_VERSION value of 1, as defined in include/uapi/rdma/bnxt_re-abi.h.
On the other hand, the libbnxt_re-228.0.133.0 package and the bnxt_re-228.0.133.0 driver embedded within https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0 are using a BNXT_RE_ABI_VERSION of 6.
This seems to indicate that the in-tree kernel driver cannot be used with the official version of libbnxt_re as distributed by Broadcom. Is this correct? If so, is there a separate version of libbnxt_re intended to be used with the in-kernel driver?
Thanks,
Chris Friesen
Hi, Chris
Added the comments inline.
Thanks,
Jiping
-----Original Message-----
From: Friesen, Chris <[email protected]>
Sent: Friday, March 1, 2024 4:31 AM
To: [email protected]; [email protected]
Cc: Asselstine, Mark <[email protected]>; Ma, Jiping <[email protected]>
Subject: question about in-tree vs out-of-tree Broadcom ROCE drivers.
Hi,
I got your address from the Linux kernel MAINTAINERS file, I was wondering if you could clear something up?
As far as I can tell, the in-tree driver at drivers/infiniband/hw/bnxt_re uses a BNXT_RE_ABI_VERSION value of 1, as defined in include/uapi/rdma/bnxt_re-abi.h.
On the other hand, the libbnxt_re-228.0.133.0 package and the
bnxt_re-228.0.133.0 driver embedded within
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0 are using a BNXT_RE_ABI_VERSION of 6.
[Jiping] The abi_version of OOT bnxt_re driver is 6. BNXT_RE_ABI_VERSION of 6 of libbnxt_re is to compatible with OOT driver. Perhaps libbnxt_re does not consider the in-tree driver.
This seems to indicate that the in-tree kernel driver cannot be used with the official version of libbnxt_re as distributed by Broadcom. Is this correct? If so, is there a separate version of libbnxt_re intended to be used with the in-kernel driver?
[Jiping] Yes, it looks like. But I did RDMA test, the test passed. So I suggest we can do more full tests for this part, such as wrcp regression test etc. In addition, I did not do more search if there is other libbnxt_re can be used for in-tree driver. Could we also confirm with Broadcom for this question?
Thanks,
Chris Friesen
Hi Chris/Jiping,
please see my comments below.
Thanks,
Selvin Xavier
Quoted 25 lines · Jiping
Hi, Chris
Added the comments inline.
Thanks,
Jiping-----Original Message-----
From: Friesen, Chris <[email protected]>
Sent: Friday, March 1, 2024 4:31 AM
To: [email protected]; [email protected]
Cc: Asselstine, Mark <[email protected]>; Ma, Jiping <[email protected]>
Subject: question about in-tree vs out-of-tree Broadcom ROCE drivers.Hi,
I got your address from the Linux kernel
MAINTAINERSfile, I was wondering if you could clear something up?As far as I can tell, the in-tree driver at
drivers/infiniband/hw/bnxt_reuses aBNXT_RE_ABI_VERSIONvalue of 1, as defined ininclude/uapi/rdma/bnxt_re-abi.h.On the other hand, the libbnxt_re-228.0.133.0 package and the
bnxt_re-228.0.133.0 driver embedded within
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0 are using aBNXT_RE_ABI_VERSIONof 6.
Quoted 5 lines
[Jiping] The
abi_versionof OOTbnxt_redriver is 6.BNXT_RE_ABI_VERSIONof 6 oflibbnxt_reis to compatible with OOT driver. Perhapslibbnxt_redoes not consider the in-tree driver.This seems to indicate that the in-tree kernel driver cannot be used with the official version of
libbnxt_reas distributed by Broadcom. Is this correct? If so, is there a separate version oflibbnxt_reintended to be used with the in-kernel driver?[Jiping] Yes, it looks like. But I did RDMA test, the test passed. So I suggest we can do more full tests for this part, such as wrcp regression test etc. In addition, I did not do more search if there is other
libbnxt_recan be used for in-tree driver. Could we also confirm with Broadcom for this question?
If you are using an in-tree kernel driver, then you can use the libbnxt_re coming with the rdma-core/libibverbs packages in the OS distro. libbnxt_re is part of rdma-core
https://github.com/linux-rdma/rdma-core/tree/master/providers/bnxt_re.
So if you are using the in tree bnxt_re driver, no need to install any libbnxt_re separately. Please install the rdma-core/libibverbs package.
If you are using the OOT driver, please use the OOT libbnxt_re and bnxt_re/bnxt_en driver from the Broadcom download section.
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0
Quoted 5 lines
Thanks,
Chris Friesen
Hi, Xavier
Could you help to confirm if the in-tree driver bnxt_re/bnxt_en in kernel 6.6.7 can replace OOT drivers? We need support two kinds of drivers OOT and in-tree at the same time if in-tree driver can not replace OOT drivers. We need two libbnxt_re and rdma-core packages in the system to support in-tree drivers and OOT drivers.
Thanks,
Jiping
-----Original Message-----
From: Selvin Xavier <[email protected]>
Sent: Friday, March 1, 2024 2:07 PM
To: Ma, Jiping <[email protected]>
Cc: Friesen, Chris <[email protected]>; [email protected]; Asselstine, Mark <[email protected]>; Tao, Yue <[email protected]>; Wang, Linda <[email protected]>; Bicakci, Vefa <[email protected]>
Subject: Re: question about in-tree vs out-of-tree Broadcom ROCE drivers.
Hi Chris/Jiping,
please see my comments below.
Thanks,
Selvin Xavier
Quoted 25 lines · Jiping
Hi, Chris
Added the comments inline.
Thanks,
Jiping-----Original Message-----
From: Friesen, Chris <[email protected]>
Sent: Friday, March 1, 2024 4:31 AM
To: [email protected]; [email protected]
Cc: Asselstine, Mark <[email protected]>; Ma, Jiping <[email protected]>
Subject: question about in-tree vs out-of-tree Broadcom ROCE drivers.Hi,
I got your address from the Linux kernel
MAINTAINERSfile, I was wondering if you could clear something up?As far as I can tell, the in-tree driver at
drivers/infiniband/hw/bnxt_reuses aBNXT_RE_ABI_VERSIONvalue of 1, as defined ininclude/uapi/rdma/bnxt_re-abi.h.On the other hand, the libbnxt_re-228.0.133.0 package and the
bnxt_re-228.0.133.0 driver embedded within
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0 are using aBNXT_RE_ABI_VERSIONof 6.
Quoted 5 lines
[Jiping] The
abi_versionof OOTbnxt_redriver is 6.BNXT_RE_ABI_VERSIONof 6 oflibbnxt_reis to compatible with OOT driver. Perhapslibbnxt_redoes not consider the in-tree driver.This seems to indicate that the in-tree kernel driver cannot be used with the official version of
libbnxt_reas distributed by Broadcom. Is this correct? If so, is there a separate version oflibbnxt_reintended to be used with the in-kernel driver?[Jiping] Yes, it looks like. But I did RDMA test, the test passed. So I suggest we can do more full tests for this part, such as wrcp regression test etc. In addition, I did not do more search if there is other
libbnxt_recan be used for in-tree driver. Could we also confirm with Broadcom for this question?
If you are using an in-tree kernel driver, then you can use the libbnxt_re coming with the rdma-core/libibverbs packages in the OS distro. libbnxt_re is part of rdma-core
https://github.com/linux-rdma/rdma-core/tree/master/providers/bnxt_re.
So if you are using the in tree bnxt_re driver, no need to install any libbnxt_re separately. Please install the rdma-core/libibverbs package.
If you are using the OOT driver, please use the OOT libbnxt_re and bnxt_re/bnxt_en driver from the Broadcom download section.
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0
Quoted 5 lines
Thanks,
Chris Friesen
can you confirm the adapter version that you use?
OOT and In-tree drivers should support all released adapters. You dont have to (or can not) install both drivers in the same system.
Quoted 71 lines · Jiping
Hi, Xavier
Could you help to confirm if the in-tree driver bnxt_re/bnxt_en in kernel 6.6.7 can replace OOT drivers? We need support two kinds of drivers OOT and in-tree at the same time if in-tree driver can not replace OOT drivers. We need two
libbnxt_reand rdma-core packages in the system to support in-tree drivers and OOT drivers.Thanks,
Jiping-----Original Message-----
From: Selvin Xavier <[email protected]>
Sent: Friday, March 1, 2024 2:07 PM
To: Ma, Jiping <[email protected]>
Cc: Friesen, Chris <[email protected]>; [email protected]; Asselstine, Mark <[email protected]>; Tao, Yue <[email protected]>; Wang, Linda <[email protected]>; Bicakci, Vefa <[email protected]>
Subject: Re: question about in-tree vs out-of-tree Broadcom ROCE drivers.Hi Chris/Jiping,
please see my comments below.Thanks,
Selvin XavierQuoted 25 lines · Jiping
Hi, Chris
Added the comments inline.
Thanks,
Jiping-----Original Message-----
From: Friesen, Chris <[email protected]>
Sent: Friday, March 1, 2024 4:31 AM
To: [email protected]; [email protected]
Cc: Asselstine, Mark <[email protected]>; Ma, Jiping <[email protected]>
Subject: question about in-tree vs out-of-tree Broadcom ROCE drivers.Hi,
I got your address from the Linux kernel
MAINTAINERSfile, I was wondering if you could clear something up?As far as I can tell, the in-tree driver at
drivers/infiniband/hw/bnxt_reuses aBNXT_RE_ABI_VERSIONvalue of 1, as defined ininclude/uapi/rdma/bnxt_re-abi.h.On the other hand, the libbnxt_re-228.0.133.0 package and the
bnxt_re-228.0.133.0 driver embedded within
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0 are using aBNXT_RE_ABI_VERSIONof 6.Quoted 5 lines
[Jiping] The
abi_versionof OOTbnxt_redriver is 6.BNXT_RE_ABI_VERSIONof 6 oflibbnxt_reis to compatible with OOT driver. Perhapslibbnxt_redoes not consider the in-tree driver.This seems to indicate that the in-tree kernel driver cannot be used with the official version of
libbnxt_reas distributed by Broadcom. Is this correct? If so, is there a separate version oflibbnxt_reintended to be used with the in-kernel driver?[Jiping] Yes, it looks like. But I did RDMA test, the test passed. So I suggest we can do more full tests for this part, such as wrcp regression test etc. In addition, I did not do more search if there is other
libbnxt_recan be used for in-tree driver. Could we also confirm with Broadcom for this question?If you are using an in-tree kernel driver, then you can use the
libbnxt_recoming with the rdma-core/libibverbs packages in the OS distro.libbnxt_reis part of rdma-core
https://github.com/linux-rdma/rdma-core/tree/master/providers/bnxt_re.
So if you are using the in treebnxt_redriver, no need to install anylibbnxt_reseparately. Please install the rdma-core/libibverbs package.If you are using the OOT driver, please use the OOT
libbnxt_reand bnxt_re/bnxt_en driver from the Broadcom download section.
https://docs.broadcom.com/docs/NXE_Linux_Installer-228.1.111.0Quoted 5 lines
Thanks,
Chris Friesen