Add rxe as a second software RDMA option
cover Add rxe as a second software RDMA option
From: Chuck Lever <[email protected]>
Zhu would like to have the ability to run file system tests with software RoCE. Add rxe as an alternative to siw.
I've tested that the Kconfig and Ansible infrastructure behaves as expected, but rxe itself might not be robust enough for the existing workflows to pass cleanly. But hey, that's why we're running these tests, right?
Chuck Lever (4):
bringup.goals: Make siw settings more generic
playbooks: Add a playbook to set up rxe on target nodes
Kconfig: Add support for testing RDMA with software RoCE Update help text that mentions SIW
Makefile | 2 +- kconfigs/Kconfig.bringup.goals | 36 ++++++++++++++++++---- playbooks/roles/rxe/tasks/main.yml | 29 +++++++++++++++++ playbooks/roles/rxe/templates/udev-rule.j2 | 2 ++ playbooks/rxe.yml | 4 +++ scripts/rdma.Makefile | 23 ++++++++++++++ scripts/siw.Makefile | 11 ------- workflows/fstests/nfs/Kconfig | 6 ++-- workflows/gitr/nfs/Kconfig | 3 +- 9 files changed, 93 insertions(+), 23 deletions(-) create mode 100644 playbooks/roles/rxe/tasks/main.yml create mode 100644 playbooks/roles/rxe/templates/udev-rule.j2 create mode 100644 playbooks/rxe.yml create mode 100644 scripts/rdma.Makefile delete mode 100644 scripts/siw.Makefile
Signature · 1 line
2.45.1
PATCH 1/4 bringup.goals: Make siw settings more generic
From: Chuck Lever <[email protected]>
I'm about to introduce another option for software RDMA. Rename the siw Makefile and Kconfig options to make way for this new option.
Since only one type of emulated RDMA device can be attached to an Ethernet device at a time, make the Kconfig logic choose between no device, siw, or, when it's added, rxe.
NB: This patch replaces CONFIG_KDEVOPS_SETUP_SIW. If you have enabled siw in your configuration, please visit the config setting for software RDMA after applying this patch.
---
Makefile | 2 +-
kconfigs/Kconfig.bringup.goals | 26 +++++++++++++++++++------
scripts/{siw.Makefile => rdma.Makefile} | 2 +-
3 files changed, 22 insertions(+), 8 deletions(-)
rename scripts/{siw.Makefile => rdma.Makefile} (79%)Makefile +1 -1
@@ -104,7 +104,7 @@ ifeq (y,$(CONFIG_WORKFLOWS)) include workflows/Makefile endif # CONFIG_WORKFLOWS -include scripts/siw.Makefile +include scripts/rdma.Makefile include scripts/ktls.Makefile include scripts/nfsd.Makefile include scripts/smbd.Makefile
kconfigs/Kconfig.bringup.goals +20 -6
@@ -77,16 +77,30 @@ config KDEVOPS_TRY_INSTALL_KDEV_TOOLS most kernel developers might prefer to have installed on target systems. -config KDEVOPS_SETUP_SIW - bool "Configure soft iWARP on all test systems" - default n +choice + prompt "Configure an RDMA device" + default KDEVOPS_SETUP_RDMA_NONE help - Configure software-emulated RDMA on all target hosts via the + Provision an RDMA device on target nodes. The default is to + provision no additional RDMA devices. + +config KDEVOPS_SETUP_RDMA_NONE + bool "none" + help + Do not provision an additional RDMA device on target nodes. + This is the default behavior. + +config KDEVOPS_SETUP_RDMA_SIW + bool "siw" + help + Provision software-emulated RDMA on target nodes via the kernel siw.ko driver. siw can be used with any standard Ethernet device, including virtio-net devices. - Because siw is currently not enabled in most Linux distro - kernels, this option defaults to N. + To use this device driver, enable the CONFIG_RDMA_SIW + option in your test kernels. + +endchoice config KDEVOPS_SETUP_KTLS bool "Configure ktls on the hosts with self-signed CA"
scripts/rdma.Makefile renamed from scripts/siw.Makefile +1 -1
@@ -1,4 +1,4 @@ -ifeq (y,$(CONFIG_KDEVOPS_SETUP_SIW)) +ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_SIW)) siw: $(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
Signature · 1 line
2.45.1
PATCH 2/4 playbooks: Add a playbook to set up rxe on target nodes
In reply to [email protected]From: Chuck Lever <[email protected]>
The playbook adds a udev rule to provision an rxe device when the node is booted.
--- playbooks/roles/rxe/tasks/main.yml | 29 ++++++++++++++++++++++ playbooks/roles/rxe/templates/udev-rule.j2 | 2 ++ playbooks/rxe.yml | 4 +++ 3 files changed, 35 insertions(+) create mode 100644 playbooks/roles/rxe/tasks/main.yml create mode 100644 playbooks/roles/rxe/templates/udev-rule.j2 create mode 100644 playbooks/rxe.yml
playbooks/roles/rxe/tasks/main.yml new file +29 -0
@@ -0,0 +1,29 @@ +--- +- name: Include optional extra_vars + ansible.builtin.include_vars: + file: "{{ item }}" + with_first_found: + - files: + - "../extra_vars.yml" + - "../extra_vars.yaml" + - "../extra_vars.json" + skip: true + failed_when: false + tags: vars + +- name: Insert a udev rule to create an rxe device + become: true + become_method: ansible.builtin.sudo + ansible.builtin.template: + src: "udev-rule.j2" + dest: "/usr/lib/udev/rules.d/99-rxe.rules" + owner: "root" + group: "root" + mode: "0644" + force: false + +- name: Reload the udev ruleset + become: true + become_method: ansible.builtin.sudo + ansible.builtin.shell: "udevadm control --reload && udevadm trigger" + changed_when: true
playbooks/roles/rxe/templates/udev-rule.j2 new file +2 -0
@@ -0,0 +1,2 @@ +SUBSYSTEM=="net", KERNEL=="{{ ansible_default_ipv4.interface }}", \ + PROGRAM="/usr/sbin/rdma link add rxe0 type rxe netdev {{ ansible_default_ipv4.interface }}"
playbooks/rxe.yml new file +4 -0
@@ -0,0 +1,4 @@ +--- +- hosts: all + roles: + - role: rxe
Signature · 1 line
2.45.1
PATCH 3/4 Kconfig: Add support for testing RDMA with software RoCE
In reply to [email protected]From: Chuck Lever <[email protected]>
Add kdevops config option to use rxe rather than siw for RDMA. rxe is the kernel's software RoCE driver, and like siw, it utilizes a standard Ethernet device.
--- kconfigs/Kconfig.bringup.goals | 10 ++++++++++ scripts/rdma.Makefile | 12 ++++++++++++ 2 files changed, 22 insertions(+)
kconfigs/Kconfig.bringup.goals +10 -0
@@ -100,6 +100,16 @@ config KDEVOPS_SETUP_RDMA_SIW To use this device driver, enable the CONFIG_RDMA_SIW option in your test kernels. +config KDEVOPS_SETUP_RDMA_RXE + bool "rxe" + help + Provision software-emulated RDMA on target nodes via the + kernel rxe.ko driver. rxe can be used with any standard + Ethernet device, including virtio-net devices. + + To use this device driver, enable the CONFIG_RDMA_RXE + option in your test kernels. + endchoice config KDEVOPS_SETUP_KTLS
scripts/rdma.Makefile +12 -0
@@ -9,3 +9,15 @@ KDEVOPS_BRING_UP_DEPS += siw PHONY += siw endif + +ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_RXE)) + +rxe: + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \ + -f 30 -i hosts playbooks/rxe.yml + +KDEVOPS_BRING_UP_DEPS += rxe + +PHONY += rxe + +endif
Signature · 1 line
2.45.1
Re: [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE
Quoted 8 lines · On Mon, Jun 3, 2024 at 6:07 PM <[email protected]> wrote:
From: Chuck Lever <[email protected]>
Add kdevops config option to use rxe rather than siw for RDMA. rxe is the kernel's software RoCE driver, and like siw, it utilizes a standard Ethernet device.
Suggested-by Zhu Yanjun <[email protected]>
I am fine with this commit.
Thanks a lot.
Zhu Yanjun
Quoted 50 lines
Signed-off-by Chuck Lever <[email protected]>--- kconfigs/Kconfig.bringup.goals | 10 ++++++++++ scripts/rdma.Makefile | 12 ++++++++++++ 2 files changed, 22 insertions(+)kconfigs/Kconfig.bringup.goals +10
-0@@ -100,6 +100,16 @@ config KDEVOPS_SETUP_RDMA_SIW To use this device driver, enable the CONFIG_RDMA_SIW option in your test kernels. +config KDEVOPS_SETUP_RDMA_RXE + bool "rxe" + help + Provision software-emulated RDMA on target nodes via the + kernel rxe.ko driver. rxe can be used with any standard + Ethernet device, including virtio-net devices. + + To use this device driver, enable the CONFIG_RDMA_RXE + option in your test kernels. + endchoice config KDEVOPS_SETUP_KTLSscripts/rdma.Makefile +12
-1@@ -9,3 +9,15 @@ KDEVOPS_BRING_UP_DEPS += siw PHONY += siw endif + +ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_RXE)) + +rxe: + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \ + -f 30 -i hosts playbooks/rxe.yml + +KDEVOPS_BRING_UP_DEPS += rxe + +PHONY += rxe + +endif --2.45.1
Re: [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE
In reply to [email protected]Quoted 5 lines · On 03.06.24 18:06, [email protected] wrote:
From: Chuck Lever <[email protected]>
Add kdevops config option to use rxe rather than siw for RDMA. rxe is the kernel's software RoCE driver, and like siw, it utilizes a standard Ethernet device.
Hi, Chuck
Thanks a lot. I am very glad that rxe is used in kdevops. I looked through this software.
It is very powerful. But because I have not an appropriate host at hand, I can not install
kdevops in the local hosts successfully.
After I can find a good host, I will install kdevops in the host and verify the functionality of rxe
of the new linux kernel release.
Chuck, thanks a lot for your efforts to bring rxe into kdevops (a powerful tool).
Zhu Yanjun
Quoted 49 lines
Suggested-by Zhu Yanjun <[email protected]>Signed-off-by Chuck Lever <[email protected]>--- kconfigs/Kconfig.bringup.goals | 10 ++++++++++ scripts/rdma.Makefile | 12 ++++++++++++ 2 files changed, 22 insertions(+)kconfigs/Kconfig.bringup.goals +10
-0@@ -100,6 +100,16 @@ config KDEVOPS_SETUP_RDMA_SIW To use this device driver, enable the CONFIG_RDMA_SIW option in your test kernels. +config KDEVOPS_SETUP_RDMA_RXE + bool "rxe" + help + Provision software-emulated RDMA on target nodes via the + kernel rxe.ko driver. rxe can be used with any standard + Ethernet device, including virtio-net devices. + + To use this device driver, enable the CONFIG_RDMA_RXE + option in your test kernels. + endchoice config KDEVOPS_SETUP_KTLSscripts/rdma.Makefile +12
-0@@ -9,3 +9,15 @@ KDEVOPS_BRING_UP_DEPS += siw PHONY += siw endif + +ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_RXE)) + +rxe: + $(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \ + -f 30 -i hosts playbooks/rxe.yml + +KDEVOPS_BRING_UP_DEPS += rxe + +PHONY += rxe + +endif
Signature · 1 line
Best
PATCH 4/4 Update help text that mentions SIW
In reply to [email protected]From: Chuck Lever <[email protected]>
Generalize the help text, now that kdevops has two software RDMA options.
--- workflows/fstests/nfs/Kconfig | 6 +++--- workflows/gitr/nfs/Kconfig | 3 +-- 2 files changed, 4 insertions(+), 5 deletions(-)
workflows/fstests/nfs/Kconfig +3 -3
@@ -50,9 +50,9 @@ config FSTESTS_NFS_SECTION_RDMA bool "Enable testing section: nfs_rdma" default n help - Enabling this will test with the "rdma" mount option. Unless the - hosts have RDMA hardware already, you probably also want to enable - CONFIG_KDEVOPS_SETUP_SIW as well. + Enabling this will test with the "rdma" mount option. Unless + the target nodes have RDMA hardware already, enable one of + the software RDMA options as well. config FSTESTS_NFS_SECTION_TLS bool "Enable testing section: nfs_tls"
workflows/gitr/nfs/Kconfig +1 -2
@@ -38,8 +38,7 @@ config GITR_NFS_SECTION_RDMA help Provision a target node to run the gitr workflow on a mount using NFS/RDMA. Unless your target nodes have RDMA hardware - already, you are likely to need to enable the - CONFIG_KDEVOPS_SETUP_SIW option as well. + already, enable one of the software RDMA options as well. config GITR_NFS_SECTION_V42 bool "Enable testing section: nfs_v42"
Signature · 1 line
2.45.1
Quoted 8 lines · On Jun 3, 2024, at 12:06 PM, [email protected] wrote:
From: Chuck Lever <[email protected]>
Zhu would like to have the ability to run file system tests with software RoCE. Add rxe as an alternative to siw.
Turns out Red Hat is also interested in testing NFS/RDMA with RoCE. I've pushed these to linux-kdevops/kdevops.git.
Quoted 16 lines
Chuck Lever (4):
bringup.goals: Make siw settings more generic
playbooks: Add a playbook to set up rxe on target nodes
Kconfig: Add support for testing RDMA with software RoCE Update help text that mentions SIWMakefile | 2 +- kconfigs/Kconfig.bringup.goals | 36 ++++++++++++++++++---- playbooks/roles/rxe/tasks/main.yml | 29 +++++++++++++++++ playbooks/roles/rxe/templates/udev-rule.j2 | 2 ++ playbooks/rxe.yml | 4 +++ scripts/rdma.Makefile | 23 ++++++++++++++ scripts/siw.Makefile | 11 ------- workflows/fstests/nfs/Kconfig | 6 ++-- workflows/gitr/nfs/Kconfig | 3 +- 9 files changed, 93 insertions(+), 23 deletions(-)
Quoted 9 lines
create mode 100644
playbooks/roles/rxe/tasks/main.ymlcreate mode 100644 playbooks/roles/rxe/templates/udev-rule.j2 create mode 100644playbooks/rxe.yml
create mode 100644scripts/rdma.Makefile
delete mode 100644scripts/siw.MakefileSignature · 1 line
2.45.1
--
Chuck Lever