Skip to contentSkip to conversation

Add rxe as a second software RDMA option

PATCH In review [email protected] 3 participants 8 messages Jun 3, 2024 – Jun 7, 2024 Kbuild

lore
Share & tools
lore
#0 lore raw

cover Add rxe as a second software RDMA option

From: Chuck Lever <[email protected]>

Zhu would like to have the ability to run file system tests with software RoCE. Add rxe as an alternative to siw.

I've tested that the Kconfig and Ansible infrastructure behaves as expected, but rxe itself might not be robust enough for the existing workflows to pass cleanly. But hey, that's why we're running these tests, right?

Chuck Lever (4):
bringup.goals: Make siw settings more generic
playbooks: Add a playbook to set up rxe on target nodes
Kconfig: Add support for testing RDMA with software RoCE Update help text that mentions SIW

 Makefile                                   |  2 +-
 kconfigs/Kconfig.bringup.goals             | 36 ++++++++++++++++++----
 playbooks/roles/rxe/tasks/main.yml         | 29 +++++++++++++++++
 playbooks/roles/rxe/templates/udev-rule.j2 |  2 ++
 playbooks/rxe.yml                          |  4 +++
 scripts/rdma.Makefile                      | 23 ++++++++++++++
 scripts/siw.Makefile                       | 11 -------
 workflows/fstests/nfs/Kconfig              |  6 ++--
 workflows/gitr/nfs/Kconfig                 |  3 +-
 9 files changed, 93 insertions(+), 23 deletions(-)
 create mode 100644 playbooks/roles/rxe/tasks/main.yml
 create mode 100644 playbooks/roles/rxe/templates/udev-rule.j2
 create mode 100644 playbooks/rxe.yml
 create mode 100644 scripts/rdma.Makefile
 delete mode 100644 scripts/siw.Makefile
Signature · 1 line
2.45.1
#1 lore raw

PATCH 1/4 bringup.goals: Make siw settings more generic

From: Chuck Lever <[email protected]>

I'm about to introduce another option for software RDMA. Rename the siw Makefile and Kconfig options to make way for this new option.

Since only one type of emulated RDMA device can be attached to an Ethernet device at a time, make the Kconfig logic choose between no device, siw, or, when it's added, rxe.

NB: This patch replaces CONFIG_KDEVOPS_SETUP_SIW. If you have enabled siw in your configuration, please visit the config setting for software RDMA after applying this patch.

---
 Makefile                                |  2 +-
 kconfigs/Kconfig.bringup.goals          | 26 +++++++++++++++++++------
 scripts/{siw.Makefile => rdma.Makefile} |  2 +-
 3 files changed, 22 insertions(+), 8 deletions(-)
 rename scripts/{siw.Makefile => rdma.Makefile} (79%)
Makefile +1 -1
index 266eccd90111..20c2081118a9 100644
@@ -104,7 +104,7 @@ ifeq (y,$(CONFIG_WORKFLOWS))
 include workflows/Makefile
 endif # CONFIG_WORKFLOWS
 
-include scripts/siw.Makefile
+include scripts/rdma.Makefile
 include scripts/ktls.Makefile
 include scripts/nfsd.Makefile
 include scripts/smbd.Makefile
kconfigs/Kconfig.bringup.goals +20 -6
index e6d3b1815f3b..e8629ebea5e0 100644
@@ -77,16 +77,30 @@ config KDEVOPS_TRY_INSTALL_KDEV_TOOLS
 	  most kernel developers might prefer to have installed on target
 	  systems.
 
-config KDEVOPS_SETUP_SIW
-	bool "Configure soft iWARP on all test systems"
-	default n
+choice
+	prompt "Configure an RDMA device"
+	default KDEVOPS_SETUP_RDMA_NONE
 	help
-	  Configure software-emulated RDMA on all target hosts via the
+	  Provision an RDMA device on target nodes. The default is to
+	  provision no additional RDMA devices.
+
+config KDEVOPS_SETUP_RDMA_NONE
+	bool "none"
+	help
+	  Do not provision an additional RDMA device on target nodes.
+	  This is the default behavior.
+
+config KDEVOPS_SETUP_RDMA_SIW
+	bool "siw"
+	help
+	  Provision software-emulated RDMA on target nodes via the
 	  kernel siw.ko driver. siw can be used with any standard
 	  Ethernet device, including virtio-net devices.
 
-	  Because siw is currently not enabled in most Linux distro
-	  kernels, this option defaults to N.
+	  To use this device driver, enable the CONFIG_RDMA_SIW
+	  option in your test kernels.
+
+endchoice
 
 config KDEVOPS_SETUP_KTLS
 	bool "Configure ktls on the hosts with self-signed CA"
scripts/rdma.Makefile renamed from scripts/siw.Makefile +1 -1
similarity index 79%
rename from scripts/siw.Makefile
rename to scripts/rdma.Makefile
index d09d1a2f82ac..d7cdf0062638 100644
@@ -1,4 +1,4 @@
-ifeq (y,$(CONFIG_KDEVOPS_SETUP_SIW))
+ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_SIW))
 
 siw:
 	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
Signature · 1 line
2.45.1
#2 lore raw

PATCH 2/4 playbooks: Add a playbook to set up rxe on target nodes

In reply to [email protected]

From: Chuck Lever <[email protected]>

The playbook adds a udev rule to provision an rxe device when the node is booted.

---
 playbooks/roles/rxe/tasks/main.yml         | 29 ++++++++++++++++++++++
 playbooks/roles/rxe/templates/udev-rule.j2 |  2 ++
 playbooks/rxe.yml                          |  4 +++
 3 files changed, 35 insertions(+)
 create mode 100644 playbooks/roles/rxe/tasks/main.yml
 create mode 100644 playbooks/roles/rxe/templates/udev-rule.j2
 create mode 100644 playbooks/rxe.yml
playbooks/roles/rxe/tasks/main.yml new file +29 -0
new file mode 100644
index 000000000000..88e655981f7f
@@ -0,0 +1,29 @@
+---
+- name: Include optional extra_vars
+  ansible.builtin.include_vars:
+    file: "{{ item }}"
+  with_first_found:
+    - files:
+        - "../extra_vars.yml"
+        - "../extra_vars.yaml"
+        - "../extra_vars.json"
+      skip: true
+  failed_when: false
+  tags: vars
+
+- name: Insert a udev rule to create an rxe device
+  become: true
+  become_method: ansible.builtin.sudo
+  ansible.builtin.template:
+    src: "udev-rule.j2"
+    dest: "/usr/lib/udev/rules.d/99-rxe.rules"
+    owner: "root"
+    group: "root"
+    mode: "0644"
+    force: false
+
+- name: Reload the udev ruleset
+  become: true
+  become_method: ansible.builtin.sudo
+  ansible.builtin.shell: "udevadm control --reload && udevadm trigger"
+  changed_when: true
playbooks/roles/rxe/templates/udev-rule.j2 new file +2 -0
new file mode 100644
index 000000000000..9f5b5e4a68f3
@@ -0,0 +1,2 @@
+SUBSYSTEM=="net", KERNEL=="{{ ansible_default_ipv4.interface }}", \
+   PROGRAM="/usr/sbin/rdma link add rxe0 type rxe netdev {{ ansible_default_ipv4.interface }}"
playbooks/rxe.yml new file +4 -0
new file mode 100644
index 000000000000..dc51c0347628
@@ -0,0 +1,4 @@
+---
+- hosts: all
+  roles:
+    - role: rxe
Signature · 1 line
2.45.1
#3 lore raw

PATCH 3/4 Kconfig: Add support for testing RDMA with software RoCE

In reply to [email protected]

From: Chuck Lever <[email protected]>

Add kdevops config option to use rxe rather than siw for RDMA. rxe is the kernel's software RoCE driver, and like siw, it utilizes a standard Ethernet device.

---
 kconfigs/Kconfig.bringup.goals | 10 ++++++++++
 scripts/rdma.Makefile          | 12 ++++++++++++
 2 files changed, 22 insertions(+)
kconfigs/Kconfig.bringup.goals +10 -0
index e8629ebea5e0..794742530e9e 100644
@@ -100,6 +100,16 @@ config KDEVOPS_SETUP_RDMA_SIW
 	  To use this device driver, enable the CONFIG_RDMA_SIW
 	  option in your test kernels.
 
+config KDEVOPS_SETUP_RDMA_RXE
+	bool "rxe"
+	help
+	  Provision software-emulated RDMA on target nodes via the
+	  kernel rxe.ko driver. rxe can be used with any standard
+	  Ethernet device, including virtio-net devices.
+
+	  To use this device driver, enable the CONFIG_RDMA_RXE
+	  option in your test kernels.
+
 endchoice
 
 config KDEVOPS_SETUP_KTLS
scripts/rdma.Makefile +12 -0
index d7cdf0062638..dd2a70120eb2 100644
@@ -9,3 +9,15 @@ KDEVOPS_BRING_UP_DEPS += siw
 PHONY += siw
 
 endif
+
+ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_RXE))
+
+rxe:
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
+		-f 30 -i hosts playbooks/rxe.yml
+
+KDEVOPS_BRING_UP_DEPS += rxe
+
+PHONY += rxe
+
+endif
Signature · 1 line
2.45.1
#6 lore raw

PATCH 4/4 Update help text that mentions SIW

In reply to [email protected]

From: Chuck Lever <[email protected]>

Generalize the help text, now that kdevops has two software RDMA options.

---
 workflows/fstests/nfs/Kconfig | 6 +++---
 workflows/gitr/nfs/Kconfig    | 3 +--
 2 files changed, 4 insertions(+), 5 deletions(-)
workflows/fstests/nfs/Kconfig +3 -3
index 9f47761712b2..926bc40b4559 100644
@@ -50,9 +50,9 @@ config FSTESTS_NFS_SECTION_RDMA
 	bool "Enable testing section: nfs_rdma"
 	default n
 	help
-	  Enabling this will test with the "rdma" mount option. Unless the
-	  hosts have RDMA hardware already, you probably also want to enable
-	  CONFIG_KDEVOPS_SETUP_SIW as well.
+	  Enabling this will test with the "rdma" mount option. Unless
+	  the target nodes have RDMA hardware already, enable one of
+	  the software RDMA options as well.
 
 config FSTESTS_NFS_SECTION_TLS
 	bool "Enable testing section: nfs_tls"
workflows/gitr/nfs/Kconfig +1 -2
index a8c81b098957..1b1e53316ac7 100644
@@ -38,8 +38,7 @@ config GITR_NFS_SECTION_RDMA
 	help
 	  Provision a target node to run the gitr workflow on a mount
 	  using NFS/RDMA. Unless your target nodes have RDMA hardware
-	  already, you are likely to need to enable the
-	  CONFIG_KDEVOPS_SETUP_SIW option as well.
+	  already, enable one of the software RDMA options as well.
 
 config GITR_NFS_SECTION_V42
 	bool "Enable testing section: nfs_v42"
Signature · 1 line
2.45.1
#4 lore raw

Re: [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE

In reply to [email protected]
Quoted 8 lines · On Mon, Jun 3, 2024 at 6:07 PM <[email protected]> wrote:

From: Chuck Lever <[email protected]>

Add kdevops config option to use rxe rather than siw for RDMA. rxe is the kernel's software RoCE driver, and like siw, it utilizes a standard Ethernet device.

I am fine with this commit.
Thanks a lot.

Zhu Yanjun

Quoted 50 lines
---
 kconfigs/Kconfig.bringup.goals | 10 ++++++++++
 scripts/rdma.Makefile          | 12 ++++++++++++
 2 files changed, 22 insertions(+)
kconfigs/Kconfig.bringup.goals +10 -0
index e8629ebea5e0..794742530e9e 100644
@@ -100,6 +100,16 @@ config KDEVOPS_SETUP_RDMA_SIW
          To use this device driver, enable the CONFIG_RDMA_SIW
          option in your test kernels.
 
+config KDEVOPS_SETUP_RDMA_RXE
+       bool "rxe"
+       help
+         Provision software-emulated RDMA on target nodes via the
+         kernel rxe.ko driver. rxe can be used with any standard
+         Ethernet device, including virtio-net devices.
+
+         To use this device driver, enable the CONFIG_RDMA_RXE
+         option in your test kernels.
+
 endchoice
 
 config KDEVOPS_SETUP_KTLS
scripts/rdma.Makefile +12 -1
index d7cdf0062638..dd2a70120eb2 100644
@@ -9,3 +9,15 @@ KDEVOPS_BRING_UP_DEPS += siw
 PHONY += siw
 
 endif
+
+ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_RXE))
+
+rxe:
+       $(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
+               -f 30 -i hosts playbooks/rxe.yml
+
+KDEVOPS_BRING_UP_DEPS += rxe
+
+PHONY += rxe
+
+endif
--

2.45.1

#7 lore raw
In reply to [email protected]
Quoted 8 lines · On Jun 3, 2024, at 12:06 PM, [email protected] wrote:

From: Chuck Lever <[email protected]>

Zhu would like to have the ability to run file system tests with software RoCE. Add rxe as an alternative to siw.

Turns out Red Hat is also interested in testing NFS/RDMA with RoCE. I've pushed these to linux-kdevops/kdevops.git.

Quoted 16 lines

Chuck Lever (4):
bringup.goals: Make siw settings more generic
playbooks: Add a playbook to set up rxe on target nodes
Kconfig: Add support for testing RDMA with software RoCE Update help text that mentions SIW

Makefile                                   |  2 +-
kconfigs/Kconfig.bringup.goals             | 36 ++++++++++++++++++----
playbooks/roles/rxe/tasks/main.yml         | 29 +++++++++++++++++
playbooks/roles/rxe/templates/udev-rule.j2 |  2 ++
playbooks/rxe.yml                          |  4 +++
scripts/rdma.Makefile                      | 23 ++++++++++++++
scripts/siw.Makefile                       | 11 -------
workflows/fstests/nfs/Kconfig              |  6 ++--
workflows/gitr/nfs/Kconfig                 |  3 +-
9 files changed, 93 insertions(+), 23 deletions(-)
Quoted 9 lines

create mode 100644 playbooks/roles/rxe/tasks/main.yml create mode 100644 playbooks/roles/rxe/templates/udev-rule.j2 create mode 100644 playbooks/rxe.yml
create mode 100644 scripts/rdma.Makefile
delete mode 100644 scripts/siw.Makefile

Signature · 1 line
2.45.1

--
Chuck Lever

#5 lore raw

Re: [PATCH 3/4] Kconfig: Add support for testing RDMA with software RoCE

In reply to [email protected]
Quoted 5 lines · On 03.06.24 18:06, [email protected] wrote:

From: Chuck Lever <[email protected]>

Add kdevops config option to use rxe rather than siw for RDMA. rxe is the kernel's software RoCE driver, and like siw, it utilizes a standard Ethernet device.

Hi, Chuck

Thanks a lot. I am very glad that rxe is used in kdevops. I looked through this software.

It is very powerful. But because I have not an appropriate host at hand, I can not install

kdevops in the local hosts successfully.

After I can find a good host, I will install kdevops in the host and verify the functionality of rxe

of the new linux kernel release.

Chuck, thanks a lot for your efforts to bring rxe into kdevops (a powerful tool).

Zhu Yanjun

Quoted 49 lines
---
  kconfigs/Kconfig.bringup.goals | 10 ++++++++++
  scripts/rdma.Makefile          | 12 ++++++++++++
  2 files changed, 22 insertions(+)
kconfigs/Kconfig.bringup.goals +10 -0
index e8629ebea5e0..794742530e9e 100644
@@ -100,6 +100,16 @@ config KDEVOPS_SETUP_RDMA_SIW
  	  To use this device driver, enable the CONFIG_RDMA_SIW
  	  option in your test kernels.
 
+config KDEVOPS_SETUP_RDMA_RXE
+	bool "rxe"
+	help
+	  Provision software-emulated RDMA on target nodes via the
+	  kernel rxe.ko driver. rxe can be used with any standard
+	  Ethernet device, including virtio-net devices.
+
+	  To use this device driver, enable the CONFIG_RDMA_RXE
+	  option in your test kernels.
+
  endchoice
 
  config KDEVOPS_SETUP_KTLS
scripts/rdma.Makefile +12 -0
index d7cdf0062638..dd2a70120eb2 100644
@@ -9,3 +9,15 @@ KDEVOPS_BRING_UP_DEPS += siw
  PHONY += siw
 
  endif
+
+ifeq (y,$(CONFIG_KDEVOPS_SETUP_RDMA_RXE))
+
+rxe:
+	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) --extra-vars=@./extra_vars.yaml \
+		-f 30 -i hosts playbooks/rxe.yml
+
+KDEVOPS_BRING_UP_DEPS += rxe
+
+PHONY += rxe
+
+endif
Signature · 1 line
Best