Skip to contentSkip to conversation

Provide an ism layer

RFC In review Alexandra Winter 8 participants 61 messages Jan 15, 2025 – Feb 25, 2025

Subsystems (5)
lore
Share & tools
lore

Alexandra Winter

author maintainer
#0 lore raw

cover Provide an ism layer

This RFC is about providing a generic shim layer between all kinds of ism devices and all kinds of ism users.

Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.

Request for comments:
---------------------
Any comments are welcome, but I am aware that this series needs more work. It may not be worth your time to do an in-depth review of the details, I am looking for feedback on the general idea.
I am mostly interested in your thoughts and recommendations about the general concept, the location of net/ism, the structure of include/linux/ism.h, the KConfig and makefiles.

Status of this RFC:
-------------------
This is a very early RFC to ask you for comments on this general idea. The RFC does not fullfill all criteria required for a patchset. The whole set compiles and runs, but I did not try all combinations of module and built-in yet. I did not check for checkpatch or any other checkers. Also I have only done very rudimentary quick tests of SMC-D. More testing is required.

Background / Status quo:
------------------------
Currently s390 hardware provides virtual PCI ISM devices (ism_vpci). Their driver is in drivers/s390/net/ism_drv.c. The main user is SMC-D (net/smc). ism_vpci driver offers a client interface so other users/protocols can also use them, but it is still heavily intermingled with the smc code. Namely, the ISM vPCI module cannot be used without the SMC module, which feels artificial.

The ISM concept is being extended:
[1] proposed an ISM loopback interface (ism_lo), that can be used on non-s390 architectures (e.g. between containers or to test SMC-D). A minimal implementation went upstream with [2]: ism_lo currently is a part of the smc protocol and rather hidden.

[3] proposed a virtio definition of ISM (ism_virtio) that can be used between kvm guests.

We will shortly send an RFC for an ISM client that uses ISM as transport for TTY.

Concept:
--------
Create a shim layer in net/ism that contains common definitions and code for all ism devices and all ism clients.
Any device or client module only needs to depend on this ism layer module and any device or client code only needs to include the definitions in include/linux/ism.h

Ideas for next steps:
---------------------
- sysfs representation? e.g. as /sys/class/ism ?
- provide a full-fledged ism loopback interface
(runtime enable/disable, sysfs device, ..)
- additional clients (tty over ism)
- additional devices (virtio-ism, ...)

Alexandra Winter (7):
net/ism: Create net/ism
net/ism: Remove dependencies between ISM_VPCI and SMC net/ism: Use uuid_t for ISM GID
net/ism: Add kernel-doc comments for ism functions net/ism: Move ism_loopback to net/ism s390/ism: Define ismvp_dev
net/smc: Use only ism_ops

 MAINTAINERS                |   7 +
 drivers/s390/net/Kconfig   |  10 +-
 drivers/s390/net/Makefile  |   4 +-
 drivers/s390/net/ism.h     |  27 ++-
 drivers/s390/net/ism_drv.c | 467 ++++++++++++-------------------------
 include/linux/ism.h        | 299 +++++++++++++++++++++---
 include/net/smc.h          |  52 +----
 net/Kconfig                |   1 +
 net/Makefile               |   1 +
 net/ism/Kconfig            |  27 +++
 net/ism/Makefile           |   8 +
 net/ism/ism_loopback.c     | 366 +++++++++++++++++++++++++++++
 net/ism/ism_loopback.h     |  59 +++++
 net/ism/ism_main.c         | 171 ++++++++++++++
 net/smc/Kconfig            |  13 --
 net/smc/Makefile           |   1 -
 net/smc/af_smc.c           |  12 +-
 net/smc/smc_clc.c          |   6 +-
 net/smc/smc_core.c         |   6 +-
 net/smc/smc_diag.c         |   2 +-
 net/smc/smc_ism.c          | 112 +++++----
 net/smc/smc_ism.h          |  29 ++-
 net/smc/smc_loopback.c     | 427 ---------------------------------
 net/smc/smc_loopback.h     |  60 -----
 net/smc/smc_pnet.c         |   8 +-
 25 files changed, 1183 insertions(+), 992 deletions(-)
 create mode 100644 net/ism/Kconfig
 create mode 100644 net/ism/Makefile
 create mode 100644 net/ism/ism_loopback.c
 create mode 100644 net/ism/ism_loopback.h
 create mode 100644 net/ism/ism_main.c
 delete mode 100644 net/smc/smc_loopback.c
 delete mode 100644 net/smc/smc_loopback.h
Signature · 1 line
2.45.2
author maintainer
#1 lore raw

RFC 1/7 net/ism: Create net/ism

Create an 'ISM' shim layer that will provide generic functionality and declarations for ism device drivers and ism clients.

Move the respective pieces from drivers/s390/net/ism_drv.* to net/ism/

When we need to distinguish between generic ism interfaces and specifically the s390 virtual pci ism device, it will be called 'ISM_vPCI'.

No optimizations are done in this patch, it only moves pieces around. Following patch will further detangle ism_vpci and smc-d.

---
 MAINTAINERS                |   7 ++
 drivers/s390/net/Kconfig   |   9 +--
 drivers/s390/net/Makefile  |   4 +-
 drivers/s390/net/ism_drv.c | 129 ++---------------------------
 include/linux/ism.h        |   8 ++
 include/net/smc.h          |   3 -
 net/Kconfig                |   1 +
 net/Makefile               |   1 +
 net/ism/Kconfig            |  14 ++++
 net/ism/Makefile           |   7 ++
 net/ism/ism_main.c         | 162 +++++++++++++++++++++++++++++++++++++
 11 files changed, 213 insertions(+), 132 deletions(-)
 create mode 100644 net/ism/Kconfig
 create mode 100644 net/ism/Makefile
 create mode 100644 net/ism/ism_main.c
MAINTAINERS +7 -0
index 4dcb849e6748..780db61f3f16 100644
@@ -12239,6 +12239,13 @@ F:	Documentation/devicetree/bindings/hwmon/renesas,isl28022.yaml
 F:	Documentation/hwmon/isl28022.rst
 F:	drivers/hwmon/isl28022.c
 
+ISM (INTERNAL SHARED MEMORY)
+M:	Alexandra Winter <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	include/linux/ism.h
+F:	net/ism/
+
 ISOFS FILESYSTEM
 M:	Jan Kara <[email protected]>
 L:	[email protected]
drivers/s390/net/Kconfig +4 -5
index c61e6427384c..2e900d3087d4 100644
@@ -100,15 +100,14 @@ config CCWGROUP
 	tristate
 	default (LCS || CTCM || QETH || SMC)
 
-config ISM
+config ISM_VPCI
 	tristate "Support for ISM vPCI Adapter"
-	depends on PCI
+	depends on PCI && ISM
 	imply SMC
-	default n
+	default y
 	help
 	  Select this option if you want to use the Internal Shared Memory
 	  vPCI Adapter. The adapter can be used with the SMC network protocol.
 
-	  To compile as a module choose M. The module name is ism.
-	  If unsure, choose N.
+	  To compile as a module choose M. The module name is ism_vpci.
 endmenu
drivers/s390/net/Makefile +2 -2
index bc55ec316adb..87461019184e 100644
@@ -16,5 +16,5 @@ obj-$(CONFIG_QETH_L2) += qeth_l2.o
 qeth_l3-y += qeth_l3_main.o qeth_l3_sys.o
 obj-$(CONFIG_QETH_L3) += qeth_l3.o
 
-ism-y := ism_drv.o
-obj-$(CONFIG_ISM) += ism.o
+ism_vpci-y += ism_drv.o
+obj-$(CONFIG_ISM_VPCI) += ism_vpci.o
drivers/s390/net/ism_drv.c +7 -122

187 lines not on this page · raw message

include/linux/ism.h +8 -0
index 5428edd90982..1462296e8ba7 100644
@@ -9,6 +9,7 @@
 #ifndef _ISM_H
 #define _ISM_H
 
+#include <linux/device.h>
 #include <linux/workqueue.h>
 
 struct ism_dmb {
@@ -24,6 +25,7 @@ struct ism_dmb {
 
 /* Unless we gain unexpected popularity, this limit should hold for a while */
 #define MAX_CLIENTS		8
+#define NO_CLIENT		0xff		/* must be >= MAX_CLIENTS */
 #define ISM_NR_DMBS		1920
 
 struct ism_dev {
@@ -76,6 +78,9 @@ static inline void *ism_get_priv(struct ism_dev *dev,
 	return dev->priv[client->id];
 }
 
+int ism_dev_register(struct ism_dev *ism);
+void ism_dev_unregister(struct ism_dev *ism);
+
 static inline void ism_set_priv(struct ism_dev *dev, struct ism_client *client,
 				void *priv) {
 	dev->priv[client->id] = priv;
@@ -87,6 +92,9 @@ int  ism_unregister_dmb(struct ism_dev *dev, struct ism_dmb *dmb);
 int  ism_move(struct ism_dev *dev, u64 dmb_tok, unsigned int idx, bool sf,
 	      unsigned int offset, void *data, unsigned int size);
 
+#define ISM_RESERVED_VLANID	0x1FFF
+#define ISM_ERROR	0xFFFF
+
 const struct smcd_ops *ism_get_smcd_ops(void);
 
 #endif	/* _ISM_H */
include/net/smc.h +0 -3
index db84e4e35080..ab732b286f91 100644
@@ -42,9 +42,6 @@ struct smcd_dmb {
 #define ISM_EVENT_GID	1
 #define ISM_EVENT_SWR	2
 
-#define ISM_RESERVED_VLANID	0x1FFF
-
-#define ISM_ERROR	0xFFFF
 
 struct smcd_dev;
 
net/Kconfig +1 -0
index c3fca69a7c83..2dbe9655f7de 100644
@@ -83,6 +83,7 @@ source "net/tls/Kconfig"
 source "net/xfrm/Kconfig"
 source "net/iucv/Kconfig"
 source "net/smc/Kconfig"
+source "net/ism/Kconfig"
 source "net/xdp/Kconfig"
 
 config NET_HANDSHAKE
net/Makefile +1 -0
index 60ed5190eda8..6f06cf00bfbb 100644
@@ -51,6 +51,7 @@ obj-$(CONFIG_TIPC)		+= tipc/
 obj-$(CONFIG_NETLABEL)		+= netlabel/
 obj-$(CONFIG_IUCV)		+= iucv/
 obj-$(CONFIG_SMC)		+= smc/
+obj-$(CONFIG_ISM)		+= ism/
 obj-$(CONFIG_RFKILL)		+= rfkill/
 obj-$(CONFIG_NET_9P)		+= 9p/
 obj-$(CONFIG_CAIF)		+= caif/
net/ism/Kconfig new file +14 -0
new file mode 100644
index 000000000000..4329489cc1e9
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0
+config ISM
+	tristate "ISM support"
+	default n
+	help
+	  Internal Shared Memory (ISM)
+	  A communication method that uses common physical memory for
+	  synchronous direct access into a remote buffer.
+
+	  Select this option to provide the abstraction layer between
+	  ISM devices and ISM users like the SMC protocol.
+
+	  To compile as a module choose M. The module name is ism.
+	  If unsure, choose N.
net/ism/Makefile new file +7 -0
new file mode 100644
index 000000000000..b752baf72003
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# ISM class module
+#
+
+ism-y += ism_main.o
+obj-$(CONFIG_ISM) += ism.o
net/ism/ism_main.c new file +162 -0

162 lines not on this page · raw message

Signature · 1 line
2.45.2
#2 lore raw

Re: [RFC net-next 1/7] net/ism: Create net/ism

MAINTAINERS F: Documentation/devicetree/bindings/hwmon/renesas,isl28022.yaml · line 12247

Quoted 6 lines
MAINTAINERS +6 -0
+ISM (INTERNAL SHARED MEMORY)
+M:	Alexandra Winter <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	include/linux/ism.h
+F:	net/ism/

Is there any high level documentation about this?

A while back, TI was trying to upstream something for one of there SoCs. It was a multi CPU system, with not all CPUs used for SMP, but one or two kept for management and real time tasks, not even running Linux. They had a block of shared memory used for communication between the CPUs/OSes, along with rproc. They layered an ethernet driver on top of this, with buffers for frames in the shared memory.

Could ISM be used for something like this?

Andrew

author maintainer
#3 lore raw

Re: [RFC net-next 1/7] net/ism: Create net/ism

Quoted 8 lines · Andrew Lunn

MAINTAINERS F: Documentation/devicetree/bindings/hwmon/renesas,isl28022.yaml · line 12247

Quoted 6 lines
MAINTAINERS +6 -0
+ISM (INTERNAL SHARED MEMORY)
+M:	Alexandra Winter <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	include/linux/ism.h
+F:	net/ism/

Is there any high level documentation about this?

As the ISM devices were developed for SMC-D, the only documentation is about their usage for SMC-D.
e.g.:
https://www.ibm.com/support/pages/system/files/inline-files/IBM%20Shared%20Memory%20Communications%20Version%202.1%20Emulated-ISM_0.pdf
(page 33)
https://community.ibm.com/community/user/ibmz-and-linuxone/viewdocument/2021-07-15-boosting-tcp-networking?CommunityKey=c1293167-6d93-448e-8854-3068846d3dfe&tab=librarydocuments
But those do not go into much detail.

We now want to provide interfaces for other usecases in Linux.

ism.h would be the place to explicitely state the assumptions, restrictions and requirements in a single place, so future devices and clients know about them.

Quoted 11 lines

A while back, TI was trying to upstream something for one of there SoCs. It was a multi CPU system, with not all CPUs used for SMP, but one or two kept for management and real time tasks, not even running Linux. They had a block of shared memory used for communication between the CPUs/OSes, along with rproc. They layered an ethernet driver on top of this, with buffers for frames in the shared memory.

Could ISM be used for something like this?

Andrew

If the communication endpoints were represented as devices, that sounds like a similar concept.

I think you could implement a client that provides network devices on top of ism devices. (mapping MACs to GIDs)
As the memory buffers are set up for 1 sender and 1 receiver, it would either create some additional latency, if you setup buffers for each message or additional memory consumption, if you try to keep and re-use the buffers.
I'm not sure what the benefit ISM would provide as ethernet device. A shared network card would probably outperform such a usecase.
SMC exploits ISM for TCP traffic. There the buffers are kept per socket connection, and a lot of the TCP/IP mechanisms are not neccessary, because transport is reliable, synchronous and in-order. Thus latency is minimal.

author maintainer
#4 lore raw

RFC 2/7 net/ism: Remove dependencies between ISM_VPCI and SMC

In reply to Alexandra Winter

The modules ISM_VPCI and SMC should not depend on each other, instead they should both depend on the ISM layer module.

use only ism_dmb:
Now that SMC depends on ISM, we can safely remove the duplicate declaration of smcd_dmb and use only ism_dmb.

Move smcd_ops away from ism_drv:
move smcd_ops from drivers/s390/net/ism_drv.c to net/smc/smc_ism.c Less exported functions, no more dependencies between ISM_VPCI and SMC. Once ism_loopback is also moved to ism layer, a follow on patch can use ism_ops directly and remove smcd_ops.

Now the ISM_VPCI module no longer needs to imply SMC.

Note:
- This patch temporarily moves smcd_gid to ism.h, a follow on patch (uuid_t gid) will restore this.
- Added a comment that vlan handling in ism_drv.c and smc is incomplete. Should be fixed by a follow-on patch.

---
 drivers/s390/net/Kconfig   |   1 -
 drivers/s390/net/ism.h     |   1 -
 drivers/s390/net/ism_drv.c | 236 +++++++++++++------------------------
 include/linux/ism.h        | 146 +++++++++++++++++------
 include/net/smc.h          |  31 ++---
 net/smc/smc_ism.c          | 123 ++++++++++++++++++-
 net/smc/smc_loopback.c     |   6 +-
 7 files changed, 319 insertions(+), 225 deletions(-)
drivers/s390/net/Kconfig +0 -1
index 2e900d3087d4..9bb3cc186510 100644
@@ -103,7 +103,6 @@ config CCWGROUP
 config ISM_VPCI
 	tristate "Support for ISM vPCI Adapter"
 	depends on PCI && ISM
-	imply SMC
 	default y
 	help
 	  Select this option if you want to use the Internal Shared Memory
drivers/s390/net/ism.h +0 -1
index 047fa6101555..8b56e1d82e6b 100644
@@ -6,7 +6,6 @@
 #include <linux/types.h>
 #include <linux/pci.h>
 #include <linux/ism.h>
-#include <net/smc.h>
 #include <asm/pci_insn.h>
 
 #define UTIL_STR_LEN	16
drivers/s390/net/ism_drv.c +80 -156

298 lines not on this page · raw message

include/linux/ism.h +110 -36

180 lines not on this page · raw message

include/net/smc.h +8 -23
index ab732b286f91..3d20c6c05056 100644
@@ -27,35 +27,20 @@ struct smc_hashinfo {
 };
 
 /* SMCD/ISM device driver interface */
-struct smcd_dmb {
-	u64 dmb_tok;
-	u64 rgid;
-	u32 dmb_len;
-	u32 sba_idx;
-	u32 vlan_valid;
-	u32 vlan_id;
-	void *cpu_addr;
-	dma_addr_t dma_addr;
-};
-
-#define ISM_EVENT_DMB	0
-#define ISM_EVENT_GID	1
-#define ISM_EVENT_SWR	2
-
 
 struct smcd_dev;
 
-struct smcd_gid {
-	u64	gid;
-	u64	gid_ext;
-};
-
+//struct smcd_gid {
+//	u64	gid;
+//	u64	gid_ext;
+//};
+//
 struct smcd_ops {
 	int (*query_remote_gid)(struct smcd_dev *dev, struct smcd_gid *rgid,
 				u32 vid_valid, u32 vid);
-	int (*register_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb,
+	int (*register_dmb)(struct smcd_dev *dev, struct ism_dmb *dmb,
 			    void *client);
-	int (*unregister_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb);
+	int (*unregister_dmb)(struct smcd_dev *dev, struct ism_dmb *dmb);
 	int (*move_data)(struct smcd_dev *dev, u64 dmb_tok, unsigned int idx,
 			 bool sf, unsigned int offset, void *data,
 			 unsigned int size);
@@ -72,7 +57,7 @@ struct smcd_ops {
 	int (*signal_event)(struct smcd_dev *dev, struct smcd_gid *rgid,
 			    u32 trigger_irq, u32 event_code, u64 info);
 	int (*support_dmb_nocopy)(struct smcd_dev *dev);
-	int (*attach_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb);
+	int (*attach_dmb)(struct smcd_dev *dev, struct ism_dmb *dmb);
 	int (*detach_dmb)(struct smcd_dev *dev, u64 token);
 };
 
net/smc/smc_ism.c +118 -5

155 lines not on this page · raw message

net/smc/smc_loopback.c +3 -3
index 3c5f64ca4115..c4020653ae20 100644
@@ -51,7 +51,7 @@ static int smc_lo_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid,
 	return 0;
 }
 
-static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
+static int smc_lo_register_dmb(struct smcd_dev *smcd, struct ism_dmb *dmb,
 			       void *client_priv)
 {
 	struct smc_lo_dmb_node *dmb_node, *tmp_node;
@@ -129,7 +129,7 @@ static void __smc_lo_unregister_dmb(struct smc_lo_dev *ldev,
 		wake_up(&ldev->ldev_release);
 }
 
-static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
+static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct ism_dmb *dmb)
 {
 	struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node;
 	struct smc_lo_dev *ldev = smcd->priv;
@@ -158,7 +158,7 @@ static int smc_lo_support_dmb_nocopy(struct smcd_dev *smcd)
 	return SMC_LO_SUPPORT_NOCOPY;
 }
 
-static int smc_lo_attach_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
+static int smc_lo_attach_dmb(struct smcd_dev *smcd, struct ism_dmb *dmb)
 {
 	struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node;
 	struct smc_lo_dev *ldev = smcd->priv;
Signature · 1 line
2.45.2
author maintainer
#5 lore raw

RFC 3/7 net/ism: Use uuid_t for ISM GID

In reply to Alexandra Winter

SMC uses 64 Bit and 128 Bit Global Identifiers (GIDs) that need to be sent via the SMC protocol.
When integers are used network endianness and host endianness need to be considered.

Avoid this in the ISM layer by using uuid_t byte arrays. Follow on patches could do the same change for SMC, for now conversion helper functions are introduced.

ISM-vPCI devices provide 64 Bit GIDs. Map them to ISM uuid_t GIDs like this:
_________________________________________
| 64 Bit ISM-vPCI GID | 00000000_00000000 |
-----------------------------------------
If interpreted as UUID, this would be interpreted as th UIID variant, that is reserved for NCS backward compatibility. So it will not collide with UUIDs that were generated according to the standard.

Future ISM devices, shall use real UUIDs as 128 Bit GIDs.

Note:
- In this RFC patch smcd_gid is now moved back to smc.h, future patchset should avoid that.
- ism_dmb and ism_event structs still contain 64 Bit rgid and info fields. A future patch could change them to uuid_t gids. This does not break anything, because ism_loopback does not use them.

---
 drivers/s390/net/ism.h     |  9 +++++++++
 drivers/s390/net/ism_drv.c | 16 ++++++++--------
 include/linux/ism.h        | 16 ++++++----------
 include/net/smc.h          | 12 ++++++------
 net/smc/smc_ism.c          | 13 ++++++++-----
 net/smc/smc_ism.h          | 21 +++++++++++++++++++++
 6 files changed, 58 insertions(+), 29 deletions(-)
drivers/s390/net/ism.h +9 -0
index 8b56e1d82e6b..61cf10334170 100644
@@ -64,6 +64,15 @@ union ism_reg_ieq {
 	} response;
 } __aligned(16);
 
+/* ISM-vPCI devices provide 64 Bit GIDs
+ * Map them to ISM UUID GIDs like this:
+ *  _________________________________________
+ * | 64 Bit ISM-vPCI GID | 00000000_00000000 |
+ *  -----------------------------------------
+ * This will be interpreted as a UIID variant, that is reserved
+ * for NCS backward compatibility. So it will not collide with
+ * proper UUIDs.
+ */
 union ism_read_gid {
 	struct {
 		struct ism_req_hdr hdr;
drivers/s390/net/ism_drv.c +8 -8
index 112e0d67cdd6..ab70debbdd9d 100644
@@ -191,14 +191,14 @@ static int ism_read_local_gid(struct ism_dev *ism)
 	if (ret)
 		goto out;
 
-	ism->gid.gid = cmd.response.gid;
-	ism->gid.gid_ext = 0;
+	memset(&ism->gid, 0, sizeof(ism->gid));
+	memcpy(&ism->gid, &cmd.response.gid, sizeof(cmd.response.gid));
 out:
 	return ret;
 }
 
-static int ism_query_rgid(struct ism_dev *ism, struct smcd_gid *rgid,
-			  u32 vid_valid, u32 vid)
+static int ism_query_rgid(struct ism_dev *ism, uuid_t *rgid, u32 vid_valid,
+			  u32 vid)
 {
 	union ism_query_rgid cmd;
 
@@ -206,7 +206,7 @@ static int ism_query_rgid(struct ism_dev *ism, struct smcd_gid *rgid,
 	cmd.request.hdr.cmd = ISM_QUERY_RGID;
 	cmd.request.hdr.len = sizeof(cmd.request);
 
-	cmd.request.rgid = rgid->gid;
+	memcpy(&cmd.request.rgid, rgid, sizeof(cmd.request.rgid));
 	cmd.request.vlan_valid = vid_valid;
 	cmd.request.vlan_id = vid;
 
@@ -364,8 +364,8 @@ static int ism_reset_vlan_required(struct ism_dev *ism)
 	return ism_cmd_simple(ism, ISM_RESET_VLAN);
 }
 
-static int ism_signal_ieq(struct ism_dev *ism, struct smcd_gid *rgid,
-			  u32 trigger_irq, u32 event_code, u64 info)
+static int ism_signal_ieq(struct ism_dev *ism, uuid_t *rgid, u32 trigger_irq,
+			  u32 event_code, u64 info)
 {
 	union ism_sig_ieq cmd;
 
@@ -373,7 +373,7 @@ static int ism_signal_ieq(struct ism_dev *ism, struct smcd_gid *rgid,
 	cmd.request.hdr.cmd = ISM_SIGNAL_IEQ;
 	cmd.request.hdr.len = sizeof(cmd.request);
 
-	cmd.request.rgid = rgid->gid;
+	memcpy(&cmd.request.rgid, rgid, sizeof(cmd.request.rgid));
 	cmd.request.trigger_irq = trigger_irq;
 	cmd.request.event_code = event_code;
 	cmd.request.info = info;
include/linux/ism.h +6 -10
index ede1a40b408e..50975847248f 100644
@@ -11,6 +11,7 @@
 
 #include <linux/device.h>
 #include <linux/workqueue.h>
+#include <linux/uuid.h>
 
 /* The remote peer rgid can use dmb_tok to write into this buffer. */
 struct ism_dmb {
@@ -24,11 +25,6 @@ struct ism_dmb {
 	dma_addr_t dma_addr;
 };
 
-struct smcd_gid {
-	u64	gid;
-	u64	gid_ext;
-};
-
 struct ism_event {
 	u32 type;
 	u32 code;
@@ -61,7 +57,7 @@ int ism_register_client(struct ism_client *client);
 int  ism_unregister_client(struct ism_client *client);
 
 /* Mandatory operations for all ism devices:
- * int (*query_remote_gid)(struct ism_dev *dev, struct smcd_gid *rgid,
+ * int (*query_remote_gid)(struct ism_dev *dev, uuid_t *rgid,
  *	                   u32 vid_valid, u32 vid);
  *	Query whether remote GID rgid is reachable via this device and this
  *	vlan id. Vlan id is only checked if vid_valid != 0.
@@ -101,14 +97,14 @@ int  ism_unregister_client(struct ism_client *client);
  *	Ability to assign dmbs to VLANs is missing
  *	- do we really want / need this?
  *
- * int (*signal_event)(struct ism_dev *dev, struct smcd_gid *rgid,
+ * int (*signal_event)(struct ism_dev *dev, uuid_t *rgid,
  *			    u32 trigger_irq, u32 event_code, u64 info);
  *	Send a control event into the event queue of a remote gid (rgid)
  *	with (1) or without (0) triggering an interrupt at the remote gid.
  */
 
 struct ism_ops {
-	int (*query_remote_gid)(struct ism_dev *dev, struct smcd_gid *rgid,
+	int (*query_remote_gid)(struct ism_dev *dev, uuid_t *rgid,
 				u32 vid_valid, u32 vid);
 	int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb,
 			    struct ism_client *client);
@@ -125,7 +121,7 @@ struct ism_ops {
 	int (*del_vlan_id)(struct ism_dev *dev, u64 vlan_id);
 	int (*set_vlan_required)(struct ism_dev *dev);
 	int (*reset_vlan_required)(struct ism_dev *dev);
-	int (*signal_event)(struct ism_dev *dev, struct smcd_gid *rgid,
+	int (*signal_event)(struct ism_dev *dev, uuid_t *rgid,
 			    u32 trigger_irq, u32 event_code, u64 info);
 };
 
@@ -150,7 +146,7 @@ struct ism_dev {
 	dma_addr_t ieq_dma_addr;
 
 	struct device dev;
-	struct smcd_gid gid;
+	uuid_t gid;
 	int ieq_idx;
 
 	struct ism_client *subs[MAX_CLIENTS];
include/net/smc.h +6 -6
index 3d20c6c05056..91aab1d44166 100644
@@ -15,7 +15,7 @@
 #include <linux/spinlock.h>
 #include <linux/types.h>
 #include <linux/wait.h>
-#include "linux/ism.h"
+#include <linux/ism.h>
 
 struct sock;
 
@@ -30,11 +30,11 @@ struct smc_hashinfo {
 
 struct smcd_dev;
 
-//struct smcd_gid {
-//	u64	gid;
-//	u64	gid_ext;
-//};
-//
+struct smcd_gid {
+	u64	gid;
+	u64	gid_ext;
+};
+
 struct smcd_ops {
 	int (*query_remote_gid)(struct smcd_dev *dev, struct smcd_gid *rgid,
 				u32 vid_valid, u32 vid);
net/smc/smc_ism.c +8 -5
index 6fbacad02f23..a49da16bafd5 100644
@@ -485,8 +485,10 @@ static int smcd_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid,
 			   u32 vid_valid, u32 vid)
 {
 	struct ism_dev *ism = smcd->priv;
+	uuid_t ism_rgid;
 
-	return ism->ops->query_remote_gid(ism, rgid, vid_valid, vid);
+	copy_to_ismgid(&ism_rgid, rgid);
+	return ism->ops->query_remote_gid(ism, &ism_rgid, vid_valid, vid);
 }
 
 static int smcd_register_dmb(struct smcd_dev *smcd, struct ism_dmb *dmb,
@@ -536,9 +538,11 @@ static int smcd_signal_ieq(struct smcd_dev *smcd, struct smcd_gid *rgid,
 			   u32 trigger_irq, u32 event_code, u64 info)
 {
 	struct ism_dev *ism = smcd->priv;
+	uuid_t ism_rgid;
 
-	return ism->ops->signal_event(ism, rgid,
-			      trigger_irq, event_code, info);
+	copy_to_ismgid(&ism_rgid, rgid);
+	return ism->ops->signal_event(ism, &ism_rgid, trigger_irq,
+				      event_code, info);
 }
 
 static int smcd_move(struct smcd_dev *smcd, u64 dmb_tok, unsigned int idx,
@@ -560,8 +564,7 @@ static void smcd_get_local_gid(struct smcd_dev *smcd,
 {
 	struct ism_dev *ism = smcd->priv;
 
-	smcd_gid->gid = ism->gid.gid;
-	smcd_gid->gid_ext = ism->gid.gid_ext;
+	copy_to_smcdgid(smcd_gid, &ism->gid);
 }
 
 static u16 smcd_get_chid(struct smcd_dev *smcd)
net/smc/smc_ism.h +21 -0
index 6763133dd8d0..d041e5a7c459 100644
@@ -12,6 +12,7 @@
 #include <linux/uio.h>
 #include <linux/types.h>
 #include <linux/mutex.h>
+#include <linux/ism.h>
 
 #include "smc.h"
 
@@ -94,4 +95,24 @@ static inline bool smc_ism_is_loopback(struct smcd_dev *smcd)
 	return (smcd->ops->get_chid(smcd) == 0xFFFF);
 }
 
+static inline void copy_to_smcdgid(struct smcd_gid *sgid, uuid_t *igid)
+{
+	__be64 temp;
+
+	memcpy(&temp, igid, sizeof(sgid->gid));
+	sgid->gid = ntohll(temp);
+	memcpy(&temp, igid + sizeof(sgid->gid), sizeof(sgid->gid_ext));
+	sgid->gid_ext = ntohll(temp);
+}
+
+static inline void copy_to_ismgid(uuid_t *igid, struct smcd_gid *sgid)
+{
+	__be64 temp;
+
+	temp = htonll(sgid->gid);
+	memcpy(igid, &temp, sizeof(sgid->gid));
+	temp = htonll(sgid->gid_ext);
+	memcpy(igid + sizeof(sgid->gid), &temp, sizeof(sgid->gid_ext));
+}
+
 #endif
Signature · 1 line
2.45.2
reviewer
#6 lore raw

Re: [RFC net-next 3/7] net/ism: Use uuid_t for ISM GID

Quoted 28 lines · Alexandra Winter

SMC uses 64 Bit and 128 Bit Global Identifiers (GIDs) that need to be sent via the SMC protocol.
When integers are used network endianness and host endianness need to be considered.

Avoid this in the ISM layer by using uuid_t byte arrays. Follow on patches could do the same change for SMC, for now conversion helper functions are introduced.

ISM-vPCI devices provide 64 Bit GIDs. Map them to ISM uuid_t GIDs like this:
_________________________________________
| 64 Bit ISM-vPCI GID | 00000000_00000000 |
-----------------------------------------
If interpreted as UUID, this would be interpreted as th UIID variant, that is reserved for NCS backward compatibility. So it will not collide with UUIDs that were generated according to the standard.

Future ISM devices, shall use real UUIDs as 128 Bit GIDs.

Note:
- In this RFC patch smcd_gid is now moved back to smc.h, future patchset should avoid that.
- ism_dmb and ism_event structs still contain 64 Bit rgid and info fields. A future patch could change them to uuid_t gids. This does not break anything, because ism_loopback does not use them.

...

Quoted 13 lines
net/smc/smc_ism.h +1 -0
index 6763133dd8d0..d041e5a7c459 100644
@@ -12,6 +12,7 @@
 #include <linux/uio.h>
 #include <linux/types.h>
 #include <linux/mutex.h>
+#include <linux/ism.h>
 
 #include "smc.h"
 

net/smc/smc_ism.h static inline bool smc_ism_is_loopback(struct smcd_dev *smcd) · line 104

Quoted 10 lines
net/smc/smc_ism.h +7 -0
@@ -94,4 +95,24 @@ static inline bool smc_ism_is_loopback(struct smcd_dev *smcd)
 	return (smcd->ops->get_chid(smcd) == 0xFFFF);
 }
 
+static inline void copy_to_smcdgid(struct smcd_gid *sgid, uuid_t *igid)
+{
+	__be64 temp;
+
+	memcpy(&temp, igid, sizeof(sgid->gid));
+	sgid->gid = ntohll(temp);
+	memcpy(&temp, igid + sizeof(sgid->gid), sizeof(sgid->gid_ext));

Hi Alexandra,

The stride of the pointer arithmetic is the width of igid so this write will be at an offset of:

sizeof(igid) + sizeof(sgid->gid) = 128 bytes

Which is beyond the end of *igid.

I think the desired operation is to write at an offset of 8 bytes, so perhaps this is a way to achieve that, as the bi field is a 16 byte array of u8:

	memcpy(&temp, igid->b + sizeof(sgid->gid), sizeof(sgid->gid_ext));

Flagged by W=1 builds with gcc-14 and clang-19, and by Smatch.

net/smc/smc_ism.h static inline bool smc_ism_is_loopback(struct smcd_dev *smcd) · line 115

Quoted 11 lines
net/smc/smc_ism.h +11 -0
+	sgid->gid_ext = ntohll(temp);
+}
+
+static inline void copy_to_ismgid(uuid_t *igid, struct smcd_gid *sgid)
+{
+	__be64 temp;
+
+	temp = htonll(sgid->gid);
+	memcpy(igid, &temp, sizeof(sgid->gid));
+	temp = htonll(sgid->gid_ext);
+	memcpy(igid + sizeof(sgid->gid), &temp, sizeof(sgid->gid_ext));

I believe there is a similar problem here too.

Quoted 6 lines
net/smc/smc_ism.h +2 -0
+}
+
 #endif
Signature · 1 line
2.45.2
author maintainer
#7 lore raw

Re: [RFC net-next 3/7] net/ism: Use uuid_t for ISM GID

Quoted 9 lines · Simon Horman

net/smc/smc_ism.h static inline bool smc_ism_is_loopback(struct smcd_dev *smcd) · line 104

Quoted 7 lines
net/smc/smc_ism.h +7 -0
+static inline void copy_to_smcdgid(struct smcd_gid *sgid, uuid_t *igid)
+{
+	__be64 temp;
+
+	memcpy(&temp, igid, sizeof(sgid->gid));
+	sgid->gid = ntohll(temp);
+	memcpy(&temp, igid + sizeof(sgid->gid), sizeof(sgid->gid_ext));

Hi Alexandra,

Quoted 6 lines

The stride of the pointer arithmetic is the width of igid so this write will be at an offset of:

   sizeof(igid) + sizeof(sgid->gid) = 128 bytes

Which is beyond the end of *igid.

Duh, what a stupid mistake. Thank you.

Quoted 5 lines

I think the desired operation is to write at an offset of 8 bytes, so perhaps this is a way to achieve that, as the bi field is a 16 byte array of u8:

	memcpy(&temp, igid->b + sizeof(sgid->gid), sizeof(sgid->gid_ext));

I propose to keep the
memcpy(&temp, (u8 *)igid + sizeof(sgid->gid), sizeof(sgid->gid_ext)); like in the orginal net/smc/smc_loopback.c

Quoted 4 lines

Flagged by W=1 builds with gcc-14 and clang-19, and by Smatch.

Review

Quoted 2 lines
(quoted diff) +2 -0
+	sgid->gid_ext = ntohll(temp);
+}

I actually overlooked it in my smatch run (too many old warnings), but I cannot get W=1 to flag it. I'll try to improve my setup.

author maintainer
#8 lore raw

RFC 4/7 net/ism: Add kernel-doc comments for ism functions

In reply to Alexandra Winter

Note that in this RFC this patch is not complete, future versions of this patch need to contain comments for all ism_ops. Especially signal_event() and handle_event() need a good generic description.

---
 include/linux/ism.h | 115 ++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 105 insertions(+), 10 deletions(-)
include/linux/ism.h +105 -10
index 50975847248f..bc165d077071 100644
@@ -13,11 +13,26 @@
 #include <linux/workqueue.h>
 #include <linux/uuid.h>
 
-/* The remote peer rgid can use dmb_tok to write into this buffer. */
+/*
+ * DMB - Direct Memory Buffer
+ * ==========================
+ * An ism client provides an DMB as input buffer for a local receiving
+ * ism device for exactly one (remote) sending ism device. Only this
+ * sending device can send data into this DMB using move_data(). Sender
+ * and receiver can be the same device.
+ * TODO: Alignment and length rules (CPU and DMA). Device specific?
+ */
 struct ism_dmb {
+	/* dmb_tok - Token for this dmb
+	 * Used by remote sender to address this dmb.
+	 * Provided by ism fabric in register_dmb().
+	 * Unique per ism fabric.
+	 */
 	u64 dmb_tok;
+	/* rgid - GID of designated remote sending device */
 	u64 rgid;
 	u32 dmb_len;
+	/* sba_idx - Index of this DMB on this receiving device */
 	u32 sba_idx;
 	u32 vlan_valid;
 	u32 vlan_id;
@@ -25,6 +40,8 @@ struct ism_dmb {
 	dma_addr_t dma_addr;
 };
 
+/* ISM event structure (currently device type specific) */
+// TODO: Define and describe generic event properties
 struct ism_event {
 	u32 type;
 	u32 code;
@@ -33,38 +50,89 @@ struct ism_event {
 	u64 info;
 };
 
+//TODO: use enum typedef
 #define ISM_EVENT_DMB	0
 #define ISM_EVENT_GID	1
 #define ISM_EVENT_SWR	2
 
 struct ism_dev;
 
+/*
+ * ISM clients
+ * ===========
+ * All ism clients have access to all ism devices
+ * and must provide the following functions to be called by
+ * ism device drivers:
+ */
 struct ism_client {
+	/* client name for logging and debugging purposes */
 	const char *name;
+	/**
+	 *  add() - add an ism device
+	 *  @dev: device that was added
+	 *
+	 * Will be called during ism_register_client() for all existing
+	 * ism devices and whenever a new ism device is registered.
+	 * *dev is valid until ism_client->remove() is called.
+	 */
 	void (*add)(struct ism_dev *dev);
+	/**
+	 * remove() - remove an ism device
+	 * @dev: device to be removed
+	 *
+	 * Will be called whenever an ism device is unregistered.
+	 * Before this call the device is already inactive: It will
+	 * no longer call client handlers.
+	 * The client must not access *dev after this call.
+	 */
 	void (*remove)(struct ism_dev *dev);
+	/**
+	 * handle_event() - Handle control information sent by device
+	 * @dev: device reporting the event
+	 * @event: ism event structure
+	 */
 	void (*handle_event)(struct ism_dev *dev, struct ism_event *event);
-	/* Parameter dmbemask contains a bit vector with updated DMBEs, if sent
-	 * via ism_move_data(). Callback function must handle all active bits
-	 * indicated by dmbemask.
+	/**
+	 * handle_irq() - Handle signalling of a DMB
+	 * @dev: device owns the dmb
+	 * @bit: sba_idx=idx of the ism_dmb that got signalled
+	 *	TODO: Pass a priv pointer to ism_dmb instead of 'bit'(?)
+	 * @dmbemask: ism signalling mask of the dmb
+	 *
+	 * Handle signalling of a dmb that was registered by this client
+	 * for this device.
+	 * The ism device can coalesce multiple signalling triggers into a
+	 * single call of handle_irq(). dmbemask can be used to indicate
+	 * different kinds of triggers.
 	 */
 	void (*handle_irq)(struct ism_dev *dev, unsigned int bit, u16 dmbemask);
-	/* Private area - don't touch! */
+	/* client index - provided by ism layer */
 	u8 id;
 };
 
 int ism_register_client(struct ism_client *client);
 int  ism_unregister_client(struct ism_client *client);
 
+//TODO: Pair descriptions with functions
+/*
+ * ISM devices
+ * ===========
+ */
 /* Mandatory operations for all ism devices:
  * int (*query_remote_gid)(struct ism_dev *dev, uuid_t *rgid,
  *	                   u32 vid_valid, u32 vid);
  *	Query whether remote GID rgid is reachable via this device and this
  *	vlan id. Vlan id is only checked if vid_valid != 0.
+ *	Returns 0 if remote gid is reachable.
  *
  * int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb,
  *			    void *client);
- *	Register an ism_dmb buffer for this device and this client.
+ *	Allocate and register an ism_dmb buffer for this device and this client.
+ *	The following fields of ism_dmb must be valid:
+ *	rgid, dmb_len, vlan_*; Optionally:requested sba_idx (non-zero)
+ *	Upon return the following fields will be valid: dmb_tok, sba_idx
+ *		cpu_addr, dma_addr (if applicable)
+ *	Returns zero on success
  *
  * int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb);
  *	Unregister an ism_dmb buffer
@@ -81,10 +149,15 @@ int  ism_unregister_client(struct ism_client *client);
  * u16 (*get_chid)(struct ism_dev *dev);
  *	Returns ism fabric identifier (channel id) of this device.
  *	Only devices on the same ism fabric can communicate.
- *	chid is unique per HW system, except for 0xFFFF, which denotes
- *	an ism_loopback device that can only communicate with itself.
- *	Use chid for fast negative checks, but only query_remote_gid()
- *	can give a reliable positive answer.
+ *	chid is unique per HW system. Use chid for fast negative checks,
+ *	but only query_remote_gid() can give a reliable positive answer:
+ *	Different chid: ism is not possible
+ *	Same chid: ism traffic may be possible or not
+ *		   (e.g. different HW systems)
+ *	EXCEPTION: A value of 0xFFFF denotes an ism_loopback device
+ *		that can only communicate with itself. Use GID or
+ *		query_remote_gid()to determine whether sender and
+ *		receiver use the same ism_loopback device.
  *
  * struct device* (*get_dev)(struct ism_dev *dev);
  *
@@ -109,6 +182,28 @@ struct ism_ops {
 	int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb,
 			    struct ism_client *client);
 	int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb);
+	/**
+	 * move_data() - write into a remote dmb
+	 * @dev: Local sending ism device
+	 * @dmb_tok: Token of the remote dmb
+	 * @idx: signalling index
+	 * @sf: signalling flag;
+	 *      if true, idx will be turned on at target ism interrupt mask
+	 *      and target device will be signalled, if required.
+	 * @offset: offset within target dmb
+	 * @data: pointer to data to be sent
+	 * @size: length of data to be sent
+	 *
+	 * Use dev to write data of size at offset into a remote dmb
+	 * identified by dmb_tok. Data is moved synchronously, *data can
+	 * be freed when this function returns.
+	 *
+	 * If signalling flag (sf) is true, bit number idx bit will be
+	 * turned on in the ism signalling mask, that belongs to the
+	 * target dmb, and handle_irq() of the ism client that owns this
+	 * dmb will be called, if required. The target device may chose to
+	 * coalesce multiple signalling triggers.
+	 */
 	int (*move_data)(struct ism_dev *dev, u64 dmb_tok, unsigned int idx,
 			 bool sf, unsigned int offset, void *data,
 			 unsigned int size);
Signature · 1 line
2.45.2
#9 lore raw

Re: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions

Quoted 4 lines · On Wed, 15 Jan 2025 20:55:24 +0100 Alexandra Winter <[email protected]>…

Note that in this RFC this patch is not complete, future versions of this patch need to contain comments for all ism_ops. Especially signal_event() and handle_event() need a good generic description.

Such notes don't belong to the commit message, but to the inline cover letter IMHO.

Quoted 3 lines

---

maintainer
#10 lore raw

Re: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions

In reply to Alexandra Winter
Quoted 175 lines · Alexandra Winter

Note that in this RFC this patch is not complete, future versions of this patch need to contain comments for all ism_ops. Especially signal_event() and handle_event() need a good generic description.

---
include/linux/ism.h | 115 ++++++++++++++++++++++++++++++++++++++++----

1 file changed, 105 insertions(+), 10 deletions(-)
include/linux/ism.h +88 -10
index 50975847248f..bc165d077071 100644
@@ -13,11 +13,26 @@
 #include <linux/workqueue.h>
 #include <linux/uuid.h>
 
-/* The remote peer rgid can use dmb_tok to write into this buffer. */
+/*
+ * DMB - Direct Memory Buffer
+ * ==========================
+ * An ism client provides an DMB as input buffer for a local receiving
+ * ism device for exactly one (remote) sending ism device. Only this
+ * sending device can send data into this DMB using move_data(). Sender
+ * and receiver can be the same device.
+ * TODO: Alignment and length rules (CPU and DMA). Device specific?
+ */
 struct ism_dmb {
+	/* dmb_tok - Token for this dmb
+	 * Used by remote sender to address this dmb.
+	 * Provided by ism fabric in register_dmb().
+	 * Unique per ism fabric.
+	 */
 	u64 dmb_tok;
+	/* rgid - GID of designated remote sending device */
 	u64 rgid;
 	u32 dmb_len;
+	/* sba_idx - Index of this DMB on this receiving device */
 	u32 sba_idx;
 	u32 vlan_valid;
 	u32 vlan_id;
@@ -25,6 +40,8 @@ struct ism_dmb {
 	dma_addr_t dma_addr;
 };
 
+/* ISM event structure (currently device type specific) */
+// TODO: Define and describe generic event properties
 struct ism_event {
 	u32 type;
 	u32 code;
@@ -33,38 +50,89 @@ struct ism_event {
 	u64 info;
 };
 
+//TODO: use enum typedef
 #define ISM_EVENT_DMB	0
 #define ISM_EVENT_GID	1
 #define ISM_EVENT_SWR	2
 
 struct ism_dev;
 
+/*
+ * ISM clients
+ * ===========
+ * All ism clients have access to all ism devices
+ * and must provide the following functions to be called by
+ * ism device drivers:
+ */
 struct ism_client {
+	/* client name for logging and debugging purposes */
 	const char *name;
+	/**
+	 *  add() - add an ism device
+	 *  @dev: device that was added
+	 *
+	 * Will be called during ism_register_client() for all existing
+	 * ism devices and whenever a new ism device is registered.
+	 * *dev is valid until ism_client->remove() is called.
+	 */
 	void (*add)(struct ism_dev *dev);
+	/**
+	 * remove() - remove an ism device
+	 * @dev: device to be removed
+	 *
+	 * Will be called whenever an ism device is unregistered.
+	 * Before this call the device is already inactive: It will
+	 * no longer call client handlers.
+	 * The client must not access *dev after this call.
+	 */
 	void (*remove)(struct ism_dev *dev);
+	/**
+	 * handle_event() - Handle control information sent by device
+	 * @dev: device reporting the event
+	 * @event: ism event structure
+	 */
 	void (*handle_event)(struct ism_dev *dev, struct ism_event *event);
-	/* Parameter dmbemask contains a bit vector with updated DMBEs, if sent
-	 * via ism_move_data(). Callback function must handle all active bits
-	 * indicated by dmbemask.
+	/**
+	 * handle_irq() - Handle signalling of a DMB
+	 * @dev: device owns the dmb
+	 * @bit: sba_idx=idx of the ism_dmb that got signalled
+	 *	TODO: Pass a priv pointer to ism_dmb instead of 'bit'(?)
+	 * @dmbemask: ism signalling mask of the dmb
+	 *
+	 * Handle signalling of a dmb that was registered by this client
+	 * for this device.
+	 * The ism device can coalesce multiple signalling triggers into a
+	 * single call of handle_irq(). dmbemask can be used to indicate
+	 * different kinds of triggers.
 	 */
 	void (*handle_irq)(struct ism_dev *dev, unsigned int bit, u16 dmbemask);
-	/* Private area - don't touch! */
+	/* client index - provided by ism layer */
 	u8 id;
 };
 
 int ism_register_client(struct ism_client *client);
 int  ism_unregister_client(struct ism_client *client);
 
+//TODO: Pair descriptions with functions
+/*
+ * ISM devices
+ * ===========
+ */
 /* Mandatory operations for all ism devices:
 * int (*query_remote_gid)(struct ism_dev *dev, uuid_t *rgid,
 *	                   u32 vid_valid, u32 vid);
 *	Query whether remote GID rgid is reachable via this device and this
 *	vlan id. Vlan id is only checked if vid_valid != 0.
+ *	Returns 0 if remote gid is reachable.
 *
 * int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb,
 *			    void *client);
- *	Register an ism_dmb buffer for this device and this client.
+ *	Allocate and register an ism_dmb buffer for this device and this client.
+ *	The following fields of ism_dmb must be valid:
+ *	rgid, dmb_len, vlan_*; Optionally:requested sba_idx (non-zero)
+ *	Upon return the following fields will be valid: dmb_tok, sba_idx
+ *		cpu_addr, dma_addr (if applicable)
+ *	Returns zero on success
 *
 * int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb);
 *	Unregister an ism_dmb buffer
@@ -81,10 +149,15 @@ int  ism_unregister_client(struct ism_client *client);
 * u16 (*get_chid)(struct ism_dev *dev);
 *	Returns ism fabric identifier (channel id) of this device.
 *	Only devices on the same ism fabric can communicate.
- *	chid is unique per HW system, except for 0xFFFF, which denotes
- *	an ism_loopback device that can only communicate with itself.
- *	Use chid for fast negative checks, but only query_remote_gid()
- *	can give a reliable positive answer.
+ *	chid is unique per HW system. Use chid for fast negative checks,
+ *	but only query_remote_gid() can give a reliable positive answer:
+ *	Different chid: ism is not possible
+ *	Same chid: ism traffic may be possible or not
+ *		   (e.g. different HW systems)
+ *	EXCEPTION: A value of 0xFFFF denotes an ism_loopback device
+ *		that can only communicate with itself. Use GID or
+ *		query_remote_gid()to determine whether sender and
+ *		receiver use the same ism_loopback device.
 *
 * struct device* (*get_dev)(struct ism_dev *dev);
 *
@@ -109,6 +182,28 @@ struct ism_ops {
 	int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb,
 			    struct ism_client *client);
 	int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb);
+	/**
+	 * move_data() - write into a remote dmb
+	 * @dev: Local sending ism device
+	 * @dmb_tok: Token of the remote dmb
+	 * @idx: signalling index

include/linux/ism.h struct ism_ops { · line 199

Quoted 10 lines
include/linux/ism.h +10 -0
+	 * @sf: signalling flag;
+	 *      if true, idx will be turned on at target ism interrupt mask
+	 *      and target device will be signalled, if required.
+	 * @offset: offset within target dmb
+	 * @data: pointer to data to be sent
+	 * @size: length of data to be sent
+	 *
+	 * Use dev to write data of size at offset into a remote dmb
+	 * identified by dmb_tok. Data is moved synchronously, *data can
+	 * be freed when this function returns.

When considering the API, I found this comment may be incorrect.

IIUC, in copy mode for PCI ISM devices, the CPU only tells the device to perform a DMA copy. As a result, when this function returns, the device may not have completed the DMA copy.

In zero-copy mode for loopback, the source and destination share the same buffer. If the source rewrites the buffer, the destination may encounter corrupted data. The source should only reuse the data after the destination has finished reading it.

Best regards,
Dust

Quoted 13 lines
include/linux/ism.h +7 -0
+	 *
+	 * If signalling flag (sf) is true, bit number idx bit will be
+	 * turned on in the ism signalling mask, that belongs to the
+	 * target dmb, and handle_irq() of the ism client that owns this
+	 * dmb will be called, if required. The target device may chose to
+	 * coalesce multiple signalling triggers.
+	 */
 	int (*move_data)(struct ism_dev *dev, u64 dmb_tok, unsigned int idx,
 			 bool sf, unsigned int offset, void *data,
 			 unsigned int size);
Signature · 1 line
2.45.2
author maintainer
#11 lore raw

Re: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions

Quoted 17 lines · Dust Li
Quoted 5 lines
include/linux/ism.h +5 -0
+	/**
+	 * move_data() - write into a remote dmb
+	 * @dev: Local sending ism device
+	 * @dmb_tok: Token of the remote dmb
+	 * @idx: signalling index

include/linux/ism.h struct ism_ops { · line 199

Quoted 10 lines
include/linux/ism.h +10 -0
+	 * @sf: signalling flag;
+	 *      if true, idx will be turned on at target ism interrupt mask
+	 *      and target device will be signalled, if required.
+	 * @offset: offset within target dmb
+	 * @data: pointer to data to be sent
+	 * @size: length of data to be sent
+	 *
+	 * Use dev to write data of size at offset into a remote dmb
+	 * identified by dmb_tok. Data is moved synchronously, *data can
+	 * be freed when this function returns.

When considering the API, I found this comment may be incorrect.

Quoted 4 lines

IIUC, in copy mode for PCI ISM devices, the CPU only tells the device to perform a DMA copy. As a result, when this function returns, the device may not have completed the DMA copy.

No, it is actually one of the properties of ISM vPCI that the data is moved synchronously inside the move_data() function. (on PCI layer the data is moved inside the __zpci_store_block() command). Obviously for loopback move_data() is also synchornous.

SMC-D does not make use of it, instead they re-use the same conn->sndbuf_desc for the lifetime of a connection.

Quoted 5 lines

In zero-copy mode for loopback, the source and destination share the same buffer. If the source rewrites the buffer, the destination may encounter corrupted data. The source should only reuse the data after the destination has finished reading it.

That is true independent of the question, whether the move is synchronous or not.
It is the clients' responsibility to make sure a sender does not overwrite unread data. SMC uses the write-pointers and read-pointer for that.

Quoted 14 lines

Best regards,
Dust

Quoted 11 lines
include/linux/ism.h +7 -0
+	 *
+	 * If signalling flag (sf) is true, bit number idx bit will be
+	 * turned on in the ism signalling mask, that belongs to the
+	 * target dmb, and handle_irq() of the ism client that owns this
+	 * dmb will be called, if required. The target device may chose to
+	 * coalesce multiple signalling triggers.
+	 */

int (*move_data)(struct ism_dev *dev, u64 dmb_tok, unsigned int idx,
bool sf, unsigned int offset, void *data, unsigned int size);

#12 lore raw

Re: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions

Quoted 25 lines · Alexandra Winter
Quoted 17 lines · Dust Li
Quoted 5 lines
include/linux/ism.h +5 -0
+	/**
+	 * move_data() - write into a remote dmb
+	 * @dev: Local sending ism device
+	 * @dmb_tok: Token of the remote dmb
+	 * @idx: signalling index

include/linux/ism.h struct ism_ops { · line 199

Quoted 10 lines
include/linux/ism.h +10 -0
+	 * @sf: signalling flag;
+	 *      if true, idx will be turned on at target ism interrupt mask
+	 *      and target device will be signalled, if required.
+	 * @offset: offset within target dmb
+	 * @data: pointer to data to be sent
+	 * @size: length of data to be sent
+	 *
+	 * Use dev to write data of size at offset into a remote dmb
+	 * identified by dmb_tok. Data is moved synchronously, *data can
+	 * be freed when this function returns.

When considering the API, I found this comment may be incorrect.

Quoted 4 lines

IIUC, in copy mode for PCI ISM devices, the CPU only tells the device to perform a DMA copy. As a result, when this function returns, the device may not have completed the DMA copy.

Quoted 4 lines

No, it is actually one of the properties of ISM vPCI that the data is moved synchronously inside the move_data() function. (on PCI layer the data is moved inside the __zpci_store_block() command). Obviously for loopback move_data() is also synchornous.

That is true for the IBM ISM vPCI device but maybe we should design the API also for future PCI devices that do not move data synchronously.

Quoted 32 lines

SMC-D does not make use of it, instead they re-use the same conn->sndbuf_desc for the lifetime of a connection.

Quoted 5 lines

In zero-copy mode for loopback, the source and destination share the same buffer. If the source rewrites the buffer, the destination may encounter corrupted data. The source should only reuse the data after the destination has finished reading it.

That is true independent of the question, whether the move is synchronous or not.
It is the clients' responsibility to make sure a sender does not overwrite unread data. SMC uses the write-pointers and read-pointer for that.

Quoted 14 lines

Best regards,
Dust

Quoted 11 lines
include/linux/ism.h +7 -0
+	 *
+	 * If signalling flag (sf) is true, bit number idx bit will be
+	 * turned on in the ism signalling mask, that belongs to the
+	 * target dmb, and handle_irq() of the ism client that owns this
+	 * dmb will be called, if required. The target device may chose to
+	 * coalesce multiple signalling triggers.
+	 */

int (*move_data)(struct ism_dev *dev, u64 dmb_tok, unsigned int idx,
bool sf, unsigned int offset, void *data, unsigned int size);

author maintainer
#13 lore raw

Re: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions

Quoted 26 lines · Julian Ruess
Quoted 25 lines · Alexandra Winter
Quoted 21 lines · Dust Li
Quoted 5 lines
include/linux/ism.h +5 -0
+	/**
+	 * move_data() - write into a remote dmb
+	 * @dev: Local sending ism device
+	 * @dmb_tok: Token of the remote dmb
+	 * @idx: signalling index

include/linux/ism.h struct ism_ops { · line 199

Quoted 10 lines
include/linux/ism.h +10 -0
+	 * @sf: signalling flag;
+	 *      if true, idx will be turned on at target ism interrupt mask
+	 *      and target device will be signalled, if required.
+	 * @offset: offset within target dmb
+	 * @data: pointer to data to be sent
+	 * @size: length of data to be sent
+	 *
+	 * Use dev to write data of size at offset into a remote dmb
+	 * identified by dmb_tok. Data is moved synchronously, *data can
+	 * be freed when this function returns.

When considering the API, I found this comment may be incorrect.

IIUC, in copy mode for PCI ISM devices, the CPU only tells the device to perform a DMA copy. As a result, when this function returns, the device may not have completed the DMA copy.

Quoted 9 lines
Quoted 4 lines

No, it is actually one of the properties of ISM vPCI that the data is moved synchronously inside the move_data() function. (on PCI layer the data is moved inside the __zpci_store_block() command). Obviously for loopback move_data() is also synchornous.

That is true for the IBM ISM vPCI device but maybe we should design the API also for future PCI devices that do not move data synchronously.

An API should always be extendable

Quoted 33 lines
Quoted 32 lines

SMC-D does not make use of it, instead they re-use the same conn->sndbuf_desc for the lifetime of a connection.

Quoted 5 lines

In zero-copy mode for loopback, the source and destination share the same buffer. If the source rewrites the buffer, the destination may encounter corrupted data. The source should only reuse the data after the destination has finished reading it.

That is true independent of the question, whether the move is synchronous or not.
It is the clients' responsibility to make sure a sender does not overwrite unread data. SMC uses the write-pointers and read-pointer for that.

Quoted 14 lines

Best regards,
Dust

Quoted 11 lines
include/linux/ism.h +7 -0
+	 *
+	 * If signalling flag (sf) is true, bit number idx bit will be
+	 * turned on in the ism signalling mask, that belongs to the
+	 * target dmb, and handle_irq() of the ism client that owns this
+	 * dmb will be called, if required. The target device may chose to
+	 * coalesce multiple signalling triggers.
+	 */

int (*move_data)(struct ism_dev *dev, u64 dmb_tok, unsigned int idx,
bool sf, unsigned int offset, void *data, unsigned int size);

#14 lore raw

Re: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions

In reply to Dust Li
Quoted 189 lines · Dust Li
Quoted 175 lines · Alexandra Winter

Note that in this RFC this patch is not complete, future versions of this patch need to contain comments for all ism_ops. Especially signal_event() and handle_event() need a good generic description.

---
include/linux/ism.h | 115 ++++++++++++++++++++++++++++++++++++++++----

1 file changed, 105 insertions(+), 10 deletions(-)
include/linux/ism.h +88 -10
index 50975847248f..bc165d077071 100644
@@ -13,11 +13,26 @@
 #include <linux/workqueue.h>
 #include <linux/uuid.h>
 
-/* The remote peer rgid can use dmb_tok to write into this buffer. */
+/*
+ * DMB - Direct Memory Buffer
+ * ==========================
+ * An ism client provides an DMB as input buffer for a local receiving
+ * ism device for exactly one (remote) sending ism device. Only this
+ * sending device can send data into this DMB using move_data(). Sender
+ * and receiver can be the same device.
+ * TODO: Alignment and length rules (CPU and DMA). Device specific?
+ */
 struct ism_dmb {
+	/* dmb_tok - Token for this dmb
+	 * Used by remote sender to address this dmb.
+	 * Provided by ism fabric in register_dmb().
+	 * Unique per ism fabric.
+	 */
 	u64 dmb_tok;
+	/* rgid - GID of designated remote sending device */
 	u64 rgid;
 	u32 dmb_len;
+	/* sba_idx - Index of this DMB on this receiving device */
 	u32 sba_idx;
 	u32 vlan_valid;
 	u32 vlan_id;
@@ -25,6 +40,8 @@ struct ism_dmb {
 	dma_addr_t dma_addr;
 };
 
+/* ISM event structure (currently device type specific) */
+// TODO: Define and describe generic event properties
 struct ism_event {
 	u32 type;
 	u32 code;
@@ -33,38 +50,89 @@ struct ism_event {
 	u64 info;
 };
 
+//TODO: use enum typedef
 #define ISM_EVENT_DMB	0
 #define ISM_EVENT_GID	1
 #define ISM_EVENT_SWR	2
 
 struct ism_dev;
 
+/*
+ * ISM clients
+ * ===========
+ * All ism clients have access to all ism devices
+ * and must provide the following functions to be called by
+ * ism device drivers:
+ */
 struct ism_client {
+	/* client name for logging and debugging purposes */
 	const char *name;
+	/**
+	 *  add() - add an ism device
+	 *  @dev: device that was added
+	 *
+	 * Will be called during ism_register_client() for all existing
+	 * ism devices and whenever a new ism device is registered.
+	 * *dev is valid until ism_client->remove() is called.
+	 */
 	void (*add)(struct ism_dev *dev);
+	/**
+	 * remove() - remove an ism device
+	 * @dev: device to be removed
+	 *
+	 * Will be called whenever an ism device is unregistered.
+	 * Before this call the device is already inactive: It will
+	 * no longer call client handlers.
+	 * The client must not access *dev after this call.
+	 */
 	void (*remove)(struct ism_dev *dev);
+	/**
+	 * handle_event() - Handle control information sent by device
+	 * @dev: device reporting the event
+	 * @event: ism event structure
+	 */
 	void (*handle_event)(struct ism_dev *dev, struct ism_event *event);
-	/* Parameter dmbemask contains a bit vector with updated DMBEs, if sent
-	 * via ism_move_data(). Callback function must handle all active bits
-	 * indicated by dmbemask.
+	/**
+	 * handle_irq() - Handle signalling of a DMB
+	 * @dev: device owns the dmb
+	 * @bit: sba_idx=idx of the ism_dmb that got signalled
+	 *	TODO: Pass a priv pointer to ism_dmb instead of 'bit'(?)
+	 * @dmbemask: ism signalling mask of the dmb
+	 *
+	 * Handle signalling of a dmb that was registered by this client
+	 * for this device.
+	 * The ism device can coalesce multiple signalling triggers into a
+	 * single call of handle_irq(). dmbemask can be used to indicate
+	 * different kinds of triggers.
 	 */
 	void (*handle_irq)(struct ism_dev *dev, unsigned int bit, u16 dmbemask);
-	/* Private area - don't touch! */
+	/* client index - provided by ism layer */
 	u8 id;
 };
 
 int ism_register_client(struct ism_client *client);
 int  ism_unregister_client(struct ism_client *client);
 
+//TODO: Pair descriptions with functions
+/*
+ * ISM devices
+ * ===========
+ */
 /* Mandatory operations for all ism devices:
 * int (*query_remote_gid)(struct ism_dev *dev, uuid_t *rgid,
 *	                   u32 vid_valid, u32 vid);
 *	Query whether remote GID rgid is reachable via this device and this
 *	vlan id. Vlan id is only checked if vid_valid != 0.
+ *	Returns 0 if remote gid is reachable.
 *
 * int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb,
 *			    void *client);
- *	Register an ism_dmb buffer for this device and this client.
+ *	Allocate and register an ism_dmb buffer for this device and this client.
+ *	The following fields of ism_dmb must be valid:
+ *	rgid, dmb_len, vlan_*; Optionally:requested sba_idx (non-zero)
+ *	Upon return the following fields will be valid: dmb_tok, sba_idx
+ *		cpu_addr, dma_addr (if applicable)
+ *	Returns zero on success
 *
 * int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb);
 *	Unregister an ism_dmb buffer
@@ -81,10 +149,15 @@ int  ism_unregister_client(struct ism_client *client);
 * u16 (*get_chid)(struct ism_dev *dev);
 *	Returns ism fabric identifier (channel id) of this device.
 *	Only devices on the same ism fabric can communicate.
- *	chid is unique per HW system, except for 0xFFFF, which denotes
- *	an ism_loopback device that can only communicate with itself.
- *	Use chid for fast negative checks, but only query_remote_gid()
- *	can give a reliable positive answer.
+ *	chid is unique per HW system. Use chid for fast negative checks,
+ *	but only query_remote_gid() can give a reliable positive answer:
+ *	Different chid: ism is not possible
+ *	Same chid: ism traffic may be possible or not
+ *		   (e.g. different HW systems)
+ *	EXCEPTION: A value of 0xFFFF denotes an ism_loopback device
+ *		that can only communicate with itself. Use GID or
+ *		query_remote_gid()to determine whether sender and
+ *		receiver use the same ism_loopback device.
 *
 * struct device* (*get_dev)(struct ism_dev *dev);
 *
@@ -109,6 +182,28 @@ struct ism_ops {
 	int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb,
 			    struct ism_client *client);
 	int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb);
+	/**
+	 * move_data() - write into a remote dmb
+	 * @dev: Local sending ism device
+	 * @dmb_tok: Token of the remote dmb
+	 * @idx: signalling index

include/linux/ism.h struct ism_ops { · line 199

Quoted 10 lines
include/linux/ism.h +10 -0
+	 * @sf: signalling flag;
+	 *      if true, idx will be turned on at target ism interrupt mask
+	 *      and target device will be signalled, if required.
+	 * @offset: offset within target dmb
+	 * @data: pointer to data to be sent
+	 * @size: length of data to be sent
+	 *
+	 * Use dev to write data of size at offset into a remote dmb
+	 * identified by dmb_tok. Data is moved synchronously, *data can
+	 * be freed when this function returns.

When considering the API, I found this comment may be incorrect.

Quoted 3 lines

IIUC, in copy mode for PCI ISM devices, the CPU only tells the device to perform a DMA copy. As a result, when this function returns, the device may not have completed the DMA copy.

For the s390 ISM device the statement is true. The move_data() function does a PCI Store Block instruction which is both the write on the sender side but also synchronously acts as the devices DMA write on the receiver side. So when the PCI Store Block instruction completes the data has been cache coherently written to the receiver DMB. And yes full synchronicity would be impossible with the posted writes of real PCIe.

That said when it comes to API design I think you have a great point here in that we need to decide if this synchronicity should be baked into the move_data() API. I think we instead want to only guarantee a weaker rule. That is the source buffer can be re-used after the move. This to me is also aligned with the word "move" here in that the data has been moved after the call not registered to be moved or such. This could be achieved with a real PCIe device by copying the data or by waiting on completion. If we ever get devices which need to wait on completion it may indeed be better to have a separate completion step in the API too. Then again I think the concept of having a single "move data" step is somewhat central to ISM and I'd hate to lose that simplicity.

I've been thinking also about a possible copy mode in a virtio-ism. That could be useful if we wanted to use virtio-ism between memory partitioned guests, or if one wanted to transparently proxy virtio-ism over s390 ISM to span multiple KVM hosts. And I think such a mode could still work with a single "move data" step and I'd love to have that in any future virtio-ism spec.

Quoted 5 lines

In zero-copy mode for loopback, the source and destination share the same buffer. If the source rewrites the buffer, the destination may encounter corrupted data. The source should only reuse the data after the destination has finished reading it.

I think there are two potential overwrite scenarios here.

1. The sender re-uses the source data buffer i.e. the @data buffer of the move_data() call. On s390 ISM this is fine because the data was copied out and into the destination DMB during the call. This could typically become an issue if the device DMA reads directly from @data after the move_data() call completed.

2. The sender does subsequent move_data() overwriting data in the destination DMB before the receiver has read the data. This can happen on s390 ISM too and needs to be prevented by DMB access rules.

For the move_data() call I think that even in a "shared i.e. same page DMB" scenario move_data() must still do a copy out of the @data buffer into the shared DMB. Otherwise it really wouldn't "move" data and it would be a very weird API since @data is just a buffer not some kind of descriptor. In other words I think scenario 1 shouldn't be possible in either copy or shared DMB mode by the semantics of move_data().

Quoted 14 lines
Quoted 14 lines
include/linux/ism.h +7 -0
+	 *
+	 * If signalling flag (sf) is true, bit number idx bit will be
+	 * turned on in the ism signalling mask, that belongs to the
+	 * target dmb, and handle_irq() of the ism client that owns this
+	 * dmb will be called, if required. The target device may chose to
+	 * coalesce multiple signalling triggers.
+	 */
 	int (*move_data)(struct ism_dev *dev, u64 dmb_tok, unsigned int idx,
 			 bool sf, unsigned int offset, void *data,
 			 unsigned int size);
Signature · 1 line
2.45.2
maintainer
#15 lore raw

Re: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions

Quoted 194 lines · Niklas Schnelle
Quoted 189 lines · Dust Li
Quoted 175 lines · Alexandra Winter

Note that in this RFC this patch is not complete, future versions of this patch need to contain comments for all ism_ops. Especially signal_event() and handle_event() need a good generic description.

---
include/linux/ism.h | 115 ++++++++++++++++++++++++++++++++++++++++----

1 file changed, 105 insertions(+), 10 deletions(-)
include/linux/ism.h +88 -10
index 50975847248f..bc165d077071 100644
@@ -13,11 +13,26 @@
 #include <linux/workqueue.h>
 #include <linux/uuid.h>
 
-/* The remote peer rgid can use dmb_tok to write into this buffer. */
+/*
+ * DMB - Direct Memory Buffer
+ * ==========================
+ * An ism client provides an DMB as input buffer for a local receiving
+ * ism device for exactly one (remote) sending ism device. Only this
+ * sending device can send data into this DMB using move_data(). Sender
+ * and receiver can be the same device.
+ * TODO: Alignment and length rules (CPU and DMA). Device specific?
+ */
 struct ism_dmb {
+	/* dmb_tok - Token for this dmb
+	 * Used by remote sender to address this dmb.
+	 * Provided by ism fabric in register_dmb().
+	 * Unique per ism fabric.
+	 */
 	u64 dmb_tok;
+	/* rgid - GID of designated remote sending device */
 	u64 rgid;
 	u32 dmb_len;
+	/* sba_idx - Index of this DMB on this receiving device */
 	u32 sba_idx;
 	u32 vlan_valid;
 	u32 vlan_id;
@@ -25,6 +40,8 @@ struct ism_dmb {
 	dma_addr_t dma_addr;
 };
 
+/* ISM event structure (currently device type specific) */
+// TODO: Define and describe generic event properties
 struct ism_event {
 	u32 type;
 	u32 code;
@@ -33,38 +50,89 @@ struct ism_event {
 	u64 info;
 };
 
+//TODO: use enum typedef
 #define ISM_EVENT_DMB	0
 #define ISM_EVENT_GID	1
 #define ISM_EVENT_SWR	2
 
 struct ism_dev;
 
+/*
+ * ISM clients
+ * ===========
+ * All ism clients have access to all ism devices
+ * and must provide the following functions to be called by
+ * ism device drivers:
+ */
 struct ism_client {
+	/* client name for logging and debugging purposes */
 	const char *name;
+	/**
+	 *  add() - add an ism device
+	 *  @dev: device that was added
+	 *
+	 * Will be called during ism_register_client() for all existing
+	 * ism devices and whenever a new ism device is registered.
+	 * *dev is valid until ism_client->remove() is called.
+	 */
 	void (*add)(struct ism_dev *dev);
+	/**
+	 * remove() - remove an ism device
+	 * @dev: device to be removed
+	 *
+	 * Will be called whenever an ism device is unregistered.
+	 * Before this call the device is already inactive: It will
+	 * no longer call client handlers.
+	 * The client must not access *dev after this call.
+	 */
 	void (*remove)(struct ism_dev *dev);
+	/**
+	 * handle_event() - Handle control information sent by device
+	 * @dev: device reporting the event
+	 * @event: ism event structure
+	 */
 	void (*handle_event)(struct ism_dev *dev, struct ism_event *event);
-	/* Parameter dmbemask contains a bit vector with updated DMBEs, if sent
-	 * via ism_move_data(). Callback function must handle all active bits
-	 * indicated by dmbemask.
+	/**
+	 * handle_irq() - Handle signalling of a DMB
+	 * @dev: device owns the dmb
+	 * @bit: sba_idx=idx of the ism_dmb that got signalled
+	 *	TODO: Pass a priv pointer to ism_dmb instead of 'bit'(?)
+	 * @dmbemask: ism signalling mask of the dmb
+	 *
+	 * Handle signalling of a dmb that was registered by this client
+	 * for this device.
+	 * The ism device can coalesce multiple signalling triggers into a
+	 * single call of handle_irq(). dmbemask can be used to indicate
+	 * different kinds of triggers.
 	 */
 	void (*handle_irq)(struct ism_dev *dev, unsigned int bit, u16 dmbemask);
-	/* Private area - don't touch! */
+	/* client index - provided by ism layer */
 	u8 id;
 };
 
 int ism_register_client(struct ism_client *client);
 int  ism_unregister_client(struct ism_client *client);
 
+//TODO: Pair descriptions with functions
+/*
+ * ISM devices
+ * ===========
+ */
 /* Mandatory operations for all ism devices:
 * int (*query_remote_gid)(struct ism_dev *dev, uuid_t *rgid,
 *	                   u32 vid_valid, u32 vid);
 *	Query whether remote GID rgid is reachable via this device and this
 *	vlan id. Vlan id is only checked if vid_valid != 0.
+ *	Returns 0 if remote gid is reachable.
 *
 * int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb,
 *			    void *client);
- *	Register an ism_dmb buffer for this device and this client.
+ *	Allocate and register an ism_dmb buffer for this device and this client.
+ *	The following fields of ism_dmb must be valid:
+ *	rgid, dmb_len, vlan_*; Optionally:requested sba_idx (non-zero)
+ *	Upon return the following fields will be valid: dmb_tok, sba_idx
+ *		cpu_addr, dma_addr (if applicable)
+ *	Returns zero on success
 *
 * int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb);
 *	Unregister an ism_dmb buffer
@@ -81,10 +149,15 @@ int  ism_unregister_client(struct ism_client *client);
 * u16 (*get_chid)(struct ism_dev *dev);
 *	Returns ism fabric identifier (channel id) of this device.
 *	Only devices on the same ism fabric can communicate.
- *	chid is unique per HW system, except for 0xFFFF, which denotes
- *	an ism_loopback device that can only communicate with itself.
- *	Use chid for fast negative checks, but only query_remote_gid()
- *	can give a reliable positive answer.
+ *	chid is unique per HW system. Use chid for fast negative checks,
+ *	but only query_remote_gid() can give a reliable positive answer:
+ *	Different chid: ism is not possible
+ *	Same chid: ism traffic may be possible or not
+ *		   (e.g. different HW systems)
+ *	EXCEPTION: A value of 0xFFFF denotes an ism_loopback device
+ *		that can only communicate with itself. Use GID or
+ *		query_remote_gid()to determine whether sender and
+ *		receiver use the same ism_loopback device.
 *
 * struct device* (*get_dev)(struct ism_dev *dev);
 *
@@ -109,6 +182,28 @@ struct ism_ops {
 	int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb,
 			    struct ism_client *client);
 	int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb);
+	/**
+	 * move_data() - write into a remote dmb
+	 * @dev: Local sending ism device
+	 * @dmb_tok: Token of the remote dmb
+	 * @idx: signalling index

include/linux/ism.h struct ism_ops { · line 199

Quoted 10 lines
include/linux/ism.h +10 -0
+	 * @sf: signalling flag;
+	 *      if true, idx will be turned on at target ism interrupt mask
+	 *      and target device will be signalled, if required.
+	 * @offset: offset within target dmb
+	 * @data: pointer to data to be sent
+	 * @size: length of data to be sent
+	 *
+	 * Use dev to write data of size at offset into a remote dmb
+	 * identified by dmb_tok. Data is moved synchronously, *data can
+	 * be freed when this function returns.

When considering the API, I found this comment may be incorrect.

Quoted 3 lines

IIUC, in copy mode for PCI ISM devices, the CPU only tells the device to perform a DMA copy. As a result, when this function returns, the device may not have completed the DMA copy.

Quoted 7 lines

For the s390 ISM device the statement is true. The move_data() function does a PCI Store Block instruction which is both the write on the sender side but also synchronously acts as the devices DMA write on the receiver side. So when the PCI Store Block instruction completes the data has been cache coherently written to the receiver DMB. And yes full synchronicity would be impossible with the posted writes of real PCIe.

Thank you guys for the detail explaination on s390! I throught it was like normal PCI device. @Niklas, @Alexandra, @Julian

Quoted 13 lines

That said when it comes to API design I think you have a great point here in that we need to decide if this synchronicity should be baked into the move_data() API. I think we instead want to only guarantee a weaker rule. That is the source buffer can be re-used after the move. This to me is also aligned with the word "move" here in that the data has been moved after the call not registered to be moved or such. This could be achieved with a real PCIe device by copying the data or by waiting on completion. If we ever get devices which need to wait on completion it may indeed be better to have a separate completion step in the API too. Then again I think the concept of having a single "move data" step is somewhat central to ISM and I'd hate to lose that simplicity.

Agree.

For SMC, it already handles zero-copy and copy modes differently. In copy mode, the current API seems appropriate, but for zero-copy mode, we may need to extend it.

Taking the loopback device as an example, SMC requires the move_data() function to copy the CDC into the DMB. However, copying the data from the source to the destination DMB is not necessary. Instead, it needs to signal the peer that the data transfer is complete, so an API for notification is required.

One solution might be to retain the move_data() function and introduce something like a notify() API. Alternatively, if we don't want to extend the API, we could use something like move_data(ismdev, data=NULL, length=0, ..., signal=1).

Quoted 7 lines

I've been thinking also about a possible copy mode in a virtio-ism. That could be useful if we wanted to use virtio-ism between memory partitioned guests, or if one wanted to transparently proxy virtio-ism over s390 ISM to span multiple KVM hosts. And I think such a mode could still work with a single "move data" step and I'd love to have that in any future virtio-ism spec.

I'm not opposed to that.

Quoted 19 lines
Quoted 6 lines

In zero-copy mode for loopback, the source and destination share the same buffer. If the source rewrites the buffer, the destination may encounter corrupted data. The source should only reuse the data after the destination has finished reading it.

I think there are two potential overwrite scenarios here.

1. The sender re-uses the source data buffer i.e. the @data buffer of the move_data() call. On s390 ISM this is fine because the data was copied out and into the destination DMB during the call. This could typically become an issue if the device DMA reads directly from @data after the move_data() call completed.

2. The sender does subsequent move_data() overwriting data in the destination DMB before the receiver has read the data. This can happen on s390 ISM too and needs to be prevented by DMB access rules.

Quoted 6 lines

For the move_data() call I think that even in a "shared i.e. same page DMB" scenario move_data() must still do a copy out of the @data buffer into the shared DMB. Otherwise it really wouldn't "move" data and it would be a very weird API since @data is just a buffer not some kind of descriptor. In other words I think scenario 1 shouldn't be possible in either copy or shared DMB mode by the semantics of move_data().

For the shared DMB, I now agree with you. Perhaps we can keep the move_data() behavior consistent with memcpy(3) or memmove(3) and leave the usage to the upper layer.

Best regards,
Dust

author maintainer
#16 lore raw

RFC 5/7 net/ism: Move ism_loopback to net/ism

In reply to Alexandra Winter

The first stage of ism_loopback was implemented as part of the SMC module [1]. Now that we have the ism layer, provide access to the ism_loopback device to all ism clients.

Move ism_loopback.* from net/smc to net/ism.
The following changes are required to ism_loopback.c:
- Change ism_lo_move_data() to no longer schedule an smcd receive tasklet, but instead call ism_client->handle_irq().
Note: In this RFC patch ism_loppback is not fully generic.
The smc-d client uses attached buffers, for moves without signalling. and not-attached buffers for moves with signalling. ism_lo_move_data() must not rely on that assumption. ism_lo_move_data() must be able to handle more than one ism client.

In addition the following changes are required to unify ism_loopback and ism_vp:

In ism layer and ism_vpci:
ism_loopback is not backed by a pci device, so use dev instead of pdev in ism_dev.

In smc-d:
in smcd_alloc_dev():
- use kernel memory instead of device memory for smcd_dev and smcd->conn. An alternative would be to ask device to alloc the memory.
- use different smcd_ops and max_dmbs for ism_vp and ism_loopback. A future patch can change smc-d to directly use ism_ops instead of smcd_ops.
- use ism dev_name instead of pci dev name for ism_evt_wq name
- allocate an event workqueue for ism_loopback, although it currently does not generate events.

---
 drivers/s390/net/ism.h     |   6 +-
 drivers/s390/net/ism_drv.c |  31 ++-
 include/linux/ism.h        |  59 +++++
 include/net/smc.h          |   4 +-
 net/ism/Kconfig            |  13 ++
 net/ism/Makefile           |   1 +
 net/ism/ism_loopback.c     | 366 +++++++++++++++++++++++++++++++
 net/ism/ism_loopback.h     |  59 +++++
 net/ism/ism_main.c         |  11 +-
 net/smc/Kconfig            |  13 --
 net/smc/Makefile           |   1 -
 net/smc/af_smc.c           |  12 +-
 net/smc/smc_ism.c          | 108 +++++++---
 net/smc/smc_loopback.c     | 427 -------------------------------------
 net/smc/smc_loopback.h     |  60 ------
 15 files changed, 606 insertions(+), 565 deletions(-)
 create mode 100644 net/ism/ism_loopback.c
 create mode 100644 net/ism/ism_loopback.h
 delete mode 100644 net/smc/smc_loopback.c
 delete mode 100644 net/smc/smc_loopback.h
drivers/s390/net/ism.h +3 -3
index 61cf10334170..0deca6d0e328 100644
@@ -202,7 +202,7 @@ struct ism_sba {
 static inline void __ism_read_cmd(struct ism_dev *ism, void *data,
 				  unsigned long offset, unsigned long len)
 {
-	struct zpci_dev *zdev = to_zpci(ism->pdev);
+	struct zpci_dev *zdev = to_zpci(to_pci_dev(ism->dev.parent));
 	u64 req = ZPCI_CREATE_REQ(zdev->fh, 2, 8);
 
 	while (len > 0) {
@@ -216,7 +216,7 @@ static inline void __ism_read_cmd(struct ism_dev *ism, void *data,
 static inline void __ism_write_cmd(struct ism_dev *ism, void *data,
 				   unsigned long offset, unsigned long len)
 {
-	struct zpci_dev *zdev = to_zpci(ism->pdev);
+	struct zpci_dev *zdev = to_zpci(to_pci_dev(ism->dev.parent));
 	u64 req = ZPCI_CREATE_REQ(zdev->fh, 2, len);
 
 	if (len)
@@ -226,7 +226,7 @@ static inline void __ism_write_cmd(struct ism_dev *ism, void *data,
 static inline int __ism_move(struct ism_dev *ism, u64 dmb_req, void *data,
 			     unsigned int size)
 {
-	struct zpci_dev *zdev = to_zpci(ism->pdev);
+	struct zpci_dev *zdev = to_zpci(to_pci_dev(ism->dev.parent));
 	u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, size);
 
 	return __zpci_store_block(data, req, dmb_req);
drivers/s390/net/ism_drv.c +13 -18
index ab70debbdd9d..c0954d6dd9f5 100644
@@ -88,7 +88,7 @@ static int register_sba(struct ism_dev *ism)
 	dma_addr_t dma_handle;
 	struct ism_sba *sba;
 
-	sba = dma_alloc_coherent(&ism->pdev->dev, PAGE_SIZE, &dma_handle,
+	sba = dma_alloc_coherent(ism->dev.parent, PAGE_SIZE, &dma_handle,
 				 GFP_KERNEL);
 	if (!sba)
 		return -ENOMEM;
@@ -99,7 +99,7 @@ static int register_sba(struct ism_dev *ism)
 	cmd.request.sba = dma_handle;
 
 	if (ism_cmd(ism, &cmd)) {
-		dma_free_coherent(&ism->pdev->dev, PAGE_SIZE, sba, dma_handle);
+		dma_free_coherent(ism->dev.parent, PAGE_SIZE, sba, dma_handle);
 		return -EIO;
 	}
 
@@ -115,7 +115,7 @@ static int register_ieq(struct ism_dev *ism)
 	dma_addr_t dma_handle;
 	struct ism_eq *ieq;
 
-	ieq = dma_alloc_coherent(&ism->pdev->dev, PAGE_SIZE, &dma_handle,
+	ieq = dma_alloc_coherent(ism->dev.parent, PAGE_SIZE, &dma_handle,
 				 GFP_KERNEL);
 	if (!ieq)
 		return -ENOMEM;
@@ -127,7 +127,7 @@ static int register_ieq(struct ism_dev *ism)
 	cmd.request.len = sizeof(*ieq);
 
 	if (ism_cmd(ism, &cmd)) {
-		dma_free_coherent(&ism->pdev->dev, PAGE_SIZE, ieq, dma_handle);
+		dma_free_coherent(ism->dev.parent, PAGE_SIZE, ieq, dma_handle);
 		return -EIO;
 	}
 
@@ -149,7 +149,7 @@ static int unregister_sba(struct ism_dev *ism)
 	if (ret && ret != ISM_ERROR)
 		return -EIO;
 
-	dma_free_coherent(&ism->pdev->dev, PAGE_SIZE,
+	dma_free_coherent(ism->dev.parent, PAGE_SIZE,
 			  ism->sba, ism->sba_dma_addr);
 
 	ism->sba = NULL;
@@ -169,7 +169,7 @@ static int unregister_ieq(struct ism_dev *ism)
 	if (ret && ret != ISM_ERROR)
 		return -EIO;
 
-	dma_free_coherent(&ism->pdev->dev, PAGE_SIZE,
+	dma_free_coherent(ism->dev.parent, PAGE_SIZE,
 			  ism->ieq, ism->ieq_dma_addr);
 
 	ism->ieq = NULL;
@@ -216,7 +216,7 @@ static int ism_query_rgid(struct ism_dev *ism, uuid_t *rgid, u32 vid_valid,
 static void ism_free_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
 {
 	clear_bit(dmb->sba_idx, ism->sba_bitmap);
-	dma_unmap_page(&ism->pdev->dev, dmb->dma_addr, dmb->dmb_len,
+	dma_unmap_page(ism->dev.parent, dmb->dma_addr, dmb->dmb_len,
 		       DMA_FROM_DEVICE);
 	folio_put(virt_to_folio(dmb->cpu_addr));
 }
@@ -227,7 +227,7 @@ static int ism_alloc_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
 	unsigned long bit;
 	int rc;
 
-	if (PAGE_ALIGN(dmb->dmb_len) > dma_get_max_seg_size(&ism->pdev->dev))
+	if (PAGE_ALIGN(dmb->dmb_len) > dma_get_max_seg_size(ism->dev.parent))
 		return -EINVAL;
 
 	if (!dmb->sba_idx) {
@@ -251,10 +251,10 @@ static int ism_alloc_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
 	}
 
 	dmb->cpu_addr = folio_address(folio);
-	dmb->dma_addr = dma_map_page(&ism->pdev->dev,
+	dmb->dma_addr = dma_map_page(ism->dev.parent,
 				     virt_to_page(dmb->cpu_addr), 0,
 				     dmb->dmb_len, DMA_FROM_DEVICE);
-	if (dma_mapping_error(&ism->pdev->dev, dmb->dma_addr)) {
+	if (dma_mapping_error(ism->dev.parent, dmb->dma_addr)) {
 		rc = -ENOMEM;
 		goto out_free;
 	}
@@ -419,10 +419,7 @@ static int ism_supports_v2(void)
 
 static u16 ism_get_chid(struct ism_dev *ism)
 {
-	if (!ism || !ism->pdev)
-		return 0;
-
-	return to_zpci(ism->pdev)->pchid;
+	return to_zpci(to_pci_dev(ism->dev.parent))->pchid;
 }
 
 static void ism_handle_event(struct ism_dev *ism)
@@ -499,7 +496,7 @@ static const struct ism_ops ism_vp_ops = {
 
 static int ism_dev_init(struct ism_dev *ism)
 {
-	struct pci_dev *pdev = ism->pdev;
+	struct pci_dev *pdev = to_pci_dev(ism->dev.parent);
 	int ret;
 
 	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
@@ -565,7 +562,6 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	spin_lock_init(&ism->lock);
 	dev_set_drvdata(&pdev->dev, ism);
-	ism->pdev = pdev;
 	ism->dev.parent = &pdev->dev;
 	device_initialize(&ism->dev);
 	dev_set_name(&ism->dev, dev_name(&pdev->dev));
@@ -603,14 +599,13 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	device_del(&ism->dev);
 err_dev:
 	dev_set_drvdata(&pdev->dev, NULL);
-	kfree(ism);
 
 	return ret;
 }
 
 static void ism_dev_exit(struct ism_dev *ism)
 {
-	struct pci_dev *pdev = ism->pdev;
+	struct pci_dev *pdev = to_pci_dev(ism->dev.parent);
 	unsigned long flags;
 	int i;
 
include/linux/ism.h +59 -0

83 lines not on this page · raw message

include/net/smc.h +2 -2
index 91aab1d44166..7a96ed2ae20c 100644
@@ -63,8 +63,8 @@ struct smcd_ops {
 
 struct smcd_dev {
 	const struct smcd_ops *ops;
-	void *priv;
-	void *client;
+	struct ism_dev *ism;
+	struct ism_client *client;
 	struct list_head list;
 	spinlock_t lock;
 	struct smc_connection **conn;
net/ism/Kconfig +13 -0
index 4329489cc1e9..ac7a9ba7c792 100644
@@ -12,3 +12,16 @@ config ISM
 
 	  To compile as a module choose M. The module name is ism.
 	  If unsure, choose N.
+
+config ISM_LO
+	bool "intra-OS shortcut with loopback-ism"
+	depends on ISM
+	default n
+	help
+	  ISM_LO enables the creation of an Emulated-ISM device named
+	  loopback-ism which can be used for transferring data
+	  when communication occurs within the same OS. This helps in
+	  convenient testing of ISM clients, since loopback-ism is
+	  independent of architecture or hardware.
+
+	  if unsure, say N.
net/ism/Makefile +1 -0
index b752baf72003..5e7c51845862 100644
@@ -5,3 +5,4 @@
 
 ism-y += ism_main.o
 obj-$(CONFIG_ISM) += ism.o
+ism-$(CONFIG_ISM_LO) += ism_loopback.o
\ No newline at end of file
net/ism/ism_loopback.c new file +366 -0

366 lines not on this page · raw message

net/ism/ism_loopback.h new file +59 -0

59 lines not on this page · raw message

net/ism/ism_main.c +10 -1
index 268408dbd691..13edccff45ea 100644
@@ -14,6 +14,8 @@
 #include <linux/err.h>
 #include <linux/ism.h>
 
+#include "ism_loopback.h"
+
 MODULE_DESCRIPTION("Internal Shared Memory class");
 MODULE_LICENSE("GPL");
 
@@ -148,14 +150,21 @@ EXPORT_SYMBOL_GPL(ism_dev_unregister);
 
 static int __init ism_init(void)
 {
+	int rc;
+
 	memset(clients, 0, sizeof(clients));
 	max_client = 0;
 
-	return 0;
+	rc = ism_loopback_init();
+	if (rc)
+		pr_err("%s: ism_loopback_init fails with %d\n", __func__, rc);
+
+	return rc;
 }
 
 static void __exit ism_exit(void)
 {
+	ism_loopback_exit();
 }
 
 module_init(ism_init);
net/smc/Kconfig +0 -13
index ba5e6a2dd2fd..746be3996768 100644
@@ -20,16 +20,3 @@ config SMC_DIAG
 	  smcss.
 
 	  if unsure, say Y.
-
-config SMC_LO
-	bool "SMC intra-OS shortcut with loopback-ism"
-	depends on SMC
-	default n
-	help
-	  SMC_LO enables the creation of an Emulated-ISM device named
-	  loopback-ism in SMC and makes use of it for transferring data
-	  when communication occurs within the same OS. This helps in
-	  convenient testing of SMC-D since loopback-ism is independent
-	  of architecture or hardware.
-
-	  if unsure, say N.
net/smc/Makefile +0 -1
index 60f1c87d5212..0e754cbc38f9 100644
@@ -6,4 +6,3 @@ smc-y := af_smc.o smc_pnet.o smc_ib.o smc_clc.o smc_core.o smc_wr.o smc_llc.o
 smc-y += smc_cdc.o smc_tx.o smc_rx.o smc_close.o smc_ism.o smc_netlink.o smc_stats.o
 smc-y += smc_tracepoint.o smc_inet.o
 smc-$(CONFIG_SYSCTL) += smc_sysctl.o
-smc-$(CONFIG_SMC_LO) += smc_loopback.o
net/smc/af_smc.c +1 -11
index 9e6c69d18581..b80cae1940e1 100644
@@ -53,7 +53,6 @@
 #include "smc_stats.h"
 #include "smc_tracepoint.h"
 #include "smc_sysctl.h"
-#include "smc_loopback.h"
 #include "smc_inet.h"
 
 static DEFINE_MUTEX(smc_server_lgr_pending);	/* serialize link group
@@ -3560,16 +3559,10 @@ static int __init smc_init(void)
 		goto out_sock;
 	}
 
-	rc = smc_loopback_init();
-	if (rc) {
-		pr_err("%s: smc_loopback_init fails with %d\n", __func__, rc);
-		goto out_ib;
-	}
-
 	rc = tcp_register_ulp(&smc_ulp_ops);
 	if (rc) {
 		pr_err("%s: tcp_ulp_register fails with %d\n", __func__, rc);
-		goto out_lo;
+		goto out_ib;
 	}
 	rc = smc_inet_init();
 	if (rc) {
@@ -3580,8 +3573,6 @@ static int __init smc_init(void)
 	return 0;
 out_ulp:
 	tcp_unregister_ulp(&smc_ulp_ops);
-out_lo:
-	smc_loopback_exit();
 out_ib:
 	smc_ib_unregister_client();
 out_sock:
@@ -3620,7 +3611,6 @@ static void __exit smc_exit(void)
 	tcp_unregister_ulp(&smc_ulp_ops);
 	sock_unregister(PF_SMC);
 	smc_core_exit();
-	smc_loopback_exit();
 	smc_ib_unregister_client();
 	smc_ism_exit();
 	destroy_workqueue(smc_close_wq);
net/smc/smc_ism.c +79 -29

228 lines not on this page · raw message

net/smc/smc_loopback.c deleted +0 -427

427 lines not on this page · raw message

net/smc/smc_loopback.h deleted +0 -60

60 lines not on this page · raw message

Signature · 1 line
2.45.2
maintainer
#17 lore raw

Re: [RFC net-next 5/7] net/ism: Move ism_loopback to net/ism

Quoted 258 lines · Alexandra Winter

The first stage of ism_loopback was implemented as part of the SMC module [1]. Now that we have the ism layer, provide access to the ism_loopback device to all ism clients.

Move ism_loopback.* from net/smc to net/ism.
The following changes are required to ism_loopback.c:
- Change ism_lo_move_data() to no longer schedule an smcd receive tasklet, but instead call ism_client->handle_irq().
Note: In this RFC patch ism_loppback is not fully generic.
The smc-d client uses attached buffers, for moves without signalling. and not-attached buffers for moves with signalling. ism_lo_move_data() must not rely on that assumption. ism_lo_move_data() must be able to handle more than one ism client.

In addition the following changes are required to unify ism_loopback and ism_vp:

In ism layer and ism_vpci:
ism_loopback is not backed by a pci device, so use dev instead of pdev in ism_dev.

In smc-d:
in smcd_alloc_dev():
- use kernel memory instead of device memory for smcd_dev and smcd->conn. An alternative would be to ask device to alloc the memory.
- use different smcd_ops and max_dmbs for ism_vp and ism_loopback. A future patch can change smc-d to directly use ism_ops instead of smcd_ops.
- use ism dev_name instead of pci dev name for ism_evt_wq name
- allocate an event workqueue for ism_loopback, although it currently does not generate events.

---

drivers/s390/net/ism.h     |   6 +-
drivers/s390/net/ism_drv.c |  31 ++-
include/linux/ism.h        |  59 +++++
include/net/smc.h          |   4 +-
net/ism/Kconfig            |  13 ++
net/ism/Makefile           |   1 +
net/ism/ism_loopback.c     | 366 +++++++++++++++++++++++++++++++
net/ism/ism_loopback.h     |  59 +++++
net/ism/ism_main.c         |  11 +-
net/smc/Kconfig            |  13 --
net/smc/Makefile           |   1 -
net/smc/af_smc.c           |  12 +-
net/smc/smc_ism.c          | 108 +++++++---
net/smc/smc_loopback.c     | 427 -------------------------------------
net/smc/smc_loopback.h     |  60 ------
15 files changed, 606 insertions(+), 565 deletions(-)

create mode 100644 net/ism/ism_loopback.c
create mode 100644 net/ism/ism_loopback.h
delete mode 100644 net/smc/smc_loopback.c
delete mode 100644 net/smc/smc_loopback.h

drivers/s390/net/ism.h +3 -3
index 61cf10334170..0deca6d0e328 100644
@@ -202,7 +202,7 @@ struct ism_sba {
 static inline void __ism_read_cmd(struct ism_dev *ism, void *data,
 				  unsigned long offset, unsigned long len)
 {
-	struct zpci_dev *zdev = to_zpci(ism->pdev);
+	struct zpci_dev *zdev = to_zpci(to_pci_dev(ism->dev.parent));
 	u64 req = ZPCI_CREATE_REQ(zdev->fh, 2, 8);
 
 	while (len > 0) {
@@ -216,7 +216,7 @@ static inline void __ism_read_cmd(struct ism_dev *ism, void *data,
 static inline void __ism_write_cmd(struct ism_dev *ism, void *data,
 				   unsigned long offset, unsigned long len)
 {
-	struct zpci_dev *zdev = to_zpci(ism->pdev);
+	struct zpci_dev *zdev = to_zpci(to_pci_dev(ism->dev.parent));
 	u64 req = ZPCI_CREATE_REQ(zdev->fh, 2, len);
 
 	if (len)
@@ -226,7 +226,7 @@ static inline void __ism_write_cmd(struct ism_dev *ism, void *data,
 static inline int __ism_move(struct ism_dev *ism, u64 dmb_req, void *data,
 			     unsigned int size)
 {
-	struct zpci_dev *zdev = to_zpci(ism->pdev);
+	struct zpci_dev *zdev = to_zpci(to_pci_dev(ism->dev.parent));
 	u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, size);
 
 	return __zpci_store_block(data, req, dmb_req);
drivers/s390/net/ism_drv.c +12 -16
index ab70debbdd9d..c0954d6dd9f5 100644
@@ -88,7 +88,7 @@ static int register_sba(struct ism_dev *ism)
 	dma_addr_t dma_handle;
 	struct ism_sba *sba;
 
-	sba = dma_alloc_coherent(&ism->pdev->dev, PAGE_SIZE, &dma_handle,
+	sba = dma_alloc_coherent(ism->dev.parent, PAGE_SIZE, &dma_handle,
 				 GFP_KERNEL);
 	if (!sba)
 		return -ENOMEM;
@@ -99,7 +99,7 @@ static int register_sba(struct ism_dev *ism)
 	cmd.request.sba = dma_handle;
 
 	if (ism_cmd(ism, &cmd)) {
-		dma_free_coherent(&ism->pdev->dev, PAGE_SIZE, sba, dma_handle);
+		dma_free_coherent(ism->dev.parent, PAGE_SIZE, sba, dma_handle);
 		return -EIO;
 	}
 
@@ -115,7 +115,7 @@ static int register_ieq(struct ism_dev *ism)
 	dma_addr_t dma_handle;
 	struct ism_eq *ieq;
 
-	ieq = dma_alloc_coherent(&ism->pdev->dev, PAGE_SIZE, &dma_handle,
+	ieq = dma_alloc_coherent(ism->dev.parent, PAGE_SIZE, &dma_handle,
 				 GFP_KERNEL);
 	if (!ieq)
 		return -ENOMEM;
@@ -127,7 +127,7 @@ static int register_ieq(struct ism_dev *ism)
 	cmd.request.len = sizeof(*ieq);
 
 	if (ism_cmd(ism, &cmd)) {
-		dma_free_coherent(&ism->pdev->dev, PAGE_SIZE, ieq, dma_handle);
+		dma_free_coherent(ism->dev.parent, PAGE_SIZE, ieq, dma_handle);
 		return -EIO;
 	}
 
@@ -149,7 +149,7 @@ static int unregister_sba(struct ism_dev *ism)
 	if (ret && ret != ISM_ERROR)
 		return -EIO;
 
-	dma_free_coherent(&ism->pdev->dev, PAGE_SIZE,
+	dma_free_coherent(ism->dev.parent, PAGE_SIZE,
 			  ism->sba, ism->sba_dma_addr);
 
 	ism->sba = NULL;
@@ -169,7 +169,7 @@ static int unregister_ieq(struct ism_dev *ism)
 	if (ret && ret != ISM_ERROR)
 		return -EIO;
 
-	dma_free_coherent(&ism->pdev->dev, PAGE_SIZE,
+	dma_free_coherent(ism->dev.parent, PAGE_SIZE,
 			  ism->ieq, ism->ieq_dma_addr);
 
 	ism->ieq = NULL;
@@ -216,7 +216,7 @@ static int ism_query_rgid(struct ism_dev *ism, uuid_t *rgid, u32 vid_valid,
 static void ism_free_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
 {
 	clear_bit(dmb->sba_idx, ism->sba_bitmap);
-	dma_unmap_page(&ism->pdev->dev, dmb->dma_addr, dmb->dmb_len,
+	dma_unmap_page(ism->dev.parent, dmb->dma_addr, dmb->dmb_len,
 		       DMA_FROM_DEVICE);
 	folio_put(virt_to_folio(dmb->cpu_addr));
 }
@@ -227,7 +227,7 @@ static int ism_alloc_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
 	unsigned long bit;
 	int rc;
 
-	if (PAGE_ALIGN(dmb->dmb_len) > dma_get_max_seg_size(&ism->pdev->dev))
+	if (PAGE_ALIGN(dmb->dmb_len) > dma_get_max_seg_size(ism->dev.parent))
 		return -EINVAL;
 
 	if (!dmb->sba_idx) {
@@ -251,10 +251,10 @@ static int ism_alloc_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
 	}
 
 	dmb->cpu_addr = folio_address(folio);
-	dmb->dma_addr = dma_map_page(&ism->pdev->dev,
+	dmb->dma_addr = dma_map_page(ism->dev.parent,
 				     virt_to_page(dmb->cpu_addr), 0,
 				     dmb->dmb_len, DMA_FROM_DEVICE);
-	if (dma_mapping_error(&ism->pdev->dev, dmb->dma_addr)) {
+	if (dma_mapping_error(ism->dev.parent, dmb->dma_addr)) {
 		rc = -ENOMEM;
 		goto out_free;
 	}
@@ -419,10 +419,7 @@ static int ism_supports_v2(void)
 
 static u16 ism_get_chid(struct ism_dev *ism)
 {
-	if (!ism || !ism->pdev)
-		return 0;
-
-	return to_zpci(ism->pdev)->pchid;
+	return to_zpci(to_pci_dev(ism->dev.parent))->pchid;
 }
 
 static void ism_handle_event(struct ism_dev *ism)
@@ -499,7 +496,7 @@ static const struct ism_ops ism_vp_ops = {
 
 static int ism_dev_init(struct ism_dev *ism)
 {
-	struct pci_dev *pdev = ism->pdev;
+	struct pci_dev *pdev = to_pci_dev(ism->dev.parent);
 	int ret;
 
 	ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI);
@@ -565,7 +562,6 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	spin_lock_init(&ism->lock);
 	dev_set_drvdata(&pdev->dev, ism);
-	ism->pdev = pdev;
 	ism->dev.parent = &pdev->dev;
 	device_initialize(&ism->dev);
 	dev_set_name(&ism->dev, dev_name(&pdev->dev));
@@ -603,14 +599,13 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	device_del(&ism->dev);

err_dev:

	dev_set_drvdata(&pdev->dev, NULL);
drivers/s390/net/ism_drv.c +0 -1
-	kfree(ism);
	return ret;
}

static void ism_dev_exit(struct ism_dev *ism)
{

drivers/s390/net/ism_drv.c +1 -1
-	struct pci_dev *pdev = ism->pdev;
+	struct pci_dev *pdev = to_pci_dev(ism->dev.parent);
	unsigned long flags;
	int i;
include/linux/ism.h +21 -0
index bc165d077071..929a1f275419 100644
@@ -144,6 +144,9 @@ int  ism_unregister_client(struct ism_client *client);
 *	identified by dmb_tok and idx. If signal flag (sf) then signal
 *	the remote peer that data has arrived in this dmb.
 *
+ * int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb);
+ *	Unregister an ism_dmb buffer
+ *
 * int (*supports_v2)(void);
 *
 * u16 (*get_chid)(struct ism_dev *dev);
@@ -218,12 +221,63 @@ struct ism_ops {
 	int (*reset_vlan_required)(struct ism_dev *dev);
 	int (*signal_event)(struct ism_dev *dev, uuid_t *rgid,
 			    u32 trigger_irq, u32 event_code, u64 info);
+/* no copy option
+ * --------------
+ */
+	/**
+	 * support_dmb_nocopy() - does this device provide no-copy option?
+	 * @dev: ism device
+	 *
+	 * In addition to using move_data(), a sender device can provide a
+	 * kernel address + length, that represents a target dmb
+	 * (like MMIO). If a sender writes into such a ghost-send-buffer
+	 * (= at this kernel address) the data will automatically
+	 * immediately appear in the target dmb, even without calling
+	 * move_data().
+	 * Note that this is NOT related to the MSG_ZEROCOPY socket flag.
+	 *
+	 * Either all 3 function pointers for support_dmb_nocopy(),
+	 * attach_dmb() and detach_dmb() are defined, or all of them must
+	 * be NULL.

include/linux/ism.h struct ism_ops { · line 251

Quoted 10 lines
include/linux/ism.h +10 -0
+	 *
+	 * Return: non-zero, if no-copy is supported.
+	 */
+	int (*support_dmb_nocopy)(struct ism_dev *dev);
+	/**
+	 * attach_dmb() - attach local memory to a remote dmb
+	 * @dev: Local sending ism device
+	 * @dmb: all other parameters are passed in the form of a
+	 *	 dmb struct
+	 *	 TODO: (THIS IS CONFUSING, should be changed)

Agree.

Quoted 22 lines
include/linux/ism.h +20 -0
+	 *  dmb_tok: (in) Token of the remote dmb, we want to attach to
+	 *  cpu_addr: (out) MMIO address
+	 *  dma_addr: (out) MMIO address (if applicable, invalid otherwise)
+	 *  dmb_len: (out) length of local MMIO region,
+	 *           equal to length of remote DMB.
+	 *  sba_idx: (out) index of remote dmb (NOT HELPFUL, should be removed)
+	 *
+	 * Provides a memory address to the sender that can be used to
+	 * directly write into the remote dmb.
+	 *
+	 * Return: Zero upon success, Error code otherwise
+	 */
+	int (*attach_dmb)(struct ism_dev *dev, struct ism_dmb *dmb);
+	/**
+	 * detach_dmb() - Detach the ghost buffer from a remote dmb
+	 * @dev: ism device
+	 * @token: dmb token of the remote dmb
+	 * Return: Zero upon success, Error code otherwise
+	 */
+	int (*detach_dmb)(struct ism_dev *dev, u64 token);
 };
 

...

Quoted 22 lines
include/linux/ism.h +22 -0
+
+static int ism_lo_move_data(struct ism_dev *ism, u64 dmb_tok,
+			    unsigned int idx, bool sf, unsigned int offset,
+			    void *data, unsigned int size)
+{
+	struct ism_lo_dmb_node *rmb_node = NULL, *tmp_node;
+	struct ism_lo_dev *ldev;
+	u16 s_mask;
+	u8 client_id;
+	u32 sba_idx;
+
+	ldev = container_of(ism, struct ism_lo_dev, ism);
+
+	if (!sf)
+		/* since sndbuf is merged with peer DMB, there is
+		 * no need to copy data from sndbuf to peer DMB.
+		 */
+		return 0;
+
+	read_lock_bh(&ldev->dmb_ht_lock);
+	hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_tok) {
+		if (tmp_node->token == dmb_tok) {

include/linux/ism.h

Quoted 10 lines
include/linux/ism.h +10 -0
+			rmb_node = tmp_node;
+			break;
+		}
+	}
+	if (!rmb_node) {
+		read_unlock_bh(&ldev->dmb_ht_lock);
+		return -EINVAL;
+	}
+	// So why copy the data now?? SMC usecase? Data buffer is attached,
+	// rw-pointer are not attached?

I understand the confusion here. I have the same confusion the first time I saw this.

This is actually the tricky part: it assumes the CDC will signal, while the data will not. We need to copy the CDC, so the copy here only to the CDC.

I think we should refine the move_data() API to make this clearer.

Best regards,
Dust

author maintainer
#18 lore raw

Re: [RFC net-next 5/7] net/ism: Move ism_loopback to net/ism

Quoted 34 lines · Dust Li
Quoted 21 lines
net/ism/ism_loopback.c new file +21 -0
+static int ism_lo_move_data(struct ism_dev *ism, u64 dmb_tok,
+			    unsigned int idx, bool sf, unsigned int offset,
+			    void *data, unsigned int size)
+{
+	struct ism_lo_dmb_node *rmb_node = NULL, *tmp_node;
+	struct ism_lo_dev *ldev;
+	u16 s_mask;
+	u8 client_id;
+	u32 sba_idx;
+
+	ldev = container_of(ism, struct ism_lo_dev, ism);
+
+	if (!sf)
+		/* since sndbuf is merged with peer DMB, there is
+		 * no need to copy data from sndbuf to peer DMB.
+		 */
+		return 0;
+
+	read_lock_bh(&ldev->dmb_ht_lock);
+	hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_tok) {
+		if (tmp_node->token == dmb_tok) {

net/ism/ism_loopback.c line 249

Quoted 10 lines
net/ism/ism_loopback.c new file +10 -0
+			rmb_node = tmp_node;
+			break;
+		}
+	}
+	if (!rmb_node) {
+		read_unlock_bh(&ldev->dmb_ht_lock);
+		return -EINVAL;
+	}
+	// So why copy the data now?? SMC usecase? Data buffer is attached,
+	// rw-pointer are not attached?

I understand the confusion here. I have the same confusion the first time I saw this.

Quoted 8 lines

This is actually the tricky part: it assumes the CDC will signal, while the data will not. We need to copy the CDC, so the copy here only to the CDC.

I think we should refine the move_data() API to make this clearer.

Best regards,
Dust

I agree. Will be refined in next version.

#19 lore raw

Re: [RFC net-next 5/7] net/ism: Move ism_loopback to net/ism

In reply to Alexandra Winter
Quoted 57 lines · Alexandra Winter

The first stage of ism_loopback was implemented as part of the SMC module [1]. Now that we have the ism layer, provide access to the ism_loopback device to all ism clients.

Move ism_loopback.* from net/smc to net/ism.
The following changes are required to ism_loopback.c:
- Change ism_lo_move_data() to no longer schedule an smcd receive tasklet, but instead call ism_client->handle_irq().
Note: In this RFC patch ism_loppback is not fully generic.
The smc-d client uses attached buffers, for moves without signalling. and not-attached buffers for moves with signalling. ism_lo_move_data() must not rely on that assumption. ism_lo_move_data() must be able to handle more than one ism client.

In addition the following changes are required to unify ism_loopback and ism_vp:

In ism layer and ism_vpci:
ism_loopback is not backed by a pci device, so use dev instead of pdev in ism_dev.

In smc-d:
in smcd_alloc_dev():
- use kernel memory instead of device memory for smcd_dev and smcd->conn. An alternative would be to ask device to alloc the memory.
- use different smcd_ops and max_dmbs for ism_vp and ism_loopback. A future patch can change smc-d to directly use ism_ops instead of smcd_ops.
- use ism dev_name instead of pci dev name for ism_evt_wq name
- allocate an event workqueue for ism_loopback, although it currently does not generate events.

---
 drivers/s390/net/ism.h     |   6 +-
 drivers/s390/net/ism_drv.c |  31 ++-
 include/linux/ism.h        |  59 +++++
 include/net/smc.h          |   4 +-
 net/ism/Kconfig            |  13 ++
 net/ism/Makefile           |   1 +
 net/ism/ism_loopback.c     | 366 +++++++++++++++++++++++++++++++
 net/ism/ism_loopback.h     |  59 +++++
 net/ism/ism_main.c         |  11 +-
 net/smc/Kconfig            |  13 --
 net/smc/Makefile           |   1 -
 net/smc/af_smc.c           |  12 +-
 net/smc/smc_ism.c          | 108 +++++++---
 net/smc/smc_loopback.c     | 427 -------------------------------------
 net/smc/smc_loopback.h     |  60 ------
 15 files changed, 606 insertions(+), 565 deletions(-)
 create mode 100644 net/ism/ism_loopback.c
 create mode 100644 net/ism/ism_loopback.h
 delete mode 100644 net/smc/smc_loopback.c
 delete mode 100644 net/smc/smc_loopback.h

...

Quoted 31 lines
net/ism/ism_loopback.c new file +25 -0
new file mode 100644
index 000000000000..47e5ef355dd7
@@ -0,0 +1,366 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ *  Functions for loopback-ism device.
+ *
+ *  Copyright (c) 2024, Alibaba Inc.
+ *
+ *  Author: Wen Gu <[email protected]>
+ *          Tony Lu <[email protected]>
+ *
+ */
+
+#include <linux/bitops.h>
+#include <linux/device.h>
+#include <linux/ism.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+
+#include "ism_loopback.h"
+
+#define ISM_LO_V2_CAPABLE	0x1 /* loopback-ism acts as ISMv2 */
+#define ISM_LO_SUPPORT_NOCOPY	0x1
+#define ISM_DMA_ADDR_INVALID	(~(dma_addr_t)0)
+
+static const char ism_lo_dev_name[] = "loopback-ism";
+/* global loopback device */

net/ism/ism_loopback.c line 35

Quoted 10 lines
net/ism/ism_loopback.c new file +10 -0
+static struct ism_lo_dev *lo_dev;
+
+static int ism_lo_query_rgid(struct ism_dev *ism, uuid_t *rgid,
+			     u32 vid_valid, u32 vid)
+{
+	/* rgid should be the same as lgid; vlan is not supported */
+	if (!vid_valid && uuid_equal(rgid, &ism->gid))
+		return 0;
+	return -ENETUNREACH;
+}

This vid_valid check breaks ism-loopback for me.

author maintainer
#20 lore raw

Re: [RFC net-next 5/7] net/ism: Move ism_loopback to net/ism

Quoted 9 lines · Julian Ruess

net/ism/ism_loopback.c line 35

Quoted 8 lines
net/ism/ism_loopback.c new file +8 -0
+static int ism_lo_query_rgid(struct ism_dev *ism, uuid_t *rgid,
+			     u32 vid_valid, u32 vid)
+{
+	/* rgid should be the same as lgid; vlan is not supported */
+	if (!vid_valid && uuid_equal(rgid, &ism->gid))
+		return 0;
+	return -ENETUNREACH;
+}

This vid_valid check breaks ism-loopback for me.

oops, I also get:

Quoted 1 line

smc_chk -C 10.44.30.50

[1] 967189
Test with target IP 10.44.30.50 and port 37373
Live test (SMC-D and SMC-R)
Server started on port 37373

     Failed  (TCP fallback), reasons:
          Client:        0x05000000   Peer declined during handshake
          Server:        0x03030007   No SMC-Dv2 device found, but required

Sorry about that.
Current upstream smc_loopback just ignores vid_valid in smc_lo_query_rgidsmc_lo_query_rgid(), but I'm not sure that is the best way to handle that.
I'll investigate and make sure it works in v2.

author maintainer
#21 lore raw

RFC 6/7 s390/ism: Define ismvp_dev

In reply to Alexandra Winter

Move the fields that are specific to the s390 ism_vpci driver out of the generic ism_dev into a local ismvp_dev structure.

---
 drivers/s390/net/ism.h     | 11 +++++
 drivers/s390/net/ism_drv.c | 87 +++++++++++++++++++++++---------------
 include/linux/ism.h        | 20 +++------
 3 files changed, 71 insertions(+), 47 deletions(-)
drivers/s390/net/ism.h +11 -0
index 0deca6d0e328..720a783ebf90 100644
@@ -196,6 +196,17 @@ struct ism_sba {
 	u16 dmbe_mask[ISM_NR_DMBS];
 };
 
+struct ismvp_dev {
+	struct ism_dev ism;
+	struct ism_sba *sba;
+	dma_addr_t sba_dma_addr;
+	DECLARE_BITMAP(sba_bitmap, ISM_NR_DMBS);
+
+	struct ism_eq *ieq;
+	dma_addr_t ieq_dma_addr;
+	int ieq_idx;
+};
+
 #define ISM_CREATE_REQ(dmb, idx, sf, offset)		\
 	((dmb) | (idx) << 24 | (sf) << 23 | (offset))
 
drivers/s390/net/ism_drv.c +53 -34

247 lines not on this page · raw message

include/linux/ism.h +7 -13
index 929a1f275419..f28238fb5d74 100644
@@ -281,24 +281,18 @@ struct ism_ops {
 
 struct ism_dev {
 	const struct ism_ops *ops;
-	spinlock_t lock; /* protects the ism device */
 	struct list_head list;
-	struct pci_dev *pdev;
-
-	struct ism_sba *sba;
-	dma_addr_t sba_dma_addr;
-	DECLARE_BITMAP(sba_bitmap, ISM_NR_DMBS);
-	u8 *sba_client_arr;	/* entries are indices into 'clients' array */
-	void *priv[MAX_CLIENTS];
-
-	struct ism_eq *ieq;
-	dma_addr_t ieq_dma_addr;
-
 	struct device dev;
 	uuid_t gid;
-	int ieq_idx;
 
+	/* get this lock before accessing any of the fields below */
+	spinlock_t lock;
+	/* indexed by dmb idx; entries are indices into priv and subs arrays: */
+	u8 *sba_client_arr;
+	/* Sparse array of all ISM clients */
 	struct ism_client *subs[MAX_CLIENTS];
+	/* priv pointer per client; for client usage only */
+	void *priv[MAX_CLIENTS];
 };
 
 int ism_dev_register(struct ism_dev *ism);
Signature · 1 line
2.45.2
author maintainer
#22 lore raw

RFC 7/7 net/smc: Use only ism_ops

In reply to Alexandra Winter

Replace smcd_ops by using ism_ops directly.

---
 include/linux/ism.h |   1 +
 include/net/smc.h   |  30 ------
 net/smc/smc_clc.c   |   6 +-
 net/smc/smc_core.c  |   6 +-
 net/smc/smc_diag.c  |   2 +-
 net/smc/smc_ism.c   | 222 ++++++++------------------------------------
 net/smc/smc_ism.h   |   8 +-
 net/smc/smc_pnet.c  |   8 +-
 8 files changed, 55 insertions(+), 228 deletions(-)
include/linux/ism.h +1 -0
index f28238fb5d74..c11de3931722 100644
@@ -30,6 +30,7 @@ struct ism_dmb {
 	 */
 	u64 dmb_tok;
 	/* rgid - GID of designated remote sending device */
+	//TODO: Change to uuid_t GID. Ok for now, because loopback ignores it.
 	u64 rgid;
 	u32 dmb_len;
 	/* sba_idx - Index of this DMB on this receiving device */
include/net/smc.h +0 -30
index 7a96ed2ae20c..a8235de6cf0a 100644
@@ -28,43 +28,13 @@ struct smc_hashinfo {
 
 /* SMCD/ISM device driver interface */
 
-struct smcd_dev;
-
 struct smcd_gid {
 	u64	gid;
 	u64	gid_ext;
 };
 
-struct smcd_ops {
-	int (*query_remote_gid)(struct smcd_dev *dev, struct smcd_gid *rgid,
-				u32 vid_valid, u32 vid);
-	int (*register_dmb)(struct smcd_dev *dev, struct ism_dmb *dmb,
-			    void *client);
-	int (*unregister_dmb)(struct smcd_dev *dev, struct ism_dmb *dmb);
-	int (*move_data)(struct smcd_dev *dev, u64 dmb_tok, unsigned int idx,
-			 bool sf, unsigned int offset, void *data,
-			 unsigned int size);
-	int (*supports_v2)(void);
-	void (*get_local_gid)(struct smcd_dev *dev, struct smcd_gid *gid);
-	u16 (*get_chid)(struct smcd_dev *dev);
-	struct device* (*get_dev)(struct smcd_dev *dev);
-
-	/* optional operations */
-	int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
-	int (*del_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
-	int (*set_vlan_required)(struct smcd_dev *dev);
-	int (*reset_vlan_required)(struct smcd_dev *dev);
-	int (*signal_event)(struct smcd_dev *dev, struct smcd_gid *rgid,
-			    u32 trigger_irq, u32 event_code, u64 info);
-	int (*support_dmb_nocopy)(struct smcd_dev *dev);
-	int (*attach_dmb)(struct smcd_dev *dev, struct ism_dmb *dmb);
-	int (*detach_dmb)(struct smcd_dev *dev, u64 token);
-};
-
 struct smcd_dev {
-	const struct smcd_ops *ops;
 	struct ism_dev *ism;
-	struct ism_client *client;
 	struct list_head list;
 	spinlock_t lock;
 	struct smc_connection **conn;
net/smc/smc_clc.c +3 -3
index 33fa787c28eb..b546999f83a4 100644
@@ -900,7 +900,7 @@ int smc_clc_send_proposal(struct smc_sock *smc, struct smc_init_info *ini)
 		/* add SMC-D specifics */
 		if (ini->ism_dev[0]) {
 			smcd = ini->ism_dev[0];
-			smcd->ops->get_local_gid(smcd, &smcd_gid);
+			copy_to_smcdgid(&smcd_gid, &smcd->ism->gid);
 			pclc_smcd->ism.gid = htonll(smcd_gid.gid);
 			pclc_smcd->ism.chid =
 				htons(smc_ism_get_chid(ini->ism_dev[0]));
@@ -950,7 +950,7 @@ int smc_clc_send_proposal(struct smc_sock *smc, struct smc_init_info *ini)
 		if (ini->ism_offered_cnt) {
 			for (i = 1; i <= ini->ism_offered_cnt; i++) {
 				smcd = ini->ism_dev[i];
-				smcd->ops->get_local_gid(smcd, &smcd_gid);
+				copy_to_smcdgid(&smcd_gid, &smcd->ism->gid);
 				gidchids[entry].chid =
 					htons(smc_ism_get_chid(ini->ism_dev[i]));
 				gidchids[entry].gid = htonll(smcd_gid.gid);
@@ -1043,7 +1043,7 @@ smcd_clc_prep_confirm_accept(struct smc_connection *conn,
 	/* SMC-D specific settings */
 	memcpy(clc->hdr.eyecatcher, SMCD_EYECATCHER,
 	       sizeof(SMCD_EYECATCHER));
-	smcd->ops->get_local_gid(smcd, &smcd_gid);
+	copy_to_smcdgid(&smcd_gid, &smcd->ism->gid);
 	clc->hdr.typev1 = SMC_TYPE_D;
 	clc->d0.gid = htonll(smcd_gid.gid);
 	clc->d0.token = htonll(conn->rmb_desc->token);
net/smc/smc_core.c +3 -3
index d489b80a4503..dca43edfc6be 100644
@@ -555,7 +555,7 @@ static int smc_nl_fill_smcd_lgr(struct smc_link_group *lgr,
 
 	if (nla_put_u32(skb, SMC_NLA_LGR_D_ID, *((u32 *)&lgr->id)))
 		goto errattr;
-	smcd->ops->get_local_gid(smcd, &smcd_gid);
+	copy_to_smcdgid(&smcd_gid, &smcd->ism->gid);
 	if (nla_put_u64_64bit(skb, SMC_NLA_LGR_D_GID,
 			      smcd_gid.gid, SMC_NLA_LGR_D_PAD))
 		goto errattr;
@@ -919,7 +919,7 @@ static int smc_lgr_create(struct smc_sock *smc, struct smc_init_info *ini)
 	if (ini->is_smcd) {
 		/* SMC-D specific settings */
 		smcd = ini->ism_dev[ini->ism_selected];
-		get_device(smcd->ops->get_dev(smcd));
+		get_device(ism_get_dev(smcd->ism));
 		lgr->peer_gid.gid =
 			ini->ism_peer_gid[ini->ism_selected].gid;
 		lgr->peer_gid.gid_ext =
@@ -1469,7 +1469,7 @@ static void smc_lgr_free(struct smc_link_group *lgr)
 	destroy_workqueue(lgr->tx_wq);
 	if (lgr->is_smcd) {
 		smc_ism_put_vlan(lgr->smcd, lgr->vlan_id);
-		put_device(lgr->smcd->ops->get_dev(lgr->smcd));
+		put_device(ism_get_dev(lgr->smcd->ism));
 	}
 	smc_lgr_put(lgr); /* theoretically last lgr_put */
 }
net/smc/smc_diag.c +1 -1
index 6fdb2d96777a..5e79345108d4 100644
@@ -175,7 +175,7 @@ static int __smc_diag_dump(struct sock *sk, struct sk_buff *skb,
 		dinfo.linkid = *((u32 *)conn->lgr->id);
 		dinfo.peer_gid = conn->lgr->peer_gid.gid;
 		dinfo.peer_gid_ext = conn->lgr->peer_gid.gid_ext;
-		smcd->ops->get_local_gid(smcd, &smcd_gid);
+		copy_to_smcdgid(&smcd_gid, &smcd->ism->gid);
 		dinfo.my_gid = smcd_gid.gid;
 		dinfo.my_gid_ext = smcd_gid.gid_ext;
 		dinfo.token = conn->rmb_desc->token;
net/smc/smc_ism.c +38 -184

360 lines not on this page · raw message

net/smc/smc_ism.h +5 -3
index d041e5a7c459..e2e8cfba2575 100644
@@ -68,7 +68,9 @@ static inline int smc_ism_write(struct smcd_dev *smcd, u64 dmb_tok,
 {
 	int rc;
 
-	rc = smcd->ops->move_data(smcd, dmb_tok, idx, sf, offset, data, len);
+	rc = smcd->ism->ops->move_data(smcd->ism, dmb_tok, idx, sf, offset,
+				       data, len);
+
 	return rc < 0 ? rc : 0;
 }
 
@@ -85,14 +87,14 @@ static inline bool __smc_ism_is_emulated(u16 chid)
 
 static inline bool smc_ism_is_emulated(struct smcd_dev *smcd)
 {
-	u16 chid = smcd->ops->get_chid(smcd);
+	u16 chid = smcd->ism->ops->get_chid(smcd->ism);
 
 	return __smc_ism_is_emulated(chid);
 }
 
 static inline bool smc_ism_is_loopback(struct smcd_dev *smcd)
 {
-	return (smcd->ops->get_chid(smcd) == 0xFFFF);
+	return (smcd->ism->ops->get_chid(smcd->ism) == ISM_LO_RESERVED_CHID);
 }
 
 static inline void copy_to_smcdgid(struct smcd_gid *sgid, uuid_t *igid)
net/smc/smc_pnet.c +4 -4
index 716808f374a8..397557f4b7d4 100644
@@ -169,7 +169,7 @@ static int smc_pnet_remove_by_pnetid(struct net *net, char *pnet_name)
 			pr_warn_ratelimited("smc: smcd device %s "
 					    "erased user defined pnetid "
 					    "%.16s\n",
-					    dev_name(smcd->ops->get_dev(smcd)),
+					    dev_name(ism_get_dev(smcd->ism)),
 					    smcd->pnetid);
 			memset(smcd->pnetid, 0, SMC_MAX_PNETID_LEN);
 			smcd->pnetid_by_user = false;
@@ -332,7 +332,7 @@ static struct smcd_dev *smc_pnet_find_smcd(char *smcd_name)
 
 	mutex_lock(&smcd_dev_list.mutex);
 	list_for_each_entry(smcd_dev, &smcd_dev_list.list, list) {
-		if (!strncmp(dev_name(smcd_dev->ops->get_dev(smcd_dev)),
+		if (!strncmp(dev_name(ism_get_dev(smcd_dev->ism)),
 			     smcd_name, IB_DEVICE_NAME_MAX - 1))
 			goto out;
 	}
@@ -431,7 +431,7 @@ static int smc_pnet_add_ib(struct smc_pnettable *pnettable, char *ib_name,
 	if (smcd) {
 		smcddev_applied = smc_pnet_apply_smcd(smcd, pnet_name);
 		if (smcddev_applied) {
-			dev = smcd->ops->get_dev(smcd);
+			dev = ism_get_dev(smcd->ism);
 			pr_warn_ratelimited("smc: smcd device %s "
 					    "applied user defined pnetid "
 					    "%.16s\n", dev_name(dev),
@@ -1190,7 +1190,7 @@ int smc_pnetid_by_table_ib(struct smc_ib_device *smcibdev, u8 ib_port)
  */
 int smc_pnetid_by_table_smcd(struct smcd_dev *smcddev)
 {
-	const char *ib_name = dev_name(smcddev->ops->get_dev(smcddev));
+	const char *ib_name = dev_name(ism_get_dev(smcddev->ism));
 	struct smc_pnettable *pnettable;
 	struct smc_pnetentry *tmp_pe;
 	struct smc_net *sn;
Signature · 1 line
2.45.2
maintainer
#23 lore raw
In reply to Alexandra Winter

On 2025-01-15 20:55:20, Alexandra Winter wrote:

Hi Winter,

I'm fully supportive of the refactor!

Interestingly, I developed a similar RFC code about a month ago while working on enhancing internal communication between guest and host systems. Here are some of my thoughts on the matter:

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory). To my knowledge, a "Shared Memory Device" better encapsulates the functionality we're aiming to implement. It might be beneficial to place it under drivers/shd/ and register it as a new class under /sys/class/shd/. That said, my initial draft also adopted the ISM terminology for simplicity.

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate ism->ops, making the implementation cleaner and more intuitive. This way, the struct ism_device would mainly serve its implementers, while the upper helper functions offer a streamlined interface for SMC.

Structuring and Naming: I recommend embedding the structure of ism_ops directly within ism_dev rather than using a pointer. Additionally, renaming it to ism_device_ops could enhance clarity and consistency.

Quoted 8 lines

This RFC is about providing a generic shim layer between all kinds of ism devices and all kinds of ism users.

Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.

Fully agree.

Quoted 47 lines

Request for comments:
---------------------
Any comments are welcome, but I am aware that this series needs more work. It may not be worth your time to do an in-depth review of the details, I am looking for feedback on the general idea.
I am mostly interested in your thoughts and recommendations about the general concept, the location of net/ism, the structure of include/linux/ism.h, the KConfig and makefiles.

Status of this RFC:
-------------------
This is a very early RFC to ask you for comments on this general idea. The RFC does not fullfill all criteria required for a patchset. The whole set compiles and runs, but I did not try all combinations of module and built-in yet. I did not check for checkpatch or any other checkers. Also I have only done very rudimentary quick tests of SMC-D. More testing is required.

Background / Status quo:
------------------------
Currently s390 hardware provides virtual PCI ISM devices (ism_vpci). Their driver is in drivers/s390/net/ism_drv.c. The main user is SMC-D (net/smc). ism_vpci driver offers a client interface so other users/protocols can also use them, but it is still heavily intermingled with the smc code. Namely, the ISM vPCI module cannot be used without the SMC module, which feels artificial.

The ISM concept is being extended:
[1] proposed an ISM loopback interface (ism_lo), that can be used on non-s390 architectures (e.g. between containers or to test SMC-D). A minimal implementation went upstream with [2]: ism_lo currently is a part of the smc protocol and rather hidden.

[3] proposed a virtio definition of ISM (ism_virtio) that can be used between kvm guests.

We will shortly send an RFC for an ISM client that uses ISM as transport for TTY.

Concept:
--------
Create a shim layer in net/ism that contains common definitions and code for all ism devices and all ism clients.
Any device or client module only needs to depend on this ism layer module and any device or client code only needs to include the definitions in include/linux/ism.h

Quoted 5 lines

Ideas for next steps:
---------------------
- sysfs representation? e.g. as /sys/class/ism ?
- provide a full-fledged ism loopback interface
(runtime enable/disable, sysfs device, ..)

I think it's better if we can make this common for all ISM devices. but yeah, that shoud be the next step.

Best regards,
Dust

Quoted 53 lines

- additional clients (tty over ism)
- additional devices (virtio-ism, ...)

Alexandra Winter (7):
net/ism: Create net/ism
net/ism: Remove dependencies between ISM_VPCI and SMC net/ism: Use uuid_t for ISM GID
net/ism: Add kernel-doc comments for ism functions net/ism: Move ism_loopback to net/ism s390/ism: Define ismvp_dev
net/smc: Use only ism_ops

MAINTAINERS                |   7 +
drivers/s390/net/Kconfig   |  10 +-
drivers/s390/net/Makefile  |   4 +-
drivers/s390/net/ism.h     |  27 ++-
drivers/s390/net/ism_drv.c | 467 ++++++++++++-------------------------
include/linux/ism.h        | 299 +++++++++++++++++++++---
include/net/smc.h          |  52 +----
net/Kconfig                |   1 +
net/Makefile               |   1 +
net/ism/Kconfig            |  27 +++
net/ism/Makefile           |   8 +
net/ism/ism_loopback.c     | 366 +++++++++++++++++++++++++++++
net/ism/ism_loopback.h     |  59 +++++
net/ism/ism_main.c         | 171 ++++++++++++++
net/smc/Kconfig            |  13 --
net/smc/Makefile           |   1 -
net/smc/af_smc.c           |  12 +-
net/smc/smc_clc.c          |   6 +-
net/smc/smc_core.c         |   6 +-
net/smc/smc_diag.c         |   2 +-
net/smc/smc_ism.c          | 112 +++++----
net/smc/smc_ism.h          |  29 ++-
net/smc/smc_loopback.c     | 427 ---------------------------------
net/smc/smc_loopback.h     |  60 -----
net/smc/smc_pnet.c         |   8 +-
25 files changed, 1183 insertions(+), 992 deletions(-)

create mode 100644 net/ism/Kconfig
create mode 100644 net/ism/Makefile
create mode 100644 net/ism/ism_loopback.c
create mode 100644 net/ism/ism_loopback.h
create mode 100644 net/ism/ism_main.c
delete mode 100644 net/smc/smc_loopback.c
delete mode 100644 net/smc/smc_loopback.h

Signature · 1 line
2.45.2
#24 lore raw
Quoted 10 lines · Dust Li

On 2025-01-15 20:55:20, Alexandra Winter wrote:

Hi Winter,

I'm fully supportive of the refactor!

Interestingly, I developed a similar RFC code about a month ago while working on enhancing internal communication between guest and host systems. Here are some of my thoughts on the matter:

Quoted 6 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory). To my knowledge, a "Shared Memory Device" better encapsulates the functionality we're aiming to implement. It might be beneficial to place it under drivers/shd/ and register it as a new class under /sys/class/shd/. That said, my initial draft also adopted the ISM terminology for simplicity.

I'm not sure if we really want to introduce a new name for the already existing ISM device. For me, having two names for the same thing just adds additional complexity.

I would go for /sys/class/ism

Quoted 76 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate ism->ops, making the implementation cleaner and more intuitive. This way, the struct ism_device would mainly serve its implementers, while the upper helper functions offer a streamlined interface for SMC.

Structuring and Naming: I recommend embedding the structure of ism_ops directly within ism_dev rather than using a pointer. Additionally, renaming it to ism_device_ops could enhance clarity and consistency.

Quoted 8 lines

This RFC is about providing a generic shim layer between all kinds of ism devices and all kinds of ism users.

Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.

Fully agree.

Quoted 47 lines

Request for comments:
---------------------
Any comments are welcome, but I am aware that this series needs more work. It may not be worth your time to do an in-depth review of the details, I am looking for feedback on the general idea.
I am mostly interested in your thoughts and recommendations about the general concept, the location of net/ism, the structure of include/linux/ism.h, the KConfig and makefiles.

Status of this RFC:
-------------------
This is a very early RFC to ask you for comments on this general idea. The RFC does not fullfill all criteria required for a patchset. The whole set compiles and runs, but I did not try all combinations of module and built-in yet. I did not check for checkpatch or any other checkers. Also I have only done very rudimentary quick tests of SMC-D. More testing is required.

Background / Status quo:
------------------------
Currently s390 hardware provides virtual PCI ISM devices (ism_vpci). Their driver is in drivers/s390/net/ism_drv.c. The main user is SMC-D (net/smc). ism_vpci driver offers a client interface so other users/protocols can also use them, but it is still heavily intermingled with the smc code. Namely, the ISM vPCI module cannot be used without the SMC module, which feels artificial.

The ISM concept is being extended:
[1] proposed an ISM loopback interface (ism_lo), that can be used on non-s390 architectures (e.g. between containers or to test SMC-D). A minimal implementation went upstream with [2]: ism_lo currently is a part of the smc protocol and rather hidden.

[3] proposed a virtio definition of ISM (ism_virtio) that can be used between kvm guests.

We will shortly send an RFC for an ISM client that uses ISM as transport for TTY.

Concept:
--------
Create a shim layer in net/ism that contains common definitions and code for all ism devices and all ism clients.
Any device or client module only needs to depend on this ism layer module and any device or client code only needs to include the definitions in include/linux/ism.h

Quoted 8 lines
Quoted 5 lines

Ideas for next steps:
---------------------
- sysfs representation? e.g. as /sys/class/ism ?
- provide a full-fledged ism loopback interface
(runtime enable/disable, sysfs device, ..)

I think it's better if we can make this common for all ISM devices. but yeah, that shoud be the next step.

I already have patches based on this series that introduce /sys/class/ism and show ism-loopback as well as s390/ism devices. I can send this soon.

Julian

author maintainer
#25 lore raw
Quoted 6 lines · Julian Ruess
Quoted 5 lines · Dust Li

On 2025-01-15 20:55:20, Alexandra Winter wrote:

Hi Winter,

I'm fully supportive of the refactor!

Thank you very much Dust Li for joining the discussion.

Quoted 3 lines
Quoted 3 lines

Interestingly, I developed a similar RFC code about a month ago while working on enhancing internal communication between guest and host systems.

But you did not send that out, did you?
I hope I did not overlook an earlier proposal by you.

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

It might be beneficial to place it under

Quoted 14 lines
Quoted 2 lines

drivers/shd/ and register it as a new class under /sys/class/shd/. That said, my initial draft also adopted the ISM terminology for simplicity.

I'm not sure if we really want to introduce a new name for the already existing ISM device. For me, having two names for the same thing just adds additional complexity.

I would go for /sys/class/ism

Quoted 5 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).

Quoted 25 lines
Quoted 25 lines

Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate ism->ops, making the implementation cleaner and more intuitive. This way, the struct ism_device would mainly serve its implementers, while the upper helper functions offer a streamlined interface for SMC.

Structuring and Naming: I recommend embedding the structure of ism_ops directly within ism_dev rather than using a pointer. Additionally, renaming it to ism_device_ops could enhance clarity and consistency.

Quoted 8 lines

This RFC is about providing a generic shim layer between all kinds of ism devices and all kinds of ism users.

Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.

Fully agree.

Quoted 1 line

[...]

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

Ideas for next steps:
---------------------
- sysfs representation? e.g. as /sys/class/ism ?
- provide a full-fledged ism loopback interface
(runtime enable/disable, sysfs device, ..)

I think it's better if we can make this common for all ISM devices. but yeah, that shoud be the next step.

The s390 ism_vpci devices are already backed by struct pci_dev. And I assume that would be represented in sysfs somehow like:
/sys/class/ism/ism_vp0/device -> /sys/devices/<pci bus no>/<pci dev no> so there is an
/sys/class/ism/<ism dev name>/device/enable entry already, because there is /sys/devices/<pci bus no>/<pci dev no>/enable today.

I remember Wen Gu's first proposal for ism_loopback had a device in /sys/devices/virtual/ and had an 'active' entry to enable/disable. Something like that could be linked to /sys/class/ism/ism_lo/device.

Quoted 7 lines

I already have patches based on this series that introduce /sys/class/ism and show ism-loopback as well as s390/ism devices. I can send this soon.

Julian

#26 lore raw
Quoted 118 lines · Alexandra Winter
Quoted 6 lines · Julian Ruess
Quoted 5 lines · Dust Li

On 2025-01-15 20:55:20, Alexandra Winter wrote:

Hi Winter,

I'm fully supportive of the refactor!

Thank you very much Dust Li for joining the discussion.

Quoted 3 lines
Quoted 3 lines

Interestingly, I developed a similar RFC code about a month ago while working on enhancing internal communication between guest and host systems.

But you did not send that out, did you?
I hope I did not overlook an earlier proposal by you.

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

It might be beneficial to place it under

Quoted 14 lines
Quoted 2 lines

drivers/shd/ and register it as a new class under /sys/class/shd/. That said, my initial draft also adopted the ISM terminology for simplicity.

I'm not sure if we really want to introduce a new name for the already existing ISM device. For me, having two names for the same thing just adds additional complexity.

I would go for /sys/class/ism

Quoted 5 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).

Quoted 25 lines
Quoted 25 lines

Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate ism->ops, making the implementation cleaner and more intuitive. This way, the struct ism_device would mainly serve its implementers, while the upper helper functions offer a streamlined interface for SMC.

Structuring and Naming: I recommend embedding the structure of ism_ops directly within ism_dev rather than using a pointer. Additionally, renaming it to ism_device_ops could enhance clarity and consistency.

Quoted 8 lines

This RFC is about providing a generic shim layer between all kinds of ism devices and all kinds of ism users.

Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.

Fully agree.

Quoted 1 line

[...]

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

Ideas for next steps:
---------------------
- sysfs representation? e.g. as /sys/class/ism ?
- provide a full-fledged ism loopback interface
(runtime enable/disable, sysfs device, ..)

I think it's better if we can make this common for all ISM devices. but yeah, that shoud be the next step.

The s390 ism_vpci devices are already backed by struct pci_dev. And I assume that would be represented in sysfs somehow like:
/sys/class/ism/ism_vp0/device -> /sys/devices/<pci bus no>/<pci dev no> so there is an
/sys/class/ism/<ism dev name>/device/enable entry already, because there is /sys/devices/<pci bus no>/<pci dev no>/enable today.

Quoted 3 lines

I remember Wen Gu's first proposal for ism_loopback had a device in /sys/devices/virtual/ and had an 'active' entry to enable/disable. Something like that could be linked to /sys/class/ism/ism_lo/device.

My current implementation represents the devices as following in '/sys/class/ism':

ism_lo -> ../../devices/virtual/ism/ism_lo
lismvpci0 -> ../../devices/pci0124:00/0124:00:00.0/ism/ismvpci0

The driver is repsonsible for the naming of its devices.

And yes, because the s390 ism_vpci is backed by a PCI device, '/sys/class/ism/ismvpci0/device/enable' exists.

I think we could implement a device attribute for ism_lo to implement this functionality. I already have a device attribute implemented in ism_main
to access the gid of each ISM device. This leads to the following sysfs entries:

'/sys/class/ism/ism_lo/gid'
'/sys/class/ism/ismvpci0/gid'

Julian

Quoted 9 lines
Quoted 9 lines

I already have patches based on this series that introduce /sys/class/ism and show ism-loopback as well as s390/ism devices. I can send this soon.

Julian

maintainer
#27 lore raw
In reply to Alexandra Winter
Quoted 14 lines · Alexandra Winter
Quoted 6 lines · Julian Ruess
Quoted 5 lines · Dust Li

On 2025-01-15 20:55:20, Alexandra Winter wrote:

Hi Winter,

I'm fully supportive of the refactor!

Thank you very much Dust Li for joining the discussion.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

Interestingly, I developed a similar RFC code about a month ago while working on enhancing internal communication between guest and host systems.

But you did not send that out, did you?
I hope I did not overlook an earlier proposal by you.

No, I just did a POC and didn't find the time to improve it. So I think we can go on with your version.

Quoted 9 lines

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

Oh, I was trying to refer to SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.

Quoted 11 lines

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.

In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.

Quoted 9 lines

It might be beneficial to place it under

Quoted 6 lines
Quoted 2 lines

drivers/shd/ and register it as a new class under /sys/class/shd/. That said, my initial draft also adopted the ISM terminology for simplicity.

I'm not sure if we really want to introduce a new name for the already existing ISM device. For me, having two names for the same thing just adds additional complexity.

I believe that if we are going to rename it, there should be no reference to "ISM" in this subsystem. IBM's PCI ISM can retain that name, as it is an implementation of the Shared Memory device (assuming we adopt that name).

Quoted 13 lines
Quoted 8 lines

I would go for /sys/class/ism

Quoted 5 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

Quoted 10 lines

But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).

This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.

Best regards,
Dust

#28 lore raw
Quoted 1 line · Dust Li
Quoted 1 line

---8<---

Quoted 23 lines
Quoted 7 lines

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

Oh, I was trying to refer to SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.

Quoted 11 lines

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

Quoted 8 lines

I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.

In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.

I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)

One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.

With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.

Thanks,
Niklas

#29 lore raw
Quoted 3 lines

One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording.

Maybe this is the trap i fell into. So are you saying it is not a dual port memory mapped into two CPUs physical address space? In another email there was reference to shm. That would be a VMM equivalent, a bunch of pages mapped into two processes address space.

This comes back to the lack of top level architecture documentation. Outside reviewers such as i will have difficultly making useful contributions, and seeing potential overlap and reuse with other systems, without having a basic understanding of what you are talking about.

Andrew

#30 lore raw
Quoted 7 lines · Andrew Lunn
Quoted 3 lines

One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording.

Maybe this is the trap i fell into. So are you saying it is not a dual port memory mapped into two CPUs physical address space?

Conceptually kind of but the existing s390 specific ISM device is a bit special. But let me start with some background. On s390 aka Mainframes OSs including Linux runs in so called logical partitions (LPARs) which are machine hypervisor VMs which use partitioned non-paging memory. The fact that memory is partitioned is important because this means LPARs can not share physical memory by mapping it.

Now at a high level an ISM device allows communication between two such Linux LPARs on the same machine. The device is discovered as a PCI device and allows Linux to take a buffer called a DMB map that in the IOMMU and generate a token specific to another LPAR which also sees an ISM device sharing the same virtual channel identifier (VCHID). This token can then be transferred out of band (e.g. as part of an extended TCP handshake in SMC-D) to that other system. With the token the other system can use its ISM device to securely (authenticated by the token, LPAR identity and the IOMMU mapping) write into the original systems DMB at throughput and latency similar to doing a memcpy() via a syscall.

On the implementation level the ISM device is actually a piece of firmware and the write to a remote DMB is a special case of our PCI Store Block instruction (no real MMIO on s390, instead there are special instructions). Sadly there are a few more quirks but in principle you can think of it as redirecting writes to a part of the ISM PCI devices' BAR to the DMB in the peer system if that makes sense. There's of course also a mechanism to cause an interrupt on the receiver as the write completes.

Quoted 3 lines

In another
email there was reference to shm. That would be a VMM equivalent, a bunch of pages mapped into two processes address space.

Yes as on a hypervisor which backs VMs with pages one can simply map the DMBin both guests (one mapping potentially being write only) with writes being literally a memcpy().

Quoted 8 lines

This comes back to the lack of top level architecture documentation. Outside reviewers such as i will have difficultly making useful contributions, and seeing potential overlap and reuse with other systems, without having a basic understanding of what you are talking about.

Andrew

I understand your frustration. I do think we're making progress here though. For one loopback ISM makes SMC-D usable/testable even on bare metal and then Alibaba is working on a virtio based ISM device and the draft is public[0]. And there is some information on SMC's use of ISM via a whitepaper[1]

[0] https://lore.kernel.org/all/Y1IqX2uVpcD7cvRF@TonyMac-Alibaba/T/
[1]
https://www.ibm.com/support/pages/system/files/inline-files/IBM%20Shared%20Memory%20Communications%20Version%202.1%20Emulated-ISM_0.pdf

#31 lore raw
Quoted 19 lines

Conceptually kind of but the existing s390 specific ISM device is a bit special. But let me start with some background. On s390 aka Mainframes OSs including Linux runs in so called logical partitions (LPARs) which are machine hypervisor VMs which use partitioned non-paging memory. The fact that memory is partitioned is important because this means LPARs can not share physical memory by mapping it.

Now at a high level an ISM device allows communication between two such Linux LPARs on the same machine. The device is discovered as a PCI device and allows Linux to take a buffer called a DMB map that in the IOMMU and generate a token specific to another LPAR which also sees an ISM device sharing the same virtual channel identifier (VCHID). This token can then be transferred out of band (e.g. as part of an extended TCP handshake in SMC-D) to that other system. With the token the other system can use its ISM device to securely (authenticated by the token, LPAR identity and the IOMMU mapping) write into the original systems DMB at throughput and latency similar to doing a memcpy() via a syscall.

Quoted 8 lines

On the implementation level the ISM device is actually a piece of firmware and the write to a remote DMB is a special case of our PCI Store Block instruction (no real MMIO on s390, instead there are special instructions). Sadly there are a few more quirks but in principle you can think of it as redirecting writes to a part of the ISM PCI devices' BAR to the DMB in the peer system if that makes sense. There's of course also a mechanism to cause an interrupt on the receiver as the write completes.

So the s390 details are interesting, but as you say, it is special. Ideally, all the special should be hidden away inside the driver.

So please take a step back. What is the abstract model?

Can the abstract model be mapped onto CLX? Could it be used with a GPU vRAM? SoC with real shared memory between a pool of CPUs.

Andrew

#32 lore raw
Quoted 28 lines · Andrew Lunn
Quoted 19 lines

Conceptually kind of but the existing s390 specific ISM device is a bit special. But let me start with some background. On s390 aka Mainframes OSs including Linux runs in so called logical partitions (LPARs) which are machine hypervisor VMs which use partitioned non-paging memory. The fact that memory is partitioned is important because this means LPARs can not share physical memory by mapping it.

Now at a high level an ISM device allows communication between two such Linux LPARs on the same machine. The device is discovered as a PCI device and allows Linux to take a buffer called a DMB map that in the IOMMU and generate a token specific to another LPAR which also sees an ISM device sharing the same virtual channel identifier (VCHID). This token can then be transferred out of band (e.g. as part of an extended TCP handshake in SMC-D) to that other system. With the token the other system can use its ISM device to securely (authenticated by the token, LPAR identity and the IOMMU mapping) write into the original systems DMB at throughput and latency similar to doing a memcpy() via a syscall.

Quoted 8 lines

On the implementation level the ISM device is actually a piece of firmware and the write to a remote DMB is a special case of our PCI Store Block instruction (no real MMIO on s390, instead there are special instructions). Sadly there are a few more quirks but in principle you can think of it as redirecting writes to a part of the ISM PCI devices' BAR to the DMB in the peer system if that makes sense. There's of course also a mechanism to cause an interrupt on the receiver as the write completes.

Quoted 3 lines

So the s390 details are interesting, but as you say, it is special. Ideally, all the special should be hidden away inside the driver.

Yes and it will be. There are some exceptions e.g. for vfio-pci pass- through but that's not unusual and why there is already the concept of vfio-pci extension module.

Quoted 2 lines

So please take a step back. What is the abstract model?

I think my high level description may be a good start. The abstract model is the ability to share a memory buffer (DMB) for writing by a communication partner, authenticated by a DMB Token. Plus stuff like triggering an interrupt on write or explicit trigger. Then Alibaba added optional support for what they called attaching the buffer which means it becomes truly shared between the peers but which IBM's ISM can't support. Plus a few more optional pieces such as VLANs, PNETIDs don't ask. The idea for the new layer then is to define this interface with operations and documentation.

Quoted 5 lines

Can the abstract model be mapped onto CLX? Could it be used with a GPU vRAM? SoC with real shared memory between a pool of CPUs.

Andrew

I'd think that yes, one could implement such a mechanism on top of CXL as well as on SoC. Or even with no special hardware between a host and a DPU (e.g. via PCIe endpoint framework). Basically anything that can DMA and IRQs between two OS instances.

#33 lore raw
Quoted 54 lines · Niklas Schnelle
Quoted 28 lines · Andrew Lunn
Quoted 19 lines

Conceptually kind of but the existing s390 specific ISM device is a bit special. But let me start with some background. On s390 aka Mainframes OSs including Linux runs in so called logical partitions (LPARs) which are machine hypervisor VMs which use partitioned non-paging memory. The fact that memory is partitioned is important because this means LPARs can not share physical memory by mapping it.

Now at a high level an ISM device allows communication between two such Linux LPARs on the same machine. The device is discovered as a PCI device and allows Linux to take a buffer called a DMB map that in the IOMMU and generate a token specific to another LPAR which also sees an ISM device sharing the same virtual channel identifier (VCHID). This token can then be transferred out of band (e.g. as part of an extended TCP handshake in SMC-D) to that other system. With the token the other system can use its ISM device to securely (authenticated by the token, LPAR identity and the IOMMU mapping) write into the original systems DMB at throughput and latency similar to doing a memcpy() via a syscall.

Quoted 8 lines

On the implementation level the ISM device is actually a piece of firmware and the write to a remote DMB is a special case of our PCI Store Block instruction (no real MMIO on s390, instead there are special instructions). Sadly there are a few more quirks but in principle you can think of it as redirecting writes to a part of the ISM PCI devices' BAR to the DMB in the peer system if that makes sense. There's of course also a mechanism to cause an interrupt on the receiver as the write completes.

Quoted 3 lines

So the s390 details are interesting, but as you say, it is special. Ideally, all the special should be hidden away inside the driver.

Yes and it will be. There are some exceptions e.g. for vfio-pci pass- through but that's not unusual and why there is already the concept of vfio-pci extension module.

Quoted 2 lines

So please take a step back. What is the abstract model?

I think my high level description may be a good start. The abstract model is the ability to share a memory buffer (DMB) for writing by a communication partner, authenticated by a DMB Token. Plus stuff like triggering an interrupt on write or explicit trigger. Then Alibaba added optional support for what they called attaching the buffer which means it becomes truly shared between the peers but which IBM's ISM can't support. Plus a few more optional pieces such as VLANs, PNETIDs don't ask. The idea for the new layer then is to define this interface with operations and documentation.

Quoted 4 lines

Can the abstract model be mapped onto CLX? Could it be used with a GPU vRAM? SoC with real shared memory between a pool of CPUs.

Quoted 6 lines
Quoted 1 line

Andrew

I'd think that yes, one could implement such a mechanism on top of CXL as well as on SoC. Or even with no special hardware between a host and a DPU (e.g. via PCIe endpoint framework). Basically anything that can DMA and IRQs between two OS instances.

Is DMA part of the abstract model? That would suggest a true shared memory system is excluded, since that would not require DMA.

Maybe take a look at subsystems like USB, I2C.

usb_submit_urb(struct urb *urb, gfp_t mem_flags)

An URB is a data structure with a block of memory associated with it, contains the detail to pass to the USB device.

i2c_transfer(struct i2c_adapter *adap, struct i2c_msg *msgs, int num)

*msgs points to num of messages which get transferred to/from the I2C device.

Could the high level API look like this? No DMA, no IRQ, no concept of a somewhat shared memory. Just an API which asks for a message to be sent to the other end? struct urb has some USB concepts in it, struct i2c_msg has some I2C concepts in it. A struct ism_msg would follow the same pattern, but does it need to care about the DMA, the IRQ, the memory which is semi shared?

Andrew

maintainer
#34 lore raw
Quoted 77 lines · Andrew Lunn
Quoted 54 lines · Niklas Schnelle
Quoted 28 lines · Andrew Lunn
Quoted 19 lines

Conceptually kind of but the existing s390 specific ISM device is a bit special. But let me start with some background. On s390 aka Mainframes OSs including Linux runs in so called logical partitions (LPARs) which are machine hypervisor VMs which use partitioned non-paging memory. The fact that memory is partitioned is important because this means LPARs can not share physical memory by mapping it.

Now at a high level an ISM device allows communication between two such Linux LPARs on the same machine. The device is discovered as a PCI device and allows Linux to take a buffer called a DMB map that in the IOMMU and generate a token specific to another LPAR which also sees an ISM device sharing the same virtual channel identifier (VCHID). This token can then be transferred out of band (e.g. as part of an extended TCP handshake in SMC-D) to that other system. With the token the other system can use its ISM device to securely (authenticated by the token, LPAR identity and the IOMMU mapping) write into the original systems DMB at throughput and latency similar to doing a memcpy() via a syscall.

Quoted 8 lines

On the implementation level the ISM device is actually a piece of firmware and the write to a remote DMB is a special case of our PCI Store Block instruction (no real MMIO on s390, instead there are special instructions). Sadly there are a few more quirks but in principle you can think of it as redirecting writes to a part of the ISM PCI devices' BAR to the DMB in the peer system if that makes sense. There's of course also a mechanism to cause an interrupt on the receiver as the write completes.

Quoted 3 lines

So the s390 details are interesting, but as you say, it is special. Ideally, all the special should be hidden away inside the driver.

Yes and it will be. There are some exceptions e.g. for vfio-pci pass- through but that's not unusual and why there is already the concept of vfio-pci extension module.

Quoted 2 lines

So please take a step back. What is the abstract model?

I think my high level description may be a good start. The abstract model is the ability to share a memory buffer (DMB) for writing by a communication partner, authenticated by a DMB Token. Plus stuff like triggering an interrupt on write or explicit trigger. Then Alibaba added optional support for what they called attaching the buffer which means it becomes truly shared between the peers but which IBM's ISM can't support. Plus a few more optional pieces such as VLANs, PNETIDs don't ask. The idea for the new layer then is to define this interface with operations and documentation.

Quoted 4 lines

Can the abstract model be mapped onto CLX? Could it be used with a GPU vRAM? SoC with real shared memory between a pool of CPUs.

Quoted 6 lines
Quoted 1 line

Andrew

I'd think that yes, one could implement such a mechanism on top of CXL as well as on SoC. Or even with no special hardware between a host and a DPU (e.g. via PCIe endpoint framework). Basically anything that can DMA and IRQs between two OS instances.

Is DMA part of the abstract model? That would suggest a true shared memory system is excluded, since that would not require DMA.

Maybe take a look at subsystems like USB, I2C.

usb_submit_urb(struct urb *urb, gfp_t mem_flags)

An URB is a data structure with a block of memory associated with it, contains the detail to pass to the USB device.

i2c_transfer(struct i2c_adapter *adap, struct i2c_msg *msgs, int num)

*msgs points to num of messages which get transferred to/from the I2C device.

Quoted 6 lines

Could the high level API look like this? No DMA, no IRQ, no concept of a somewhat shared memory. Just an API which asks for a message to be sent to the other end? struct urb has some USB concepts in it, struct i2c_msg has some I2C concepts in it. A struct ism_msg would follow the same pattern, but does it need to care about the DMA, the IRQ, the memory which is semi shared?

I don’t have a clear picture of what the API should look like yet, but I believe it’s possible to avoid DMA and IRQ. In fact, the current data transfer API, ops->move_data() in include/linux/ism.h, already abstracts away the DMA and IRQ details.

One thing we cannot hide, however, is whether the operation is zero-copy or copy. This distinction is important because we can reuse the data at different times in copy mode and zero-copy mode.

Best regards,
Dust

author maintainer
#35 lore raw
Quoted 85 lines · Dust Li
Quoted 77 lines · Andrew Lunn
Quoted 54 lines · Niklas Schnelle
Quoted 28 lines · Andrew Lunn
Quoted 19 lines

Conceptually kind of but the existing s390 specific ISM device is a bit special. But let me start with some background. On s390 aka Mainframes OSs including Linux runs in so called logical partitions (LPARs) which are machine hypervisor VMs which use partitioned non-paging memory. The fact that memory is partitioned is important because this means LPARs can not share physical memory by mapping it.

Now at a high level an ISM device allows communication between two such Linux LPARs on the same machine. The device is discovered as a PCI device and allows Linux to take a buffer called a DMB map that in the IOMMU and generate a token specific to another LPAR which also sees an ISM device sharing the same virtual channel identifier (VCHID). This token can then be transferred out of band (e.g. as part of an extended TCP handshake in SMC-D) to that other system. With the token the other system can use its ISM device to securely (authenticated by the token, LPAR identity and the IOMMU mapping) write into the original systems DMB at throughput and latency similar to doing a memcpy() via a syscall.

Quoted 8 lines

On the implementation level the ISM device is actually a piece of firmware and the write to a remote DMB is a special case of our PCI Store Block instruction (no real MMIO on s390, instead there are special instructions). Sadly there are a few more quirks but in principle you can think of it as redirecting writes to a part of the ISM PCI devices' BAR to the DMB in the peer system if that makes sense. There's of course also a mechanism to cause an interrupt on the receiver as the write completes.

Quoted 3 lines

So the s390 details are interesting, but as you say, it is special. Ideally, all the special should be hidden away inside the driver.

Yes and it will be. There are some exceptions e.g. for vfio-pci pass- through but that's not unusual and why there is already the concept of vfio-pci extension module.

Quoted 2 lines

So please take a step back. What is the abstract model?

I think my high level description may be a good start. The abstract model is the ability to share a memory buffer (DMB) for writing by a communication partner, authenticated by a DMB Token. Plus stuff like triggering an interrupt on write or explicit trigger. Then Alibaba added optional support for what they called attaching the buffer which means it becomes truly shared between the peers but which IBM's ISM can't support. Plus a few more optional pieces such as VLANs, PNETIDs don't ask. The idea for the new layer then is to define this interface with operations and documentation.

Quoted 4 lines

Can the abstract model be mapped onto CLX? Could it be used with a GPU vRAM? SoC with real shared memory between a pool of CPUs.

Quoted 6 lines
Quoted 1 line

Andrew

I'd think that yes, one could implement such a mechanism on top of CXL as well as on SoC. Or even with no special hardware between a host and a DPU (e.g. via PCIe endpoint framework). Basically anything that can DMA and IRQs between two OS instances.

Is DMA part of the abstract model? That would suggest a true shared memory system is excluded, since that would not require DMA.

Maybe take a look at subsystems like USB, I2C.

usb_submit_urb(struct urb *urb, gfp_t mem_flags)

An URB is a data structure with a block of memory associated with it, contains the detail to pass to the USB device.

i2c_transfer(struct i2c_adapter *adap, struct i2c_msg *msgs, int num)

*msgs points to num of messages which get transferred to/from the I2C device.

Quoted 6 lines

Could the high level API look like this? No DMA, no IRQ, no concept of a somewhat shared memory. Just an API which asks for a message to be sent to the other end? struct urb has some USB concepts in it, struct i2c_msg has some I2C concepts in it. A struct ism_msg would follow the same pattern, but does it need to care about the DMA, the IRQ, the memory which is semi shared?

Quoted 5 lines

I don’t have a clear picture of what the API should look like yet, but I believe it’s possible to avoid DMA and IRQ. In fact, the current data transfer API, ops->move_data() in include/linux/ism.h, already abstracts away the DMA and IRQ details.

What is central to ISM is the DMB (Direct Memory Buffer). The concept that there is a DMB dedicated to one writer and one reader. It is owned by the reader and only this writer can write at any offset into the DMB (Fabric controlled). (Reader can technically read/write as well).

So for the client API I think the core functions are
- move_data(*data, target_dmb_token, offset) - called by the sending client, to move data at some offset into a DMB.
- receive_signal(dmb_token, some_signal_info) - called by the ism layer to signal the client, that this DMB needs handling. (currently called handle_irq)

I would not want to abstract that to a message based API, because then we need queues etc and are almost at a net_device. All that is not needed for ism, because DMBs are dedicated to a single writer (who has the responsibility).

Quoted 7 lines

One thing we cannot hide, however, is whether the operation is zero-copy or copy. This distinction is important because we can reuse the data at different times in copy mode and zero-copy mode.

Best regards,
Dust

See my reply on 4/7, as well as Niklas' reply. Currently you can always re-use the send buffer. So zero-copy can be a property of the DMB (attach() function, etc. )

#36 lore raw
Quoted 8 lines

What is central to ISM is the DMB (Direct Memory Buffer). The concept that there is a DMB dedicated to one writer and one reader. It is owned by the reader and only this writer can write at any offset into the DMB (Fabric controlled). (Reader can technically read/write as well).

So for the client API I think the core functions are
- move_data(*data, target_dmb_token, offset) - called by the sending client, to move data at some offset into a DMB.

Missing a length, but otherwise this looks O.K.

Quoted 3 lines

- receive_signal(dmb_token, some_signal_info) - called by the ism layer to signal the client, that this DMB needs handling. (currently called handle_irq)

So there is no indication where in the DMB there is new content?

And when you say "This DMB" does that imply there are multiple DMB shared between two peers?

Maybe i have the wrong idea about a DMB. I was thinking of maybe 64K to a few Mega bytes of memory, in a memory which could truly be shared by CPUs. But maybe a DMB is just a 4K Page, and you have lots of them? If you are 'faking' a shared memory with DMA, they can be anywhere in the address space where the DMA engine can access them.

Quoted 4 lines

I would not want to abstract that to a message based API, because then we need queues etc and are almost at a net_device. All that is not needed for ism, because DMBs are dedicated to a single writer (who has the responsibility).

But i assume there are "protocols" above this. You talked about running a TTY over this. That should be standardized, so everybody implements TTYs in exactly the same way.

Quoted 3 lines
Quoted 3 lines

One thing we cannot hide, however, is whether the operation is zero-copy or copy. This distinction is important because we can reuse the data at different times in copy mode and zero-copy mode.

This needs more explanation. Are you talking about putting data into the DMB, or moving the DMB to the peer?

If you have a DMA engine
moving stuff around, the data can be anywhere the DMA engine can access. But if you have a true shared memory, ideally you want to avoid copying into it.

Then you have the API used by your protocol drivers above. For a TTY running at 9600 baud, a copy into the DMB does not matter. But if you are talking about a network protocol stack on top, your copy from user space to kernel space probably wants to go direct into the DMB. So maybe your API also needs to include allocating/freeing DMBs in an abstract way so it can hide the difference between true shared memory, and kernel memory which can be DMAed?

Andrew

author maintainer
#37 lore raw
Quoted 10 lines · Andrew Lunn
Quoted 8 lines

What is central to ISM is the DMB (Direct Memory Buffer). The concept that there is a DMB dedicated to one writer and one reader. It is owned by the reader and only this writer can write at any offset into the DMB (Fabric controlled). (Reader can technically read/write as well).

So for the client API I think the core functions are
- move_data(*data, target_dmb_token, offset) - called by the sending client, to move data at some offset into a DMB.

Missing a length, but otherwise this looks O.K.

Right, move_data() has a length field. My bad.

Quoted 7 lines
Quoted 4 lines

- receive_signal(dmb_token, some_signal_info) - called by the ism layer to signal the client, that this DMB needs handling. (currently called handle_irq)

So there is no indication where in the DMB there is new content?

The existing ism implementations pass a bit mask in 'some_signal_info' that can be used to signal which parts of the DMB have data to look at.

Quoted 3 lines

And when you say "This DMB" does that imply there are multiple DMB shared between two peers?

Yes, there can be multiple DMBs between the same two peers. And/or an ism device can provide multiple DMBs that are shared with different peers.

Quoted 6 lines

Maybe i have the wrong idea about a DMB. I was thinking of maybe 64K to a few Mega bytes of memory, in a memory which could truly be shared by CPUs. But maybe a DMB is just a 4K Page, and you have lots of them? If you are 'faking' a shared memory with DMA, they can be anywhere in the address space where the DMA engine can access them.

More the latter. Although they can be large, if the client or application wants to spend so much memory.

Which brings us back to the other thread, that ISM may not be the best name for this concept. MCD - 'Memory Communication Device', was a proposals without 'Shared' in the name...

Quoted 9 lines
Quoted 4 lines

I would not want to abstract that to a message based API, because then we need queues etc and are almost at a net_device. All that is not needed for ism, because DMBs are dedicated to a single writer (who has the responsibility).

But i assume there are "protocols" above this. You talked about running a TTY over this. That should be standardized, so everybody implements TTYs in exactly the same way.

Yes, the 'clients' are the protocols above this.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

One thing we cannot hide, however, is whether the operation is zero-copy or copy. This distinction is important because we can reuse the data at different times in copy mode and zero-copy mode.

This needs more explanation. Are you talking about putting data into the DMB, or moving the DMB to the peer?

The former: putting data into the DMB.
But yes the concept of attached, no-copy DMBs, that was introduced by ism-loopback needs a better description.

Quoted 13 lines

If you have a DMA engine
moving stuff around, the data can be anywhere the DMA engine can access. But if you have a true shared memory, ideally you want to avoid copying into it.

Then you have the API used by your protocol drivers above. For a TTY running at 9600 baud, a copy into the DMB does not matter. But if you are talking about a network protocol stack on top, your copy from user space to kernel space probably wants to go direct into the DMB. So maybe your API also needs to include allocating/freeing DMBs in an abstract way so it can hide the difference between true shared memory, and kernel memory which can be DMAed?

Quoted 3 lines

Andrew

The ism_ops register_dmb() and unregister_dmb() are meant to provide that API.

maintainer
#38 lore raw
In reply to Niklas Schnelle
Quoted 47 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line

---8<---

Quoted 23 lines
Quoted 7 lines

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

Oh, I was trying to refer to SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.

Quoted 11 lines

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

Quoted 8 lines

I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.

In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.

I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)

One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.

Quoted 6 lines

With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.

Make sense, I agree with MCD.

Best regard,
Dust

author maintainer
#39 lore raw
Quoted 55 lines · Dust Li
Quoted 47 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line

---8<---

Quoted 23 lines
Quoted 7 lines

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

Oh, I was trying to refer to SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.

Quoted 11 lines

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

Quoted 8 lines

I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.

In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.

I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)

One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.

Quoted 6 lines

With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.

Quoted 5 lines

Make sense, I agree with MCD.

Best regard,
Dust

In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.

So what do your think of Direct Memory Communication (DMC) as class name for these devices?

I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.

maintainer
#40 lore raw
Quoted 58 lines · Alexandra Winter
Quoted 55 lines · Dust Li
Quoted 53 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line

---8<---

Quoted 23 lines
Quoted 7 lines

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

Oh, I was trying to refer to SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.

Quoted 11 lines

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

Quoted 8 lines

I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.

In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.

I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)

One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.

With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.

Quoted 7 lines
Quoted 5 lines

Make sense, I agree with MCD.

Best regard,
Dust

Hi Winter,

Sorry for the late reply; we were on break for the Chinese Spring Festival.

Quoted 9 lines

In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.

So what do your think of Direct Memory Communication (DMC) as class name for these devices?

I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.

I personally prefer MCD over Direct Memory Communication (DMC).

For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?

Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."

Best regards,
Dust

author maintainer
#41 lore raw
Quoted 88 lines · Dust Li
Quoted 58 lines · Alexandra Winter
Quoted 55 lines · Dust Li
Quoted 53 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line

---8<---

Quoted 23 lines
Quoted 7 lines

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

Oh, I was trying to refer to SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.

Quoted 11 lines

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

Quoted 8 lines

I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.

In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.

I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)

One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.

With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.

Quoted 7 lines
Quoted 5 lines

Make sense, I agree with MCD.

Best regard,
Dust

Hi Winter,

Sorry for the late reply; we were on break for the Chinese Spring Festival.

Quoted 9 lines

In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.

So what do your think of Direct Memory Communication (DMC) as class name for these devices?

I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.

I personally prefer MCD over Direct Memory Communication (DMC).

For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?

Quoted 5 lines

Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."

Best regards,
Dust

Thank you for your thoughts, Dust.
For me the 'D as 'direct' is not so much about the number of copies, but more about the aspect, that you can directly write at any offset into the buffer. I.e. no queues. More like the D in DMA or RDMA.

I am preparing a talk for netdev in March about this subject, and the more I work on it, it seems to me that the buffers ('B'), that are
a) only authorized for a single remote device and
b) can be accessed at any offset
are the important differentiator compared other virtual devices.
So maybe 'D' for Dedicated?

I even came up with
dibs - Dedicated Internal Buffer Sharing or
dibc - Dedicated Internal Buffer Communication
(ok, I like the sound and look of the 'I'. But being on the same hardware as opposed to RDMA is also an important aspect.)

MCD - 'memory communication device' sounds rather vague to me. But if it is the smallest common denominator, i.e. the only thing we can all agree on, I could live with it.

maintainer
#42 lore raw
Quoted 94 lines · Alexandra Winter
Quoted 91 lines · Dust Li
Quoted 58 lines · Alexandra Winter
Quoted 55 lines · Dust Li
Quoted 53 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line

---8<---

Quoted 23 lines
Quoted 7 lines

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

Oh, I was trying to refer to SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.

Quoted 11 lines

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

Quoted 8 lines

I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.

In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.

I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)

One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.

With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.

Quoted 7 lines
Quoted 5 lines

Make sense, I agree with MCD.

Best regard,
Dust

Hi Winter,

Sorry for the late reply; we were on break for the Chinese Spring Festival.

Quoted 9 lines

In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.

So what do your think of Direct Memory Communication (DMC) as class name for these devices?

I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.

I personally prefer MCD over Direct Memory Communication (DMC).

For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?

Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."

Quoted 7 lines
Quoted 2 lines

Best regards,
Dust

Thank you for your thoughts, Dust.
For me the 'D as 'direct' is not so much about the number of copies, but more about the aspect, that you can directly write at any offset into the buffer. I.e. no queues. More like the D in DMA or RDMA.

Thanks for you explaintion. I think I understand you now.

Quoted 15 lines

I am preparing a talk for netdev in March about this subject, and the more I work on it, it seems to me that the buffers ('B'), that are
a) only authorized for a single remote device and
b) can be accessed at any offset
are the important differentiator compared other virtual devices.
So maybe 'D' for Dedicated?

I even came up with
dibs - Dedicated Internal Buffer Sharing or
dibc - Dedicated Internal Buffer Communication
(ok, I like the sound and look of the 'I'. But being on the same hardware as opposed to RDMA is also an important aspect.)

Quoted 3 lines

MCD - 'memory communication device' sounds rather vague to me. But if it is the smallest common denominator, i.e. the only thing we can all agree on, I could live with it.

I've thought about it a bit more. Since DMC is at the same level as RDMA and fits well with the "D" in SMC-D, either DMC or MCD works for me.

Best regards,
Dust

reviewer
#43 lore raw
In reply to Alexandra Winter
Quoted 94 lines · Alexandra Winter
Quoted 91 lines · Dust Li
Quoted 58 lines · Alexandra Winter
Quoted 55 lines · Dust Li
Quoted 53 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line

---8<---

Quoted 23 lines
Quoted 7 lines

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

Oh, I was trying to refer to SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.

Quoted 11 lines

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

Quoted 8 lines

I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.

In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.

I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)

One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.

With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.

Quoted 7 lines
Quoted 5 lines

Make sense, I agree with MCD.

Best regard,
Dust

Hi Winter,

Sorry for the late reply; we were on break for the Chinese Spring Festival.

Quoted 9 lines

In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.

So what do your think of Direct Memory Communication (DMC) as class name for these devices?

I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.

I personally prefer MCD over Direct Memory Communication (DMC).

For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?

Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."

Quoted 8 lines
Quoted 2 lines

Best regards,
Dust

Thank you for your thoughts, Dust.
For me the 'D as 'direct' is not so much about the number of copies, but more about the aspect, that you can directly write at any offset into the buffer. I.e. no queues. More like the D in DMA or RDMA.

IMHO the 'D' means that the CPU copy does not need to be involved, and memory access only involves between memory and IO devices. So under this semantics, I think 'DMC' also applies to s390 ism device, since IIUC the s390 ism directly access to the memory which is passed down by move_data(). The exception is lo-ism, where the device actually doesn't need to access the memory(DMB), since the data has been put into the shared memory once the sendmsg() is called and no copy or move is needed. But this is not a violation of name, just a special kind of short-cut. So DMC makes sense to me.

Quoted 17 lines

I am preparing a talk for netdev in March about this subject, and the more I work on it, it seems to me that the buffers ('B'), that are
a) only authorized for a single remote device and
b) can be accessed at any offset
are the important differentiator compared other virtual devices.
So maybe 'D' for Dedicated?

I even came up with
dibs - Dedicated Internal Buffer Sharing or
dibc - Dedicated Internal Buffer Communication
(ok, I like the sound and look of the 'I'. But being on the same hardware as opposed to RDMA is also an important aspect.)

MCD - 'memory communication device' sounds rather vague to me. But if it is the smallest common denominator, i.e. the only thing we can all agree on, I could live with it.

author maintainer
#44 lore raw

Re: [RFC net-next 0/7] Provide an ism layer - naming

Quoted 106 lines · Wen Gu
Quoted 94 lines · Alexandra Winter
Quoted 91 lines · Dust Li
Quoted 58 lines · Alexandra Winter
Quoted 55 lines · Dust Li
Quoted 53 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line

---8<---

Quoted 23 lines
Quoted 7 lines

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

Oh, I was trying to refer to SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.

Quoted 11 lines

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

Quoted 8 lines

I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.

In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.

I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)

One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.

With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.

Quoted 7 lines
Quoted 5 lines

Make sense, I agree with MCD.

Best regard,
Dust

Hi Winter,

Sorry for the late reply; we were on break for the Chinese Spring Festival.

Quoted 9 lines

In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.

So what do your think of Direct Memory Communication (DMC) as class name for these devices?

I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.

I personally prefer MCD over Direct Memory Communication (DMC).

For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?

Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."

Quoted 8 lines
Quoted 2 lines

Best regards,
Dust

Thank you for your thoughts, Dust.
For me the 'D as 'direct' is not so much about the number of copies, but more about the aspect, that you can directly write at any offset into the buffer. I.e. no queues. More like the D in DMA or RDMA.

Quoted 26 lines

IMHO the 'D' means that the CPU copy does not need to be involved, and memory access only involves between memory and IO devices. So under this semantics, I think 'DMC' also applies to s390 ism device, since IIUC the s390 ism directly access to the memory which is passed down by move_data(). The exception is lo-ism, where the device actually doesn't need to access the memory(DMB), since the data has been put into the shared memory once the sendmsg() is called and no copy or move is needed. But this is not a violation of name, just a special kind of short-cut. So DMC makes sense to me.

Quoted 17 lines

I am preparing a talk for netdev in March about this subject, and the more I work on it, it seems to me that the buffers ('B'), that are
a) only authorized for a single remote device and
b) can be accessed at any offset
are the important differentiator compared other virtual devices.
So maybe 'D' for Dedicated?

I even came up with
dibs - Dedicated Internal Buffer Sharing or
dibc - Dedicated Internal Buffer Communication
(ok, I like the sound and look of the 'I'. But being on the same hardware as opposed to RDMA is also an important aspect.)

MCD - 'memory communication device' sounds rather vague to me. But if it is the smallest common denominator, i.e. the only thing we can all agree on, I could live with it.

Could you guys accept
'DIBS - Dedicated Internal Buffer Sharing'
as well?
-> dibs_layer, /class/dibs/, dibs_dev

That is currently my favourite.

maintainer
#45 lore raw

Re: [RFC net-next 0/7] Provide an ism layer - naming

Quoted 137 lines · Alexandra Winter
Quoted 106 lines · Wen Gu
Quoted 94 lines · Alexandra Winter
Quoted 91 lines · Dust Li
Quoted 58 lines · Alexandra Winter
Quoted 55 lines · Dust Li
Quoted 53 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line

---8<---

Quoted 23 lines
Quoted 7 lines

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

Oh, I was trying to refer to SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.

Quoted 11 lines

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

Quoted 8 lines

I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.

In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.

I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)

One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.

With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.

Quoted 7 lines
Quoted 5 lines

Make sense, I agree with MCD.

Best regard,
Dust

Hi Winter,

Sorry for the late reply; we were on break for the Chinese Spring Festival.

Quoted 9 lines

In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.

So what do your think of Direct Memory Communication (DMC) as class name for these devices?

I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.

I personally prefer MCD over Direct Memory Communication (DMC).

For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?

Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."

Quoted 8 lines
Quoted 2 lines

Best regards,
Dust

Thank you for your thoughts, Dust.
For me the 'D as 'direct' is not so much about the number of copies, but more about the aspect, that you can directly write at any offset into the buffer. I.e. no queues. More like the D in DMA or RDMA.

Quoted 26 lines

IMHO the 'D' means that the CPU copy does not need to be involved, and memory access only involves between memory and IO devices. So under this semantics, I think 'DMC' also applies to s390 ism device, since IIUC the s390 ism directly access to the memory which is passed down by move_data(). The exception is lo-ism, where the device actually doesn't need to access the memory(DMB), since the data has been put into the shared memory once the sendmsg() is called and no copy or move is needed. But this is not a violation of name, just a special kind of short-cut. So DMC makes sense to me.

Quoted 17 lines

I am preparing a talk for netdev in March about this subject, and the more I work on it, it seems to me that the buffers ('B'), that are
a) only authorized for a single remote device and
b) can be accessed at any offset
are the important differentiator compared other virtual devices.
So maybe 'D' for Dedicated?

I even came up with
dibs - Dedicated Internal Buffer Sharing or
dibc - Dedicated Internal Buffer Communication
(ok, I like the sound and look of the 'I'. But being on the same hardware as opposed to RDMA is also an important aspect.)

MCD - 'memory communication device' sounds rather vague to me. But if it is the smallest common denominator, i.e. the only thing we can all agree on, I could live with it.

Quoted 7 lines

Could you guys accept
'DIBS - Dedicated Internal Buffer Sharing'
as well?
-> dibs_layer, /class/dibs/, dibs_dev

That is currently my favourite.

I think you might prefer a name that describes shared memory, but I personally believe that something reflecting the device itself would be more fitting.

To be honest, here’s my ranking:

MCD > DMC > DIBS

Best regards,
Dust

author maintainer
#46 lore raw

Re: [RFC net-next 0/7] Provide an ism layer - naming

Quoted 95 lines · Dust Li
Quoted 82 lines · Alexandra Winter
Quoted 51 lines · Wen Gu
Quoted 39 lines · Alexandra Winter
Quoted 36 lines · Dust Li
Quoted 10 lines · Alexandra Winter
Quoted 4 lines · Dust Li
Quoted 3 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line

---8<---

Hi Winter,

Sorry for the late reply; we were on break for the Chinese Spring Festival.

Quoted 9 lines

In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.

So what do your think of Direct Memory Communication (DMC) as class name for these devices?

I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.

I personally prefer MCD over Direct Memory Communication (DMC).

For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?

Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."

Quoted 8 lines
Quoted 2 lines

Best regards,
Dust

Thank you for your thoughts, Dust.
For me the 'D as 'direct' is not so much about the number of copies, but more about the aspect, that you can directly write at any offset into the buffer. I.e. no queues. More like the D in DMA or RDMA.

Quoted 26 lines

IMHO the 'D' means that the CPU copy does not need to be involved, and memory access only involves between memory and IO devices. So under this semantics, I think 'DMC' also applies to s390 ism device, since IIUC the s390 ism directly access to the memory which is passed down by move_data(). The exception is lo-ism, where the device actually doesn't need to access the memory(DMB), since the data has been put into the shared memory once the sendmsg() is called and no copy or move is needed. But this is not a violation of name, just a special kind of short-cut. So DMC makes sense to me.

Quoted 17 lines

I am preparing a talk for netdev in March about this subject, and the more I work on it, it seems to me that the buffers ('B'), that are
a) only authorized for a single remote device and
b) can be accessed at any offset
are the important differentiator compared other virtual devices.
So maybe 'D' for Dedicated?

I even came up with
dibs - Dedicated Internal Buffer Sharing or
dibc - Dedicated Internal Buffer Communication
(ok, I like the sound and look of the 'I'. But being on the same hardware as opposed to RDMA is also an important aspect.)

MCD - 'memory communication device' sounds rather vague to me. But if it is the smallest common denominator, i.e. the only thing we can all agree on, I could live with it.

Quoted 7 lines

Could you guys accept
'DIBS - Dedicated Internal Buffer Sharing'
as well?
-> dibs_layer, /class/dibs/, dibs_dev

That is currently my favourite.

I think you might prefer a name that describes shared memory, but I personally believe that something reflecting the device itself would be more fitting.

Quoted 6 lines

To be honest, here’s my ranking:

MCD > DMC > DIBS

Best regards,
Dust

Thank you keeping the discussion going, Dust.
I know there is no perfect answer, but imo good names can make things easier to understand.
For reasons described above, I would like to have a 'B for buffer' in the prefix. There are many I/O concepts that share memory somehow, but the concept of a dmb dedicated to exactly 2 devices is a differentiator, imo.
I thought 'D for device' was somehow redundant and obvious. But you now you are saying that for you the device is a differentiator? As opposed to other memory sharing techniques that work without devices? Maybe you have a point...

So maybe DMB - Direct Memory Buffers is a good term?
I know we use it for the buffers already, but they are actually a common aspect for all devices and clients, right? So we could define a dmb layer with generic dmb_devices that can be used by dmb_clients to communicate via dmb_bufs.

author maintainer
#47 lore raw
In reply to Dust Li
Quoted 18 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

Quoted 10 lines

But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).

Quoted 6 lines

This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.

Best regards,
Dust

Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.

#48 lore raw
Quoted 25 lines · Alexandra Winter
Quoted 22 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

Quoted 10 lines

But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).

This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.

Quoted 7 lines
Quoted 2 lines

Best regards,
Dust

Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.

Again, lack of basic understanding.... but why is it not a network device? Network devices are not just Ethernet. We also have CAN, SLIP, FDDI, etc.

Andrew

author maintainer
#49 lore raw
Quoted 34 lines · Andrew Lunn
Quoted 25 lines · Alexandra Winter
Quoted 22 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

Quoted 10 lines

But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).

This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.

Quoted 7 lines
Quoted 2 lines

Best regards,
Dust

Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.

Quoted 6 lines

Again, lack of basic understanding.... but why is it not a network device? Network devices are not just Ethernet. We also have CAN, SLIP, FDDI, etc.

Andrew

Thank you very much Andrew for spending the time and discussing this.

At the moment there is not usecase for attaching an ism interface via struct net_device to the linux network layer.
Current client is SMC-D and they bypass the network stack. Next client is a tty-over-ism console driver.
And Niklas Schnelle envisions TTYs, block devices, framebuffers over ISM. None of them do need queues or headers or other things the network stack offers, as long as the source can directly write into the target buffer. As mentioned earlier, probably one could write an network-over-ism driver, but nobody asks for it at the moment.

I have read about the Linux device model with devices and buses, but I have a bit of a hard time mapping that to an existing machine with classes, susbsystems, slots, subtypes, buses over buses and virtual things, etc that often have the same names for different things. So any advice for alternative placements of ism is welcome.

This is the kind of picture I have in my head:

SMC-sockets console
ISM clients: | |

             +-----------------+        +-----------------+
             |     SMC-D       |        |     tty-ism     |
             +-----------------+        +-----------------+
		     |			         |
+----------------------------------------------------------------------+
|            ism layer						       |
|  ism interfaces:     						       |
| ism_vp0	     ism_vp1,        ism_lo,           ism_virtio0, .. |
+----------------------------------------------------------------------+
     |			|		|
+--------------+  +--------------+ +----------------+	
| 0000:00:00.0 |  | 0000:00:00.1 | | virtual/ism/lo |	
+--------------+  +--------------+ +----------------+	
	\		/
+------------------------+
|     pci bus            |
+------------------------+

ls /sys/class/ism
ism_vp0 -> ../../devices/pci0124:00/0124:00:00.0/ism/ism_vp0 ism_vp1 -> ../../devices/pci0125:00/0125:00:00.0/ism/ism_vp1 ism_lo -> ../../devices/virtual/ism/ism_lo

Maybe all that is overkill for ISM?
I think it would be very helpful, if they show up in sysfs, but not absolutely required.

author maintainer
#50 lore raw
In reply to Alexandra Winter
Quoted 25 lines · Alexandra Winter
Quoted 22 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

Quoted 10 lines

But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).

This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.

Quoted 7 lines
Quoted 2 lines

Best regards,
Dust

Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.

Dust for some reason, you did not 'Reply-all':

Quoted 3 lines · Dust Li

I think dev_close()/dev_open() are the high-level APIs, while ndo_stop()/ndo_open() are the underlying device operations that we can reference.

I hear you, it can be beneficial to have a way for upper layers to enable/disable an ism device.
But all this is typically a tricky area. The device driver can also have reasons to enable/disable a device, then hardware could do that or even hotplug a device. Error recovery on different levels may want to run a disable/enable sequence as a reset, etc. And all this has potential for deadlocks.
All this is rather trivial for ism-loopback, as there is not much of a lower layer.
ism-vpci already has 'HW' / device driver configure on/off and device add/remove.
For a future ism-virtio, the Hipervisor may want to add/remove devices.

I wonder what could be the simplest definition of an enable/disable for the ism layer, that we can start with? More sophisticated functionality can always be added later.
Maybe support for add/remove ism-device by the device driver is sufficient as starting point?

maintainer
#51 lore raw
Quoted 32 lines · Alexandra Winter
Quoted 29 lines · Alexandra Winter
Quoted 24 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

Quoted 10 lines

But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).

This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.

Best regards,
Dust

Quoted 6 lines
Quoted 3 lines

Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.

Dust for some reason, you did not 'Reply-all':

Oh, sorry I didn't notice that

Quoted 19 lines
Quoted 3 lines · Dust Li

I think dev_close()/dev_open() are the high-level APIs, while ndo_stop()/ndo_open() are the underlying device operations that we can reference.

I hear you, it can be beneficial to have a way for upper layers to enable/disable an ism device.
But all this is typically a tricky area. The device driver can also have reasons to enable/disable a device, then hardware could do that or even hotplug a device. Error recovery on different levels may want to run a disable/enable sequence as a reset, etc. And all this has potential for deadlocks.
All this is rather trivial for ism-loopback, as there is not much of a lower layer.
ism-vpci already has 'HW' / device driver configure on/off and device add/remove.
For a future ism-virtio, the Hipervisor may want to add/remove devices.

Quoted 5 lines

I wonder what could be the simplest definition of an enable/disable for the ism layer, that we can start with? More sophisticated functionality can always be added later.
Maybe support for add/remove ism-device by the device driver is sufficient as starting point?

I agree; this can be added later. For now, we can simply support unregistering a device from the device driver. Which is already handled by ism_dev_unregister() IIUC.

However, I believe we still need an API and the ability to enable or disable ISM devices from the upper layer. For example, if we want to disable a specific ISM device (such as the loopback device) in SMC, we should not do so by disabling the loopback device at the device layer, as it may also serve other clients beyond SMC.

Further more, I think removing the loopback from the loopback device driver seems unnecessory ? Since we should support that from the upper layer in the future.

Best regards,
Dust

author maintainer
#52 lore raw
Quoted 71 lines · Dust Li
Quoted 32 lines · Alexandra Winter
Quoted 29 lines · Alexandra Winter
Quoted 24 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

Quoted 10 lines

But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).

This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.

Best regards,
Dust

Quoted 6 lines
Quoted 3 lines

Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.

Dust for some reason, you did not 'Reply-all':

Oh, sorry I didn't notice that

Quoted 19 lines
Quoted 3 lines · Dust Li

I think dev_close()/dev_open() are the high-level APIs, while ndo_stop()/ndo_open() are the underlying device operations that we can reference.

I hear you, it can be beneficial to have a way for upper layers to enable/disable an ism device.
But all this is typically a tricky area. The device driver can also have reasons to enable/disable a device, then hardware could do that or even hotplug a device. Error recovery on different levels may want to run a disable/enable sequence as a reset, etc. And all this has potential for deadlocks.
All this is rather trivial for ism-loopback, as there is not much of a lower layer.
ism-vpci already has 'HW' / device driver configure on/off and device add/remove.
For a future ism-virtio, the Hipervisor may want to add/remove devices.

Quoted 5 lines

I wonder what could be the simplest definition of an enable/disable for the ism layer, that we can start with? More sophisticated functionality can always be added later.
Maybe support for add/remove ism-device by the device driver is sufficient as starting point?

I agree; this can be added later. For now, we can simply support unregistering a device from the device driver. Which is already handled by ism_dev_unregister() IIUC.

Quoted 5 lines

However, I believe we still need an API and the ability to enable or disable ISM devices from the upper layer. For example, if we want to disable a specific ISM device (such as the loopback device) in SMC, we should not do so by disabling the loopback device at the device layer, as it may also serve other clients beyond SMC.

Just a thought: not all clients have to use all available ism devices. The client could opt out without removing the device.

Quoted 7 lines

Further more, I think removing the loopback from the loopback device driver seems unnecessory ? Since we should support that from the upper layer in the future.

Best regards,
Dust

All good points. But it also shows that there are many options how to extend ism device handling of the upper layers / clients. e.g. I can image a loop macro ism_for_each_dev() might be nice... I'd prefer to take one step at a time. Start with a minimal useful ism layer and extend by usecase.

author maintainer
#53 lore raw
Quoted 81 lines · Alexandra Winter
Quoted 71 lines · Dust Li
Quoted 32 lines · Alexandra Winter
Quoted 29 lines · Alexandra Winter
Quoted 24 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

Quoted 10 lines

But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).

This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.

Best regards,
Dust

Quoted 6 lines
Quoted 3 lines

Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.

Dust for some reason, you did not 'Reply-all':

Oh, sorry I didn't notice that

Quoted 19 lines
Quoted 3 lines · Dust Li

I think dev_close()/dev_open() are the high-level APIs, while ndo_stop()/ndo_open() are the underlying device operations that we can reference.

I hear you, it can be beneficial to have a way for upper layers to enable/disable an ism device.
But all this is typically a tricky area. The device driver can also have reasons to enable/disable a device, then hardware could do that or even hotplug a device. Error recovery on different levels may want to run a disable/enable sequence as a reset, etc. And all this has potential for deadlocks.
All this is rather trivial for ism-loopback, as there is not much of a lower layer.
ism-vpci already has 'HW' / device driver configure on/off and device add/remove.
For a future ism-virtio, the Hipervisor may want to add/remove devices.

Quoted 5 lines

I wonder what could be the simplest definition of an enable/disable for the ism layer, that we can start with? More sophisticated functionality can always be added later.
Maybe support for add/remove ism-device by the device driver is sufficient as starting point?

I agree; this can be added later. For now, we can simply support unregistering a device from the device driver. Which is already handled by ism_dev_unregister() IIUC.

Quoted 5 lines

However, I believe we still need an API and the ability to enable or disable ISM devices from the upper layer. For example, if we want to disable a specific ISM device (such as the loopback device) in SMC, we should not do so by disabling the loopback device at the device layer, as it may also serve other clients beyond SMC.

Quoted 7 lines

Just a thought: not all clients have to use all available ism devices. The client could opt out without removing the device.

Quoted 4 lines

Further more, I think removing the loopback from the loopback device driver seems unnecessory ? Since we should support that from the upper layer in the future.

If it is not too much effort, I would like to have a simple remove for ism_loopback soon, as it would allow for simple variations of testcases.

Quoted 19 lines
Quoted 3 lines

Best regards,
Dust

All good points. But it also shows that there are many options how to extend ism device handling of the upper layers / clients. e.g. I can image a loop macro ism_for_each_dev() might be nice... I'd prefer to take one step at a time. Start with a minimal useful ism layer and extend by usecase.

maintainer
#54 lore raw
Quoted 88 lines · Alexandra Winter
Quoted 85 lines · Alexandra Winter
Quoted 71 lines · Dust Li
Quoted 32 lines · Alexandra Winter
Quoted 29 lines · Alexandra Winter
Quoted 24 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

Quoted 10 lines

But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).

This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.

Best regards,
Dust

Quoted 6 lines
Quoted 3 lines

Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.

Dust for some reason, you did not 'Reply-all':

Oh, sorry I didn't notice that

Quoted 19 lines
Quoted 3 lines · Dust Li

I think dev_close()/dev_open() are the high-level APIs, while ndo_stop()/ndo_open() are the underlying device operations that we can reference.

I hear you, it can be beneficial to have a way for upper layers to enable/disable an ism device.
But all this is typically a tricky area. The device driver can also have reasons to enable/disable a device, then hardware could do that or even hotplug a device. Error recovery on different levels may want to run a disable/enable sequence as a reset, etc. And all this has potential for deadlocks.
All this is rather trivial for ism-loopback, as there is not much of a lower layer.
ism-vpci already has 'HW' / device driver configure on/off and device add/remove.
For a future ism-virtio, the Hipervisor may want to add/remove devices.

Quoted 5 lines

I wonder what could be the simplest definition of an enable/disable for the ism layer, that we can start with? More sophisticated functionality can always be added later.
Maybe support for add/remove ism-device by the device driver is sufficient as starting point?

I agree; this can be added later. For now, we can simply support unregistering a device from the device driver. Which is already handled by ism_dev_unregister() IIUC.

Quoted 5 lines

However, I believe we still need an API and the ability to enable or disable ISM devices from the upper layer. For example, if we want to disable a specific ISM device (such as the loopback device) in SMC, we should not do so by disabling the loopback device at the device layer, as it may also serve other clients beyond SMC.

Just a thought: not all clients have to use all available ism devices. The client could opt out without removing the device.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

Further more, I think removing the loopback from the loopback device driver seems unnecessory ? Since we should support that from the upper layer in the future.

If it is not too much effort, I would like to have a simple remove for ism_loopback soon, as it would allow for simple variations of testcases.

Yes, this is very useful for testing before we can do that from the upper layer.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Best regards,
Dust

Quoted 5 lines
Quoted 5 lines

All good points. But it also shows that there are many options how to extend ism device handling of the upper layers / clients. e.g. I can image a loop macro ism_for_each_dev() might be nice... I'd prefer to take one step at a time. Start with a minimal useful ism layer and extend by usecase.

That works for me.

Best regards,
Dust

#55 lore raw
In reply to Alexandra Winter
Quoted 2 lines

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

In Linux terminology, a device is something which has a struct device, and a device lives on some sort of bus, even if it is a virtual bus. Will ISM devices properly fit into the Linux device driver model?

Andrew

author maintainer
#56 lore raw
Quoted 9 lines · Andrew Lunn
Quoted 2 lines

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

In Linux terminology, a device is something which has a struct device, and a device lives on some sort of bus, even if it is a virtual bus. Will ISM devices properly fit into the Linux device driver model?

Andrew

ism_vpci lives on a pci bus (zpci flavor) today. The fact that it is not backed by a real hardware PCI slot, but emulated by s390 firmware is not visible to Linux.

In the first proposal, ism_lo lived in on a virtual bus, afaiu. I liked that. In the current stage 1 implementation that is currently upstream, it is not visible in sysfs :-(

ism_dev is a bit modeled after net_device. So it is contains a pointer to a struct device, but it is not the device itself.

I have to admit that the sysfs details are a bit confusing to me, so I wanted to discuss them first before adding them to the RFC. But I tried to bring all the prereqs in place.

reviewer
#57 lore raw
In reply to Alexandra Winter
Quoted 118 lines · Alexandra Winter
Quoted 6 lines · Julian Ruess
Quoted 5 lines · Dust Li

On 2025-01-15 20:55:20, Alexandra Winter wrote:

Hi Winter,

I'm fully supportive of the refactor!

Thank you very much Dust Li for joining the discussion.

Quoted 3 lines
Quoted 3 lines

Interestingly, I developed a similar RFC code about a month ago while working on enhancing internal communication between guest and host systems.

But you did not send that out, did you?
I hope I did not overlook an earlier proposal by you.

Here are some of my thoughts on the matter:

Quoted 3 lines
Quoted 3 lines

Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).

So where does the 'H' come from? If you want to call it Shared Memory _D_evice?

To my knowledge, a

Quoted 2 lines
Quoted 2 lines

"Shared Memory Device" better encapsulates the functionality we're aiming to implement.

Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?

It might be beneficial to place it under

Quoted 14 lines
Quoted 2 lines

drivers/shd/ and register it as a new class under /sys/class/shd/. That said, my initial draft also adopted the ISM terminology for simplicity.

I'm not sure if we really want to introduce a new name for the already existing ISM device. For me, having two names for the same thing just adds additional complexity.

I would go for /sys/class/ism

Quoted 5 lines

Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.

With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.

But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).

Quoted 25 lines
Quoted 25 lines

Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate ism->ops, making the implementation cleaner and more intuitive. This way, the struct ism_device would mainly serve its implementers, while the upper helper functions offer a streamlined interface for SMC.

Structuring and Naming: I recommend embedding the structure of ism_ops directly within ism_dev rather than using a pointer. Additionally, renaming it to ism_device_ops could enhance clarity and consistency.

Quoted 8 lines

This RFC is about providing a generic shim layer between all kinds of ism devices and all kinds of ism users.

Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.

Fully agree.

Quoted 1 line

[...]

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

Ideas for next steps:
---------------------
- sysfs representation? e.g. as /sys/class/ism ?
- provide a full-fledged ism loopback interface
(runtime enable/disable, sysfs device, ..)

I think it's better if we can make this common for all ISM devices. but yeah, that shoud be the next step.

The s390 ism_vpci devices are already backed by struct pci_dev. And I assume that would be represented in sysfs somehow like:
/sys/class/ism/ism_vp0/device -> /sys/devices/<pci bus no>/<pci dev no> so there is an
/sys/class/ism/<ism dev name>/device/enable entry already, because there is /sys/devices/<pci bus no>/<pci dev no>/enable today.

Quoted 4 lines

I remember Wen Gu's first proposal for ism_loopback had a device in /sys/devices/virtual/ and had an 'active' entry to enable/disable. Something like that could be linked to /sys/class/ism/ism_lo/device.

Yes, the previous proposal can be refered to [1]. And the hierarchy you mentioned makes sense to me.

[1] https://lore.kernel.org/netdev/[email protected]/

Quoted 8 lines
Quoted 8 lines

I already have patches based on this series that introduce /sys/class/ism and show ism-loopback as well as s390/ism devices. I can send this soon.

Julian

author maintainer
#58 lore raw
In reply to Dust Li

I hit the send button to early, sorry about that. Let me comment on the other proposals from Dust Li as well.

Quoted 4 lines · Dust Li

Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate ism->ops, making the implementation cleaner and more intuitive.

Maybe I misunderstand what you mean by helper functions.. Why would you encapsulate ism->ops functions in another set of wrappers? I was happy to remove the helper functions in 2/7 and 7/7.

This way, the struct ism_device would mainly serve its

Quoted 2 lines

implementers, while the upper helper functions offer a streamlined interface for SMC.

I was actually also wondering, whether the clients should access ism_device at all. Or whether they should only use the ism_ops.
I can give that a try in the next version. I think this RFC almost there already. The clients would still need to pass a poitner to ism_dev as a parameter.

Quoted 2 lines

Structuring and Naming: I recommend embedding the structure of ism_ops directly within ism_dev rather than using a pointer.

I think it is a common method to have the const struct xy_ops in the device driver code and then use pointer to register the device with an upper layer.
What would be the benefit of duplicating that struct in every ism_dev?

Additionally,

Quoted 1 line

renaming it to ism_device_ops could enhance clarity and consistency.

Yes, that would help to distinguish them from the ism_client functions. I' rename them to ism_dev_ops in the next version.

maintainer
#59 lore raw
Quoted 10 lines · Alexandra Winter

I hit the send button to early, sorry about that. Let me comment on the other proposals from Dust Li as well.

Quoted 4 lines · Dust Li

Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate ism->ops, making the implementation cleaner and more intuitive.

Quoted 3 lines

Maybe I misunderstand what you mean by helper functions.. Why would you encapsulate ism->ops functions in another set of wrappers? I was happy to remove the helper functions in 2/7 and 7/7.

What I mean is similar to how IB handles it in include/rdma/ib_verbs.h. A good example is ib_post_send or ibv_post_send in user space:

```c
static inline int ib_post_send(struct ib_qp *qp,
                               const struct ib_send_wr *send_wr,
                               const struct ib_send_wr **bad_send_wr)
{
        const struct ib_send_wr *dummy;

        return qp->device->ops.post_send(qp, send_wr, bad_send_wr ? : &dummy);
}
```

By following this approach, we can "hide" all the implementations behind ism_xxx. Our users (SMC) should only interact with these APIs. The ism->ops would then be used by our device implementers (vISM, loopback, etc.). This would help make the layers clearer, which is the same approach IB takes.

The layout would somehow like this:

| -------------------- |-----------------------------|
|  ism_register_dmb()  |                             |
|  ism_move_data()     | <---  API for our users     |
|  ism_xxx() ...       |                             |
| -------------------- |-----------------------------|
|   ism_device_ops     | <---for our implementers    |
|                      |    (PCI-ISM/loopback, etc)  |
|----------------------|-----------------------------|
Quoted 9 lines

This way, the struct ism_device would mainly serve its

Quoted 2 lines

implementers, while the upper helper functions offer a streamlined interface for SMC.

I was actually also wondering, whether the clients should access ism_device at all. Or whether they should only use the ism_ops.

I believe the client should only pass an ism_dev pointer to the ism_xxx() helper functions. They should never directly access any of the fields inside the ism_dev.

Quoted 10 lines

I can give that a try in the next version. I think this RFC almost there already. The clients would still need to pass a poitner to ism_dev as a parameter.

Quoted 2 lines

Structuring and Naming: I recommend embedding the structure of ism_ops directly within ism_dev rather than using a pointer.

I think it is a common method to have the const struct xy_ops in the device driver code and then use pointer to register the device with an upper layer.

Right, If we have many ism_devs for each one ISM type, then using pointer should save us some memory.

Quoted 1 line

What would be the benefit of duplicating that struct in every ism_dev?

The main benefit of embedding ism_device_ops within ism_dev is that it reduces the dereferencing of an extra pointer. We already have too many dereference in the datapath, it is not good for performance :(

For example:

rc = smcd->ism->ops->move_data(smcd->ism, dmb_tok, idx, sf, offset,

                               data, len);

Best regards,
Dust

author maintainer
#60 lore raw
Quoted 36 lines · Dust Li
Quoted 10 lines · Alexandra Winter

I hit the send button to early, sorry about that. Let me comment on the other proposals from Dust Li as well.

Quoted 4 lines · Dust Li

Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate ism->ops, making the implementation cleaner and more intuitive.

Quoted 3 lines

Maybe I misunderstand what you mean by helper functions.. Why would you encapsulate ism->ops functions in another set of wrappers? I was happy to remove the helper functions in 2/7 and 7/7.

What I mean is similar to how IB handles it in include/rdma/ib_verbs.h. A good example is ib_post_send or ibv_post_send in user space:

```c
static inline int ib_post_send(struct ib_qp *qp,
                               const struct ib_send_wr *send_wr,
                               const struct ib_send_wr **bad_send_wr)
{
        const struct ib_send_wr *dummy;

        return qp->device->ops.post_send(qp, send_wr, bad_send_wr ? : &dummy);
}
```

By following this approach, we can "hide" all the implementations behind ism_xxx. Our users (SMC) should only interact with these APIs. The ism->ops would then be used by our device implementers (vISM, loopback, etc.). This would help make the layers clearer, which is the same approach IB takes.

The layout would somehow like this:

Quoted 17 lines
| -------------------- |-----------------------------|
|  ism_register_dmb()  |                             |
|  ism_move_data()     | <---  API for our users     |
|  ism_xxx() ...       |                             |
| -------------------- |-----------------------------|
|   ism_device_ops     | <---for our implementers    |
|                      |    (PCI-ISM/loopback, etc)  |
|----------------------|-----------------------------|
Quoted 7 lines

This way, the struct ism_device would mainly serve its

Quoted 2 lines

implementers, while the upper helper functions offer a streamlined interface for SMC.

Thanks for the explanations.
Yes, probably makes sense to further decouple the client API from the device API. I'll give that a try in the next version.

Quoted 28 lines
Quoted 2 lines

I was actually also wondering, whether the clients should access ism_device at all. Or whether they should only use the ism_ops.

I believe the client should only pass an ism_dev pointer to the ism_xxx() helper functions. They should never directly access any of the fields inside the ism_dev.

Quoted 10 lines

I can give that a try in the next version. I think this RFC almost there already. The clients would still need to pass a poitner to ism_dev as a parameter.

Quoted 2 lines

Structuring and Naming: I recommend embedding the structure of ism_ops directly within ism_dev rather than using a pointer.

I think it is a common method to have the const struct xy_ops in the device driver code and then use pointer to register the device with an upper layer.

Right, If we have many ism_devs for each one ISM type, then using pointer should save us some memory.

Quoted 1 line

What would be the benefit of duplicating that struct in every ism_dev?

The main benefit of embedding ism_device_ops within ism_dev is that it reduces the dereferencing of an extra pointer. We already have too many dereference in the datapath, it is not good for performance :(

Quoted 8 lines

For example:

rc = smcd->ism->ops->move_data(smcd->ism, dmb_tok, idx, sf, offset,

                               data, len);

Best regards,
Dust

I see your point. I'm not yet convinced. I'll think more about it.