Provide an ism layer
Subsystems (5)
Alexandra Winter
maintainercover Provide an ism layer
This RFC is about providing a generic shim layer between all kinds of ism devices and all kinds of ism users.
Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.
Request for comments:
---------------------
Any comments are welcome, but I am aware that this series needs more work. It may not be worth your time to do an in-depth review of the details, I am looking for feedback on the general idea.
I am mostly interested in your thoughts and recommendations about the general concept, the location of net/ism, the structure of include/linux/ism.h, the KConfig and makefiles.
Status of this RFC:
-------------------
This is a very early RFC to ask you for comments on this general idea. The RFC does not fullfill all criteria required for a patchset. The whole set compiles and runs, but I did not try all combinations of module and built-in yet. I did not check for checkpatch or any other checkers. Also I have only done very rudimentary quick tests of SMC-D. More testing is required.
Background / Status quo:
------------------------
Currently s390 hardware provides virtual PCI ISM devices (ism_vpci). Their driver is in drivers/s390/net/ism_drv.c. The main user is SMC-D (net/smc). ism_vpci driver offers a client interface so other users/protocols can also use them, but it is still heavily intermingled with the smc code. Namely, the ISM vPCI module cannot be used without the SMC module, which feels artificial.
The ISM concept is being extended:
[1] proposed an ISM loopback interface (ism_lo), that can be used on non-s390 architectures (e.g. between containers or to test SMC-D). A minimal implementation went upstream with [2]: ism_lo currently is a part of the smc protocol and rather hidden.
[3] proposed a virtio definition of ISM (ism_virtio) that can be used between kvm guests.
We will shortly send an RFC for an ISM client that uses ISM as transport for TTY.
Concept:
--------
Create a shim layer in net/ism that contains common definitions and code for all ism devices and all ism clients.
Any device or client module only needs to depend on this ism layer module and any device or client code only needs to include the definitions in include/linux/ism.h
Ideas for next steps:
---------------------
- sysfs representation? e.g. as /sys/class/ism ?
- provide a full-fledged ism loopback interface
(runtime enable/disable, sysfs device, ..)
- additional clients (tty over ism)
- additional devices (virtio-ism, ...)
Alexandra Winter (7):
net/ism: Create net/ism
net/ism: Remove dependencies between ISM_VPCI and SMC net/ism: Use uuid_t for ISM GID
net/ism: Add kernel-doc comments for ism functions net/ism: Move ism_loopback to net/ism s390/ism: Define ismvp_dev
net/smc: Use only ism_ops
MAINTAINERS | 7 + drivers/s390/net/Kconfig | 10 +- drivers/s390/net/Makefile | 4 +- drivers/s390/net/ism.h | 27 ++- drivers/s390/net/ism_drv.c | 467 ++++++++++++------------------------- include/linux/ism.h | 299 +++++++++++++++++++++--- include/net/smc.h | 52 +---- net/Kconfig | 1 + net/Makefile | 1 + net/ism/Kconfig | 27 +++ net/ism/Makefile | 8 + net/ism/ism_loopback.c | 366 +++++++++++++++++++++++++++++ net/ism/ism_loopback.h | 59 +++++ net/ism/ism_main.c | 171 ++++++++++++++ net/smc/Kconfig | 13 -- net/smc/Makefile | 1 - net/smc/af_smc.c | 12 +- net/smc/smc_clc.c | 6 +- net/smc/smc_core.c | 6 +- net/smc/smc_diag.c | 2 +- net/smc/smc_ism.c | 112 +++++---- net/smc/smc_ism.h | 29 ++- net/smc/smc_loopback.c | 427 --------------------------------- net/smc/smc_loopback.h | 60 ----- net/smc/smc_pnet.c | 8 +- 25 files changed, 1183 insertions(+), 992 deletions(-) create mode 100644 net/ism/Kconfig create mode 100644 net/ism/Makefile create mode 100644 net/ism/ism_loopback.c create mode 100644 net/ism/ism_loopback.h create mode 100644 net/ism/ism_main.c delete mode 100644 net/smc/smc_loopback.c delete mode 100644 net/smc/smc_loopback.h
Signature · 1 line
2.45.2
Alexandra Winter
maintainerRFC 1/7 net/ism: Create net/ism
Create an 'ISM' shim layer that will provide generic functionality and declarations for ism device drivers and ism clients.
Move the respective pieces from drivers/s390/net/ism_drv.* to net/ism/
When we need to distinguish between generic ism interfaces and specifically the s390 virtual pci ism device, it will be called 'ISM_vPCI'.
No optimizations are done in this patch, it only moves pieces around. Following patch will further detangle ism_vpci and smc-d.
--- MAINTAINERS | 7 ++ drivers/s390/net/Kconfig | 9 +-- drivers/s390/net/Makefile | 4 +- drivers/s390/net/ism_drv.c | 129 ++--------------------------- include/linux/ism.h | 8 ++ include/net/smc.h | 3 - net/Kconfig | 1 + net/Makefile | 1 + net/ism/Kconfig | 14 ++++ net/ism/Makefile | 7 ++ net/ism/ism_main.c | 162 +++++++++++++++++++++++++++++++++++++ 11 files changed, 213 insertions(+), 132 deletions(-) create mode 100644 net/ism/Kconfig create mode 100644 net/ism/Makefile create mode 100644 net/ism/ism_main.c
MAINTAINERS +7 -0
@@ -12239,6 +12239,13 @@ F: Documentation/devicetree/bindings/hwmon/renesas,isl28022.yaml F: Documentation/hwmon/isl28022.rst F: drivers/hwmon/isl28022.c +ISM (INTERNAL SHARED MEMORY) +M: Alexandra Winter <[email protected]> +L: [email protected] +S: Supported +F: include/linux/ism.h +F: net/ism/ + ISOFS FILESYSTEM M: Jan Kara <[email protected]> L: [email protected]
drivers/s390/net/Kconfig +4 -5
@@ -100,15 +100,14 @@ config CCWGROUP tristate default (LCS || CTCM || QETH || SMC) -config ISM +config ISM_VPCI tristate "Support for ISM vPCI Adapter" - depends on PCI + depends on PCI && ISM imply SMC - default n + default y help Select this option if you want to use the Internal Shared Memory vPCI Adapter. The adapter can be used with the SMC network protocol. - To compile as a module choose M. The module name is ism. - If unsure, choose N. + To compile as a module choose M. The module name is ism_vpci. endmenu
drivers/s390/net/Makefile +2 -2
@@ -16,5 +16,5 @@ obj-$(CONFIG_QETH_L2) += qeth_l2.o qeth_l3-y += qeth_l3_main.o qeth_l3_sys.o obj-$(CONFIG_QETH_L3) += qeth_l3.o -ism-y := ism_drv.o -obj-$(CONFIG_ISM) += ism.o +ism_vpci-y += ism_drv.o +obj-$(CONFIG_ISM_VPCI) += ism_vpci.o
drivers/s390/net/ism_drv.c +7 -122
187 lines not on this page · raw message
include/linux/ism.h +8 -0
@@ -9,6 +9,7 @@#ifndef _ISM_H#define _ISM_H +#include <linux/device.h> #include <linux/workqueue.h> struct ism_dmb {
@@ -24,6 +25,7 @@ struct ism_dmb {/* Unless we gain unexpected popularity, this limit should hold for a while */ #define MAX_CLIENTS 8 +#define NO_CLIENT 0xff/* must be >= MAX_CLIENTS */ #define ISM_NR_DMBS 1920struct ism_dev {
@@ -76,6 +78,9 @@ static inline void *ism_get_priv(struct ism_dev *dev,return dev->priv[client->id]; } +int ism_dev_register (struct ism_dev *ism); +void ism_dev_unregister (struct ism_dev *ism); +static inline void ism_set_priv (struct ism_dev *dev,struct ism_client *client,void *priv) { dev->priv[client->id] = priv;
@@ -87,6 +92,9 @@ int ism_unregister_dmb(struct ism_dev *dev, struct ism_dmb *dmb);int ism_move (struct ism_dev *dev,u64 dmb_tok,unsigned int idx,bool sf,unsigned int offset,void *data,unsigned int size); +#define ISM_RESERVED_VLANID 0x1FFF +#define ISM_ERROR 0xFFFF +const struct smcd_ops *ism_get_smcd_ops (void );#endif /* _ISM_H */
include/net/smc.h +0 -3
@@ -42,9 +42,6 @@ struct smcd_dmb {#define ISM_EVENT_GID 1#define ISM_EVENT_SWR 2 -#define ISM_RESERVED_VLANID 0x1FFF - -#define ISM_ERROR 0xFFFFstruct smcd_dev ;
net/Kconfig +1 -0
@@ -83,6 +83,7 @@ source "net/tls/Kconfig" source "net/xfrm/Kconfig" source "net/iucv/Kconfig" source "net/smc/Kconfig" +source "net/ism/Kconfig" source "net/xdp/Kconfig" config NET_HANDSHAKE
net/Makefile +1 -0
@@ -51,6 +51,7 @@ obj-$(CONFIG_TIPC) += tipc/ obj-$(CONFIG_NETLABEL) += netlabel/ obj-$(CONFIG_IUCV) += iucv/ obj-$(CONFIG_SMC) += smc/ +obj-$(CONFIG_ISM) += ism/ obj-$(CONFIG_RFKILL) += rfkill/ obj-$(CONFIG_NET_9P) += 9p/ obj-$(CONFIG_CAIF) += caif/
net/ism/Kconfig new file +14 -0
@@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0 +config ISM + tristate "ISM support" + default n + help + Internal Shared Memory (ISM) + A communication method that uses common physical memory for + synchronous direct access into a remote buffer. + + Select this option to provide the abstraction layer between + ISM devices and ISM users like the SMC protocol. + + To compile as a module choose M. The module name is ism. + If unsure, choose N.
net/ism/Makefile new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# ISM class module +# + +ism-y += ism_main.o +obj-$(CONFIG_ISM) += ism.o
net/ism/ism_main.c new file +162 -0
162 lines not on this page · raw message
Signature · 1 line
2.45.2
Re: [RFC net-next 1/7] net/ism: Create net/ism
MAINTAINERS F: Documentation/devicetree/bindings/hwmon/renesas,isl28022.yaml · line 12247
Quoted 6 lines
MAINTAINERS +6
-0+ISM (INTERNAL SHARED MEMORY) +M: Alexandra Winter <[email protected]> +L: [email protected] +S: Supported +F: include/linux/ism.h +F: net/ism/
Is there any high level documentation about this?
A while back, TI was trying to upstream something for one of there SoCs. It was a multi CPU system, with not all CPUs used for SMP, but one or two kept for management and real time tasks, not even running Linux. They had a block of shared memory used for communication between the CPUs/OSes, along with rproc. They layered an ethernet driver on top of this, with buffers for frames in the shared memory.
Could ISM be used for something like this?
Andrew
Alexandra Winter
maintainerRe: [RFC net-next 1/7] net/ism: Create net/ism
Quoted 8 lines · Andrew Lunn
MAINTAINERS F: Documentation/devicetree/bindings/hwmon/renesas,isl28022.yaml · line 12247
Quoted 6 lines
MAINTAINERS +6
-0+ISM (INTERNAL SHARED MEMORY) +M: Alexandra Winter <[email protected]> +L: [email protected] +S: Supported +F: include/linux/ism.h +F: net/ism/Is there any high level documentation about this?
As the ISM devices were developed for SMC-D, the only documentation is about their usage for SMC-D.
e.g.:
https://www.ibm.com/support/pages/system/files/inline-files/IBM%20Shared%20Memory%20Communications%20Version%202.1%20Emulated-ISM_0.pdf
(page 33)
https://community.ibm.com/community/user/ibmz-and-linuxone/viewdocument/2021-07-15-boosting-tcp-networking?CommunityKey=c1293167-6d93-448e-8854-3068846d3dfe&tab=librarydocuments
But those do not go into much detail.
We now want to provide interfaces for other usecases in Linux.
ism.h would be the place to explicitely state the assumptions, restrictions and requirements in a single place, so future devices and clients know about them.
Quoted 11 lines
A while back, TI was trying to upstream something for one of there SoCs. It was a multi CPU system, with not all CPUs used for SMP, but one or two kept for management and real time tasks, not even running Linux. They had a block of shared memory used for communication between the CPUs/OSes, along with rproc. They layered an ethernet driver on top of this, with buffers for frames in the shared memory.
Could ISM be used for something like this?
Andrew
If the communication endpoints were represented as devices, that sounds like a similar concept.
I think you could implement a client that provides network devices on top of ism devices. (mapping MACs to GIDs)
As the memory buffers are set up for 1 sender and 1 receiver, it would either create some additional latency, if you setup buffers for each message or additional memory consumption, if you try to keep and re-use the buffers.
I'm not sure what the benefit ISM would provide as ethernet device. A shared network card would probably outperform such a usecase.
SMC exploits ISM for TCP traffic. There the buffers are kept per socket connection, and a lot of the TCP/IP mechanisms are not neccessary, because transport is reliable, synchronous and in-order. Thus latency is minimal.
Alexandra Winter
maintainerRFC 2/7 net/ism: Remove dependencies between ISM_VPCI and SMC
In reply to Alexandra WinterThe modules ISM_VPCI and SMC should not depend on each other, instead they should both depend on the ISM layer module.
use only ism_dmb:
Now that SMC depends on ISM, we can safely remove the duplicate declaration of smcd_dmb and use only ism_dmb.
Move smcd_ops away from ism_drv:
move smcd_ops from drivers/s390/net/ism_drv.c to net/smc/smc_ism.c Less exported functions, no more dependencies between ISM_VPCI and SMC. Once ism_loopback is also moved to ism layer, a follow on patch can use ism_ops directly and remove smcd_ops.
Now the ISM_VPCI module no longer needs to imply SMC.
Note:
- This patch temporarily moves smcd_gid to ism.h, a follow on patch (uuid_t gid) will restore this.
- Added a comment that vlan handling in ism_drv.c and smc is incomplete. Should be fixed by a follow-on patch.
--- drivers/s390/net/Kconfig | 1 - drivers/s390/net/ism.h | 1 - drivers/s390/net/ism_drv.c | 236 +++++++++++++------------------------ include/linux/ism.h | 146 +++++++++++++++++------ include/net/smc.h | 31 ++--- net/smc/smc_ism.c | 123 ++++++++++++++++++- net/smc/smc_loopback.c | 6 +- 7 files changed, 319 insertions(+), 225 deletions(-)
drivers/s390/net/Kconfig +0 -1
@@ -103,7 +103,6 @@ config CCWGROUP config ISM_VPCI tristate "Support for ISM vPCI Adapter" depends on PCI && ISM - imply SMC default y help Select this option if you want to use the Internal Shared Memory
drivers/s390/net/ism.h +0 -1
@@ -6,7 +6,6 @@#include <linux/types.h> #include <linux/pci.h> #include <linux/ism.h> -#include <net/smc.h> #include <asm/pci_insn.h> #define UTIL_STR_LEN 16
drivers/s390/net/ism_drv.c +80 -156
298 lines not on this page · raw message
include/linux/ism.h +110 -36
180 lines not on this page · raw message
include/net/smc.h +8 -23
@@ -27,35 +27,20 @@ struct smc_hashinfo { };/* SMCD/ISM device driver interface */ -struct smcd_dmb { -u64 dmb_tok; -u64 rgid; -u32 dmb_len; -u32 sba_idx; -u32 vlan_valid; -u32 vlan_id; -void *cpu_addr; -dma_addr_t dma_addr; -}; - -#define ISM_EVENT_DMB 0 -#define ISM_EVENT_GID 1 -#define ISM_EVENT_SWR 2 -struct smcd_dev ; -struct smcd_gid { -u64 gid; -u64 gid_ext; -}; - +//struct smcd_gid { +// u64 gid; +// u64 gid_ext; +//}; +// struct smcd_ops {int (*query_remote_gid)(struct smcd_dev *dev,struct smcd_gid *rgid,u32 vid_valid,u32 vid); -int (*register_dmb)(struct smcd_dev *dev,struct smcd_dmb *dmb, +int (*register_dmb)(struct smcd_dev *dev,struct ism_dmb *dmb,void *client); -int (*unregister_dmb)(struct smcd_dev *dev,struct smcd_dmb *dmb); +int (*unregister_dmb)(struct smcd_dev *dev,struct ism_dmb *dmb);int (*move_data)(struct smcd_dev *dev,u64 dmb_tok,unsigned int idx,bool sf,unsigned int offset,void *data,unsigned int size);
@@ -72,7 +57,7 @@ struct smcd_ops {int (*signal_event)(struct smcd_dev *dev,struct smcd_gid *rgid,u32 trigger_irq,u32 event_code,u64 info);int (*support_dmb_nocopy)(struct smcd_dev *dev); -int (*attach_dmb)(struct smcd_dev *dev,struct smcd_dmb *dmb); +int (*attach_dmb)(struct smcd_dev *dev,struct ism_dmb *dmb);int (*detach_dmb)(struct smcd_dev *dev,u64 token); };
net/smc/smc_ism.c +118 -5
155 lines not on this page · raw message
net/smc/smc_loopback.c +3 -3
@@ -51,7 +51,7 @@ static int smc_lo_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid,return 0 ; } -static int smc_lo_register_dmb (struct smcd_dev *smcd,struct smcd_dmb *dmb, +static int smc_lo_register_dmb (struct smcd_dev *smcd,struct ism_dmb *dmb,void *client_priv) {struct smc_lo_dmb_node *dmb_node, *tmp_node;
@@ -129,7 +129,7 @@ static void __smc_lo_unregister_dmb(struct smc_lo_dev *ldev,wake_up (&ldev->ldev_release); } -static int smc_lo_unregister_dmb (struct smcd_dev *smcd,struct smcd_dmb *dmb) +static int smc_lo_unregister_dmb (struct smcd_dev *smcd,struct ism_dmb *dmb) {struct smc_lo_dmb_node *dmb_node =NULL , *tmp_node;struct smc_lo_dev *ldev = smcd->priv;
@@ -158,7 +158,7 @@ static int smc_lo_support_dmb_nocopy(struct smcd_dev *smcd)return SMC_LO_SUPPORT_NOCOPY ; } -static int smc_lo_attach_dmb (struct smcd_dev *smcd,struct smcd_dmb *dmb) +static int smc_lo_attach_dmb (struct smcd_dev *smcd,struct ism_dmb *dmb) {struct smc_lo_dmb_node *dmb_node =NULL , *tmp_node;struct smc_lo_dev *ldev = smcd->priv;
Signature · 1 line
2.45.2
Alexandra Winter
maintainerRFC 3/7 net/ism: Use uuid_t for ISM GID
In reply to Alexandra WinterSMC uses 64 Bit and 128 Bit Global Identifiers (GIDs) that need to be sent via the SMC protocol.
When integers are used network endianness and host endianness need to be considered.
Avoid this in the ISM layer by using uuid_t byte arrays. Follow on patches could do the same change for SMC, for now conversion helper functions are introduced.
ISM-vPCI devices provide 64 Bit GIDs. Map them to ISM uuid_t GIDs like this:
_________________________________________
| 64 Bit ISM-vPCI GID | 00000000_00000000 |
-----------------------------------------
If interpreted as UUID, this would be interpreted as th UIID variant, that is reserved for NCS backward compatibility. So it will not collide with UUIDs that were generated according to the standard.
Future ISM devices, shall use real UUIDs as 128 Bit GIDs.
Note:
- In this RFC patch smcd_gid is now moved back to smc.h, future patchset should avoid that.
- ism_dmb and ism_event structs still contain 64 Bit rgid and info fields. A future patch could change them to uuid_t gids. This does not break anything, because ism_loopback does not use them.
--- drivers/s390/net/ism.h | 9 +++++++++ drivers/s390/net/ism_drv.c | 16 ++++++++-------- include/linux/ism.h | 16 ++++++---------- include/net/smc.h | 12 ++++++------ net/smc/smc_ism.c | 13 ++++++++----- net/smc/smc_ism.h | 21 +++++++++++++++++++++ 6 files changed, 58 insertions(+), 29 deletions(-)
drivers/s390/net/ism.h +9 -0
@@ -64,6 +64,15 @@ union ism_reg_ieq { } response; }__aligned(16) ; +/* ISM-vPCI devices provide 64 Bit GIDs +* Map them to ISM UUID GIDs like this: +* _________________________________________ +* | 64 Bit ISM-vPCI GID | 00000000_00000000 | +* ----------------------------------------- +* This will be interpreted as a UIID variant, that is reserved +* for NCS backward compatibility. So it will not collide with +* proper UUIDs. +*/ union ism_read_gid { struct { struct ism_req_hdr hdr;
drivers/s390/net/ism_drv.c +8 -8
@@ -191,14 +191,14 @@ static int ism_read_local_gid(struct ism_dev *ism)if (ret) goto out; - ism->gid.gid = cmd.response.gid; - ism->gid.gid_ext =0 ; +memset (&ism->gid,0 ,sizeof (ism->gid)); +memcpy (&ism->gid, &cmd.response.gid,sizeof (cmd.response.gid)); out:return ret; } -static int ism_query_rgid (struct ism_dev *ism,struct smcd_gid *rgid, -u32 vid_valid,u32 vid) +static int ism_query_rgid (struct ism_dev *ism,uuid_t *rgid,u32 vid_valid, +u32 vid) {union ism_query_rgid cmd;
@@ -206,7 +206,7 @@ static int ism_query_rgid(struct ism_dev *ism, struct smcd_gid *rgid, cmd.request.hdr.cmd =ISM_QUERY_RGID ; cmd.request.hdr.len =sizeof (cmd.request); - cmd.request.rgid = rgid->gid; +memcpy (&cmd.request.rgid, rgid,sizeof (cmd.request.rgid)); cmd.request.vlan_valid = vid_valid; cmd.request.vlan_id = vid;
@@ -364,8 +364,8 @@ static int ism_reset_vlan_required(struct ism_dev *ism)return ism_cmd_simple (ism,ISM_RESET_VLAN ); } -static int ism_signal_ieq (struct ism_dev *ism,struct smcd_gid *rgid, -u32 trigger_irq,u32 event_code,u64 info) +static int ism_signal_ieq (struct ism_dev *ism,uuid_t *rgid,u32 trigger_irq, +u32 event_code,u64 info) {union ism_sig_ieq cmd;
@@ -373,7 +373,7 @@ static int ism_signal_ieq(struct ism_dev *ism, struct smcd_gid *rgid, cmd.request.hdr.cmd =ISM_SIGNAL_IEQ ; cmd.request.hdr.len =sizeof (cmd.request); - cmd.request.rgid = rgid->gid; +memcpy (&cmd.request.rgid, rgid,sizeof (cmd.request.rgid)); cmd.request.trigger_irq = trigger_irq; cmd.request.event_code = event_code; cmd.request.info = info;
include/linux/ism.h +6 -10
@@ -11,6 +11,7 @@#include <linux/device.h> #include <linux/workqueue.h> +#include <linux/uuid.h> /* The remote peer rgid can use dmb_tok to write into this buffer. */ struct ism_dmb {
@@ -24,11 +25,6 @@ struct ism_dmb {dma_addr_t dma_addr; }; -struct smcd_gid { -u64 gid; -u64 gid_ext; -}; -struct ism_event {u32 type;u32 code;
@@ -61,7 +57,7 @@ int ism_register_client(struct ism_client *client);int ism_unregister_client (struct ism_client *client);/* Mandatory operations for all ism devices: -* int (*query_remote_gid)(struct ism_dev *dev, struct smcd_gid *rgid, +* int (*query_remote_gid)(struct ism_dev *dev, uuid_t *rgid, * u32 vid_valid, u32 vid); * Query whether remote GID rgid is reachable via this device and this * vlan id. Vlan id is only checked if vid_valid != 0.
@@ -101,14 +97,14 @@ int ism_unregister_client(struct ism_client *client);* Ability to assign dmbs to VLANs is missing * - do we really want / need this? * -* int (*signal_event)(struct ism_dev *dev, struct smcd_gid *rgid, +* int (*signal_event)(struct ism_dev *dev, uuid_t *rgid, * u32 trigger_irq, u32 event_code, u64 info); * Send a control event into the event queue of a remote gid (rgid) * with (1) or without (0) triggering an interrupt at the remote gid. */ struct ism_ops { -int (*query_remote_gid)(struct ism_dev *dev,struct smcd_gid *rgid, +int (*query_remote_gid)(struct ism_dev *dev,uuid_t *rgid,u32 vid_valid,u32 vid);int (*register_dmb)(struct ism_dev *dev,struct ism_dmb *dmb,struct ism_client *client);
@@ -125,7 +121,7 @@ struct ism_ops {int (*del_vlan_id)(struct ism_dev *dev,u64 vlan_id);int (*set_vlan_required)(struct ism_dev *dev);int (*reset_vlan_required)(struct ism_dev *dev); -int (*signal_event)(struct ism_dev *dev,struct smcd_gid *rgid, +int (*signal_event)(struct ism_dev *dev,uuid_t *rgid,u32 trigger_irq,u32 event_code,u64 info); };
@@ -150,7 +146,7 @@ struct ism_dev {dma_addr_t ieq_dma_addr;struct device dev; -struct smcd_gid gid; +uuid_t gid;int ieq_idx;struct ism_client *subs[MAX_CLIENTS ];
include/net/smc.h +6 -6
@@ -15,7 +15,7 @@#include <linux/spinlock.h> #include <linux/types.h> #include <linux/wait.h> -#include "linux/ism.h" +#include <linux/ism.h> struct sock ;
@@ -30,11 +30,11 @@ struct smc_hashinfo {struct smcd_dev ; -//struct smcd_gid { -// u64 gid; -// u64 gid_ext; -//}; -// +struct smcd_gid { +u64 gid; +u64 gid_ext; +}; +struct smcd_ops {int (*query_remote_gid)(struct smcd_dev *dev,struct smcd_gid *rgid,u32 vid_valid,u32 vid);
net/smc/smc_ism.c +8 -5
@@ -485,8 +485,10 @@ static int smcd_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid, u32 vid_valid, u32 vid) {struct ism_dev *ism = smcd->priv; +uuid_t ism_rgid; -return ism->ops->query_remote_gid (ism, rgid, vid_valid, vid); +copy_to_ismgid (&ism_rgid, rgid); +return ism->ops->query_remote_gid (ism, &ism_rgid, vid_valid, vid); } static int smcd_register_dmb(struct smcd_dev *smcd, struct ism_dmb *dmb,
@@ -536,9 +538,11 @@ static int smcd_signal_ieq(struct smcd_dev *smcd, struct smcd_gid *rgid, u32 trigger_irq, u32 event_code, u64 info) { struct ism_dev *ism = smcd->priv; +uuid_t ism_rgid; -return ism->ops->signal_event (ism, rgid, - trigger_irq, event_code, info); +copy_to_ismgid (&ism_rgid, rgid); +return ism->ops->signal_event (ism, &ism_rgid, trigger_irq, + event_code, info); } static int smcd_move(struct smcd_dev *smcd, u64 dmb_tok, unsigned int idx,
@@ -560,8 +564,7 @@ static void smcd_get_local_gid(struct smcd_dev *smcd, {struct ism_dev *ism = smcd->priv; - smcd_gid->gid = ism->gid.gid; - smcd_gid->gid_ext = ism->gid.gid_ext; +copy_to_smcdgid (smcd_gid, &ism->gid); }static u16 smcd_get_chid (struct smcd_dev *smcd)
net/smc/smc_ism.h +21 -0
@@ -12,6 +12,7 @@#include <linux/uio.h> #include <linux/types.h> #include <linux/mutex.h> +#include <linux/ism.h> #include "smc.h"
@@ -94,4 +95,24 @@ static inline bool smc_ism_is_loopback(struct smcd_dev *smcd)return (smcd->ops->get_chid (smcd) ==0xFFFF ); } +static inline void copy_to_smcdgid (struct smcd_gid *sgid,uuid_t *igid) +{ +__be64 temp; + +memcpy (&temp, igid,sizeof (sgid->gid)); + sgid->gid =ntohll (temp); +memcpy (&temp, igid +sizeof (sgid->gid),sizeof (sgid->gid_ext)); + sgid->gid_ext =ntohll (temp); +} + +static inline void copy_to_ismgid (uuid_t *igid,struct smcd_gid *sgid) +{ +__be64 temp; + + temp =htonll (sgid->gid); +memcpy (igid, &temp,sizeof (sgid->gid)); + temp =htonll (sgid->gid_ext); +memcpy (igid +sizeof (sgid->gid), &temp,sizeof (sgid->gid_ext)); +} +#endif
Signature · 1 line
2.45.2
Simon Horman
reviewerRe: [RFC net-next 3/7] net/ism: Use uuid_t for ISM GID
Quoted 28 lines · Alexandra Winter
SMC uses 64 Bit and 128 Bit Global Identifiers (GIDs) that need to be sent via the SMC protocol.
When integers are used network endianness and host endianness need to be considered.Avoid this in the ISM layer by using
uuid_tbyte arrays. Follow on patches could do the same change for SMC, for now conversion helper functions are introduced.ISM-vPCI devices provide 64 Bit GIDs. Map them to ISM
uuid_tGIDs like this:
_________________________________________
| 64 Bit ISM-vPCI GID | 00000000_00000000 |
-----------------------------------------
If interpreted as UUID, this would be interpreted as th UIID variant, that is reserved for NCS backward compatibility. So it will not collide with UUIDs that were generated according to the standard.Future ISM devices, shall use real UUIDs as 128 Bit GIDs.
Note:
- In this RFC patchsmcd_gidis now moved back tosmc.h, future patchset should avoid that.
-ism_dmbandism_eventstructs still contain 64 Bit rgid and info fields. A future patch could change them touuid_tgids. This does not break anything, becauseism_loopbackdoes not use them.Signed-off-by Alexandra Winter <[email protected]>
...
Quoted 13 lines
net/smc/smc_ism.h +1
-0@@ -12,6 +12,7 @@#include <linux/uio.h> #include <linux/types.h> #include <linux/mutex.h> +#include <linux/ism.h> #include "smc.h"
net/smc/smc_ism.h static inline bool smc_ism_is_loopback(struct smcd_dev *smcd) · line 104
Quoted 10 lines
net/smc/smc_ism.h +7
-0@@ -94,4 +95,24 @@ static inline bool smc_ism_is_loopback(struct smcd_dev *smcd)return (smcd->ops->get_chid (smcd) ==0xFFFF ); } +static inline void copy_to_smcdgid (struct smcd_gid *sgid,uuid_t *igid) +{ +__be64 temp; + +memcpy (&temp, igid,sizeof (sgid->gid)); + sgid->gid =ntohll (temp); +memcpy (&temp, igid +sizeof (sgid->gid),sizeof (sgid->gid_ext));
Hi Alexandra,
The stride of the pointer arithmetic is the width of igid so this write will be at an offset of:
sizeof(igid) + sizeof(sgid->gid) = 128 bytes
Which is beyond the end of *igid.
I think the desired operation is to write at an offset of 8 bytes, so perhaps this is a way to achieve that, as the bi field is a 16 byte array of u8:
memcpy(&temp, igid->b + sizeof(sgid->gid), sizeof(sgid->gid_ext));
Flagged by W=1 builds with gcc-14 and clang-19, and by Smatch.
net/smc/smc_ism.h static inline bool smc_ism_is_loopback(struct smcd_dev *smcd) · line 115
Quoted 11 lines
net/smc/smc_ism.h +11
-0+ sgid->gid_ext =ntohll (temp); +} + +static inline void copy_to_ismgid (uuid_t *igid,struct smcd_gid *sgid) +{ +__be64 temp; + + temp =htonll (sgid->gid); +memcpy (igid, &temp,sizeof (sgid->gid)); + temp =htonll (sgid->gid_ext); +memcpy (igid +sizeof (sgid->gid), &temp,sizeof (sgid->gid_ext));
I believe there is a similar problem here too.
Quoted 6 lines
net/smc/smc_ism.h +2
-0+} +#endif Signature · 1 line
2.45.2
Alexandra Winter
maintainerRe: [RFC net-next 3/7] net/ism: Use uuid_t for ISM GID
Quoted 9 lines · Simon Horman
net/smc/smc_ism.h static inline bool smc_ism_is_loopback(struct smcd_dev *smcd) · line 104
Quoted 7 lines
net/smc/smc_ism.h +7
-0+static inline void copy_to_smcdgid (struct smcd_gid *sgid,uuid_t *igid) +{ +__be64 temp; + +memcpy (&temp, igid,sizeof (sgid->gid)); + sgid->gid =ntohll (temp); +memcpy (&temp, igid +sizeof (sgid->gid),sizeof (sgid->gid_ext));Hi Alexandra,
Quoted 6 lines
The stride of the pointer arithmetic is the width of igid so this write will be at an offset of:
sizeof(igid) + sizeof(sgid->gid) = 128 bytesWhich is beyond the end of *igid.
Duh, what a stupid mistake. Thank you.
Quoted 5 lines
I think the desired operation is to write at an offset of 8 bytes, so perhaps this is a way to achieve that, as the bi field is a 16 byte array of u8:
memcpy(&temp, igid->b + sizeof(sgid->gid), sizeof(sgid->gid_ext));
I propose to keep thememcpy(&temp, (u8 *)igid + sizeof(sgid->gid), sizeof(sgid->gid_ext)); like in the orginal net/smc/smc_loopback.c
Quoted 4 lines
Flagged by W=1 builds with gcc-14 and clang-19, and by Smatch.
Review
Quoted 2 lines
(quoted diff) +2
-0+ sgid->gid_ext = ntohll(temp); +}
I actually overlooked it in my smatch run (too many old warnings), but I cannot get W=1 to flag it. I'll try to improve my setup.
Alexandra Winter
maintainerRFC 4/7 net/ism: Add kernel-doc comments for ism functions
In reply to Alexandra WinterNote that in this RFC this patch is not complete, future versions of this patch need to contain comments for all ism_ops. Especially signal_event() and handle_event() need a good generic description.
--- include/linux/ism.h | 115 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 105 insertions(+), 10 deletions(-)
include/linux/ism.h +105 -10
@@ -13,11 +13,26 @@#include <linux/workqueue.h> #include <linux/uuid.h> -/* The remote peer rgid can use dmb_tok to write into this buffer. */ +/* +* DMB - Direct Memory Buffer +* ========================== +* An ism client provides an DMB as input buffer for a local receiving +* ism device for exactly one (remote) sending ism device. Only this +* sending device can send data into this DMB using move_data(). Sender +* and receiver can be the same device. +* TODO: Alignment and length rules (CPU and DMA). Device specific? +*/ struct ism_dmb { +/* dmb_tok - Token for this dmb +* Used by remote sender to address this dmb. +* Provided by ism fabric in register_dmb(). +* Unique per ism fabric. +*/ u64 dmb_tok; +/* rgid - GID of designated remote sending device */ u64 rgid;u32 dmb_len; +/* sba_idx - Index of this DMB on this receiving device */ u32 sba_idx;u32 vlan_valid;u32 vlan_id;
@@ -25,6 +40,8 @@ struct ism_dmb {dma_addr_t dma_addr; }; +/* ISM event structure (currently device type specific) */ +// TODO: Define and describe generic event properties struct ism_event {u32 type;u32 code;
@@ -33,38 +50,89 @@ struct ism_event {u64 info; }; +//TODO: use enum typedef #define ISM_EVENT_DMB 0#define ISM_EVENT_GID 1#define ISM_EVENT_SWR 2struct ism_dev ; +/* +* ISM clients +* =========== +* All ism clients have access to all ism devices +* and must provide the following functions to be called by +* ism device drivers: +*/ struct ism_client { +/* client name for logging and debugging purposes */ const char *name; +/** +* add() - add an ism device +* @dev: device that was added +* +* Will be called during ism_register_client() for all existing +* ism devices and whenever a new ism device is registered. +* *dev is valid until ism_client->remove() is called. +*/ void (*add)(struct ism_dev *dev); +/** +* remove() - remove an ism device +* @dev: device to be removed +* +* Will be called whenever an ism device is unregistered. +* Before this call the device is already inactive: It will +* no longer call client handlers. +* The client must not access *dev after this call. +*/ void (*remove)(struct ism_dev *dev); +/** +* handle_event() - Handle control information sent by device +* @dev: device reporting the event +* @event: ism event structure +*/ void (*handle_event)(struct ism_dev *dev,struct ism_event *event); -/* Parameter dmbemask contains a bit vector with updated DMBEs, if sent -* via ism_move_data(). Callback function must handle all active bits -* indicated by dmbemask. +/** +* handle_irq() - Handle signalling of a DMB +* @dev: device owns the dmb +* @bit: sba_idx=idx of the ism_dmb that got signalled +* TODO: Pass a priv pointer to ism_dmb instead of 'bit'(?) +* @dmbemask: ism signalling mask of the dmb +* +* Handle signalling of a dmb that was registered by this client +* for this device. +* The ism device can coalesce multiple signalling triggers into a +* single call of handle_irq(). dmbemask can be used to indicate +* different kinds of triggers. */ void (*handle_irq)(struct ism_dev *dev,unsigned int bit,u16 dmbemask); -/* Private area - don't touch! */ +/* client index - provided by ism layer */ u8 id; };int ism_register_client (struct ism_client *client);int ism_unregister_client (struct ism_client *client); +//TODO: Pair descriptions with functions +/* +* ISM devices +* =========== +*/ /* Mandatory operations for all ism devices: * int (*query_remote_gid)(struct ism_dev *dev, uuid_t *rgid, * u32 vid_valid, u32 vid); * Query whether remote GID rgid is reachable via this device and this * vlan id. Vlan id is only checked if vid_valid != 0. +* Returns 0 if remote gid is reachable. * * int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb, * void *client); -* Register an ism_dmb buffer for this device and this client. +* Allocate and register an ism_dmb buffer for this device and this client. +* The following fields of ism_dmb must be valid: +* rgid, dmb_len, vlan_*; Optionally:requested sba_idx (non-zero) +* Upon return the following fields will be valid: dmb_tok, sba_idx +* cpu_addr, dma_addr (if applicable) +* Returns zero on success * * int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb); * Unregister an ism_dmb buffer
@@ -81,10 +149,15 @@ int ism_unregister_client(struct ism_client *client);* u16 (*get_chid)(struct ism_dev *dev); * Returns ism fabric identifier (channel id) of this device. * Only devices on the same ism fabric can communicate. -* chid is unique per HW system, except for 0xFFFF, which denotes -* an ism_loopback device that can only communicate with itself. -* Use chid for fast negative checks, but only query_remote_gid() -* can give a reliable positive answer. +* chid is unique per HW system. Use chid for fast negative checks, +* but only query_remote_gid() can give a reliable positive answer: +* Different chid: ism is not possible +* Same chid: ism traffic may be possible or not +* (e.g. different HW systems) +* EXCEPTION: A value of 0xFFFF denotes an ism_loopback device +* that can only communicate with itself. Use GID or +* query_remote_gid()to determine whether sender and +* receiver use the same ism_loopback device. * * struct device* (*get_dev)(struct ism_dev *dev); *
@@ -109,6 +182,28 @@ struct ism_ops {int (*register_dmb)(struct ism_dev *dev,struct ism_dmb *dmb,struct ism_client *client);int (*unregister_dmb)(struct ism_dev *dev,struct ism_dmb *dmb); +/** +* move_data() - write into a remote dmb +* @dev: Local sending ism device +* @dmb_tok: Token of the remote dmb +* @idx: signalling index +* @sf: signalling flag; +* if true, idx will be turned on at target ism interrupt mask +* and target device will be signalled, if required. +* @offset: offset within target dmb +* @data: pointer to data to be sent +* @size: length of data to be sent +* +* Use dev to write data of size at offset into a remote dmb +* identified by dmb_tok. Data is moved synchronously, *data can +* be freed when this function returns. +* +* If signalling flag (sf) is true, bit number idx bit will be +* turned on in the ism signalling mask, that belongs to the +* target dmb, and handle_irq() of the ism client that owns this +* dmb will be called, if required. The target device may chose to +* coalesce multiple signalling triggers. +*/ int (*move_data)(struct ism_dev *dev,u64 dmb_tok,unsigned int idx,bool sf,unsigned int offset,void *data,unsigned int size);
Signature · 1 line
2.45.2
Re: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions
Quoted 4 lines · On Wed, 15 Jan 2025 20:55:24 +0100 Alexandra Winter <[email protected]>…
Note that in this RFC this patch is not complete, future versions of this patch need to contain comments for all
ism_ops. Especiallysignal_event()andhandle_event()need a good generic description.
Such notes don't belong to the commit message, but to the inline cover letter IMHO.
Quoted 3 lines
Signed-off-by Alexandra Winter <[email protected]>---
Dust Li
maintainerRe: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions
In reply to Alexandra WinterQuoted 175 lines · Alexandra Winter
Note that in this RFC this patch is not complete, future versions of this patch need to contain comments for all
ism_ops. Especiallysignal_event()andhandle_event()need a good generic description.Signed-off-by Alexandra Winter <[email protected]>---
include/linux/ism.h| 115 ++++++++++++++++++++++++++++++++++++++++----1 file changed, 105 insertions(+), 10 deletions(-)include/linux/ism.h +88
-10@@ -13,11 +13,26 @@#include <linux/workqueue.h> #include <linux/uuid.h> -/* The remote peer rgid can use dmb_tok to write into this buffer. */ +/* +* DMB - Direct Memory Buffer +* ========================== +* An ism client provides an DMB as input buffer for a local receiving +* ism device for exactly one (remote) sending ism device. Only this +* sending device can send data into this DMB using move_data(). Sender +* and receiver can be the same device. +* TODO: Alignment and length rules (CPU and DMA). Device specific? +*/ struct ism_dmb { +/* dmb_tok - Token for this dmb +* Used by remote sender to address this dmb. +* Provided by ism fabric in register_dmb(). +* Unique per ism fabric. +*/ u64 dmb_tok; +/* rgid - GID of designated remote sending device */ u64 rgid;u32 dmb_len; +/* sba_idx - Index of this DMB on this receiving device */ u32 sba_idx;u32 vlan_valid;u32 vlan_id;@@ -25,6 +40,8 @@ struct ism_dmb {dma_addr_t dma_addr; }; +/* ISM event structure (currently device type specific) */ +// TODO: Define and describe generic event properties struct ism_event {u32 type;u32 code;@@ -33,38 +50,89 @@ struct ism_event {u64 info; }; +//TODO: use enum typedef #define ISM_EVENT_DMB 0#define ISM_EVENT_GID 1#define ISM_EVENT_SWR 2struct ism_dev ; +/* +* ISM clients +* =========== +* All ism clients have access to all ism devices +* and must provide the following functions to be called by +* ism device drivers: +*/ struct ism_client { +/* client name for logging and debugging purposes */ const char *name; +/** +* add() - add an ism device +* @dev: device that was added +* +* Will be called during ism_register_client() for all existing +* ism devices and whenever a new ism device is registered. +* *dev is valid until ism_client->remove() is called. +*/ void (*add)(struct ism_dev *dev); +/** +* remove() - remove an ism device +* @dev: device to be removed +* +* Will be called whenever an ism device is unregistered. +* Before this call the device is already inactive: It will +* no longer call client handlers. +* The client must not access *dev after this call. +*/ void (*remove)(struct ism_dev *dev); +/** +* handle_event() - Handle control information sent by device +* @dev: device reporting the event +* @event: ism event structure +*/ void (*handle_event)(struct ism_dev *dev,struct ism_event *event); -/* Parameter dmbemask contains a bit vector with updated DMBEs, if sent -* via ism_move_data(). Callback function must handle all active bits -* indicated by dmbemask. +/** +* handle_irq() - Handle signalling of a DMB +* @dev: device owns the dmb +* @bit: sba_idx=idx of the ism_dmb that got signalled +* TODO: Pass a priv pointer to ism_dmb instead of 'bit'(?) +* @dmbemask: ism signalling mask of the dmb +* +* Handle signalling of a dmb that was registered by this client +* for this device. +* The ism device can coalesce multiple signalling triggers into a +* single call of handle_irq(). dmbemask can be used to indicate +* different kinds of triggers. */ void (*handle_irq)(struct ism_dev *dev,unsigned int bit,u16 dmbemask); -/* Private area - don't touch! */ +/* client index - provided by ism layer */ u8 id; };int ism_register_client (struct ism_client *client);int ism_unregister_client (struct ism_client *client); +//TODO: Pair descriptions with functions +/* +* ISM devices +* =========== +*/ /* Mandatory operations for all ism devices: * int (*query_remote_gid)(struct ism_dev *dev, uuid_t *rgid, * u32 vid_valid, u32 vid); * Query whether remote GID rgid is reachable via this device and this * vlan id. Vlan id is only checked if vid_valid != 0. +* Returns 0 if remote gid is reachable. * * int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb, * void *client); -* Register an ism_dmb buffer for this device and this client. +* Allocate and register an ism_dmb buffer for this device and this client. +* The following fields of ism_dmb must be valid: +* rgid, dmb_len, vlan_*; Optionally:requested sba_idx (non-zero) +* Upon return the following fields will be valid: dmb_tok, sba_idx +* cpu_addr, dma_addr (if applicable) +* Returns zero on success * * int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb); * Unregister an ism_dmb buffer @@ -81,10 +149,15 @@ int ism_unregister_client(struct ism_client *client);* u16 (*get_chid)(struct ism_dev *dev); * Returns ism fabric identifier (channel id) of this device. * Only devices on the same ism fabric can communicate. -* chid is unique per HW system, except for 0xFFFF, which denotes -* an ism_loopback device that can only communicate with itself. -* Use chid for fast negative checks, but only query_remote_gid() -* can give a reliable positive answer. +* chid is unique per HW system. Use chid for fast negative checks, +* but only query_remote_gid() can give a reliable positive answer: +* Different chid: ism is not possible +* Same chid: ism traffic may be possible or not +* (e.g. different HW systems) +* EXCEPTION: A value of 0xFFFF denotes an ism_loopback device +* that can only communicate with itself. Use GID or +* query_remote_gid()to determine whether sender and +* receiver use the same ism_loopback device. * * struct device* (*get_dev)(struct ism_dev *dev); * @@ -109,6 +182,28 @@ struct ism_ops {int (*register_dmb)(struct ism_dev *dev,struct ism_dmb *dmb,struct ism_client *client);int (*unregister_dmb)(struct ism_dev *dev,struct ism_dmb *dmb); +/** +* move_data() - write into a remote dmb +* @dev: Local sending ism device +* @dmb_tok: Token of the remote dmb +* @idx: signalling index
include/linux/ism.h struct ism_ops { · line 199
Quoted 10 lines
include/linux/ism.h +10
-0+* @sf: signalling flag; +* if true, idx will be turned on at target ism interrupt mask +* and target device will be signalled, if required. +* @offset: offset within target dmb +* @data: pointer to data to be sent +* @size: length of data to be sent +* +* Use dev to write data of size at offset into a remote dmb +* identified by dmb_tok. Data is moved synchronously, *data can +* be freed when this function returns.
When considering the API, I found this comment may be incorrect.
IIUC, in copy mode for PCI ISM devices, the CPU only tells the device to perform a DMA copy. As a result, when this function returns, the device may not have completed the DMA copy.
In zero-copy mode for loopback, the source and destination share the same buffer. If the source rewrites the buffer, the destination may encounter corrupted data. The source should only reuse the data after the destination has finished reading it.
Best regards,
Dust
Quoted 13 lines
include/linux/ism.h +7
-0+* +* If signalling flag (sf) is true, bit number idx bit will be +* turned on in the ism signalling mask, that belongs to the +* target dmb, and handle_irq() of the ism client that owns this +* dmb will be called, if required. The target device may chose to +* coalesce multiple signalling triggers. +*/ int (*move_data)(struct ism_dev *dev,u64 dmb_tok,unsigned int idx,bool sf,unsigned int offset,void *data,unsigned int size);Signature · 1 line
2.45.2
Alexandra Winter
maintainerRe: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions
Quoted 17 lines · Dust Li
Quoted 5 lines
include/linux/ism.h +5
-0+/** +* move_data() - write into a remote dmb +* @dev: Local sending ism device +* @dmb_tok: Token of the remote dmb +* @idx: signalling index include/linux/ism.h struct ism_ops { · line 199
Quoted 10 lines
include/linux/ism.h +10
-0+* @sf: signalling flag; +* if true, idx will be turned on at target ism interrupt mask +* and target device will be signalled, if required. +* @offset: offset within target dmb +* @data: pointer to data to be sent +* @size: length of data to be sent +* +* Use dev to write data of size at offset into a remote dmb +* identified by dmb_tok. Data is moved synchronously, *data can +* be freed when this function returns. When considering the API, I found this comment may be incorrect.
Quoted 4 lines
IIUC, in copy mode for PCI ISM devices, the CPU only tells the device to perform a DMA copy. As a result, when this function returns, the device may not have completed the DMA copy.
No, it is actually one of the properties of ISM vPCI that the data is moved synchronously inside the move_data() function. (on PCI layer the data is moved inside the __zpci_store_block() command). Obviously for loopback move_data() is also synchornous.
SMC-D does not make use of it, instead they re-use the same conn->sndbuf_desc for the lifetime of a connection.
Quoted 5 lines
In zero-copy mode for loopback, the source and destination share the same buffer. If the source rewrites the buffer, the destination may encounter corrupted data. The source should only reuse the data after the destination has finished reading it.
That is true independent of the question, whether the move is synchronous or not.
It is the clients' responsibility to make sure a sender does not overwrite unread data. SMC uses the write-pointers and read-pointer for that.
Quoted 14 lines
Best regards,
DustQuoted 11 lines
include/linux/ism.h +7
-0+* +* If signalling flag (sf) is true, bit number idx bit will be +* turned on in the ism signalling mask, that belongs to the +* target dmb, and handle_irq() of the ism client that owns this +* dmb will be called, if required. The target device may chose to +* coalesce multiple signalling triggers. +*/ int (*move_data)(struct
ism_dev*dev, u64dmb_tok, unsigned int idx,
bool sf, unsigned int offset, void *data, unsigned int size);
Re: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions
Quoted 25 lines · Alexandra Winter
Quoted 17 lines · Dust Li
Quoted 5 lines
include/linux/ism.h +5
-0+/** +* move_data() - write into a remote dmb +* @dev: Local sending ism device +* @dmb_tok: Token of the remote dmb +* @idx: signalling index include/linux/ism.h struct ism_ops { · line 199
Quoted 10 lines
include/linux/ism.h +10
-0+* @sf: signalling flag; +* if true, idx will be turned on at target ism interrupt mask +* and target device will be signalled, if required. +* @offset: offset within target dmb +* @data: pointer to data to be sent +* @size: length of data to be sent +* +* Use dev to write data of size at offset into a remote dmb +* identified by dmb_tok. Data is moved synchronously, *data can +* be freed when this function returns. When considering the API, I found this comment may be incorrect.
Quoted 4 lines
IIUC, in copy mode for PCI ISM devices, the CPU only tells the device to perform a DMA copy. As a result, when this function returns, the device may not have completed the DMA copy.
Quoted 4 lines
No, it is actually one of the properties of ISM vPCI that the data is moved synchronously inside the
move_data()function. (on PCI layer the data is moved inside the__zpci_store_block()command). Obviously for loopbackmove_data()is also synchornous.
That is true for the IBM ISM vPCI device but maybe we should design the API also for future PCI devices that do not move data synchronously.
Quoted 32 lines
SMC-D does not make use of it, instead they re-use the same
conn->sndbuf_descfor the lifetime of a connection.Quoted 5 lines
In zero-copy mode for loopback, the source and destination share the same buffer. If the source rewrites the buffer, the destination may encounter corrupted data. The source should only reuse the data after the destination has finished reading it.
That is true independent of the question, whether the move is synchronous or not.
It is the clients' responsibility to make sure a sender does not overwrite unread data. SMC uses the write-pointers and read-pointer for that.Quoted 14 lines
Best regards,
DustQuoted 11 lines
include/linux/ism.h +7
-0+* +* If signalling flag (sf) is true, bit number idx bit will be +* turned on in the ism signalling mask, that belongs to the +* target dmb, and handle_irq() of the ism client that owns this +* dmb will be called, if required. The target device may chose to +* coalesce multiple signalling triggers. +*/ int (*move_data)(struct
ism_dev*dev, u64dmb_tok, unsigned int idx,
bool sf, unsigned int offset, void *data, unsigned int size);
Alexandra Winter
maintainerRe: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions
Quoted 26 lines · Julian Ruess
Quoted 25 lines · Alexandra Winter
Quoted 21 lines · Dust Li
Quoted 5 lines
include/linux/ism.h +5
-0+/** +* move_data() - write into a remote dmb +* @dev: Local sending ism device +* @dmb_tok: Token of the remote dmb +* @idx: signalling index include/linux/ism.h struct ism_ops { · line 199
Quoted 10 lines
include/linux/ism.h +10
-0+* @sf: signalling flag; +* if true, idx will be turned on at target ism interrupt mask +* and target device will be signalled, if required. +* @offset: offset within target dmb +* @data: pointer to data to be sent +* @size: length of data to be sent +* +* Use dev to write data of size at offset into a remote dmb +* identified by dmb_tok. Data is moved synchronously, *data can +* be freed when this function returns. When considering the API, I found this comment may be incorrect.
IIUC, in copy mode for PCI ISM devices, the CPU only tells the device to perform a DMA copy. As a result, when this function returns, the device may not have completed the DMA copy.
Quoted 9 lines
Quoted 4 lines
No, it is actually one of the properties of ISM vPCI that the data is moved synchronously inside the
move_data()function. (on PCI layer the data is moved inside the__zpci_store_block()command). Obviously for loopbackmove_data()is also synchornous.That is true for the IBM ISM vPCI device but maybe we should design the API also for future PCI devices that do not move data synchronously.
An API should always be extendable
Quoted 33 lines
Quoted 32 lines
SMC-D does not make use of it, instead they re-use the same
conn->sndbuf_descfor the lifetime of a connection.Quoted 5 lines
In zero-copy mode for loopback, the source and destination share the same buffer. If the source rewrites the buffer, the destination may encounter corrupted data. The source should only reuse the data after the destination has finished reading it.
That is true independent of the question, whether the move is synchronous or not.
It is the clients' responsibility to make sure a sender does not overwrite unread data. SMC uses the write-pointers and read-pointer for that.Quoted 14 lines
Best regards,
DustQuoted 11 lines
include/linux/ism.h +7
-0+* +* If signalling flag (sf) is true, bit number idx bit will be +* turned on in the ism signalling mask, that belongs to the +* target dmb, and handle_irq() of the ism client that owns this +* dmb will be called, if required. The target device may chose to +* coalesce multiple signalling triggers. +*/ int (*move_data)(struct
ism_dev*dev, u64dmb_tok, unsigned int idx,
bool sf, unsigned int offset, void *data, unsigned int size);
Re: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions
In reply to Dust LiQuoted 189 lines · Dust Li
Quoted 175 lines · Alexandra Winter
Note that in this RFC this patch is not complete, future versions of this patch need to contain comments for all
ism_ops. Especiallysignal_event()andhandle_event()need a good generic description.Signed-off-by Alexandra Winter <[email protected]>---
include/linux/ism.h| 115 ++++++++++++++++++++++++++++++++++++++++----1 file changed, 105 insertions(+), 10 deletions(-)include/linux/ism.h +88
-10@@ -13,11 +13,26 @@#include <linux/workqueue.h> #include <linux/uuid.h> -/* The remote peer rgid can use dmb_tok to write into this buffer. */ +/* +* DMB - Direct Memory Buffer +* ========================== +* An ism client provides an DMB as input buffer for a local receiving +* ism device for exactly one (remote) sending ism device. Only this +* sending device can send data into this DMB using move_data(). Sender +* and receiver can be the same device. +* TODO: Alignment and length rules (CPU and DMA). Device specific? +*/ struct ism_dmb { +/* dmb_tok - Token for this dmb +* Used by remote sender to address this dmb. +* Provided by ism fabric in register_dmb(). +* Unique per ism fabric. +*/ u64 dmb_tok; +/* rgid - GID of designated remote sending device */ u64 rgid;u32 dmb_len; +/* sba_idx - Index of this DMB on this receiving device */ u32 sba_idx;u32 vlan_valid;u32 vlan_id;@@ -25,6 +40,8 @@ struct ism_dmb {dma_addr_t dma_addr; }; +/* ISM event structure (currently device type specific) */ +// TODO: Define and describe generic event properties struct ism_event {u32 type;u32 code;@@ -33,38 +50,89 @@ struct ism_event {u64 info; }; +//TODO: use enum typedef #define ISM_EVENT_DMB 0#define ISM_EVENT_GID 1#define ISM_EVENT_SWR 2struct ism_dev ; +/* +* ISM clients +* =========== +* All ism clients have access to all ism devices +* and must provide the following functions to be called by +* ism device drivers: +*/ struct ism_client { +/* client name for logging and debugging purposes */ const char *name; +/** +* add() - add an ism device +* @dev: device that was added +* +* Will be called during ism_register_client() for all existing +* ism devices and whenever a new ism device is registered. +* *dev is valid until ism_client->remove() is called. +*/ void (*add)(struct ism_dev *dev); +/** +* remove() - remove an ism device +* @dev: device to be removed +* +* Will be called whenever an ism device is unregistered. +* Before this call the device is already inactive: It will +* no longer call client handlers. +* The client must not access *dev after this call. +*/ void (*remove)(struct ism_dev *dev); +/** +* handle_event() - Handle control information sent by device +* @dev: device reporting the event +* @event: ism event structure +*/ void (*handle_event)(struct ism_dev *dev,struct ism_event *event); -/* Parameter dmbemask contains a bit vector with updated DMBEs, if sent -* via ism_move_data(). Callback function must handle all active bits -* indicated by dmbemask. +/** +* handle_irq() - Handle signalling of a DMB +* @dev: device owns the dmb +* @bit: sba_idx=idx of the ism_dmb that got signalled +* TODO: Pass a priv pointer to ism_dmb instead of 'bit'(?) +* @dmbemask: ism signalling mask of the dmb +* +* Handle signalling of a dmb that was registered by this client +* for this device. +* The ism device can coalesce multiple signalling triggers into a +* single call of handle_irq(). dmbemask can be used to indicate +* different kinds of triggers. */ void (*handle_irq)(struct ism_dev *dev,unsigned int bit,u16 dmbemask); -/* Private area - don't touch! */ +/* client index - provided by ism layer */ u8 id; };int ism_register_client (struct ism_client *client);int ism_unregister_client (struct ism_client *client); +//TODO: Pair descriptions with functions +/* +* ISM devices +* =========== +*/ /* Mandatory operations for all ism devices: * int (*query_remote_gid)(struct ism_dev *dev, uuid_t *rgid, * u32 vid_valid, u32 vid); * Query whether remote GID rgid is reachable via this device and this * vlan id. Vlan id is only checked if vid_valid != 0. +* Returns 0 if remote gid is reachable. * * int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb, * void *client); -* Register an ism_dmb buffer for this device and this client. +* Allocate and register an ism_dmb buffer for this device and this client. +* The following fields of ism_dmb must be valid: +* rgid, dmb_len, vlan_*; Optionally:requested sba_idx (non-zero) +* Upon return the following fields will be valid: dmb_tok, sba_idx +* cpu_addr, dma_addr (if applicable) +* Returns zero on success * * int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb); * Unregister an ism_dmb buffer @@ -81,10 +149,15 @@ int ism_unregister_client(struct ism_client *client);* u16 (*get_chid)(struct ism_dev *dev); * Returns ism fabric identifier (channel id) of this device. * Only devices on the same ism fabric can communicate. -* chid is unique per HW system, except for 0xFFFF, which denotes -* an ism_loopback device that can only communicate with itself. -* Use chid for fast negative checks, but only query_remote_gid() -* can give a reliable positive answer. +* chid is unique per HW system. Use chid for fast negative checks, +* but only query_remote_gid() can give a reliable positive answer: +* Different chid: ism is not possible +* Same chid: ism traffic may be possible or not +* (e.g. different HW systems) +* EXCEPTION: A value of 0xFFFF denotes an ism_loopback device +* that can only communicate with itself. Use GID or +* query_remote_gid()to determine whether sender and +* receiver use the same ism_loopback device. * * struct device* (*get_dev)(struct ism_dev *dev); * @@ -109,6 +182,28 @@ struct ism_ops {int (*register_dmb)(struct ism_dev *dev,struct ism_dmb *dmb,struct ism_client *client);int (*unregister_dmb)(struct ism_dev *dev,struct ism_dmb *dmb); +/** +* move_data() - write into a remote dmb +* @dev: Local sending ism device +* @dmb_tok: Token of the remote dmb +* @idx: signalling index include/linux/ism.h struct ism_ops { · line 199
Quoted 10 lines
include/linux/ism.h +10
-0+* @sf: signalling flag; +* if true, idx will be turned on at target ism interrupt mask +* and target device will be signalled, if required. +* @offset: offset within target dmb +* @data: pointer to data to be sent +* @size: length of data to be sent +* +* Use dev to write data of size at offset into a remote dmb +* identified by dmb_tok. Data is moved synchronously, *data can +* be freed when this function returns. When considering the API, I found this comment may be incorrect.
Quoted 3 lines
IIUC, in copy mode for PCI ISM devices, the CPU only tells the device to perform a DMA copy. As a result, when this function returns, the device may not have completed the DMA copy.
For the s390 ISM device the statement is true. The move_data() function does a PCI Store Block instruction which is both the write on the sender side but also synchronously acts as the devices DMA write on the receiver side. So when the PCI Store Block instruction completes the data has been cache coherently written to the receiver DMB. And yes full synchronicity would be impossible with the posted writes of real PCIe.
That said when it comes to API design I think you have a great point here in that we need to decide if this synchronicity should be baked into the move_data() API. I think we instead want to only guarantee a weaker rule. That is the source buffer can be re-used after the move. This to me is also aligned with the word "move" here in that the data has been moved after the call not registered to be moved or such. This could be achieved with a real PCIe device by copying the data or by waiting on completion. If we ever get devices which need to wait on completion it may indeed be better to have a separate completion step in the API too. Then again I think the concept of having a single "move data" step is somewhat central to ISM and I'd hate to lose that simplicity.
I've been thinking also about a possible copy mode in a virtio-ism. That could be useful if we wanted to use virtio-ism between memory partitioned guests, or if one wanted to transparently proxy virtio-ism over s390 ISM to span multiple KVM hosts. And I think such a mode could still work with a single "move data" step and I'd love to have that in any future virtio-ism spec.
Quoted 5 lines
In zero-copy mode for loopback, the source and destination share the same buffer. If the source rewrites the buffer, the destination may encounter corrupted data. The source should only reuse the data after the destination has finished reading it.
I think there are two potential overwrite scenarios here.
1. The sender re-uses the source data buffer i.e. the @data buffer of the move_data() call. On s390 ISM this is fine because the data was copied out and into the destination DMB during the call. This could typically become an issue if the device DMA reads directly from @data after the move_data() call completed.
2. The sender does subsequent move_data() overwriting data in the destination DMB before the receiver has read the data. This can happen on s390 ISM too and needs to be prevented by DMB access rules.
For the move_data() call I think that even in a "shared i.e. same page DMB" scenario move_data() must still do a copy out of the @data buffer into the shared DMB. Otherwise it really wouldn't "move" data and it would be a very weird API since @data is just a buffer not some kind of descriptor. In other words I think scenario 1 shouldn't be possible in either copy or shared DMB mode by the semantics of move_data().
Quoted 14 lines
Quoted 14 lines
include/linux/ism.h +7
-0+* +* If signalling flag (sf) is true, bit number idx bit will be +* turned on in the ism signalling mask, that belongs to the +* target dmb, and handle_irq() of the ism client that owns this +* dmb will be called, if required. The target device may chose to +* coalesce multiple signalling triggers. +*/ int (*move_data)(struct ism_dev *dev,u64 dmb_tok,unsigned int idx,bool sf,unsigned int offset,void *data,unsigned int size);Signature · 1 line
2.45.2
Dust Li
maintainerRe: [RFC net-next 4/7] net/ism: Add kernel-doc comments for ism functions
Quoted 194 lines · Niklas Schnelle
Quoted 189 lines · Dust Li
Quoted 175 lines · Alexandra Winter
Note that in this RFC this patch is not complete, future versions of this patch need to contain comments for all
ism_ops. Especiallysignal_event()andhandle_event()need a good generic description.Signed-off-by Alexandra Winter <[email protected]>---
include/linux/ism.h| 115 ++++++++++++++++++++++++++++++++++++++++----1 file changed, 105 insertions(+), 10 deletions(-)include/linux/ism.h +88
-10@@ -13,11 +13,26 @@#include <linux/workqueue.h> #include <linux/uuid.h> -/* The remote peer rgid can use dmb_tok to write into this buffer. */ +/* +* DMB - Direct Memory Buffer +* ========================== +* An ism client provides an DMB as input buffer for a local receiving +* ism device for exactly one (remote) sending ism device. Only this +* sending device can send data into this DMB using move_data(). Sender +* and receiver can be the same device. +* TODO: Alignment and length rules (CPU and DMA). Device specific? +*/ struct ism_dmb { +/* dmb_tok - Token for this dmb +* Used by remote sender to address this dmb. +* Provided by ism fabric in register_dmb(). +* Unique per ism fabric. +*/ u64 dmb_tok; +/* rgid - GID of designated remote sending device */ u64 rgid;u32 dmb_len; +/* sba_idx - Index of this DMB on this receiving device */ u32 sba_idx;u32 vlan_valid;u32 vlan_id;@@ -25,6 +40,8 @@ struct ism_dmb {dma_addr_t dma_addr; }; +/* ISM event structure (currently device type specific) */ +// TODO: Define and describe generic event properties struct ism_event {u32 type;u32 code;@@ -33,38 +50,89 @@ struct ism_event {u64 info; }; +//TODO: use enum typedef #define ISM_EVENT_DMB 0#define ISM_EVENT_GID 1#define ISM_EVENT_SWR 2struct ism_dev ; +/* +* ISM clients +* =========== +* All ism clients have access to all ism devices +* and must provide the following functions to be called by +* ism device drivers: +*/ struct ism_client { +/* client name for logging and debugging purposes */ const char *name; +/** +* add() - add an ism device +* @dev: device that was added +* +* Will be called during ism_register_client() for all existing +* ism devices and whenever a new ism device is registered. +* *dev is valid until ism_client->remove() is called. +*/ void (*add)(struct ism_dev *dev); +/** +* remove() - remove an ism device +* @dev: device to be removed +* +* Will be called whenever an ism device is unregistered. +* Before this call the device is already inactive: It will +* no longer call client handlers. +* The client must not access *dev after this call. +*/ void (*remove)(struct ism_dev *dev); +/** +* handle_event() - Handle control information sent by device +* @dev: device reporting the event +* @event: ism event structure +*/ void (*handle_event)(struct ism_dev *dev,struct ism_event *event); -/* Parameter dmbemask contains a bit vector with updated DMBEs, if sent -* via ism_move_data(). Callback function must handle all active bits -* indicated by dmbemask. +/** +* handle_irq() - Handle signalling of a DMB +* @dev: device owns the dmb +* @bit: sba_idx=idx of the ism_dmb that got signalled +* TODO: Pass a priv pointer to ism_dmb instead of 'bit'(?) +* @dmbemask: ism signalling mask of the dmb +* +* Handle signalling of a dmb that was registered by this client +* for this device. +* The ism device can coalesce multiple signalling triggers into a +* single call of handle_irq(). dmbemask can be used to indicate +* different kinds of triggers. */ void (*handle_irq)(struct ism_dev *dev,unsigned int bit,u16 dmbemask); -/* Private area - don't touch! */ +/* client index - provided by ism layer */ u8 id; };int ism_register_client (struct ism_client *client);int ism_unregister_client (struct ism_client *client); +//TODO: Pair descriptions with functions +/* +* ISM devices +* =========== +*/ /* Mandatory operations for all ism devices: * int (*query_remote_gid)(struct ism_dev *dev, uuid_t *rgid, * u32 vid_valid, u32 vid); * Query whether remote GID rgid is reachable via this device and this * vlan id. Vlan id is only checked if vid_valid != 0. +* Returns 0 if remote gid is reachable. * * int (*register_dmb)(struct ism_dev *dev, struct ism_dmb *dmb, * void *client); -* Register an ism_dmb buffer for this device and this client. +* Allocate and register an ism_dmb buffer for this device and this client. +* The following fields of ism_dmb must be valid: +* rgid, dmb_len, vlan_*; Optionally:requested sba_idx (non-zero) +* Upon return the following fields will be valid: dmb_tok, sba_idx +* cpu_addr, dma_addr (if applicable) +* Returns zero on success * * int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb); * Unregister an ism_dmb buffer @@ -81,10 +149,15 @@ int ism_unregister_client(struct ism_client *client);* u16 (*get_chid)(struct ism_dev *dev); * Returns ism fabric identifier (channel id) of this device. * Only devices on the same ism fabric can communicate. -* chid is unique per HW system, except for 0xFFFF, which denotes -* an ism_loopback device that can only communicate with itself. -* Use chid for fast negative checks, but only query_remote_gid() -* can give a reliable positive answer. +* chid is unique per HW system. Use chid for fast negative checks, +* but only query_remote_gid() can give a reliable positive answer: +* Different chid: ism is not possible +* Same chid: ism traffic may be possible or not +* (e.g. different HW systems) +* EXCEPTION: A value of 0xFFFF denotes an ism_loopback device +* that can only communicate with itself. Use GID or +* query_remote_gid()to determine whether sender and +* receiver use the same ism_loopback device. * * struct device* (*get_dev)(struct ism_dev *dev); * @@ -109,6 +182,28 @@ struct ism_ops {int (*register_dmb)(struct ism_dev *dev,struct ism_dmb *dmb,struct ism_client *client);int (*unregister_dmb)(struct ism_dev *dev,struct ism_dmb *dmb); +/** +* move_data() - write into a remote dmb +* @dev: Local sending ism device +* @dmb_tok: Token of the remote dmb +* @idx: signalling index include/linux/ism.h struct ism_ops { · line 199
Quoted 10 lines
include/linux/ism.h +10
-0+* @sf: signalling flag; +* if true, idx will be turned on at target ism interrupt mask +* and target device will be signalled, if required. +* @offset: offset within target dmb +* @data: pointer to data to be sent +* @size: length of data to be sent +* +* Use dev to write data of size at offset into a remote dmb +* identified by dmb_tok. Data is moved synchronously, *data can +* be freed when this function returns. When considering the API, I found this comment may be incorrect.
Quoted 3 lines
IIUC, in copy mode for PCI ISM devices, the CPU only tells the device to perform a DMA copy. As a result, when this function returns, the device may not have completed the DMA copy.
Quoted 7 lines
For the s390 ISM device the statement is true. The
move_data()function does a PCI Store Block instruction which is both the write on the sender side but also synchronously acts as the devices DMA write on the receiver side. So when the PCI Store Block instruction completes the data has been cache coherently written to the receiver DMB. And yes full synchronicity would be impossible with the posted writes of real PCIe.
Thank you guys for the detail explaination on s390! I throught it was like normal PCI device. @Niklas, @Alexandra, @Julian
Quoted 13 lines
That said when it comes to API design I think you have a great point here in that we need to decide if this synchronicity should be baked into the
move_data()API. I think we instead want to only guarantee a weaker rule. That is the source buffer can be re-used after the move. This to me is also aligned with the word "move" here in that the data has been moved after the call not registered to be moved or such. This could be achieved with a real PCIe device by copying the data or by waiting on completion. If we ever get devices which need to wait on completion it may indeed be better to have a separate completion step in the API too. Then again I think the concept of having a single "move data" step is somewhat central to ISM and I'd hate to lose that simplicity.
Agree.
For SMC, it already handles zero-copy and copy modes differently. In copy mode, the current API seems appropriate, but for zero-copy mode, we may need to extend it.
Taking the loopback device as an example, SMC requires the move_data() function to copy the CDC into the DMB. However, copying the data from the source to the destination DMB is not necessary. Instead, it needs to signal the peer that the data transfer is complete, so an API for notification is required.
One solution might be to retain the move_data() function and introduce something like a notify() API. Alternatively, if we don't want to extend the API, we could use something like move_data(ismdev, data=NULL, length=0, ..., signal=1).
Quoted 7 lines
I've been thinking also about a possible copy mode in a virtio-ism. That could be useful if we wanted to use virtio-ism between memory partitioned guests, or if one wanted to transparently proxy virtio-ism over s390 ISM to span multiple KVM hosts. And I think such a mode could still work with a single "move data" step and I'd love to have that in any future virtio-ism spec.
I'm not opposed to that.
Quoted 19 lines
Quoted 6 lines
In zero-copy mode for loopback, the source and destination share the same buffer. If the source rewrites the buffer, the destination may encounter corrupted data. The source should only reuse the data after the destination has finished reading it.
I think there are two potential overwrite scenarios here.
1. The sender re-uses the source data buffer i.e. the @data buffer of the
move_data()call. On s390 ISM this is fine because the data was copied out and into the destination DMB during the call. This could typically become an issue if the device DMA reads directly from @data after themove_data()call completed.2. The sender does subsequent
move_data()overwriting data in the destination DMB before the receiver has read the data. This can happen on s390 ISM too and needs to be prevented by DMB access rules.
Quoted 6 lines
For the
move_data()call I think that even in a "shared i.e. same page DMB" scenariomove_data()must still do a copy out of the @data buffer into the shared DMB. Otherwise it really wouldn't "move" data and it would be a very weird API since @data is just a buffer not some kind of descriptor. In other words I think scenario 1 shouldn't be possible in either copy or shared DMB mode by the semantics ofmove_data().
For the shared DMB, I now agree with you. Perhaps we can keep the move_data() behavior consistent with memcpy(3) or memmove(3) and leave the usage to the upper layer.
Best regards,
Dust
Alexandra Winter
maintainerRFC 5/7 net/ism: Move ism_loopback to net/ism
In reply to Alexandra WinterThe first stage of ism_loopback was implemented as part of the SMC module [1]. Now that we have the ism layer, provide access to the ism_loopback device to all ism clients.
Move ism_loopback.* from net/smc to net/ism.
The following changes are required to ism_loopback.c:
- Change ism_lo_move_data() to no longer schedule an smcd receive tasklet, but instead call ism_client->handle_irq().
Note: In this RFC patch ism_loppback is not fully generic.
The smc-d client uses attached buffers, for moves without signalling. and not-attached buffers for moves with signalling. ism_lo_move_data() must not rely on that assumption. ism_lo_move_data() must be able to handle more than one ism client.
In addition the following changes are required to unify ism_loopback and ism_vp:
In ism layer and ism_vpci:ism_loopback is not backed by a pci device, so use dev instead of pdev in ism_dev.
In smc-d:
in smcd_alloc_dev():
- use kernel memory instead of device memory for smcd_dev and smcd->conn. An alternative would be to ask device to alloc the memory.
- use different smcd_ops and max_dmbs for ism_vp and ism_loopback. A future patch can change smc-d to directly use ism_ops instead of smcd_ops.
- use ism dev_name instead of pci dev name for ism_evt_wq name
- allocate an event workqueue for ism_loopback, although it currently does not generate events.
--- drivers/s390/net/ism.h | 6 +- drivers/s390/net/ism_drv.c | 31 ++- include/linux/ism.h | 59 +++++ include/net/smc.h | 4 +- net/ism/Kconfig | 13 ++ net/ism/Makefile | 1 + net/ism/ism_loopback.c | 366 +++++++++++++++++++++++++++++++ net/ism/ism_loopback.h | 59 +++++ net/ism/ism_main.c | 11 +- net/smc/Kconfig | 13 -- net/smc/Makefile | 1 - net/smc/af_smc.c | 12 +- net/smc/smc_ism.c | 108 +++++++--- net/smc/smc_loopback.c | 427 ------------------------------------- net/smc/smc_loopback.h | 60 ------ 15 files changed, 606 insertions(+), 565 deletions(-) create mode 100644 net/ism/ism_loopback.c create mode 100644 net/ism/ism_loopback.h delete mode 100644 net/smc/smc_loopback.c delete mode 100644 net/smc/smc_loopback.h
drivers/s390/net/ism.h +3 -3
@@ -202,7 +202,7 @@ struct ism_sba {static inline void __ism_read_cmd (struct ism_dev *ism,void *data,unsigned long offset,unsigned long len) { -struct zpci_dev *zdev =to_zpci (ism->pdev); +struct zpci_dev *zdev =to_zpci (to_pci_dev (ism->dev.parent));u64 req =ZPCI_CREATE_REQ (zdev->fh,2 ,8 );while (len >0 ) {
@@ -216,7 +216,7 @@ static inline void __ism_read_cmd(struct ism_dev *ism, void *data,static inline void __ism_write_cmd (struct ism_dev *ism,void *data,unsigned long offset,unsigned long len) { -struct zpci_dev *zdev =to_zpci (ism->pdev); +struct zpci_dev *zdev =to_zpci (to_pci_dev (ism->dev.parent));u64 req =ZPCI_CREATE_REQ (zdev->fh,2 , len);if (len)
@@ -226,7 +226,7 @@ static inline void __ism_write_cmd(struct ism_dev *ism, void *data,static inline int __ism_move (struct ism_dev *ism,u64 dmb_req,void *data,unsigned int size) { -struct zpci_dev *zdev =to_zpci (ism->pdev); +struct zpci_dev *zdev =to_zpci (to_pci_dev (ism->dev.parent));u64 req =ZPCI_CREATE_REQ (zdev->fh,0 , size);return __zpci_store_block (data, req, dmb_req);
drivers/s390/net/ism_drv.c +13 -18
@@ -88,7 +88,7 @@ static int register_sba(struct ism_dev *ism)dma_addr_t dma_handle;struct ism_sba *sba; - sba =dma_alloc_coherent (&ism->pdev->dev,PAGE_SIZE , &dma_handle, + sba =dma_alloc_coherent (ism->dev.parent,PAGE_SIZE , &dma_handle,GFP_KERNEL );if (!sba)return -ENOMEM ;
@@ -99,7 +99,7 @@ static int register_sba(struct ism_dev *ism) cmd.request.sba = dma_handle;if (ism_cmd (ism, &cmd)) { -dma_free_coherent (&ism->pdev->dev,PAGE_SIZE , sba, dma_handle); +dma_free_coherent (ism->dev.parent,PAGE_SIZE , sba, dma_handle);return -EIO ; }
@@ -115,7 +115,7 @@ static int register_ieq(struct ism_dev *ism)dma_addr_t dma_handle;struct ism_eq *ieq; - ieq =dma_alloc_coherent (&ism->pdev->dev,PAGE_SIZE , &dma_handle, + ieq =dma_alloc_coherent (ism->dev.parent,PAGE_SIZE , &dma_handle,GFP_KERNEL );if (!ieq)return -ENOMEM ;
@@ -127,7 +127,7 @@ static int register_ieq(struct ism_dev *ism) cmd.request.len =sizeof (*ieq);if (ism_cmd (ism, &cmd)) { -dma_free_coherent (&ism->pdev->dev,PAGE_SIZE , ieq, dma_handle); +dma_free_coherent (ism->dev.parent,PAGE_SIZE , ieq, dma_handle);return -EIO ; }
@@ -149,7 +149,7 @@ static int unregister_sba(struct ism_dev *ism)if (ret && ret !=ISM_ERROR )return -EIO ; -dma_free_coherent (&ism->pdev->dev,PAGE_SIZE , +dma_free_coherent (ism->dev.parent,PAGE_SIZE , ism->sba, ism->sba_dma_addr); ism->sba =NULL ;
@@ -169,7 +169,7 @@ static int unregister_ieq(struct ism_dev *ism)if (ret && ret !=ISM_ERROR )return -EIO ; -dma_free_coherent (&ism->pdev->dev,PAGE_SIZE , +dma_free_coherent (ism->dev.parent,PAGE_SIZE , ism->ieq, ism->ieq_dma_addr); ism->ieq =NULL ;
@@ -216,7 +216,7 @@ static int ism_query_rgid(struct ism_dev *ism, uuid_t *rgid, u32 vid_valid,static void ism_free_dmb (struct ism_dev *ism,struct ism_dmb *dmb) {clear_bit (dmb->sba_idx, ism->sba_bitmap); -dma_unmap_page (&ism->pdev->dev, dmb->dma_addr, dmb->dmb_len, +dma_unmap_page (ism->dev.parent, dmb->dma_addr, dmb->dmb_len,DMA_FROM_DEVICE );folio_put (virt_to_folio (dmb->cpu_addr)); }
@@ -227,7 +227,7 @@ static int ism_alloc_dmb(struct ism_dev *ism, struct ism_dmb *dmb)unsigned long bit;int rc; -if (PAGE_ALIGN (dmb->dmb_len) >dma_get_max_seg_size (&ism->pdev->dev)) +if (PAGE_ALIGN (dmb->dmb_len) >dma_get_max_seg_size (ism->dev.parent))return -EINVAL ;if (!dmb->sba_idx) {
@@ -251,10 +251,10 @@ static int ism_alloc_dmb(struct ism_dev *ism, struct ism_dmb *dmb) } dmb->cpu_addr =folio_address (folio); - dmb->dma_addr =dma_map_page (&ism->pdev->dev, + dmb->dma_addr =dma_map_page (ism->dev.parent,virt_to_page (dmb->cpu_addr),0 , dmb->dmb_len,DMA_FROM_DEVICE ); -if (dma_mapping_error (&ism->pdev->dev, dmb->dma_addr)) { +if (dma_mapping_error (ism->dev.parent, dmb->dma_addr)) { rc = -ENOMEM ; goto out_free; }
@@ -419,10 +419,7 @@ static int ism_supports_v2(void)static u16 ism_get_chid (struct ism_dev *ism) { -if (!ism || !ism->pdev) -return 0 ; - -return to_zpci (ism->pdev)->pchid; +return to_zpci (to_pci_dev (ism->dev.parent))->pchid; }static void ism_handle_event (struct ism_dev *ism)
@@ -499,7 +496,7 @@ static const struct ism_ops ism_vp_ops = {static int ism_dev_init (struct ism_dev *ism) { -struct pci_dev *pdev = ism->pdev; +struct pci_dev *pdev =to_pci_dev (ism->dev.parent);int ret; ret =pci_alloc_irq_vectors (pdev,1 ,1 ,PCI_IRQ_MSI );
@@ -565,7 +562,6 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)spin_lock_init (&ism->lock);dev_set_drvdata (&pdev->dev, ism); - ism->pdev = pdev; ism->dev.parent = &pdev->dev;device_initialize (&ism->dev);dev_set_name (&ism->dev,dev_name (&pdev->dev));
@@ -603,14 +599,13 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)device_del (&ism->dev); err_dev:dev_set_drvdata (&pdev->dev,NULL ); -kfree (ism);return ret; }static void ism_dev_exit (struct ism_dev *ism) { -struct pci_dev *pdev = ism->pdev; +struct pci_dev *pdev =to_pci_dev (ism->dev.parent);unsigned long flags;int i;
include/linux/ism.h +59 -0
83 lines not on this page · raw message
include/net/smc.h +2 -2
@@ -63,8 +63,8 @@ struct smcd_ops {struct smcd_dev {const struct smcd_ops *ops; -void *priv; -void *client; +struct ism_dev *ism; +struct ism_client *client;struct list_head list;spinlock_t lock;struct smc_connection **conn;
net/ism/Kconfig +13 -0
@@ -12,3 +12,16 @@ config ISM To compile as a module choose M. The module name is ism. If unsure, choose N. + +config ISM_LO + bool "intra-OS shortcut with loopback-ism" + depends on ISM + default n + help + ISM_LO enables the creation of an Emulated-ISM device named + loopback-ism which can be used for transferring data + when communication occurs within the same OS. This helps in + convenient testing of ISM clients, since loopback-ism is + independent of architecture or hardware. + + if unsure, say N.
net/ism/Makefile +1 -0
@@ -5,3 +5,4 @@ ism-y += ism_main.o obj-$(CONFIG_ISM) += ism.o +ism-$(CONFIG_ISM_LO) += ism_loopback.o
net/ism/ism_loopback.c new file +366 -0
366 lines not on this page · raw message
net/ism/ism_loopback.h new file +59 -0
59 lines not on this page · raw message
net/ism/ism_main.c +10 -1
@@ -14,6 +14,8 @@#include <linux/err.h> #include <linux/ism.h> +#include "ism_loopback.h" +MODULE_DESCRIPTION ("Internal Shared Memory class" );MODULE_LICENSE ("GPL" );
@@ -148,14 +150,21 @@ EXPORT_SYMBOL_GPL(ism_dev_unregister);static int __init ism_init (void ) { +int rc; +memset (clients,0 ,sizeof (clients)); max_client =0 ; -return 0 ; + rc =ism_loopback_init (); +if (rc) +pr_err ("%s: ism_loopback_init fails with %d\n" , __func__, rc); + +return rc; } static void __exit ism_exit(void) { +ism_loopback_exit (); }module_init (ism_init);
net/smc/Kconfig +0 -13
@@ -20,16 +20,3 @@ config SMC_DIAG smcss. if unsure, say Y. - -config SMC_LO - bool "SMC intra-OS shortcut with loopback-ism" - depends on SMC - default n - help - SMC_LO enables the creation of an Emulated-ISM device named - loopback-ism in SMC and makes use of it for transferring data - when communication occurs within the same OS. This helps in - convenient testing of SMC-D since loopback-ism is independent - of architecture or hardware. - - if unsure, say N.
net/smc/Makefile +0 -1
@@ -6,4 +6,3 @@ smc-y := af_smc.o smc_pnet.o smc_ib.o smc_clc.o smc_core.o smc_wr.o smc_llc.o smc-y += smc_cdc.o smc_tx.o smc_rx.o smc_close.o smc_ism.o smc_netlink.o smc_stats.o smc-y += smc_tracepoint.o smc_inet.o smc-$(CONFIG_SYSCTL) += smc_sysctl.o -smc-$(CONFIG_SMC_LO) += smc_loopback.o
net/smc/af_smc.c +1 -11
@@ -53,7 +53,6 @@#include "smc_stats.h" #include "smc_tracepoint.h" #include "smc_sysctl.h" -#include "smc_loopback.h" #include "smc_inet.h" static DEFINE_MUTEX (smc_server_lgr_pending);/* serialize link group
@@ -3560,16 +3559,10 @@ static int __init smc_init(void) goto out_sock; } - rc =smc_loopback_init (); -if (rc) { -pr_err ("%s: smc_loopback_init fails with %d\n" , __func__, rc); - goto out_ib; - } - rc =tcp_register_ulp (&smc_ulp_ops);if (rc) {pr_err ("%s: tcp_ulp_register fails with %d\n" , __func__, rc); - goto out_lo; + goto out_ib; } rc =smc_inet_init ();if (rc) {
@@ -3580,8 +3573,6 @@ static int __init smc_init(void)return 0 ; out_ulp:tcp_unregister_ulp (&smc_ulp_ops); -out_lo: -smc_loopback_exit (); out_ib:smc_ib_unregister_client (); out_sock:
@@ -3620,7 +3611,6 @@ static void __exit smc_exit(void)tcp_unregister_ulp (&smc_ulp_ops);sock_unregister (PF_SMC );smc_core_exit (); -smc_loopback_exit ();smc_ib_unregister_client ();smc_ism_exit ();destroy_workqueue (smc_close_wq);
net/smc/smc_ism.c +79 -29
228 lines not on this page · raw message
net/smc/smc_loopback.c deleted +0 -427
427 lines not on this page · raw message
net/smc/smc_loopback.h deleted +0 -60
60 lines not on this page · raw message
Signature · 1 line
2.45.2
Dust Li
maintainerRe: [RFC net-next 5/7] net/ism: Move ism_loopback to net/ism
Quoted 258 lines · Alexandra Winter
The first stage of
ism_loopbackwas implemented as part of the SMC module [1]. Now that we have the ism layer, provide access to theism_loopbackdevice to all ism clients.Move ism_loopback.* from net/smc to net/ism.
The following changes are required toism_loopback.c:
- Changeism_lo_move_data()to no longer schedule an smcd receive tasklet, but instead callism_client->handle_irq().
Note: In this RFC patchism_loppbackis not fully generic.
The smc-d client uses attached buffers, for moves without signalling. and not-attached buffers for moves with signalling.ism_lo_move_data()must not rely on that assumption.ism_lo_move_data()must be able to handle more than one ism client.In addition the following changes are required to unify
ism_loopbackandism_vp:In ism layer and
ism_vpci:ism_loopbackis not backed by a pci device, so use dev instead of pdev inism_dev.In smc-d:
insmcd_alloc_dev():
- use kernel memory instead of device memory forsmcd_devandsmcd->conn. An alternative would be to ask device to alloc the memory.
- use differentsmcd_opsandmax_dmbsforism_vpandism_loopback. A future patch can change smc-d to directly useism_opsinstead ofsmcd_ops.
- use ismdev_nameinstead of pci dev name forism_evt_wqname
- allocate an event workqueue forism_loopback, although it currently does not generate events.Signed-off-by Alexandra Winter <[email protected]>---
drivers/s390/net/ism.h | 6 +- drivers/s390/net/ism_drv.c | 31 ++- include/linux/ism.h | 59 +++++ include/net/smc.h | 4 +- net/ism/Kconfig | 13 ++ net/ism/Makefile | 1 + net/ism/ism_loopback.c | 366 +++++++++++++++++++++++++++++++ net/ism/ism_loopback.h | 59 +++++ net/ism/ism_main.c | 11 +- net/smc/Kconfig | 13 -- net/smc/Makefile | 1 - net/smc/af_smc.c | 12 +- net/smc/smc_ism.c | 108 +++++++--- net/smc/smc_loopback.c | 427 ------------------------------------- net/smc/smc_loopback.h | 60 ------ 15 files changed, 606 insertions(+), 565 deletions(-)create mode 100644
net/ism/ism_loopback.c
create mode 100644net/ism/ism_loopback.h
delete mode 100644net/smc/smc_loopback.c
delete mode 100644net/smc/smc_loopback.hdrivers/s390/net/ism.h +3
-3@@ -202,7 +202,7 @@ struct ism_sba {static inline void __ism_read_cmd (struct ism_dev *ism,void *data,unsigned long offset,unsigned long len) { -struct zpci_dev *zdev =to_zpci (ism->pdev); +struct zpci_dev *zdev =to_zpci (to_pci_dev (ism->dev.parent));u64 req =ZPCI_CREATE_REQ (zdev->fh,2 ,8 );while (len >0 ) {@@ -216,7 +216,7 @@ static inline void __ism_read_cmd(struct ism_dev *ism, void *data,static inline void __ism_write_cmd (struct ism_dev *ism,void *data,unsigned long offset,unsigned long len) { -struct zpci_dev *zdev =to_zpci (ism->pdev); +struct zpci_dev *zdev =to_zpci (to_pci_dev (ism->dev.parent));u64 req =ZPCI_CREATE_REQ (zdev->fh,2 , len);if (len)@@ -226,7 +226,7 @@ static inline void __ism_write_cmd(struct ism_dev *ism, void *data,static inline int __ism_move (struct ism_dev *ism,u64 dmb_req,void *data,unsigned int size) { -struct zpci_dev *zdev =to_zpci (ism->pdev); +struct zpci_dev *zdev =to_zpci (to_pci_dev (ism->dev.parent));u64 req =ZPCI_CREATE_REQ (zdev->fh,0 , size);return __zpci_store_block (data, req, dmb_req);drivers/s390/net/ism_drv.c +12
-16@@ -88,7 +88,7 @@ static int register_sba(struct ism_dev *ism)dma_addr_t dma_handle;struct ism_sba *sba; - sba =dma_alloc_coherent (&ism->pdev->dev,PAGE_SIZE , &dma_handle, + sba =dma_alloc_coherent (ism->dev.parent,PAGE_SIZE , &dma_handle,GFP_KERNEL );if (!sba)return -ENOMEM ;@@ -99,7 +99,7 @@ static int register_sba(struct ism_dev *ism) cmd.request.sba = dma_handle;if (ism_cmd (ism, &cmd)) { -dma_free_coherent (&ism->pdev->dev,PAGE_SIZE , sba, dma_handle); +dma_free_coherent (ism->dev.parent,PAGE_SIZE , sba, dma_handle);return -EIO ; }@@ -115,7 +115,7 @@ static int register_ieq(struct ism_dev *ism)dma_addr_t dma_handle;struct ism_eq *ieq; - ieq =dma_alloc_coherent (&ism->pdev->dev,PAGE_SIZE , &dma_handle, + ieq =dma_alloc_coherent (ism->dev.parent,PAGE_SIZE , &dma_handle,GFP_KERNEL );if (!ieq)return -ENOMEM ;@@ -127,7 +127,7 @@ static int register_ieq(struct ism_dev *ism) cmd.request.len =sizeof (*ieq);if (ism_cmd (ism, &cmd)) { -dma_free_coherent (&ism->pdev->dev,PAGE_SIZE , ieq, dma_handle); +dma_free_coherent (ism->dev.parent,PAGE_SIZE , ieq, dma_handle);return -EIO ; }@@ -149,7 +149,7 @@ static int unregister_sba(struct ism_dev *ism)if (ret && ret !=ISM_ERROR )return -EIO ; -dma_free_coherent (&ism->pdev->dev,PAGE_SIZE , +dma_free_coherent (ism->dev.parent,PAGE_SIZE , ism->sba, ism->sba_dma_addr); ism->sba =NULL ;@@ -169,7 +169,7 @@ static int unregister_ieq(struct ism_dev *ism)if (ret && ret !=ISM_ERROR )return -EIO ; -dma_free_coherent (&ism->pdev->dev,PAGE_SIZE , +dma_free_coherent (ism->dev.parent,PAGE_SIZE , ism->ieq, ism->ieq_dma_addr); ism->ieq =NULL ;@@ -216,7 +216,7 @@ static int ism_query_rgid(struct ism_dev *ism, uuid_t *rgid, u32 vid_valid,static void ism_free_dmb (struct ism_dev *ism,struct ism_dmb *dmb) {clear_bit (dmb->sba_idx, ism->sba_bitmap); -dma_unmap_page (&ism->pdev->dev, dmb->dma_addr, dmb->dmb_len, +dma_unmap_page (ism->dev.parent, dmb->dma_addr, dmb->dmb_len,DMA_FROM_DEVICE );folio_put (virt_to_folio (dmb->cpu_addr)); }@@ -227,7 +227,7 @@ static int ism_alloc_dmb(struct ism_dev *ism, struct ism_dmb *dmb)unsigned long bit;int rc; -if (PAGE_ALIGN (dmb->dmb_len) >dma_get_max_seg_size (&ism->pdev->dev)) +if (PAGE_ALIGN (dmb->dmb_len) >dma_get_max_seg_size (ism->dev.parent))return -EINVAL ;if (!dmb->sba_idx) {@@ -251,10 +251,10 @@ static int ism_alloc_dmb(struct ism_dev *ism, struct ism_dmb *dmb) } dmb->cpu_addr =folio_address (folio); - dmb->dma_addr =dma_map_page (&ism->pdev->dev, + dmb->dma_addr =dma_map_page (ism->dev.parent,virt_to_page (dmb->cpu_addr),0 , dmb->dmb_len,DMA_FROM_DEVICE ); -if (dma_mapping_error (&ism->pdev->dev, dmb->dma_addr)) { +if (dma_mapping_error (ism->dev.parent, dmb->dma_addr)) { rc = -ENOMEM ; goto out_free; }@@ -419,10 +419,7 @@ static int ism_supports_v2(void)static u16 ism_get_chid (struct ism_dev *ism) { -if (!ism || !ism->pdev) -return 0 ; - -return to_zpci (ism->pdev)->pchid; +return to_zpci (to_pci_dev (ism->dev.parent))->pchid; }static void ism_handle_event (struct ism_dev *ism)@@ -499,7 +496,7 @@ static const struct ism_ops ism_vp_ops = {static int ism_dev_init (struct ism_dev *ism) { -struct pci_dev *pdev = ism->pdev; +struct pci_dev *pdev =to_pci_dev (ism->dev.parent);int ret; ret =pci_alloc_irq_vectors (pdev,1 ,1 ,PCI_IRQ_MSI );@@ -565,7 +562,6 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)spin_lock_init (&ism->lock);dev_set_drvdata (&pdev->dev, ism); - ism->pdev = pdev; ism->dev.parent = &pdev->dev;device_initialize (&ism->dev);dev_set_name (&ism->dev,dev_name (&pdev->dev));@@ -603,14 +599,13 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)device_del (&ism->dev);
err_dev:dev_set_drvdata(&pdev->dev, NULL);drivers/s390/net/ism_drv.c +0
-1-kfree (ism);return ret; }static void
ism_dev_exit(struct ism_dev *ism)
{drivers/s390/net/ism_drv.c +1
-1-struct pci_dev *pdev = ism->pdev; +struct pci_dev *pdev =to_pci_dev (ism->dev.parent);unsigned long flags; int i;include/linux/ism.h +21
-0@@ -144,6 +144,9 @@ int ism_unregister_client(struct ism_client *client);* identified by dmb_tok and idx. If signal flag (sf) then signal * the remote peer that data has arrived in this dmb. * +* int (*unregister_dmb)(struct ism_dev *dev, struct ism_dmb *dmb); +* Unregister an ism_dmb buffer +* * int (*supports_v2)(void); * * u16 (*get_chid)(struct ism_dev *dev); @@ -218,12 +221,63 @@ struct ism_ops {int (*reset_vlan_required)(struct ism_dev *dev);int (*signal_event)(struct ism_dev *dev,uuid_t *rgid,u32 trigger_irq,u32 event_code,u64 info); +/* no copy option +* -------------- +*/ +/** +* support_dmb_nocopy() - does this device provide no-copy option? +* @dev: ism device +* +* In addition to using move_data(), a sender device can provide a +* kernel address + length, that represents a target dmb +* (like MMIO). If a sender writes into such a ghost-send-buffer +* (= at this kernel address) the data will automatically +* immediately appear in the target dmb, even without calling +* move_data(). +* Note that this is NOT related to the MSG_ZEROCOPY socket flag. +* +* Either all 3 function pointers for support_dmb_nocopy(), +* attach_dmb() and detach_dmb() are defined, or all of them must +* be NULL.
include/linux/ism.h struct ism_ops { · line 251
Quoted 10 lines
include/linux/ism.h +10
-0+* +* Return: non-zero, if no-copy is supported. +*/ +int (*support_dmb_nocopy)(struct ism_dev *dev); +/** +* attach_dmb() - attach local memory to a remote dmb +* @dev: Local sending ism device +* @dmb: all other parameters are passed in the form of a +* dmb struct +* TODO: (THIS IS CONFUSING, should be changed)
Agree.
Quoted 22 lines
include/linux/ism.h +20
-0+* dmb_tok: (in) Token of the remote dmb, we want to attach to +* cpu_addr: (out) MMIO address +* dma_addr: (out) MMIO address (if applicable, invalid otherwise) +* dmb_len: (out) length of local MMIO region, +* equal to length of remote DMB. +* sba_idx: (out) index of remote dmb (NOT HELPFUL, should be removed) +* +* Provides a memory address to the sender that can be used to +* directly write into the remote dmb. +* +* Return: Zero upon success, Error code otherwise +*/ +int (*attach_dmb)(struct ism_dev *dev,struct ism_dmb *dmb); +/** +* detach_dmb() - Detach the ghost buffer from a remote dmb +* @dev: ism device +* @token: dmb token of the remote dmb +* Return: Zero upon success, Error code otherwise +*/ +int (*detach_dmb)(struct ism_dev *dev,u64 token); };
...
Quoted 22 lines
include/linux/ism.h +22
-0+ +static int ism_lo_move_data (struct ism_dev *ism,u64 dmb_tok, +unsigned int idx,bool sf,unsigned int offset, +void *data,unsigned int size) +{ +struct ism_lo_dmb_node *rmb_node =NULL , *tmp_node; +struct ism_lo_dev *ldev; +u16 s_mask; +u8 client_id; +u32 sba_idx; + + ldev = container_of(ism, struct ism_lo_dev, ism); + +if (!sf) +/* since sndbuf is merged with peer DMB, there is +* no need to copy data from sndbuf to peer DMB. +*/ +return 0 ; + +read_lock_bh (&ldev->dmb_ht_lock); +hash_for_each_possible (ldev->dmb_ht, tmp_node, list, dmb_tok) { +if (tmp_node->token == dmb_tok) {
include/linux/ism.h
Quoted 10 lines
include/linux/ism.h +10
-0+ rmb_node = tmp_node; +break ; + } + } +if (!rmb_node) { +read_unlock_bh (&ldev->dmb_ht_lock); +return -EINVAL ; + } +// So why copy the data now?? SMC usecase? Data buffer is attached, +// rw-pointer are not attached?
I understand the confusion here. I have the same confusion the first time I saw this.
This is actually the tricky part: it assumes the CDC will signal, while the data will not. We need to copy the CDC, so the copy here only to the CDC.
I think we should refine the move_data() API to make this clearer.
Best regards,
Dust
Alexandra Winter
maintainerRe: [RFC net-next 5/7] net/ism: Move ism_loopback to net/ism
Quoted 34 lines · Dust Li
Quoted 21 lines
net/ism/ism_loopback.c new file +21
-0+static int ism_lo_move_data (struct ism_dev *ism,u64 dmb_tok, +unsigned int idx,bool sf,unsigned int offset, +void *data,unsigned int size) +{ +struct ism_lo_dmb_node *rmb_node =NULL , *tmp_node; +struct ism_lo_dev *ldev; +u16 s_mask; +u8 client_id; +u32 sba_idx; + + ldev = container_of(ism, struct ism_lo_dev, ism); + +if (!sf) +/* since sndbuf is merged with peer DMB, there is +* no need to copy data from sndbuf to peer DMB. +*/ +return 0 ; + +read_lock_bh (&ldev->dmb_ht_lock); +hash_for_each_possible (ldev->dmb_ht, tmp_node, list, dmb_tok) { +if (tmp_node->token == dmb_tok) {net/ism/ism_loopback.c line 249
Quoted 10 lines
net/ism/ism_loopback.c new file +10
-0+ rmb_node = tmp_node; +break ; + } + } +if (!rmb_node) { +read_unlock_bh (&ldev->dmb_ht_lock); +return -EINVAL ; + } +// So why copy the data now?? SMC usecase? Data buffer is attached, +// rw-pointer are not attached? I understand the confusion here. I have the same confusion the first time I saw this.
Quoted 8 lines
This is actually the tricky part: it assumes the CDC will signal, while the data will not. We need to copy the CDC, so the copy here only to the CDC.
I think we should refine the
move_data()API to make this clearer.Best regards,
Dust
I agree. Will be refined in next version.
Re: [RFC net-next 5/7] net/ism: Move ism_loopback to net/ism
In reply to Alexandra WinterQuoted 57 lines · Alexandra Winter
The first stage of
ism_loopbackwas implemented as part of the SMC module [1]. Now that we have the ism layer, provide access to theism_loopbackdevice to all ism clients.Move ism_loopback.* from net/smc to net/ism.
The following changes are required toism_loopback.c:
- Changeism_lo_move_data()to no longer schedule an smcd receive tasklet, but instead callism_client->handle_irq().
Note: In this RFC patchism_loppbackis not fully generic.
The smc-d client uses attached buffers, for moves without signalling. and not-attached buffers for moves with signalling.ism_lo_move_data()must not rely on that assumption.ism_lo_move_data()must be able to handle more than one ism client.In addition the following changes are required to unify
ism_loopbackandism_vp:In ism layer and
ism_vpci:ism_loopbackis not backed by a pci device, so use dev instead of pdev inism_dev.In smc-d:
insmcd_alloc_dev():
- use kernel memory instead of device memory forsmcd_devandsmcd->conn. An alternative would be to ask device to alloc the memory.
- use differentsmcd_opsandmax_dmbsforism_vpandism_loopback. A future patch can change smc-d to directly useism_opsinstead ofsmcd_ops.
- use ismdev_nameinstead of pci dev name forism_evt_wqname
- allocate an event workqueue forism_loopback, although it currently does not generate events.Signed-off-by Alexandra Winter <[email protected]>--- drivers/s390/net/ism.h | 6 +- drivers/s390/net/ism_drv.c | 31 ++- include/linux/ism.h | 59 +++++ include/net/smc.h | 4 +- net/ism/Kconfig | 13 ++ net/ism/Makefile | 1 + net/ism/ism_loopback.c | 366 +++++++++++++++++++++++++++++++ net/ism/ism_loopback.h | 59 +++++ net/ism/ism_main.c | 11 +- net/smc/Kconfig | 13 -- net/smc/Makefile | 1 - net/smc/af_smc.c | 12 +- net/smc/smc_ism.c | 108 +++++++--- net/smc/smc_loopback.c | 427 ------------------------------------- net/smc/smc_loopback.h | 60 ------ 15 files changed, 606 insertions(+), 565 deletions(-) create mode 100644 net/ism/ism_loopback.c create mode 100644 net/ism/ism_loopback.h delete mode 100644 net/smc/smc_loopback.c delete mode 100644 net/smc/smc_loopback.h
...
Quoted 31 lines
net/ism/ism_loopback.c new file +25
-0@@ -0,0 +1,366 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Functions for loopback-ism device. +* +* Copyright (c) 2024, Alibaba Inc. +* +* Author: Wen Gu <[email protected]> +* Tony Lu <[email protected]> +* +*/ + +#include <linux/bitops.h> +#include <linux/device.h> +#include <linux/ism.h> +#include <linux/spinlock.h> +#include <linux/types.h> + +#include "ism_loopback.h" + +#define ISM_LO_V2_CAPABLE 0x1/* loopback-ism acts as ISMv2 */ +#define ISM_LO_SUPPORT_NOCOPY 0x1 +#define ISM_DMA_ADDR_INVALID (~(dma_addr_t)0) + +static const char ism_lo_dev_name[] ="loopback-ism" ; +/* global loopback device */
net/ism/ism_loopback.c line 35
Quoted 10 lines
net/ism/ism_loopback.c new file +10
-0+static struct ism_lo_dev *lo_dev; + +static int ism_lo_query_rgid (struct ism_dev *ism,uuid_t *rgid, +u32 vid_valid,u32 vid) +{ +/* rgid should be the same as lgid; vlan is not supported */ +if (!vid_valid &&uuid_equal (rgid, &ism->gid)) +return 0 ; +return -ENETUNREACH ; +}
This vid_valid check breaks ism-loopback for me.
Alexandra Winter
maintainerRe: [RFC net-next 5/7] net/ism: Move ism_loopback to net/ism
Quoted 9 lines · Julian Ruess
net/ism/ism_loopback.c line 35
Quoted 8 lines
net/ism/ism_loopback.c new file +8
-0+static int ism_lo_query_rgid (struct ism_dev *ism,uuid_t *rgid, +u32 vid_valid,u32 vid) +{ +/* rgid should be the same as lgid; vlan is not supported */ +if (!vid_valid &&uuid_equal (rgid, &ism->gid)) +return 0 ; +return -ENETUNREACH ; +}This
vid_validcheck breaks ism-loopback for me.
oops, I also get:
Quoted 1 line
smc_chk-C 10.44.30.50
[1] 967189
Test with target IP 10.44.30.50 and port 37373
Live test (SMC-D and SMC-R)
Server started on port 37373
Failed (TCP fallback), reasons:
Client: 0x05000000 Peer declined during handshake
Server: 0x03030007 No SMC-Dv2 device found, but requiredSorry about that.
Current upstream smc_loopback just ignores vid_valid in smc_lo_query_rgidsmc_lo_query_rgid(), but I'm not sure that is the best way to handle that.
I'll investigate and make sure it works in v2.
Alexandra Winter
maintainerRFC 6/7 s390/ism: Define ismvp_dev
In reply to Alexandra WinterMove the fields that are specific to the s390 ism_vpci driver out of the generic ism_dev into a local ismvp_dev structure.
--- drivers/s390/net/ism.h | 11 +++++ drivers/s390/net/ism_drv.c | 87 +++++++++++++++++++++++--------------- include/linux/ism.h | 20 +++------ 3 files changed, 71 insertions(+), 47 deletions(-)
drivers/s390/net/ism.h +11 -0
@@ -196,6 +196,17 @@ struct ism_sba {u16 dmbe_mask[ISM_NR_DMBS ]; }; +struct ismvp_dev { +struct ism_dev ism; +struct ism_sba *sba; +dma_addr_t sba_dma_addr; + DECLARE_BITMAP(sba_bitmap, ISM_NR_DMBS); + +struct ism_eq *ieq; +dma_addr_t ieq_dma_addr; +int ieq_idx; +}; +#define ISM_CREATE_REQ (dmb, idx, sf, offset) \ ((dmb) | (idx) << 24 | (sf) << 23 | (offset))
drivers/s390/net/ism_drv.c +53 -34
247 lines not on this page · raw message
include/linux/ism.h +7 -13
@@ -281,24 +281,18 @@ struct ism_ops {struct ism_dev {const struct ism_ops *ops; -spinlock_t lock;/* protects the ism device */ struct list_head list; -struct pci_dev *pdev; - -struct ism_sba *sba; -dma_addr_t sba_dma_addr; - DECLARE_BITMAP(sba_bitmap, ISM_NR_DMBS); -u8 *sba_client_arr;/* entries are indices into 'clients' array */ -void *priv[MAX_CLIENTS ]; - -struct ism_eq *ieq; -dma_addr_t ieq_dma_addr; -struct device dev;uuid_t gid; -int ieq_idx; +/* get this lock before accessing any of the fields below */ +spinlock_t lock; +/* indexed by dmb idx; entries are indices into priv and subs arrays: */ +u8 *sba_client_arr; +/* Sparse array of all ISM clients */ struct ism_client *subs[MAX_CLIENTS ]; +/* priv pointer per client; for client usage only */ +void *priv[MAX_CLIENTS ]; };int ism_dev_register (struct ism_dev *ism);
Signature · 1 line
2.45.2
Alexandra Winter
maintainerRFC 7/7 net/smc: Use only ism_ops
In reply to Alexandra WinterReplace smcd_ops by using ism_ops directly.
--- include/linux/ism.h | 1 + include/net/smc.h | 30 ------ net/smc/smc_clc.c | 6 +- net/smc/smc_core.c | 6 +- net/smc/smc_diag.c | 2 +- net/smc/smc_ism.c | 222 ++++++++------------------------------------ net/smc/smc_ism.h | 8 +- net/smc/smc_pnet.c | 8 +- 8 files changed, 55 insertions(+), 228 deletions(-)
include/linux/ism.h +1 -0
@@ -30,6 +30,7 @@ struct ism_dmb {*/ u64 dmb_tok;/* rgid - GID of designated remote sending device */ +//TODO: Change to uuid_t GID. Ok for now, because loopback ignores it. u64 rgid;u32 dmb_len;/* sba_idx - Index of this DMB on this receiving device */
include/net/smc.h +0 -30
@@ -28,43 +28,13 @@ struct smc_hashinfo {/* SMCD/ISM device driver interface */ -struct smcd_dev ; -struct smcd_gid {u64 gid;u64 gid_ext; }; -struct smcd_ops { -int (*query_remote_gid)(struct smcd_dev *dev,struct smcd_gid *rgid, -u32 vid_valid,u32 vid); -int (*register_dmb)(struct smcd_dev *dev,struct ism_dmb *dmb, -void *client); -int (*unregister_dmb)(struct smcd_dev *dev,struct ism_dmb *dmb); -int (*move_data)(struct smcd_dev *dev,u64 dmb_tok,unsigned int idx, -bool sf,unsigned int offset,void *data, -unsigned int size); -int (*supports_v2)(void ); -void (*get_local_gid)(struct smcd_dev *dev,struct smcd_gid *gid); -u16 (*get_chid)(struct smcd_dev *dev); -struct device * (*get_dev)(struct smcd_dev *dev); - -/* optional operations */ -int (*add_vlan_id)(struct smcd_dev *dev,u64 vlan_id); -int (*del_vlan_id)(struct smcd_dev *dev,u64 vlan_id); -int (*set_vlan_required)(struct smcd_dev *dev); -int (*reset_vlan_required)(struct smcd_dev *dev); -int (*signal_event)(struct smcd_dev *dev,struct smcd_gid *rgid, -u32 trigger_irq,u32 event_code,u64 info); -int (*support_dmb_nocopy)(struct smcd_dev *dev); -int (*attach_dmb)(struct smcd_dev *dev,struct ism_dmb *dmb); -int (*detach_dmb)(struct smcd_dev *dev,u64 token); -}; -struct smcd_dev { -const struct smcd_ops *ops;struct ism_dev *ism; -struct ism_client *client;struct list_head list;spinlock_t lock;struct smc_connection **conn;
net/smc/smc_clc.c +3 -3
@@ -900,7 +900,7 @@ int smc_clc_send_proposal(struct smc_sock *smc, struct smc_init_info *ini)/* add SMC-D specifics */ if (ini->ism_dev[0 ]) { smcd = ini->ism_dev[0 ]; - smcd->ops->get_local_gid (smcd, &smcd_gid); +copy_to_smcdgid (&smcd_gid, &smcd->ism->gid); pclc_smcd->ism.gid =htonll (smcd_gid.gid); pclc_smcd->ism.chid =htons (smc_ism_get_chid (ini->ism_dev[0 ]));
@@ -950,7 +950,7 @@ int smc_clc_send_proposal(struct smc_sock *smc, struct smc_init_info *ini)if (ini->ism_offered_cnt) {for (i =1 ; i <= ini->ism_offered_cnt; i++) { smcd = ini->ism_dev[i]; - smcd->ops->get_local_gid (smcd, &smcd_gid); +copy_to_smcdgid (&smcd_gid, &smcd->ism->gid); gidchids[entry].chid =htons (smc_ism_get_chid (ini->ism_dev[i])); gidchids[entry].gid =htonll (smcd_gid.gid);
@@ -1043,7 +1043,7 @@ smcd_clc_prep_confirm_accept(struct smc_connection *conn,/* SMC-D specific settings */ memcpy (clc->hdr.eyecatcher,SMCD_EYECATCHER ,sizeof (SMCD_EYECATCHER )); - smcd->ops->get_local_gid (smcd, &smcd_gid); +copy_to_smcdgid (&smcd_gid, &smcd->ism->gid); clc->hdr.typev1 =SMC_TYPE_D ; clc->d0.gid =htonll (smcd_gid.gid); clc->d0.token =htonll (conn->rmb_desc->token);
net/smc/smc_core.c +3 -3
@@ -555,7 +555,7 @@ static int smc_nl_fill_smcd_lgr(struct smc_link_group *lgr,if (nla_put_u32 (skb,SMC_NLA_LGR_D_ID , *((u32 *)&lgr->id))) goto errattr; - smcd->ops->get_local_gid (smcd, &smcd_gid); +copy_to_smcdgid (&smcd_gid, &smcd->ism->gid);if (nla_put_u64_64bit (skb,SMC_NLA_LGR_D_GID , smcd_gid.gid,SMC_NLA_LGR_D_PAD )) goto errattr;
@@ -919,7 +919,7 @@ static int smc_lgr_create(struct smc_sock *smc, struct smc_init_info *ini)if (ini->is_smcd) {/* SMC-D specific settings */ smcd = ini->ism_dev[ini->ism_selected]; -get_device (smcd->ops->get_dev (smcd)); +get_device (ism_get_dev (smcd->ism)); lgr->peer_gid.gid = ini->ism_peer_gid[ini->ism_selected].gid; lgr->peer_gid.gid_ext =
@@ -1469,7 +1469,7 @@ static void smc_lgr_free(struct smc_link_group *lgr)destroy_workqueue (lgr->tx_wq);if (lgr->is_smcd) {smc_ism_put_vlan (lgr->smcd, lgr->vlan_id); -put_device (lgr->smcd->ops->get_dev (lgr->smcd)); +put_device (ism_get_dev (lgr->smcd->ism)); }smc_lgr_put (lgr);/* theoretically last lgr_put */ }
net/smc/smc_diag.c +1 -1
@@ -175,7 +175,7 @@ static int __smc_diag_dump(struct sock *sk, struct sk_buff *skb, dinfo.linkid = *((u32 *)conn->lgr->id); dinfo.peer_gid = conn->lgr->peer_gid.gid; dinfo.peer_gid_ext = conn->lgr->peer_gid.gid_ext; - smcd->ops->get_local_gid (smcd, &smcd_gid); +copy_to_smcdgid (&smcd_gid, &smcd->ism->gid); dinfo.my_gid = smcd_gid.gid; dinfo.my_gid_ext = smcd_gid.gid_ext; dinfo.token = conn->rmb_desc->token;
net/smc/smc_ism.c +38 -184
360 lines not on this page · raw message
net/smc/smc_ism.h +5 -3
@@ -68,7 +68,9 @@ static inline int smc_ism_write(struct smcd_dev *smcd, u64 dmb_tok, {int rc; - rc = smcd->ops->move_data (smcd, dmb_tok, idx, sf, offset, data, len); + rc = smcd->ism->ops->move_data (smcd->ism, dmb_tok, idx, sf, offset, + data, len); +return rc <0 ? rc :0 ; }
@@ -85,14 +87,14 @@ static inline bool __smc_ism_is_emulated(u16 chid)static inline bool smc_ism_is_emulated (struct smcd_dev *smcd) { -u16 chid = smcd->ops->get_chid (smcd); +u16 chid = smcd->ism->ops->get_chid (smcd->ism);return __smc_ism_is_emulated (chid); }static inline bool smc_ism_is_loopback (struct smcd_dev *smcd) { -return (smcd->ops->get_chid (smcd) ==0xFFFF ); +return (smcd->ism->ops->get_chid (smcd->ism) ==ISM_LO_RESERVED_CHID ); }static inline void copy_to_smcdgid (struct smcd_gid *sgid,uuid_t *igid)
net/smc/smc_pnet.c +4 -4
@@ -169,7 +169,7 @@ static int smc_pnet_remove_by_pnetid(struct net *net, char *pnet_name)pr_warn_ratelimited ("smc: smcd device %s " "erased user defined pnetid " "%.16s\n" , -dev_name (smcd->ops->get_dev (smcd)), +dev_name (ism_get_dev (smcd->ism)), smcd->pnetid);memset (smcd->pnetid,0 ,SMC_MAX_PNETID_LEN ); smcd->pnetid_by_user = false;
@@ -332,7 +332,7 @@ static struct smcd_dev *smc_pnet_find_smcd(char *smcd_name)mutex_lock (&smcd_dev_list.mutex);list_for_each_entry (smcd_dev, &smcd_dev_list.list, list) { -if (!strncmp (dev_name (smcd_dev->ops->get_dev (smcd_dev)), +if (!strncmp (dev_name (ism_get_dev (smcd_dev->ism)), smcd_name,IB_DEVICE_NAME_MAX -1 )) goto out; }
@@ -431,7 +431,7 @@ static int smc_pnet_add_ib(struct smc_pnettable *pnettable, char *ib_name,if (smcd) { smcddev_applied =smc_pnet_apply_smcd (smcd, pnet_name);if (smcddev_applied) { - dev = smcd->ops->get_dev (smcd); + dev =ism_get_dev (smcd->ism); pr_warn_ratelimited("smc: smcd device %s " "applied user defined pnetid " "%.16s\n", dev_name(dev),
@@ -1190,7 +1190,7 @@ int smc_pnetid_by_table_ib(struct smc_ib_device *smcibdev, u8 ib_port)*/ int smc_pnetid_by_table_smcd (struct smcd_dev *smcddev) { -const char *ib_name =dev_name (smcddev->ops->get_dev (smcddev)); +const char *ib_name =dev_name (ism_get_dev (smcddev->ism));struct smc_pnettable *pnettable;struct smc_pnetentry *tmp_pe;struct smc_net *sn;
Signature · 1 line
2.45.2
Dust Li
maintainerOn 2025-01-15 20:55:20, Alexandra Winter wrote:
Hi Winter,
I'm fully supportive of the refactor!
Interestingly, I developed a similar RFC code about a month ago while working on enhancing internal communication between guest and host systems. Here are some of my thoughts on the matter:
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory). To my knowledge, a "Shared Memory Device" better encapsulates the functionality we're aiming to implement. It might be beneficial to place it under drivers/shd/ and register it as a new class under /sys/class/shd/. That said, my initial draft also adopted the ISM terminology for simplicity.
Modular Approach: I've made the ism_loopback an independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.
Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate ism->ops, making the implementation cleaner and more intuitive. This way, the struct ism_device would mainly serve its implementers, while the upper helper functions offer a streamlined interface for SMC.
Structuring and Naming: I recommend embedding the structure of ism_ops directly within ism_dev rather than using a pointer. Additionally, renaming it to ism_device_ops could enhance clarity and consistency.
Quoted 8 lines
This RFC is about providing a generic shim layer between all kinds of ism devices and all kinds of ism users.
Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.
Fully agree.
Quoted 47 lines
Request for comments:
---------------------
Any comments are welcome, but I am aware that this series needs more work. It may not be worth your time to do an in-depth review of the details, I am looking for feedback on the general idea.
I am mostly interested in your thoughts and recommendations about the general concept, the location of net/ism, the structure ofinclude/linux/ism.h, the KConfig and makefiles.Status of this RFC:
-------------------
This is a very early RFC to ask you for comments on this general idea. The RFC does not fullfill all criteria required for a patchset. The whole set compiles and runs, but I did not try all combinations of module and built-in yet. I did not check for checkpatch or any other checkers. Also I have only done very rudimentary quick tests of SMC-D. More testing is required.Background / Status quo:
------------------------
Currently s390 hardware provides virtual PCI ISM devices (ism_vpci). Their driver is indrivers/s390/net/ism_drv.c. The main user is SMC-D (net/smc).ism_vpcidriver offers a client interface so other users/protocols can also use them, but it is still heavily intermingled with the smc code. Namely, the ISM vPCI module cannot be used without the SMC module, which feels artificial.The ISM concept is being extended:
[1] proposed an ISM loopback interface (ism_lo), that can be used on non-s390 architectures (e.g. between containers or to test SMC-D). A minimal implementation went upstream with [2]:ism_locurrently is a part of the smc protocol and rather hidden.[3] proposed a virtio definition of ISM (
ism_virtio) that can be used between kvm guests.We will shortly send an RFC for an ISM client that uses ISM as transport for TTY.
Concept:
--------
Create a shim layer in net/ism that contains common definitions and code for all ism devices and all ism clients.
Any device or client module only needs to depend on this ism layer module and any device or client code only needs to include the definitions ininclude/linux/ism.h
Quoted 5 lines
Ideas for next steps:
---------------------
- sysfs representation? e.g. as/sys/class/ism?
- provide a full-fledged ism loopback interface
(runtime enable/disable, sysfs device, ..)
I think it's better if we can make this common for all ISM devices. but yeah, that shoud be the next step.
Best regards,
Dust
Quoted 53 lines
- additional clients (tty over ism)
- additional devices (virtio-ism, ...)Alexandra Winter (7):
net/ism: Create net/ism
net/ism: Remove dependencies betweenISM_VPCIand SMC net/ism: Useuuid_tfor ISM GID
net/ism: Add kernel-doc comments for ism functions net/ism: Moveism_loopbackto net/ism s390/ism: Defineismvp_dev
net/smc: Use onlyism_opsMAINTAINERS | 7 + drivers/s390/net/Kconfig | 10 +- drivers/s390/net/Makefile | 4 +- drivers/s390/net/ism.h | 27 ++- drivers/s390/net/ism_drv.c | 467 ++++++++++++------------------------- include/linux/ism.h | 299 +++++++++++++++++++++--- include/net/smc.h | 52 +---- net/Kconfig | 1 + net/Makefile | 1 + net/ism/Kconfig | 27 +++ net/ism/Makefile | 8 + net/ism/ism_loopback.c | 366 +++++++++++++++++++++++++++++ net/ism/ism_loopback.h | 59 +++++ net/ism/ism_main.c | 171 ++++++++++++++ net/smc/Kconfig | 13 -- net/smc/Makefile | 1 - net/smc/af_smc.c | 12 +- net/smc/smc_clc.c | 6 +- net/smc/smc_core.c | 6 +- net/smc/smc_diag.c | 2 +- net/smc/smc_ism.c | 112 +++++---- net/smc/smc_ism.h | 29 ++- net/smc/smc_loopback.c | 427 --------------------------------- net/smc/smc_loopback.h | 60 ----- net/smc/smc_pnet.c | 8 +- 25 files changed, 1183 insertions(+), 992 deletions(-)create mode 100644
net/ism/Kconfig
create mode 100644net/ism/Makefile
create mode 100644net/ism/ism_loopback.c
create mode 100644net/ism/ism_loopback.h
create mode 100644net/ism/ism_main.c
delete mode 100644net/smc/smc_loopback.c
delete mode 100644net/smc/smc_loopback.hSignature · 1 line
2.45.2
Quoted 10 lines · Dust Li
On 2025-01-15 20:55:20, Alexandra Winter wrote:
Hi Winter,
I'm fully supportive of the refactor!
Interestingly, I developed a similar RFC code about a month ago while working on enhancing internal communication between guest and host systems. Here are some of my thoughts on the matter:
Quoted 6 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory). To my knowledge, a "Shared Memory Device" better encapsulates the functionality we're aiming to implement. It might be beneficial to place it under
drivers/shd/and register it as a new class under/sys/class/shd/. That said, my initial draft also adopted the ISM terminology for simplicity.
I'm not sure if we really want to introduce a new name for the already existing ISM device. For me, having two names for the same thing just adds additional complexity.
I would go for /sys/class/ism
Quoted 76 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate
ism->ops, making the implementation cleaner and more intuitive. This way, the structism_devicewould mainly serve its implementers, while the upper helper functions offer a streamlined interface for SMC.Structuring and Naming: I recommend embedding the structure of
ism_opsdirectly withinism_devrather than using a pointer. Additionally, renaming it toism_device_opscould enhance clarity and consistency.Quoted 8 lines
This RFC is about providing a generic shim layer between all kinds of ism devices and all kinds of ism users.
Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.Fully agree.
Quoted 47 lines
Request for comments:
---------------------
Any comments are welcome, but I am aware that this series needs more work. It may not be worth your time to do an in-depth review of the details, I am looking for feedback on the general idea.
I am mostly interested in your thoughts and recommendations about the general concept, the location of net/ism, the structure ofinclude/linux/ism.h, the KConfig and makefiles.Status of this RFC:
-------------------
This is a very early RFC to ask you for comments on this general idea. The RFC does not fullfill all criteria required for a patchset. The whole set compiles and runs, but I did not try all combinations of module and built-in yet. I did not check for checkpatch or any other checkers. Also I have only done very rudimentary quick tests of SMC-D. More testing is required.Background / Status quo:
------------------------
Currently s390 hardware provides virtual PCI ISM devices (ism_vpci). Their driver is indrivers/s390/net/ism_drv.c. The main user is SMC-D (net/smc).ism_vpcidriver offers a client interface so other users/protocols can also use them, but it is still heavily intermingled with the smc code. Namely, the ISM vPCI module cannot be used without the SMC module, which feels artificial.The ISM concept is being extended:
[1] proposed an ISM loopback interface (ism_lo), that can be used on non-s390 architectures (e.g. between containers or to test SMC-D). A minimal implementation went upstream with [2]:ism_locurrently is a part of the smc protocol and rather hidden.[3] proposed a virtio definition of ISM (
ism_virtio) that can be used between kvm guests.We will shortly send an RFC for an ISM client that uses ISM as transport for TTY.
Concept:
--------
Create a shim layer in net/ism that contains common definitions and code for all ism devices and all ism clients.
Any device or client module only needs to depend on this ism layer module and any device or client code only needs to include the definitions ininclude/linux/ism.h
Quoted 8 lines
Quoted 5 lines
Ideas for next steps:
---------------------
- sysfs representation? e.g. as/sys/class/ism?
- provide a full-fledged ism loopback interface
(runtime enable/disable, sysfs device, ..)I think it's better if we can make this common for all ISM devices. but yeah, that shoud be the next step.
I already have patches based on this series that introduce /sys/class/ism and show ism-loopback as well as s390/ism devices. I can send this soon.
Julian
Alexandra Winter
maintainerQuoted 6 lines · Julian Ruess
Quoted 5 lines · Dust Li
On 2025-01-15 20:55:20, Alexandra Winter wrote:
Hi Winter,
I'm fully supportive of the refactor!
Thank you very much Dust Li for joining the discussion.
Quoted 3 lines
Quoted 3 lines
Interestingly, I developed a similar RFC code about a month ago while working on enhancing internal communication between guest and host systems.
But you did not send that out, did you?
I hope I did not overlook an earlier proposal by you.
Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory _D_evice?
To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and by ism_loopback. So what is the benefit in changing it?
It might be beneficial to place it under
Quoted 14 lines
Quoted 2 lines
drivers/shd/and register it as a new class under/sys/class/shd/. That said, my initial draft also adopted the ISM terminology for simplicity.I'm not sure if we really want to introduce a new name for the already existing ISM device. For me, having two names for the same thing just adds additional complexity.
I would go for
/sys/class/ismQuoted 5 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.
With this proposal ism_loopback is just another ism device and SMC-D will handle removal just like ism_client.remove(ism_dev) of other ism devices.
But I understand that net/smc/ism_loopback.c today does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal by ism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separate ism_loopback kernel module, like you say. Today ism_loopback is only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).
Quoted 25 lines
Quoted 25 lines
Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate
ism->ops, making the implementation cleaner and more intuitive. This way, the structism_devicewould mainly serve its implementers, while the upper helper functions offer a streamlined interface for SMC.Structuring and Naming: I recommend embedding the structure of
ism_opsdirectly withinism_devrather than using a pointer. Additionally, renaming it toism_device_opscould enhance clarity and consistency.Quoted 8 lines
This RFC is about providing a generic shim layer between all kinds of ism devices and all kinds of ism users.
Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.Fully agree.
Quoted 1 line
[...]
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Ideas for next steps:
---------------------
- sysfs representation? e.g. as/sys/class/ism?
- provide a full-fledged ism loopback interface
(runtime enable/disable, sysfs device, ..)I think it's better if we can make this common for all ISM devices. but yeah, that shoud be the next step.
The s390 ism_vpci devices are already backed by struct pci_dev. And I assume that would be represented in sysfs somehow like:/sys/class/ism/ism_vp0/device -> /sys/devices/<pci bus no>/<pci dev no> so there is an
/sys/class/ism/<ism dev name>/device/enable entry already, because there is /sys/devices/<pci bus no>/<pci dev no>/enable today.
I remember Wen Gu's first proposal for ism_loopback had a device in /sys/devices/virtual/ and had an 'active' entry to enable/disable. Something like that could be linked to /sys/class/ism/ism_lo/device.
Quoted 7 lines
I already have patches based on this series that introduce
/sys/class/ismand show ism-loopback as well as s390/ism devices. I can send this soon.Julian
Quoted 118 lines · Alexandra Winter
Quoted 6 lines · Julian Ruess
Quoted 5 lines · Dust Li
On 2025-01-15 20:55:20, Alexandra Winter wrote:
Hi Winter,
I'm fully supportive of the refactor!
Thank you very much Dust Li for joining the discussion.
Quoted 3 lines
Quoted 3 lines
Interestingly, I developed a similar RFC code about a month ago while working on enhancing internal communication between guest and host systems.
But you did not send that out, did you?
I hope I did not overlook an earlier proposal by you.Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory
_D_evice?To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and byism_loopback. So what is the benefit in changing it?It might be beneficial to place it under
Quoted 14 lines
Quoted 2 lines
drivers/shd/and register it as a new class under/sys/class/shd/. That said, my initial draft also adopted the ISM terminology for simplicity.I'm not sure if we really want to introduce a new name for the already existing ISM device. For me, having two names for the same thing just adds additional complexity.
I would go for
/sys/class/ismQuoted 5 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.But I understand that
net/smc/ism_loopback.ctoday does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal byism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separateism_loopbackkernel module, like you say. Todayism_loopbackis only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).Quoted 25 lines
Quoted 25 lines
Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate
ism->ops, making the implementation cleaner and more intuitive. This way, the structism_devicewould mainly serve its implementers, while the upper helper functions offer a streamlined interface for SMC.Structuring and Naming: I recommend embedding the structure of
ism_opsdirectly withinism_devrather than using a pointer. Additionally, renaming it toism_device_opscould enhance clarity and consistency.Quoted 8 lines
This RFC is about providing a generic shim layer between all kinds of ism devices and all kinds of ism users.
Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.Fully agree.
Quoted 1 line
[...]
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Ideas for next steps:
---------------------
- sysfs representation? e.g. as/sys/class/ism?
- provide a full-fledged ism loopback interface
(runtime enable/disable, sysfs device, ..)I think it's better if we can make this common for all ISM devices. but yeah, that shoud be the next step.
The s390
ism_vpcidevices are already backed by structpci_dev. And I assume that would be represented in sysfs somehow like:/sys/class/ism/ism_vp0/device-> /sys/devices/<pci bus no>/<pci dev no> so there is an
/sys/class/ism/<ism dev name>/device/enable entry already, because there is /sys/devices/<pci bus no>/<pci dev no>/enable today.
Quoted 3 lines
I remember Wen Gu's first proposal for
ism_loopbackhad a device in/sys/devices/virtual/and had an 'active' entry to enable/disable. Something like that could be linked to/sys/class/ism/ism_lo/device.
My current implementation represents the devices as following in '/sys/class/ism':
ism_lo -> ../../devices/virtual/ism/ism_lo
lismvpci0 -> ../../devices/pci0124:00/0124:00:00.0/ism/ismvpci0
The driver is repsonsible for the naming of its devices.
And yes, because the s390 ism_vpci is backed by a PCI device, '/sys/class/ism/ismvpci0/device/enable' exists.
I think we could implement a device attribute for ism_lo to implement this functionality. I already have a device attribute implemented in ism_main
to access the gid of each ISM device. This leads to the following sysfs entries:
'/sys/class/ism/ism_lo/gid'
'/sys/class/ism/ismvpci0/gid'
Julian
Quoted 9 lines
Quoted 9 lines
I already have patches based on this series that introduce
/sys/class/ismand show ism-loopback as well as s390/ism devices. I can send this soon.Julian
Dust Li
maintainerQuoted 14 lines · Alexandra Winter
Quoted 6 lines · Julian Ruess
Quoted 5 lines · Dust Li
On 2025-01-15 20:55:20, Alexandra Winter wrote:
Hi Winter,
I'm fully supportive of the refactor!
Thank you very much Dust Li for joining the discussion.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Interestingly, I developed a similar RFC code about a month ago while working on enhancing internal communication between guest and host systems.
But you did not send that out, did you?
I hope I did not overlook an earlier proposal by you.
No, I just did a POC and didn't find the time to improve it. So I think we can go on with your version.
Quoted 9 lines
Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory
_D_evice?
Oh, I was trying to refer to SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.
Quoted 11 lines
To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and byism_loopback. So what is the benefit in changing it?
I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.
In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.
Quoted 9 lines
It might be beneficial to place it under
Quoted 6 lines
Quoted 2 lines
drivers/shd/and register it as a new class under/sys/class/shd/. That said, my initial draft also adopted the ISM terminology for simplicity.I'm not sure if we really want to introduce a new name for the already existing ISM device. For me, having two names for the same thing just adds additional complexity.
I believe that if we are going to rename it, there should be no reference to "ISM" in this subsystem. IBM's PCI ISM can retain that name, as it is an implementation of the Shared Memory device (assuming we adopt that name).
Quoted 13 lines
Quoted 8 lines
I would go for
/sys/class/ismQuoted 5 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.
Quoted 10 lines
But I understand that
net/smc/ism_loopback.ctoday does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal byism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separateism_loopbackkernel module, like you say. Todayism_loopbackis only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).
This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.
Best regards,
Dust
Quoted 1 line · Dust Li
Quoted 1 line
---8<---
Quoted 23 lines
Quoted 7 lines
Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory
_D_evice?Oh, I was trying to refer to
SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.Quoted 11 lines
To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and byism_loopback. So what is the benefit in changing it?
Quoted 8 lines
I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.
In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.
I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)
One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.
With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.
Thanks,
Niklas
Quoted 3 lines
One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording.
Maybe this is the trap i fell into. So are you saying it is not a dual port memory mapped into two CPUs physical address space? In another email there was reference to shm. That would be a VMM equivalent, a bunch of pages mapped into two processes address space.
This comes back to the lack of top level architecture documentation. Outside reviewers such as i will have difficultly making useful contributions, and seeing potential overlap and reuse with other systems, without having a basic understanding of what you are talking about.
Andrew
Quoted 7 lines · Andrew Lunn
Quoted 3 lines
One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording.
Maybe this is the trap i fell into. So are you saying it is not a dual port memory mapped into two CPUs physical address space?
Conceptually kind of but the existing s390 specific ISM device is a bit special. But let me start with some background. On s390 aka Mainframes OSs including Linux runs in so called logical partitions (LPARs) which are machine hypervisor VMs which use partitioned non-paging memory. The fact that memory is partitioned is important because this means LPARs can not share physical memory by mapping it.
Now at a high level an ISM device allows communication between two such Linux LPARs on the same machine. The device is discovered as a PCI device and allows Linux to take a buffer called a DMB map that in the IOMMU and generate a token specific to another LPAR which also sees an ISM device sharing the same virtual channel identifier (VCHID). This token can then be transferred out of band (e.g. as part of an extended TCP handshake in SMC-D) to that other system. With the token the other system can use its ISM device to securely (authenticated by the token, LPAR identity and the IOMMU mapping) write into the original systems DMB at throughput and latency similar to doing a memcpy() via a syscall.
On the implementation level the ISM device is actually a piece of firmware and the write to a remote DMB is a special case of our PCI Store Block instruction (no real MMIO on s390, instead there are special instructions). Sadly there are a few more quirks but in principle you can think of it as redirecting writes to a part of the ISM PCI devices' BAR to the DMB in the peer system if that makes sense. There's of course also a mechanism to cause an interrupt on the receiver as the write completes.
Quoted 3 lines
In another
email there was reference to shm. That would be a VMM equivalent, a bunch of pages mapped into two processes address space.
Yes as on a hypervisor which backs VMs with pages one can simply map the DMBin both guests (one mapping potentially being write only) with writes being literally a memcpy().
Quoted 8 lines
This comes back to the lack of top level architecture documentation. Outside reviewers such as i will have difficultly making useful contributions, and seeing potential overlap and reuse with other systems, without having a basic understanding of what you are talking about.
Andrew
I understand your frustration. I do think we're making progress here though. For one loopback ISM makes SMC-D usable/testable even on bare metal and then Alibaba is working on a virtio based ISM device and the draft is public[0]. And there is some information on SMC's use of ISM via a whitepaper[1]
[0] https://lore.kernel.org/all/Y1IqX2uVpcD7cvRF@TonyMac-Alibaba/T/
[1]
https://www.ibm.com/support/pages/system/files/inline-files/IBM%20Shared%20Memory%20Communications%20Version%202.1%20Emulated-ISM_0.pdf
Quoted 19 lines
Conceptually kind of but the existing s390 specific ISM device is a bit special. But let me start with some background. On s390 aka Mainframes OSs including Linux runs in so called logical partitions (LPARs) which are machine hypervisor VMs which use partitioned non-paging memory. The fact that memory is partitioned is important because this means LPARs can not share physical memory by mapping it.
Now at a high level an ISM device allows communication between two such Linux LPARs on the same machine. The device is discovered as a PCI device and allows Linux to take a buffer called a DMB map that in the IOMMU and generate a token specific to another LPAR which also sees an ISM device sharing the same virtual channel identifier (VCHID). This token can then be transferred out of band (e.g. as part of an extended TCP handshake in SMC-D) to that other system. With the token the other system can use its ISM device to securely (authenticated by the token, LPAR identity and the IOMMU mapping) write into the original systems DMB at throughput and latency similar to doing a
memcpy()via a syscall.
Quoted 8 lines
On the implementation level the ISM device is actually a piece of firmware and the write to a remote DMB is a special case of our PCI Store Block instruction (no real MMIO on s390, instead there are special instructions). Sadly there are a few more quirks but in principle you can think of it as redirecting writes to a part of the ISM PCI devices' BAR to the DMB in the peer system if that makes sense. There's of course also a mechanism to cause an interrupt on the receiver as the write completes.
So the s390 details are interesting, but as you say, it is special. Ideally, all the special should be hidden away inside the driver.
So please take a step back. What is the abstract model?
Can the abstract model be mapped onto CLX? Could it be used with a GPU vRAM? SoC with real shared memory between a pool of CPUs.
Andrew
Quoted 28 lines · Andrew Lunn
Quoted 19 lines
Conceptually kind of but the existing s390 specific ISM device is a bit special. But let me start with some background. On s390 aka Mainframes OSs including Linux runs in so called logical partitions (LPARs) which are machine hypervisor VMs which use partitioned non-paging memory. The fact that memory is partitioned is important because this means LPARs can not share physical memory by mapping it.
Now at a high level an ISM device allows communication between two such Linux LPARs on the same machine. The device is discovered as a PCI device and allows Linux to take a buffer called a DMB map that in the IOMMU and generate a token specific to another LPAR which also sees an ISM device sharing the same virtual channel identifier (VCHID). This token can then be transferred out of band (e.g. as part of an extended TCP handshake in SMC-D) to that other system. With the token the other system can use its ISM device to securely (authenticated by the token, LPAR identity and the IOMMU mapping) write into the original systems DMB at throughput and latency similar to doing a
memcpy()via a syscall.Quoted 8 lines
On the implementation level the ISM device is actually a piece of firmware and the write to a remote DMB is a special case of our PCI Store Block instruction (no real MMIO on s390, instead there are special instructions). Sadly there are a few more quirks but in principle you can think of it as redirecting writes to a part of the ISM PCI devices' BAR to the DMB in the peer system if that makes sense. There's of course also a mechanism to cause an interrupt on the receiver as the write completes.
Quoted 3 lines
So the s390 details are interesting, but as you say, it is special. Ideally, all the special should be hidden away inside the driver.
Yes and it will be. There are some exceptions e.g. for vfio-pci pass- through but that's not unusual and why there is already the concept of vfio-pci extension module.
Quoted 2 lines
So please take a step back. What is the abstract model?
I think my high level description may be a good start. The abstract model is the ability to share a memory buffer (DMB) for writing by a communication partner, authenticated by a DMB Token. Plus stuff like triggering an interrupt on write or explicit trigger. Then Alibaba added optional support for what they called attaching the buffer which means it becomes truly shared between the peers but which IBM's ISM can't support. Plus a few more optional pieces such as VLANs, PNETIDs don't ask. The idea for the new layer then is to define this interface with operations and documentation.
Quoted 5 lines
Can the abstract model be mapped onto CLX? Could it be used with a GPU vRAM? SoC with real shared memory between a pool of CPUs.
Andrew
I'd think that yes, one could implement such a mechanism on top of CXL as well as on SoC. Or even with no special hardware between a host and a DPU (e.g. via PCIe endpoint framework). Basically anything that can DMA and IRQs between two OS instances.
Quoted 54 lines · Niklas Schnelle
Quoted 28 lines · Andrew Lunn
Quoted 19 lines
Conceptually kind of but the existing s390 specific ISM device is a bit special. But let me start with some background. On s390 aka Mainframes OSs including Linux runs in so called logical partitions (LPARs) which are machine hypervisor VMs which use partitioned non-paging memory. The fact that memory is partitioned is important because this means LPARs can not share physical memory by mapping it.
Now at a high level an ISM device allows communication between two such Linux LPARs on the same machine. The device is discovered as a PCI device and allows Linux to take a buffer called a DMB map that in the IOMMU and generate a token specific to another LPAR which also sees an ISM device sharing the same virtual channel identifier (VCHID). This token can then be transferred out of band (e.g. as part of an extended TCP handshake in SMC-D) to that other system. With the token the other system can use its ISM device to securely (authenticated by the token, LPAR identity and the IOMMU mapping) write into the original systems DMB at throughput and latency similar to doing a
memcpy()via a syscall.Quoted 8 lines
On the implementation level the ISM device is actually a piece of firmware and the write to a remote DMB is a special case of our PCI Store Block instruction (no real MMIO on s390, instead there are special instructions). Sadly there are a few more quirks but in principle you can think of it as redirecting writes to a part of the ISM PCI devices' BAR to the DMB in the peer system if that makes sense. There's of course also a mechanism to cause an interrupt on the receiver as the write completes.
Quoted 3 lines
So the s390 details are interesting, but as you say, it is special. Ideally, all the special should be hidden away inside the driver.
Yes and it will be. There are some exceptions e.g. for vfio-pci pass- through but that's not unusual and why there is already the concept of vfio-pci extension module.
Quoted 2 lines
So please take a step back. What is the abstract model?
I think my high level description may be a good start. The abstract model is the ability to share a memory buffer (DMB) for writing by a communication partner, authenticated by a DMB Token. Plus stuff like triggering an interrupt on write or explicit trigger. Then Alibaba added optional support for what they called attaching the buffer which means it becomes truly shared between the peers but which IBM's ISM can't support. Plus a few more optional pieces such as VLANs, PNETIDs don't ask. The idea for the new layer then is to define this interface with operations and documentation.
Quoted 4 lines
Can the abstract model be mapped onto CLX? Could it be used with a GPU vRAM? SoC with real shared memory between a pool of CPUs.
Quoted 6 lines
Quoted 1 line
Andrew
I'd think that yes, one could implement such a mechanism on top of CXL as well as on SoC. Or even with no special hardware between a host and a DPU (e.g. via PCIe endpoint framework). Basically anything that can DMA and IRQs between two OS instances.
Is DMA part of the abstract model? That would suggest a true shared memory system is excluded, since that would not require DMA.
Maybe take a look at subsystems like USB, I2C.
usb_submit_urb(struct urb *urb, gfp_t mem_flags)
An URB is a data structure with a block of memory associated with it, contains the detail to pass to the USB device.
i2c_transfer(struct i2c_adapter *adap, struct i2c_msg *msgs, int num)
*msgs points to num of messages which get transferred to/from the I2C device.
Could the high level API look like this? No DMA, no IRQ, no concept of a somewhat shared memory. Just an API which asks for a message to be sent to the other end? struct urb has some USB concepts in it, struct i2c_msg has some I2C concepts in it. A struct ism_msg would follow the same pattern, but does it need to care about the DMA, the IRQ, the memory which is semi shared?
Andrew
Dust Li
maintainerQuoted 77 lines · Andrew Lunn
Quoted 54 lines · Niklas Schnelle
Quoted 28 lines · Andrew Lunn
Quoted 19 lines
Conceptually kind of but the existing s390 specific ISM device is a bit special. But let me start with some background. On s390 aka Mainframes OSs including Linux runs in so called logical partitions (LPARs) which are machine hypervisor VMs which use partitioned non-paging memory. The fact that memory is partitioned is important because this means LPARs can not share physical memory by mapping it.
Now at a high level an ISM device allows communication between two such Linux LPARs on the same machine. The device is discovered as a PCI device and allows Linux to take a buffer called a DMB map that in the IOMMU and generate a token specific to another LPAR which also sees an ISM device sharing the same virtual channel identifier (VCHID). This token can then be transferred out of band (e.g. as part of an extended TCP handshake in SMC-D) to that other system. With the token the other system can use its ISM device to securely (authenticated by the token, LPAR identity and the IOMMU mapping) write into the original systems DMB at throughput and latency similar to doing a
memcpy()via a syscall.Quoted 8 lines
On the implementation level the ISM device is actually a piece of firmware and the write to a remote DMB is a special case of our PCI Store Block instruction (no real MMIO on s390, instead there are special instructions). Sadly there are a few more quirks but in principle you can think of it as redirecting writes to a part of the ISM PCI devices' BAR to the DMB in the peer system if that makes sense. There's of course also a mechanism to cause an interrupt on the receiver as the write completes.
Quoted 3 lines
So the s390 details are interesting, but as you say, it is special. Ideally, all the special should be hidden away inside the driver.
Yes and it will be. There are some exceptions e.g. for vfio-pci pass- through but that's not unusual and why there is already the concept of vfio-pci extension module.
Quoted 2 lines
So please take a step back. What is the abstract model?
I think my high level description may be a good start. The abstract model is the ability to share a memory buffer (DMB) for writing by a communication partner, authenticated by a DMB Token. Plus stuff like triggering an interrupt on write or explicit trigger. Then Alibaba added optional support for what they called attaching the buffer which means it becomes truly shared between the peers but which IBM's ISM can't support. Plus a few more optional pieces such as VLANs, PNETIDs don't ask. The idea for the new layer then is to define this interface with operations and documentation.
Quoted 4 lines
Can the abstract model be mapped onto CLX? Could it be used with a GPU vRAM? SoC with real shared memory between a pool of CPUs.
Quoted 6 lines
Quoted 1 line
Andrew
I'd think that yes, one could implement such a mechanism on top of CXL as well as on SoC. Or even with no special hardware between a host and a DPU (e.g. via PCIe endpoint framework). Basically anything that can DMA and IRQs between two OS instances.
Is DMA part of the abstract model? That would suggest a true shared memory system is excluded, since that would not require DMA.
Maybe take a look at subsystems like USB, I2C.
usb_submit_urb(struct urb *urb, gfp_t mem_flags)An URB is a data structure with a block of memory associated with it, contains the detail to pass to the USB device.
i2c_transfer(struct i2c_adapter *adap, struct i2c_msg *msgs, int num)*msgs points to num of messages which get transferred to/from the I2C device.
Quoted 6 lines
Could the high level API look like this? No DMA, no IRQ, no concept of a somewhat shared memory. Just an API which asks for a message to be sent to the other end? struct urb has some USB concepts in it, struct
i2c_msghas some I2C concepts in it. A structism_msgwould follow the same pattern, but does it need to care about the DMA, the IRQ, the memory which is semi shared?
I don’t have a clear picture of what the API should look like yet, but I believe it’s possible to avoid DMA and IRQ. In fact, the current data transfer API, ops->move_data() in include/linux/ism.h, already abstracts away the DMA and IRQ details.
One thing we cannot hide, however, is whether the operation is zero-copy or copy. This distinction is important because we can reuse the data at different times in copy mode and zero-copy mode.
Best regards,
Dust
Alexandra Winter
maintainerQuoted 85 lines · Dust Li
Quoted 77 lines · Andrew Lunn
Quoted 54 lines · Niklas Schnelle
Quoted 28 lines · Andrew Lunn
Quoted 19 lines
Conceptually kind of but the existing s390 specific ISM device is a bit special. But let me start with some background. On s390 aka Mainframes OSs including Linux runs in so called logical partitions (LPARs) which are machine hypervisor VMs which use partitioned non-paging memory. The fact that memory is partitioned is important because this means LPARs can not share physical memory by mapping it.
Now at a high level an ISM device allows communication between two such Linux LPARs on the same machine. The device is discovered as a PCI device and allows Linux to take a buffer called a DMB map that in the IOMMU and generate a token specific to another LPAR which also sees an ISM device sharing the same virtual channel identifier (VCHID). This token can then be transferred out of band (e.g. as part of an extended TCP handshake in SMC-D) to that other system. With the token the other system can use its ISM device to securely (authenticated by the token, LPAR identity and the IOMMU mapping) write into the original systems DMB at throughput and latency similar to doing a
memcpy()via a syscall.Quoted 8 lines
On the implementation level the ISM device is actually a piece of firmware and the write to a remote DMB is a special case of our PCI Store Block instruction (no real MMIO on s390, instead there are special instructions). Sadly there are a few more quirks but in principle you can think of it as redirecting writes to a part of the ISM PCI devices' BAR to the DMB in the peer system if that makes sense. There's of course also a mechanism to cause an interrupt on the receiver as the write completes.
Quoted 3 lines
So the s390 details are interesting, but as you say, it is special. Ideally, all the special should be hidden away inside the driver.
Yes and it will be. There are some exceptions e.g. for vfio-pci pass- through but that's not unusual and why there is already the concept of vfio-pci extension module.
Quoted 2 lines
So please take a step back. What is the abstract model?
I think my high level description may be a good start. The abstract model is the ability to share a memory buffer (DMB) for writing by a communication partner, authenticated by a DMB Token. Plus stuff like triggering an interrupt on write or explicit trigger. Then Alibaba added optional support for what they called attaching the buffer which means it becomes truly shared between the peers but which IBM's ISM can't support. Plus a few more optional pieces such as VLANs, PNETIDs don't ask. The idea for the new layer then is to define this interface with operations and documentation.
Quoted 4 lines
Can the abstract model be mapped onto CLX? Could it be used with a GPU vRAM? SoC with real shared memory between a pool of CPUs.
Quoted 6 lines
Quoted 1 line
Andrew
I'd think that yes, one could implement such a mechanism on top of CXL as well as on SoC. Or even with no special hardware between a host and a DPU (e.g. via PCIe endpoint framework). Basically anything that can DMA and IRQs between two OS instances.
Is DMA part of the abstract model? That would suggest a true shared memory system is excluded, since that would not require DMA.
Maybe take a look at subsystems like USB, I2C.
usb_submit_urb(struct urb *urb, gfp_t mem_flags)An URB is a data structure with a block of memory associated with it, contains the detail to pass to the USB device.
i2c_transfer(struct i2c_adapter *adap, struct i2c_msg *msgs, int num)*msgs points to num of messages which get transferred to/from the I2C device.
Quoted 6 lines
Could the high level API look like this? No DMA, no IRQ, no concept of a somewhat shared memory. Just an API which asks for a message to be sent to the other end? struct urb has some USB concepts in it, struct
i2c_msghas some I2C concepts in it. A structism_msgwould follow the same pattern, but does it need to care about the DMA, the IRQ, the memory which is semi shared?
Quoted 5 lines
I don’t have a clear picture of what the API should look like yet, but I believe it’s possible to avoid DMA and IRQ. In fact, the current data transfer API,
ops->move_data()ininclude/linux/ism.h, already abstracts away the DMA and IRQ details.
What is central to ISM is the DMB (Direct Memory Buffer). The concept that there is a DMB dedicated to one writer and one reader. It is owned by the reader and only this writer can write at any offset into the DMB (Fabric controlled). (Reader can technically read/write as well).
So for the client API I think the core functions are
- move_data(*data, target_dmb_token, offset) - called by the sending client, to move data at some offset into a DMB.
- receive_signal(dmb_token, some_signal_info) - called by the ism layer to signal the client, that this DMB needs handling. (currently called handle_irq)
I would not want to abstract that to a message based API, because then we need queues etc and are almost at a net_device. All that is not needed for ism, because DMBs are dedicated to a single writer (who has the responsibility).
Quoted 7 lines
One thing we cannot hide, however, is whether the operation is zero-copy or copy. This distinction is important because we can reuse the data at different times in copy mode and zero-copy mode.
Best regards,
Dust
See my reply on 4/7, as well as Niklas' reply. Currently you can always re-use the send buffer. So zero-copy can be a property of the DMB (attach() function, etc. )
Quoted 8 lines
What is central to ISM is the DMB (Direct Memory Buffer). The concept that there is a DMB dedicated to one writer and one reader. It is owned by the reader and only this writer can write at any offset into the DMB (Fabric controlled). (Reader can technically read/write as well).
So for the client API I think the core functions are
-move_data(*data, target_dmb_token, offset)- called by the sending client, to move data at some offset into a DMB.
Missing a length, but otherwise this looks O.K.
Quoted 3 lines
-
receive_signal(dmb_token, some_signal_info)- called by the ism layer to signal the client, that this DMB needs handling. (currently calledhandle_irq)
So there is no indication where in the DMB there is new content?
And when you say "This DMB" does that imply there are multiple DMB shared between two peers?
Maybe i have the wrong idea about a DMB. I was thinking of maybe 64K to a few Mega bytes of memory, in a memory which could truly be shared by CPUs. But maybe a DMB is just a 4K Page, and you have lots of them? If you are 'faking' a shared memory with DMA, they can be anywhere in the address space where the DMA engine can access them.
Quoted 4 lines
I would not want to abstract that to a message based API, because then we need queues etc and are almost at a
net_device. All that is not needed for ism, because DMBs are dedicated to a single writer (who has the responsibility).
But i assume there are "protocols" above this. You talked about running a TTY over this. That should be standardized, so everybody implements TTYs in exactly the same way.
Quoted 3 lines
Quoted 3 lines
One thing we cannot hide, however, is whether the operation is zero-copy or copy. This distinction is important because we can reuse the data at different times in copy mode and zero-copy mode.
This needs more explanation. Are you talking about putting data into the DMB, or moving the DMB to the peer?
If you have a DMA engine
moving stuff around, the data can be anywhere the DMA engine can access. But if you have a true shared memory, ideally you want to avoid copying into it.
Then you have the API used by your protocol drivers above. For a TTY running at 9600 baud, a copy into the DMB does not matter. But if you are talking about a network protocol stack on top, your copy from user space to kernel space probably wants to go direct into the DMB. So maybe your API also needs to include allocating/freeing DMBs in an abstract way so it can hide the difference between true shared memory, and kernel memory which can be DMAed?
Andrew
Alexandra Winter
maintainerQuoted 10 lines · Andrew Lunn
Quoted 8 lines
What is central to ISM is the DMB (Direct Memory Buffer). The concept that there is a DMB dedicated to one writer and one reader. It is owned by the reader and only this writer can write at any offset into the DMB (Fabric controlled). (Reader can technically read/write as well).
So for the client API I think the core functions are
-move_data(*data, target_dmb_token, offset)- called by the sending client, to move data at some offset into a DMB.Missing a length, but otherwise this looks O.K.
Right, move_data() has a length field. My bad.
Quoted 7 lines
Quoted 4 lines
-
receive_signal(dmb_token, some_signal_info)- called by the ism layer to signal the client, that this DMB needs handling. (currently calledhandle_irq)So there is no indication where in the DMB there is new content?
The existing ism implementations pass a bit mask in 'some_signal_info' that can be used to signal which parts of the DMB have data to look at.
Quoted 3 lines
And when you say "This DMB" does that imply there are multiple DMB shared between two peers?
Yes, there can be multiple DMBs between the same two peers. And/or an ism device can provide multiple DMBs that are shared with different peers.
Quoted 6 lines
Maybe i have the wrong idea about a DMB. I was thinking of maybe 64K to a few Mega bytes of memory, in a memory which could truly be shared by CPUs. But maybe a DMB is just a 4K Page, and you have lots of them? If you are 'faking' a shared memory with DMA, they can be anywhere in the address space where the DMA engine can access them.
More the latter. Although they can be large, if the client or application wants to spend so much memory.
Which brings us back to the other thread, that ISM may not be the best name for this concept. MCD - 'Memory Communication Device', was a proposals without 'Shared' in the name...
Quoted 9 lines
Quoted 4 lines
I would not want to abstract that to a message based API, because then we need queues etc and are almost at a
net_device. All that is not needed for ism, because DMBs are dedicated to a single writer (who has the responsibility).But i assume there are "protocols" above this. You talked about running a TTY over this. That should be standardized, so everybody implements TTYs in exactly the same way.
Yes, the 'clients' are the protocols above this.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
One thing we cannot hide, however, is whether the operation is zero-copy or copy. This distinction is important because we can reuse the data at different times in copy mode and zero-copy mode.
This needs more explanation. Are you talking about putting data into the DMB, or moving the DMB to the peer?
The former: putting data into the DMB.
But yes the concept of attached, no-copy DMBs, that was introduced by ism-loopback needs a better description.
Quoted 13 lines
If you have a DMA engine
moving stuff around, the data can be anywhere the DMA engine can access. But if you have a true shared memory, ideally you want to avoid copying into it.Then you have the API used by your protocol drivers above. For a TTY running at 9600 baud, a copy into the DMB does not matter. But if you are talking about a network protocol stack on top, your copy from user space to kernel space probably wants to go direct into the DMB. So maybe your API also needs to include allocating/freeing DMBs in an abstract way so it can hide the difference between true shared memory, and kernel memory which can be DMAed?
Quoted 3 lines
Andrew
The ism_ops register_dmb() and unregister_dmb() are meant to provide that API.
Dust Li
maintainerQuoted 47 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line
---8<---
Quoted 23 lines
Quoted 7 lines
Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory
_D_evice?Oh, I was trying to refer to
SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.Quoted 11 lines
To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and byism_loopback. So what is the benefit in changing it?Quoted 8 lines
I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.
In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.
I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)
One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.
Quoted 6 lines
With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.
Make sense, I agree with MCD.
Best regard,
Dust
Alexandra Winter
maintainerQuoted 55 lines · Dust Li
Quoted 47 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line
---8<---
Quoted 23 lines
Quoted 7 lines
Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory
_D_evice?Oh, I was trying to refer to
SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.Quoted 11 lines
To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and byism_loopback. So what is the benefit in changing it?Quoted 8 lines
I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.
In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.
I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)
One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.
Quoted 6 lines
With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.
Quoted 5 lines
Make sense, I agree with MCD.
Best regard,
Dust
In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.
So what do your think of Direct Memory Communication (DMC) as class name for these devices?
I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.
Dust Li
maintainerQuoted 58 lines · Alexandra Winter
Quoted 55 lines · Dust Li
Quoted 53 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line
---8<---
Quoted 23 lines
Quoted 7 lines
Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory
_D_evice?Oh, I was trying to refer to
SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.Quoted 11 lines
To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and byism_loopback. So what is the benefit in changing it?Quoted 8 lines
I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.
In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.
I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)
One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.
With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.
Quoted 7 lines
Quoted 5 lines
Make sense, I agree with MCD.
Best regard,
Dust
Hi Winter,
Sorry for the late reply; we were on break for the Chinese Spring Festival.
Quoted 9 lines
In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.So what do your think of Direct Memory Communication (DMC) as class name for these devices?
I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.
I personally prefer MCD over Direct Memory Communication (DMC).
For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?
Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."
Best regards,
Dust
Alexandra Winter
maintainerQuoted 88 lines · Dust Li
Quoted 58 lines · Alexandra Winter
Quoted 55 lines · Dust Li
Quoted 53 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line
---8<---
Quoted 23 lines
Quoted 7 lines
Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory
_D_evice?Oh, I was trying to refer to
SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.Quoted 11 lines
To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and byism_loopback. So what is the benefit in changing it?Quoted 8 lines
I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.
In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.
I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)
One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.
With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.
Quoted 7 lines
Quoted 5 lines
Make sense, I agree with MCD.
Best regard,
DustHi Winter,
Sorry for the late reply; we were on break for the Chinese Spring Festival.
Quoted 9 lines
In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.So what do your think of Direct Memory Communication (DMC) as class name for these devices?
I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.
I personally prefer MCD over Direct Memory Communication (DMC).
For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?
Quoted 5 lines
Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."
Best regards,
Dust
Thank you for your thoughts, Dust.
For me the 'D as 'direct' is not so much about the number of copies, but more about the aspect, that you can directly write at any offset into the buffer. I.e. no queues. More like the D in DMA or RDMA.
I am preparing a talk for netdev in March about this subject, and the more I work on it, it seems to me that the buffers ('B'), that are
a) only authorized for a single remote device and
b) can be accessed at any offset
are the important differentiator compared other virtual devices.
So maybe 'D' for Dedicated?
I even came up with
dibs - Dedicated Internal Buffer Sharing or
dibc - Dedicated Internal Buffer Communication
(ok, I like the sound and look of the 'I'. But being on the same hardware as opposed to RDMA is also an important aspect.)
MCD - 'memory communication device' sounds rather vague to me. But if it is the smallest common denominator, i.e. the only thing we can all agree on, I could live with it.
Dust Li
maintainerQuoted 94 lines · Alexandra Winter
Quoted 91 lines · Dust Li
Quoted 58 lines · Alexandra Winter
Quoted 55 lines · Dust Li
Quoted 53 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line
---8<---
Quoted 23 lines
Quoted 7 lines
Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory
_D_evice?Oh, I was trying to refer to
SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.Quoted 11 lines
To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and byism_loopback. So what is the benefit in changing it?Quoted 8 lines
I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.
In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.
I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)
One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.
With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.
Quoted 7 lines
Quoted 5 lines
Make sense, I agree with MCD.
Best regard,
DustHi Winter,
Sorry for the late reply; we were on break for the Chinese Spring Festival.
Quoted 9 lines
In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.So what do your think of Direct Memory Communication (DMC) as class name for these devices?
I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.
I personally prefer MCD over Direct Memory Communication (DMC).
For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?
Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."
Quoted 7 lines
Quoted 2 lines
Best regards,
DustThank you for your thoughts, Dust.
For me the 'D as 'direct' is not so much about the number of copies, but more about the aspect, that you can directly write at any offset into the buffer. I.e. no queues. More like the D in DMA or RDMA.
Thanks for you explaintion. I think I understand you now.
Quoted 15 lines
I am preparing a talk for netdev in March about this subject, and the more I work on it, it seems to me that the buffers ('B'), that are
a) only authorized for a single remote device and
b) can be accessed at any offset
are the important differentiator compared other virtual devices.
So maybe 'D' for Dedicated?I even came up with
dibs - Dedicated Internal Buffer Sharing or
dibc - Dedicated Internal Buffer Communication
(ok, I like the sound and look of the 'I'. But being on the same hardware as opposed to RDMA is also an important aspect.)
Quoted 3 lines
MCD - 'memory communication device' sounds rather vague to me. But if it is the smallest common denominator, i.e. the only thing we can all agree on, I could live with it.
I've thought about it a bit more. Since DMC is at the same level as RDMA and fits well with the "D" in SMC-D, either DMC or MCD works for me.
Best regards,
Dust
Wen Gu
reviewerQuoted 94 lines · Alexandra Winter
Quoted 91 lines · Dust Li
Quoted 58 lines · Alexandra Winter
Quoted 55 lines · Dust Li
Quoted 53 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line
---8<---
Quoted 23 lines
Quoted 7 lines
Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory
_D_evice?Oh, I was trying to refer to
SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.Quoted 11 lines
To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and byism_loopback. So what is the benefit in changing it?Quoted 8 lines
I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.
In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.
I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)
One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.
With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.
Quoted 7 lines
Quoted 5 lines
Make sense, I agree with MCD.
Best regard,
DustHi Winter,
Sorry for the late reply; we were on break for the Chinese Spring Festival.
Quoted 9 lines
In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.So what do your think of Direct Memory Communication (DMC) as class name for these devices?
I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.
I personally prefer MCD over Direct Memory Communication (DMC).
For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?
Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."
Quoted 8 lines
Quoted 2 lines
Best regards,
DustThank you for your thoughts, Dust.
For me the 'D as 'direct' is not so much about the number of copies, but more about the aspect, that you can directly write at any offset into the buffer. I.e. no queues. More like the D in DMA or RDMA.
IMHO the 'D' means that the CPU copy does not need to be involved, and memory access only involves between memory and IO devices. So under this semantics, I think 'DMC' also applies to s390 ism device, since IIUC the s390 ism directly access to the memory which is passed down by move_data(). The exception is lo-ism, where the device actually doesn't need to access the memory(DMB), since the data has been put into the shared memory once the sendmsg() is called and no copy or move is needed. But this is not a violation of name, just a special kind of short-cut. So DMC makes sense to me.
Quoted 17 lines
I am preparing a talk for netdev in March about this subject, and the more I work on it, it seems to me that the buffers ('B'), that are
a) only authorized for a single remote device and
b) can be accessed at any offset
are the important differentiator compared other virtual devices.
So maybe 'D' for Dedicated?I even came up with
dibs - Dedicated Internal Buffer Sharing or
dibc - Dedicated Internal Buffer Communication
(ok, I like the sound and look of the 'I'. But being on the same hardware as opposed to RDMA is also an important aspect.)MCD - 'memory communication device' sounds rather vague to me. But if it is the smallest common denominator, i.e. the only thing we can all agree on, I could live with it.
Alexandra Winter
maintainerRe: [RFC net-next 0/7] Provide an ism layer - naming
Quoted 106 lines · Wen Gu
Quoted 94 lines · Alexandra Winter
Quoted 91 lines · Dust Li
Quoted 58 lines · Alexandra Winter
Quoted 55 lines · Dust Li
Quoted 53 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line
---8<---
Quoted 23 lines
Quoted 7 lines
Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory
_D_evice?Oh, I was trying to refer to
SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.Quoted 11 lines
To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and byism_loopback. So what is the benefit in changing it?Quoted 8 lines
I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.
In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.
I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)
One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.
With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.
Quoted 7 lines
Quoted 5 lines
Make sense, I agree with MCD.
Best regard,
DustHi Winter,
Sorry for the late reply; we were on break for the Chinese Spring Festival.
Quoted 9 lines
In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.So what do your think of Direct Memory Communication (DMC) as class name for these devices?
I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.
I personally prefer MCD over Direct Memory Communication (DMC).
For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?
Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."
Quoted 8 lines
Quoted 2 lines
Best regards,
DustThank you for your thoughts, Dust.
For me the 'D as 'direct' is not so much about the number of copies, but more about the aspect, that you can directly write at any offset into the buffer. I.e. no queues. More like the D in DMA or RDMA.
Quoted 26 lines
IMHO the 'D' means that the CPU copy does not need to be involved, and memory access only involves between memory and IO devices. So under this semantics, I think 'DMC' also applies to s390 ism device, since IIUC the s390 ism directly access to the memory which is passed down by
move_data(). The exception is lo-ism, where the device actually doesn't need to access the memory(DMB), since the data has been put into the shared memory once thesendmsg()is called and no copy or move is needed. But this is not a violation of name, just a special kind of short-cut. So DMC makes sense to me.Quoted 17 lines
I am preparing a talk for netdev in March about this subject, and the more I work on it, it seems to me that the buffers ('B'), that are
a) only authorized for a single remote device and
b) can be accessed at any offset
are the important differentiator compared other virtual devices.
So maybe 'D' for Dedicated?I even came up with
dibs - Dedicated Internal Buffer Sharing or
dibc - Dedicated Internal Buffer Communication
(ok, I like the sound and look of the 'I'. But being on the same hardware as opposed to RDMA is also an important aspect.)MCD - 'memory communication device' sounds rather vague to me. But if it is the smallest common denominator, i.e. the only thing we can all agree on, I could live with it.
Could you guys accept
'DIBS - Dedicated Internal Buffer Sharing'
as well?
-> dibs_layer, /class/dibs/, dibs_dev
That is currently my favourite.
Dust Li
maintainerRe: [RFC net-next 0/7] Provide an ism layer - naming
Quoted 137 lines · Alexandra Winter
Quoted 106 lines · Wen Gu
Quoted 94 lines · Alexandra Winter
Quoted 91 lines · Dust Li
Quoted 58 lines · Alexandra Winter
Quoted 55 lines · Dust Li
Quoted 53 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line
---8<---
Quoted 23 lines
Quoted 7 lines
Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory
_D_evice?Oh, I was trying to refer to
SHM(Share memory file in the userspace, see man shm_open(3)). SMD is also OK.Quoted 11 lines
To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and byism_loopback. So what is the benefit in changing it?Quoted 8 lines
I believe that if we are going to separate and refine the code, and add a common subsystem, we should choose the most appropriate name.
In my opinion, "ISM" doesn’t quite capture what the device provides. Since we’re adding a "Device" that enables different entities (such as processes or VMs) to perform shared memory communication, I think a more fitting name would be better. If you have any alternative suggestions, I’m open to them.
I kept thinking about this a bit and I'd like to propose yet another name for this group of devices: Memory Communication Devices (MCD)
One important point I see is that there is a bit of a misnomer in the existing ISM name in that our ISM device does in fact *not* share memory in the common sense of the "shared memory" wording. Instead it copies data between partitions of memory that share a common cache/memory hierarchy while not sharing the memory itself. loopback- ism and a possibly future virtio-ism on the other hand would share memory in the "shared memory" sense. Though I'd very much hope they will retain a copy mode to allow use in partition scenarios.
With that background I think the common denominator between them and the main idea behind ISM is that they facilitate communication via memory buffers and very simple and reliable copy/share operations. I think this would also capture our planned use-case of devices (TTYs, block devices, framebuffers + HID etc) provided by a peer on top of such a memory communication device.
Quoted 7 lines
Quoted 5 lines
Make sense, I agree with MCD.
Best regard,
DustHi Winter,
Sorry for the late reply; we were on break for the Chinese Spring Festival.
Quoted 9 lines
In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.So what do your think of Direct Memory Communication (DMC) as class name for these devices?
I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.
I personally prefer MCD over Direct Memory Communication (DMC).
For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?
Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."
Quoted 8 lines
Quoted 2 lines
Best regards,
DustThank you for your thoughts, Dust.
For me the 'D as 'direct' is not so much about the number of copies, but more about the aspect, that you can directly write at any offset into the buffer. I.e. no queues. More like the D in DMA or RDMA.Quoted 26 lines
IMHO the 'D' means that the CPU copy does not need to be involved, and memory access only involves between memory and IO devices. So under this semantics, I think 'DMC' also applies to s390 ism device, since IIUC the s390 ism directly access to the memory which is passed down by
move_data(). The exception is lo-ism, where the device actually doesn't need to access the memory(DMB), since the data has been put into the shared memory once thesendmsg()is called and no copy or move is needed. But this is not a violation of name, just a special kind of short-cut. So DMC makes sense to me.Quoted 17 lines
I am preparing a talk for netdev in March about this subject, and the more I work on it, it seems to me that the buffers ('B'), that are
a) only authorized for a single remote device and
b) can be accessed at any offset
are the important differentiator compared other virtual devices.
So maybe 'D' for Dedicated?I even came up with
dibs - Dedicated Internal Buffer Sharing or
dibc - Dedicated Internal Buffer Communication
(ok, I like the sound and look of the 'I'. But being on the same hardware as opposed to RDMA is also an important aspect.)MCD - 'memory communication device' sounds rather vague to me. But if it is the smallest common denominator, i.e. the only thing we can all agree on, I could live with it.
Quoted 7 lines
Could you guys accept
'DIBS - Dedicated Internal Buffer Sharing'
as well?
->dibs_layer, /class/dibs/,dibs_devThat is currently my favourite.
I think you might prefer a name that describes shared memory, but I personally believe that something reflecting the device itself would be more fitting.
To be honest, here’s my ranking:
MCD > DMC > DIBS
Best regards,
Dust
Alexandra Winter
maintainerRe: [RFC net-next 0/7] Provide an ism layer - naming
Quoted 95 lines · Dust Li
Quoted 82 lines · Alexandra Winter
Quoted 51 lines · Wen Gu
Quoted 39 lines · Alexandra Winter
Quoted 36 lines · Dust Li
Quoted 10 lines · Alexandra Winter
Quoted 4 lines · Dust Li
Quoted 3 lines · Niklas Schnelle
Quoted 1 line · Dust Li
Quoted 1 line
---8<---
Hi Winter,
Sorry for the late reply; we were on break for the Chinese Spring Festival.
Quoted 9 lines
In the discussion with Andrew Lunn, it showed that
a) we need an abstract description of 'ISM' devices (noted)
b) DMBs (Direct Memory Buffers) are a critical differentiator.So what do your think of Direct Memory Communication (DMC) as class name for these devices?
I don't have a strong preference (we could also stay with ISM). But DMC may be a bit more concrete than MCD or ISM.
I personally prefer MCD over Direct Memory Communication (DMC).
For loopback or Virtio-ISM, DMC seems like a good choice. However, for IBM ISM, since there's a DMA copy involved, it doesn’t seem truly "Direct," does it?
Additionally, since we are providing a device, MCD feels like a more fitting choice, as it aligns better with the concept of a "device."
Quoted 8 lines
Quoted 2 lines
Best regards,
DustThank you for your thoughts, Dust.
For me the 'D as 'direct' is not so much about the number of copies, but more about the aspect, that you can directly write at any offset into the buffer. I.e. no queues. More like the D in DMA or RDMA.Quoted 26 lines
IMHO the 'D' means that the CPU copy does not need to be involved, and memory access only involves between memory and IO devices. So under this semantics, I think 'DMC' also applies to s390 ism device, since IIUC the s390 ism directly access to the memory which is passed down by
move_data(). The exception is lo-ism, where the device actually doesn't need to access the memory(DMB), since the data has been put into the shared memory once thesendmsg()is called and no copy or move is needed. But this is not a violation of name, just a special kind of short-cut. So DMC makes sense to me.Quoted 17 lines
I am preparing a talk for netdev in March about this subject, and the more I work on it, it seems to me that the buffers ('B'), that are
a) only authorized for a single remote device and
b) can be accessed at any offset
are the important differentiator compared other virtual devices.
So maybe 'D' for Dedicated?I even came up with
dibs - Dedicated Internal Buffer Sharing or
dibc - Dedicated Internal Buffer Communication
(ok, I like the sound and look of the 'I'. But being on the same hardware as opposed to RDMA is also an important aspect.)MCD - 'memory communication device' sounds rather vague to me. But if it is the smallest common denominator, i.e. the only thing we can all agree on, I could live with it.
Quoted 7 lines
Could you guys accept
'DIBS - Dedicated Internal Buffer Sharing'
as well?
->dibs_layer, /class/dibs/,dibs_devThat is currently my favourite.
I think you might prefer a name that describes shared memory, but I personally believe that something reflecting the device itself would be more fitting.
Quoted 6 lines
To be honest, here’s my ranking:
MCD > DMC > DIBS
Best regards,
Dust
Thank you keeping the discussion going, Dust.
I know there is no perfect answer, but imo good names can make things easier to understand.
For reasons described above, I would like to have a 'B for buffer' in the prefix. There are many I/O concepts that share memory somehow, but the concept of a dmb dedicated to exactly 2 devices is a differentiator, imo.
I thought 'D for device' was somehow redundant and obvious. But you now you are saying that for you the device is a differentiator? As opposed to other memory sharing techniques that work without devices? Maybe you have a point...
So maybe DMB - Direct Memory Buffers is a good term?
I know we use it for the buffers already, but they are actually a common aspect for all devices and clients, right? So we could define a dmb layer with generic dmb_devices that can be used by dmb_clients to communicate via dmb_bufs.
Alexandra Winter
maintainerQuoted 18 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.Quoted 10 lines
But I understand that
net/smc/ism_loopback.ctoday does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal byism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separateism_loopbackkernel module, like you say. Todayism_loopbackis only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).
Quoted 6 lines
This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.
Best regards,
Dust
Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.
Quoted 25 lines · Alexandra Winter
Quoted 22 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.Quoted 10 lines
But I understand that
net/smc/ism_loopback.ctoday does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal byism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separateism_loopbackkernel module, like you say. Todayism_loopbackis only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.
Quoted 7 lines
Quoted 2 lines
Best regards,
DustIs there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.
Again, lack of basic understanding.... but why is it not a network device? Network devices are not just Ethernet. We also have CAN, SLIP, FDDI, etc.
Andrew
Alexandra Winter
maintainerQuoted 34 lines · Andrew Lunn
Quoted 25 lines · Alexandra Winter
Quoted 22 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.Quoted 10 lines
But I understand that
net/smc/ism_loopback.ctoday does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal byism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separateism_loopbackkernel module, like you say. Todayism_loopbackis only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.
Quoted 7 lines
Quoted 2 lines
Best regards,
DustIs there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.
Quoted 6 lines
Again, lack of basic understanding.... but why is it not a network device? Network devices are not just Ethernet. We also have CAN, SLIP, FDDI, etc.
Andrew
Thank you very much Andrew for spending the time and discussing this.
At the moment there is not usecase for attaching an ism interface via struct net_device to the linux network layer.
Current client is SMC-D and they bypass the network stack. Next client is a tty-over-ism console driver.
And Niklas Schnelle envisions TTYs, block devices, framebuffers over ISM. None of them do need queues or headers or other things the network stack offers, as long as the source can directly write into the target buffer. As mentioned earlier, probably one could write an network-over-ism driver, but nobody asks for it at the moment.
I have read about the Linux device model with devices and buses, but I have a bit of a hard time mapping that to an existing machine with classes, susbsystems, slots, subtypes, buses over buses and virtual things, etc that often have the same names for different things. So any advice for alternative placements of ism is welcome.
This is the kind of picture I have in my head:
SMC-sockets console
ISM clients: | |
+-----------------+ +-----------------+
| SMC-D | | tty-ism |
+-----------------+ +-----------------+
| |
+----------------------------------------------------------------------+
| ism layer |
| ism interfaces: |
| ism_vp0 ism_vp1, ism_lo, ism_virtio0, .. |
+----------------------------------------------------------------------+
| | |
+--------------+ +--------------+ +----------------+
| 0000:00:00.0 | | 0000:00:00.1 | | virtual/ism/lo |
+--------------+ +--------------+ +----------------+
\ /
+------------------------+
| pci bus |
+------------------------+ls /sys/class/ismism_vp0 -> ../../devices/pci0124:00/0124:00:00.0/ism/ism_vp0 ism_vp1 -> ../../devices/pci0125:00/0125:00:00.0/ism/ism_vp1 ism_lo -> ../../devices/virtual/ism/ism_lo
Maybe all that is overkill for ISM?
I think it would be very helpful, if they show up in sysfs, but not absolutely required.
Alexandra Winter
maintainerQuoted 25 lines · Alexandra Winter
Quoted 22 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.Quoted 10 lines
But I understand that
net/smc/ism_loopback.ctoday does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal byism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separateism_loopbackkernel module, like you say. Todayism_loopbackis only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.
Quoted 7 lines
Quoted 2 lines
Best regards,
DustIs there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.
Dust for some reason, you did not 'Reply-all':
Quoted 3 lines · Dust Li
I think dev_close()/dev_open() are the high-level APIs, while ndo_stop()/ndo_open() are the underlying device operations that we can reference.
I hear you, it can be beneficial to have a way for upper layers to enable/disable an ism device.
But all this is typically a tricky area. The device driver can also have reasons to enable/disable a device, then hardware could do that or even hotplug a device. Error recovery on different levels may want to run a disable/enable sequence as a reset, etc. And all this has potential for deadlocks.
All this is rather trivial for ism-loopback, as there is not much of a lower layer.
ism-vpci already has 'HW' / device driver configure on/off and device add/remove.
For a future ism-virtio, the Hipervisor may want to add/remove devices.
I wonder what could be the simplest definition of an enable/disable for the ism layer, that we can start with? More sophisticated functionality can always be added later.
Maybe support for add/remove ism-device by the device driver is sufficient as starting point?
Dust Li
maintainerQuoted 32 lines · Alexandra Winter
Quoted 29 lines · Alexandra Winter
Quoted 24 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.Quoted 10 lines
But I understand that
net/smc/ism_loopback.ctoday does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal byism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separateism_loopbackkernel module, like you say. Todayism_loopbackis only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.
Best regards,
Dust
Quoted 6 lines
Quoted 3 lines
Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.Dust for some reason, you did not 'Reply-all':
Oh, sorry I didn't notice that
Quoted 19 lines
Quoted 3 lines · Dust Li
I think dev_close()/dev_open() are the high-level APIs, while ndo_stop()/ndo_open() are the underlying device operations that we can reference.
I hear you, it can be beneficial to have a way for upper layers to enable/disable an ism device.
But all this is typically a tricky area. The device driver can also have reasons to enable/disable a device, then hardware could do that or even hotplug a device. Error recovery on different levels may want to run a disable/enable sequence as a reset, etc. And all this has potential for deadlocks.
All this is rather trivial for ism-loopback, as there is not much of a lower layer.
ism-vpci already has 'HW' / device driver configure on/off and device add/remove.
For a future ism-virtio, the Hipervisor may want to add/remove devices.
Quoted 5 lines
I wonder what could be the simplest definition of an enable/disable for the ism layer, that we can start with? More sophisticated functionality can always be added later.
Maybe support for add/remove ism-device by the device driver is sufficient as starting point?
I agree; this can be added later. For now, we can simply support unregistering a device from the device driver. Which is already handled by ism_dev_unregister() IIUC.
However, I believe we still need an API and the ability to enable or disable ISM devices from the upper layer. For example, if we want to disable a specific ISM device (such as the loopback device) in SMC, we should not do so by disabling the loopback device at the device layer, as it may also serve other clients beyond SMC.
Further more, I think removing the loopback from the loopback device driver seems unnecessory ? Since we should support that from the upper layer in the future.
Best regards,
Dust
Alexandra Winter
maintainerQuoted 71 lines · Dust Li
Quoted 32 lines · Alexandra Winter
Quoted 29 lines · Alexandra Winter
Quoted 24 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.Quoted 10 lines
But I understand that
net/smc/ism_loopback.ctoday does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal byism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separateism_loopbackkernel module, like you say. Todayism_loopbackis only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.
Best regards,
DustQuoted 6 lines
Quoted 3 lines
Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.Dust for some reason, you did not 'Reply-all':
Oh, sorry I didn't notice that
Quoted 19 lines
Quoted 3 lines · Dust Li
I think dev_close()/dev_open() are the high-level APIs, while ndo_stop()/ndo_open() are the underlying device operations that we can reference.
I hear you, it can be beneficial to have a way for upper layers to enable/disable an ism device.
But all this is typically a tricky area. The device driver can also have reasons to enable/disable a device, then hardware could do that or even hotplug a device. Error recovery on different levels may want to run a disable/enable sequence as a reset, etc. And all this has potential for deadlocks.
All this is rather trivial for ism-loopback, as there is not much of a lower layer.
ism-vpci already has 'HW' / device driver configure on/off and device add/remove.
For a future ism-virtio, the Hipervisor may want to add/remove devices.Quoted 5 lines
I wonder what could be the simplest definition of an enable/disable for the ism layer, that we can start with? More sophisticated functionality can always be added later.
Maybe support for add/remove ism-device by the device driver is sufficient as starting point?I agree; this can be added later. For now, we can simply support unregistering a device from the device driver. Which is already handled by
ism_dev_unregister()IIUC.
Quoted 5 lines
However, I believe we still need an API and the ability to enable or disable ISM devices from the upper layer. For example, if we want to disable a specific ISM device (such as the loopback device) in SMC, we should not do so by disabling the loopback device at the device layer, as it may also serve other clients beyond SMC.
Just a thought: not all clients have to use all available ism devices. The client could opt out without removing the device.
Quoted 7 lines
Further more, I think removing the loopback from the loopback device driver seems unnecessory ? Since we should support that from the upper layer in the future.
Best regards,
Dust
All good points. But it also shows that there are many options how to extend ism device handling of the upper layers / clients. e.g. I can image a loop macro ism_for_each_dev() might be nice... I'd prefer to take one step at a time. Start with a minimal useful ism layer and extend by usecase.
Alexandra Winter
maintainerQuoted 81 lines · Alexandra Winter
Quoted 71 lines · Dust Li
Quoted 32 lines · Alexandra Winter
Quoted 29 lines · Alexandra Winter
Quoted 24 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.Quoted 10 lines
But I understand that
net/smc/ism_loopback.ctoday does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal byism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separateism_loopbackkernel module, like you say. Todayism_loopbackis only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.
Best regards,
DustQuoted 6 lines
Quoted 3 lines
Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.Dust for some reason, you did not 'Reply-all':
Oh, sorry I didn't notice that
Quoted 19 lines
Quoted 3 lines · Dust Li
I think dev_close()/dev_open() are the high-level APIs, while ndo_stop()/ndo_open() are the underlying device operations that we can reference.
I hear you, it can be beneficial to have a way for upper layers to enable/disable an ism device.
But all this is typically a tricky area. The device driver can also have reasons to enable/disable a device, then hardware could do that or even hotplug a device. Error recovery on different levels may want to run a disable/enable sequence as a reset, etc. And all this has potential for deadlocks.
All this is rather trivial for ism-loopback, as there is not much of a lower layer.
ism-vpci already has 'HW' / device driver configure on/off and device add/remove.
For a future ism-virtio, the Hipervisor may want to add/remove devices.Quoted 5 lines
I wonder what could be the simplest definition of an enable/disable for the ism layer, that we can start with? More sophisticated functionality can always be added later.
Maybe support for add/remove ism-device by the device driver is sufficient as starting point?I agree; this can be added later. For now, we can simply support unregistering a device from the device driver. Which is already handled by
ism_dev_unregister()IIUC.Quoted 5 lines
However, I believe we still need an API and the ability to enable or disable ISM devices from the upper layer. For example, if we want to disable a specific ISM device (such as the loopback device) in SMC, we should not do so by disabling the loopback device at the device layer, as it may also serve other clients beyond SMC.
Quoted 7 lines
Just a thought: not all clients have to use all available ism devices. The client could opt out without removing the device.
Quoted 4 lines
Further more, I think removing the loopback from the loopback device driver seems unnecessory ? Since we should support that from the upper layer in the future.
If it is not too much effort, I would like to have a simple remove for ism_loopback soon, as it would allow for simple variations of testcases.
Quoted 19 lines
Quoted 3 lines
Best regards,
DustAll good points. But it also shows that there are many options how to extend ism device handling of the upper layers / clients. e.g. I can image a loop macro
ism_for_each_dev()might be nice... I'd prefer to take one step at a time. Start with a minimal useful ism layer and extend by usecase.
Dust Li
maintainerQuoted 88 lines · Alexandra Winter
Quoted 85 lines · Alexandra Winter
Quoted 71 lines · Dust Li
Quoted 32 lines · Alexandra Winter
Quoted 29 lines · Alexandra Winter
Quoted 24 lines · Dust Li
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.Quoted 10 lines
But I understand that
net/smc/ism_loopback.ctoday does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal byism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separateism_loopbackkernel module, like you say. Todayism_loopbackis only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).This works for me as well. I think it would be better to implement this within the common ISM layer, rather than duplicating the code in each device. Similar to how it's done in netdevice.
Best regards,
DustQuoted 6 lines
Quoted 3 lines
Is there a specific example for enable/disable in the netdevice code, you have in mind? Or do you mean in general how netdevice provides a common layer?
Yes, everything that is common for all devices should be provided by the network layer.Dust for some reason, you did not 'Reply-all':
Oh, sorry I didn't notice that
Quoted 19 lines
Quoted 3 lines · Dust Li
I think dev_close()/dev_open() are the high-level APIs, while ndo_stop()/ndo_open() are the underlying device operations that we can reference.
I hear you, it can be beneficial to have a way for upper layers to enable/disable an ism device.
But all this is typically a tricky area. The device driver can also have reasons to enable/disable a device, then hardware could do that or even hotplug a device. Error recovery on different levels may want to run a disable/enable sequence as a reset, etc. And all this has potential for deadlocks.
All this is rather trivial for ism-loopback, as there is not much of a lower layer.
ism-vpci already has 'HW' / device driver configure on/off and device add/remove.
For a future ism-virtio, the Hipervisor may want to add/remove devices.Quoted 5 lines
I wonder what could be the simplest definition of an enable/disable for the ism layer, that we can start with? More sophisticated functionality can always be added later.
Maybe support for add/remove ism-device by the device driver is sufficient as starting point?I agree; this can be added later. For now, we can simply support unregistering a device from the device driver. Which is already handled by
ism_dev_unregister()IIUC.Quoted 5 lines
However, I believe we still need an API and the ability to enable or disable ISM devices from the upper layer. For example, if we want to disable a specific ISM device (such as the loopback device) in SMC, we should not do so by disabling the loopback device at the device layer, as it may also serve other clients beyond SMC.
Just a thought: not all clients have to use all available ism devices. The client could opt out without removing the device.
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Further more, I think removing the loopback from the loopback device driver seems unnecessory ? Since we should support that from the upper layer in the future.
If it is not too much effort, I would like to have a simple remove for
ism_loopbacksoon, as it would allow for simple variations of testcases.
Yes, this is very useful for testing before we can do that from the upper layer.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Best regards,
Dust
Quoted 5 lines
Quoted 5 lines
All good points. But it also shows that there are many options how to extend ism device handling of the upper layers / clients. e.g. I can image a loop macro
ism_for_each_dev()might be nice... I'd prefer to take one step at a time. Start with a minimal useful ism layer and extend by usecase.
That works for me.
Best regards,
Dust
Quoted 2 lines
With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.
In Linux terminology, a device is something which has a struct device, and a device lives on some sort of bus, even if it is a virtual bus. Will ISM devices properly fit into the Linux device driver model?
Andrew
Alexandra Winter
maintainerQuoted 9 lines · Andrew Lunn
Quoted 2 lines
With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.In Linux terminology, a device is something which has a struct device, and a device lives on some sort of bus, even if it is a virtual bus. Will ISM devices properly fit into the Linux device driver model?
Andrew
ism_vpci lives on a pci bus (zpci flavor) today. The fact that it is not backed by a real hardware PCI slot, but emulated by s390 firmware is not visible to Linux.
In the first proposal, ism_lo lived in on a virtual bus, afaiu. I liked that. In the current stage 1 implementation that is currently upstream, it is not visible in sysfs :-(
ism_dev is a bit modeled after net_device. So it is contains a pointer to a struct device, but it is not the device itself.
I have to admit that the sysfs details are a bit confusing to me, so I wanted to discuss them first before adding them to the RFC. But I tried to bring all the prereqs in place.
Wen Gu
reviewerQuoted 118 lines · Alexandra Winter
Quoted 6 lines · Julian Ruess
Quoted 5 lines · Dust Li
On 2025-01-15 20:55:20, Alexandra Winter wrote:
Hi Winter,
I'm fully supportive of the refactor!
Thank you very much Dust Li for joining the discussion.
Quoted 3 lines
Quoted 3 lines
Interestingly, I developed a similar RFC code about a month ago while working on enhancing internal communication between guest and host systems.
But you did not send that out, did you?
I hope I did not overlook an earlier proposal by you.Here are some of my thoughts on the matter:
Quoted 3 lines
Quoted 3 lines
Naming and Structure: I suggest we refer to it as SHD (Shared Memory Device) instead of ISM (Internal Shared Memory).
So where does the 'H' come from? If you want to call it Shared Memory
_D_evice?To my knowledge, a
Quoted 2 lines
Quoted 2 lines
"Shared Memory Device" better encapsulates the functionality we're aiming to implement.
Could you explain why that would be better?
'Internal Shared Memory' is supposed to be a bit of a counterpart to the Remote 'R' in RoCE. Not the greatest name, but it is used already by our ISM devices and byism_loopback. So what is the benefit in changing it?It might be beneficial to place it under
Quoted 14 lines
Quoted 2 lines
drivers/shd/and register it as a new class under/sys/class/shd/. That said, my initial draft also adopted the ISM terminology for simplicity.I'm not sure if we really want to introduce a new name for the already existing ISM device. For me, having two names for the same thing just adds additional complexity.
I would go for
/sys/class/ismQuoted 5 lines
Modular Approach: I've made the
ism_loopbackan independent kernel module since dynamic enable/disable functionality is not yet supported in SMC. Using insmod and rmmod for module management could provide the flexibility needed in practical scenarios.With this proposal
ism_loopbackis just another ism device and SMC-D will handle removal just likeism_client.remove(ism_dev)of other ism devices.But I understand that
net/smc/ism_loopback.ctoday does not provide enable/disable, which is a big disadvantage, I agree. The ism layer is prepared for dynamic removal byism_dev_unregister(). In case of this RFC that would only happen in case of rmmod ism. Which should be improved.
One way to do that would be a separateism_loopbackkernel module, like you say. Todayism_loopbackis only 10k LOC, so I'd be fine with leaving it in the ism module. I also think it is a great way for testing any ISM client, so it has benefit for anybody using the ism module.
Another way would be e.g. an 'enable' entry in the sysfs of the loopback device. (Once we agree if and how to represent ism devices in genera in sysfs).Quoted 25 lines
Quoted 25 lines
Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate
ism->ops, making the implementation cleaner and more intuitive. This way, the structism_devicewould mainly serve its implementers, while the upper helper functions offer a streamlined interface for SMC.Structuring and Naming: I recommend embedding the structure of
ism_opsdirectly withinism_devrather than using a pointer. Additionally, renaming it toism_device_opscould enhance clarity and consistency.Quoted 8 lines
This RFC is about providing a generic shim layer between all kinds of ism devices and all kinds of ism users.
Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.Fully agree.
Quoted 1 line
[...]
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Ideas for next steps:
---------------------
- sysfs representation? e.g. as/sys/class/ism?
- provide a full-fledged ism loopback interface
(runtime enable/disable, sysfs device, ..)I think it's better if we can make this common for all ISM devices. but yeah, that shoud be the next step.
The s390
ism_vpcidevices are already backed by structpci_dev. And I assume that would be represented in sysfs somehow like:/sys/class/ism/ism_vp0/device-> /sys/devices/<pci bus no>/<pci dev no> so there is an
/sys/class/ism/<ism dev name>/device/enable entry already, because there is /sys/devices/<pci bus no>/<pci dev no>/enable today.
Quoted 4 lines
I remember Wen Gu's first proposal for
ism_loopbackhad a device in/sys/devices/virtual/and had an 'active' entry to enable/disable. Something like that could be linked to/sys/class/ism/ism_lo/device.
Yes, the previous proposal can be refered to [1]. And the hierarchy you mentioned makes sense to me.
[1] https://lore.kernel.org/netdev/[email protected]/
Quoted 8 lines
Quoted 8 lines
I already have patches based on this series that introduce
/sys/class/ismand show ism-loopback as well as s390/ism devices. I can send this soon.Julian
Alexandra Winter
maintainerI hit the send button to early, sorry about that. Let me comment on the other proposals from Dust Li as well.
Quoted 4 lines · Dust Li
Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate
ism->ops, making the implementation cleaner and more intuitive.
Maybe I misunderstand what you mean by helper functions.. Why would you encapsulate ism->ops functions in another set of wrappers? I was happy to remove the helper functions in 2/7 and 7/7.
This way, the struct ism_device would mainly serve its
Quoted 2 lines
implementers, while the upper helper functions offer a streamlined interface for SMC.
I was actually also wondering, whether the clients should access ism_device at all. Or whether they should only use the ism_ops.
I can give that a try in the next version. I think this RFC almost there already. The clients would still need to pass a poitner to ism_dev as a parameter.
Quoted 2 lines
Structuring and Naming: I recommend embedding the structure of
ism_opsdirectly withinism_devrather than using a pointer.
I think it is a common method to have the const struct xy_ops in the device driver code and then use pointer to register the device with an upper layer.
What would be the benefit of duplicating that struct in every ism_dev?
Additionally,
Quoted 1 line
renaming it to
ism_device_opscould enhance clarity and consistency.
Yes, that would help to distinguish them from the ism_client functions. I' rename them to ism_dev_ops in the next version.
Dust Li
maintainerQuoted 10 lines · Alexandra Winter
I hit the send button to early, sorry about that. Let me comment on the other proposals from Dust Li as well.
Quoted 4 lines · Dust Li
Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate
ism->ops, making the implementation cleaner and more intuitive.
Quoted 3 lines
Maybe I misunderstand what you mean by helper functions.. Why would you encapsulate
ism->opsfunctions in another set of wrappers? I was happy to remove the helper functions in 2/7 and 7/7.
What I mean is similar to how IB handles it in include/rdma/ib_verbs.h. A good example is ib_post_send or ibv_post_send in user space:
```c
static inline int ib_post_send(struct ib_qp *qp,
const struct ib_send_wr *send_wr,
const struct ib_send_wr **bad_send_wr)
{
const struct ib_send_wr *dummy;
return qp->device->ops.post_send(qp, send_wr, bad_send_wr ? : &dummy);
}
```By following this approach, we can "hide" all the implementations behind ism_xxx. Our users (SMC) should only interact with these APIs. The ism->ops would then be used by our device implementers (vISM, loopback, etc.). This would help make the layers clearer, which is the same approach IB takes.
The layout would somehow like this:
| -------------------- |-----------------------------| | ism_register_dmb() | | | ism_move_data() | <--- API for our users | | ism_xxx() ... | | | -------------------- |-----------------------------| | ism_device_ops | <---for our implementers | | | (PCI-ISM/loopback, etc) | |----------------------|-----------------------------|
Quoted 9 lines
This way, the struct
ism_devicewould mainly serve itsQuoted 2 lines
implementers, while the upper helper functions offer a streamlined interface for SMC.
I was actually also wondering, whether the clients should access
ism_deviceat all. Or whether they should only use theism_ops.
I believe the client should only pass an ism_dev pointer to the ism_xxx() helper functions. They should never directly access any of the fields inside the ism_dev.
Quoted 10 lines
I can give that a try in the next version. I think this RFC almost there already. The clients would still need to pass a poitner to
ism_devas a parameter.Quoted 2 lines
Structuring and Naming: I recommend embedding the structure of
ism_opsdirectly withinism_devrather than using a pointer.I think it is a common method to have the const struct
xy_opsin the device driver code and then use pointer to register the device with an upper layer.
Right, If we have many ism_devs for each one ISM type, then using pointer should save us some memory.
Quoted 1 line
What would be the benefit of duplicating that struct in every
ism_dev?
The main benefit of embedding ism_device_ops within ism_dev is that it reduces the dereferencing of an extra pointer. We already have too many dereference in the datapath, it is not good for performance :(
For example:
rc = smcd->ism->ops->move_data(smcd->ism, dmb_tok, idx, sf, offset,
data, len);
Best regards,
Dust
Alexandra Winter
maintainerQuoted 36 lines · Dust Li
Quoted 10 lines · Alexandra Winter
I hit the send button to early, sorry about that. Let me comment on the other proposals from Dust Li as well.
Quoted 4 lines · Dust Li
Abstraction of ISM Device Details: I propose we abstract the ISM device details by providing SMC with helper functions. These functions could encapsulate
ism->ops, making the implementation cleaner and more intuitive.Quoted 3 lines
Maybe I misunderstand what you mean by helper functions.. Why would you encapsulate
ism->opsfunctions in another set of wrappers? I was happy to remove the helper functions in 2/7 and 7/7.What I mean is similar to how IB handles it in
include/rdma/ib_verbs.h. A good example isib_post_sendoribv_post_sendin user space:```c static inline int ib_post_send(struct ib_qp *qp, const struct ib_send_wr *send_wr, const struct ib_send_wr **bad_send_wr) { const struct ib_send_wr *dummy; return qp->device->ops.post_send(qp, send_wr, bad_send_wr ? : &dummy); } ```By following this approach, we can "hide" all the implementations behind
ism_xxx. Our users (SMC) should only interact with these APIs. Theism->opswould then be used by our device implementers (vISM, loopback, etc.). This would help make the layers clearer, which is the same approach IB takes.The layout would somehow like this:
Quoted 17 lines
| -------------------- |-----------------------------| | ism_register_dmb() | | | ism_move_data() | <--- API for our users | | ism_xxx() ... | | | -------------------- |-----------------------------| | ism_device_ops | <---for our implementers | | | (PCI-ISM/loopback, etc) | |----------------------|-----------------------------|Quoted 7 lines
This way, the struct
ism_devicewould mainly serve itsQuoted 2 lines
implementers, while the upper helper functions offer a streamlined interface for SMC.
Thanks for the explanations.
Yes, probably makes sense to further decouple the client API from the device API. I'll give that a try in the next version.
Quoted 28 lines
Quoted 2 lines
I was actually also wondering, whether the clients should access
ism_deviceat all. Or whether they should only use theism_ops.I believe the client should only pass an
ism_devpointer to theism_xxx()helper functions. They should never directly access any of the fields inside theism_dev.Quoted 10 lines
I can give that a try in the next version. I think this RFC almost there already. The clients would still need to pass a poitner to
ism_devas a parameter.Quoted 2 lines
Structuring and Naming: I recommend embedding the structure of
ism_opsdirectly withinism_devrather than using a pointer.I think it is a common method to have the const struct
xy_opsin the device driver code and then use pointer to register the device with an upper layer.Right, If we have many
ism_devsfor each one ISM type, then using pointer should save us some memory.Quoted 1 line
What would be the benefit of duplicating that struct in every
ism_dev?The main benefit of embedding
ism_device_opswithinism_devis that it reduces the dereferencing of an extra pointer. We already have too many dereference in the datapath, it is not good for performance :(
Quoted 8 lines
For example:
rc = smcd->ism->ops->move_data(smcd->ism,
dmb_tok, idx, sf, offset,data, len);Best regards,
Dust
I see your point. I'm not yet convinced. I'll think more about it.