Skip to contentSkip to conversation

dibs - Direct Internal Buffer Sharing

RFC In review Alexandra Winter 5 participants 52 messages Aug 6, 2025 – Sep 1, 2025

Subsystems (7)
lore
Share & tools
lore

Alexandra Winter

author maintainer
#0 lore raw

cover dibs - Direct Internal Buffer Sharing

This series introduces a generic abstraction of existing components like:
- the s390 specific ISM device (Internal Shared Memory),
- the SMC-D loopback mechanism (Shared Memory Communication - Direct)
- the client interface of the SMC-D module to the transport devices This generic shim layer can be extended with more devices, more clients and more features in the future.

This layer is called 'dibs' for Direct Internal Buffer Sharing based on the common scheme that these mechanisms enable controlled sharing of memory buffers within some containing entity such as a hypervisor or a Linux instance.

Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.

An overview was given at the Netdev 0x19 conference, recordings and slides are available [1].

Background / Status quo:
------------------------
Currently s390 hardware provides virtual PCI ISM devices (Internal Shared Memory). Their driver is in drivers/s390/net/ism_drv.c. The main user is SMC-D (net/smc). The ism driver offers a client interface so other users/protocols can also use them, but it is still heavily intermingled with the smc code. Namely, the ism module cannot be used without the smc module, which feels artificial.

There is ongoing work to extend the ISM concept of shared buffers that can be accessed directly by another instance on the same hardware: [2] proposed a loopback interface (ism_lo), that can be used on non-s390 architectures (e.g. between containers or to test SMC-D). A minimal implementation went upstream with [3]: ism_lo currently is a part of the smc protocol and rather hidden.

[4] proposed a virtio definition of ism (ism_virtio) that can be used between kvm guests.

We will shortly send an RFC for an dibs client that uses dibs as transport for TTY.

Concept:
--------
Create a shim layer in net/dibs that contains common definitions and code for all dibs devices and all dibs clients. Any device or client module only needs to depend on this dibs layer module and any device or client code only needs to include the definitions in include/linux/dibs.h.

The name dibs was chosen to clearly distinguish it from the existing s390 ism devices. And to emphasize that it is not about sharing whole memory regions with anybody, but dedicating single buffers for another system.

Implementation:
---------------
The end result of this series is: A dibs shim layer with One dibs client: smc-d
Two dibs device drivers: ism and dibs-loopback Everything prepared to add more clients and more device drivers.

As net-next is still closed, I am sending this series as RFC to give reviewers an early start (vacation season). This series is based on net-next tag 'bpf-next-6.17' and contains everything I want to include for the initial implementation.

Patches 1-5 contain some issues that were found along the way. They make sense on their own, but also enable a better structured dibs series.

There are three components that exist today:
a) smc module (especially SMC-D functionality, which is an ism client today)
b) ism device driver (supports multiple ism clients today)
c) smc-loopback (integrated with smc today)
In order to preserve existing functionality at each step, these are not moved to dibs layer by component, instead:
- the dibs layer is established in parallel to existing code [patches 6-9]
- then some service functions are moved to the dibs layer [patches 10-15]
- the actual data movement is moved to the dibs layer [patch 16]
- and last event handling is moved to the dibs layer [patch 17]

Future:
-------
Items that are not part of this patchset but could be added later:
- dynamically add or remove dibs_loopback. That will be allow for simple testing of add_dev()/del_dev()
- handle_irq(): Call clients without interrupt context. e.g using threaded interrupts. I left this for a follow-on, because it includes conceptual changes for the smcd receive code.
- Any improvements of locking scopes. I mainly moved some of the the existing locks to dibs layer. I have the feeling there is room for improvements.

Alexandra Winter (14):
net/smc: Remove __init marker from smc_core_init() s390/ism: Log module load/unload
net/smc: Remove error handling of unregister_dmb() net/smc: Decouple sf and attached send_buf in smc_loopback net/smc: Improve log message for devices w/o pnetid net/dibs: Create net/dibs
net/dibs: Register smc as dibs_client
net/dibs: Register ism as dibs device
net/dibs: Define dibs loopback
net/dibs: Define dibs_client_ops and dibs_dev_ops net/dibs: Local gid for dibs devices
net/dibs: Move vlan support to dibs_dev_ops net/dibs: Move query_remote_gid() to dibs_dev_ops net/dibs: Move data path to dibs layer

Julian Ruess (3):
net/dibs: Move struct device to dibs_dev
net/dibs: Create class dibs
net/dibs: Move event handling to dibs layer

 MAINTAINERS                |   9 +-
 drivers/s390/net/Kconfig   |   3 +-
 drivers/s390/net/ism.h     |  53 +++-
 drivers/s390/net/ism_drv.c | 580 ++++++++++++++-----------------------
 include/linux/dibs.h       | 479 ++++++++++++++++++++++++++++++
 include/linux/ism.h        |  93 ------
 include/net/smc.h          |  51 +---
 net/Kconfig                |   1 +
 net/Makefile               |   1 +
 net/dibs/Kconfig           |  27 ++
 net/dibs/Makefile          |   8 +
 net/dibs/dibs_loopback.c   | 355 +++++++++++++++++++++++
 net/dibs/dibs_loopback.h   |  57 ++++
 net/dibs/dibs_main.c       | 280 ++++++++++++++++++
 net/smc/Kconfig            |  16 +-
 net/smc/Makefile           |   1 -
 net/smc/af_smc.c           |  12 +-
 net/smc/smc_clc.c          |   6 +-
 net/smc/smc_core.c         |   8 +-
 net/smc/smc_core.h         |   5 +
 net/smc/smc_diag.c         |   2 +-
 net/smc/smc_ib.c           |  18 +-
 net/smc/smc_ism.c          | 233 ++++++++-------
 net/smc/smc_ism.h          |  36 ++-
 net/smc/smc_loopback.c     | 421 ---------------------------
 net/smc/smc_loopback.h     |  60 ----
 net/smc/smc_pnet.c         |   8 +-
 net/smc/smc_tx.c           |   3 +
 28 files changed, 1672 insertions(+), 1154 deletions(-)
 create mode 100644 include/linux/dibs.h
 delete mode 100644 include/linux/ism.h
 create mode 100644 net/dibs/Kconfig
 create mode 100644 net/dibs/Makefile
 create mode 100644 net/dibs/dibs_loopback.c
 create mode 100644 net/dibs/dibs_loopback.h
 create mode 100644 net/dibs/dibs_main.c
 delete mode 100644 net/smc/smc_loopback.c
 delete mode 100644 net/smc/smc_loopback.h
Signature · 1 line
2.48.1
author maintainer
#1 lore raw

RFC 1/17 net/smc: Remove __init marker from smc_core_init()

Remove the __init marker because smc_core_init() is not the init function of the smc module and for consistency with smc_core_exit() which neither has an __exit marker.

---
 net/smc/smc_core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/smc/smc_core.c +1 -1
index 262746e304dd..67f9e0b83ebc 100644
@@ -2758,7 +2758,7 @@ static struct notifier_block smc_reboot_notifier = {
 	.notifier_call = smc_core_reboot_event,
 };
 
-int __init smc_core_init(void)
+int smc_core_init(void)
 {
 	return register_reboot_notifier(&smc_reboot_notifier);
 }
Signature · 1 line
2.48.1
author maintainer
#5 lore raw

RFC 3/17 net/smc: Remove error handling of unregister_dmb()

In reply to Alexandra Winter

smcd_buf_free() calls smc_ism_unregister_dmb(lgr->smcd, buf_desc) and then unconditionally frees buf_desc.

Remove the cleaning up of fields of buf_desc in smc_ism_unregister_dmb(), because it is not helpful.

This removes the only usage of ISM_ERROR from the smc module. So move it to drivers/s390/net/ism.h.

---
 drivers/s390/net/ism.h |  1 +
 include/net/smc.h      |  2 --
 net/smc/smc_ism.c      | 14 +++++---------
 net/smc/smc_ism.h      |  3 ++-
 4 files changed, 8 insertions(+), 12 deletions(-)
drivers/s390/net/ism.h +1 -0
index 047fa6101555..b5b03db52fce 100644
@@ -10,6 +10,7 @@
 #include <asm/pci_insn.h>
 
 #define UTIL_STR_LEN	16
+#define ISM_ERROR	0xFFFF
 
 /*
  * Do not use the first word of the DMB bits to ensure 8 byte aligned access.
include/net/smc.h +0 -2
index db84e4e35080..a9c023dd1380 100644
@@ -44,8 +44,6 @@ struct smcd_dmb {
 
 #define ISM_RESERVED_VLANID	0x1FFF
 
-#define ISM_ERROR	0xFFFF
-
 struct smcd_dev;
 
 struct smcd_gid {
net/smc/smc_ism.c +5 -9
index 84f98e18c7db..a94e1450d095 100644
@@ -205,13 +205,13 @@ int smc_ism_put_vlan(struct smcd_dev *smcd, unsigned short vlanid)
 	return rc;
 }
 
-int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc)
+void smc_ism_unregister_dmb(struct smcd_dev *smcd,
+			    struct smc_buf_desc *dmb_desc)
 {
 	struct smcd_dmb dmb;
-	int rc = 0;
 
 	if (!dmb_desc->dma_addr)
-		return rc;
+		return;
 
 	memset(&dmb, 0, sizeof(dmb));
 	dmb.dmb_tok = dmb_desc->token;
@@ -219,13 +219,9 @@ int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc)
 	dmb.cpu_addr = dmb_desc->cpu_addr;
 	dmb.dma_addr = dmb_desc->dma_addr;
 	dmb.dmb_len = dmb_desc->len;
-	rc = smcd->ops->unregister_dmb(smcd, &dmb);
-	if (!rc || rc == ISM_ERROR) {
-		dmb_desc->cpu_addr = NULL;
-		dmb_desc->dma_addr = 0;
-	}
+	smcd->ops->unregister_dmb(smcd, &dmb);
 
-	return rc;
+	return;
 }
 
 int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len,
net/smc/smc_ism.h +2 -1
index 6763133dd8d0..765aa8fae6fa 100644
@@ -47,7 +47,8 @@ int smc_ism_get_vlan(struct smcd_dev *dev, unsigned short vlan_id);
 int smc_ism_put_vlan(struct smcd_dev *dev, unsigned short vlan_id);
 int smc_ism_register_dmb(struct smc_link_group *lgr, int buf_size,
 			 struct smc_buf_desc *dmb_desc);
-int smc_ism_unregister_dmb(struct smcd_dev *dev, struct smc_buf_desc *dmb_desc);
+void smc_ism_unregister_dmb(struct smcd_dev *dev,
+			    struct smc_buf_desc *dmb_desc);
 bool smc_ism_support_dmb_nocopy(struct smcd_dev *smcd);
 int smc_ism_attach_dmb(struct smcd_dev *dev, u64 token,
 		       struct smc_buf_desc *dmb_desc);
Signature · 1 line
2.48.1
author maintainer
#43 lore raw

RFC 14/17 net/dibs: Move vlan support to dibs_dev_ops

In reply to Alexandra Winter

It can be debated how much benefit definition of vlan ids for dibs devices brings, as the dmbs are accessible only by a single peer anyhow. But ism provides vlan support and smcd exploits it, so move it to dibs layer as an optional feature.

smcd_loopback simply ignores all vlan settings, do the same in dibs_loopback.

SMC-D and ISM have a method to use the invalid VLAN ID 1FFF (ISM_RESERVED_VLANID), to indicate that both communication peers support routable SMC-Dv2. Tolerate it in dibs, but move it to SMC only.

---
 drivers/s390/net/ism_drv.c | 47 +++++---------------------------------
 include/linux/dibs.h       | 19 +++++++++++++++
 include/net/smc.h          |  5 ----
 net/smc/smc_ism.c          | 14 ++++++++----
 net/smc/smc_loopback.c     |  5 ----
 5 files changed, 34 insertions(+), 56 deletions(-)
drivers/s390/net/ism_drv.c +6 -41
index 2dbf4ebb6bff..636594b2fab2 100644
@@ -36,7 +36,6 @@ static struct ism_client *clients[MAX_CLIENTS];	/* use an array rather than */
 						/* a list for fast mapping  */
 static u8 max_client;
 static DEFINE_MUTEX(clients_lock);
-static bool ism_v2_capable;
 struct ism_dev_list {
 	struct list_head list;
 	struct mutex mutex; /* protects ism device list */
@@ -409,8 +408,9 @@ int ism_unregister_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
 }
 EXPORT_SYMBOL_GPL(ism_unregister_dmb);
 
-static int ism_add_vlan_id(struct ism_dev *ism, u64 vlan_id)
+static int ism_add_vlan_id(struct dibs_dev *dibs, u64 vlan_id)
 {
+	struct ism_dev *ism = dibs->drv_priv;
 	union ism_set_vlan_id cmd;
 
 	memset(&cmd, 0, sizeof(cmd));
@@ -422,8 +422,9 @@ static int ism_add_vlan_id(struct ism_dev *ism, u64 vlan_id)
 	return ism_cmd(ism, &cmd);
 }
 
-static int ism_del_vlan_id(struct ism_dev *ism, u64 vlan_id)
+static int ism_del_vlan_id(struct dibs_dev *dibs, u64 vlan_id)
 {
+	struct ism_dev *ism = dibs->drv_priv;
 	union ism_set_vlan_id cmd;
 
 	memset(&cmd, 0, sizeof(cmd));
@@ -536,6 +537,8 @@ static irqreturn_t ism_handle_irq(int irq, void *data)
 
 static const struct dibs_dev_ops ism_ops = {
 	.get_fabric_id = ism_get_chid,
+	.add_vlan_id = ism_add_vlan_id,
+	.del_vlan_id = ism_del_vlan_id,
 };
 
 static int ism_dev_init(struct ism_dev *ism)
@@ -565,12 +568,6 @@ static int ism_dev_init(struct ism_dev *ism)
 	if (ret)
 		goto unreg_sba;
 
-	if (!ism_add_vlan_id(ism, ISM_RESERVED_VLANID))
-		/* hardware is V2 capable */
-		ism_v2_capable = true;
-	else
-		ism_v2_capable = false;
-
 	mutex_lock(&ism_dev_list.mutex);
 	mutex_lock(&clients_lock);
 	for (i = 0; i < max_client; ++i) {
@@ -611,8 +608,6 @@ static void ism_dev_exit(struct ism_dev *ism)
 
 	mutex_lock(&ism_dev_list.mutex);
 
-	if (ism_v2_capable)
-		ism_del_vlan_id(ism, ISM_RESERVED_VLANID);
 	unregister_ieq(ism);
 	unregister_sba(ism);
 	free_irq(pci_irq_vector(pdev, 0), ism);
@@ -791,26 +786,6 @@ static int smcd_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
 	return ism_unregister_dmb(smcd->priv, (struct ism_dmb *)dmb);
 }
 
-static int smcd_add_vlan_id(struct smcd_dev *smcd, u64 vlan_id)
-{
-	return ism_add_vlan_id(smcd->priv, vlan_id);
-}
-
-static int smcd_del_vlan_id(struct smcd_dev *smcd, u64 vlan_id)
-{
-	return ism_del_vlan_id(smcd->priv, vlan_id);
-}
-
-static int smcd_set_vlan_required(struct smcd_dev *smcd)
-{
-	return ism_cmd_simple(smcd->priv, ISM_SET_VLAN);
-}
-
-static int smcd_reset_vlan_required(struct smcd_dev *smcd)
-{
-	return ism_cmd_simple(smcd->priv, ISM_RESET_VLAN);
-}
-
 static int ism_signal_ieq(struct ism_dev *ism, u64 rgid, u32 trigger_irq,
 			  u32 event_code, u64 info)
 {
@@ -842,22 +817,12 @@ static int smcd_move(struct smcd_dev *smcd, u64 dmb_tok, unsigned int idx,
 	return ism_move(smcd->priv, dmb_tok, idx, sf, offset, data, size);
 }
 
-static int smcd_supports_v2(void)
-{
-	return ism_v2_capable;
-}
-
 static const struct smcd_ops ism_smcd_ops = {
 	.query_remote_gid = smcd_query_rgid,
 	.register_dmb = smcd_register_dmb,
 	.unregister_dmb = smcd_unregister_dmb,
-	.add_vlan_id = smcd_add_vlan_id,
-	.del_vlan_id = smcd_del_vlan_id,
-	.set_vlan_required = smcd_set_vlan_required,
-	.reset_vlan_required = smcd_reset_vlan_required,
 	.signal_event = smcd_signal_ieq,
 	.move_data = smcd_move,
-	.supports_v2 = smcd_supports_v2,
 };
 
 const struct smcd_ops *ism_get_smcd_ops(void)
include/linux/dibs.h +19 -0
index 058dfc14aa5c..10be10ae4660 100644
@@ -133,6 +133,25 @@ struct dibs_dev_ops {
 	 * Return: 2 byte dibs fabric id
 	 */
 	u16 (*get_fabric_id)(struct dibs_dev *dev);
+	/**
+	 * add_vlan_id() - add dibs device to vlan (optional, deprecated)
+	 * @dev: dibs device
+	 * @vlan_id: vlan id
+	 *
+	 * In order to write into a vlan-tagged dmb, the remote device needs
+	 * to belong to the this vlan. A device can belong to more than 1 vlan.
+	 * Any device can access an untagged dmb.
+	 * Deprecated, only supported for backwards compatibility.
+	 * Return: zero on success
+	 */
+	int (*add_vlan_id)(struct dibs_dev *dev, u64 vlan_id);
+	/**
+	 * del_vlan_id() - remove dibs device from vlan (optional, deprecated)
+	 * @dev: dibs device
+	 * @vlan_id: vlan id
+	 * Return: zero on success
+	 */
+	int (*del_vlan_id)(struct dibs_dev *dev, u64 vlan_id);
 };
 
 struct dibs_dev {
include/net/smc.h +0 -5
index 9cb8385bbc6e..51b4aefc106a 100644
@@ -61,13 +61,8 @@ struct smcd_ops {
 	int (*move_data)(struct smcd_dev *dev, u64 dmb_tok, unsigned int idx,
 			 bool sf, unsigned int offset, void *data,
 			 unsigned int size);
-	int (*supports_v2)(void);
 
 	/* optional operations */
-	int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
-	int (*del_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
-	int (*set_vlan_required)(struct smcd_dev *dev);
-	int (*reset_vlan_required)(struct smcd_dev *dev);
 	int (*signal_event)(struct smcd_dev *dev, struct smcd_gid *rgid,
 			    u32 trigger_irq, u32 event_code, u64 info);
 	int (*support_dmb_nocopy)(struct smcd_dev *dev);
net/smc/smc_ism.c +9 -5
index dd46d8000381..ce9d2195e14a 100644
@@ -140,7 +140,7 @@ int smc_ism_get_vlan(struct smcd_dev *smcd, unsigned short vlanid)
 
 	if (!vlanid)			/* No valid vlan id */
 		return -EINVAL;
-	if (!smcd->ops->add_vlan_id)
+	if (!smcd->dibs->ops->add_vlan_id)
 		return -EOPNOTSUPP;
 
 	/* create new vlan entry, in case we need it */
@@ -163,7 +163,7 @@ int smc_ism_get_vlan(struct smcd_dev *smcd, unsigned short vlanid)
 	/* no existing entry found.
 	 * add new entry to device; might fail, e.g., if HW limit reached
 	 */
-	if (smcd->ops->add_vlan_id(smcd, vlanid)) {
+	if (smcd->dibs->ops->add_vlan_id(smcd->dibs, vlanid)) {
 		kfree(new_vlan);
 		rc = -EIO;
 		goto out;
@@ -187,7 +187,7 @@ int smc_ism_put_vlan(struct smcd_dev *smcd, unsigned short vlanid)
 
 	if (!vlanid)			/* No valid vlan id */
 		return -EINVAL;
-	if (!smcd->ops->del_vlan_id)
+	if (!smcd->dibs->ops->del_vlan_id)
 		return -EOPNOTSUPP;
 
 	spin_lock_irqsave(&smcd->lock, flags);
@@ -205,7 +205,7 @@ int smc_ism_put_vlan(struct smcd_dev *smcd, unsigned short vlanid)
 	}
 
 	/* Found and the last reference just gone */
-	if (smcd->ops->del_vlan_id(smcd, vlanid))
+	if (smcd->dibs->ops->del_vlan_id(smcd->dibs, vlanid))
 		rc = -EIO;
 	list_del(&vlan->list);
 	kfree(vlan);
@@ -536,8 +536,12 @@ static void smcd_register_dev(struct dibs_dev *dibs)
 
 	smcd->client = &smc_ism_client;
 
-	if (smcd->ops->supports_v2())
+	if (smc_ism_is_loopback(dibs) ||
+	    (dibs->ops->add_vlan_id &&
+	     !dibs->ops->add_vlan_id(dibs, ISM_RESERVED_VLANID))) {
 		smc_ism_set_v2_capable();
+	}
+
 	mutex_lock(&smcd_dev_list.mutex);
 	/* sort list:
 	 * - devices without pnetid before devices with pnetid;
net/smc/smc_loopback.c +0 -5
index 454d9d6a6e8f..982a19430313 100644
@@ -20,7 +20,6 @@
 #include "smc_ism.h"
 #include "smc_loopback.h"
 
-#define SMC_LO_V2_CAPABLE	0x1 /* loopback-ism acts as ISMv2 */
 #define SMC_LO_SUPPORT_NOCOPY	0x1
 #define SMC_DMA_ADDR_INVALID	(~(dma_addr_t)0)
 
@@ -242,10 +241,6 @@ static const struct smcd_ops lo_ops = {
 	.support_dmb_nocopy = smc_lo_support_dmb_nocopy,
 	.attach_dmb = smc_lo_attach_dmb,
 	.detach_dmb = smc_lo_detach_dmb,
-	.add_vlan_id		= NULL,
-	.del_vlan_id		= NULL,
-	.set_vlan_required	= NULL,
-	.reset_vlan_required	= NULL,
 	.signal_event		= NULL,
 	.move_data = smc_lo_move_data,
 };
Signature · 1 line
2.48.1
author maintainer
#9 lore raw

RFC 4/17 net/smc: Decouple sf and attached send_buf in smc_loopback

In reply to Alexandra Winter

Before this patch there was the following assumption in smc_loopback.c>smc_lo_move_data():
sf (signalling flag) == 0 : data is already in an attached target dmb sf == 1 : data is not yet in the target dmb

This is true for the 2 callers in smc client
smcd_cdc_msg_send() : sf=1
smcd_tx_rdma_writes() : sf=0
but should not be a general assumption.

Add a bool to struct smc_buf_desc to indicate whether an SMC-D sndbuf_desc is an attached buffer. Don't call move_data() for attached send_buffers, because it is not necessary.

Move the data in smc_lo_move_data() if len != 0 and signal when requested.

---
 net/smc/smc_core.h     | 5 +++++
 net/smc/smc_ism.c      | 1 +
 net/smc/smc_loopback.c | 9 +++------
 net/smc/smc_tx.c       | 3 +++
 4 files changed, 12 insertions(+), 6 deletions(-)
net/smc/smc_core.h +5 -0
index 48a1b1dcb576..fe5f48d14323 100644
@@ -13,6 +13,7 @@
 #define _SMC_CORE_H
 
 #include <linux/atomic.h>
+#include <linux/types.h>
 #include <linux/smc.h>
 #include <linux/pci.h>
 #include <rdma/ib_verbs.h>
@@ -221,12 +222,16 @@ struct smc_buf_desc {
 					/* virtually contiguous */
 		};
 		struct { /* SMC-D */
+			 /* SMC-D rx buffer: */
 			unsigned short	sba_idx;
 					/* SBA index number */
 			u64		token;
 					/* DMB token number */
 			dma_addr_t	dma_addr;
 					/* DMA address */
+			/* SMC-D tx buffer */
+			bool		is_attached;
+					/* no need for explicit writes */
 		};
 	};
 };
net/smc/smc_ism.c +1 -0
index a94e1450d095..7363f8be9f94 100644
@@ -274,6 +274,7 @@ int smc_ism_attach_dmb(struct smcd_dev *dev, u64 token,
 		dmb_desc->cpu_addr = dmb.cpu_addr;
 		dmb_desc->dma_addr = dmb.dma_addr;
 		dmb_desc->len = dmb.dmb_len;
+		dmb_desc->is_attached = true;
 	}
 	return rc;
 }
net/smc/smc_loopback.c +3 -6
index 0eb00bbefd17..1853c26fbbbb 100644
@@ -224,12 +224,6 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok,
 	struct smc_lo_dev *ldev = smcd->priv;
 	struct smc_connection *conn;
 
-	if (!sf)
-		/* since sndbuf is merged with peer DMB, there is
-		 * no need to copy data from sndbuf to peer DMB.
-		 */
-		return 0;
-
 	read_lock_bh(&ldev->dmb_ht_lock);
 	hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_tok) {
 		if (tmp_node->token == dmb_tok) {
@@ -244,6 +238,9 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok,
 	memcpy((char *)rmb_node->cpu_addr + offset, data, size);
 	read_unlock_bh(&ldev->dmb_ht_lock);
 
+	if (!sf)
+		return 0;
+
 	conn = smcd->conn[rmb_node->sba_idx];
 	if (!conn || conn->killed)
 		return -EPIPE;
net/smc/smc_tx.c +3 -0
index 214ac3cbcf9a..3144b4b1fe29 100644
@@ -426,6 +426,9 @@ static int smcd_tx_rdma_writes(struct smc_connection *conn, size_t len,
 	int srcchunk, dstchunk;
 	int rc;
 
+	if (conn->sndbuf_desc->is_attached)
+		return 0;
+
 	for (dstchunk = 0; dstchunk < 2; dstchunk++) {
 		for (srcchunk = 0; srcchunk < 2; srcchunk++) {
 			void *data = conn->sndbuf_desc->cpu_addr + src_off;
Signature · 1 line
2.48.1
author maintainer
#41 lore raw

RFC 12/17 net/dibs: Create class dibs

In reply to Alexandra Winter

From: Julian Ruess <[email protected]>

Create '/sys/class/dibs' to represent multiple kinds of dibs devices in sysfs. Show s390/ism devices as well as dibs_loopback devices.

Show attribute fabric_id using dibs_ops.get_fabric_id(). This can help users understand which dibs devices are connected to the same fabric in different systems and which dibs devices are loopback devices (fabric_id 0xffff)

Instead of using the same name as the pci device, give the ism devices their own readable names based on uid or fid from the HW definition.

smc_loopback was never visible in sysfs. dibs_loopback is now represented as a virtual device.

Examples:
---------
ism before:

Quoted 1 line

ls /sys/bus/pci/devices/0000:00:00.0/0000:00:00.0

uevent

ism now:

Quoted 1 line

ls /sys/bus/pci/devices/0000:00:00.0/dibs/ism30

device -> ../../../0000:00:00.0/
fabric_id
subsystem -> ../../../../../class/dibs/
uevent

dibs loopback:

Quoted 1 line

ls /sys/devices/virtual/dibs/lo/

fabric_id
subsystem -> ../../../../class/dibs/
uevent

dibs class:

Quoted 1 line

ls -l /sys/class/dibs/

ism30 -> ../../devices/pci0000:00/0000:00:00.0/dibs/ism30/ lo -> ../../devices/virtual/dibs/lo/

For comparison:

Quoted 1 line

ls -l /sys/class/net/

enc8410 -> ../../devices/qeth/0.0.8410/net/enc8410/ ens1693 -> ../../devices/pci0001:00/0001:00:00.0/net/ens1693/ lo -> ../../devices/virtual/net/lo/

---
 drivers/s390/net/ism_drv.c |  5 ++++-
 net/dibs/dibs_main.c       | 40 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 1 deletion(-)
drivers/s390/net/ism_drv.c +4 -1
index 0ddfd47a3a7c..31f50cad4b39 100644
@@ -629,6 +629,7 @@ static void ism_dev_exit(struct ism_dev *ism)
 static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
 	struct dibs_dev *dibs;
+	struct zpci_dev *zdev;
 	struct ism_dev *ism;
 	int ret;
 
@@ -672,7 +673,9 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		goto err_dibs;
 
 	dibs->dev.parent = &pdev->dev;
-	dev_set_name(&dibs->dev, "%s", dev_name(&pdev->dev));
+
+	zdev = to_zpci(pdev);
+	dev_set_name(&dibs->dev, "ism%x", zdev->uid ? zdev->uid : zdev->fid);
 
 	ret = dibs_dev_add(dibs);
 	if (ret)
net/dibs/dibs_main.c +40 -0
index 8ffe1b4c90ef..e530c90a2934 100644
@@ -19,6 +19,8 @@
 MODULE_DESCRIPTION("Direct Internal Buffer Sharing class");
 MODULE_LICENSE("GPL");
 
+static struct class *dibs_class;
+
 /* use an array rather a list for fast mapping: */
 static struct dibs_client *clients[MAX_DIBS_CLIENTS];
 static u8 max_client;
@@ -104,12 +106,35 @@ struct dibs_dev *dibs_dev_alloc(void)
 	if (!dibs)
 		return dibs;
 	dibs->dev.release = dibs_dev_release;
+	dibs->dev.class = dibs_class;
 	device_initialize(&dibs->dev);
 
 	return dibs;
 }
 EXPORT_SYMBOL_GPL(dibs_dev_alloc);
 
+static ssize_t fabric_id_show(struct device *dev, struct device_attribute *attr,
+			      char *buf)
+{
+	struct dibs_dev *dibs;
+	u16 fabric_id;
+
+	dibs = container_of(dev, struct dibs_dev, dev);
+	fabric_id = dibs->ops->get_fabric_id(dibs);
+
+	return sysfs_emit(buf, "0x%04x\n", fabric_id);
+}
+static DEVICE_ATTR_RO(fabric_id);
+
+static struct attribute *dibs_dev_attrs[] = {
+	&dev_attr_fabric_id.attr,
+	NULL,
+};
+
+static const struct attribute_group dibs_dev_attr_group = {
+	.attrs = dibs_dev_attrs,
+};
+
 int dibs_dev_add(struct dibs_dev *dibs)
 {
 	int i, ret;
@@ -118,6 +143,11 @@ int dibs_dev_add(struct dibs_dev *dibs)
 	if (ret)
 		return ret;
 
+	ret = sysfs_create_group(&dibs->dev.kobj, &dibs_dev_attr_group);
+	if (ret) {
+		dev_err(&dibs->dev, "sysfs_create_group failed for dibs_dev\n");
+		goto err_device_del;
+	}
 	mutex_lock(&dibs_dev_list.mutex);
 	mutex_lock(&clients_lock);
 	for (i = 0; i < max_client; ++i) {
@@ -129,6 +159,11 @@ int dibs_dev_add(struct dibs_dev *dibs)
 	mutex_unlock(&dibs_dev_list.mutex);
 
 	return 0;
+
+err_device_del:
+	device_del(&dibs->dev);
+	return ret;
+
 }
 EXPORT_SYMBOL_GPL(dibs_dev_add);
 
@@ -157,6 +192,10 @@ static int __init dibs_init(void)
 	memset(clients, 0, sizeof(clients));
 	max_client = 0;
 
+	dibs_class = class_create("dibs");
+	if (IS_ERR(&dibs_class))
+		return PTR_ERR(&dibs_class);
+
 	rc = dibs_loopback_init();
 	if (rc)
 		pr_err("%s fails with %d\n", __func__, rc);
@@ -169,6 +208,7 @@ static int __init dibs_init(void)
 static void __exit dibs_exit(void)
 {
 	dibs_loopback_exit();
+	class_destroy(dibs_class);
 	pr_info("module unloaded\n");
 }
 
Signature · 1 line
2.48.1
author maintainer
#4 lore raw

RFC 2/17 s390/ism: Log module load/unload

In reply to Alexandra Winter

Add log messages to visualize timeline of module loads and unloads.

---
 drivers/s390/net/ism_drv.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/s390/net/ism_drv.c +6 -1
index 6cd60b174315..a543e59818bb 100644
@@ -718,8 +718,12 @@ static int __init ism_init(void)
 	debug_register_view(ism_debug_info, &debug_hex_ascii_view);
 	ret = pci_register_driver(&ism_driver);
 	if (ret)
-		debug_unregister(ism_debug_info);
+		goto err_dbg_unreg;
+	pr_info("module loaded\n");
+	return 0;
 
+err_dbg_unreg:
+	debug_unregister(ism_debug_info);
 	return ret;
 }
 
@@ -727,6 +731,7 @@ static void __exit ism_exit(void)
 {
 	pci_unregister_driver(&ism_driver);
 	debug_unregister(ism_debug_info);
+	pr_info("module unloaded\n");
 }
 
 module_init(ism_init);
Signature · 1 line
2.48.1
author maintainer
#13 lore raw

RFC 5/17 net/smc: Improve log message for devices w/o pnetid

In reply to Alexandra Winter

Explicitly state in the log message, when a device has no pnetid. "with pnetid" and "has pnetid" was misleading for devices without pnetid.

---
 net/smc/smc_ib.c  | 18 +++++++++++-------
 net/smc/smc_ism.c | 13 +++++++++----
 2 files changed, 20 insertions(+), 11 deletions(-)
net/smc/smc_ib.c +11 -7
index 53828833a3f7..f2de12990b5b 100644
@@ -971,13 +971,17 @@ static int smc_ib_add_dev(struct ib_device *ibdev)
 					   smcibdev->pnetid[i]))
 			smc_pnetid_by_table_ib(smcibdev, i + 1);
 		smc_copy_netdev_ifindex(smcibdev, i);
-		pr_warn_ratelimited("smc:    ib device %s port %d has pnetid "
-				    "%.16s%s\n",
-				    smcibdev->ibdev->name, i + 1,
-				    smcibdev->pnetid[i],
-				    smcibdev->pnetid_by_user[i] ?
-				     " (user defined)" :
-				     "");
+		if (smc_pnet_is_pnetid_set(smcibdev->pnetid[i]))
+			pr_warn_ratelimited("smc:    ib device %s port %d has pnetid %.16s%s\n",
+					    smcibdev->ibdev->name, i + 1,
+					    smcibdev->pnetid[i],
+					    smcibdev->pnetid_by_user[i] ?
+						" (user defined)" :
+						"");
+		else
+			pr_warn_ratelimited("smc:    ib device %s port %d has no pnetid\n",
+					    smcibdev->ibdev->name, i + 1);
+
 	}
 	schedule_work(&smcibdev->port_event_work);
 	return 0;
net/smc/smc_ism.c +9 -4
index 7363f8be9f94..503a9f93b392 100644
@@ -515,10 +515,15 @@ static void smcd_register_dev(struct ism_dev *ism)
 	}
 	mutex_unlock(&smcd_dev_list.mutex);
 
-	pr_warn_ratelimited("smc: adding smcd device %s with pnetid %.16s%s\n",
-			    dev_name(&ism->dev), smcd->pnetid,
-			    smcd->pnetid_by_user ? " (user defined)" : "");
-
+	if (smc_pnet_is_pnetid_set(smcd->pnetid))
+		pr_warn_ratelimited("smc: adding smcd device %s with pnetid %.16s%s\n",
+				    dev_name(&ism->dev), smcd->pnetid,
+				    smcd->pnetid_by_user ?
+					" (user defined)" :
+					"");
+	else
+		pr_warn_ratelimited("smc: adding smcd device %s without pnetid\n",
+				    dev_name(&ism->dev));
 	return;
 }
 
Signature · 1 line
2.48.1
author maintainer
#17 lore raw

RFC 7/17 net/dibs: Register smc as dibs_client

In reply to Alexandra Winter

Formally register smc as dibs client. Functionality will be moved by follow-on patches from ism_client to dibs_client until eventually ism_client can be removed.
As DIBS is only a shim layer without any dependencies, we can depend SMC on DIBS without adding indirect dependencies. A follow-on patch will remove dependency of SMC on ISM.

---
 include/linux/dibs.h | 24 +++++++++++++++++++++++-
 net/dibs/dibs_main.c | 35 +++++++++++++++++++++++++++++++++++
 net/smc/Kconfig      |  2 +-
 net/smc/smc_ism.c    |  6 ++++++
 4 files changed, 65 insertions(+), 2 deletions(-)
include/linux/dibs.h +23 -1
index 3f4175aaa732..5c432699becb 100644
@@ -33,10 +33,32 @@
  * clients.
  */
 
+/* DIBS client
+ * -----------
+ */
 #define MAX_DIBS_CLIENTS	8
-
 struct dibs_client {
+	/* client name for logging and debugging purposes */
 	const char *name;
+	/* client index - provided and used by dibs layer */
+	u8 id;
 };
 
+/* Functions to be called by dibs clients:
+ */
+/**
+ * dibs_register_client() - register a client with dibs layer
+ * @client: this client
+ *
+ * Return: zero on success.
+ */
+int dibs_register_client(struct dibs_client *client);
+/**
+ * dibs_unregister_client() - unregister a client with dibs layer
+ * @client: this client
+ *
+ * Return: zero on success.
+ */
+int dibs_unregister_client(struct dibs_client *client);
+
 #endif	/* _DIBS_H */
net/dibs/dibs_main.c +35 -0
index 348242fd05dd..d91e461f2f06 100644
@@ -20,6 +20,41 @@ MODULE_LICENSE("GPL");
 /* use an array rather a list for fast mapping: */
 static struct dibs_client *clients[MAX_DIBS_CLIENTS];
 static u8 max_client;
+static DEFINE_MUTEX(clients_lock);
+
+int dibs_register_client(struct dibs_client *client)
+{
+	int i, rc = -ENOSPC;
+
+	mutex_lock(&clients_lock);
+	for (i = 0; i < MAX_DIBS_CLIENTS; ++i) {
+		if (!clients[i]) {
+			clients[i] = client;
+			client->id = i;
+			if (i == max_client)
+				max_client++;
+			rc = 0;
+			break;
+		}
+	}
+	mutex_unlock(&clients_lock);
+
+	return rc;
+}
+EXPORT_SYMBOL_GPL(dibs_register_client);
+
+int dibs_unregister_client(struct dibs_client *client)
+{
+	int rc = 0;
+
+	mutex_lock(&clients_lock);
+	clients[client->id] = NULL;
+	if (client->id + 1 == max_client)
+		max_client--;
+	mutex_unlock(&clients_lock);
+	return rc;
+}
+EXPORT_SYMBOL_GPL(dibs_unregister_client);
 
 static int __init dibs_init(void)
 {
net/smc/Kconfig +1 -1
index ba5e6a2dd2fd..40dd60c1d23f 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 config SMC
 	tristate "SMC socket protocol family"
-	depends on INET && INFINIBAND
+	depends on INET && INFINIBAND && DIBS
 	depends on m || ISM != m
 	help
 	  SMC-R provides a "sockets over RDMA" solution making use of
net/smc/smc_ism.c +6 -0
index 503a9f93b392..a7a965e3c0ce 100644
@@ -18,6 +18,7 @@
 #include "smc_pnet.h"
 #include "smc_netlink.h"
 #include "linux/ism.h"
+#include "linux/dibs.h"
 
 struct smcd_dev_list smcd_dev_list = {
 	.list = LIST_HEAD_INIT(smcd_dev_list.list),
@@ -42,6 +43,9 @@ static struct ism_client smc_ism_client = {
 	.handle_irq = smcd_handle_irq,
 };
 #endif
+static struct dibs_client smc_dibs_client = {
+	.name = "SMC-D",
+};
 
 static void smc_ism_create_system_eid(void)
 {
@@ -623,11 +627,13 @@ int smc_ism_init(void)
 #if IS_ENABLED(CONFIG_ISM)
 	rc = ism_register_client(&smc_ism_client);
 #endif
+	rc = dibs_register_client(&smc_dibs_client);
 	return rc;
 }
 
 void smc_ism_exit(void)
 {
+	dibs_unregister_client(&smc_dibs_client);
 #if IS_ENABLED(CONFIG_ISM)
 	ism_unregister_client(&smc_ism_client);
 #endif
Signature · 1 line
2.48.1
author maintainer
#16 lore raw

RFC 6/17 net/dibs: Create net/dibs

In reply to Alexandra Winter

Create an 'DIBS' shim layer that will provide generic functionality and declarations for dibs device drivers and dibs clients.

Following patches will add functionality.

---
 MAINTAINERS          |  7 +++++++
 include/linux/dibs.h | 42 ++++++++++++++++++++++++++++++++++++++++++
 net/Kconfig          |  1 +
 net/Makefile         |  1 +
 net/dibs/Kconfig     | 14 ++++++++++++++
 net/dibs/Makefile    |  7 +++++++
 net/dibs/dibs_main.c | 39 +++++++++++++++++++++++++++++++++++++++
 7 files changed, 111 insertions(+)
 create mode 100644 include/linux/dibs.h
 create mode 100644 net/dibs/Kconfig
 create mode 100644 net/dibs/Makefile
 create mode 100644 net/dibs/dibs_main.c
MAINTAINERS +7 -0
index b968bc6959d1..e28d92f64236 100644
@@ -12948,6 +12948,13 @@ F:	Documentation/devicetree/bindings/hwmon/renesas,isl28022.yaml
 F:	Documentation/hwmon/isl28022.rst
 F:	drivers/hwmon/isl28022.c
 
+DIBS (DIRECT INTERNAL BUFFER SHARING)
+M:	Alexandra Winter <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	include/linux/dibs.h
+F:	net/dibs/
+
 ISOFS FILESYSTEM
 M:	Jan Kara <[email protected]>
 L:	[email protected]
include/linux/dibs.h new file +42 -0

42 lines not on this page · raw message

net/Kconfig +1 -0
index d5865cf19799..5410226f314b 100644
@@ -87,6 +87,7 @@ source "net/tls/Kconfig"
 source "net/xfrm/Kconfig"
 source "net/iucv/Kconfig"
 source "net/smc/Kconfig"
+source "net/dibs/Kconfig"
 source "net/xdp/Kconfig"
 
 config NET_HANDSHAKE
net/Makefile +1 -0
index aac960c41db6..3e4771cfe48e 100644
@@ -50,6 +50,7 @@ obj-$(CONFIG_TIPC)		+= tipc/
 obj-$(CONFIG_NETLABEL)		+= netlabel/
 obj-$(CONFIG_IUCV)		+= iucv/
 obj-$(CONFIG_SMC)		+= smc/
+obj-$(CONFIG_DIBS)		+= dibs/
 obj-$(CONFIG_RFKILL)		+= rfkill/
 obj-$(CONFIG_NET_9P)		+= 9p/
 obj-$(CONFIG_CAIF)		+= caif/
net/dibs/Kconfig new file +14 -0
new file mode 100644
index 000000000000..968c52938708
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0
+config DIBS
+	tristate "DIBS support"
+	default n
+	help
+	  Direct Internal Buffer Sharing (DIBS)
+	  A communication method that uses common physical (internal) memory
+	  for synchronous direct access into a remote buffer.
+
+	  Select this option to provide the abstraction layer between
+	  dibs devices and dibs clients like the SMC protocol.
+
+	  To compile as a module choose M. The module name is dibs.
+	  If unsure, choose N.
net/dibs/Makefile new file +7 -0
new file mode 100644
index 000000000000..825dec431bfc
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# DIBS class module
+#
+
+dibs-y += dibs_main.o
+obj-$(CONFIG_DIBS) += dibs.o
net/dibs/dibs_main.c new file +39 -0
new file mode 100644
index 000000000000..348242fd05dd
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ *  DIBS - Direct Internal Buffer Sharing
+ *
+ *  Implementation of the DIBS class module
+ *
+ *  Copyright IBM Corp. 2025
+ */
+#define KMSG_COMPONENT "dibs"
+#define pr_fmt(fmt) KMSG_COMPONENT ": " fmt
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <linux/err.h>
+#include <linux/dibs.h>
+
+MODULE_DESCRIPTION("Direct Internal Buffer Sharing class");
+MODULE_LICENSE("GPL");
+
+/* use an array rather a list for fast mapping: */
+static struct dibs_client *clients[MAX_DIBS_CLIENTS];
+static u8 max_client;
+
+static int __init dibs_init(void)
+{
+	memset(clients, 0, sizeof(clients));
+	max_client = 0;
+
+	pr_info("module loaded\n");
+	return 0;
+}
+
+static void __exit dibs_exit(void)
+{
+	pr_info("module unloaded\n");
+}
+
+module_init(dibs_init);
+module_exit(dibs_exit);
Signature · 1 line
2.48.1
author maintainer
#42 lore raw

RFC 13/17 net/dibs: Local gid for dibs devices

In reply to Alexandra Winter

Define a uuid_t GID attribute to identify a dibs device.

SMC uses 64 Bit and 128 Bit Global Identifiers (GIDs) per device, that need to be sent via the SMC protocol. Because the smc code uses integers, network endianness and host endianness need to be considered. Avoid this in the dibs layer by using uuid_t byte arrays. Future patches could change SMC to use uuid_t. For now conversion helper functions are introduced.

ISM devices provide 64 Bit GIDs. Map them to dibs uuid_t GIDs like this:
_________________________________________
| 64 Bit ISM-vPCI GID | 00000000_00000000 |
-----------------------------------------
If interpreted as UUID [1], this would be interpreted as the UIID variant, that is reserved for NCS backward compatibility. So it will not collide with UUIDs that were generated according to the standard.

smc_loopback already uses version 4 UUIDs as 128 Bit GIDs, move that to dibs loopback. A temporary change to smc_lo_query_rgid() is required, that will be moved to dibs_loopback with a follow-on patch.

Provide gid of a dibs device as sysfs read-only attribute.

---
 drivers/s390/net/ism.h     |  9 +++++++++
 drivers/s390/net/ism_drv.c | 30 +++++++++---------------------
 include/linux/dibs.h       |  3 +++
 include/linux/ism.h        |  1 -
 include/net/smc.h          |  1 -
 net/dibs/dibs_loopback.c   |  1 +
 net/dibs/dibs_main.c       | 12 ++++++++++++
 net/smc/smc_clc.c          |  6 +++---
 net/smc/smc_core.c         |  2 +-
 net/smc/smc_diag.c         |  2 +-
 net/smc/smc_ism.h          | 22 ++++++++++++++++++++++
 net/smc/smc_loopback.c     | 29 ++++-------------------------
 net/smc/smc_loopback.h     |  1 -
 13 files changed, 65 insertions(+), 54 deletions(-)
drivers/s390/net/ism.h +9 -0
index 3078779fa71e..1b9fa14da20c 100644
@@ -67,6 +67,15 @@ union ism_reg_ieq {
 	} response;
 } __aligned(16);
 
+/* ISM-vPCI devices provide 64 Bit GIDs
+ * Map them to ISM UUID GIDs like this:
+ *  _________________________________________
+ * | 64 Bit ISM-vPCI GID | 00000000_00000000 |
+ *  -----------------------------------------
+ * This will be interpreted as a UIID variant, that is reserved
+ * for NCS backward compatibility. So it will not collide with
+ * proper UUIDs.
+ */
 union ism_read_gid {
 	struct {
 		struct ism_req_hdr hdr;
drivers/s390/net/ism_drv.c +9 -21
index 31f50cad4b39..2dbf4ebb6bff 100644
@@ -272,8 +272,9 @@ static int unregister_ieq(struct ism_dev *ism)
 	return 0;
 }
 
-static int ism_read_local_gid(struct ism_dev *ism)
+static int ism_read_local_gid(struct dibs_dev *dibs)
 {
+	struct ism_dev *ism = dibs->drv_priv;
 	union ism_read_gid cmd;
 	int ret;
 
@@ -285,7 +286,8 @@ static int ism_read_local_gid(struct ism_dev *ism)
 	if (ret)
 		goto out;
 
-	ism->local_gid = cmd.response.gid;
+	memset(&dibs->gid, 0, sizeof(dibs->gid));
+	memcpy(&dibs->gid, &cmd.response.gid, sizeof(cmd.response.gid));
 out:
 	return ret;
 }
@@ -563,10 +565,6 @@ static int ism_dev_init(struct ism_dev *ism)
 	if (ret)
 		goto unreg_sba;
 
-	ret = ism_read_local_gid(ism);
-	if (ret)
-		goto unreg_ieq;
-
 	if (!ism_add_vlan_id(ism, ISM_RESERVED_VLANID))
 		/* hardware is V2 capable */
 		ism_v2_capable = true;
@@ -588,8 +586,6 @@ static int ism_dev_init(struct ism_dev *ism)
 	query_info(ism);
 	return 0;
 
-unreg_ieq:
-	unregister_ieq(ism);
 unreg_sba:
 	unregister_sba(ism);
 free_irq:
@@ -672,6 +668,11 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (ret)
 		goto err_dibs;
 
+	/* after ism_dev_init() we can call ism function to set gid */
+	ret = ism_read_local_gid(dibs);
+	if (ret)
+		goto err_ism;
+
 	dibs->dev.parent = &pdev->dev;
 
 	zdev = to_zpci(pdev);
@@ -846,18 +847,6 @@ static int smcd_supports_v2(void)
 	return ism_v2_capable;
 }
 
-static u64 ism_get_local_gid(struct ism_dev *ism)
-{
-	return ism->local_gid;
-}
-
-static void smcd_get_local_gid(struct smcd_dev *smcd,
-			       struct smcd_gid *smcd_gid)
-{
-	smcd_gid->gid = ism_get_local_gid(smcd->priv);
-	smcd_gid->gid_ext = 0;
-}
-
 static const struct smcd_ops ism_smcd_ops = {
 	.query_remote_gid = smcd_query_rgid,
 	.register_dmb = smcd_register_dmb,
@@ -869,7 +858,6 @@ static const struct smcd_ops ism_smcd_ops = {
 	.signal_event = smcd_signal_ieq,
 	.move_data = smcd_move,
 	.supports_v2 = smcd_supports_v2,
-	.get_local_gid = smcd_get_local_gid,
 };
 
 const struct smcd_ops *ism_get_smcd_ops(void)
include/linux/dibs.h +3 -0
index 4459b9369dc0..058dfc14aa5c 100644
@@ -10,6 +10,8 @@
 #define _DIBS_H
 
 #include <linux/device.h>
+#include <linux/uuid.h>
+
 /* DIBS - Direct Internal Buffer Sharing - concept
  * -----------------------------------------------
  * In the case of multiple system sharing the same hardware, dibs fabrics can
@@ -138,6 +140,7 @@ struct dibs_dev {
 	struct device dev;
 	/* To be filled by device driver, before calling dibs_dev_add(): */
 	const struct dibs_dev_ops *ops;
+	uuid_t gid;
 	/* priv pointer for device driver */
 	void *drv_priv;
 
include/linux/ism.h +0 -1
index 84f1afb3dded..a926dd61b5a1 100644
@@ -42,7 +42,6 @@ struct ism_dev {
 	struct ism_eq *ieq;
 	dma_addr_t ieq_dma_addr;
 
-	u64 local_gid;
 	int ieq_idx;
 
 	struct ism_client *subs[MAX_CLIENTS];
include/net/smc.h +0 -1
index 05faac83371e..9cb8385bbc6e 100644
@@ -62,7 +62,6 @@ struct smcd_ops {
 			 bool sf, unsigned int offset, void *data,
 			 unsigned int size);
 	int (*supports_v2)(void);
-	void (*get_local_gid)(struct smcd_dev *dev, struct smcd_gid *gid);
 
 	/* optional operations */
 	int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
net/dibs/dibs_loopback.c +1 -0
index bf02563527b4..d3800264fbba 100644
@@ -45,6 +45,7 @@ static int dibs_lo_dev_probe(void)
 
 	ldev->dibs = dibs;
 	dibs->drv_priv = ldev;
+	uuid_gen(&dibs->gid);
 	dibs->ops = &dibs_lo_ops;
 
 	dibs->dev.parent = NULL;
net/dibs/dibs_main.c +12 -0
index e530c90a2934..3e8b7e31b57c 100644
@@ -113,6 +113,17 @@ struct dibs_dev *dibs_dev_alloc(void)
 }
 EXPORT_SYMBOL_GPL(dibs_dev_alloc);
 
+static ssize_t gid_show(struct device *dev, struct device_attribute *attr,
+			char *buf)
+{
+	struct dibs_dev *dibs;
+
+	dibs = container_of(dev, struct dibs_dev, dev);
+
+	return sysfs_emit(buf, "%pUb\n", &dibs->gid);
+}
+static DEVICE_ATTR_RO(gid);
+
 static ssize_t fabric_id_show(struct device *dev, struct device_attribute *attr,
 			      char *buf)
 {
@@ -127,6 +138,7 @@ static ssize_t fabric_id_show(struct device *dev, struct device_attribute *attr,
 static DEVICE_ATTR_RO(fabric_id);
 
 static struct attribute *dibs_dev_attrs[] = {
+	&dev_attr_gid.attr,
 	&dev_attr_fabric_id.attr,
 	NULL,
 };
net/smc/smc_clc.c +3 -3
index 5a4db151fe95..e44cbfd2619d 100644
@@ -915,7 +915,7 @@ int smc_clc_send_proposal(struct smc_sock *smc, struct smc_init_info *ini)
 		/* add SMC-D specifics */
 		if (ini->ism_dev[0]) {
 			smcd = ini->ism_dev[0];
-			smcd->ops->get_local_gid(smcd, &smcd_gid);
+			copy_to_smcdgid(&smcd_gid, &smcd->dibs->gid);
 			pclc_smcd->ism.gid = htonll(smcd_gid.gid);
 			pclc_smcd->ism.chid =
 				htons(smc_ism_get_chid(ini->ism_dev[0]));
@@ -965,7 +965,7 @@ int smc_clc_send_proposal(struct smc_sock *smc, struct smc_init_info *ini)
 		if (ini->ism_offered_cnt) {
 			for (i = 1; i <= ini->ism_offered_cnt; i++) {
 				smcd = ini->ism_dev[i];
-				smcd->ops->get_local_gid(smcd, &smcd_gid);
+				copy_to_smcdgid(&smcd_gid, &smcd->dibs->gid);
 				gidchids[entry].chid =
 					htons(smc_ism_get_chid(ini->ism_dev[i]));
 				gidchids[entry].gid = htonll(smcd_gid.gid);
@@ -1058,7 +1058,7 @@ smcd_clc_prep_confirm_accept(struct smc_connection *conn,
 	/* SMC-D specific settings */
 	memcpy(clc->hdr.eyecatcher, SMCD_EYECATCHER,
 	       sizeof(SMCD_EYECATCHER));
-	smcd->ops->get_local_gid(smcd, &smcd_gid);
+	copy_to_smcdgid(&smcd_gid, &smcd->dibs->gid);
 	clc->hdr.typev1 = SMC_TYPE_D;
 	clc->d0.gid = htonll(smcd_gid.gid);
 	clc->d0.token = htonll(conn->rmb_desc->token);
net/smc/smc_core.c +1 -1
index 71c410dc3658..405cb0003241 100644
@@ -555,7 +555,7 @@ static int smc_nl_fill_smcd_lgr(struct smc_link_group *lgr,
 
 	if (nla_put_u32(skb, SMC_NLA_LGR_D_ID, *((u32 *)&lgr->id)))
 		goto errattr;
-	smcd->ops->get_local_gid(smcd, &smcd_gid);
+	copy_to_smcdgid(&smcd_gid, &smcd->dibs->gid);
 	if (nla_put_u64_64bit(skb, SMC_NLA_LGR_D_GID,
 			      smcd_gid.gid, SMC_NLA_LGR_D_PAD))
 		goto errattr;
net/smc/smc_diag.c +1 -1
index 8ed2f6689b01..bf0beaa23bdb 100644
@@ -175,7 +175,7 @@ static int __smc_diag_dump(struct sock *sk, struct sk_buff *skb,
 		dinfo.linkid = *((u32 *)conn->lgr->id);
 		dinfo.peer_gid = conn->lgr->peer_gid.gid;
 		dinfo.peer_gid_ext = conn->lgr->peer_gid.gid_ext;
-		smcd->ops->get_local_gid(smcd, &smcd_gid);
+		copy_to_smcdgid(&smcd_gid, &smcd->dibs->gid);
 		dinfo.my_gid = smcd_gid.gid;
 		dinfo.my_gid_ext = smcd_gid.gid_ext;
 		dinfo.token = conn->rmb_desc->token;
net/smc/smc_ism.h +22 -0
index 04699951d03f..139e99da2c9f 100644
@@ -96,4 +96,26 @@ static inline bool smc_ism_is_loopback(struct dibs_dev *dibs)
 	return (dibs->ops->get_fabric_id(dibs) == DIBS_LOOPBACK_FABRIC);
 }
 
+static inline void copy_to_smcdgid(struct smcd_gid *sgid, uuid_t *dibs_gid)
+{
+	__be64 temp;
+
+	memcpy(&temp, dibs_gid, sizeof(sgid->gid));
+	sgid->gid = ntohll(temp);
+	memcpy(&temp, (uint8_t *)dibs_gid + sizeof(sgid->gid),
+	       sizeof(sgid->gid_ext));
+	sgid->gid_ext = ntohll(temp);
+}
+
+static inline void copy_to_dibsgid(uuid_t *dibs_gid, struct smcd_gid *sgid)
+{
+	__be64 temp;
+
+	temp = htonll(sgid->gid);
+	memcpy(dibs_gid, &temp, sizeof(sgid->gid));
+	temp = htonll(sgid->gid_ext);
+	memcpy((uint8_t *)dibs_gid + sizeof(sgid->gid), &temp,
+	       sizeof(sgid->gid_ext));
+}
+
 #endif
net/smc/smc_loopback.c +4 -25
index 262d0d0df4d0..454d9d6a6e8f 100644
@@ -13,6 +13,7 @@
 
 #include <linux/device.h>
 #include <linux/types.h>
+#include <linux/dibs.h>
 #include <net/smc.h>
 
 #include "smc_cdc.h"
@@ -25,25 +26,14 @@
 
 static struct smc_lo_dev *lo_dev;
 
-static void smc_lo_generate_ids(struct smc_lo_dev *ldev)
-{
-	struct smcd_gid *lgid = &ldev->local_gid;
-	uuid_t uuid;
-
-	uuid_gen(&uuid);
-	memcpy(&lgid->gid, &uuid, sizeof(lgid->gid));
-	memcpy(&lgid->gid_ext, (u8 *)&uuid + sizeof(lgid->gid),
-	       sizeof(lgid->gid_ext));
-}
-
 static int smc_lo_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid,
 			     u32 vid_valid, u32 vid)
 {
-	struct smc_lo_dev *ldev = smcd->priv;
+	uuid_t temp;
 
+	copy_to_dibsgid(&temp, rgid);
 	/* rgid should be the same as lgid */
-	if (!ldev || rgid->gid != ldev->local_gid.gid ||
-	    rgid->gid_ext != ldev->local_gid.gid_ext)
+	if (!uuid_equal(&temp, &smcd->dibs->gid))
 		return -ENETUNREACH;
 	return 0;
 }
@@ -245,15 +235,6 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok,
 	return 0;
 }
 
-static void smc_lo_get_local_gid(struct smcd_dev *smcd,
-				 struct smcd_gid *smcd_gid)
-{
-	struct smc_lo_dev *ldev = smcd->priv;
-
-	smcd_gid->gid = ldev->local_gid.gid;
-	smcd_gid->gid_ext = ldev->local_gid.gid_ext;
-}
-
 static const struct smcd_ops lo_ops = {
 	.query_remote_gid = smc_lo_query_rgid,
 	.register_dmb = smc_lo_register_dmb,
@@ -267,7 +248,6 @@ static const struct smcd_ops lo_ops = {
 	.reset_vlan_required	= NULL,
 	.signal_event		= NULL,
 	.move_data = smc_lo_move_data,
-	.get_local_gid = smc_lo_get_local_gid,
 };
 
 const struct smcd_ops *smc_lo_get_smcd_ops(void)
@@ -277,7 +257,6 @@ const struct smcd_ops *smc_lo_get_smcd_ops(void)
 
 static void smc_lo_dev_init(struct smc_lo_dev *ldev)
 {
-	smc_lo_generate_ids(ldev);
 	rwlock_init(&ldev->dmb_ht_lock);
 	hash_init(ldev->dmb_ht);
 	atomic_set(&ldev->dmb_cnt, 0);
net/smc/smc_loopback.h +0 -1
index a033bf10890a..33bb96ec8b77 100644
@@ -32,7 +32,6 @@ struct smc_lo_dmb_node {
 
 struct smc_lo_dev {
 	struct smcd_dev *smcd;
-	struct smcd_gid local_gid;
 	atomic_t dmb_cnt;
 	rwlock_t dmb_ht_lock;
 	DECLARE_BITMAP(sba_idx_mask, SMC_LO_MAX_DMBS);
Signature · 1 line
2.48.1
author maintainer
#44 lore raw

RFC 15/17 net/dibs: Move query_remote_gid() to dibs_dev_ops

In reply to Alexandra Winter

Provide the dibs_dev_ops->query_remote_gid() in ism and dibs_loopback dibs_devices. And call it in smc dibs_client.

---
 drivers/s390/net/ism_drv.c | 41 +++++++++++++++++---------------------
 include/linux/dibs.h       | 14 +++++++++++++
 include/net/smc.h          |  2 --
 net/dibs/dibs_loopback.c   | 10 ++++++++++
 net/smc/smc_ism.c          |  8 ++++++--
 net/smc/smc_loopback.c     | 13 ------------
 6 files changed, 48 insertions(+), 40 deletions(-)
drivers/s390/net/ism_drv.c +18 -23
index 636594b2fab2..6a30178b1b06 100644
@@ -291,6 +291,23 @@ static int ism_read_local_gid(struct dibs_dev *dibs)
 	return ret;
 }
 
+static int ism_query_rgid(struct dibs_dev *dibs, uuid_t *rgid, u32 vid_valid,
+			  u32 vid)
+{
+	struct ism_dev *ism = dibs->drv_priv;
+	union ism_query_rgid cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.request.hdr.cmd = ISM_QUERY_RGID;
+	cmd.request.hdr.len = sizeof(cmd.request);
+
+	memcpy(&cmd.request.rgid, rgid, sizeof(cmd.request.rgid));
+	cmd.request.vlan_valid = vid_valid;
+	cmd.request.vlan_id = vid;
+
+	return ism_cmd(ism, &cmd);
+}
+
 static void ism_free_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
 {
 	clear_bit(dmb->sba_idx, ism->sba_bitmap);
@@ -537,6 +554,7 @@ static irqreturn_t ism_handle_irq(int irq, void *data)
 
 static const struct dibs_dev_ops ism_ops = {
 	.get_fabric_id = ism_get_chid,
+	.query_remote_gid = ism_query_rgid,
 	.add_vlan_id = ism_add_vlan_id,
 	.del_vlan_id = ism_del_vlan_id,
 };
@@ -753,28 +771,6 @@ module_exit(ism_exit);
 /*************************** SMC-D Implementation *****************************/
 
 #if IS_ENABLED(CONFIG_SMC)
-static int ism_query_rgid(struct ism_dev *ism, u64 rgid, u32 vid_valid,
-			  u32 vid)
-{
-	union ism_query_rgid cmd;
-
-	memset(&cmd, 0, sizeof(cmd));
-	cmd.request.hdr.cmd = ISM_QUERY_RGID;
-	cmd.request.hdr.len = sizeof(cmd.request);
-
-	cmd.request.rgid = rgid;
-	cmd.request.vlan_valid = vid_valid;
-	cmd.request.vlan_id = vid;
-
-	return ism_cmd(ism, &cmd);
-}
-
-static int smcd_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid,
-			   u32 vid_valid, u32 vid)
-{
-	return ism_query_rgid(smcd->priv, rgid->gid, vid_valid, vid);
-}
-
 static int smcd_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
 			     void *client)
 {
@@ -818,7 +814,6 @@ static int smcd_move(struct smcd_dev *smcd, u64 dmb_tok, unsigned int idx,
 }
 
 static const struct smcd_ops ism_smcd_ops = {
-	.query_remote_gid = smcd_query_rgid,
 	.register_dmb = smcd_register_dmb,
 	.unregister_dmb = smcd_unregister_dmb,
 	.signal_event = smcd_signal_ieq,
include/linux/dibs.h +14 -0
index 10be10ae4660..d940411aa179 100644
@@ -133,6 +133,20 @@ struct dibs_dev_ops {
 	 * Return: 2 byte dibs fabric id
 	 */
 	u16 (*get_fabric_id)(struct dibs_dev *dev);
+	/**
+	 * query_remote_gid()
+	 * @dev: local dibs device
+	 * @rgid: gid of remote dibs device
+	 * @vid_valid: if zero, vid will be ignored;
+	 *	       deprecated, ignored if device does not support vlan
+	 * @vid: VLAN id; deprecated, ignored if device does not support vlan
+	 *
+	 * Query whether a remote dibs device is reachable via this local device
+	 * and this vlan id.
+	 * Return: 0 if remote gid is reachable.
+	 */
+	int (*query_remote_gid)(struct dibs_dev *dev, uuid_t *rgid,
+				u32 vid_valid, u32 vid);
 	/**
 	 * add_vlan_id() - add dibs device to vlan (optional, deprecated)
 	 * @dev: dibs device
include/net/smc.h +0 -2
index 51b4aefc106a..5bd135fb4d49 100644
@@ -53,8 +53,6 @@ struct smcd_gid {
 };
 
 struct smcd_ops {
-	int (*query_remote_gid)(struct smcd_dev *dev, struct smcd_gid *rgid,
-				u32 vid_valid, u32 vid);
 	int (*register_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb,
 			    void *client);
 	int (*unregister_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb);
net/dibs/dibs_loopback.c +10 -0
index d3800264fbba..b56579d0dfe7 100644
@@ -23,8 +23,18 @@ static u16 dibs_lo_get_fabric_id(struct dibs_dev *dibs)
 	return DIBS_LOOPBACK_FABRIC;
 }
 
+static int dibs_lo_query_rgid(struct dibs_dev *dibs, uuid_t *rgid,
+			      u32 vid_valid, u32 vid)
+{
+	/* rgid should be the same as lgid */
+	if (!uuid_equal(rgid, &dibs->gid))
+		return -ENETUNREACH;
+	return 0;
+}
+
 static const struct dibs_dev_ops dibs_lo_ops = {
 	.get_fabric_id = dibs_lo_get_fabric_id,
+	.query_remote_gid = dibs_lo_query_rgid,
 };
 
 static int dibs_lo_dev_probe(void)
net/smc/smc_ism.c +6 -2
index ce9d2195e14a..265c8b383b71 100644
@@ -77,8 +77,12 @@ static void smc_ism_create_system_eid(void)
 int smc_ism_cantalk(struct smcd_gid *peer_gid, unsigned short vlan_id,
 		    struct smcd_dev *smcd)
 {
-	return smcd->ops->query_remote_gid(smcd, peer_gid, vlan_id ? 1 : 0,
-					   vlan_id);
+	struct dibs_dev *dibs = smcd->dibs;
+	uuid_t ism_rgid;
+
+	copy_to_dibsgid(&ism_rgid, peer_gid);
+	return dibs->ops->query_remote_gid(dibs, &ism_rgid, vlan_id ? 1 : 0,
+					  vlan_id);
 }
 
 void smc_ism_get_system_eid(u8 **eid)
net/smc/smc_loopback.c +0 -13
index 982a19430313..52cba01cb209 100644
@@ -25,18 +25,6 @@
 
 static struct smc_lo_dev *lo_dev;
 
-static int smc_lo_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid,
-			     u32 vid_valid, u32 vid)
-{
-	uuid_t temp;
-
-	copy_to_dibsgid(&temp, rgid);
-	/* rgid should be the same as lgid */
-	if (!uuid_equal(&temp, &smcd->dibs->gid))
-		return -ENETUNREACH;
-	return 0;
-}
-
 static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
 			       void *client_priv)
 {
@@ -235,7 +223,6 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok,
 }
 
 static const struct smcd_ops lo_ops = {
-	.query_remote_gid = smc_lo_query_rgid,
 	.register_dmb = smc_lo_register_dmb,
 	.unregister_dmb = smc_lo_unregister_dmb,
 	.support_dmb_nocopy = smc_lo_support_dmb_nocopy,
Signature · 1 line
2.48.1
author maintainer
#51 lore raw

RFC 17/17 net/dibs: Move event handling to dibs layer

In reply to Alexandra Winter

From: Julian Ruess <[email protected]>

Add defines for all event types and subtypes an ism device is known to produce as it can be helpful for debugging purposes.

Introduces a generic 'struct dibs_event' and adopt ism device driver and smc-d client accordingly. Tolerate and ignore other type and subtype values to enable future device extensions.

SMC-D and ISM are now independent.
struct ism_dev can be moved to drivers/s390/net/ism.h.

Note that in smc, the term 'ism' is still used. Future patches could replace that with 'dibs' or 'smc-d' as appropriate.

---
 MAINTAINERS                |   2 -
 drivers/s390/net/Kconfig   |   1 -
 drivers/s390/net/ism.h     |  42 ++++++-
 drivers/s390/net/ism_drv.c | 221 ++++++++++++-------------------------
 include/linux/dibs.h       |  62 +++++++++++
 include/linux/ism.h        |  67 -----------
 include/net/smc.h          |  15 ---
 net/dibs/dibs_main.c       |   2 +-
 net/smc/Kconfig            |   1 -
 net/smc/smc_ism.c          |  94 ++++++----------
 10 files changed, 206 insertions(+), 301 deletions(-)
 delete mode 100644 include/linux/ism.h
MAINTAINERS +0 -2
index e28d92f64236..02d7374913fa 100644
@@ -17410,7 +17410,6 @@ F:	include/linux/fddidevice.h
 F:	include/linux/hippidevice.h
 F:	include/linux/if_*
 F:	include/linux/inetdevice.h
-F:	include/linux/ism.h
 F:	include/linux/netdev*
 F:	include/linux/platform_data/wiznet.h
 F:	include/uapi/linux/cn_proc.h
@@ -22010,7 +22009,6 @@ L:	[email protected]
 L:	[email protected]
 S:	Supported
 F:	drivers/s390/net/
-F:	include/linux/ism.h
 
 S390 PCI SUBSYSTEM
 M:	Niklas Schnelle <[email protected]>
drivers/s390/net/Kconfig +0 -1
index 92985f595d59..0fd700c5745a 100644
@@ -82,7 +82,6 @@ config CCWGROUP
 config ISM
 	tristate "Support for ISM vPCI Adapter"
 	depends on PCI && DIBS
-	imply SMC
 	default n
 	help
 	  Select this option if you want to use the Internal Shared Memory
drivers/s390/net/ism.h +40 -2

71 lines not on this page · raw message

drivers/s390/net/ism_drv.c +71 -150

293 lines not on this page · raw message

include/linux/dibs.h +62 -0

80 lines not on this page · raw message

include/linux/ism.h deleted +0 -67

67 lines not on this page · raw message

include/net/smc.h +0 -15
index 8e3debcf7db5..08bee529ed8d 100644
@@ -16,7 +16,6 @@
 #include <linux/types.h>
 #include <linux/wait.h>
 #include <linux/dibs.h>
-#include "linux/ism.h"
 
 struct sock;
 
@@ -28,28 +27,14 @@ struct smc_hashinfo {
 };
 
 /* SMCD/ISM device driver interface */
-#define ISM_EVENT_DMB	0
-#define ISM_EVENT_GID	1
-#define ISM_EVENT_SWR	2
-
 #define ISM_RESERVED_VLANID	0x1FFF
 
-struct smcd_dev;
-
 struct smcd_gid {
 	u64	gid;
 	u64	gid_ext;
 };
 
-struct smcd_ops {
-	/* optional operations */
-	int (*signal_event)(struct smcd_dev *dev, struct smcd_gid *rgid,
-			    u32 trigger_irq, u32 event_code, u64 info);
-};
-
 struct smcd_dev {
-	const struct smcd_ops *ops;
-	void *priv;
 	struct dibs_dev *dibs;
 	struct list_head list;
 	spinlock_t lock;
net/dibs/dibs_main.c +1 -1
index dc7f16fb15b6..1d7c6095f4ff 100644
@@ -97,7 +97,7 @@ int dibs_unregister_client(struct dibs_client *client)
 				goto err_reg_dmb;
 			}
 		}
-		/* Stop forwarding IRQs */
+		/* Stop forwarding IRQs and events */
 		dibs->subs[client->id] = NULL;
 		spin_unlock_irqrestore(&dibs->lock, flags);
 		clients[client->id]->ops->del_dev(dibs);
net/smc/Kconfig +0 -1
index 9535d88c2acb..99ecd59d1f4b 100644
@@ -2,7 +2,6 @@
 config SMC
 	tristate "SMC socket protocol family"
 	depends on INET && INFINIBAND && DIBS
-	depends on m || ISM != m
 	help
 	  SMC-R provides a "sockets over RDMA" solution making use of
 	  RDMA over Converged Ethernet (RoCE) technology to upgrade
net/smc/smc_ism.c +32 -62

219 lines not on this page · raw message

Signature · 1 line
2.48.1
author maintainer
#27 lore raw

RFC 9/17 net/dibs: Define dibs loopback

In reply to Alexandra Winter

The first stage of loopback-ism was implemented as part of the SMC module [1]. Now that we have the dibs layer, provide access to a dibs_loopback device to all dibs clients.

This is the first step of moving loopback-ism from net/smc/smc_loopback.* to net/dibs/dibs_loopback.*. One global structure lo_dev is allocated and added to the dibs devices. Follow-on patches will move functionality.

Same as smc_loopback, dibs_loopback is provided by a config option. Note that there is no way to dynamically add or remove the loopback device. That could be a future improvement.

When moving code to net/dibs, replace ism_ prefix with dibs_ prefix. As this is mostly a move of existing code, copyright and authors are unchanged.

---
 net/dibs/Kconfig         | 13 +++++++
 net/dibs/Makefile        |  1 +
 net/dibs/dibs_loopback.c | 77 ++++++++++++++++++++++++++++++++++++++++
 net/dibs/dibs_loopback.h | 38 ++++++++++++++++++++
 net/dibs/dibs_main.c     | 14 ++++++--
 5 files changed, 141 insertions(+), 2 deletions(-)
 create mode 100644 net/dibs/dibs_loopback.c
 create mode 100644 net/dibs/dibs_loopback.h
net/dibs/Kconfig +13 -0
index 968c52938708..9a6182391b25 100644
@@ -12,3 +12,16 @@ config DIBS
 
 	  To compile as a module choose M. The module name is dibs.
 	  If unsure, choose N.
+
+config DIBS_LO
+	bool "intra-OS shortcut with dibs loopback"
+	depends on DIBS
+	default n
+	help
+	  DIBS_LO enables the creation of an software-emulated dibs device
+	  named lo which can be used for transferring data when communication
+	  occurs within the same OS. This helps in convenient testing of
+	  dibs clients, since dibs loopback is independent of architecture or
+	  hardware.
+
+	  if unsure, say N.
net/dibs/Makefile +1 -0
index 825dec431bfc..85805490c77f 100644
@@ -5,3 +5,4 @@
 
 dibs-y += dibs_main.o
 obj-$(CONFIG_DIBS) += dibs.o
+dibs-$(CONFIG_DIBS_LO) += dibs_loopback.o
\ No newline at end of file
net/dibs/dibs_loopback.c new file +77 -0

77 lines not on this page · raw message

net/dibs/dibs_loopback.h new file +38 -0
new file mode 100644
index 000000000000..fd03b6333a24
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ *  dibs loopback (aka loopback-ism) device structure definitions.
+ *
+ *  Copyright (c) 2024, Alibaba Inc.
+ *
+ *  Author: Wen Gu <[email protected]>
+ *          Tony Lu <[email protected]>
+ *
+ */
+
+#ifndef _DIBS_LOOPBACK_H
+#define _DIBS_LOOPBACK_H
+
+#include <linux/dibs.h>
+#include <linux/types.h>
+#include <linux/wait.h>
+
+#if IS_ENABLED(CONFIG_DIBS_LO)
+
+struct dibs_lo_dev {
+	struct dibs_dev *dibs;
+};
+
+int dibs_loopback_init(void);
+void dibs_loopback_exit(void);
+#else
+static inline int dibs_loopback_init(void)
+{
+	return 0;
+}
+
+static inline void dibs_loopback_exit(void)
+{
+}
+#endif
+
+#endif /* _DIBS_LOOPBACK_H */
net/dibs/dibs_main.c +12 -2
index 5345e41ae5e4..2c213e1f8f93 100644
@@ -14,6 +14,8 @@
 #include <linux/err.h>
 #include <linux/dibs.h>
 
+#include "dibs_loopback.h"
+
 MODULE_DESCRIPTION("Direct Internal Buffer Sharing class");
 MODULE_LICENSE("GPL");
 
@@ -94,15 +96,23 @@ EXPORT_SYMBOL_GPL(dibs_dev_del);
 
 static int __init dibs_init(void)
 {
+	int rc;
+
 	memset(clients, 0, sizeof(clients));
 	max_client = 0;
 
-	pr_info("module loaded\n");
-	return 0;
+	rc = dibs_loopback_init();
+	if (rc)
+		pr_err("%s fails with %d\n", __func__, rc);
+	else
+		pr_info("module loaded\n");
+
+	return rc;
 }
 
 static void __exit dibs_exit(void)
 {
+	dibs_loopback_exit();
 	pr_info("module unloaded\n");
 }
 
Signature · 1 line
2.48.1
author maintainer
#28 lore raw

RFC 10/17 net/dibs: Define dibs_client_ops and dibs_dev_ops

In reply to Alexandra Winter

Move the device add() and remove() functions from ism_client to dibs_client_ops and call add_dev()/del_dev() for ism devices and dibs_loopback devices. dibs_client_ops->add_dev() = smcd_register_dev() for the smc_dibs_client. This is the first step to handle ism and loopback devices alike (as dibs devices) in the smc dibs client.

Define dibs_dev->ops and move smcd_ops->get_chid to dibs_dev_ops->get_fabric_id() for ism and loopback devices. See below for why this needs to be in the same patch as dibs_client_ops->add_dev().

The following changes contain intermediate steps, that will be obsoleted by follow-on patches, once more functionality has been moved to dibs:

Use different smcd_ops and max_dmbs for ism and loopback. Follow-on patches will change SMC-D to directly use dibs_ops instead of smcd_ops.

In smcd_register_dev() it is now necessary to identify a dibs_loopback device before smcd_dev and smcd_ops->get_chid() are available. So provide dibs_dev_ops->get_fabric_id() in this patch and evaluate it in smc_ism_is_loopback().

Call smc_loopback_init() in smcd_register_dev() and call smc_loopback_exit() in smcd_unregister_dev() to handle the functionality that is still in smc_loopback. Follow-on patches will move all smc_loopback code to dibs_loopback.

In smcd_unregister_dev() use only ism device name, this will be replaced by dibs device name by a follow-on patch.

End of changes with intermediate parts.

Allocate an smcd event workqueue for all dibs devices, although dibs_loopback does not generate events.

Use kernel memory instead of devres memory for smcd_dev and smcd->conn. Since commit a72178cfe855 ("net/smc: Fix dependency of SMC on ISM") an ism device and its driver can have a longer lifetime than the smc module, so smc should not rely on devres to free its resources [1]. It is now the responsibility of the smc client to free smcd and smcd->conn for all dibs devices, ism devices as well as loopback. Call client->ops->del_dev() for all existing dibs devices in dibs_unregister_client(), so all device related structures can be freed in the client.

When dibs_unregister_client() is called in the context of smc_exit() or smc_core_reboot_event(), these functions have already called smc_lgrs_shutdown() which calls smc_smcd_terminate_all(smcd) and sets going_away. This is done a second time in smcd_unregister_dev(). This is analogous to how smcr is handled in these functions, by calling first smc_lgrs_shutdown() and then smc_ib_unregister_client() > smc_ib_remove_dev(), so leave it that way. It may be worth investigating, whether smc_lgrs_shutdown() is still required or useful.

Remove CONFIG_SMC_LO. CONFIG_DIBS_LO now controls whether a dibs loopback device exists or not.

---
 drivers/s390/net/ism_drv.c |  43 +++++++---------
 include/linux/dibs.h       |  90 +++++++++++++++++++++++++++++++-
 include/linux/ism.h        |   2 -
 include/net/smc.h          |   3 +-
 net/dibs/dibs_loopback.c   |  11 ++++
 net/dibs/dibs_main.c       |  37 +++++++++++++
 net/smc/Kconfig            |  13 -----
 net/smc/Makefile           |   2 +-
 net/smc/af_smc.c           |  12 +----
 net/smc/smc_ism.c          | 103 ++++++++++++++++++++++++++-----------
 net/smc/smc_ism.h          |   7 +--
 net/smc/smc_loopback.c     |  94 +++++----------------------------
 net/smc/smc_loopback.h     |  17 ++----
 13 files changed, 254 insertions(+), 180 deletions(-)
drivers/s390/net/ism_drv.c +19 -24
index 7948334f8d30..84a6e9ae2e64 100644
@@ -79,7 +79,6 @@ int ism_register_client(struct ism_client *client)
 		/* initialize with all devices that we got so far */
 		list_for_each_entry(ism, &ism_dev_list.list, list) {
 			ism->priv[i] = NULL;
-			client->add(ism);
 			ism_setup_forwarding(client, ism);
 		}
 	}
@@ -465,6 +464,16 @@ int ism_move(struct ism_dev *ism, u64 dmb_tok, unsigned int idx, bool sf,
 }
 EXPORT_SYMBOL_GPL(ism_move);
 
+static u16 ism_get_chid(struct dibs_dev *dibs)
+{
+	struct ism_dev *ism = dibs->drv_priv;
+
+	if (!ism || !ism->pdev)
+		return 0;
+
+	return to_zpci(ism->pdev)->pchid;
+}
+
 static void ism_handle_event(struct ism_dev *ism)
 {
 	struct ism_event *entry;
@@ -523,6 +532,10 @@ static irqreturn_t ism_handle_irq(int irq, void *data)
 	return IRQ_HANDLED;
 }
 
+static const struct dibs_dev_ops ism_ops = {
+	.get_fabric_id = ism_get_chid,
+};
+
 static int ism_dev_init(struct ism_dev *ism)
 {
 	struct pci_dev *pdev = ism->pdev;
@@ -564,7 +577,6 @@ static int ism_dev_init(struct ism_dev *ism)
 	mutex_lock(&clients_lock);
 	for (i = 0; i < max_client; ++i) {
 		if (clients[i]) {
-			clients[i]->add(ism);
 			ism_setup_forwarding(clients[i], ism);
 		}
 	}
@@ -611,12 +623,6 @@ static void ism_dev_exit(struct ism_dev *ism)
 	spin_unlock_irqrestore(&ism->lock, flags);
 
 	mutex_lock(&ism_dev_list.mutex);
-	mutex_lock(&clients_lock);
-	for (i = 0; i < max_client; ++i) {
-		if (clients[i])
-			clients[i]->remove(ism);
-	}
-	mutex_unlock(&clients_lock);
 
 	if (ism_v2_capable)
 		ism_del_vlan_id(ism, ISM_RESERVED_VLANID);
@@ -672,7 +678,10 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		ret = -ENOMEM;
 		goto err_resource;
 	}
+	/* set this up before we enable interrupts */
 	ism->dibs = dibs;
+	dibs->drv_priv = ism;
+	dibs->ops = &ism_ops;
 
 	ret = ism_dev_init(ism);
 	if (ret)
@@ -862,19 +871,6 @@ static void smcd_get_local_gid(struct smcd_dev *smcd,
 	smcd_gid->gid_ext = 0;
 }
 
-static u16 ism_get_chid(struct ism_dev *ism)
-{
-	if (!ism || !ism->pdev)
-		return 0;
-
-	return to_zpci(ism->pdev)->pchid;
-}
-
-static u16 smcd_get_chid(struct smcd_dev *smcd)
-{
-	return ism_get_chid(smcd->priv);
-}
-
 static inline struct device *smcd_get_dev(struct smcd_dev *dev)
 {
 	struct ism_dev *ism = dev->priv;
@@ -882,7 +878,7 @@ static inline struct device *smcd_get_dev(struct smcd_dev *dev)
 	return &ism->dev;
 }
 
-static const struct smcd_ops ism_ops = {
+static const struct smcd_ops ism_smcd_ops = {
 	.query_remote_gid = smcd_query_rgid,
 	.register_dmb = smcd_register_dmb,
 	.unregister_dmb = smcd_unregister_dmb,
@@ -894,13 +890,12 @@ static const struct smcd_ops ism_ops = {
 	.move_data = smcd_move,
 	.supports_v2 = smcd_supports_v2,
 	.get_local_gid = smcd_get_local_gid,
-	.get_chid = smcd_get_chid,
 	.get_dev = smcd_get_dev,
 };
 
 const struct smcd_ops *ism_get_smcd_ops(void)
 {
-	return &ism_ops;
+	return &ism_smcd_ops;
 }
 EXPORT_SYMBOL_GPL(ism_get_smcd_ops);
 #endif
include/linux/dibs.h +88 -2

128 lines not on this page · raw message

include/linux/ism.h +0 -2
index 9a53d3c48c16..c818a25996db 100644
@@ -59,8 +59,6 @@ struct ism_event {
 
 struct ism_client {
 	const char *name;
-	void (*add)(struct ism_dev *dev);
-	void (*remove)(struct ism_dev *dev);
 	void (*handle_event)(struct ism_dev *dev, struct ism_event *event);
 	/* Parameter dmbemask contains a bit vector with updated DMBEs, if sent
 	 * via ism_move_data(). Callback function must handle all active bits
include/net/smc.h +2 -1
index a9c023dd1380..e271891b85e6 100644
@@ -15,6 +15,7 @@
 #include <linux/spinlock.h>
 #include <linux/types.h>
 #include <linux/wait.h>
+#include <linux/dibs.h>
 #include "linux/ism.h"
 
 struct sock;
@@ -62,7 +63,6 @@ struct smcd_ops {
 			 unsigned int size);
 	int (*supports_v2)(void);
 	void (*get_local_gid)(struct smcd_dev *dev, struct smcd_gid *gid);
-	u16 (*get_chid)(struct smcd_dev *dev);
 	struct device* (*get_dev)(struct smcd_dev *dev);
 
 	/* optional operations */
@@ -81,6 +81,7 @@ struct smcd_dev {
 	const struct smcd_ops *ops;
 	void *priv;
 	void *client;
+	struct dibs_dev *dibs;
 	struct list_head list;
 	spinlock_t lock;
 	struct smc_connection **conn;
net/dibs/dibs_loopback.c +11 -0
index c221b55d24a2..1d9d3081c020 100644
@@ -17,6 +17,15 @@
 /* global loopback device */
 static struct dibs_lo_dev *lo_dev;
 
+static u16 dibs_lo_get_fabric_id(struct dibs_dev *dibs)
+{
+	return DIBS_LOOPBACK_FABRIC;
+}
+
+static const struct dibs_dev_ops dibs_lo_ops = {
+	.get_fabric_id = dibs_lo_get_fabric_id,
+};
+
 static void dibs_lo_dev_exit(struct dibs_lo_dev *ldev)
 {
 	dibs_dev_del(ldev->dibs);
@@ -39,6 +48,8 @@ static int dibs_lo_dev_probe(void)
 	}
 
 	ldev->dibs = dibs;
+	dibs->drv_priv = ldev;
+	dibs->ops = &dibs_lo_ops;
 
 	ret = dibs_dev_add(dibs);
 	if (ret)
net/dibs/dibs_main.c +37 -0
index 2c213e1f8f93..d8fa4a0b5935 100644
@@ -35,8 +35,10 @@ static struct dibs_dev_list dibs_dev_list = {
 
 int dibs_register_client(struct dibs_client *client)
 {
+	struct dibs_dev *dibs;
 	int i, rc = -ENOSPC;
 
+	mutex_lock(&dibs_dev_list.mutex);
 	mutex_lock(&clients_lock);
 	for (i = 0; i < MAX_DIBS_CLIENTS; ++i) {
 		if (!clients[i]) {
@@ -50,19 +52,37 @@ int dibs_register_client(struct dibs_client *client)
 	}
 	mutex_unlock(&clients_lock);
 
+	if (i < MAX_DIBS_CLIENTS) {
+		/* initialize with all devices that we got so far */
+		list_for_each_entry(dibs, &dibs_dev_list.list, list) {
+			dibs->priv[i] = NULL;
+			client->ops->add_dev(dibs);
+		}
+	}
+	mutex_unlock(&dibs_dev_list.mutex);
+
 	return rc;
 }
 EXPORT_SYMBOL_GPL(dibs_register_client);
 
 int dibs_unregister_client(struct dibs_client *client)
 {
+	struct dibs_dev *dibs;
 	int rc = 0;
 
+	mutex_lock(&dibs_dev_list.mutex);
+	list_for_each_entry(dibs, &dibs_dev_list.list, list) {
+		clients[client->id]->ops->del_dev(dibs);
+		dibs->priv[client->id] = NULL;
+	}
+
 	mutex_lock(&clients_lock);
 	clients[client->id] = NULL;
 	if (client->id + 1 == max_client)
 		max_client--;
 	mutex_unlock(&clients_lock);
+
+	mutex_unlock(&dibs_dev_list.mutex);
 	return rc;
 }
 EXPORT_SYMBOL_GPL(dibs_unregister_client);
@@ -72,13 +92,22 @@ struct dibs_dev *dibs_dev_alloc(void)
 	struct dibs_dev *dibs;
 
 	dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
+
 	return dibs;
 }
 EXPORT_SYMBOL_GPL(dibs_dev_alloc);
 
 int dibs_dev_add(struct dibs_dev *dibs)
 {
+	int i;
+
 	mutex_lock(&dibs_dev_list.mutex);
+	mutex_lock(&clients_lock);
+	for (i = 0; i < max_client; ++i) {
+		if (clients[i])
+			clients[i]->ops->add_dev(dibs);
+	}
+	mutex_unlock(&clients_lock);
 	list_add(&dibs->list, &dibs_dev_list.list);
 	mutex_unlock(&dibs_dev_list.mutex);
 
@@ -88,7 +117,15 @@ EXPORT_SYMBOL_GPL(dibs_dev_add);
 
 void dibs_dev_del(struct dibs_dev *dibs)
 {
+	int i;
+
 	mutex_lock(&dibs_dev_list.mutex);
+	mutex_lock(&clients_lock);
+	for (i = 0; i < max_client; ++i) {
+		if (clients[i])
+			clients[i]->ops->del_dev(dibs);
+	}
+	mutex_unlock(&clients_lock);
 	list_del_init(&dibs->list);
 	mutex_unlock(&dibs_dev_list.mutex);
 }
net/smc/Kconfig +0 -13
index 40dd60c1d23f..9535d88c2acb 100644
@@ -20,16 +20,3 @@ config SMC_DIAG
 	  smcss.
 
 	  if unsure, say Y.
-
-config SMC_LO
-	bool "SMC intra-OS shortcut with loopback-ism"
-	depends on SMC
-	default n
-	help
-	  SMC_LO enables the creation of an Emulated-ISM device named
-	  loopback-ism in SMC and makes use of it for transferring data
-	  when communication occurs within the same OS. This helps in
-	  convenient testing of SMC-D since loopback-ism is independent
-	  of architecture or hardware.
-
-	  if unsure, say N.
net/smc/Makefile +1 -1
index 60f1c87d5212..96ccfdf246df 100644
@@ -6,4 +6,4 @@ smc-y := af_smc.o smc_pnet.o smc_ib.o smc_clc.o smc_core.o smc_wr.o smc_llc.o
 smc-y += smc_cdc.o smc_tx.o smc_rx.o smc_close.o smc_ism.o smc_netlink.o smc_stats.o
 smc-y += smc_tracepoint.o smc_inet.o
 smc-$(CONFIG_SYSCTL) += smc_sysctl.o
-smc-$(CONFIG_SMC_LO) += smc_loopback.o
+smc-y += smc_loopback.o
net/smc/af_smc.c +1 -11
index 9311c38f7abe..a5f95db7928c 100644
@@ -57,7 +57,6 @@
 #include "smc_stats.h"
 #include "smc_tracepoint.h"
 #include "smc_sysctl.h"
-#include "smc_loopback.h"
 #include "smc_inet.h"
 
 static DEFINE_MUTEX(smc_server_lgr_pending);	/* serialize link group
@@ -3592,16 +3591,10 @@ static int __init smc_init(void)
 		goto out_sock;
 	}
 
-	rc = smc_loopback_init();
-	if (rc) {
-		pr_err("%s: smc_loopback_init fails with %d\n", __func__, rc);
-		goto out_ib;
-	}
-
 	rc = tcp_register_ulp(&smc_ulp_ops);
 	if (rc) {
 		pr_err("%s: tcp_ulp_register fails with %d\n", __func__, rc);
-		goto out_lo;
+		goto out_ib;
 	}
 	rc = smc_inet_init();
 	if (rc) {
@@ -3612,8 +3605,6 @@ static int __init smc_init(void)
 	return 0;
 out_ulp:
 	tcp_unregister_ulp(&smc_ulp_ops);
-out_lo:
-	smc_loopback_exit();
 out_ib:
 	smc_ib_unregister_client();
 out_sock:
@@ -3652,7 +3643,6 @@ static void __exit smc_exit(void)
 	tcp_unregister_ulp(&smc_ulp_ops);
 	sock_unregister(PF_SMC);
 	smc_core_exit();
-	smc_loopback_exit();
 	smc_ib_unregister_client();
 	smc_ism_exit();
 	destroy_workqueue(smc_close_wq);
net/smc/smc_ism.c +74 -29

200 lines not on this page · raw message

net/smc/smc_ism.h +4 -3
index 765aa8fae6fa..04699951d03f 100644
@@ -12,6 +12,7 @@
 #include <linux/uio.h>
 #include <linux/types.h>
 #include <linux/mutex.h>
+#include <linux/dibs.h>
 
 #include "smc.h"
 
@@ -85,14 +86,14 @@ static inline bool __smc_ism_is_emulated(u16 chid)
 
 static inline bool smc_ism_is_emulated(struct smcd_dev *smcd)
 {
-	u16 chid = smcd->ops->get_chid(smcd);
+	u16 chid = smcd->dibs->ops->get_fabric_id(smcd->dibs);
 
 	return __smc_ism_is_emulated(chid);
 }
 
-static inline bool smc_ism_is_loopback(struct smcd_dev *smcd)
+static inline bool smc_ism_is_loopback(struct dibs_dev *dibs)
 {
-	return (smcd->ops->get_chid(smcd) == 0xFFFF);
+	return (dibs->ops->get_fabric_id(dibs) == DIBS_LOOPBACK_FABRIC);
 }
 
 #endif
net/smc/smc_loopback.c +14 -80

152 lines not on this page · raw message

net/smc/smc_loopback.h +3 -14
index 04dc6808d2e1..76c62526e2e5 100644
@@ -17,10 +17,8 @@
 #include <linux/device.h>
 #include <net/smc.h>
 
-#if IS_ENABLED(CONFIG_SMC_LO)
 #define SMC_LO_MAX_DMBS		5000
 #define SMC_LO_DMBS_HASH_BITS	12
-#define SMC_LO_RESERVED_CHID	0xFFFF
 
 struct smc_lo_dmb_node {
 	struct hlist_node list;
@@ -35,7 +33,6 @@ struct smc_lo_dmb_node {
 struct smc_lo_dev {
 	struct smcd_dev *smcd;
 	struct device dev;
-	u16 chid;
 	struct smcd_gid local_gid;
 	atomic_t dmb_cnt;
 	rwlock_t dmb_ht_lock;
@@ -44,17 +41,9 @@ struct smc_lo_dev {
 	wait_queue_head_t ldev_release;
 };
 
-int smc_loopback_init(void);
+const struct smcd_ops *smc_lo_get_smcd_ops(void);
+
+int smc_loopback_init(struct smc_lo_dev **smc_lb);
 void smc_loopback_exit(void);
-#else
-static inline int smc_loopback_init(void)
-{
-	return 0;
-}
-
-static inline void smc_loopback_exit(void)
-{
-}
-#endif
 
 #endif /* _SMC_LOOPBACK_H */
Signature · 1 line
2.48.1
author maintainer
#34 lore raw

RFC 11/17 net/dibs: Move struct device to dibs_dev

In reply to Alexandra Winter

From: Julian Ruess <[email protected]>

Move struct device from ism_dev and smc_lo_dev to dibs_dev, and define a corresponding release function. Free ism_dev in ism_remove() and smc_lo_dev in smc_lo_dev_remove().

Replace smcd->ops->get_dev(smcd) by dibs_get_dev().

An alternative design would be to embed dibs_dev as a field in ism_dev and do the same for other dibs device driver specific structs. However that would have the disadvantage that each dibs device driver needs to allocate dibs_dev and each dibs device driver needs a different device release function. The advantage would be that ism_dev and other device driver specific structs would be covered by device reference counts.

---
 drivers/s390/net/ism_drv.c | 40 ++++++++------------------------------
 include/linux/dibs.h       | 16 +++++++++++++++
 include/linux/ism.h        |  1 -
 include/net/smc.h          |  1 -
 net/dibs/dibs_loopback.c   | 15 +++++++-------
 net/dibs/dibs_main.c       | 21 +++++++++++++++++++-
 net/smc/smc_core.c         |  4 ++--
 net/smc/smc_ism.c          | 28 ++++++++++++--------------
 net/smc/smc_loopback.c     | 21 +-------------------
 net/smc/smc_loopback.h     |  1 -
 net/smc/smc_pnet.c         |  8 ++++----
 11 files changed, 70 insertions(+), 86 deletions(-)
drivers/s390/net/ism_drv.c +8 -32
index 84a6e9ae2e64..0ddfd47a3a7c 100644
@@ -602,15 +602,6 @@ static int ism_dev_init(struct ism_dev *ism)
 	return ret;
 }
 
-static void ism_dev_release(struct device *dev)
-{
-	struct ism_dev *ism;
-
-	ism = container_of(dev, struct ism_dev, dev);
-
-	kfree(ism);
-}
-
 static void ism_dev_exit(struct ism_dev *ism)
 {
 	struct pci_dev *pdev = ism->pdev;
@@ -649,17 +640,10 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	spin_lock_init(&ism->cmd_lock);
 	dev_set_drvdata(&pdev->dev, ism);
 	ism->pdev = pdev;
-	ism->dev.parent = &pdev->dev;
-	ism->dev.release = ism_dev_release;
-	device_initialize(&ism->dev);
-	dev_set_name(&ism->dev, "%s", dev_name(&pdev->dev));
-	ret = device_add(&ism->dev);
-	if (ret)
-		goto err_dev;
 
 	ret = pci_enable_device_mem(pdev);
 	if (ret)
-		goto err;
+		goto err_dev;
 
 	ret = pci_request_mem_regions(pdev, DRV_NAME);
 	if (ret)
@@ -687,6 +671,9 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (ret)
 		goto err_dibs;
 
+	dibs->dev.parent = &pdev->dev;
+	dev_set_name(&dibs->dev, "%s", dev_name(&pdev->dev));
+
 	ret = dibs_dev_add(dibs);
 	if (ret)
 		goto err_ism;
@@ -697,16 +684,14 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	ism_dev_exit(ism);
 err_dibs:
 	/* pairs with dibs_dev_alloc() */
-	kfree(dibs);
+	put_device(dibs_get_dev(dibs));
 err_resource:
 	pci_release_mem_regions(pdev);
 err_disable:
 	pci_disable_device(pdev);
-err:
-	device_del(&ism->dev);
 err_dev:
 	dev_set_drvdata(&pdev->dev, NULL);
-	put_device(&ism->dev);
+	kfree(ism);
 
 	return ret;
 }
@@ -719,13 +704,12 @@ static void ism_remove(struct pci_dev *pdev)
 	dibs_dev_del(dibs);
 	ism_dev_exit(ism);
 	/* pairs with dibs_dev_alloc() */
-	kfree(dibs);
+	put_device(dibs_get_dev(dibs));
 
 	pci_release_mem_regions(pdev);
 	pci_disable_device(pdev);
-	device_del(&ism->dev);
 	dev_set_drvdata(&pdev->dev, NULL);
-	put_device(&ism->dev);
+	kfree(ism);
 }
 
 static struct pci_driver ism_driver = {
@@ -871,13 +855,6 @@ static void smcd_get_local_gid(struct smcd_dev *smcd,
 	smcd_gid->gid_ext = 0;
 }
 
-static inline struct device *smcd_get_dev(struct smcd_dev *dev)
-{
-	struct ism_dev *ism = dev->priv;
-
-	return &ism->dev;
-}
-
 static const struct smcd_ops ism_smcd_ops = {
 	.query_remote_gid = smcd_query_rgid,
 	.register_dmb = smcd_register_dmb,
@@ -890,7 +867,6 @@ static const struct smcd_ops ism_smcd_ops = {
 	.move_data = smcd_move,
 	.supports_v2 = smcd_supports_v2,
 	.get_local_gid = smcd_get_local_gid,
-	.get_dev = smcd_get_dev,
 };
 
 const struct smcd_ops *ism_get_smcd_ops(void)
include/linux/dibs.h +16 -0
index 805ab33271b5..4459b9369dc0 100644
@@ -135,6 +135,7 @@ struct dibs_dev_ops {
 
 struct dibs_dev {
 	struct list_head list;
+	struct device dev;
 	/* To be filled by device driver, before calling dibs_dev_add(): */
 	const struct dibs_dev_ops *ops;
 	/* priv pointer for device driver */
@@ -158,6 +159,21 @@ static inline void *dibs_get_priv(struct dibs_dev *dev,
 
 /* ------- End of client-only functions ----------- */
 
+/* Functions to be called by dibs clients and dibs device drivers:
+ */
+/**
+ * dibs_get_dev()
+ * @dev: dibs device
+ * @token: dmb token of the remote dmb
+ *
+ * TODO: provide get and put functions
+ * Return: struct device* to be used for device refcounting
+ */
+static inline struct device *dibs_get_dev(struct dibs_dev *dibs)
+{
+	return &dibs->dev;
+}
+
 /* Functions to be called by dibs device drivers:
  */
 /**
include/linux/ism.h +0 -1
index c818a25996db..84f1afb3dded 100644
@@ -42,7 +42,6 @@ struct ism_dev {
 	struct ism_eq *ieq;
 	dma_addr_t ieq_dma_addr;
 
-	struct device dev;
 	u64 local_gid;
 	int ieq_idx;
 
include/net/smc.h +0 -1
index e271891b85e6..05faac83371e 100644
@@ -63,7 +63,6 @@ struct smcd_ops {
 			 unsigned int size);
 	int (*supports_v2)(void);
 	void (*get_local_gid)(struct smcd_dev *dev, struct smcd_gid *gid);
-	struct device* (*get_dev)(struct smcd_dev *dev);
 
 	/* optional operations */
 	int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
net/dibs/dibs_loopback.c +7 -8
index 1d9d3081c020..bf02563527b4 100644
@@ -14,6 +14,7 @@
 
 #include "dibs_loopback.h"
 
+static const char dibs_lo_dev_name[] = "lo";
 /* global loopback device */
 static struct dibs_lo_dev *lo_dev;
 
@@ -26,11 +27,6 @@ static const struct dibs_dev_ops dibs_lo_ops = {
 	.get_fabric_id = dibs_lo_get_fabric_id,
 };
 
-static void dibs_lo_dev_exit(struct dibs_lo_dev *ldev)
-{
-	dibs_dev_del(ldev->dibs);
-}
-
 static int dibs_lo_dev_probe(void)
 {
 	struct dibs_lo_dev *ldev;
@@ -51,6 +47,9 @@ static int dibs_lo_dev_probe(void)
 	dibs->drv_priv = ldev;
 	dibs->ops = &dibs_lo_ops;
 
+	dibs->dev.parent = NULL;
+	dev_set_name(&dibs->dev, "%s", dibs_lo_dev_name);
+
 	ret = dibs_dev_add(dibs);
 	if (ret)
 		goto err_reg;
@@ -59,7 +58,7 @@ static int dibs_lo_dev_probe(void)
 
 err_reg:
 	/* pairs with dibs_dev_alloc() */
-	kfree(dibs);
+	put_device(&dibs->dev);
 	kfree(ldev);
 
 	return ret;
@@ -70,9 +69,9 @@ static void dibs_lo_dev_remove(void)
 	if (!lo_dev)
 		return;
 
-	dibs_lo_dev_exit(lo_dev);
+	dibs_dev_del(lo_dev->dibs);
 	/* pairs with dibs_dev_alloc() */
-	kfree(lo_dev->dibs);
+	put_device(&lo_dev->dibs->dev);
 	kfree(lo_dev);
 	lo_dev = NULL;
 }
net/dibs/dibs_main.c +20 -1
index d8fa4a0b5935..8ffe1b4c90ef 100644
@@ -87,11 +87,24 @@ int dibs_unregister_client(struct dibs_client *client)
 }
 EXPORT_SYMBOL_GPL(dibs_unregister_client);
 
+static void dibs_dev_release(struct device *dev)
+{
+	struct dibs_dev *dibs;
+
+	dibs = container_of(dev, struct dibs_dev, dev);
+
+	kfree(dibs);
+}
+
 struct dibs_dev *dibs_dev_alloc(void)
 {
 	struct dibs_dev *dibs;
 
 	dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
+	if (!dibs)
+		return dibs;
+	dibs->dev.release = dibs_dev_release;
+	device_initialize(&dibs->dev);
 
 	return dibs;
 }
@@ -99,7 +112,11 @@ EXPORT_SYMBOL_GPL(dibs_dev_alloc);
 
 int dibs_dev_add(struct dibs_dev *dibs)
 {
-	int i;
+	int i, ret;
+
+	ret = device_add(&dibs->dev);
+	if (ret)
+		return ret;
 
 	mutex_lock(&dibs_dev_list.mutex);
 	mutex_lock(&clients_lock);
@@ -128,6 +145,8 @@ void dibs_dev_del(struct dibs_dev *dibs)
 	mutex_unlock(&clients_lock);
 	list_del_init(&dibs->list);
 	mutex_unlock(&dibs_dev_list.mutex);
+
+	device_del(&dibs->dev);
 }
 EXPORT_SYMBOL_GPL(dibs_dev_del);
 
net/smc/smc_core.c +2 -2
index 67f9e0b83ebc..71c410dc3658 100644
@@ -924,7 +924,7 @@ static int smc_lgr_create(struct smc_sock *smc, struct smc_init_info *ini)
 	if (ini->is_smcd) {
 		/* SMC-D specific settings */
 		smcd = ini->ism_dev[ini->ism_selected];
-		get_device(smcd->ops->get_dev(smcd));
+		get_device(dibs_get_dev(smcd->dibs));
 		lgr->peer_gid.gid =
 			ini->ism_peer_gid[ini->ism_selected].gid;
 		lgr->peer_gid.gid_ext =
@@ -1474,7 +1474,7 @@ static void smc_lgr_free(struct smc_link_group *lgr)
 	destroy_workqueue(lgr->tx_wq);
 	if (lgr->is_smcd) {
 		smc_ism_put_vlan(lgr->smcd, lgr->vlan_id);
-		put_device(lgr->smcd->ops->get_dev(lgr->smcd));
+		put_device(dibs_get_dev(lgr->smcd->dibs));
 	}
 	smc_lgr_put(lgr); /* theoretically last lgr_put */
 }
net/smc/smc_ism.c +12 -16
index 0943e7d4cd2a..dd46d8000381 100644
@@ -303,12 +303,12 @@ static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd,
 	char smc_pnet[SMC_MAX_PNETID_LEN + 1];
 	struct smc_pci_dev smc_pci_dev;
 	struct nlattr *port_attrs;
+	struct dibs_dev *dibs;
 	struct nlattr *attrs;
-	struct ism_dev *ism;
 	int use_cnt = 0;
 	void *nlh;
 
-	ism = smcd->priv;
+	dibs = smcd->dibs;
 	nlh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq,
 			  &smc_gen_nl_family, NLM_F_MULTI,
 			  SMC_NETLINK_GET_DEV_SMCD);
@@ -323,7 +323,7 @@ static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd,
 	if (nla_put_u8(skb, SMC_NLA_DEV_IS_CRIT, use_cnt > 0))
 		goto errattr;
 	memset(&smc_pci_dev, 0, sizeof(smc_pci_dev));
-	smc_set_pci_values(ism->pdev, &smc_pci_dev);
+	smc_set_pci_values(to_pci_dev(dibs->dev.parent), &smc_pci_dev);
 	if (nla_put_u32(skb, SMC_NLA_DEV_PCI_FID, smc_pci_dev.pci_fid))
 		goto errattr;
 	if (nla_put_u16(skb, SMC_NLA_DEV_PCI_CHID, smc_pci_dev.pci_pchid))
@@ -509,12 +509,12 @@ static void smcd_register_dev(struct dibs_dev *dibs)
 
 	if (smc_ism_is_loopback(dibs)) {
 		ops = smc_lo_get_smcd_ops();
-		smcd = smcd_alloc_dev(dev_name(&smc_lo->dev), ops,
+		smcd = smcd_alloc_dev(dev_name(&dibs->dev), ops,
 				      SMC_LO_MAX_DMBS);
 	} else {
 		ism = dibs->drv_priv;
 		ops = ism_get_smcd_ops();
-		smcd = smcd_alloc_dev(dev_name(&ism->pdev->dev), ops,
+		smcd = smcd_alloc_dev(dev_name(&dibs->dev), ops,
 				      ISM_NR_DMBS);
 	}
 	if (!smcd)
@@ -529,10 +529,11 @@ static void smcd_register_dev(struct dibs_dev *dibs)
 	} else {
 		smcd->priv = ism;
 		ism_set_priv(ism, &smc_ism_client, smcd);
-		if (smc_pnetid_by_dev_port(&ism->pdev->dev, 0, smcd->pnetid))
-			smc_pnetid_by_table_smcd(smcd);
 	}
 
+	if (smc_pnetid_by_dev_port(dibs->dev.parent, 0, smcd->pnetid))
+		smc_pnetid_by_table_smcd(smcd);
+
 	smcd->client = &smc_ism_client;
 
 	if (smcd->ops->supports_v2())
@@ -556,27 +557,22 @@ static void smcd_register_dev(struct dibs_dev *dibs)
 
 	if (smc_pnet_is_pnetid_set(smcd->pnetid))
 		pr_warn_ratelimited("smc: adding smcd device %s with pnetid %.16s%s\n",
-				    dev_name(&ism->dev), smcd->pnetid,
+				    dev_name(&dibs->dev), smcd->pnetid,
 				    smcd->pnetid_by_user ?
 					" (user defined)" :
 					"");
 	else
 		pr_warn_ratelimited("smc: adding smcd device %s without pnetid\n",
-				    dev_name(&ism->dev));
+				    dev_name(&dibs->dev));
 	return;
 }
 
 static void smcd_unregister_dev(struct dibs_dev *dibs)
 {
 	struct smcd_dev *smcd = dibs_get_priv(dibs, &smc_dibs_client);
-	struct ism_dev *ism = dibs->drv_priv;
 
-	if (smc_ism_is_loopback(dibs)) {
-		pr_warn_ratelimited("smc: removing smcd loopback device\n");
-	} else {
-		pr_warn_ratelimited("smc: removing smcd device %s\n",
-				    dev_name(&ism->dev));
-	}
+	pr_warn_ratelimited("smc: removing smcd device %s\n",
+			    dev_name(&dibs->dev));
 	smcd->going_away = 1;
 	smc_smcd_terminate_all(smcd);
 	mutex_lock(&smcd_dev_list.mutex);
net/smc/smc_loopback.c +1 -20
index 37d8366419f7..262d0d0df4d0 100644
@@ -23,7 +23,6 @@
 #define SMC_LO_SUPPORT_NOCOPY	0x1
 #define SMC_DMA_ADDR_INVALID	(~(dma_addr_t)0)
 
-static const char smc_lo_dev_name[] = "loopback-ism";
 static struct smc_lo_dev *lo_dev;
 
 static void smc_lo_generate_ids(struct smc_lo_dev *ldev)
@@ -255,11 +254,6 @@ static void smc_lo_get_local_gid(struct smcd_dev *smcd,
 	smcd_gid->gid_ext = ldev->local_gid.gid_ext;
 }
 
-static struct device *smc_lo_get_dev(struct smcd_dev *smcd)
-{
-	return &((struct smc_lo_dev *)smcd->priv)->dev;
-}
-
 static const struct smcd_ops lo_ops = {
 	.query_remote_gid = smc_lo_query_rgid,
 	.register_dmb = smc_lo_register_dmb,
@@ -274,7 +268,6 @@ static const struct smcd_ops lo_ops = {
 	.signal_event		= NULL,
 	.move_data = smc_lo_move_data,
 	.get_local_gid = smc_lo_get_local_gid,
-	.get_dev = smc_lo_get_dev,
 };
 
 const struct smcd_ops *smc_lo_get_smcd_ops(void)
@@ -299,14 +292,6 @@ static void smc_lo_dev_exit(struct smc_lo_dev *ldev)
 		wait_event(ldev->ldev_release, !atomic_read(&ldev->dmb_cnt));
 }
 
-static void smc_lo_dev_release(struct device *dev)
-{
-	struct smc_lo_dev *ldev =
-		container_of(dev, struct smc_lo_dev, dev);
-
-	kfree(ldev);
-}
-
 static int smc_lo_dev_probe(void)
 {
 	struct smc_lo_dev *ldev;
@@ -315,10 +300,6 @@ static int smc_lo_dev_probe(void)
 	if (!ldev)
 		return -ENOMEM;
 
-	ldev->dev.parent = NULL;
-	ldev->dev.release = smc_lo_dev_release;
-	device_initialize(&ldev->dev);
-	dev_set_name(&ldev->dev, smc_lo_dev_name);
 	smc_lo_dev_init(ldev);
 
 	lo_dev = ldev; /* global loopback device */
@@ -332,7 +313,7 @@ static void smc_lo_dev_remove(void)
 		return;
 
 	smc_lo_dev_exit(lo_dev);
-	put_device(&lo_dev->dev); /* device_initialize in smc_lo_dev_probe */
+	kfree(lo_dev);
 	lo_dev = NULL;
 }
 
net/smc/smc_loopback.h +0 -1
index 76c62526e2e5..a033bf10890a 100644
@@ -32,7 +32,6 @@ struct smc_lo_dmb_node {
 
 struct smc_lo_dev {
 	struct smcd_dev *smcd;
-	struct device dev;
 	struct smcd_gid local_gid;
 	atomic_t dmb_cnt;
 	rwlock_t dmb_ht_lock;
net/smc/smc_pnet.c +4 -4
index 76ad29e31d60..bbdd875731f2 100644
@@ -169,7 +169,7 @@ static int smc_pnet_remove_by_pnetid(struct net *net, char *pnet_name)
 			pr_warn_ratelimited("smc: smcd device %s "
 					    "erased user defined pnetid "
 					    "%.16s\n",
-					    dev_name(smcd->ops->get_dev(smcd)),
+					    dev_name(dibs_get_dev(smcd->dibs)),
 					    smcd->pnetid);
 			memset(smcd->pnetid, 0, SMC_MAX_PNETID_LEN);
 			smcd->pnetid_by_user = false;
@@ -332,7 +332,7 @@ static struct smcd_dev *smc_pnet_find_smcd(char *smcd_name)
 
 	mutex_lock(&smcd_dev_list.mutex);
 	list_for_each_entry(smcd_dev, &smcd_dev_list.list, list) {
-		if (!strncmp(dev_name(smcd_dev->ops->get_dev(smcd_dev)),
+		if (!strncmp(dev_name(dibs_get_dev(smcd_dev->dibs)),
 			     smcd_name, IB_DEVICE_NAME_MAX - 1))
 			goto out;
 	}
@@ -431,7 +431,7 @@ static int smc_pnet_add_ib(struct smc_pnettable *pnettable, char *ib_name,
 	if (smcd) {
 		smcddev_applied = smc_pnet_apply_smcd(smcd, pnet_name);
 		if (smcddev_applied) {
-			dev = smcd->ops->get_dev(smcd);
+			dev = dibs_get_dev(smcd->dibs);
 			pr_warn_ratelimited("smc: smcd device %s "
 					    "applied user defined pnetid "
 					    "%.16s\n", dev_name(dev),
@@ -1192,7 +1192,7 @@ int smc_pnetid_by_table_ib(struct smc_ib_device *smcibdev, u8 ib_port)
  */
 int smc_pnetid_by_table_smcd(struct smcd_dev *smcddev)
 {
-	const char *ib_name = dev_name(smcddev->ops->get_dev(smcddev));
+	const char *ib_name = dev_name(dibs_get_dev(smcddev->dibs));
 	struct smc_pnettable *pnettable;
 	struct smc_pnetentry *tmp_pe;
 	struct smc_net *sn;
Signature · 1 line
2.48.1
author maintainer
#47 lore raw

RFC 16/17 net/dibs: Move data path to dibs layer

In reply to Alexandra Winter

Use struct dibs_dmb instead of struct smc_dmb and move the corresponding client tables to dibs_dev. Leave driver specific implementation details like sba in the device drivers.

Register and unregister dmbs via dibs_dev_ops. A dmb is dedicated to a single client, but a dibs device can have dmbs for more than one client.

Trigger dibs clients via dibs_client_ops->handle_irq(), when data is received into a dmb. For dibs_loopback replace scheduling an smcd receive tasklet with calling dibs_client_ops->handle_irq().

For loopback devices attach_dmb(), detach_dmb() and move_data() need to access the dmb tables, so move those to dibs_dev_ops in this patch as well.

Remove remaining definitions of smc_loopback as they are no longer required, now that everything is in dibs_loopback.

Note that struct ism_client and struct ism_dev are still required in smc until a follow-on patch moves event handling to dibs. (Loopback does not use events).

---
 drivers/s390/net/ism_drv.c | 121 +++++++--------
 include/linux/dibs.h       | 177 ++++++++++++++++++++++
 include/linux/ism.h        |  23 ---
 include/net/smc.h          |  22 ---
 net/dibs/dibs_loopback.c   | 257 ++++++++++++++++++++++++++++++++
 net/dibs/dibs_loopback.h   |  19 +++
 net/dibs/dibs_main.c       |  56 ++++++-
 net/smc/Makefile           |   1 -
 net/smc/smc_ism.c          |  66 ++++-----
 net/smc/smc_ism.h          |   4 +-
 net/smc/smc_loopback.c     | 294 -------------------------------------
 net/smc/smc_loopback.h     |  47 ------
 12 files changed, 588 insertions(+), 499 deletions(-)
 delete mode 100644 net/smc/smc_loopback.c
 delete mode 100644 net/smc/smc_loopback.h
drivers/s390/net/ism_drv.c +49 -72
index 6a30178b1b06..919b32a152f4 100644
@@ -98,14 +98,6 @@ int ism_unregister_client(struct ism_client *client)
 		spin_lock_irqsave(&ism->lock, flags);
 		/* Stop forwarding IRQs and events */
 		ism->subs[client->id] = NULL;
-		for (int i = 0; i < ISM_NR_DMBS; ++i) {
-			if (ism->sba_client_arr[i] == client->id) {
-				WARN(1, "%s: attempt to unregister '%s' with registered dmb(s)\n",
-				     __func__, client->name);
-				rc = -EBUSY;
-				goto err_reg_dmb;
-			}
-		}
 		spin_unlock_irqrestore(&ism->lock, flags);
 	}
 	mutex_unlock(&ism_dev_list.mutex);
@@ -116,11 +108,6 @@ int ism_unregister_client(struct ism_client *client)
 		max_client--;
 	mutex_unlock(&clients_lock);
 	return rc;
-
-err_reg_dmb:
-	spin_unlock_irqrestore(&ism->lock, flags);
-	mutex_unlock(&ism_dev_list.mutex);
-	return rc;
 }
 EXPORT_SYMBOL_GPL(ism_unregister_client);
 
@@ -308,15 +295,20 @@ static int ism_query_rgid(struct dibs_dev *dibs, uuid_t *rgid, u32 vid_valid,
 	return ism_cmd(ism, &cmd);
 }
 
-static void ism_free_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
+static int ism_max_dmbs(void)
+{
+	return ISM_NR_DMBS;
+}
+
+static void ism_free_dmb(struct ism_dev *ism, struct dibs_dmb *dmb)
 {
-	clear_bit(dmb->sba_idx, ism->sba_bitmap);
+	clear_bit(dmb->idx, ism->sba_bitmap);
 	dma_unmap_page(&ism->pdev->dev, dmb->dma_addr, dmb->dmb_len,
 		       DMA_FROM_DEVICE);
 	folio_put(virt_to_folio(dmb->cpu_addr));
 }
 
-static int ism_alloc_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
+static int ism_alloc_dmb(struct ism_dev *ism, struct dibs_dmb *dmb)
 {
 	struct folio *folio;
 	unsigned long bit;
@@ -325,16 +317,16 @@ static int ism_alloc_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
 	if (PAGE_ALIGN(dmb->dmb_len) > dma_get_max_seg_size(&ism->pdev->dev))
 		return -EINVAL;
 
-	if (!dmb->sba_idx) {
+	if (!dmb->idx) {
 		bit = find_next_zero_bit(ism->sba_bitmap, ISM_NR_DMBS,
 					 ISM_DMB_BIT_OFFSET);
 		if (bit == ISM_NR_DMBS)
 			return -ENOSPC;
 
-		dmb->sba_idx = bit;
+		dmb->idx = bit;
 	}
-	if (dmb->sba_idx < ISM_DMB_BIT_OFFSET ||
-	    test_and_set_bit(dmb->sba_idx, ism->sba_bitmap))
+	if (dmb->idx < ISM_DMB_BIT_OFFSET ||
+	    test_and_set_bit(dmb->idx, ism->sba_bitmap))
 		return -EINVAL;
 
 	folio = folio_alloc(GFP_KERNEL | __GFP_NOWARN | __GFP_NOMEMALLOC |
@@ -359,13 +351,14 @@ static int ism_alloc_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
 out_free:
 	kfree(dmb->cpu_addr);
 out_bit:
-	clear_bit(dmb->sba_idx, ism->sba_bitmap);
+	clear_bit(dmb->idx, ism->sba_bitmap);
 	return rc;
 }
 
-int ism_register_dmb(struct ism_dev *ism, struct ism_dmb *dmb,
-		     struct ism_client *client)
+static int ism_register_dmb(struct dibs_dev *dibs, struct dibs_dmb *dmb,
+			    struct dibs_client *client)
 {
+	struct ism_dev *ism = dibs->drv_priv;
 	union ism_reg_dmb cmd;
 	unsigned long flags;
 	int ret;
@@ -380,10 +373,10 @@ int ism_register_dmb(struct ism_dev *ism, struct ism_dmb *dmb,
 
 	cmd.request.dmb = dmb->dma_addr;
 	cmd.request.dmb_len = dmb->dmb_len;
-	cmd.request.sba_idx = dmb->sba_idx;
+	cmd.request.sba_idx = dmb->idx;
 	cmd.request.vlan_valid = dmb->vlan_valid;
 	cmd.request.vlan_id = dmb->vlan_id;
-	cmd.request.rgid = dmb->rgid;
+	memcpy(&cmd.request.rgid, &dmb->rgid, sizeof(u64));
 
 	ret = ism_cmd(ism, &cmd);
 	if (ret) {
@@ -391,16 +384,16 @@ int ism_register_dmb(struct ism_dev *ism, struct ism_dmb *dmb,
 		goto out;
 	}
 	dmb->dmb_tok = cmd.response.dmb_tok;
-	spin_lock_irqsave(&ism->lock, flags);
-	ism->sba_client_arr[dmb->sba_idx - ISM_DMB_BIT_OFFSET] = client->id;
-	spin_unlock_irqrestore(&ism->lock, flags);
+	spin_lock_irqsave(&dibs->lock, flags);
+	dibs->dmb_clientid_arr[dmb->idx - ISM_DMB_BIT_OFFSET] = client->id;
+	spin_unlock_irqrestore(&dibs->lock, flags);
 out:
 	return ret;
 }
-EXPORT_SYMBOL_GPL(ism_register_dmb);
 
-int ism_unregister_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
+static int ism_unregister_dmb(struct dibs_dev *dibs, struct dibs_dmb *dmb)
 {
+	struct ism_dev *ism = dibs->drv_priv;
 	union ism_unreg_dmb cmd;
 	unsigned long flags;
 	int ret;
@@ -411,9 +404,9 @@ int ism_unregister_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
 
 	cmd.request.dmb_tok = dmb->dmb_tok;
 
-	spin_lock_irqsave(&ism->lock, flags);
-	ism->sba_client_arr[dmb->sba_idx - ISM_DMB_BIT_OFFSET] = NO_CLIENT;
-	spin_unlock_irqrestore(&ism->lock, flags);
+	spin_lock_irqsave(&dibs->lock, flags);
+	dibs->dmb_clientid_arr[dmb->idx - ISM_DMB_BIT_OFFSET] = NO_DIBS_CLIENT;
+	spin_unlock_irqrestore(&dibs->lock, flags);
 
 	ret = ism_cmd(ism, &cmd);
 	if (ret && ret != ISM_ERROR)
@@ -423,7 +416,6 @@ int ism_unregister_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
 out:
 	return ret;
 }
-EXPORT_SYMBOL_GPL(ism_unregister_dmb);
 
 static int ism_add_vlan_id(struct dibs_dev *dibs, u64 vlan_id)
 {
@@ -459,9 +451,11 @@ static unsigned int max_bytes(unsigned int start, unsigned int len,
 	return min(boundary - (start & (boundary - 1)), len);
 }
 
-int ism_move(struct ism_dev *ism, u64 dmb_tok, unsigned int idx, bool sf,
-	     unsigned int offset, void *data, unsigned int size)
+static int ism_move(struct dibs_dev *dibs, u64 dmb_tok, unsigned int idx,
+		    bool sf, unsigned int offset, void *data,
+		    unsigned int size)
 {
+	struct ism_dev *ism = dibs->drv_priv;
 	unsigned int bytes;
 	u64 dmb_req;
 	int ret;
@@ -482,7 +476,6 @@ int ism_move(struct ism_dev *ism, u64 dmb_tok, unsigned int idx, bool sf,
 
 	return 0;
 }
-EXPORT_SYMBOL_GPL(ism_move);
 
 static u16 ism_get_chid(struct dibs_dev *dibs)
 {
@@ -518,14 +511,17 @@ static irqreturn_t ism_handle_irq(int irq, void *data)
 {
 	struct ism_dev *ism = data;
 	unsigned long bit, end;
+	struct dibs_dev *dibs;
 	unsigned long *bv;
 	u16 dmbemask;
 	u8 client_id;
 
+	dibs = ism->dibs;
+
 	bv = (void *) &ism->sba->dmb_bits[ISM_DMB_WORD_OFFSET];
 	end = sizeof(ism->sba->dmb_bits) * BITS_PER_BYTE - ISM_DMB_BIT_OFFSET;
 
-	spin_lock(&ism->lock);
+	spin_lock(&dibs->lock);
 	ism->sba->s = 0;
 	barrier();
 	for (bit = 0;;) {
@@ -537,10 +533,13 @@ static irqreturn_t ism_handle_irq(int irq, void *data)
 		dmbemask = ism->sba->dmbe_mask[bit + ISM_DMB_BIT_OFFSET];
 		ism->sba->dmbe_mask[bit + ISM_DMB_BIT_OFFSET] = 0;
 		barrier();
-		client_id = ism->sba_client_arr[bit];
-		if (unlikely(client_id == NO_CLIENT || !ism->subs[client_id]))
+		client_id = dibs->dmb_clientid_arr[bit];
+		if (unlikely(client_id == NO_DIBS_CLIENT ||
+			     !dibs->subs[client_id]))
 			continue;
-		ism->subs[client_id]->handle_irq(ism, bit + ISM_DMB_BIT_OFFSET, dmbemask);
+		dibs->subs[client_id]->ops->handle_irq(dibs,
+						       bit + ISM_DMB_BIT_OFFSET,
+						       dmbemask);
 	}
 
 	if (ism->sba->e) {
@@ -548,13 +547,17 @@ static irqreturn_t ism_handle_irq(int irq, void *data)
 		barrier();
 		ism_handle_event(ism);
 	}
-	spin_unlock(&ism->lock);
+	spin_unlock(&dibs->lock);
 	return IRQ_HANDLED;
 }
 
 static const struct dibs_dev_ops ism_ops = {
 	.get_fabric_id = ism_get_chid,
 	.query_remote_gid = ism_query_rgid,
+	.max_dmbs = ism_max_dmbs,
+	.register_dmb = ism_register_dmb,
+	.unregister_dmb = ism_unregister_dmb,
+	.move_data = ism_move,
 	.add_vlan_id = ism_add_vlan_id,
 	.del_vlan_id = ism_del_vlan_id,
 };
@@ -568,15 +571,10 @@ static int ism_dev_init(struct ism_dev *ism)
 	if (ret <= 0)
 		goto out;
 
-	ism->sba_client_arr = kzalloc(ISM_NR_DMBS, GFP_KERNEL);
-	if (!ism->sba_client_arr)
-		goto free_vectors;
-	memset(ism->sba_client_arr, NO_CLIENT, ISM_NR_DMBS);
-
 	ret = request_irq(pci_irq_vector(pdev, 0), ism_handle_irq, 0,
 			  pci_name(pdev), ism);
 	if (ret)
-		goto free_client_arr;
+		goto free_vectors;
 
 	ret = register_sba(ism);
 	if (ret)
@@ -605,8 +603,6 @@ static int ism_dev_init(struct ism_dev *ism)
 	unregister_sba(ism);
 free_irq:
 	free_irq(pci_irq_vector(pdev, 0), ism);
-free_client_arr:
-	kfree(ism->sba_client_arr);
 free_vectors:
 	pci_free_irq_vectors(pdev);
 out:
@@ -629,7 +625,6 @@ static void ism_dev_exit(struct ism_dev *ism)
 	unregister_ieq(ism);
 	unregister_sba(ism);
 	free_irq(pci_irq_vector(pdev, 0), ism);
-	kfree(ism->sba_client_arr);
 	pci_free_irq_vectors(pdev);
 	list_del_init(&ism->list);
 	mutex_unlock(&ism_dev_list.mutex);
@@ -677,6 +672,9 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	dibs->drv_priv = ism;
 	dibs->ops = &ism_ops;
 
+	/* enable ism device, but any interrupts and events will be ignored
+	 * before dibs_dev_add() adds it to any clients.
+	 */
 	ret = ism_dev_init(ism);
 	if (ret)
 		goto err_dibs;
@@ -771,17 +769,6 @@ module_exit(ism_exit);
 /*************************** SMC-D Implementation *****************************/
 
 #if IS_ENABLED(CONFIG_SMC)
-static int smcd_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
-			     void *client)
-{
-	return ism_register_dmb(smcd->priv, (struct ism_dmb *)dmb, client);
-}
-
-static int smcd_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
-{
-	return ism_unregister_dmb(smcd->priv, (struct ism_dmb *)dmb);
-}
-
 static int ism_signal_ieq(struct ism_dev *ism, u64 rgid, u32 trigger_irq,
 			  u32 event_code, u64 info)
 {
@@ -806,18 +793,8 @@ static int smcd_signal_ieq(struct smcd_dev *smcd, struct smcd_gid *rgid,
 			      trigger_irq, event_code, info);
 }
 
-static int smcd_move(struct smcd_dev *smcd, u64 dmb_tok, unsigned int idx,
-		     bool sf, unsigned int offset, void *data,
-		     unsigned int size)
-{
-	return ism_move(smcd->priv, dmb_tok, idx, sf, offset, data, size);
-}
-
 static const struct smcd_ops ism_smcd_ops = {
-	.register_dmb = smcd_register_dmb,
-	.unregister_dmb = smcd_unregister_dmb,
 	.signal_event = smcd_signal_ieq,
-	.move_data = smcd_move,
 };
 
 const struct smcd_ops *ism_get_smcd_ops(void)
include/linux/dibs.h +177 -0

213 lines not on this page · raw message

include/linux/ism.h +0 -23
index a926dd61b5a1..b7feb4dcd5a8 100644
@@ -11,17 +11,6 @@
 
 #include <linux/workqueue.h>
 
-struct ism_dmb {
-	u64 dmb_tok;
-	u64 rgid;
-	u32 dmb_len;
-	u32 sba_idx;
-	u32 vlan_valid;
-	u32 vlan_id;
-	void *cpu_addr;
-	dma_addr_t dma_addr;
-};
-
 /* Unless we gain unexpected popularity, this limit should hold for a while */
 #define MAX_CLIENTS		8
 #define ISM_NR_DMBS		1920
@@ -36,7 +25,6 @@ struct ism_dev {
 	struct ism_sba *sba;
 	dma_addr_t sba_dma_addr;
 	DECLARE_BITMAP(sba_bitmap, ISM_NR_DMBS);
-	u8 *sba_client_arr;	/* entries are indices into 'clients' array */
 	void *priv[MAX_CLIENTS];
 
 	struct ism_eq *ieq;
@@ -58,11 +46,6 @@ struct ism_event {
 struct ism_client {
 	const char *name;
 	void (*handle_event)(struct ism_dev *dev, struct ism_event *event);
-	/* Parameter dmbemask contains a bit vector with updated DMBEs, if sent
-	 * via ism_move_data(). Callback function must handle all active bits
-	 * indicated by dmbemask.
-	 */
-	void (*handle_irq)(struct ism_dev *dev, unsigned int bit, u16 dmbemask);
 	/* Private area - don't touch! */
 	u8 id;
 };
@@ -79,12 +62,6 @@ static inline void ism_set_priv(struct ism_dev *dev, struct ism_client *client,
 	dev->priv[client->id] = priv;
 }
 
-int  ism_register_dmb(struct ism_dev *dev, struct ism_dmb *dmb,
-		      struct ism_client *client);
-int  ism_unregister_dmb(struct ism_dev *dev, struct ism_dmb *dmb);
-int  ism_move(struct ism_dev *dev, u64 dmb_tok, unsigned int idx, bool sf,
-	      unsigned int offset, void *data, unsigned int size);
-
 const struct smcd_ops *ism_get_smcd_ops(void);
 
 #endif	/* _ISM_H */
include/net/smc.h +0 -22
index 5bd135fb4d49..8e3debcf7db5 100644
@@ -28,17 +28,6 @@ struct smc_hashinfo {
 };
 
 /* SMCD/ISM device driver interface */
-struct smcd_dmb {
-	u64 dmb_tok;
-	u64 rgid;
-	u32 dmb_len;
-	u32 sba_idx;
-	u32 vlan_valid;
-	u32 vlan_id;
-	void *cpu_addr;
-	dma_addr_t dma_addr;
-};
-
 #define ISM_EVENT_DMB	0
 #define ISM_EVENT_GID	1
 #define ISM_EVENT_SWR	2
@@ -53,25 +42,14 @@ struct smcd_gid {
 };
 
 struct smcd_ops {
-	int (*register_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb,
-			    void *client);
-	int (*unregister_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb);
-	int (*move_data)(struct smcd_dev *dev, u64 dmb_tok, unsigned int idx,
-			 bool sf, unsigned int offset, void *data,
-			 unsigned int size);
-
 	/* optional operations */
 	int (*signal_event)(struct smcd_dev *dev, struct smcd_gid *rgid,
 			    u32 trigger_irq, u32 event_code, u64 info);
-	int (*support_dmb_nocopy)(struct smcd_dev *dev);
-	int (*attach_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb);
-	int (*detach_dmb)(struct smcd_dev *dev, u64 token);
 };
 
 struct smcd_dev {
 	const struct smcd_ops *ops;
 	void *priv;
-	void *client;
 	struct dibs_dev *dibs;
 	struct list_head list;
 	spinlock_t lock;
net/dibs/dibs_loopback.c +257 -0

297 lines not on this page · raw message

net/dibs/dibs_loopback.h +19 -0
index fd03b6333a24..0664f6a8e662 100644
@@ -13,13 +13,32 @@
 #define _DIBS_LOOPBACK_H
 
 #include <linux/dibs.h>
+#include <linux/hashtable.h>
+#include <linux/spinlock.h>
 #include <linux/types.h>
 #include <linux/wait.h>
 
 #if IS_ENABLED(CONFIG_DIBS_LO)
+#define DIBS_LO_DMBS_HASH_BITS	12
+#define DIBS_LO_MAX_DMBS	5000
+
+struct dibs_lo_dmb_node {
+	struct hlist_node list;
+	u64 token;
+	u32 len;
+	u32 sba_idx;
+	void *cpu_addr;
+	dma_addr_t dma_addr;
+	refcount_t refcnt;
+};
 
 struct dibs_lo_dev {
 	struct dibs_dev *dibs;
+	atomic_t dmb_cnt;
+	rwlock_t dmb_ht_lock;
+	DECLARE_BITMAP(sba_idx_mask, DIBS_LO_MAX_DMBS);
+	DECLARE_HASHTABLE(dmb_ht, DIBS_LO_DMBS_HASH_BITS);
+	wait_queue_head_t ldev_release;
 };
 
 int dibs_loopback_init(void);
net/dibs/dibs_main.c +54 -2

121 lines not on this page · raw message

net/smc/Makefile +0 -1
index 96ccfdf246df..0e754cbc38f9 100644
@@ -6,4 +6,3 @@ smc-y := af_smc.o smc_pnet.o smc_ib.o smc_clc.o smc_core.o smc_wr.o smc_llc.o
 smc-y += smc_cdc.o smc_tx.o smc_rx.o smc_close.o smc_ism.o smc_netlink.o smc_stats.o
 smc-y += smc_tracepoint.o smc_inet.o
 smc-$(CONFIG_SYSCTL) += smc_sysctl.o
-smc-y += smc_loopback.o
net/smc/smc_ism.c +29 -37

181 lines not on this page · raw message

net/smc/smc_ism.h +3 -1
index 139e99da2c9f..a1575e31df73 100644
@@ -69,7 +69,9 @@ static inline int smc_ism_write(struct smcd_dev *smcd, u64 dmb_tok,
 {
 	int rc;
 
-	rc = smcd->ops->move_data(smcd, dmb_tok, idx, sf, offset, data, len);
+	rc = smcd->dibs->ops->move_data(smcd->dibs, dmb_tok, idx, sf, offset,
+				       data, len);
+
 	return rc < 0 ? rc : 0;
 }
 
net/smc/smc_loopback.c deleted +0 -294

294 lines not on this page · raw message

net/smc/smc_loopback.h deleted +0 -47

47 lines not on this page · raw message

Signature · 1 line
2.48.1
author maintainer
#18 lore raw

RFC 8/17 net/dibs: Register ism as dibs device

In reply to Alexandra Winter

Register ism devices with the dibs layer. Follow-on patches will move functionality to the dibs layer.

As DIBS is only a shim layer without any dependencies, we can depend ISM on DIBS without adding indirect dependencies. A follow-on patch will remove implication of SMC by ISM.

Define struct dibs_dev. Follow-on patches will move more content into dibs_dev. The goal of follow-on patches is that ism_dev will only contain fields that are special for this device driver. The same concept will apply to other dibs device drivers.

Define dibs_dev_alloc(), dibs_dev_add() and dibs_dev_del() to be called by dibs device drivers and call them from ism_drv.c
Use ism_dev.dibs for a pointer to dibs_dev.

---
 drivers/s390/net/Kconfig   |  2 +-
 drivers/s390/net/ism.h     |  1 +
 drivers/s390/net/ism_drv.c | 83 ++++++++++++++++++++++++--------------
 include/linux/dibs.h       | 38 +++++++++++++++++
 include/linux/ism.h        |  1 +
 net/dibs/dibs_main.c       | 36 +++++++++++++++++
 6 files changed, 129 insertions(+), 32 deletions(-)
drivers/s390/net/Kconfig +1 -1
index 2b43f6f28362..92985f595d59 100644
@@ -81,7 +81,7 @@ config CCWGROUP
 
 config ISM
 	tristate "Support for ISM vPCI Adapter"
-	depends on PCI
+	depends on PCI && DIBS
 	imply SMC
 	default n
 	help
drivers/s390/net/ism.h +1 -0
index b5b03db52fce..3078779fa71e 100644
@@ -5,6 +5,7 @@
 #include <linux/spinlock.h>
 #include <linux/types.h>
 #include <linux/pci.h>
+#include <linux/dibs.h>
 #include <linux/ism.h>
 #include <net/smc.h>
 #include <asm/pci_insn.h>
drivers/s390/net/ism_drv.c +52 -31

113 lines not on this page · raw message

include/linux/dibs.h +38 -0
index 5c432699becb..e9a66cc7f25d 100644
@@ -9,6 +9,7 @@
 #ifndef _DIBS_H
 #define _DIBS_H
 
+#include <linux/device.h>
 /* DIBS - Direct Internal Buffer Sharing - concept
  * -----------------------------------------------
  * In the case of multiple system sharing the same hardware, dibs fabrics can
@@ -61,4 +62,41 @@ int dibs_register_client(struct dibs_client *client);
  */
 int dibs_unregister_client(struct dibs_client *client);
 
+/* DIBS devices
+ * ------------
+ */
+struct dibs_dev {
+	struct list_head list;
+};
+
+/* ------- End of client-only functions ----------- */
+
+/*
+ * Functions to be called by dibs device drivers:
+ */
+/**
+ * dibs_dev_alloc() - allocate and reference device structure
+ *
+ * The following fields will be valid upon successful return: dev
+ * NOTE: Use put_device(dibs_get_dev(@dibs)) to give up your reference instead
+ * of freeing @dibs @dev directly once you have successfully called this
+ * function.
+ * Return: Pointer to dibs device structure
+ */
+struct dibs_dev *dibs_dev_alloc(void);
+/**
+ * dibs_dev_add() - register with dibs layer and all clients
+ * @dibs: dibs device
+ *
+ * The following fields must be valid upon entry: dev, ops, drv_priv
+ * All fields will be valid upon successful return.
+ * Return: zero on success
+ */
+int dibs_dev_add(struct dibs_dev *dibs);
+/**
+ * dibs_dev_del() - unregister from dibs layer and all clients
+ * @dibs: dibs device
+ */
+void dibs_dev_del(struct dibs_dev *dibs);
+
 #endif	/* _DIBS_H */
include/linux/ism.h +1 -0
index 8358b4cd7ba6..9a53d3c48c16 100644
@@ -30,6 +30,7 @@ struct ism_dev {
 	spinlock_t lock; /* protects the ism device */
 	spinlock_t cmd_lock; /* serializes cmds */
 	struct list_head list;
+	struct dibs_dev *dibs;
 	struct pci_dev *pdev;
 
 	struct ism_sba *sba;
net/dibs/dibs_main.c +36 -0
index d91e461f2f06..5345e41ae5e4 100644
@@ -21,6 +21,15 @@ MODULE_LICENSE("GPL");
 static struct dibs_client *clients[MAX_DIBS_CLIENTS];
 static u8 max_client;
 static DEFINE_MUTEX(clients_lock);
+struct dibs_dev_list {
+	struct list_head list;
+	struct mutex mutex; /* protects dibs device list */
+};
+
+static struct dibs_dev_list dibs_dev_list = {
+	.list = LIST_HEAD_INIT(dibs_dev_list.list),
+	.mutex = __MUTEX_INITIALIZER(dibs_dev_list.mutex),
+};
 
 int dibs_register_client(struct dibs_client *client)
 {
@@ -56,6 +65,33 @@ int dibs_unregister_client(struct dibs_client *client)
 }
 EXPORT_SYMBOL_GPL(dibs_unregister_client);
 
+struct dibs_dev *dibs_dev_alloc(void)
+{
+	struct dibs_dev *dibs;
+
+	dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
+	return dibs;
+}
+EXPORT_SYMBOL_GPL(dibs_dev_alloc);
+
+int dibs_dev_add(struct dibs_dev *dibs)
+{
+	mutex_lock(&dibs_dev_list.mutex);
+	list_add(&dibs->list, &dibs_dev_list.list);
+	mutex_unlock(&dibs_dev_list.mutex);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(dibs_dev_add);
+
+void dibs_dev_del(struct dibs_dev *dibs)
+{
+	mutex_lock(&dibs_dev_list.mutex);
+	list_del_init(&dibs->list);
+	mutex_unlock(&dibs_dev_list.mutex);
+}
+EXPORT_SYMBOL_GPL(dibs_dev_del);
+
 static int __init dibs_init(void)
 {
 	memset(clients, 0, sizeof(clients));
Signature · 1 line
2.48.1
maintainer
#2 lore raw

Re: [RFC net-next 01/17] net/smc: Remove __init marker from smc_core_init()

In reply to Alexandra Winter
Quoted 3 lines · Alexandra Winter

Remove the __init marker because smc_core_init() is not the init function of the smc module and for consistency with smc_core_exit() which neither has an __exit marker.

Have you seen a real warning or error because of the __init marker ?

I think the __init marker is just to tell the kernel this function will only be called during initialization. So it doesn't need to be the module's init function.

Best regards,
Dust

Quoted 21 lines

---
net/smc/smc_core.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
net/smc/smc_core.c +1 -1
index 262746e304dd..67f9e0b83ebc 100644
@@ -2758,7 +2758,7 @@ static struct notifier_block smc_reboot_notifier = {
 	.notifier_call = smc_core_reboot_event,
 };
 
-int __init smc_core_init(void)
+int smc_core_init(void)
 {
 	return register_reboot_notifier(&smc_reboot_notifier);
 }
Signature · 1 line
2.48.1
author maintainer
#3 lore raw

Re: [RFC net-next 01/17] net/smc: Remove __init marker from smc_core_init()

Quoted 4 lines · Dust Li
Quoted 3 lines · Alexandra Winter

Remove the __init marker because smc_core_init() is not the init function of the smc module and for consistency with smc_core_exit() which neither has an __exit marker.

Quoted 8 lines

Have you seen a real warning or error because of the __init marker ?

I think the __init marker is just to tell the kernel this function will only be called during initialization. So it doesn't need to be the module's init function.

Best regards,
Dust

My bad. Thank you Dust, for pointing this out.
This patch will be removed from the series.

#19 lore raw

Re: [RFC net-next 08/17] net/dibs: Register ism as dibs device

In reply to Alexandra Winter
kernel test robot noticed the following build errors:

Hi Alexandra,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on net-next/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Alexandra-Winter/net-smc-Remove-__init-marker-from-smc_core_init/20250806-234621
base:   net-next/main
patch link:    https://lore.kernel.org/r/20250806154122.3413330-9-wintera%40linux.ibm.com
patch subject: [RFC net-next 08/17] net/dibs: Register ism as dibs device
config: m68k-allmodconfig (https://download.01.org/0day-ci/archive/20250807/[email protected]/config)
compiler: m68k-linux-gcc (GCC) 15.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250807/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

net/dibs/dibs_main.c: In function 'dibs_dev_alloc':

>> net/dibs/dibs_main.c:72:16: error: implicit declaration of function 'kzalloc' [-Wimplicit-function-declaration]
      72 |         dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
         |                ^~~~~~~
>> net/dibs/dibs_main.c:72:14: error: assignment to 'struct dibs_dev *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
      72 |         dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
         |              ^

vim +/kzalloc +72 net/dibs/dibs_main.c

    67	
    68	struct dibs_dev *dibs_dev_alloc(void)
    69	{
    70		struct dibs_dev *dibs;
    71	
  > 72		dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
    73		return dibs;
    74	}
    75	EXPORT_SYMBOL_GPL(dibs_dev_alloc);
    76	
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#35 lore raw

Re: [RFC net-next 11/17] net/dibs: Move struct device to dibs_dev

In reply to Alexandra Winter
kernel test robot noticed the following build errors:

Hi Alexandra,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on net-next/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Alexandra-Winter/net-smc-Remove-__init-marker-from-smc_core_init/20250806-234621
base:   net-next/main
patch link:    https://lore.kernel.org/r/20250806154122.3413330-12-wintera%40linux.ibm.com
patch subject: [RFC net-next 11/17] net/dibs: Move struct device to dibs_dev
config: m68k-allmodconfig (https://download.01.org/0day-ci/archive/20250808/[email protected]/config)
compiler: m68k-linux-gcc (GCC) 15.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250808/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

net/dibs/dibs_main.c: In function 'dibs_dev_release':

>> net/dibs/dibs_main.c:96:9: error: implicit declaration of function 'kfree' [-Wimplicit-function-declaration]
      96 |         kfree(dibs);
         |         ^~~~~
   net/dibs/dibs_main.c: In function 'dibs_dev_alloc':
   net/dibs/dibs_main.c:103:16: error: implicit declaration of function 'kzalloc' [-Wimplicit-function-declaration]
     103 |         dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
         |                ^~~~~~~
   net/dibs/dibs_main.c:103:14: error: assignment to 'struct dibs_dev *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
     103 |         dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
         |              ^

vim +/kfree +96 net/dibs/dibs_main.c

    89	
    90	static void dibs_dev_release(struct device *dev)
    91	{
    92		struct dibs_dev *dibs;
    93	
    94		dibs = container_of(dev, struct dibs_dev, dev);
    95	
  > 96		kfree(dibs);
    97	}
    98	
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
reviewer
#20 lore raw

Re: [RFC net-next 08/17] net/dibs: Register ism as dibs device

In reply to Alexandra Winter
Quoted 17 lines · Alexandra Winter

Register ism devices with the dibs layer. Follow-on patches will move functionality to the dibs layer.

As DIBS is only a shim layer without any dependencies, we can depend ISM on DIBS without adding indirect dependencies. A follow-on patch will remove implication of SMC by ISM.

Define struct dibs_dev. Follow-on patches will move more content into dibs_dev. The goal of follow-on patches is that ism_dev will only contain fields that are special for this device driver. The same concept will apply to other dibs device drivers.

Define dibs_dev_alloc(), dibs_dev_add() and dibs_dev_del() to be called by dibs device drivers and call them from ism_drv.c
Use ism_dev.dibs for a pointer to dibs_dev.

...

Quoted 1 line
net/dibs/dibs_main.c +0 -0

...

net/dibs/dibs_main.c int dibs_unregister_client(struct dibs_client *client) · line 72

Quoted 9 lines
net/dibs/dibs_main.c +5 -0
@@ -56,6 +65,33 @@ int dibs_unregister_client(struct dibs_client *client)
 }
 EXPORT_SYMBOL_GPL(dibs_unregister_client);
 
+struct dibs_dev *dibs_dev_alloc(void)
+{
+	struct dibs_dev *dibs;
+
+	dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);

Hi Alexandra,

It is not the case for x86_64, arm64, or s390 (at least). But for x86_32 and arm (at least) it seems that linux/slab.h should be included in order for kzalloc to be available for compilation.

Quoted 3 lines
net/dibs/dibs_main.c +3 -0
+	return dibs;
+}
+EXPORT_SYMBOL_GPL(dibs_dev_alloc);

...

#23 lore raw

Re: [RFC net-next 08/17] net/dibs: Register ism as dibs device

In reply to Alexandra Winter
kernel test robot noticed the following build warnings:

Hi Alexandra,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:

[auto build test WARNING on net-next/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Alexandra-Winter/net-smc-Remove-__init-marker-from-smc_core_init/20250806-234621
base:   net-next/main
patch link:    https://lore.kernel.org/r/20250806154122.3413330-9-wintera%40linux.ibm.com
patch subject: [RFC net-next 08/17] net/dibs: Register ism as dibs device
config: um-allyesconfig (https://download.01.org/0day-ci/archive/20250808/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14+deb12u1) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250808/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

net/dibs/dibs_main.c: In function 'dibs_dev_alloc':
net/dibs/dibs_main.c:72:16: error: implicit declaration of function 'kzalloc' [-Werror=implicit-function-declaration]

      72 |         dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
         |                ^~~~~~~
>> net/dibs/dibs_main.c:72:14: warning: assignment to 'struct dibs_dev *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
      72 |         dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
         |              ^

cc1: some warnings being treated as errors

vim +72 net/dibs/dibs_main.c

    67	
    68	struct dibs_dev *dibs_dev_alloc(void)
    69	{
    70		struct dibs_dev *dibs;
    71	
  > 72		dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
    73		return dibs;
    74	}
    75	EXPORT_SYMBOL_GPL(dibs_dev_alloc);
    76	
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#48 lore raw

Re: [RFC net-next 16/17] net/dibs: Move data path to dibs layer

In reply to Alexandra Winter
kernel test robot noticed the following build errors:

Hi Alexandra,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on net-next/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Alexandra-Winter/net-smc-Remove-__init-marker-from-smc_core_init/20250806-234621
base:   net-next/main
patch link:    https://lore.kernel.org/r/20250806154122.3413330-17-wintera%40linux.ibm.com
patch subject: [RFC net-next 16/17] net/dibs: Move data path to dibs layer
config: m68k-allmodconfig (https://download.01.org/0day-ci/archive/20250808/[email protected]/config)
compiler: m68k-linux-gcc (GCC) 15.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250808/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

net/dibs/dibs_main.c: In function 'dibs_dev_release':
net/dibs/dibs_main.c:129:9: error: implicit declaration of function 'kfree' [-Wimplicit-function-declaration]

     129 |         kfree(dibs);
         |         ^~~~~
   net/dibs/dibs_main.c: In function 'dibs_dev_alloc':
   net/dibs/dibs_main.c:136:16: error: implicit declaration of function 'kzalloc' [-Wimplicit-function-declaration]
     136 |         dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
         |                ^~~~~~~
   net/dibs/dibs_main.c:136:14: error: assignment to 'struct dibs_dev *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
     136 |         dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
         |              ^
   net/dibs/dibs_main.c: In function 'dibs_dev_add':
>> net/dibs/dibs_main.c:188:32: error: assignment to 'u8 *' {aka 'unsigned char *'} from 'int' makes pointer from integer without a cast [-Wint-conversion]
     188 |         dibs->dmb_clientid_arr = kzalloc(max_dmbs, GFP_KERNEL);
         |                                ^
--
   net/dibs/dibs_loopback.c: In function 'dibs_lo_register_dmb':
   net/dibs/dibs_loopback.c:64:20: error: implicit declaration of function 'kzalloc' [-Wimplicit-function-declaration]
      64 |         dmb_node = kzalloc(sizeof(*dmb_node), GFP_KERNEL);
         |                    ^~~~~~~
>> net/dibs/dibs_loopback.c:64:18: error: assignment to 'struct dibs_lo_dmb_node *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
      64 |         dmb_node = kzalloc(sizeof(*dmb_node), GFP_KERNEL);
         |                  ^
>> net/dibs/dibs_loopback.c:72:28: error: assignment to 'void *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
      72 |         dmb_node->cpu_addr = kzalloc(dmb_node->len, GFP_KERNEL |
         |                            ^
   net/dibs/dibs_loopback.c:109:9: error: implicit declaration of function 'kfree' [-Wimplicit-function-declaration]
     109 |         kfree(dmb_node);
         |         ^~~~~
   net/dibs/dibs_loopback.c: In function '__dibs_lo_unregister_dmb':
>> net/dibs/dibs_loopback.c:124:9: error: implicit declaration of function 'kvfree' [-Wimplicit-function-declaration]
     124 |         kvfree(dmb_node->cpu_addr);
         |         ^~~~~~
   net/dibs/dibs_loopback.c: In function 'dibs_lo_dev_probe':
   net/dibs/dibs_loopback.c:300:14: error: assignment to 'struct dibs_lo_dev *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
     300 |         ldev = kzalloc(sizeof(*ldev), GFP_KERNEL);
         |              ^

vim +188 net/dibs/dibs_main.c

131
132 struct dibs_dev *dibs_dev_alloc(void)
133 {
134 struct dibs_dev *dibs;
135

 > 136		dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
   137		if (!dibs)
   138			return dibs;
   139		dibs->dev.release = dibs_dev_release;
   140		dibs->dev.class = dibs_class;
   141		device_initialize(&dibs->dev);
   142	
   143		return dibs;
   144	}
   145	EXPORT_SYMBOL_GPL(dibs_dev_alloc);
   146	
   147	static ssize_t gid_show(struct device *dev, struct device_attribute *attr,
   148				char *buf)
   149	{
   150		struct dibs_dev *dibs;
   151	
   152		dibs = container_of(dev, struct dibs_dev, dev);
   153	
   154		return sysfs_emit(buf, "%pUb\n", &dibs->gid);
   155	}
   156	static DEVICE_ATTR_RO(gid);
   157	
   158	static ssize_t fabric_id_show(struct device *dev, struct device_attribute *attr,
   159				      char *buf)
   160	{
   161		struct dibs_dev *dibs;
   162		u16 fabric_id;
   163	
   164		dibs = container_of(dev, struct dibs_dev, dev);
   165		fabric_id = dibs->ops->get_fabric_id(dibs);
   166	
   167		return sysfs_emit(buf, "0x%04x\n", fabric_id);
   168	}
   169	static DEVICE_ATTR_RO(fabric_id);
   170	
   171	static struct attribute *dibs_dev_attrs[] = {
   172		&dev_attr_gid.attr,
   173		&dev_attr_fabric_id.attr,
   174		NULL,
   175	};
   176	
   177	static const struct attribute_group dibs_dev_attr_group = {
   178		.attrs = dibs_dev_attrs,
   179	};
   180	
   181	int dibs_dev_add(struct dibs_dev *dibs)
   182	{
   183		int max_dmbs;
   184		int i, ret;
   185	
   186		max_dmbs = dibs->ops->max_dmbs();
   187		spin_lock_init(&dibs->lock);
 > 188		dibs->dmb_clientid_arr = kzalloc(max_dmbs, GFP_KERNEL);

189 if (!dibs->dmb_clientid_arr)
190 return -ENOMEM;
191 memset(dibs->dmb_clientid_arr, NO_DIBS_CLIENT, max_dmbs);
192
193 ret = device_add(&dibs->dev);
194 if (ret)
195 goto free_client_arr;
196
197 ret = sysfs_create_group(&dibs->dev.kobj, &dibs_dev_attr_group);
198 if (ret) {
199 dev_err(&dibs->dev, "sysfs_create_group failed for dibs_dev\n");
200 goto err_device_del;
201 }
202 mutex_lock(&dibs_dev_list.mutex);
203 mutex_lock(&clients_lock);
204 for (i = 0; i < max_client; ++i) {
205 if (clients[i]) {
206 clients[i]->ops->add_dev(dibs);
207 dibs_setup_forwarding(clients[i], dibs);
208 }
209 }
210 mutex_unlock(&clients_lock);
211 list_add(&dibs->list, &dibs_dev_list.list);
212 mutex_unlock(&dibs_dev_list.mutex);
213
214 return 0;
215
216 err_device_del:
217 device_del(&dibs->dev);
218 free_client_arr:
219 kfree(dibs->dmb_clientid_arr);
220 return ret;
221

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
reviewer
#21 lore raw

Re: [RFC net-next 08/17] net/dibs: Register ism as dibs device

In reply to Simon Horman
Quoted 37 lines · Simon Horman
Quoted 17 lines · Alexandra Winter

Register ism devices with the dibs layer. Follow-on patches will move functionality to the dibs layer.

As DIBS is only a shim layer without any dependencies, we can depend ISM on DIBS without adding indirect dependencies. A follow-on patch will remove implication of SMC by ISM.

Define struct dibs_dev. Follow-on patches will move more content into dibs_dev. The goal of follow-on patches is that ism_dev will only contain fields that are special for this device driver. The same concept will apply to other dibs device drivers.

Define dibs_dev_alloc(), dibs_dev_add() and dibs_dev_del() to be called by dibs device drivers and call them from ism_drv.c
Use ism_dev.dibs for a pointer to dibs_dev.

...

Quoted 1 line
net/dibs/dibs_main.c +0 -0

...

net/dibs/dibs_main.c int dibs_unregister_client(struct dibs_client *client) · line 72

Quoted 9 lines
net/dibs/dibs_main.c +5 -0
@@ -56,6 +65,33 @@ int dibs_unregister_client(struct dibs_client *client)
 }
 EXPORT_SYMBOL_GPL(dibs_unregister_client);
 
+struct dibs_dev *dibs_dev_alloc(void)
+{
+	struct dibs_dev *dibs;
+
+	dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);

Hi Alexandra,

Quoted 3 lines

It is not the case for x86_64, arm64, or s390 (at least). But for x86_32 and arm (at least) it seems that linux/slab.h should be included in order for kzalloc to be available for compilation.

Similarly for dibs_loopback.c in the following patch.

reviewer
#29 lore raw

Re: [RFC net-next 10/17] net/dibs: Define dibs_client_ops and dibs_dev_ops

In reply to Alexandra Winter

On Wed, Aug 06, 2025 at 05:41:15PM +0200, Alexandra Winter wrote:

...

Quoted 1 line
net/smc/smc_ism.c +0 -0

...

Quoted 18 lines
net/smc/smc_ism.c +8 -6
-static void smcd_register_dev(struct ism_dev *ism)
+static void smcd_register_dev(struct dibs_dev *dibs)
 {
-	const struct smcd_ops *ops = ism_get_smcd_ops();
 	struct smcd_dev *smcd, *fentry;
+	const struct smcd_ops *ops;
+	struct smc_lo_dev *smc_lo;
+	struct ism_dev *ism;
 
-	if (!ops)
-		return;
+	if (smc_ism_is_loopback(dibs)) {
+		if (smc_loopback_init(&smc_lo))
+			return;
+	}
 
-	smcd = smcd_alloc_dev(&ism->pdev->dev, dev_name(&ism->pdev->dev), ops,
-			      ISM_NR_DMBS);

net/smc/smc_ism.c static struct smcd_dev *smcd_alloc_dev(struct device *parent, const char *name, · line 519

Quoted 10 lines
net/smc/smc_ism.c +10 -0
+	if (smc_ism_is_loopback(dibs)) {
+		ops = smc_lo_get_smcd_ops();
+		smcd = smcd_alloc_dev(dev_name(&smc_lo->dev), ops,
+				      SMC_LO_MAX_DMBS);
+	} else {
+		ism = dibs->drv_priv;
+		ops = ism_get_smcd_ops();
+		smcd = smcd_alloc_dev(dev_name(&ism->pdev->dev), ops,
+				      ISM_NR_DMBS);
+	}

Hi Alexandra,

ism is initialised conditionally here.

But towards the end of this function the following dereferences ism unconditionally. And it's not clear to me this won't occur even if ism wasn't initialised above.

        if (smc_pnet_is_pnetid_set(smcd->pnetid))
                pr_warn_ratelimited("smc: adding smcd device %s with pnetid %.16s%s\n",
                                    dev_name(&ism->dev), smcd->pnetid,
                                    smcd->pnetid_by_user ?
                                        " (user defined)" :
                                        "");
        else
                pr_warn_ratelimited("smc: adding smcd device %s without pnetid\n",
                                    dev_name(&ism->dev));

net/smc/smc_ism.c static struct smcd_dev *smcd_alloc_dev(struct device *parent, const char *name, · line 531

Quoted 13 lines
net/smc/smc_ism.c +10 -1
 	if (!smcd)
 		return;
-	smcd->priv = ism;
+
+	smcd->dibs = dibs;
+	dibs_set_priv(dibs, &smc_dibs_client, smcd);
+
+	if (smc_ism_is_loopback(dibs)) {
+		smcd->priv = smc_lo;
+		smc_lo->smcd = smcd;
+	} else {
+		smcd->priv = ism;
+		ism_set_priv(ism, &smc_ism_client, smcd);

This function is now compiled even if CONFIG_ISM is not enabled. But smc_ism_client is only defined if CONFIG_ISM is enabled.

I think this code is removed by later patches. But nonetheless I also think this leads to a build error and it's best to avoid transient build errors as they break bisection.

net/smc/smc_ism.c static struct smcd_dev *smcd_alloc_dev(struct device *parent, const char *name, · line 536

Quoted 5 lines
net/smc/smc_ism.c +4 -0
+		if (smc_pnetid_by_dev_port(&ism->pdev->dev, 0, smcd->pnetid))
+			smc_pnetid_by_table_smcd(smcd);
+	}
+
 	smcd->client = &smc_ism_client;

Ditto.

Quoted 6 lines
net/smc/smc_ism.c +0 -3
-	ism_set_priv(ism, &smc_ism_client, smcd);
-	if (smc_pnetid_by_dev_port(&ism->pdev->dev, 0, smcd->pnetid))
-		smc_pnetid_by_table_smcd(smcd);
 
 	if (smcd->ops->supports_v2())
 		smc_ism_set_v2_capable();

...

reviewer
#49 lore raw

Re: [RFC net-next 16/17] net/dibs: Move data path to dibs layer

In reply to Alexandra Winter

On Wed, Aug 06, 2025 at 05:41:21PM +0200, Alexandra Winter wrote:

...

Quoted 1 line
net/smc/smc_ism.c +0 -0

...

net/smc/smc_ism.c static void smcd_register_dev(struct dibs_dev *dibs); · line 36

Quoted 7 lines
net/smc/smc_ism.c +1 -1
@@ -33,18 +32,18 @@ static void smcd_register_dev(struct dibs_dev *dibs);
 static void smcd_unregister_dev(struct dibs_dev *dibs);
 #if IS_ENABLED(CONFIG_ISM)
 static void smcd_handle_event(struct ism_dev *ism, struct ism_event *event);
-static void smcd_handle_irq(struct ism_dev *ism, unsigned int dmbno,
+static void smcd_handle_irq(struct dibs_dev *dibs, unsigned int dmbno,
 			    u16 dmbemask);

Hi Alexandria,

smcd_handle_irq is only declared (and defined) if CONFIG_ISM is enabled.

net/smc/smc_ism.c static void smcd_register_dev(struct dibs_dev *dibs); · line 46

Quoted 11 lines
net/smc/smc_ism.c +1 -1
 static struct ism_client smc_ism_client = {
 	.name = "SMC-D",
 	.handle_event = smcd_handle_event,
-	.handle_irq = smcd_handle_irq,
 };
 #endif
 static struct dibs_client_ops smc_client_ops = {
 	.add_dev = smcd_register_dev,
 	.del_dev = smcd_unregister_dev,
+	.handle_irq = smcd_handle_irq,

But here smcd_handle_irq is used regardless of if CONFIG_ISM is enabled.

I believe this is addressed by the following patch in this series. However, this does result ina transient build failure. And they should be avoided as they break bisection.

Quoted 3 lines
net/smc/smc_ism.c +0 -0
 };
 
 static struct dibs_client smc_dibs_client = {

...

author maintainer
#22 lore raw

Re: [RFC net-next 08/17] net/dibs: Register ism as dibs device

In reply to Simon Horman
Quoted 38 lines · Simon Horman
Quoted 37 lines · Simon Horman
Quoted 17 lines · Alexandra Winter

Register ism devices with the dibs layer. Follow-on patches will move functionality to the dibs layer.

As DIBS is only a shim layer without any dependencies, we can depend ISM on DIBS without adding indirect dependencies. A follow-on patch will remove implication of SMC by ISM.

Define struct dibs_dev. Follow-on patches will move more content into dibs_dev. The goal of follow-on patches is that ism_dev will only contain fields that are special for this device driver. The same concept will apply to other dibs device drivers.

Define dibs_dev_alloc(), dibs_dev_add() and dibs_dev_del() to be called by dibs device drivers and call them from ism_drv.c
Use ism_dev.dibs for a pointer to dibs_dev.

...

Quoted 1 line
net/dibs/dibs_main.c +0 -0

...

net/dibs/dibs_main.c int dibs_unregister_client(struct dibs_client *client) · line 72

Quoted 9 lines
net/dibs/dibs_main.c +5 -0
@@ -56,6 +65,33 @@ int dibs_unregister_client(struct dibs_client *client)
 }
 EXPORT_SYMBOL_GPL(dibs_unregister_client);
 
+struct dibs_dev *dibs_dev_alloc(void)
+{
+	struct dibs_dev *dibs;
+
+	dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);

Hi Alexandra,

Quoted 5 lines
Quoted 3 lines

It is not the case for x86_64, arm64, or s390 (at least). But for x86_32 and arm (at least) it seems that linux/slab.h should be included in order for kzalloc to be available for compilation.

Similarly for dibs_loopback.c in the following patch.

Thank you Simon. It will be included in next version.
(I also got notified by kernel test robot <[email protected]> about this).

author maintainer
#30 lore raw

Re: [RFC net-next 10/17] net/dibs: Define dibs_client_ops and dibs_dev_ops

In reply to Simon Horman
Quoted 39 lines · Simon Horman

On Wed, Aug 06, 2025 at 05:41:15PM +0200, Alexandra Winter wrote:

...

Quoted 1 line
net/smc/smc_ism.c +0 -0

...

Quoted 18 lines
net/smc/smc_ism.c +8 -6
-static void smcd_register_dev(struct ism_dev *ism)
+static void smcd_register_dev(struct dibs_dev *dibs)
 {
-	const struct smcd_ops *ops = ism_get_smcd_ops();
 	struct smcd_dev *smcd, *fentry;
+	const struct smcd_ops *ops;
+	struct smc_lo_dev *smc_lo;
+	struct ism_dev *ism;
 
-	if (!ops)
-		return;
+	if (smc_ism_is_loopback(dibs)) {
+		if (smc_loopback_init(&smc_lo))
+			return;
+	}
 
-	smcd = smcd_alloc_dev(&ism->pdev->dev, dev_name(&ism->pdev->dev), ops,
-			      ISM_NR_DMBS);

net/smc/smc_ism.c static struct smcd_dev *smcd_alloc_dev(struct device *parent, const char *name, · line 519

Quoted 10 lines
net/smc/smc_ism.c +10 -0
+	if (smc_ism_is_loopback(dibs)) {
+		ops = smc_lo_get_smcd_ops();
+		smcd = smcd_alloc_dev(dev_name(&smc_lo->dev), ops,
+				      SMC_LO_MAX_DMBS);
+	} else {
+		ism = dibs->drv_priv;
+		ops = ism_get_smcd_ops();
+		smcd = smcd_alloc_dev(dev_name(&ism->pdev->dev), ops,
+				      ISM_NR_DMBS);
+	}

Hi Alexandra,

Quoted 6 lines

ism is initialised conditionally here.

But towards the end of this function the following dereferences ism unconditionally. And it's not clear to me this won't occur even if ism wasn't initialised above.

No idea, how this could escape my testing. Thx, fixed in next version.

Quoted 28 lines
        if (smc_pnet_is_pnetid_set(smcd->pnetid))
                pr_warn_ratelimited("smc: adding smcd device %s with pnetid %.16s%s\n",
                                    dev_name(&ism->dev), smcd->pnetid,
                                    smcd->pnetid_by_user ?
                                        " (user defined)" :
                                        "");
        else
                pr_warn_ratelimited("smc: adding smcd device %s without pnetid\n",
                                    dev_name(&ism->dev));

net/smc/smc_ism.c static struct smcd_dev *smcd_alloc_dev(struct device *parent, const char *name, · line 531

Quoted 13 lines
net/smc/smc_ism.c +10 -1
 	if (!smcd)
 		return;
-	smcd->priv = ism;
+
+	smcd->dibs = dibs;
+	dibs_set_priv(dibs, &smc_dibs_client, smcd);
+
+	if (smc_ism_is_loopback(dibs)) {
+		smcd->priv = smc_lo;
+		smc_lo->smcd = smcd;
+	} else {
+		smcd->priv = ism;
+		ism_set_priv(ism, &smc_ism_client, smcd);

This function is now compiled even if CONFIG_ISM is not enabled. But smc_ism_client is only defined if CONFIG_ISM is enabled.

Quoted 4 lines

I think this code is removed by later patches. But nonetheless I also think this leads to a build error and it's best to avoid transient build errors as they break bisection.

Fixed in next version.
I'll make sure, all patches build without ISM. Great catch, thank you Simon.

Quoted 16 lines

net/smc/smc_ism.c static struct smcd_dev *smcd_alloc_dev(struct device *parent, const char *name, · line 536

Quoted 5 lines
net/smc/smc_ism.c +4 -0
+		if (smc_pnetid_by_dev_port(&ism->pdev->dev, 0, smcd->pnetid))
+			smc_pnetid_by_table_smcd(smcd);
+	}
+
 	smcd->client = &smc_ism_client;

Ditto.

Quoted 6 lines
net/smc/smc_ism.c +0 -3
-	ism_set_priv(ism, &smc_ism_client, smcd);
-	if (smc_pnetid_by_dev_port(&ism->pdev->dev, 0, smcd->pnetid))
-		smc_pnetid_by_table_smcd(smcd);
 
 	if (smcd->ops->supports_v2())
 		smc_ism_set_v2_capable();

...

author maintainer
#50 lore raw

Re: [RFC net-next 16/17] net/dibs: Move data path to dibs layer

In reply to Simon Horman
Quoted 38 lines · Simon Horman

On Wed, Aug 06, 2025 at 05:41:21PM +0200, Alexandra Winter wrote:

...

Quoted 1 line
net/smc/smc_ism.c +0 -0

...

net/smc/smc_ism.c static void smcd_register_dev(struct dibs_dev *dibs); · line 36

Quoted 7 lines
net/smc/smc_ism.c +1 -1
@@ -33,18 +32,18 @@ static void smcd_register_dev(struct dibs_dev *dibs);
 static void smcd_unregister_dev(struct dibs_dev *dibs);
 #if IS_ENABLED(CONFIG_ISM)
 static void smcd_handle_event(struct ism_dev *ism, struct ism_event *event);
-static void smcd_handle_irq(struct ism_dev *ism, unsigned int dmbno,
+static void smcd_handle_irq(struct dibs_dev *dibs, unsigned int dmbno,
 			    u16 dmbemask);

Hi Alexandria,

smcd_handle_irq is only declared (and defined) if CONFIG_ISM is enabled.

net/smc/smc_ism.c static void smcd_register_dev(struct dibs_dev *dibs); · line 46

Quoted 11 lines
net/smc/smc_ism.c +1 -1
 static struct ism_client smc_ism_client = {
 	.name = "SMC-D",
 	.handle_event = smcd_handle_event,
-	.handle_irq = smcd_handle_irq,
 };
 #endif
 static struct dibs_client_ops smc_client_ops = {
 	.add_dev = smcd_register_dev,
 	.del_dev = smcd_unregister_dev,
+	.handle_irq = smcd_handle_irq,

But here smcd_handle_irq is used regardless of if CONFIG_ISM is enabled.

I believe this is addressed by the following patch in this series. However, this does result ina transient build failure. And they should be avoided as they break bisection.

Quoted 5 lines
Quoted 3 lines
net/smc/smc_ism.c +0 -0
 };
 
 static struct dibs_client smc_dibs_client = {

...

Fixed in next version.
I'll make sure, all patches build without ISM.
Thank you Simon.

maintainer
#6 lore raw

Re: [RFC net-next 03/17] net/smc: Remove error handling of unregister_dmb()

In reply to Alexandra Winter
Quoted 65 lines · Alexandra Winter

smcd_buf_free() calls smc_ism_unregister_dmb(lgr->smcd, buf_desc) and then unconditionally frees buf_desc.

Remove the cleaning up of fields of buf_desc in smc_ism_unregister_dmb(), because it is not helpful.

This removes the only usage of ISM_ERROR from the smc module. So move it to drivers/s390/net/ism.h.

---

drivers/s390/net/ism.h |  1 +
include/net/smc.h      |  2 --
net/smc/smc_ism.c      | 14 +++++---------
net/smc/smc_ism.h      |  3 ++-
4 files changed, 8 insertions(+), 12 deletions(-)
drivers/s390/net/ism.h +1 -0
index 047fa6101555..b5b03db52fce 100644
@@ -10,6 +10,7 @@
 #include <asm/pci_insn.h>
 
 #define UTIL_STR_LEN	16
+#define ISM_ERROR	0xFFFF
 
 /*
 * Do not use the first word of the DMB bits to ensure 8 byte aligned access.
include/net/smc.h +0 -2
index db84e4e35080..a9c023dd1380 100644
@@ -44,8 +44,6 @@ struct smcd_dmb {
 
 #define ISM_RESERVED_VLANID	0x1FFF
 
-#define ISM_ERROR	0xFFFF
-
 struct smcd_dev;
 
 struct smcd_gid {
net/smc/smc_ism.c +3 -3
index 84f98e18c7db..a94e1450d095 100644
@@ -205,13 +205,13 @@ int smc_ism_put_vlan(struct smcd_dev *smcd, unsigned short vlanid)
 	return rc;
 }
 
-int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc)
+void smc_ism_unregister_dmb(struct smcd_dev *smcd,
+			    struct smc_buf_desc *dmb_desc)
 {
 	struct smcd_dmb dmb;
-	int rc = 0;
 
 	if (!dmb_desc->dma_addr)
-		return rc;
+		return;
 
 	memset(&dmb, 0, sizeof(dmb));
 	dmb.dmb_tok = dmb_desc->token;

net/smc/smc_ism.c int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc) · line 222

Quoted 9 lines
net/smc/smc_ism.c +1 -5
@@ -219,13 +219,9 @@ int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc)
 	dmb.cpu_addr = dmb_desc->cpu_addr;
 	dmb.dma_addr = dmb_desc->dma_addr;
 	dmb.dmb_len = dmb_desc->len;
-	rc = smcd->ops->unregister_dmb(smcd, &dmb);
-	if (!rc || rc == ISM_ERROR) {
-		dmb_desc->cpu_addr = NULL;
-		dmb_desc->dma_addr = 0;
-	}
+	smcd->ops->unregister_dmb(smcd, &dmb);

Hmm, I think the old way of handling error here is certainly not good. But completely ignoring error handling here would make bugs harder to detect.

What about adding a WARN_ON_ONCE(rc) ?

Also, I think we can just remove the rc == ISM_ERROR to remove the dependency of ISM_ERROR in smc.

Best regards,
Dust

maintainer Reviewed-by on v1 4/17
#10 lore raw

Re: [RFC net-next 04/17] net/smc: Decouple sf and attached send_buf in smc_loopback

In reply to Alexandra Winter
Quoted 41 lines · Alexandra Winter

Before this patch there was the following assumption in smc_loopback.c>smc_lo_move_data():
sf (signalling flag) == 0 : data is already in an attached target dmb sf == 1 : data is not yet in the target dmb

This is true for the 2 callers in smc client
smcd_cdc_msg_send() : sf=1
smcd_tx_rdma_writes() : sf=0
but should not be a general assumption.

Add a bool to struct smc_buf_desc to indicate whether an SMC-D sndbuf_desc is an attached buffer. Don't call move_data() for attached send_buffers, because it is not necessary.

Move the data in smc_lo_move_data() if len != 0 and signal when requested.

---

net/smc/smc_core.h     | 5 +++++
net/smc/smc_ism.c      | 1 +
net/smc/smc_loopback.c | 9 +++------
net/smc/smc_tx.c       | 3 +++
4 files changed, 12 insertions(+), 6 deletions(-)
net/smc/smc_core.h +1 -0
index 48a1b1dcb576..fe5f48d14323 100644
@@ -13,6 +13,7 @@
 #define _SMC_CORE_H
 
 #include <linux/atomic.h>
+#include <linux/types.h>
 #include <linux/smc.h>
 #include <linux/pci.h>
 #include <rdma/ib_verbs.h>
@@ -221,12 +222,16 @@ struct smc_buf_desc {
 					/* virtually contiguous */
 		};
 		struct { /* SMC-D */

net/smc/smc_core.h struct smc_buf_desc { · line 234

Quoted 10 lines
net/smc/smc_core.h +4 -0
+			 /* SMC-D rx buffer: */
 			unsigned short	sba_idx;
 					/* SBA index number */
 			u64		token;
 					/* DMB token number */
 			dma_addr_t	dma_addr;
 					/* DMA address */
+			/* SMC-D tx buffer */
+			bool		is_attached;
+					/* no need for explicit writes */

A small sugguestion: there is a hole between sba_idx and token, we can put is_attached in that hole.
Not a big deal because this is a union and SMC-R use a much large space.

Best regards,
Dust

maintainer Reviewed-by on v1 5/17
#14 lore raw

Re: [RFC net-next 05/17] net/smc: Improve log message for devices w/o pnetid

In reply to Alexandra Winter
Quoted 4 lines · Alexandra Winter

Explicitly state in the log message, when a device has no pnetid. "with pnetid" and "has pnetid" was misleading for devices without pnetid.

This patch doesn't seem strongly related to this patchset, so it might be better to send this patch separately.

Best regards,
Dust

Quoted 60 lines

---
net/smc/smc_ib.c | 18 +++++++++++-------
net/smc/smc_ism.c | 13 +++++++++----

2 files changed, 20 insertions(+), 11 deletions(-)
net/smc/smc_ib.c +11 -7
index 53828833a3f7..f2de12990b5b 100644
@@ -971,13 +971,17 @@ static int smc_ib_add_dev(struct ib_device *ibdev)
 					   smcibdev->pnetid[i]))
 			smc_pnetid_by_table_ib(smcibdev, i + 1);
 		smc_copy_netdev_ifindex(smcibdev, i);
-		pr_warn_ratelimited("smc:    ib device %s port %d has pnetid "
-				    "%.16s%s\n",
-				    smcibdev->ibdev->name, i + 1,
-				    smcibdev->pnetid[i],
-				    smcibdev->pnetid_by_user[i] ?
-				     " (user defined)" :
-				     "");
+		if (smc_pnet_is_pnetid_set(smcibdev->pnetid[i]))
+			pr_warn_ratelimited("smc:    ib device %s port %d has pnetid %.16s%s\n",
+					    smcibdev->ibdev->name, i + 1,
+					    smcibdev->pnetid[i],
+					    smcibdev->pnetid_by_user[i] ?
+						" (user defined)" :
+						"");
+		else
+			pr_warn_ratelimited("smc:    ib device %s port %d has no pnetid\n",
+					    smcibdev->ibdev->name, i + 1);
+
 	}
 	schedule_work(&smcibdev->port_event_work);
 	return 0;
net/smc/smc_ism.c +9 -4
index 7363f8be9f94..503a9f93b392 100644
@@ -515,10 +515,15 @@ static void smcd_register_dev(struct ism_dev *ism)
 	}
 	mutex_unlock(&smcd_dev_list.mutex);
 
-	pr_warn_ratelimited("smc: adding smcd device %s with pnetid %.16s%s\n",
-			    dev_name(&ism->dev), smcd->pnetid,
-			    smcd->pnetid_by_user ? " (user defined)" : "");
-
+	if (smc_pnet_is_pnetid_set(smcd->pnetid))
+		pr_warn_ratelimited("smc: adding smcd device %s with pnetid %.16s%s\n",
+				    dev_name(&ism->dev), smcd->pnetid,
+				    smcd->pnetid_by_user ?
+					" (user defined)" :
+					"");
+	else
+		pr_warn_ratelimited("smc: adding smcd device %s without pnetid\n",
+				    dev_name(&ism->dev));
 	return;
 }
 
Signature · 1 line
2.48.1
maintainer
#24 lore raw

Re: [RFC net-next 08/17] net/dibs: Register ism as dibs device

In reply to Alexandra Winter
Quoted 7 lines · Alexandra Winter

Register ism devices with the dibs layer. Follow-on patches will move functionality to the dibs layer.

As DIBS is only a shim layer without any dependencies, we can depend ISM on DIBS without adding indirect dependencies. A follow-on patch will remove implication of SMC by ISM.

Quoted 8 lines

Define struct dibs_dev. Follow-on patches will move more content into dibs_dev. The goal of follow-on patches is that ism_dev will only contain fields that are special for this device driver. The same concept will apply to other dibs device drivers.

Define dibs_dev_alloc(), dibs_dev_add() and dibs_dev_del() to be called by dibs device drivers and call them from ism_drv.c
Use ism_dev.dibs for a pointer to dibs_dev.

I've been wondering whether we should completely remove the ISM concept from SMC. Including rename smc_ism.c into smc_dibs.c.

Since DIBS already serves as the replacement for ISM, having both ISM and DIBS coexist in the codebase seems a bit confusing and inconsistent. Removing ISM could help streamline the code and improve clarity.

Best regards,
Dust

Quoted 278 lines

---

drivers/s390/net/Kconfig   |  2 +-
drivers/s390/net/ism.h     |  1 +
drivers/s390/net/ism_drv.c | 83 ++++++++++++++++++++++++--------------
include/linux/dibs.h       | 38 +++++++++++++++++
include/linux/ism.h        |  1 +
net/dibs/dibs_main.c       | 36 +++++++++++++++++
6 files changed, 129 insertions(+), 32 deletions(-)
drivers/s390/net/Kconfig +0 -0
index 2b43f6f28362..92985f595d59 100644
@@ -81,7 +81,7 @@ config CCWGROUP
 

config ISM
tristate "Support for ISM vPCI Adapter"

drivers/s390/net/Kconfig +1 -1
-	depends on PCI
+	depends on PCI && DIBS

imply SMC
default n
help

drivers/s390/net/ism.h +1 -0
index b5b03db52fce..3078779fa71e 100644
@@ -5,6 +5,7 @@
 #include <linux/spinlock.h>
 #include <linux/types.h>
 #include <linux/pci.h>
+#include <linux/dibs.h>
 #include <linux/ism.h>
 #include <net/smc.h>
 #include <asm/pci_insn.h>
drivers/s390/net/ism_drv.c +48 -1
index a543e59818bb..7948334f8d30 100644
@@ -599,8 +599,39 @@ static void ism_dev_release(struct device *dev)
 	kfree(ism);
 }
 
+static void ism_dev_exit(struct ism_dev *ism)
+{
+	struct pci_dev *pdev = ism->pdev;
+	unsigned long flags;
+	int i;
+
+	spin_lock_irqsave(&ism->lock, flags);
+	for (i = 0; i < max_client; ++i)
+		ism->subs[i] = NULL;
+	spin_unlock_irqrestore(&ism->lock, flags);
+
+	mutex_lock(&ism_dev_list.mutex);
+	mutex_lock(&clients_lock);
+	for (i = 0; i < max_client; ++i) {
+		if (clients[i])
+			clients[i]->remove(ism);
+	}
+	mutex_unlock(&clients_lock);
+
+	if (ism_v2_capable)
+		ism_del_vlan_id(ism, ISM_RESERVED_VLANID);
+	unregister_ieq(ism);
+	unregister_sba(ism);
+	free_irq(pci_irq_vector(pdev, 0), ism);
+	kfree(ism->sba_client_arr);
+	pci_free_irq_vectors(pdev);
+	list_del_init(&ism->list);
+	mutex_unlock(&ism_dev_list.mutex);
+}
+
 static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
+	struct dibs_dev *dibs;
 	struct ism_dev *ism;
 	int ret;
 
@@ -636,12 +667,28 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	dma_set_max_seg_size(&pdev->dev, SZ_1M);
 	pci_set_master(pdev);
 
+	dibs = dibs_dev_alloc();
+	if (!dibs) {
+		ret = -ENOMEM;
+		goto err_resource;
+	}
+	ism->dibs = dibs;
+
 	ret = ism_dev_init(ism);
 	if (ret)
-		goto err_resource;
+		goto err_dibs;
+
+	ret = dibs_dev_add(dibs);
+	if (ret)
+		goto err_ism;
 
 	return 0;
 
+err_ism:
+	ism_dev_exit(ism);
+err_dibs:
+	/* pairs with dibs_dev_alloc() */
+	kfree(dibs);

err_resource:

	pci_release_mem_regions(pdev);

err_disable:

drivers/s390/net/ism_drv.c +4 -30
@@ -655,41 +702,15 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	return ret;
 }
 
-static void ism_dev_exit(struct ism_dev *ism)
-{
-	struct pci_dev *pdev = ism->pdev;
-	unsigned long flags;
-	int i;
-
-	spin_lock_irqsave(&ism->lock, flags);
-	for (i = 0; i < max_client; ++i)
-		ism->subs[i] = NULL;
-	spin_unlock_irqrestore(&ism->lock, flags);
-
-	mutex_lock(&ism_dev_list.mutex);
-	mutex_lock(&clients_lock);
-	for (i = 0; i < max_client; ++i) {
-		if (clients[i])
-			clients[i]->remove(ism);
-	}
-	mutex_unlock(&clients_lock);
-
-	if (ism_v2_capable)
-		ism_del_vlan_id(ism, ISM_RESERVED_VLANID);
-	unregister_ieq(ism);
-	unregister_sba(ism);
-	free_irq(pci_irq_vector(pdev, 0), ism);
-	kfree(ism->sba_client_arr);
-	pci_free_irq_vectors(pdev);
-	list_del_init(&ism->list);
-	mutex_unlock(&ism_dev_list.mutex);
-}
-
 static void ism_remove(struct pci_dev *pdev)
 {
 	struct ism_dev *ism = dev_get_drvdata(&pdev->dev);
+	struct dibs_dev *dibs = ism->dibs;
 
+	dibs_dev_del(dibs);
 	ism_dev_exit(ism);
+	/* pairs with dibs_dev_alloc() */
+	kfree(dibs);
 
 	pci_release_mem_regions(pdev);
 	pci_disable_device(pdev);
include/linux/dibs.h +38 -0
index 5c432699becb..e9a66cc7f25d 100644
@@ -9,6 +9,7 @@
 #ifndef _DIBS_H
 #define _DIBS_H
 
+#include <linux/device.h>
 /* DIBS - Direct Internal Buffer Sharing - concept
 * -----------------------------------------------
 * In the case of multiple system sharing the same hardware, dibs fabrics can
@@ -61,4 +62,41 @@ int dibs_register_client(struct dibs_client *client);
 */
 int dibs_unregister_client(struct dibs_client *client);
 
+/* DIBS devices
+ * ------------
+ */
+struct dibs_dev {
+	struct list_head list;
+};
+
+/* ------- End of client-only functions ----------- */
+
+/*
+ * Functions to be called by dibs device drivers:
+ */
+/**
+ * dibs_dev_alloc() - allocate and reference device structure
+ *
+ * The following fields will be valid upon successful return: dev
+ * NOTE: Use put_device(dibs_get_dev(@dibs)) to give up your reference instead
+ * of freeing @dibs @dev directly once you have successfully called this
+ * function.
+ * Return: Pointer to dibs device structure
+ */
+struct dibs_dev *dibs_dev_alloc(void);
+/**
+ * dibs_dev_add() - register with dibs layer and all clients
+ * @dibs: dibs device
+ *
+ * The following fields must be valid upon entry: dev, ops, drv_priv
+ * All fields will be valid upon successful return.
+ * Return: zero on success
+ */
+int dibs_dev_add(struct dibs_dev *dibs);
+/**
+ * dibs_dev_del() - unregister from dibs layer and all clients
+ * @dibs: dibs device
+ */
+void dibs_dev_del(struct dibs_dev *dibs);
+
 #endif	/* _DIBS_H */
include/linux/ism.h +1 -0
index 8358b4cd7ba6..9a53d3c48c16 100644
@@ -30,6 +30,7 @@ struct ism_dev {
 	spinlock_t lock; /* protects the ism device */
 	spinlock_t cmd_lock; /* serializes cmds */
 	struct list_head list;
+	struct dibs_dev *dibs;
 	struct pci_dev *pdev;
 
 	struct ism_sba *sba;
net/dibs/dibs_main.c +36 -0
index d91e461f2f06..5345e41ae5e4 100644
@@ -21,6 +21,15 @@ MODULE_LICENSE("GPL");
 static struct dibs_client *clients[MAX_DIBS_CLIENTS];
 static u8 max_client;
 static DEFINE_MUTEX(clients_lock);
+struct dibs_dev_list {
+	struct list_head list;
+	struct mutex mutex; /* protects dibs device list */
+};
+
+static struct dibs_dev_list dibs_dev_list = {
+	.list = LIST_HEAD_INIT(dibs_dev_list.list),
+	.mutex = __MUTEX_INITIALIZER(dibs_dev_list.mutex),
+};
 
 int dibs_register_client(struct dibs_client *client)
 {
@@ -56,6 +65,33 @@ int dibs_unregister_client(struct dibs_client *client)
 }
 EXPORT_SYMBOL_GPL(dibs_unregister_client);
 
+struct dibs_dev *dibs_dev_alloc(void)
+{
+	struct dibs_dev *dibs;
+
+	dibs = kzalloc(sizeof(*dibs), GFP_KERNEL);
+	return dibs;
+}
+EXPORT_SYMBOL_GPL(dibs_dev_alloc);
+
+int dibs_dev_add(struct dibs_dev *dibs)
+{
+	mutex_lock(&dibs_dev_list.mutex);
+	list_add(&dibs->list, &dibs_dev_list.list);
+	mutex_unlock(&dibs_dev_list.mutex);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(dibs_dev_add);
+
+void dibs_dev_del(struct dibs_dev *dibs)
+{
+	mutex_lock(&dibs_dev_list.mutex);
+	list_del_init(&dibs->list);
+	mutex_unlock(&dibs_dev_list.mutex);
+}
+EXPORT_SYMBOL_GPL(dibs_dev_del);
+
 static int __init dibs_init(void)
 {
 	memset(clients, 0, sizeof(clients));
Signature · 1 line
2.48.1
maintainer
#31 lore raw

Re: [RFC net-next 10/17] net/dibs: Define dibs_client_ops and dibs_dev_ops

In reply to Alexandra Winter
Quoted 52 lines · Alexandra Winter

Move the device add() and remove() functions from ism_client to dibs_client_ops and call add_dev()/del_dev() for ism devices and dibs_loopback devices. dibs_client_ops->add_dev() = smcd_register_dev() for the smc_dibs_client. This is the first step to handle ism and loopback devices alike (as dibs devices) in the smc dibs client.

Define dibs_dev->ops and move smcd_ops->get_chid to dibs_dev_ops->get_fabric_id() for ism and loopback devices. See below for why this needs to be in the same patch as dibs_client_ops->add_dev().

The following changes contain intermediate steps, that will be obsoleted by follow-on patches, once more functionality has been moved to dibs:

Use different smcd_ops and max_dmbs for ism and loopback. Follow-on patches will change SMC-D to directly use dibs_ops instead of smcd_ops.

In smcd_register_dev() it is now necessary to identify a dibs_loopback device before smcd_dev and smcd_ops->get_chid() are available. So provide dibs_dev_ops->get_fabric_id() in this patch and evaluate it in smc_ism_is_loopback().

Call smc_loopback_init() in smcd_register_dev() and call smc_loopback_exit() in smcd_unregister_dev() to handle the functionality that is still in smc_loopback. Follow-on patches will move all smc_loopback code to dibs_loopback.

In smcd_unregister_dev() use only ism device name, this will be replaced by dibs device name by a follow-on patch.

End of changes with intermediate parts.

Allocate an smcd event workqueue for all dibs devices, although dibs_loopback does not generate events.

Use kernel memory instead of devres memory for smcd_dev and smcd->conn. Since commit a72178cfe855 ("net/smc: Fix dependency of SMC on ISM") an ism device and its driver can have a longer lifetime than the smc module, so smc should not rely on devres to free its resources [1]. It is now the responsibility of the smc client to free smcd and smcd->conn for all dibs devices, ism devices as well as loopback. Call client->ops->del_dev() for all existing dibs devices in dibs_unregister_client(), so all device related structures can be freed in the client.

When dibs_unregister_client() is called in the context of smc_exit() or smc_core_reboot_event(), these functions have already called smc_lgrs_shutdown() which calls smc_smcd_terminate_all(smcd) and sets going_away. This is done a second time in smcd_unregister_dev(). This is analogous to how smcr is handled in these functions, by calling first smc_lgrs_shutdown() and then smc_ib_unregister_client() > smc_ib_remove_dev(), so leave it that way. It may be worth investigating, whether smc_lgrs_shutdown() is still required or useful.

Quoted 6 lines

Remove CONFIG_SMC_LO. CONFIG_DIBS_LO now controls whether a dibs loopback device exists or not.

Hi Winter,

I feel a bit hard for me to review the code especially with so many intermediate parts. I may need more time to review these.

Seperate such a big refine patch is hard. Maybe put the small parts in the front and the final one in the last to reduce the intermediate part as much as possible ? I'm not sure.

Best regards,
Dust

#45 lore raw

Re: [RFC net-next 15/17] net/dibs: Move query_remote_gid() to dibs_dev_ops

In reply to Alexandra Winter
Quoted 5 lines · Alexandra Winter

Provide the dibs_dev_ops->query_remote_gid() in ism and dibs_loopback dibs_devices. And call it in smc dibs_client.

Quoted 9 lines
---
 drivers/s390/net/ism_drv.c | 41 +++++++++++++++++---------------------
 include/linux/dibs.h       | 14 +++++++++++++
 include/net/smc.h          |  2 --
 net/dibs/dibs_loopback.c   | 10 ++++++++++
 net/smc/smc_ism.c          |  8 ++++++--
 net/smc/smc_loopback.c     | 13 ------------
 6 files changed, 48 insertions(+), 40 deletions(-)

-- snip --

Quoted 12 lines
include/linux/dibs.h +4 -0
index 10be10ae4660..d940411aa179 100644
@@ -133,6 +133,20 @@ struct dibs_dev_ops {
 	 * Return: 2 byte dibs fabric id
 	 */
 	u16 (*get_fabric_id)(struct dibs_dev *dev);
+	/**
+	 * query_remote_gid()
+	 * @dev: local dibs device
+	 * @rgid: gid of remote dibs device

include/linux/dibs.h struct dibs_dev_ops { · line 149

Quoted 10 lines
include/linux/dibs.h +10 -0
+	 * @vid_valid: if zero, vid will be ignored;
+	 *	       deprecated, ignored if device does not support vlan
+	 * @vid: VLAN id; deprecated, ignored if device does not support vlan
+	 *
+	 * Query whether a remote dibs device is reachable via this local device
+	 * and this vlan id.
+	 * Return: 0 if remote gid is reachable.
+	 */
+	int (*query_remote_gid)(struct dibs_dev *dev, uuid_t *rgid,
+				u32 vid_valid, u32 vid);

Shouldn't this be 'const uuid_t *rgid'?

-- snip --

Thanks,
Julian

author maintainer
#7 lore raw

Re: [RFC net-next 03/17] net/smc: Remove error handling of unregister_dmb()

In reply to Dust Li
Quoted 80 lines · Dust Li
Quoted 65 lines · Alexandra Winter

smcd_buf_free() calls smc_ism_unregister_dmb(lgr->smcd, buf_desc) and then unconditionally frees buf_desc.

Remove the cleaning up of fields of buf_desc in smc_ism_unregister_dmb(), because it is not helpful.

This removes the only usage of ISM_ERROR from the smc module. So move it to drivers/s390/net/ism.h.

---

drivers/s390/net/ism.h |  1 +
include/net/smc.h      |  2 --
net/smc/smc_ism.c      | 14 +++++---------
net/smc/smc_ism.h      |  3 ++-
4 files changed, 8 insertions(+), 12 deletions(-)
drivers/s390/net/ism.h +1 -0
index 047fa6101555..b5b03db52fce 100644
@@ -10,6 +10,7 @@
 #include <asm/pci_insn.h>
 
 #define UTIL_STR_LEN	16
+#define ISM_ERROR	0xFFFF
 
 /*
 * Do not use the first word of the DMB bits to ensure 8 byte aligned access.
include/net/smc.h +0 -2
index db84e4e35080..a9c023dd1380 100644
@@ -44,8 +44,6 @@ struct smcd_dmb {
 
 #define ISM_RESERVED_VLANID	0x1FFF
 
-#define ISM_ERROR	0xFFFF
-
 struct smcd_dev;
 
 struct smcd_gid {
net/smc/smc_ism.c +3 -3
index 84f98e18c7db..a94e1450d095 100644
@@ -205,13 +205,13 @@ int smc_ism_put_vlan(struct smcd_dev *smcd, unsigned short vlanid)
 	return rc;
 }
 
-int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc)
+void smc_ism_unregister_dmb(struct smcd_dev *smcd,
+			    struct smc_buf_desc *dmb_desc)
 {
 	struct smcd_dmb dmb;
-	int rc = 0;
 
 	if (!dmb_desc->dma_addr)
-		return rc;
+		return;
 
 	memset(&dmb, 0, sizeof(dmb));
 	dmb.dmb_tok = dmb_desc->token;

net/smc/smc_ism.c int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc) · line 222

Quoted 9 lines
net/smc/smc_ism.c +1 -5
@@ -219,13 +219,9 @@ int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc)
 	dmb.cpu_addr = dmb_desc->cpu_addr;
 	dmb.dma_addr = dmb_desc->dma_addr;
 	dmb.dmb_len = dmb_desc->len;
-	rc = smcd->ops->unregister_dmb(smcd, &dmb);
-	if (!rc || rc == ISM_ERROR) {
-		dmb_desc->cpu_addr = NULL;
-		dmb_desc->dma_addr = 0;
-	}
+	smcd->ops->unregister_dmb(smcd, &dmb);

Hmm, I think the old way of handling error here is certainly not good. But completely ignoring error handling here would make bugs harder to detect.

Quoted 8 lines

What about adding a WARN_ON_ONCE(rc) ?

Also, I think we can just remove the rc == ISM_ERROR to remove the dependency of ISM_ERROR in smc.

Best regards,
Dust

As I wrote in the commit message, I removed rc, because it is ignored by the caller anyhow today. If you want to I can add it back into this function and then you can think about how SMC should handle such an error.

My thoughts on this are:
There is not really much smc can do about a problem in unregister_dmb.

I think it is Linux strategy to report and handle errors at the lowest level possible. I have some patches on my harddisk to improve error handling and error reporting of the ism device diver. And we are already discussing internally which errors should do a WARN_ON_ONCE in the ism device driver.

So I don't think there is much to do in the smc layer at the moment. How does that sound to you?

author maintainer
#11 lore raw

Re: [RFC net-next 04/17] net/smc: Decouple sf and attached send_buf in smc_loopback

In reply to Dust Li
Quoted 1 line · Dust Li

On 2025-08-06 17:41:09, Alexandra Winter wrote:

[...]

Quoted 30 lines
Quoted 17 lines
net/smc/smc_core.h +1 -0
index 48a1b1dcb576..fe5f48d14323 100644
@@ -13,6 +13,7 @@
 #define _SMC_CORE_H
 
 #include <linux/atomic.h>
+#include <linux/types.h>
 #include <linux/smc.h>
 #include <linux/pci.h>
 #include <rdma/ib_verbs.h>
@@ -221,12 +222,16 @@ struct smc_buf_desc {
 					/* virtually contiguous */
 		};
 		struct { /* SMC-D */

net/smc/smc_core.h struct smc_buf_desc { · line 234

Quoted 10 lines
net/smc/smc_core.h +4 -0
+			 /* SMC-D rx buffer: */
 			unsigned short	sba_idx;
 					/* SBA index number */
 			u64		token;
 					/* DMB token number */
 			dma_addr_t	dma_addr;
 					/* DMA address */
+			/* SMC-D tx buffer */
+			bool		is_attached;
+					/* no need for explicit writes */
Quoted 7 lines

A small sugguestion: there is a hole between sba_idx and token, we can put is_attached in that hole.
Not a big deal because this is a union and SMC-R use a much large space.

Best regards,
Dust

Thank you very much for your throrough reviews of this series, Dust.

I put 'bool is_attached' in this place, so I could add the comments about which members are used for rx-buffers and which for tx-buffers.
I find the struct smc_buf_desc a bit confusing and thought these comments would be helpful. Is it ok for you to leave it that way?

author maintainer
#12 lore raw

Re: [RFC net-next 04/17] net/smc: Decouple sf and attached send_buf in smc_loopback

Quoted 43 lines · Alexandra Winter
Quoted 1 line · Dust Li

On 2025-08-06 17:41:09, Alexandra Winter wrote:

[...]

Quoted 30 lines
Quoted 17 lines
net/smc/smc_core.h +1 -0
index 48a1b1dcb576..fe5f48d14323 100644
@@ -13,6 +13,7 @@
 #define _SMC_CORE_H
 
 #include <linux/atomic.h>
+#include <linux/types.h>
 #include <linux/smc.h>
 #include <linux/pci.h>
 #include <rdma/ib_verbs.h>
@@ -221,12 +222,16 @@ struct smc_buf_desc {
 					/* virtually contiguous */
 		};
 		struct { /* SMC-D */

net/smc/smc_core.h struct smc_buf_desc { · line 234

Quoted 10 lines
net/smc/smc_core.h +4 -0
+			 /* SMC-D rx buffer: */
 			unsigned short	sba_idx;
 					/* SBA index number */
 			u64		token;
 					/* DMB token number */
 			dma_addr_t	dma_addr;
 					/* DMA address */
+			/* SMC-D tx buffer */
+			bool		is_attached;
+					/* no need for explicit writes */
Quoted 7 lines

A small sugguestion: there is a hole between sba_idx and token, we can put is_attached in that hole.
Not a big deal because this is a union and SMC-R use a much large space.

Best regards,
Dust

Quoted 6 lines

Thank you very much for your throrough reviews of this series, Dust.

I put 'bool is_attached' in this place, so I could add the comments about which members are used for rx-buffers and which for tx-buffers.
I find the struct smc_buf_desc a bit confusing and thought these comments would be helpful. Is it ok for you to leave it that way?

I hit send too fast. Obviously I can put it above sba_idx. That will reduce the hole by 1 byte. Changed for next version.

author maintainer
#15 lore raw

Re: [RFC net-next 05/17] net/smc: Improve log message for devices w/o pnetid

In reply to Dust Li
Quoted 5 lines · Dust Li

This patch doesn't seem strongly related to this patchset, so it might be better to send this patch separately.

Best regards,
Dust

You are right, and the same is true for
[RFC net-next 02/17] s390/ism: Log module load/unload

At the moment I would like to keep them in the series, so people don't have to worry about applying the series with or without them, because they touch the same files. In case the series gets delayed, I will split them off. That's why they are in the front.

author maintainer
#25 lore raw

Re: [RFC net-next 08/17] net/dibs: Register ism as dibs device

In reply to Dust Li
Quoted 9 lines · Dust Li

I've been wondering whether we should completely remove the ISM concept from SMC. Including rename smc_ism.c into smc_dibs.c.

Since DIBS already serves as the replacement for ISM, having both ISM and DIBS coexist in the codebase seems a bit confusing and inconsistent. Removing ISM could help streamline the code and improve clarity.

Best regards,
Dust

I second that.
Like I wrote in the last commit message:
"[RFC net-next 17/17] net/dibs: Move event handling to dibs layer
...
SMC-D and ISM are now independent.
struct ism_dev can be moved to drivers/s390/net/ism.h.

Note that in smc, the term 'ism' is still used. Future patches could replace that with 'dibs' or 'smc-d' as appropriate."

I am not sure what would be the best way to do such a global replacement. One big patch on top of dibs-series? That would be a lot of changes without adding any functionality.
Or do you have other clarity improvements in the pipeline that could be combined? I would like to defer that decision to the smc maintainers. Would that be ok for you?

author maintainer
#32 lore raw

Re: [RFC net-next 10/17] net/dibs: Define dibs_client_ops and dibs_dev_ops

In reply to Dust Li
Quoted 11 lines · Dust Li

Hi Winter,

I feel a bit hard for me to review the code especially with so many intermediate parts. I may need more time to review these.

Seperate such a big refine patch is hard. Maybe put the small parts in the front and the final one in the last to reduce the intermediate part as much as possible ? I'm not sure.

Best regards,
Dust

I can understand that very well. I tried hard to split up the dibs layer implementation into consumable pieces, while preserving the functionality of smc-d, ism and loopback at each intermediate step, so it is always bisectable.

I know this patch "[RFC net-next 10/17] net/dibs: Define dibs_client_ops and dibs_dev_ops" and "[RFC net-next 16/17] net/dibs: Move data path to dibs layer" are rather large, but I could not find a way to split them up without temporarily breaking functionality. If you have any ideas, please let me know.

You write:

Quoted 1 line

Maybe put the small parts in the front and the final one in the last

I am not sure I understand, what exactly you have in mind here. Are you asking for even larger patches?

Quoted 1 line

I may need more time to review these.

FYI: I will be on vacation the last 2 weeks of August.
Would you rather have an RFC v2 this week with all the changes, I made so far? Or would you prefer that you continue reviewing this RFC and I send a new version in the first week of September?

maintainer
#8 lore raw

Re: [RFC net-next 03/17] net/smc: Remove error handling of unregister_dmb()

In reply to Alexandra Winter
Quoted 99 lines · Alexandra Winter
Quoted 80 lines · Dust Li
Quoted 65 lines · Alexandra Winter

smcd_buf_free() calls smc_ism_unregister_dmb(lgr->smcd, buf_desc) and then unconditionally frees buf_desc.

Remove the cleaning up of fields of buf_desc in smc_ism_unregister_dmb(), because it is not helpful.

This removes the only usage of ISM_ERROR from the smc module. So move it to drivers/s390/net/ism.h.

---

drivers/s390/net/ism.h |  1 +
include/net/smc.h      |  2 --
net/smc/smc_ism.c      | 14 +++++---------
net/smc/smc_ism.h      |  3 ++-
4 files changed, 8 insertions(+), 12 deletions(-)
drivers/s390/net/ism.h +1 -0
index 047fa6101555..b5b03db52fce 100644
@@ -10,6 +10,7 @@
 #include <asm/pci_insn.h>
 
 #define UTIL_STR_LEN	16
+#define ISM_ERROR	0xFFFF
 
 /*
 * Do not use the first word of the DMB bits to ensure 8 byte aligned access.
include/net/smc.h +0 -2
index db84e4e35080..a9c023dd1380 100644
@@ -44,8 +44,6 @@ struct smcd_dmb {
 
 #define ISM_RESERVED_VLANID	0x1FFF
 
-#define ISM_ERROR	0xFFFF
-
 struct smcd_dev;
 
 struct smcd_gid {
net/smc/smc_ism.c +3 -3
index 84f98e18c7db..a94e1450d095 100644
@@ -205,13 +205,13 @@ int smc_ism_put_vlan(struct smcd_dev *smcd, unsigned short vlanid)
 	return rc;
 }
 
-int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc)
+void smc_ism_unregister_dmb(struct smcd_dev *smcd,
+			    struct smc_buf_desc *dmb_desc)
 {
 	struct smcd_dmb dmb;
-	int rc = 0;
 
 	if (!dmb_desc->dma_addr)
-		return rc;
+		return;
 
 	memset(&dmb, 0, sizeof(dmb));
 	dmb.dmb_tok = dmb_desc->token;

net/smc/smc_ism.c int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc) · line 222

Quoted 9 lines
net/smc/smc_ism.c +1 -5
@@ -219,13 +219,9 @@ int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc)
 	dmb.cpu_addr = dmb_desc->cpu_addr;
 	dmb.dma_addr = dmb_desc->dma_addr;
 	dmb.dmb_len = dmb_desc->len;
-	rc = smcd->ops->unregister_dmb(smcd, &dmb);
-	if (!rc || rc == ISM_ERROR) {
-		dmb_desc->cpu_addr = NULL;
-		dmb_desc->dma_addr = 0;
-	}
+	smcd->ops->unregister_dmb(smcd, &dmb);

Hmm, I think the old way of handling error here is certainly not good. But completely ignoring error handling here would make bugs harder to detect.

Quoted 8 lines

What about adding a WARN_ON_ONCE(rc) ?

Also, I think we can just remove the rc == ISM_ERROR to remove the dependency of ISM_ERROR in smc.

Best regards,
Dust

As I wrote in the commit message, I removed rc, because it is ignored by the caller anyhow today. If you want to I can add it back into this function and then you can think about how SMC should handle such an error.

My thoughts on this are:
There is not really much smc can do about a problem in unregister_dmb.

Quoted 6 lines

I think it is Linux strategy to report and handle errors at the lowest level possible. I have some patches on my harddisk to improve error handling and error reporting of the ism device diver. And we are already discussing internally which errors should do a WARN_ON_ONCE in the ism device driver.

So I don't think there is much to do in the smc layer at the moment. How does that sound to you?

LGTM.

Best regards,
Dust

maintainer
#26 lore raw

Re: [RFC net-next 08/17] net/dibs: Register ism as dibs device

In reply to Alexandra Winter
Quoted 20 lines · Alexandra Winter
Quoted 9 lines · Dust Li

I've been wondering whether we should completely remove the ISM concept from SMC. Including rename smc_ism.c into smc_dibs.c.

Since DIBS already serves as the replacement for ISM, having both ISM and DIBS coexist in the codebase seems a bit confusing and inconsistent. Removing ISM could help streamline the code and improve clarity.

Best regards,
Dust

I second that.
Like I wrote in the last commit message:
"[RFC net-next 17/17] net/dibs: Move event handling to dibs layer
...
SMC-D and ISM are now independent.
struct ism_dev can be moved to drivers/s390/net/ism.h.

Quoted 7 lines

Note that in smc, the term 'ism' is still used. Future patches could replace that with 'dibs' or 'smc-d' as appropriate."

I am not sure what would be the best way to do such a global replacement. One big patch on top of dibs-series? That would be a lot of changes without adding any functionality.

I prefer this approach. Renaming without changing functionality keeps the patch clean and makes it easier to cherry-pick.

Best regards,
Dust

Quoted 3 lines

Or do you have other clarity improvements in the pipeline that could be combined? I would like to defer that decision to the smc maintainers. Would that be ok for you?

maintainer
#33 lore raw

Re: [RFC net-next 10/17] net/dibs: Define dibs_client_ops and dibs_dev_ops

In reply to Alexandra Winter
Quoted 24 lines · Alexandra Winter
Quoted 11 lines · Dust Li

Hi Winter,

I feel a bit hard for me to review the code especially with so many intermediate parts. I may need more time to review these.

Seperate such a big refine patch is hard. Maybe put the small parts in the front and the final one in the last to reduce the intermediate part as much as possible ? I'm not sure.

Best regards,
Dust

I can understand that very well. I tried hard to split up the dibs layer implementation into consumable pieces, while preserving the functionality of smc-d, ism and loopback at each intermediate step, so it is always bisectable.

I know this patch "[RFC net-next 10/17] net/dibs: Define dibs_client_ops and dibs_dev_ops" and "[RFC net-next 16/17] net/dibs: Move data path to dibs layer" are rather large, but I could not find a way to split them up without temporarily breaking functionality. If you have any ideas, please let me know.

Quoted 4 lines

You write:

Quoted 1 line

Maybe put the small parts in the front and the final one in the last

I am not sure I understand, what exactly you have in mind here. Are you asking for even larger patches?

Yes, that’s what I had in mind. :)
Of course, if it seems to complicate things, please disregard it.

Quoted 6 lines
Quoted 2 lines

I may need more time to review these.

FYI: I will be on vacation the last 2 weeks of August.
Would you rather have an RFC v2 this week with all the changes, I made so far? Or would you prefer that you continue reviewing this RFC and I send a new version in the first week of September?

No warries, both OK for me now.

Best regards,
Dust

author maintainer
#36 lore raw

Re: [RFC net-next 11/17] net/dibs: Move struct device to dibs_dev

In reply to Alexandra Winter

On 06.08.25 17:41, Alexandra Winter wrote:
[...]

Quoted 3 lines

Replace smcd->ops->get_dev(smcd) by dibs_get_dev().

Looking at the resulting code, I don't really like this concept of a *_get_dev() function, that does not call get_device().
I plan to replace that by using dibs->dev directly in the next version.

See below for the code pieces I am referring to:

Quoted 4 lines
drivers/s390/net/ism_drv.c +0 -0
index 84a6e9ae2e64..0ddfd47a3a7c 100644

[...]

Quoted 6 lines
drivers/s390/net/ism_drv.c +1 -1
@@ -697,16 +684,14 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	ism_dev_exit(ism);
 err_dibs:
 	/* pairs with dibs_dev_alloc() */
-	kfree(dibs);
+	put_device(dibs_get_dev(dibs));

[...]

Quoted 8 lines
drivers/s390/net/ism_drv.c +1 -1
@@ -719,13 +704,12 @@ static void ism_remove(struct pci_dev *pdev)
 	dibs_dev_del(dibs);
 	ism_dev_exit(ism);
 	/* pairs with dibs_dev_alloc() */
-	kfree(dibs);
+	put_device(dibs_get_dev(dibs));
 
 	pci_release_mem_regions(pdev);

[...]

Quoted 26 lines
drivers/s390/net/ism_drv.c +0 -8
@@ -871,13 +855,6 @@ static void smcd_get_local_gid(struct smcd_dev *smcd,
 	smcd_gid->gid_ext = 0;
 }
 
-static inline struct device *smcd_get_dev(struct smcd_dev *dev)
-{
-	struct ism_dev *ism = dev->priv;
-
-	return &ism->dev;
-}
-
 static const struct smcd_ops ism_smcd_ops = {
 	.query_remote_gid = smcd_query_rgid,
 	.register_dmb = smcd_register_dmb,
@@ -890,7 +867,6 @@ static const struct smcd_ops ism_smcd_ops = {
 	.move_data = smcd_move,
 	.supports_v2 = smcd_supports_v2,
 	.get_local_gid = smcd_get_local_gid,
-	.get_dev = smcd_get_dev,
 };
 
 const struct smcd_ops *ism_get_smcd_ops(void)
include/linux/dibs.h +0 -0
index 805ab33271b5..4459b9369dc0 100644

[...]

Quoted 19 lines
include/linux/dibs.h +15 -0
@@ -158,6 +159,21 @@ static inline void *dibs_get_priv(struct dibs_dev *dev,
 
 /* ------- End of client-only functions ----------- */
 
+/* Functions to be called by dibs clients and dibs device drivers:
+ */
+/**
+ * dibs_get_dev()
+ * @dev: dibs device
+ * @token: dmb token of the remote dmb
+ *
+ * TODO: provide get and put functions
+ * Return: struct device* to be used for device refcounting
+ */
+static inline struct device *dibs_get_dev(struct dibs_dev *dibs)
+{
+	return &dibs->dev;
+}
+

[...]

Quoted 12 lines
include/net/smc.h +0 -1
index e271891b85e6..05faac83371e 100644
@@ -63,7 +63,6 @@ struct smcd_ops {
 			 unsigned int size);
 	int (*supports_v2)(void);
 	void (*get_local_gid)(struct smcd_dev *dev, struct smcd_gid *gid);
-	struct device* (*get_dev)(struct smcd_dev *dev);
 
 	/* optional operations */
 	int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id);

[...]

Quoted 23 lines
net/smc/smc_core.c +2 -2
index 67f9e0b83ebc..71c410dc3658 100644
@@ -924,7 +924,7 @@ static int smc_lgr_create(struct smc_sock *smc, struct smc_init_info *ini)
 	if (ini->is_smcd) {
 		/* SMC-D specific settings */
 		smcd = ini->ism_dev[ini->ism_selected];
-		get_device(smcd->ops->get_dev(smcd));
+		get_device(dibs_get_dev(smcd->dibs));
 		lgr->peer_gid.gid =
 			ini->ism_peer_gid[ini->ism_selected].gid;
 		lgr->peer_gid.gid_ext =
@@ -1474,7 +1474,7 @@ static void smc_lgr_free(struct smc_link_group *lgr)
 	destroy_workqueue(lgr->tx_wq);
 	if (lgr->is_smcd) {
 		smc_ism_put_vlan(lgr->smcd, lgr->vlan_id);
-		put_device(lgr->smcd->ops->get_dev(lgr->smcd));
+		put_device(dibs_get_dev(lgr->smcd->dibs));
 	}
 	smc_lgr_put(lgr); /* theoretically last lgr_put */
 }

[...]

Quoted 31 lines
net/smc/smc_loopback.c +0 -7
index 37d8366419f7..262d0d0df4d0 100644
@@ -23,7 +23,6 @@
 #define SMC_LO_SUPPORT_NOCOPY	0x1
 #define SMC_DMA_ADDR_INVALID	(~(dma_addr_t)0)
 
-static const char smc_lo_dev_name[] = "loopback-ism";
 static struct smc_lo_dev *lo_dev;
 
 static void smc_lo_generate_ids(struct smc_lo_dev *ldev)
@@ -255,11 +254,6 @@ static void smc_lo_get_local_gid(struct smcd_dev *smcd,
 	smcd_gid->gid_ext = ldev->local_gid.gid_ext;
 }
 
-static struct device *smc_lo_get_dev(struct smcd_dev *smcd)
-{
-	return &((struct smc_lo_dev *)smcd->priv)->dev;
-}
-
 static const struct smcd_ops lo_ops = {
 	.query_remote_gid = smc_lo_query_rgid,
 	.register_dmb = smc_lo_register_dmb,
@@ -274,7 +268,6 @@ static const struct smcd_ops lo_ops = {
 	.signal_event		= NULL,
 	.move_data = smc_lo_move_data,
 	.get_local_gid = smc_lo_get_local_gid,
-	.get_dev = smc_lo_get_dev,
 };
 

[...]

Quoted 40 lines
net/smc/smc_pnet.c +4 -4
index 76ad29e31d60..bbdd875731f2 100644
@@ -169,7 +169,7 @@ static int smc_pnet_remove_by_pnetid(struct net *net, char *pnet_name)
 			pr_warn_ratelimited("smc: smcd device %s "
 					    "erased user defined pnetid "
 					    "%.16s\n",
-					    dev_name(smcd->ops->get_dev(smcd)),
+					    dev_name(dibs_get_dev(smcd->dibs)),
 					    smcd->pnetid);
 			memset(smcd->pnetid, 0, SMC_MAX_PNETID_LEN);
 			smcd->pnetid_by_user = false;
@@ -332,7 +332,7 @@ static struct smcd_dev *smc_pnet_find_smcd(char *smcd_name)
 
 	mutex_lock(&smcd_dev_list.mutex);
 	list_for_each_entry(smcd_dev, &smcd_dev_list.list, list) {
-		if (!strncmp(dev_name(smcd_dev->ops->get_dev(smcd_dev)),
+		if (!strncmp(dev_name(dibs_get_dev(smcd_dev->dibs)),
 			     smcd_name, IB_DEVICE_NAME_MAX - 1))
 			goto out;
 	}
@@ -431,7 +431,7 @@ static int smc_pnet_add_ib(struct smc_pnettable *pnettable, char *ib_name,
 	if (smcd) {
 		smcddev_applied = smc_pnet_apply_smcd(smcd, pnet_name);
 		if (smcddev_applied) {
-			dev = smcd->ops->get_dev(smcd);
+			dev = dibs_get_dev(smcd->dibs);
 			pr_warn_ratelimited("smc: smcd device %s "
 					    "applied user defined pnetid "
 					    "%.16s\n", dev_name(dev),
@@ -1192,7 +1192,7 @@ int smc_pnetid_by_table_ib(struct smc_ib_device *smcibdev, u8 ib_port)
  */
 int smc_pnetid_by_table_smcd(struct smcd_dev *smcddev)
 {
-	const char *ib_name = dev_name(smcddev->ops->get_dev(smcddev));
+	const char *ib_name = dev_name(dibs_get_dev(smcddev->dibs));
 	struct smc_pnettable *pnettable;
 	struct smc_pnetentry *tmp_pe;
 	struct smc_net *sn;
author maintainer
#46 lore raw

Re: [RFC net-next 15/17] net/dibs: Move query_remote_gid() to dibs_dev_ops

In reply to Julian Ruess
Quoted 43 lines · Julian Ruess
Quoted 5 lines · Alexandra Winter

Provide the dibs_dev_ops->query_remote_gid() in ism and dibs_loopback dibs_devices. And call it in smc dibs_client.

Quoted 9 lines
---
 drivers/s390/net/ism_drv.c | 41 +++++++++++++++++---------------------
 include/linux/dibs.h       | 14 +++++++++++++
 include/net/smc.h          |  2 --
 net/dibs/dibs_loopback.c   | 10 ++++++++++
 net/smc/smc_ism.c          |  8 ++++++--
 net/smc/smc_loopback.c     | 13 ------------
 6 files changed, 48 insertions(+), 40 deletions(-)

-- snip --

Quoted 12 lines
include/linux/dibs.h +4 -0
index 10be10ae4660..d940411aa179 100644
@@ -133,6 +133,20 @@ struct dibs_dev_ops {
 	 * Return: 2 byte dibs fabric id
 	 */
 	u16 (*get_fabric_id)(struct dibs_dev *dev);
+	/**
+	 * query_remote_gid()
+	 * @dev: local dibs device
+	 * @rgid: gid of remote dibs device

include/linux/dibs.h struct dibs_dev_ops { · line 149

Quoted 10 lines
include/linux/dibs.h +10 -0
+	 * @vid_valid: if zero, vid will be ignored;
+	 *	       deprecated, ignored if device does not support vlan
+	 * @vid: VLAN id; deprecated, ignored if device does not support vlan
+	 *
+	 * Query whether a remote dibs device is reachable via this local device
+	 * and this vlan id.
+	 * Return: 0 if remote gid is reachable.
+	 */
+	int (*query_remote_gid)(struct dibs_dev *dev, uuid_t *rgid,
+				u32 vid_valid, u32 vid);

Shouldn't this be 'const uuid_t *rgid'?

Quoted 4 lines

-- snip --

Thanks,
Julian

Good point. Same for the 'uuid_t *' in signal_event() in [RFC net-next 17/17] net/dibs: Move event handling to dibs layer Changed in next version.

maintainer
#37 lore raw

Re: [RFC net-next 11/17] net/dibs: Move struct device to dibs_dev

In reply to Alexandra Winter
Quoted 11 lines · Alexandra Winter

On 06.08.25 17:41, Alexandra Winter wrote:
[...]

Quoted 3 lines

Replace smcd->ops->get_dev(smcd) by dibs_get_dev().

Looking at the resulting code, I don't really like this concept of a *_get_dev() function, that does not call get_device().
I plan to replace that by using dibs->dev directly in the next version.

May I ask why? Because of the function name ? If so, maybe we can change the name.

While I don't have a strong preference either way, I personally favor hiding the members of the dibs_dev structure from the upper layer. In my opinion, it would be better to avoid direct access to dibs members from upper layers and instead provide dedicated interface functions.

For example, I even think we should not expose dibs->ops->xxx directly to the SMC layer. Encapsulating such details would improve modularity and maintainability. Just like what IB subsystem has done before :)

For example:

# git grep dibs net/smc
[...]
net/smc/smc_ism.c:      return dibs->ops->query_remote_gid(dibs, &ism_rgid, vlan_id ? 1 : 0,
net/smc/smc_ism.c:      return smcd->dibs->ops->get_fabric_id(smcd->dibs);
net/smc/smc_ism.c:      if (!smcd->dibs->ops->add_vlan_id)
net/smc/smc_ism.c:      if (smcd->dibs->ops->add_vlan_id(smcd->dibs, vlanid)) {
net/smc/smc_ism.c:      if (!smcd->dibs->ops->del_vlan_id)
net/smc/smc_ism.c:      if (smcd->dibs->ops->del_vlan_id(smcd->dibs, vlanid))
[...]

Best regards,
Dust

Quoted 183 lines

See below for the code pieces I am referring to:

Quoted 4 lines
drivers/s390/net/ism_drv.c +0 -0
index 84a6e9ae2e64..0ddfd47a3a7c 100644

[...]

Quoted 6 lines
drivers/s390/net/ism_drv.c +1 -1
@@ -697,16 +684,14 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	ism_dev_exit(ism);
 err_dibs:
 	/* pairs with dibs_dev_alloc() */
-	kfree(dibs);
+	put_device(dibs_get_dev(dibs));

[...]

Quoted 8 lines
drivers/s390/net/ism_drv.c +1 -1
@@ -719,13 +704,12 @@ static void ism_remove(struct pci_dev *pdev)
 	dibs_dev_del(dibs);
 	ism_dev_exit(ism);
 	/* pairs with dibs_dev_alloc() */
-	kfree(dibs);
+	put_device(dibs_get_dev(dibs));
 
 	pci_release_mem_regions(pdev);

[...]

Quoted 26 lines
drivers/s390/net/ism_drv.c +0 -8
@@ -871,13 +855,6 @@ static void smcd_get_local_gid(struct smcd_dev *smcd,
 	smcd_gid->gid_ext = 0;
 }
 
-static inline struct device *smcd_get_dev(struct smcd_dev *dev)
-{
-	struct ism_dev *ism = dev->priv;
-
-	return &ism->dev;
-}
-
 static const struct smcd_ops ism_smcd_ops = {
 	.query_remote_gid = smcd_query_rgid,
 	.register_dmb = smcd_register_dmb,
@@ -890,7 +867,6 @@ static const struct smcd_ops ism_smcd_ops = {
 	.move_data = smcd_move,
 	.supports_v2 = smcd_supports_v2,
 	.get_local_gid = smcd_get_local_gid,
-	.get_dev = smcd_get_dev,
 };
 
 const struct smcd_ops *ism_get_smcd_ops(void)
include/linux/dibs.h +0 -0
index 805ab33271b5..4459b9369dc0 100644

[...]

Quoted 19 lines
include/linux/dibs.h +15 -0
@@ -158,6 +159,21 @@ static inline void *dibs_get_priv(struct dibs_dev *dev,
 
 /* ------- End of client-only functions ----------- */
 
+/* Functions to be called by dibs clients and dibs device drivers:
+ */
+/**
+ * dibs_get_dev()
+ * @dev: dibs device
+ * @token: dmb token of the remote dmb
+ *
+ * TODO: provide get and put functions
+ * Return: struct device* to be used for device refcounting
+ */
+static inline struct device *dibs_get_dev(struct dibs_dev *dibs)
+{
+	return &dibs->dev;
+}
+

[...]

Quoted 12 lines
include/net/smc.h +0 -1
index e271891b85e6..05faac83371e 100644
@@ -63,7 +63,6 @@ struct smcd_ops {
 			 unsigned int size);
 	int (*supports_v2)(void);
 	void (*get_local_gid)(struct smcd_dev *dev, struct smcd_gid *gid);
-	struct device* (*get_dev)(struct smcd_dev *dev);
 
 	/* optional operations */
 	int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id);

[...]

Quoted 23 lines
net/smc/smc_core.c +2 -2
index 67f9e0b83ebc..71c410dc3658 100644
@@ -924,7 +924,7 @@ static int smc_lgr_create(struct smc_sock *smc, struct smc_init_info *ini)
 	if (ini->is_smcd) {
 		/* SMC-D specific settings */
 		smcd = ini->ism_dev[ini->ism_selected];
-		get_device(smcd->ops->get_dev(smcd));
+		get_device(dibs_get_dev(smcd->dibs));
 		lgr->peer_gid.gid =
 			ini->ism_peer_gid[ini->ism_selected].gid;
 		lgr->peer_gid.gid_ext =
@@ -1474,7 +1474,7 @@ static void smc_lgr_free(struct smc_link_group *lgr)
 	destroy_workqueue(lgr->tx_wq);
 	if (lgr->is_smcd) {
 		smc_ism_put_vlan(lgr->smcd, lgr->vlan_id);
-		put_device(lgr->smcd->ops->get_dev(lgr->smcd));
+		put_device(dibs_get_dev(lgr->smcd->dibs));
 	}
 	smc_lgr_put(lgr); /* theoretically last lgr_put */
 }

[...]

Quoted 31 lines
net/smc/smc_loopback.c +0 -7
index 37d8366419f7..262d0d0df4d0 100644
@@ -23,7 +23,6 @@
 #define SMC_LO_SUPPORT_NOCOPY	0x1
 #define SMC_DMA_ADDR_INVALID	(~(dma_addr_t)0)
 
-static const char smc_lo_dev_name[] = "loopback-ism";
 static struct smc_lo_dev *lo_dev;
 
 static void smc_lo_generate_ids(struct smc_lo_dev *ldev)
@@ -255,11 +254,6 @@ static void smc_lo_get_local_gid(struct smcd_dev *smcd,
 	smcd_gid->gid_ext = ldev->local_gid.gid_ext;
 }
 
-static struct device *smc_lo_get_dev(struct smcd_dev *smcd)
-{
-	return &((struct smc_lo_dev *)smcd->priv)->dev;
-}
-
 static const struct smcd_ops lo_ops = {
 	.query_remote_gid = smc_lo_query_rgid,
 	.register_dmb = smc_lo_register_dmb,
@@ -274,7 +268,6 @@ static const struct smcd_ops lo_ops = {
 	.signal_event		= NULL,
 	.move_data = smc_lo_move_data,
 	.get_local_gid = smc_lo_get_local_gid,
-	.get_dev = smc_lo_get_dev,
 };
 

[...]

Quoted 40 lines
net/smc/smc_pnet.c +4 -4
index 76ad29e31d60..bbdd875731f2 100644
@@ -169,7 +169,7 @@ static int smc_pnet_remove_by_pnetid(struct net *net, char *pnet_name)
 			pr_warn_ratelimited("smc: smcd device %s "
 					    "erased user defined pnetid "
 					    "%.16s\n",
-					    dev_name(smcd->ops->get_dev(smcd)),
+					    dev_name(dibs_get_dev(smcd->dibs)),
 					    smcd->pnetid);
 			memset(smcd->pnetid, 0, SMC_MAX_PNETID_LEN);
 			smcd->pnetid_by_user = false;
@@ -332,7 +332,7 @@ static struct smcd_dev *smc_pnet_find_smcd(char *smcd_name)
 
 	mutex_lock(&smcd_dev_list.mutex);
 	list_for_each_entry(smcd_dev, &smcd_dev_list.list, list) {
-		if (!strncmp(dev_name(smcd_dev->ops->get_dev(smcd_dev)),
+		if (!strncmp(dev_name(dibs_get_dev(smcd_dev->dibs)),
 			     smcd_name, IB_DEVICE_NAME_MAX - 1))
 			goto out;
 	}
@@ -431,7 +431,7 @@ static int smc_pnet_add_ib(struct smc_pnettable *pnettable, char *ib_name,
 	if (smcd) {
 		smcddev_applied = smc_pnet_apply_smcd(smcd, pnet_name);
 		if (smcddev_applied) {
-			dev = smcd->ops->get_dev(smcd);
+			dev = dibs_get_dev(smcd->dibs);
 			pr_warn_ratelimited("smc: smcd device %s "
 					    "applied user defined pnetid "
 					    "%.16s\n", dev_name(dev),
@@ -1192,7 +1192,7 @@ int smc_pnetid_by_table_ib(struct smc_ib_device *smcibdev, u8 ib_port)
  */
 int smc_pnetid_by_table_smcd(struct smcd_dev *smcddev)
 {
-	const char *ib_name = dev_name(smcddev->ops->get_dev(smcddev));
+	const char *ib_name = dev_name(dibs_get_dev(smcddev->dibs));
 	struct smc_pnettable *pnettable;
 	struct smc_pnetentry *tmp_pe;
 	struct smc_net *sn;
author maintainer
#38 lore raw

Re: [RFC net-next 11/17] net/dibs: Move struct device to dibs_dev

Quoted 10 lines · Dust Li
Quoted 8 lines · Alexandra Winter

On 06.08.25 17:41, Alexandra Winter wrote:
[...]

Quoted 2 lines

Replace smcd->ops->get_dev(smcd) by dibs_get_dev().

Looking at the resulting code, I don't really like this concept of a *_get_dev() function, that does not call get_device().
I plan to replace that by using dibs->dev directly in the next version.

May I ask why? Because of the function name ? If so, maybe we can change the name.

Yes the name. Especially, as it is often used as argument for get_device() or put_device(). Eventually I would like to provide dibs_get_dev()/dibs_put_dev() that actually do refcounting.
And then I thought defining dibs_read_dev() is not helping readability.

Quoted 21 lines

While I don't have a strong preference either way, I personally favor hiding the members of the dibs_dev structure from the upper layer. In my opinion, it would be better to avoid direct access to dibs members from upper layers and instead provide dedicated interface functions.

For example, I even think we should not expose dibs->ops->xxx directly to the SMC layer. Encapsulating such details would improve modularity and maintainability. Just like what IB subsystem has done before :)

For example:

# git grep dibs net/smc
[...]
net/smc/smc_ism.c:      return dibs->ops->query_remote_gid(dibs, &ism_rgid, vlan_id ? 1 : 0,
net/smc/smc_ism.c:      return smcd->dibs->ops->get_fabric_id(smcd->dibs);
net/smc/smc_ism.c:      if (!smcd->dibs->ops->add_vlan_id)
net/smc/smc_ism.c:      if (smcd->dibs->ops->add_vlan_id(smcd->dibs, vlanid)) {
net/smc/smc_ism.c:      if (!smcd->dibs->ops->del_vlan_id)
net/smc/smc_ism.c:      if (smcd->dibs->ops->del_vlan_id(smcd->dibs, vlanid))
[...]
Quoted 2 lines

Best regards,
Dust

I see your point and I remember you brought that up in your review of [RFC net-next 0/7] Provide an ism layer
already.

I tried to keep this series to a meaningful minimum, which is a tradeoff. If possible, I just wanted to move code around and add the dibs layer in-between. There are several areas where I would like to see even more de-coupling. eg.:
- handle_irq(): Clients should not run in interrupt context, a receive_data() callback function would be better.
- The device drivers should not loop through the client array
- dibs_dev_op.*_dmb() functions reveal unnecessary details of the internal dmb struct to the clients
- ...

So instead of adding a set of 1:1 caller functions / interface functions for dibs_dev_ops and dibs_client_ops now, I would like to propose to work on further decoupling devices and clients by adding more abstractions that bring benefit. And then replace the remaining calls to ops by 1:1 caller functions. Does that make sense? Or does anybody feel strongly that I need to provide interface functions now?

BTW, there are some client-only functions and some device-driver-only functions in dibs.h already. So that is the direction.

maintainer
#39 lore raw

Re: [RFC net-next 11/17] net/dibs: Move struct device to dibs_dev

Quoted 12 lines · Alexandra Winter
Quoted 9 lines · Dust Li
Quoted 8 lines · Alexandra Winter

On 06.08.25 17:41, Alexandra Winter wrote:
[...]

Quoted 2 lines

Replace smcd->ops->get_dev(smcd) by dibs_get_dev().

Looking at the resulting code, I don't really like this concept of a *_get_dev() function, that does not call get_device().
I plan to replace that by using dibs->dev directly in the next version.

Quoted 6 lines
Quoted 1 line

May I ask why? Because of the function name ? If so, maybe we can change the name.

Yes the name. Especially, as it is often used as argument for get_device() or put_device(). Eventually I would like to provide dibs_get_dev()/dibs_put_dev() that actually do refcounting.
And then I thought defining dibs_read_dev() is not helping readability.

I see. I don't like dibs_get_dev() either. What about dibs_device_to_dev() or dibs_to_dev() ?

If we can't agree on a name we’re all happy with, I agree we can leave it as is for now.

Quoted 40 lines
Quoted 21 lines

While I don't have a strong preference either way, I personally favor hiding the members of the dibs_dev structure from the upper layer. In my opinion, it would be better to avoid direct access to dibs members from upper layers and instead provide dedicated interface functions.

For example, I even think we should not expose dibs->ops->xxx directly to the SMC layer. Encapsulating such details would improve modularity and maintainability. Just like what IB subsystem has done before :)

For example:

# git grep dibs net/smc
[...]
net/smc/smc_ism.c:      return dibs->ops->query_remote_gid(dibs, &ism_rgid, vlan_id ? 1 : 0,
net/smc/smc_ism.c:      return smcd->dibs->ops->get_fabric_id(smcd->dibs);
net/smc/smc_ism.c:      if (!smcd->dibs->ops->add_vlan_id)
net/smc/smc_ism.c:      if (smcd->dibs->ops->add_vlan_id(smcd->dibs, vlanid)) {
net/smc/smc_ism.c:      if (!smcd->dibs->ops->del_vlan_id)
net/smc/smc_ism.c:      if (smcd->dibs->ops->del_vlan_id(smcd->dibs, vlanid))
[...]
Quoted 2 lines

Best regards,
Dust

I see your point and I remember you brought that up in your review of [RFC net-next 0/7] Provide an ism layer
already.

I tried to keep this series to a meaningful minimum, which is a tradeoff. If possible, I just wanted to move code around and add the dibs layer in-between. There are several areas where I would like to see even more de-coupling. eg.:
- handle_irq(): Clients should not run in interrupt context, a receive_data() callback function would be better.
- The device drivers should not loop through the client array
- dibs_dev_op.*_dmb() functions reveal unnecessary details of the internal dmb struct to the clients
- ...

Quoted 6 lines

So instead of adding a set of 1:1 caller functions / interface functions for dibs_dev_ops and dibs_client_ops now, I would like to propose to work on further decoupling devices and clients by adding more abstractions that bring benefit. And then replace the remaining calls to ops by 1:1 caller functions. Does that make sense? Or does anybody feel strongly that I need to provide interface functions now?

Yes, I agree we can do that in the future.

Best regards,
Dust

author maintainer
#40 lore raw

Re: [RFC net-next 11/17] net/dibs: Move struct device to dibs_dev

Quoted 16 lines · Dust Li
Quoted 7 lines · Alexandra Winter
Quoted 5 lines · Dust Li
Quoted 4 lines · Alexandra Winter

On 06.08.25 17:41, Alexandra Winter wrote:
[...]

Quoted 2 lines

Replace smcd->ops->get_dev(smcd) by dibs_get_dev().

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

Looking at the resulting code, I don't really like this concept of a *_get_dev() function, that does not call get_device().
I plan to replace that by using dibs->dev directly in the next version.

May I ask why? Because of the function name ? If so, maybe we can change the name.

Yes the name. Especially, as it is often used as argument for get_device() or put_device(). Eventually I would like to provide dibs_get_dev()/dibs_put_dev() that actually do refcounting.
And then I thought defining dibs_read_dev() is not helping readability.

Quoted 6 lines

I see. I don't like dibs_get_dev() either. What about dibs_device_to_dev() or dibs_to_dev() ?

If we can't agree on a name we’re all happy with, I agree we can leave it as is for now.

I'd rather leave it as it is and leave it to future patches to provide dibs_get_dev()/dibs_put_dev() and maybe also dibs_devname() service functions.