Skip to contentSkip to conversation

net/smc: implement virtual ISM extension and loopback-ism

PATCH In review Wen Gu 7 participants 50 messages Sep 24, 2023 – Oct 18, 2023

Subsystems (5)

v4 of 5 In review Latest is v5 →

lore
Share & tools
lore

Wen Gu

author reviewer
#0 lore raw

v4 cover net/smc: implement virtual ISM extension and loopback-ism

Hi, all

# Background

SMC-D is now used in IBM z with ISM function to optimize network interconnect for intra-CPC communications. Inspired by this, we try to make SMC-D available on the non-s390 architecture through a software-simulated virtual ISM device, such as loopback-ism device here, to accelerate inter-process or inter-containers communication within the same OS.

# Design

This patch set includes 4 parts:

- Patch #1-#3: decouple ISM device hard code from SMC-D stack.
- Patch #4-#8: implement virtual ISM extension defined in SMCv2.1.
- Patch #9-#13: implement loopback-ism device.
- Patch #14-#18: memory copy optimization for the case using loopback.

The loopback-ism device is designed as a kernel device and not be limited to a specific net namespace, ends of both inter-process connection (1/1' in diagram below) or inter-container connection (2/2' in diagram below) will find that peer shares the same loopback-ism device during the CLC handshake. Then loopback-ism device will be chosen.

 Container 1 (ns1)                              Container 2 (ns2)
 +-----------------------------------------+    +-------------------------+
 | +-------+      +-------+      +-------+ |    |        +-------+        |
 | | App A |      | App B |      | App C | |    |        | App D |<-+     |
 | +-------+      +---^---+      +-------+ |    |        +-------+  |(2') |
 |     |127.0.0.1 (1')|             |192.168.0.11       192.168.0.12|     |
 |  (1)|   +--------+ | +--------+  |(2)   |    | +--------+   +--------+ |
 |     `-->|   lo   |-` |  eth0  |<-`      |    | |   lo   |   |  eth0  | |
 +---------+--|---^-+---+-----|--+---------+    +-+--------+---+-^------+-+
              |   |           |                                  |
 Kernel       |   |           |                                  |
 +----+-------v---+-----------v----------------------------------+---+----+
 |    |                            TCP                               |    |
 |    |                                                              |    |
 |    +--------------------------------------------------------------+    |
 |                                                                        |
 |                           +--------------+                             |
 |                           | smc loopback |                             |
 +---------------------------+--------------+-----------------------------+

loopback-ism device allocs RMBs and sndbufs for each connection peer and 'moves' data from sndbuf at one end to RMB at the other end. Since communication occurs within the same kernel, the sndbuf can be mapped to peer RMB so that the data copy in loopback-ism case can be avoided.

 Container 1 (ns1)                              Container 2 (ns2)
 +-----------------------------------------+    +-------------------------+
 | +-------+      +-------+      +-------+ |    |        +-------+        |
 | | App A |      | App B |      | App C | |    |        | App D |        |
 | +-------+      +--^----+      +-------+ |    |        +---^---+        |
 |       |           |               |     |    |            |            |
 |   (1) |      (1') |           (2) |     |    |       (2') |            |
 |       |           |               |     |    |            |            |
 +-------|-----------|---------------|-----+    +------------|------------+
         |           |               |                       |
 Kernel  |           |               |                       |
 +-------|-----------|---------------|-----------------------|------------+
 | +-----v-+      +-------+      +---v---+               +-------+        |
 | | snd A |-+    | RMB B |<--+  | snd C |-+          +->| RMB D |        |
 | +-------+ |    +-------+   |  +-------+ |          |  +-------+        |
 | +-------+ |    +-------+   |  +-------+ |          |  +-------+        |
 | | RMB A | |    | snd B |   |  | RMB C | |          |  | snd D |        |
 | +-------+ |    +-------+   |  +-------+ |          |  +-------+        |
 |           |               +-------------v+         |                   |
 |           +-------------->| smc loopback |---------+                   |
 +---------------------------+--------------+-----------------------------+

# Benchmark Test

* Test environments:
- VM with Intel Xeon Platinum 8 core 2.50GHz, 16 GiB mem.
- SMC sndbuf/RMB size 1MB.

* Test object:
- TCP: run on TCP loopback.
- domain: run on UNIX domain.
- SMC lo: run on SMC loopback device.

1. ipc-benchmark (see [1])

- ./<foo> -c 1000000 -s 100

TCP SMC-lo
Message
rate (msg/s) 81539 151251(+85.50%)

2. sockperf

- serv: <smc_run> taskset -c <cpu> sockperf sr --tcp
- clnt: <smc_run> taskset -c <cpu> sockperf { tp | pp } --tcp --msg-size={ 64000 for tp | 14 for pp } -i 127.0.0.1 -t 30

TCP SMC-lo
Bandwidth(MBps) 5313.66 8270.51(+55.65%)
Latency(us) 5.806 3.207(-44.76%)

3. nginx/wrk

- serv: <smc_run> nginx
- clnt: <smc_run> wrk -t 8 -c 1000 -d 30 http://127.0.0.1:80

TCP SMC-lo
Requests/s 194641.79 258656.13(+32.89%)

4. redis-benchmark

- serv: <smc_run> redis-server
- clnt: <smc_run> redis-benchmark -h 127.0.0.1 -q -t set,get -n 400000 -c 200 -d 1024

TCP SMC-lo

GET(Requests/s)       85855.34                115640.35(+34.69%)
SET(Requests/s)       86337.15                118203.30(+36.90%)

[1] https://github.com/goldsborough/ipc-bench

v3->v4:
- Fix build warning of patch#12 about:
dmb_node->dma_addr = (dma_addr_t)dmb_node->cpu_addr;
- Replace kzalloc with vzalloc to alloc DMB in loopback-ism, which may cause throughput or QPS to drop by 2% to 8%. see:
https://lore.kernel.org/netdev/[email protected]/
- Add SMC_LO in Kconfig to turn on/off loopback-ism.
- Make extension GID cleaner.

v2->v3:
- Fix build warning of patch#1 and patch#10.

v1->v2:
- Fix build error on s390 arch.

Wen Gu (18):
net/smc: decouple ism_dev from SMC-D device dump net/smc: decouple ism_dev from SMC-D DMB registration net/smc: extract v2 check helper from SMC-D device registration net/smc: support SMCv2.x supplemental features negotiation net/smc: reserve CHID range for SMC-D virtual device net/smc: extend GID to 128bits only for virtual ISM device net/smc: disable SEID on non-s390 architecture net/smc: enable virtual ISM device feature bit net/smc: introduce SMC-D loopback device
net/smc: implement ID-related operations of loopback net/smc: implement some unsupported operations of loopback net/smc: implement DMB-related operations of loopback net/smc: register loopback device as SMC-Dv2 device net/smc: add operation for getting DMB attribute net/smc: add operations for DMB attach and detach net/smc: avoid data copy from sndbuf to peer RMB in SMC-D net/smc: modify cursor update logic when sndbuf mapped to RMB net/smc: add interface implementation of loopback device

 drivers/s390/net/ism_drv.c    |  20 +-
 include/net/smc.h             |  32 ++-
 include/uapi/linux/smc.h      |   3 +
 include/uapi/linux/smc_diag.h |   2 +
 net/smc/Kconfig               |  13 ++
 net/smc/Makefile              |   2 +-
 net/smc/af_smc.c              |  88 ++++++--
 net/smc/smc.h                 |   7 +
 net/smc/smc_cdc.c             |  56 ++++-
 net/smc/smc_cdc.h             |   1 +
 net/smc/smc_clc.c             |  64 ++++--
 net/smc/smc_clc.h             |  10 +-
 net/smc/smc_core.c            | 111 +++++++++-
 net/smc/smc_core.h            |   9 +-
 net/smc/smc_diag.c            |  11 +-
 net/smc/smc_ism.c             | 100 ++++++---
 net/smc/smc_ism.h             |  24 ++-
 net/smc/smc_loopback.c        | 489 ++++++++++++++++++++++++++++++++++++++++++
 net/smc/smc_loopback.h        |  54 +++++
 net/smc/smc_pnet.c            |   4 +-
 20 files changed, 996 insertions(+), 104 deletions(-)
 create mode 100644 net/smc/smc_loopback.c
 create mode 100644 net/smc/smc_loopback.h
Signature · 1 line
1.8.3.1
author reviewer
#1 lore raw

PATCH v4 1/18 net/smc: decouple ism_dev from SMC-D device dump

This patch helps to decouple ISM device from SMC-D device, allowing different underlying device forms, such as virtual ISM devices.

---
 net/smc/smc_ism.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
net/smc/smc_ism.c +4 -3
index fbee249..fc551f4 100644
@@ -230,12 +230,11 @@ static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd,
 	char smc_pnet[SMC_MAX_PNETID_LEN + 1];
 	struct smc_pci_dev smc_pci_dev;
 	struct nlattr *port_attrs;
+	struct device *priv_dev;
 	struct nlattr *attrs;
-	struct ism_dev *ism;
 	int use_cnt = 0;
 	void *nlh;
 
-	ism = smcd->priv;
 	nlh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq,
 			  &smc_gen_nl_family, NLM_F_MULTI,
 			  SMC_NETLINK_GET_DEV_SMCD);
@@ -250,7 +249,9 @@ static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd,
 	if (nla_put_u8(skb, SMC_NLA_DEV_IS_CRIT, use_cnt > 0))
 		goto errattr;
 	memset(&smc_pci_dev, 0, sizeof(smc_pci_dev));
-	smc_set_pci_values(to_pci_dev(ism->dev.parent), &smc_pci_dev);
+	priv_dev = smcd->ops->get_dev(smcd);
+	if (priv_dev && priv_dev->parent)
+		smc_set_pci_values(to_pci_dev(priv_dev->parent), &smc_pci_dev);
 	if (nla_put_u32(skb, SMC_NLA_DEV_PCI_FID, smc_pci_dev.pci_fid))
 		goto errattr;
 	if (nla_put_u16(skb, SMC_NLA_DEV_PCI_CHID, smc_pci_dev.pci_pchid))
Signature · 1 line
1.8.3.1
author reviewer
#2 lore raw

PATCH v4 2/18 net/smc: decouple ism_dev from SMC-D DMB registration

In reply to Wen Gu

This patch tries to decouple ISM device from SMC-D DMB registration, So that the register_dmb option is not restricted to ISM devices.

---
 drivers/s390/net/ism_drv.c | 2 +-
 include/net/smc.h          | 4 ++--
 net/smc/smc_ism.c          | 7 ++-----
 3 files changed, 5 insertions(+), 8 deletions(-)
drivers/s390/net/ism_drv.c +1 -1
index 6df7f37..a34e913 100644
@@ -781,7 +781,7 @@ static int smcd_query_rgid(struct smcd_dev *smcd, u64 rgid, u32 vid_valid,
 }
 
 static int smcd_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
-			     struct ism_client *client)
+			     void *client)
 {
 	return ism_register_dmb(smcd->priv, (struct ism_dmb *)dmb, client);
 }
include/net/smc.h +2 -2
index a002552..f75116e 100644
@@ -50,13 +50,12 @@ struct smcd_dmb {
 #define ISM_ERROR	0xFFFF
 
 struct smcd_dev;
-struct ism_client;
 
 struct smcd_ops {
 	int (*query_remote_gid)(struct smcd_dev *dev, u64 rgid, u32 vid_valid,
 				u32 vid);
 	int (*register_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb,
-			    struct ism_client *client);
+			    void *client);
 	int (*unregister_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb);
 	int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
 	int (*del_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
@@ -77,6 +76,7 @@ struct smcd_ops {
 struct smcd_dev {
 	const struct smcd_ops *ops;
 	void *priv;
+	void *client;
 	struct list_head list;
 	spinlock_t lock;
 	struct smc_connection **conn;
net/smc/smc_ism.c +2 -5
index fc551f4..455ae0a 100644
@@ -200,7 +200,6 @@ int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc)
 int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len,
 			 struct smc_buf_desc *dmb_desc)
 {
-#if IS_ENABLED(CONFIG_ISM)
 	struct smcd_dmb dmb;
 	int rc;
 
@@ -209,7 +208,7 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len,
 	dmb.sba_idx = dmb_desc->sba_idx;
 	dmb.vlan_id = lgr->vlan_id;
 	dmb.rgid = lgr->peer_gid;
-	rc = lgr->smcd->ops->register_dmb(lgr->smcd, &dmb, &smc_ism_client);
+	rc = lgr->smcd->ops->register_dmb(lgr->smcd, &dmb, lgr->smcd->client);
 	if (!rc) {
 		dmb_desc->sba_idx = dmb.sba_idx;
 		dmb_desc->token = dmb.dmb_tok;
@@ -218,9 +217,6 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len,
 		dmb_desc->len = dmb.dmb_len;
 	}
 	return rc;
-#else
-	return 0;
-#endif
 }
 
 static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd,
@@ -421,6 +417,7 @@ static void smcd_register_dev(struct ism_dev *ism)
 	if (!smcd)
 		return;
 	smcd->priv = ism;
+	smcd->client = &smc_ism_client;
 	ism_set_priv(ism, &smc_ism_client, smcd);
 	if (smc_pnetid_by_dev_port(&ism->pdev->dev, 0, smcd->pnetid))
 		smc_pnetid_by_table_smcd(smcd);
Signature · 1 line
1.8.3.1
author reviewer
#3 lore raw

PATCH v4 3/18 net/smc: extract v2 check helper from SMC-D device registration

In reply to Wen Gu

This patch extracts v2-capable logic from the process of registering the ISM device as an SMC-D device, so that the registration process of other underlying devices can reuse it.

---
 net/smc/smc_ism.c | 29 ++++++++++++++++++-----------
 net/smc/smc_ism.h |  1 +
 2 files changed, 19 insertions(+), 11 deletions(-)
net/smc/smc_ism.c +18 -11
index 455ae0a..8f1ba74 100644
@@ -69,6 +69,22 @@ bool smc_ism_is_v2_capable(void)
 	return smc_ism_v2_capable;
 }
 
+/* must be called under smcd_dev_list.mutex lock */
+void smc_ism_check_v2_capable(struct smcd_dev *smcd)
+{
+	u8 *system_eid = NULL;
+
+	if (smc_ism_v2_capable)
+		return;
+
+	system_eid = smcd->ops->get_system_eid();
+	if (smcd->ops->supports_v2()) {
+		smc_ism_v2_capable = true;
+		memcpy(smc_ism_v2_system_eid, system_eid,
+		       SMC_MAX_EID_LEN);
+	}
+}
+
 /* Set a connection using this DMBE. */
 void smc_ism_set_conn(struct smc_connection *conn)
 {
@@ -423,16 +439,7 @@ static void smcd_register_dev(struct ism_dev *ism)
 		smc_pnetid_by_table_smcd(smcd);
 
 	mutex_lock(&smcd_dev_list.mutex);
-	if (list_empty(&smcd_dev_list.list)) {
-		u8 *system_eid = NULL;
-
-		system_eid = smcd->ops->get_system_eid();
-		if (smcd->ops->supports_v2()) {
-			smc_ism_v2_capable = true;
-			memcpy(smc_ism_v2_system_eid, system_eid,
-			       SMC_MAX_EID_LEN);
-		}
-	}
+	smc_ism_check_v2_capable(smcd);
 	/* sort list: devices without pnetid before devices with pnetid */
 	if (smcd->pnetid[0])
 		list_add_tail(&smcd->list, &smcd_dev_list.list);
@@ -535,10 +542,10 @@ int smc_ism_init(void)
 {
 	int rc = 0;
 
-#if IS_ENABLED(CONFIG_ISM)
 	smc_ism_v2_capable = false;
 	memset(smc_ism_v2_system_eid, 0, SMC_MAX_EID_LEN);
 
+#if IS_ENABLED(CONFIG_ISM)
 	rc = ism_register_client(&smc_ism_client);
 #endif
 	return rc;
net/smc/smc_ism.h +1 -0
index 832b2f4..14d2e77 100644
@@ -42,6 +42,7 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int buf_size,
 void smc_ism_get_system_eid(u8 **eid);
 u16 smc_ism_get_chid(struct smcd_dev *dev);
 bool smc_ism_is_v2_capable(void);
+void smc_ism_check_v2_capable(struct smcd_dev *dev);
 int smc_ism_init(void);
 void smc_ism_exit(void);
 int smcd_nl_get_device(struct sk_buff *skb, struct netlink_callback *cb);
Signature · 1 line
1.8.3.1
#4 lore raw

Re: [PATCH net-next v4 03/18] net/smc: extract v2 check helper from SMC-D device registration

Quoted 42 lines · Wen Gu

This patch extracts v2-capable logic from the process of registering the ISM device as an SMC-D device, so that the registration process of other underlying devices can reuse it.

---
  net/smc/smc_ism.c | 29 ++++++++++++++++++-----------
  net/smc/smc_ism.h |  1 +
  2 files changed, 19 insertions(+), 11 deletions(-)
net/smc/smc_ism.c +16 -1
index 455ae0a..8f1ba74 100644
@@ -69,6 +69,22 @@ bool smc_ism_is_v2_capable(void)
  	return smc_ism_v2_capable;
  }
 
+/* must be called under smcd_dev_list.mutex lock */
+void smc_ism_check_v2_capable(struct smcd_dev *smcd)
+{
+	u8 *system_eid = NULL;
+
+	if (smc_ism_v2_capable)
+		return;
+
+	system_eid = smcd->ops->get_system_eid();
+	if (smcd->ops->supports_v2()) {
+		smc_ism_v2_capable = true;
+		memcpy(smc_ism_v2_system_eid, system_eid,
+		       SMC_MAX_EID_LEN);
+	}
+}
+
  /* Set a connection using this DMBE. */
  void smc_ism_set_conn(struct smc_connection *conn)
  {
@@ -423,16 +439,7 @@ static void smcd_register_dev(struct ism_dev *ism)
  		smc_pnetid_by_table_smcd(smcd);
 
  	mutex_lock(&smcd_dev_list.mutex);
-	if (list_empty(&smcd_dev_list.list)) {

net/smc/smc_ism.c static void smcd_register_dev(struct ism_dev *ism) · line 442

Quoted 10 lines
net/smc/smc_ism.c +1 -9
-		u8 *system_eid = NULL;
-
-		system_eid = smcd->ops->get_system_eid();
-		if (smcd->ops->supports_v2()) {
-			smc_ism_v2_capable = true;
-			memcpy(smc_ism_v2_system_eid, system_eid,
-			       SMC_MAX_EID_LEN);
-		}
-	}
+	smc_ism_check_v2_capable(smcd);

The list_empty check is omitted here which means the smc_ism_check_v2_capable does not touch the list. So i think the call could be placed prior the mutex_lock.

Quoted 27 lines
net/smc/smc_ism.c +1 -1
  	/* sort list: devices without pnetid before devices with pnetid */
  	if (smcd->pnetid[0])
  		list_add_tail(&smcd->list, &smcd_dev_list.list);
@@ -535,10 +542,10 @@ int smc_ism_init(void)
  {
  	int rc = 0;
 
-#if IS_ENABLED(CONFIG_ISM)
  	smc_ism_v2_capable = false;
  	memset(smc_ism_v2_system_eid, 0, SMC_MAX_EID_LEN);
 
+#if IS_ENABLED(CONFIG_ISM)
  	rc = ism_register_client(&smc_ism_client);
  #endif
  	return rc;
net/smc/smc_ism.h +1 -0
index 832b2f4..14d2e77 100644
@@ -42,6 +42,7 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int buf_size,
  void smc_ism_get_system_eid(u8 **eid);
  u16 smc_ism_get_chid(struct smcd_dev *dev);
  bool smc_ism_is_v2_capable(void);
+void smc_ism_check_v2_capable(struct smcd_dev *dev);
  int smc_ism_init(void);
  void smc_ism_exit(void);
  int smcd_nl_get_device(struct sk_buff *skb, struct netlink_callback *cb);
author reviewer
#5 lore raw

Re: [PATCH net-next v4 03/18] net/smc: extract v2 check helper from SMC-D device registration

Quoted 48 lines · Jan Karcher
Quoted 45 lines · Wen Gu

This patch extracts v2-capable logic from the process of registering the ISM device as an SMC-D device, so that the registration process of other underlying devices can reuse it.

---
  net/smc/smc_ism.c | 29 ++++++++++++++++++----------- net/smc/smc_ism.h |  1 +

  2 files changed, 19 insertions(+), 11 deletions(-)
net/smc/smc_ism.c +16 -6
index 455ae0a..8f1ba74 100644
@@ -69,6 +69,22 @@ bool smc_ism_is_v2_capable(void)
       return smc_ism_v2_capable;
   }
+/* must be called under smcd_dev_list.mutex lock */
+void smc_ism_check_v2_capable(struct smcd_dev *smcd)
+{
+    u8 *system_eid = NULL;
+
+    if (smc_ism_v2_capable)
+        return;
+
+    system_eid = smcd->ops->get_system_eid();
+    if (smcd->ops->supports_v2()) {
+        smc_ism_v2_capable = true;
+        memcpy(smc_ism_v2_system_eid, system_eid,
+               SMC_MAX_EID_LEN);
+    }
+}
+
   /* Set a connection using this DMBE. */
   void smc_ism_set_conn(struct smc_connection *conn)
   {
@@ -423,16 +439,7 @@ static void smcd_register_dev(struct ism_dev *ism)
           smc_pnetid_by_table_smcd(smcd);
       mutex_lock(&smcd_dev_list.mutex);
-    if (list_empty(&smcd_dev_list.list)) {
-        u8 *system_eid = NULL;
-
-        system_eid = smcd->ops->get_system_eid();
-        if (smcd->ops->supports_v2()) {
-            smc_ism_v2_capable = true;
Quoted 9 lines

net/smc/smc_ism.c static void smcd_register_dev(struct ism_dev *ism) · line 442

Quoted 5 lines
net/smc/smc_ism.c +1 -4
-            memcpy(smc_ism_v2_system_eid, system_eid,
-                   SMC_MAX_EID_LEN);
-        }
-    }
+    smc_ism_check_v2_capable(smcd);

The list_empty check is omitted here which means the smc_ism_check_v2_capable does not touch the list. So i think the call could be placed prior the mutex_lock.

Good catch. I omitted the list_empty check in this version but forget to remove 'the lock comments' and place the helper prior to the mutex_lock. It will be fixed.

Thank you.

Quoted 25 lines
Quoted 25 lines
net/smc/smc_ism.c +1 -1
       /* sort list: devices without pnetid before devices with pnetid */
       if (smcd->pnetid[0])
           list_add_tail(&smcd->list, &smcd_dev_list.list);
@@ -535,10 +542,10 @@ int smc_ism_init(void)
   {
       int rc = 0;
-#if IS_ENABLED(CONFIG_ISM)
       smc_ism_v2_capable = false;
       memset(smc_ism_v2_system_eid, 0, SMC_MAX_EID_LEN);
+#if IS_ENABLED(CONFIG_ISM)
       rc = ism_register_client(&smc_ism_client);
   #endif
       return rc;
net/smc/smc_ism.h +1 -0
index 832b2f4..14d2e77 100644
@@ -42,6 +42,7 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int buf_size,
   void smc_ism_get_system_eid(u8 **eid);
   u16 smc_ism_get_chid(struct smcd_dev *dev);
   bool smc_ism_is_v2_capable(void);
+void smc_ism_check_v2_capable(struct smcd_dev *dev);
   int smc_ism_init(void);
   void smc_ism_exit(void);
   int smcd_nl_get_device(struct sk_buff *skb, struct netlink_callback *cb);
author reviewer
#6 lore raw

PATCH v4 4/18 net/smc: support SMCv2.x supplemental features negotiation

In reply to Wen Gu

This patch adds SMCv2.1 supplemental features negotiation. Supported SMCv2.1 supplemental features are represented by feature_mask in FCE header of CLC messages.

Server Client

        Proposal(features(c-mask bits))
      <-----------------------------------
         Accept(features(s-mask bits))
      ----------------------------------->
        Confirm(features(s&c-mask bits))
      <-----------------------------------
---
 net/smc/smc_clc.c  | 7 +++++++
 net/smc/smc_clc.h  | 6 ++++--
 net/smc/smc_core.h | 1 +
 3 files changed, 12 insertions(+), 2 deletions(-)
net/smc/smc_clc.c +7 -0
index 8deb46c..125b0d2 100644
@@ -426,6 +426,7 @@ static int smc_clc_fill_fce(struct smc_clc_first_contact_ext_v2x *fce,
 	memset(fce, 0, sizeof(*fce));
 	fce->fce_v2_base.os_type = SMC_CLC_OS_LINUX;
 	fce->fce_v2_base.release = ini->release_nr;
+	fce->fce_v2_base.feature_mask = htons(ini->feature_mask);
 	memcpy(fce->fce_v2_base.hostname, smc_hostname, sizeof(smc_hostname));
 	if (ini->is_smcd && ini->release_nr < SMC_RELEASE_1) {
 		ret = sizeof(struct smc_clc_first_contact_ext);
@@ -906,6 +907,7 @@ int smc_clc_send_proposal(struct smc_sock *smc, struct smc_init_info *ini)
 		pclc_smcd->v2_ext_offset = htons(v2_ext_offset);
 		plen += sizeof(*v2_ext);
 
+		v2_ext->feature_mask = htons(ini->feature_mask);
 		read_lock(&smc_clc_eid_table.lock);
 		v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
 		plen += smc_clc_eid_table.ueid_cnt * SMC_MAX_EID_LEN;
@@ -1219,6 +1221,7 @@ int smc_clc_clnt_v2x_features_validate(struct smc_clc_first_contact_ext *fce,
 			return SMC_CLC_DECL_MAXLINKERR;
 		ini->max_links = fce_v2x->max_links;
 	}
+	ini->feature_mask &= ntohs(fce->feature_mask);
 
 	return 0;
 }
@@ -1250,6 +1253,10 @@ int smc_clc_v2x_features_confirm_check(struct smc_clc_msg_accept_confirm *cclc,
 			return SMC_CLC_DECL_MAXLINKERR;
 	}
 
+	if (~(ini->feature_mask) & ntohs(fce->feature_mask))
+		return SMC_CLC_DECL_FEATUNSUPP;
+	ini->feature_mask = ntohs(fce->feature_mask);
+
 	return 0;
 }
 
net/smc/smc_clc.h +4 -2
index c5c8e7d..bcf37c8 100644
@@ -48,6 +48,7 @@
 #define SMC_CLC_DECL_RELEASEERR	0x03030009  /* release version negotiate failed */
 #define SMC_CLC_DECL_MAXCONNERR	0x0303000a  /* max connections negotiate failed */
 #define SMC_CLC_DECL_MAXLINKERR	0x0303000b  /* max links negotiate failed */
+#define SMC_CLC_DECL_FEATUNSUPP	0x0303000c  /* supplemental features not supported */
 #define SMC_CLC_DECL_MODEUNSUPP	0x03040000  /* smc modes do not match (R or D)*/
 #define SMC_CLC_DECL_RMBE_EC	0x03050000  /* peer has eyecatcher in RMBE    */
 #define SMC_CLC_DECL_OPTUNSUPP	0x03060000  /* fastopen sockopt not supported */
@@ -138,7 +139,8 @@ struct smc_clc_v2_extension {
 	u8 roce[16];		/* RoCEv2 GID */
 	u8 max_conns;
 	u8 max_links;
-	u8 reserved[14];
+	__be16 feature_mask;
+	u8 reserved[12];
 	u8 user_eids[][SMC_MAX_EID_LEN];
 };
 
@@ -234,7 +236,7 @@ struct smc_clc_first_contact_ext {
 	u8 release : 4,
 	   os_type : 4;
 #endif
-	u8 reserved2[2];
+	__be16 feature_mask;
 	u8 hostname[SMC_MAX_HOSTNAME_LEN];
 };
 
net/smc/smc_core.h +1 -0
index 120027d..9f65678 100644
@@ -401,6 +401,7 @@ struct smc_init_info {
 	u8			max_links;
 	u8			first_contact_peer;
 	u8			first_contact_local;
+	u16			feature_mask;
 	unsigned short		vlan_id;
 	u32			rc;
 	u8			negotiated_eid[SMC_MAX_EID_LEN];
Signature · 1 line
1.8.3.1
author reviewer
#7 lore raw

PATCH v4 5/18 net/smc: reserve CHID range for SMC-D virtual device

In reply to Wen Gu

This patch reserve CHID range from 0xFF00 to 0xFFFF for SMC-D virtual device and introduces helpers to identify them.

---
 net/smc/smc_ism.h | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
net/smc/smc_ism.h +15 -0
index 14d2e77..2ecc8de 100644
@@ -15,6 +15,9 @@
 
 #include "smc.h"
 
+#define SMC_VIRT_ISM_CHID_MAX		0xFFFF
+#define SMC_VIRT_ISM_CHID_MIN		0xFF00
+
 struct smcd_dev_list {	/* List of SMCD devices */
 	struct list_head list;
 	struct mutex mutex;	/* Protects list of devices */
@@ -57,4 +60,16 @@ static inline int smc_ism_write(struct smcd_dev *smcd, u64 dmb_tok,
 	return rc < 0 ? rc : 0;
 }
 
+static inline bool __smc_ism_is_virtdev(u16 chid)
+{
+	return (chid >= SMC_VIRT_ISM_CHID_MIN && chid <= SMC_VIRT_ISM_CHID_MAX);
+}
+
+static inline bool smc_ism_is_virtdev(struct smcd_dev *smcd)
+{
+	u16 chid = smcd->ops->get_chid(smcd);
+
+	return __smc_ism_is_virtdev(chid);
+}
+
 #endif
Signature · 1 line
1.8.3.1
#8 lore raw

Re: [PATCH net-next v4 05/18] net/smc: reserve CHID range for SMC-D virtual device

Quoted 1 line · Wen Gu

This patch reserve CHID range from 0xFF00 to 0xFFFF for SMC-D virtual

The current state is that 0xFF00 - 0xFFFF is the range of all virtual SMC-D devices. This range devides into:
- 0xFF00 - 0xFFFE is for virto-ism
- 0xFFFF is for loopback

Quoted 12 lines

device and introduces helpers to identify them.

---
  net/smc/smc_ism.h | 15 +++++++++++++++
  1 file changed, 15 insertions(+)
net/smc/smc_ism.h +0 -0
index 14d2e77..2ecc8de 100644

net/smc/smc_ism.h line 18

Quoted 4 lines
net/smc/smc_ism.h +1 -0
@@ -15,6 +15,9 @@
 
  #include "smc.h"
 
+#define SMC_VIRT_ISM_CHID_MAX		0xFFFF

SMC_VIRT_ISM_MAX is 0xFFFE. Or do you mean virtual devices as the whole group. If yes i think that this naming will be very confusing in a few months/years. Maybe something like SMC_VIRTUAL_DEV_CHID_{MIN|MAX}?

Quoted 22 lines
net/smc/smc_ism.h +14 -0
+#define SMC_VIRT_ISM_CHID_MIN		0xFF00
+
  struct smcd_dev_list {	/* List of SMCD devices */
  	struct list_head list;
  	struct mutex mutex;	/* Protects list of devices */
@@ -57,4 +60,16 @@ static inline int smc_ism_write(struct smcd_dev *smcd, u64 dmb_tok,
  	return rc < 0 ? rc : 0;
  }
 
+static inline bool __smc_ism_is_virtdev(u16 chid)
+{
+	return (chid >= SMC_VIRT_ISM_CHID_MIN && chid <= SMC_VIRT_ISM_CHID_MAX);
+}
+
+static inline bool smc_ism_is_virtdev(struct smcd_dev *smcd)
+{
+	u16 chid = smcd->ops->get_chid(smcd);
+
+	return __smc_ism_is_virtdev(chid);
+}
+
  #endif
maintainer
#9 lore raw

Re: [PATCH net-next v4 05/18] net/smc: reserve CHID range for SMC-D virtual device

Quoted 15 lines · Jan Karcher
Quoted 1 line · Wen Gu

This patch reserve CHID range from 0xFF00 to 0xFFFF for SMC-D virtual

The current state is that 0xFF00 - 0xFFFF is the range of all virtual SMC-D devices. This range devides into:
- 0xFF00 - 0xFFFE is for virto-ism
- 0xFFFF is for loopback

Quoted 7 lines

device and introduces helpers to identify them.

---
  net/smc/smc_ism.h | 15 +++++++++++++++

  1 file changed, 15 insertions(+)
Quoted 10 lines
Quoted 7 lines
net/smc/smc_ism.h +0 -0
index 14d2e77..2ecc8de 100644
@@ -15,6 +15,9 @@
     #include "smc.h"

  +#define SMC_VIRT_ISM_CHID_MAX        0xFFFF

SMC_VIRT_ISM_MAX is 0xFFFE. Or do you mean virtual devices as the whole group. If yes i think that this naming will be very confusing in a few months/years. Maybe something like SMC_VIRTUAL_DEV_CHID_{MIN|MAX}?

IMO names are important. They can make future lives easier or harder.

Your first group of patches aims at 'decouple ISM device hard code from SMC-D stack' Maybe now would be a good point in time to decide what ISM should mean in net/smc.
a) the s390 ISM devices
b) SMC-D devices in general
I would vote for a). (today a) and b) can be found in the code, as well as the term smcd_dev)

Then like Jan wrote above:
"0xFF00 - 0xFFFF is the range of all virtual SMC-D devices" and it should NOT be called SMC_VIRT_ISM_CHID_MAX.

Then in many places in net/smc 'ism' should be replaces by 'smcd_dev' or something similar. Wen Gu, is that something you would offer to do as part of the preparation work for this series?

author reviewer
#10 lore raw

Re: [PATCH net-next v4 05/18] net/smc: reserve CHID range for SMC-D virtual device

Quoted 30 lines · Alexandra Winter
Quoted 15 lines · Jan Karcher
Quoted 1 line · Wen Gu

This patch reserve CHID range from 0xFF00 to 0xFFFF for SMC-D virtual

The current state is that 0xFF00 - 0xFFFF is the range of all virtual SMC-D devices. This range devides into:
- 0xFF00 - 0xFFFE is for virto-ism
- 0xFFFF is for loopback

Quoted 7 lines

device and introduces helpers to identify them.

---
   net/smc/smc_ism.h | 15 +++++++++++++++
   1 file changed, 15 insertions(+)
Quoted 10 lines

net/smc/smc_ism.h line 16

Quoted 7 lines
net/smc/smc_ism.h +0 -0
index 14d2e77..2ecc8de 100644
@@ -15,6 +15,9 @@
     #include "smc.h"
   +#define SMC_VIRT_ISM_CHID_MAX        0xFFFF

SMC_VIRT_ISM_MAX is 0xFFFE. Or do you mean virtual devices as the whole group. If yes i think that this naming will be very confusing in a few months/years. Maybe something like SMC_VIRTUAL_DEV_CHID_{MIN|MAX}?

Quoted 2 lines

IMO names are important. They can make future lives easier or harder.

Hi Sandy and Jan,

I agree with your opinion that names are important.

I view these terms in this way:

SMC-D devices (smcd_dev)

   |
   |- s390 ISM devices (ISM, ism_dev)
   |
   |- virtual ISM devices (virtual ISM, smc_lo_dev)
   |     |
   |     |- loopback-ism
   |     |
   |     |- virtio-ism
   |
   |- maybe future devices

SMC_VIRT_ISM_CHID_MAX was introduced to represent the maximum CHID of virtual ISM devices. CHIDs used by virtual ISM devices should be in range of [SMC_VIRT_ISM_CHID_MIN, SMC_VIRT_ISM_CHID_MAX].

I think the problem here is that SMC_VIRT_ISM_CHID_MAX might be misunderstood as CHID of virtio-ism? Then I will change them to SMC_VIRTUAL_ISM_CHID_{MAX|MIN}.

Quoted 9 lines

Your first group of patches aims at 'decouple ISM device hard code from SMC-D stack' Maybe now would be a good point in time to decide what ISM should mean in net/smc.
a) the s390 ISM devices
b) SMC-D devices in general
I would vote for a). (today a) and b) can be found in the code, as well as the term smcd_dev)

Then like Jan wrote above:
"0xFF00 - 0xFFFF is the range of all virtual SMC-D devices" and it should NOT be called SMC_VIRT_ISM_CHID_MAX.

Yes, I also vote for a).

But IMHO, loopback-ism and virtio-ism should be better classified as 'virtual ISM devices', like what describes in the specification, rather than 'virtual SMC-D devices', since they are intended to emulate ISM devices for using SMC-D on non-s390 systems.

Quoted 3 lines

Then in many places in net/smc 'ism' should be replaces by 'smcd_dev' or something similar. Wen Gu, is that something you would offer to do as part of the preparation work for this series?

Yes. But I'm not sure which 'ism' words you suggested to be replaced with 'smcd_dev'/'smcd'?

IMHO, in some generic codes like SMC-D operations (smcd_ops) or SMC-D device dump, they should be generic to all kinds of SMC-D devices, so struct ism_dev or struct ism_client should not be used, that is what patch#1 & #2 want to do.

But in some operations related to underlay device, like smcd_ism_register_dmb(), smc_ism_cantalk(), and etc in smc_ism.c. They works for both s390 ISM devices and virtual ISM devices. I think they can keep 'ism' in the helpers' name as they are now.

What do you think?

Thanks and regards,
Wen Gu

author reviewer
#11 lore raw

PATCH v4 6/18 net/smc: extend GID to 128bits only for virtual ISM device

In reply to Wen Gu

Virtual ISM devices are introduced to SMC-Dv2.1 protocal, whose GIDs are 128-bits UUIDs as defined by RFC4122. And note that the GIDs of ISM devices still remain 64-bits.

This patch adapts the relevant codes, such as CLC handshake, to make it compatible with 128 bits GID.

---
 drivers/s390/net/ism_drv.c    | 18 +++++++------
 include/net/smc.h             | 15 +++++++----
 include/uapi/linux/smc.h      |  3 +++
 include/uapi/linux/smc_diag.h |  2 ++
 net/smc/af_smc.c              | 60 +++++++++++++++++++++++++++++++++----------
 net/smc/smc_clc.c             | 43 +++++++++++++++++++++----------
 net/smc/smc_clc.h             |  4 +--
 net/smc/smc_core.c            | 41 +++++++++++++++++++++--------
 net/smc/smc_core.h            |  7 ++---
 net/smc/smc_diag.c            | 11 ++++++--
 net/smc/smc_ism.c             | 18 ++++++++-----
 net/smc/smc_ism.h             |  3 ++-
 net/smc/smc_pnet.c            |  4 +--
 13 files changed, 163 insertions(+), 66 deletions(-)
drivers/s390/net/ism_drv.c +10 -8
index a34e913..8e42eb2 100644
@@ -774,10 +774,10 @@ static void __exit ism_exit(void)
 /*************************** SMC-D Implementation *****************************/
 
 #if IS_ENABLED(CONFIG_SMC)
-static int smcd_query_rgid(struct smcd_dev *smcd, u64 rgid, u32 vid_valid,
-			   u32 vid)
+static int smcd_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid,
+			   u32 vid_valid, u32 vid)
 {
-	return ism_query_rgid(smcd->priv, rgid, vid_valid, vid);
+	return ism_query_rgid(smcd->priv, rgid->gid, vid_valid, vid);
 }
 
 static int smcd_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
@@ -811,10 +811,10 @@ static int smcd_reset_vlan_required(struct smcd_dev *smcd)
 	return ism_cmd_simple(smcd->priv, ISM_RESET_VLAN);
 }
 
-static int smcd_signal_ieq(struct smcd_dev *smcd, u64 rgid, u32 trigger_irq,
-			   u32 event_code, u64 info)
+static int smcd_signal_ieq(struct smcd_dev *smcd, struct smcd_gid *rgid,
+			   u32 trigger_irq, u32 event_code, u64 info)
 {
-	return ism_signal_ieq(smcd->priv, rgid, trigger_irq, event_code, info);
+	return ism_signal_ieq(smcd->priv, rgid->gid, trigger_irq, event_code, info);
 }
 
 static int smcd_move(struct smcd_dev *smcd, u64 dmb_tok, unsigned int idx,
@@ -830,9 +830,11 @@ static int smcd_supports_v2(void)
 		SYSTEM_EID.type[0] != '0';
 }
 
-static u64 smcd_get_local_gid(struct smcd_dev *smcd)
+static void smcd_get_local_gid(struct smcd_dev *smcd,
+			       struct smcd_gid *smcd_gid)
 {
-	return ism_get_local_gid(smcd->priv);
+	smcd_gid->gid = ism_get_local_gid(smcd->priv);
+	smcd_gid->gid_ext = 0;
 }
 
 static u16 smcd_get_chid(struct smcd_dev *smcd)
include/net/smc.h +10 -5
index f75116e..d8db5e1 100644
@@ -51,9 +51,14 @@ struct smcd_dmb {
 
 struct smcd_dev;
 
+struct smcd_gid {
+	u64	gid;
+	u64	gid_ext;
+};
+
 struct smcd_ops {
-	int (*query_remote_gid)(struct smcd_dev *dev, u64 rgid, u32 vid_valid,
-				u32 vid);
+	int (*query_remote_gid)(struct smcd_dev *dev, struct smcd_gid *rgid,
+				u32 vid_valid, u32 vid);
 	int (*register_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb,
 			    void *client);
 	int (*unregister_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb);
@@ -61,14 +66,14 @@ struct smcd_ops {
 	int (*del_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
 	int (*set_vlan_required)(struct smcd_dev *dev);
 	int (*reset_vlan_required)(struct smcd_dev *dev);
-	int (*signal_event)(struct smcd_dev *dev, u64 rgid, u32 trigger_irq,
-			    u32 event_code, u64 info);
+	int (*signal_event)(struct smcd_dev *dev, struct smcd_gid *rgid,
+			    u32 trigger_irq, u32 event_code, u64 info);
 	int (*move_data)(struct smcd_dev *dev, u64 dmb_tok, unsigned int idx,
 			 bool sf, unsigned int offset, void *data,
 			 unsigned int size);
 	int (*supports_v2)(void);
 	u8* (*get_system_eid)(void);
-	u64 (*get_local_gid)(struct smcd_dev *dev);
+	void (*get_local_gid)(struct smcd_dev *dev, struct smcd_gid *gid);
 	u16 (*get_chid)(struct smcd_dev *dev);
 	struct device* (*get_dev)(struct smcd_dev *dev);
 };
include/uapi/linux/smc.h +3 -0
index 837fcd4..0d2f020 100644
@@ -99,6 +99,9 @@ enum {
 	SMC_NLA_LGR_V2_OS,		/* u8 */
 	SMC_NLA_LGR_V2_NEG_EID,		/* string */
 	SMC_NLA_LGR_V2_PEER_HOST,	/* string */
+	SMC_NLA_LGR_V2_PAD,		/* flag */
+	SMC_NLA_LGR_V2_GID_EXT,		/* u64 */
+	SMC_NLA_LGR_V2_PEER_GID_EXT,	/* u64 */
 	__SMC_NLA_LGR_V2_MAX,
 	SMC_NLA_LGR_V2_MAX = __SMC_NLA_LGR_V2_MAX - 1
 };
include/uapi/linux/smc_diag.h +2 -0
index 8cb3a6f..78b7dfb 100644
@@ -107,6 +107,8 @@ struct smcd_diag_dmbinfo {		/* SMC-D Socket internals */
 	__aligned_u64	my_gid;		/* My GID */
 	__aligned_u64	token;		/* Token of DMB */
 	__aligned_u64	peer_token;	/* Token of remote DMBE */
+	__aligned_u64	peer_gid_ext;	/* Peer GID extension */
+	__aligned_u64	my_gid_ext;	/* My GID extension */
 };
 
 #endif /* _UAPI_SMC_DIAG_H_ */
net/smc/af_smc.c +47 -13

131 lines not on this page · raw message

net/smc/smc_clc.c +29 -14

94 lines not on this page · raw message

net/smc/smc_clc.h +2 -2
index bcf37c8..611763a 100644
@@ -281,8 +281,8 @@ struct smc_clc_msg_accept_confirm_v2 {	/* clc accept / confirm message */
 			struct smcd_clc_msg_accept_confirm_common d0;
 			__be16 chid;
 			u8 eid[SMC_MAX_EID_LEN];
-			u8 reserved5[8];
-		} d1;
+			__be64 gid_ext;
+		} __packed d1;
 	};
 };
 
net/smc/smc_core.c +31 -10

98 lines not on this page · raw message

net/smc/smc_core.h +4 -3
index 9f65678..d57eb9b 100644
@@ -17,6 +17,7 @@
 #include <linux/pci.h>
 #include <rdma/ib_verbs.h>
 #include <net/genetlink.h>
+#include <net/smc.h>
 
 #include "smc.h"
 #include "smc_ib.h"
@@ -355,7 +356,7 @@ struct smc_link_group {
 						/* max links can be added in lgr */
 		};
 		struct { /* SMC-D */
-			u64			peer_gid;
+			struct smcd_gid		peer_gid;
 						/* Peer GID (remote) */
 			struct smcd_dev		*smcd;
 						/* ISM device for VLAN reg. */
@@ -417,7 +418,7 @@ struct smc_init_info {
 	u32			ib_clcqpn;
 	struct smc_init_info_smcrv2 smcrv2;
 	/* SMC-D */
-	u64			ism_peer_gid[SMC_MAX_ISM_DEVS + 1];
+	struct smcd_gid		ism_peer_gid[SMC_MAX_ISM_DEVS + 1];
 	struct smcd_dev		*ism_dev[SMC_MAX_ISM_DEVS + 1];
 	u16			ism_chid[SMC_MAX_ISM_DEVS + 1];
 	u8			ism_offered_cnt; /* # of ISM devices offered */
@@ -545,7 +546,7 @@ static inline void smc_set_pci_values(struct pci_dev *pci_dev,
 void smc_lgr_put(struct smc_link_group *lgr);
 void smcr_port_add(struct smc_ib_device *smcibdev, u8 ibport);
 void smcr_port_err(struct smc_ib_device *smcibdev, u8 ibport);
-void smc_smcd_terminate(struct smcd_dev *dev, u64 peer_gid,
+void smc_smcd_terminate(struct smcd_dev *dev, struct smcd_gid *peer_gid,
 			unsigned short vlan);
 void smc_smcd_terminate_all(struct smcd_dev *dev);
 void smc_smcr_terminate_all(struct smc_ib_device *smcibdev);
net/smc/smc_diag.c +9 -2
index 7ff2152..a7a48b6 100644
@@ -21,6 +21,7 @@
 
 #include "smc.h"
 #include "smc_core.h"
+#include "smc_ism.h"
 
 struct smc_diag_dump_ctx {
 	int pos[2];
@@ -168,12 +169,18 @@ static int __smc_diag_dump(struct sock *sk, struct sk_buff *skb,
 		struct smc_connection *conn = &smc->conn;
 		struct smcd_diag_dmbinfo dinfo;
 		struct smcd_dev *smcd = conn->lgr->smcd;
+		struct smcd_gid smcd_gid;
 
 		memset(&dinfo, 0, sizeof(dinfo));
 
 		dinfo.linkid = *((u32 *)conn->lgr->id);
-		dinfo.peer_gid = conn->lgr->peer_gid;
-		dinfo.my_gid = smcd->ops->get_local_gid(smcd);
+		dinfo.peer_gid = conn->lgr->peer_gid.gid;
+		smcd->ops->get_local_gid(smcd, &smcd_gid);
+		dinfo.my_gid = smcd_gid.gid;
+		if (smc_ism_is_virtdev(smcd)) {
+			dinfo.peer_gid_ext = conn->lgr->peer_gid.gid_ext;
+			dinfo.my_gid_ext = smcd_gid.gid_ext;
+		}
 		dinfo.token = conn->rmb_desc->token;
 		dinfo.peer_token = conn->peer_token;
 
net/smc/smc_ism.c +12 -6
index 8f1ba74..0922fab 100644
@@ -44,7 +44,8 @@ static void smcd_handle_irq(struct ism_dev *ism, unsigned int dmbno,
 #endif
 
 /* Test if an ISM communication is possible - same CPC */
-int smc_ism_cantalk(u64 peer_gid, unsigned short vlan_id, struct smcd_dev *smcd)
+int smc_ism_cantalk(struct smcd_gid *peer_gid, unsigned short vlan_id,
+		    struct smcd_dev *smcd)
 {
 	return smcd->ops->query_remote_gid(smcd, peer_gid, vlan_id ? 1 : 0,
 					   vlan_id);
@@ -223,7 +224,7 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len,
 	dmb.dmb_len = dmb_len;
 	dmb.sba_idx = dmb_desc->sba_idx;
 	dmb.vlan_id = lgr->vlan_id;
-	dmb.rgid = lgr->peer_gid;
+	dmb.rgid = lgr->peer_gid.gid;
 	rc = lgr->smcd->ops->register_dmb(lgr->smcd, &dmb, lgr->smcd->client);
 	if (!rc) {
 		dmb_desc->sba_idx = dmb.sba_idx;
@@ -353,18 +354,20 @@ struct smc_ism_event_work {
 
 static void smcd_handle_sw_event(struct smc_ism_event_work *wrk)
 {
+	struct smcd_gid peer_gid = { .gid = wrk->event.tok,
+				     .gid_ext = 0 };
 	union smcd_sw_event_info ev_info;
 
 	ev_info.info = wrk->event.info;
 	switch (wrk->event.code) {
 	case ISM_EVENT_CODE_SHUTDOWN:	/* Peer shut down DMBs */
-		smc_smcd_terminate(wrk->smcd, wrk->event.tok, ev_info.vlan_id);
+		smc_smcd_terminate(wrk->smcd, &peer_gid, ev_info.vlan_id);
 		break;
 	case ISM_EVENT_CODE_TESTLINK:	/* Activity timer */
 		if (ev_info.code == ISM_EVENT_REQUEST) {
 			ev_info.code = ISM_EVENT_RESPONSE;
 			wrk->smcd->ops->signal_event(wrk->smcd,
-						     wrk->event.tok,
+						     &peer_gid,
 						     ISM_EVENT_REQUEST_IR,
 						     ISM_EVENT_CODE_TESTLINK,
 						     ev_info.info);
@@ -378,10 +381,13 @@ static void smc_ism_event_work(struct work_struct *work)
 {
 	struct smc_ism_event_work *wrk =
 		container_of(work, struct smc_ism_event_work, work);
+	struct smcd_gid smcd_gid = { .gid = wrk->event.tok,
+				     .gid_ext = 0 };
 
 	switch (wrk->event.type) {
 	case ISM_EVENT_GID:	/* GID event, token is peer GID */
-		smc_smcd_terminate(wrk->smcd, wrk->event.tok, VLAN_VID_MASK);
+		smcd_gid.gid = wrk->event.tok;
+		smc_smcd_terminate(wrk->smcd, &smcd_gid, VLAN_VID_MASK);
 		break;
 	case ISM_EVENT_DMB:
 		break;
@@ -530,7 +536,7 @@ int smc_ism_signal_shutdown(struct smc_link_group *lgr)
 	memcpy(ev_info.uid, lgr->id, SMC_LGR_ID_SIZE);
 	ev_info.vlan_id = lgr->vlan_id;
 	ev_info.code = ISM_EVENT_REQUEST;
-	rc = lgr->smcd->ops->signal_event(lgr->smcd, lgr->peer_gid,
+	rc = lgr->smcd->ops->signal_event(lgr->smcd, &lgr->peer_gid,
 					  ISM_EVENT_REQUEST_IR,
 					  ISM_EVENT_CODE_SHUTDOWN,
 					  ev_info.info);
net/smc/smc_ism.h +2 -1
index 2ecc8de..e6ea08c 100644
@@ -33,7 +33,8 @@ struct smc_ism_vlanid {			/* VLAN id set on ISM device */
 
 struct smcd_dev;
 
-int smc_ism_cantalk(u64 peer_gid, unsigned short vlan_id, struct smcd_dev *dev);
+int smc_ism_cantalk(struct smcd_gid *peer_gid, unsigned short vlan_id,
+		    struct smcd_dev *dev);
 void smc_ism_set_conn(struct smc_connection *conn);
 void smc_ism_unset_conn(struct smc_connection *conn);
 int smc_ism_get_vlan(struct smcd_dev *dev, unsigned short vlan_id);
net/smc/smc_pnet.c +2 -2
index 1177540..9f2c58c 100644
@@ -1103,8 +1103,8 @@ static void smc_pnet_find_ism_by_pnetid(struct net_device *ndev,
 	list_for_each_entry(ismdev, &smcd_dev_list.list, list) {
 		if (smc_pnet_match(ismdev->pnetid, ndev_pnetid) &&
 		    !ismdev->going_away &&
-		    (!ini->ism_peer_gid[0] ||
-		     !smc_ism_cantalk(ini->ism_peer_gid[0], ini->vlan_id,
+		    (!ini->ism_peer_gid[0].gid ||
+		     !smc_ism_cantalk(&ini->ism_peer_gid[0], ini->vlan_id,
 				      ismdev))) {
 			ini->ism_dev[0] = ismdev;
 			break;
Signature · 1 line
1.8.3.1
maintainer
#12 lore raw

Re: [PATCH net-next v4 06/18] net/smc: extend GID to 128bits only for virtual ISM device

Quoted 114 lines · Wen Gu

Virtual ISM devices are introduced to SMC-Dv2.1 protocal, whose GIDs are 128-bits UUIDs as defined by RFC4122. And note that the GIDs of ISM devices still remain 64-bits.

This patch adapts the relevant codes, such as CLC handshake, to make it compatible with 128 bits GID.

---

drivers/s390/net/ism_drv.c    | 18 +++++++------
include/net/smc.h             | 15 +++++++----
include/uapi/linux/smc.h      |  3 +++
include/uapi/linux/smc_diag.h |  2 ++
net/smc/af_smc.c              | 60 +++++++++++++++++++++++++++++++++----------
net/smc/smc_clc.c             | 43 +++++++++++++++++++++----------
net/smc/smc_clc.h             |  4 +--
net/smc/smc_core.c            | 41 +++++++++++++++++++++--------
net/smc/smc_core.h            |  7 ++---
net/smc/smc_diag.c            | 11 ++++++--
net/smc/smc_ism.c             | 18 ++++++++-----
net/smc/smc_ism.h             |  3 ++-
net/smc/smc_pnet.c            |  4 +--
13 files changed, 163 insertions(+), 66 deletions(-)
drivers/s390/net/ism_drv.c +10 -8
index a34e913..8e42eb2 100644
@@ -774,10 +774,10 @@ static void __exit ism_exit(void)
 /*************************** SMC-D Implementation *****************************/
 
 #if IS_ENABLED(CONFIG_SMC)
-static int smcd_query_rgid(struct smcd_dev *smcd, u64 rgid, u32 vid_valid,
-			   u32 vid)
+static int smcd_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid,
+			   u32 vid_valid, u32 vid)
 {
-	return ism_query_rgid(smcd->priv, rgid, vid_valid, vid);
+	return ism_query_rgid(smcd->priv, rgid->gid, vid_valid, vid);
 }
 
 static int smcd_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
@@ -811,10 +811,10 @@ static int smcd_reset_vlan_required(struct smcd_dev *smcd)
 	return ism_cmd_simple(smcd->priv, ISM_RESET_VLAN);
 }
 
-static int smcd_signal_ieq(struct smcd_dev *smcd, u64 rgid, u32 trigger_irq,
-			   u32 event_code, u64 info)
+static int smcd_signal_ieq(struct smcd_dev *smcd, struct smcd_gid *rgid,
+			   u32 trigger_irq, u32 event_code, u64 info)
 {
-	return ism_signal_ieq(smcd->priv, rgid, trigger_irq, event_code, info);
+	return ism_signal_ieq(smcd->priv, rgid->gid, trigger_irq, event_code, info);
 }
 
 static int smcd_move(struct smcd_dev *smcd, u64 dmb_tok, unsigned int idx,
@@ -830,9 +830,11 @@ static int smcd_supports_v2(void)
 		SYSTEM_EID.type[0] != '0';
 }
 
-static u64 smcd_get_local_gid(struct smcd_dev *smcd)
+static void smcd_get_local_gid(struct smcd_dev *smcd,
+			       struct smcd_gid *smcd_gid)
 {
-	return ism_get_local_gid(smcd->priv);
+	smcd_gid->gid = ism_get_local_gid(smcd->priv);
+	smcd_gid->gid_ext = 0;
 }
 
 static u16 smcd_get_chid(struct smcd_dev *smcd)
include/net/smc.h +10 -5
index f75116e..d8db5e1 100644
@@ -51,9 +51,14 @@ struct smcd_dmb {
 
 struct smcd_dev;
 
+struct smcd_gid {
+	u64	gid;
+	u64	gid_ext;
+};
+
 struct smcd_ops {
-	int (*query_remote_gid)(struct smcd_dev *dev, u64 rgid, u32 vid_valid,
-				u32 vid);
+	int (*query_remote_gid)(struct smcd_dev *dev, struct smcd_gid *rgid,
+				u32 vid_valid, u32 vid);
 	int (*register_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb,
 			    void *client);
 	int (*unregister_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb);
@@ -61,14 +66,14 @@ struct smcd_ops {
 	int (*del_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
 	int (*set_vlan_required)(struct smcd_dev *dev);
 	int (*reset_vlan_required)(struct smcd_dev *dev);
-	int (*signal_event)(struct smcd_dev *dev, u64 rgid, u32 trigger_irq,
-			    u32 event_code, u64 info);
+	int (*signal_event)(struct smcd_dev *dev, struct smcd_gid *rgid,
+			    u32 trigger_irq, u32 event_code, u64 info);
 	int (*move_data)(struct smcd_dev *dev, u64 dmb_tok, unsigned int idx,
 			 bool sf, unsigned int offset, void *data,
 			 unsigned int size);
 	int (*supports_v2)(void);
 	u8* (*get_system_eid)(void);
-	u64 (*get_local_gid)(struct smcd_dev *dev);
+	void (*get_local_gid)(struct smcd_dev *dev, struct smcd_gid *gid);
 	u16 (*get_chid)(struct smcd_dev *dev);
 	struct device* (*get_dev)(struct smcd_dev *dev);
 };
include/uapi/linux/smc.h +0 -0
index 837fcd4..0d2f020 100644

include/uapi/linux/smc.h enum { · line 104

Quoted 6 lines
include/uapi/linux/smc.h +3 -0
@@ -99,6 +99,9 @@ enum {
 	SMC_NLA_LGR_V2_OS,		/* u8 */
 	SMC_NLA_LGR_V2_NEG_EID,		/* string */
 	SMC_NLA_LGR_V2_PEER_HOST,	/* string */
+	SMC_NLA_LGR_V2_PAD,		/* flag */
+	SMC_NLA_LGR_V2_GID_EXT,		/* u64 */
+	SMC_NLA_LGR_V2_PEER_GID_EXT,	/* u64 */

Why those abbributes are add here, which was supposed to add common abbritubtes ?

Quoted 296 lines
include/uapi/linux/smc.h +0 -0
 	__SMC_NLA_LGR_V2_MAX,
 	SMC_NLA_LGR_V2_MAX = __SMC_NLA_LGR_V2_MAX - 1
 };
include/uapi/linux/smc_diag.h +2 -0
index 8cb3a6f..78b7dfb 100644
@@ -107,6 +107,8 @@ struct smcd_diag_dmbinfo {		/* SMC-D Socket internals */
 	__aligned_u64	my_gid;		/* My GID */
 	__aligned_u64	token;		/* Token of DMB */
 	__aligned_u64	peer_token;	/* Token of remote DMBE */
+	__aligned_u64	peer_gid_ext;	/* Peer GID extension */
+	__aligned_u64	my_gid_ext;	/* My GID extension */
 };
 
 #endif /* _UAPI_SMC_DIAG_H_ */
net/smc/af_smc.c +47 -13
index bacdd97..5bb41404 100644
@@ -719,7 +719,7 @@ static void smcd_conn_save_peer_info(struct smc_sock *smc,
 	int bufsize = smc_uncompress_bufsize(clc->d0.dmbe_size);
 
 	smc->conn.peer_rmbe_idx = clc->d0.dmbe_idx;
-	smc->conn.peer_token = clc->d0.token;
+	smc->conn.peer_token = ntohll(clc->d0.token);
 	/* msg header takes up space in the buffer */
 	smc->conn.peer_rmbe_size = bufsize - sizeof(struct smcd_cdc_msg);
 	atomic_set(&smc->conn.peer_rmbe_space, smc->conn.peer_rmbe_size);
@@ -1044,7 +1044,8 @@ static int smc_find_ism_v2_device_clnt(struct smc_sock *smc,
 {
 	int rc = SMC_CLC_DECL_NOSMCDDEV;
 	struct smcd_dev *smcd;
-	int i = 1;
+	int i = 1, slot = 1;
+	bool is_virtdev;
 	u16 chid;
 
 	if (smcd_indicated(ini->smc_type_v1))
@@ -1056,14 +1057,21 @@ static int smc_find_ism_v2_device_clnt(struct smc_sock *smc,
 		chid = smc_ism_get_chid(smcd);
 		if (!smc_find_ism_v2_is_unique_chid(chid, ini, i))
 			continue;
+		is_virtdev = __smc_ism_is_virtdev(chid);
 		if (!smc_pnet_is_pnetid_set(smcd->pnetid) ||
 		    smc_pnet_is_ndev_pnetid(sock_net(&smc->sk), smcd->pnetid)) {
+			if (is_virtdev && slot == SMC_MAX_ISM_DEVS)
+				/* no enough space for virtual ISM devices, whose GID
+				 * takes 2 slots. Try the next potential ISM device.
+				 */
+				continue;
 			ini->ism_dev[i] = smcd;
 			ini->ism_chid[i] = chid;
 			ini->is_smcd = true;
 			rc = 0;
 			i++;
-			if (i > SMC_MAX_ISM_DEVS)
+			slot = is_virtdev ? slot + 2 : slot + 1;
+			if (slot > SMC_MAX_ISM_DEVS)
 				break;
 		}
 	}
@@ -1409,8 +1417,13 @@ static int smc_connect_ism(struct smc_sock *smc,
 		rc = smc_v2_determine_accepted_chid(aclc_v2, ini);
 		if (rc)
 			return rc;
+
+		if (__smc_ism_is_virtdev(ini->ism_chid[ini->ism_selected]))
+			ini->ism_peer_gid[ini->ism_selected].gid_ext =
+						ntohll(aclc_v2->d1.gid_ext);
+		/* for non-virtual ISM devices, peer gid_ext remains 0. */
 	}
-	ini->ism_peer_gid[ini->ism_selected] = aclc->d0.gid;
+	ini->ism_peer_gid[ini->ism_selected].gid = ntohll(aclc->d0.gid);
 
 	/* there is only one lgr role for SMC-D; use server lock */
 	mutex_lock(&smc_server_lgr_pending);
@@ -2101,7 +2114,8 @@ static bool smc_is_already_selected(struct smcd_dev *smcd,
 
 /* check for ISM devices matching proposed ISM devices */
 static void smc_check_ism_v2_match(struct smc_init_info *ini,
-				   u16 proposed_chid, u64 proposed_gid,
+				   u16 proposed_chid,
+				   struct smcd_gid *proposed_gid,
 				   unsigned int *matches)
 {
 	struct smcd_dev *smcd;
@@ -2113,7 +2127,11 @@ static void smc_check_ism_v2_match(struct smc_init_info *ini,
 			continue;
 		if (smc_ism_get_chid(smcd) == proposed_chid &&
 		    !smc_ism_cantalk(proposed_gid, ISM_RESERVED_VLANID, smcd)) {
-			ini->ism_peer_gid[*matches] = proposed_gid;
+			ini->ism_peer_gid[*matches].gid = proposed_gid->gid;
+			if (__smc_ism_is_virtdev(proposed_chid))
+				ini->ism_peer_gid[*matches].gid_ext =
+							proposed_gid->gid_ext;
+				/* non-virtual ISM's peer gid_ext remains 0. */
 			ini->ism_dev[*matches] = smcd;
 			(*matches)++;
 			break;
@@ -2135,9 +2153,11 @@ static void smc_find_ism_v2_device_serv(struct smc_sock *new_smc,
 	struct smc_clc_v2_extension *smc_v2_ext;
 	struct smc_clc_msg_smcd *pclc_smcd;
 	unsigned int matches = 0;
+	struct smcd_gid smcd_gid;
 	u8 smcd_version;
 	u8 *eid = NULL;
 	int i, rc;
+	u16 chid;
 
 	if (!(ini->smcd_version & SMC_V2) || !smcd_indicated(ini->smc_type_v2))
 		goto not_found;
@@ -2147,18 +2167,31 @@ static void smc_find_ism_v2_device_serv(struct smc_sock *new_smc,
 	smcd_v2_ext = smc_get_clc_smcd_v2_ext(smc_v2_ext);
 
 	mutex_lock(&smcd_dev_list.mutex);
-	if (pclc_smcd->ism.chid)
+	if (pclc_smcd->ism.chid) {
 		/* check for ISM device matching proposed native ISM device */
+		smcd_gid.gid = ntohll(pclc_smcd->ism.gid);
+		smcd_gid.gid_ext = 0;
 		smc_check_ism_v2_match(ini, ntohs(pclc_smcd->ism.chid),
-				       ntohll(pclc_smcd->ism.gid), &matches);
+				       &smcd_gid, &matches);
+	}
 	for (i = 1; i <= smc_v2_ext->hdr.ism_gid_cnt; i++) {
 		/* check for ISM devices matching proposed non-native ISM
 		 * devices
 		 */
-		smc_check_ism_v2_match(ini,
-				       ntohs(smcd_v2_ext->gidchid[i - 1].chid),
-				       ntohll(smcd_v2_ext->gidchid[i - 1].gid),
-				       &matches);
+		smcd_gid.gid = ntohll(smcd_v2_ext->gidchid[i - 1].gid);
+		smcd_gid.gid_ext = 0;
+		chid = ntohs(smcd_v2_ext->gidchid[i - 1].chid);
+		if (__smc_ism_is_virtdev(chid)) {
+			if (i >= smc_v2_ext->hdr.ism_gid_cnt ||
+			    chid != ntohs(smcd_v2_ext->gidchid[i].chid))
+				/* check if next slot exists and the next slot's
+				 * chid is consistent with the current slot.
+				 */
+				continue;
+
+			smcd_gid.gid_ext = ntohll(smcd_v2_ext->gidchid[i++].gid);
+		}
+		smc_check_ism_v2_match(ini, chid, &smcd_gid, &matches);
 	}
 	mutex_unlock(&smcd_dev_list.mutex);
 
@@ -2207,7 +2240,8 @@ static void smc_find_ism_v1_device_serv(struct smc_sock *new_smc,
 	if (!(ini->smcd_version & SMC_V1) || !smcd_indicated(ini->smc_type_v1))
 		goto not_found;
 	ini->is_smcd = true; /* prepare ISM check */
-	ini->ism_peer_gid[0] = ntohll(pclc_smcd->ism.gid);
+	ini->ism_peer_gid[0].gid = ntohll(pclc_smcd->ism.gid);
+	ini->ism_peer_gid[0].gid_ext = 0;
 	rc = smc_find_ism_device(new_smc, ini);
 	if (rc)
 		goto not_found;
net/smc/smc_clc.c +29 -14
index 125b0d2..aeb9643 100644
@@ -882,11 +882,13 @@ int smc_clc_send_proposal(struct smc_sock *smc, struct smc_init_info *ini)
 		       ETH_ALEN);
 	}
 	if (smcd_indicated(ini->smc_type_v1)) {
+		struct smcd_gid smcd_gid;
+
 		/* add SMC-D specifics */
 		if (ini->ism_dev[0]) {
 			smcd = ini->ism_dev[0];
-			pclc_smcd->ism.gid =
-				htonll(smcd->ops->get_local_gid(smcd));
+			smcd->ops->get_local_gid(smcd, &smcd_gid);
+			pclc_smcd->ism.gid = htonll(smcd_gid.gid);
 			pclc_smcd->ism.chid =
 				htons(smc_ism_get_chid(ini->ism_dev[0]));
 		}
@@ -919,10 +921,11 @@ int smc_clc_send_proposal(struct smc_sock *smc, struct smc_init_info *ini)
 		read_unlock(&smc_clc_eid_table.lock);
 	}
 	if (smcd_indicated(ini->smc_type_v2)) {
+		struct smcd_gid smcd_gid;
 		u8 *eid = NULL;
+		int slot = 0;
 
 		v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
-		v2_ext->hdr.ism_gid_cnt = ini->ism_offered_cnt;
 		v2_ext->hdr.smcd_v2_ext_offset = htons(sizeof(*v2_ext) -
 				offsetofend(struct smc_clnt_opts_area_hdr,
 					    smcd_v2_ext_offset) +
@@ -934,14 +937,21 @@ int smc_clc_send_proposal(struct smc_sock *smc, struct smc_init_info *ini)
 		if (ini->ism_offered_cnt) {
 			for (i = 1; i <= ini->ism_offered_cnt; i++) {
 				smcd = ini->ism_dev[i];
-				gidchids[i - 1].gid =
-					htonll(smcd->ops->get_local_gid(smcd));
-				gidchids[i - 1].chid =
+				smcd->ops->get_local_gid(smcd, &smcd_gid);
+				gidchids[slot].chid =
 					htons(smc_ism_get_chid(ini->ism_dev[i]));
+				gidchids[slot].gid = htonll(smcd_gid.gid);
+				if (__smc_ism_is_virtdev(gidchids[slot].chid)) {
+					/* virtual-ism takes two slots */
+					gidchids[slot + 1].chid = gidchids[slot].chid;
+					gidchids[slot + 1].gid = htonll(smcd_gid.gid_ext);
+					slot++;
+				}
+				slot++;
 			}
-			plen += ini->ism_offered_cnt *
-				sizeof(struct smc_clc_smcd_gid_chid);
+			plen += slot * sizeof(struct smc_clc_smcd_gid_chid);
 		}
+		v2_ext->hdr.ism_gid_cnt = slot;
 	}
 	if (smcr_indicated(ini->smc_type_v2)) {
 		memcpy(v2_ext->roce, ini->smcrv2.ib_gid_v2, SMC_GID_SIZE);
@@ -977,7 +987,7 @@ int smc_clc_send_proposal(struct smc_sock *smc, struct smc_init_info *ini)
 			vec[i++].iov_len = sizeof(*smcd_v2_ext);
 			if (ini->ism_offered_cnt) {
 				vec[i].iov_base = gidchids;
-				vec[i++].iov_len = ini->ism_offered_cnt *
+				vec[i++].iov_len = v2_ext->hdr.ism_gid_cnt *
 					sizeof(struct smc_clc_smcd_gid_chid);
 			}
 		}
@@ -1019,23 +1029,28 @@ static int smc_clc_send_confirm_accept(struct smc_sock *smc,
 	if (first_contact)
 		clc->hdr.typev2 |= SMC_FIRST_CONTACT_MASK;
 	if (conn->lgr->is_smcd) {
+		struct smcd_gid smcd_gid;
+		u16 chid;
+
 		/* SMC-D specific settings */
 		memcpy(clc->hdr.eyecatcher, SMCD_EYECATCHER,
 		       sizeof(SMCD_EYECATCHER));
+		conn->lgr->smcd->ops->get_local_gid(conn->lgr->smcd, &smcd_gid);
 		clc->hdr.typev1 = SMC_TYPE_D;
-		clc->d0.gid =
-			conn->lgr->smcd->ops->get_local_gid(conn->lgr->smcd);
-		clc->d0.token = conn->rmb_desc->token;
+		clc->d0.gid = htonll(smcd_gid.gid);
+		clc->d0.token = htonll(conn->rmb_desc->token);
 		clc->d0.dmbe_size = conn->rmbe_size_comp;
 		clc->d0.dmbe_idx = 0;
 		memcpy(&clc->d0.linkid, conn->lgr->id, SMC_LGR_ID_SIZE);
 		if (version == SMC_V1) {
 			clc->hdr.length = htons(SMCD_CLC_ACCEPT_CONFIRM_LEN);
 		} else {
-			clc_v2->d1.chid =
-				htons(smc_ism_get_chid(conn->lgr->smcd));
+			chid = smc_ism_get_chid(conn->lgr->smcd);
+			clc_v2->d1.chid = htons(chid);
 			if (eid && eid[0])
 				memcpy(clc_v2->d1.eid, eid, SMC_MAX_EID_LEN);
+			if (__smc_ism_is_virtdev(chid))
+				clc_v2->d1.gid_ext = htonll(smcd_gid.gid_ext);
 			len = SMCD_CLC_ACCEPT_CONFIRM_LEN_V2;
 			if (first_contact) {
 				fce_len = smc_clc_fill_fce(&fce, ini);
net/smc/smc_clc.h +2 -2
index bcf37c8..611763a 100644
@@ -281,8 +281,8 @@ struct smc_clc_msg_accept_confirm_v2 {	/* clc accept / confirm message */
 			struct smcd_clc_msg_accept_confirm_common d0;
 			__be16 chid;
 			u8 eid[SMC_MAX_EID_LEN];
-			u8 reserved5[8];
-		} d1;
+			__be64 gid_ext;
+		} __packed d1;
 	};
 };
 
net/smc/smc_core.c +3 -0
index d520ee6..6d7c738 100644
@@ -284,6 +284,9 @@ static int smc_nl_fill_lgr_v2_common(struct smc_link_group *lgr,
 {
 	char smc_host[SMC_MAX_HOSTNAME_LEN + 1];
 	char smc_eid[SMC_MAX_EID_LEN + 1];
+	struct smcd_dev *smcd = lgr->smcd;
+	struct smcd_gid smcd_gid;
+	bool is_virtdev;
 
 	if (nla_put_u8(skb, SMC_NLA_LGR_V2_VER, lgr->smc_version))
 		goto errv2attr;
@@ -299,6 +302,16 @@ static int smc_nl_fill_lgr_v2_common(struct smc_link_group *lgr,
 	smc_eid[SMC_MAX_EID_LEN] = 0;
 	if (nla_put_string(skb, SMC_NLA_LGR_V2_NEG_EID, smc_eid))
 		goto errv2attr;

net/smc/smc_core.c static int smc_nl_fill_lgr_v2_common(struct smc_link_group *lgr, · line 314

Quoted 10 lines
net/smc/smc_core.c +10 -0
+	smcd->ops->get_local_gid(smcd, &smcd_gid);
+	is_virtdev = smc_ism_is_virtdev(smcd);
+	if (nla_put_u64_64bit(skb, SMC_NLA_LGR_V2_GID_EXT,
+			      is_virtdev ? smcd_gid.gid_ext : 0,
+			      SMC_NLA_LGR_V2_PAD))
+		goto errv2attr;
+	if (nla_put_u64_64bit(skb, SMC_NLA_LGR_V2_PEER_GID_EXT,
+			      is_virtdev ? lgr->peer_gid.gid_ext : 0,
+			      SMC_NLA_LGR_V2_PAD))
+		goto errv2attr;

I ran into a kernel panic which pointed to here, and it turns out the smcd here is NULL. See below:

But taking a closer look at the code, I'm wondering why those SMCD related attributes are filled in smc_nl_fill_lgr_v2_common() which should only fill the common attributes ?

[17567.395214] BUG: kernel NULL pointer dereference, address:
0000000000000000
[17567.395729] #PF: supervisor read access in kernel mode
[17567.396086] #PF: error_code(0x0000) - not-present page
[17567.396442] PGD 0 P4D 0 
[17567.396623] Oops: 0000 [#1] SMP NOPTI
[17567.396873] CPU: 7 PID: 34729 Comm: smcr Tainted: G        W   E
6.6.0-rc2-00669-gf4cfa8873d90-dirty #577
[17567.397528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[17567.398272] RIP: 0010:smc_nl_fill_lgr_v2_common.isra.0+0x1e0/0x2e0
[smc]
[17567.398734] Code: 21 0f 87 0e 01 00 00 0f 84 fc 00 00 00 8d 50 01 48
8d 4c 24 3f be 03 00 00 00 48 89 ef e8 28 c4 1a e1 85 c0 0f 85 72 fe ff
ff <49> 8b 45 00 4c 89 ef 48 8d 74 24 08 48 8b 40 68 ff d0 0f 1f 00 49
[17567.399950] RSP: 0018:ffffc90002adb820 EFLAGS: 00010246
[17567.400297] RAX: 0000000000000000 RBX: ffff8881cad2402c RCX:
0000000000000057
[17567.400769] RDX: 0000000000000000 RSI: ffffc90002adb85f RDI:
ffff8881cad24074
[17567.401237] RBP: ffff888107d4cd00 R08: 0020202020202020 R09:
2020202020202020
[17567.401709] R10: 2020202020202020 R11: 2020202020415955 R12:
ffff8882673e0000
[17567.402176] R13: 0000000000000000 R14: ffff8881cad23fe4 R15:
ffff88813fbe4b60
[17567.402650] FS:  00007f679f655740(0000) GS:ffff88842fdc0000(0000)
knlGS:0000000000000000
[17567.403179] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[17567.403561] CR2: 0000000000000000 CR3: 00000001680d2005 CR4:
0000000000370ee0
[17567.404029] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[17567.404501] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[17567.404968] Call Trace:
[17567.405140]  <TASK>
[17567.405287]  ? __die_body+0x1f/0x70
[17567.405533]  ? page_fault_oops+0x14c/0x440
[17567.405813]  ? exc_page_fault+0x69/0x120
[17567.406079]  ? asm_exc_page_fault+0x26/0x30
[17567.406361]  ? smc_nl_fill_lgr_v2_common.isra.0+0x1e0/0x2e0 [smc]
[17567.406778]  smc_nl_fill_lgr_list.constprop.0+0x368/0x4e0 [smc]
[17567.407180]  smcr_nl_get_link+0x17/0x20 [smc]
[17567.407484]  genl_dumpit+0x32/0x90
[17567.407722]  netlink_dump+0x19d/0x3b0
[17567.407971]  __netlink_dump_start+0x1d3/0x290
[17567.408266]  genl_family_rcv_msg_dumpit.isra.0+0x7d/0xd0
[17567.408625]  ? __pfx_genl_start+0x10/0x10
[17567.408897]  ? __pfx_genl_dumpit+0x10/0x10
[17567.409172]  ? __pfx_genl_done+0x10/0x10
[17567.409437]  genl_rcv_msg+0x113/0x2a0
[17567.409690]  ? __pfx_smcr_nl_get_link+0x10/0x10 [smc]
[17567.410040]  ? __pfx_genl_rcv_msg+0x10/0x10
[17567.410312]  netlink_rcv_skb+0x58/0x110
[17567.410567]  genl_rcv+0x28/0x40
[17567.410777]  netlink_unicast+0x181/0x240
[17567.411033]  netlink_sendmsg+0x240/0x4a0
[17567.411288]  sock_sendmsg+0xb1/0xc0
[17567.411524]  ____sys_sendmsg+0x20f/0x300
[17567.411780]  ? copy_msghdr_from_user+0x62/0x80
[17567.412067]  ___sys_sendmsg+0x81/0xc0
[17567.412306]  ? folio_add_lru+0x2b/0x30
[17567.412555]  ? do_anonymous_page+0x18d/0x4e0
[17567.412837]  ? __handle_mm_fault+0x47f/0x7c0
[17567.413115]  __sys_sendmsg+0x4d/0x80
[17567.413349]  ? exit_to_user_mode_prepare+0x3c/0x190
[17567.413669]  do_syscall_64+0x3c/0x90
[17567.413908]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[17567.414234] RIP: 0033:0x7f679f79c177
[17567.414467] Code: 0c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f
1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f
05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10
[17567.415640] RSP: 002b:00007ffe6d08c168 EFLAGS: 00000246 ORIG_RAX:
000000000000002e
[17567.416119] RAX: ffffffffffffffda RBX: 00005619cb470390 RCX:
00007f679f79c177
[17567.416574] RDX: 0000000000000000 RSI: 00007ffe6d08c1a0 RDI:
0000000000000003
[17567.417026] RBP: 00005619cb4702a0 R08: 0000000000000004 R09:
0000000000000300
[17567.417481] R10: 0000000000000004 R11: 0000000000000246 R12:
00005619cb4704b0
[17567.417932] R13: 00007ffe6d08c1a0 R14: 0000000000000000 R15:
0000000000000000
[17567.418385]  </TASK>
[17567.418535] Modules linked in: smc_diag(E) smc(E) rpcrdma(E)
sunrpc(E) ib_srpt(E) ib_isert(E) iscsi_target_mod(E) target_core_mod(E)
ib_ipoib(E) ib_iser(E) libiscsi(E) scsi_transport_iscsi(E) mlx5_ib(E)
rfkill(E)
[17567.419722] CR2: 0000000000000000
[17567.419940] ---[ end trace 0000000000000000 ]---
[17567.420237] RIP: 0010:smc_nl_fill_lgr_v2_common.isra.0+0x1e0/0x2e0
[smc]
[17567.420676] Code: 21 0f 87 0e 01 00 00 0f 84 fc 00 00 00 8d 50 01 48
8d 4c 24 3f be 03 00 00 00 48 89 ef e8 28 c4 1a e1 85 c0 0f 85 72 fe ff
ff <49> 8b 45 00 4c 89 ef 48 8d 74 24 08 48 8b 40 68 ff d0 0f 1f 00 49
[17567.421842] RSP: 0018:ffffc90002adb820 EFLAGS: 00010246
[17567.422176] RAX: 0000000000000000 RBX: ffff8881cad2402c RCX:
0000000000000057
[17567.422631] RDX: 0000000000000000 RSI: ffffc90002adb85f RDI:
ffff8881cad24074
[17567.423082] RBP: ffff888107d4cd00 R08: 0020202020202020 R09:
2020202020202020
[17567.423536] R10: 2020202020202020 R11: 2020202020415955 R12:
ffff8882673e0000
[17567.423987] R13: 0000000000000000 R14: ffff8881cad23fe4 R15:
ffff88813fbe4b60
[17567.424438] FS:  00007f679f655740(0000) GS:ffff88842fdc0000(0000)
knlGS:0000000000000000
[17567.424950] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[17567.425317] CR2: 0000000000000000 CR3: 00000001680d2005 CR4:
0000000000370ee0
[17567.425773] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[17567.426225] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[17567.426677] Kernel panic - not syncing: Fatal exception in interrupt
[17567.427583] Kernel Offset: disabled
[17567.427817] ---[ end Kernel panic - not syncing: Fatal exception in

interrupt ]---

Best regards,
Dust

Quoted 3 lines
net/smc/smc_core.c +0 -0
 	nla_nest_end(skb, v2_attrs);
 	return 0;
author reviewer
#13 lore raw

Re: [PATCH net-next v4 06/18] net/smc: extend GID to 128bits only for virtual ISM device

Quoted 25 lines · Dust Li
Quoted 24 lines · Wen Gu

Virtual ISM devices are introduced to SMC-Dv2.1 protocal, whose GIDs are 128-bits UUIDs as defined by RFC4122. And note that the GIDs of ISM devices still remain 64-bits.

This patch adapts the relevant codes, such as CLC handshake, to make it compatible with 128 bits GID.

---

drivers/s390/net/ism_drv.c    | 18 +++++++------
include/net/smc.h             | 15 +++++++----
include/uapi/linux/smc.h      |  3 +++
include/uapi/linux/smc_diag.h |  2 ++
net/smc/af_smc.c              | 60 +++++++++++++++++++++++++++++++++----------
net/smc/smc_clc.c             | 43 +++++++++++++++++++++----------
net/smc/smc_clc.h             |  4 +--
net/smc/smc_core.c            | 41 +++++++++++++++++++++--------
net/smc/smc_core.h            |  7 ++---
net/smc/smc_diag.c            | 11 ++++++--
net/smc/smc_ism.c             | 18 ++++++++-----
net/smc/smc_ism.h             |  3 ++-
net/smc/smc_pnet.c            |  4 +--
13 files changed, 163 insertions(+), 66 deletions(-)

<...>

Quoted 6 lines
Quoted 6 lines
include/uapi/linux/smc.h +0 -0
index 837fcd4..0d2f020 100644
@@ -99,6 +99,9 @@ enum {
 	SMC_NLA_LGR_V2_OS,		/* u8 */
Quoted 9 lines

include/uapi/linux/smc.h enum { · line 104

Quoted 5 lines
include/uapi/linux/smc.h +3 -0
 	SMC_NLA_LGR_V2_NEG_EID,		/* string */
 	SMC_NLA_LGR_V2_PEER_HOST,	/* string */
+	SMC_NLA_LGR_V2_PAD,		/* flag */
+	SMC_NLA_LGR_V2_GID_EXT,		/* u64 */
+	SMC_NLA_LGR_V2_PEER_GID_EXT,	/* u64 */

Why those abbributes are add here, which was supposed to add common abbritubtes ?

Thanks! After looking back at this code, I also feel that the extended GID attributes should be moved to other places. The nested attributes here are shared by both SMC-R and SMC-D.

So similar to the SMC_NLA_LGR_R_V2 nested attributes introduced by SMC-R, a new SMC_NLA_LGR_D_V2 nested attributes may be more suitable,
and processed by a new smc_nl_fill_smcr_lgr_v2() helper in smc_nl_fill_smcd_lgr().

<...>

Quoted 32 lines
Quoted 18 lines
net/smc/smc_core.c +3 -0
index d520ee6..6d7c738 100644
@@ -284,6 +284,9 @@ static int smc_nl_fill_lgr_v2_common(struct smc_link_group *lgr,
 {
 	char smc_host[SMC_MAX_HOSTNAME_LEN + 1];
 	char smc_eid[SMC_MAX_EID_LEN + 1];
+	struct smcd_dev *smcd = lgr->smcd;
+	struct smcd_gid smcd_gid;
+	bool is_virtdev;
 
 	if (nla_put_u8(skb, SMC_NLA_LGR_V2_VER, lgr->smc_version))
 		goto errv2attr;
@@ -299,6 +302,16 @@ static int smc_nl_fill_lgr_v2_common(struct smc_link_group *lgr,
 	smc_eid[SMC_MAX_EID_LEN] = 0;
 	if (nla_put_string(skb, SMC_NLA_LGR_V2_NEG_EID, smc_eid))
 		goto errv2attr;

net/smc/smc_core.c static int smc_nl_fill_lgr_v2_common(struct smc_link_group *lgr, · line 314

Quoted 10 lines
net/smc/smc_core.c +10 -0
+	smcd->ops->get_local_gid(smcd, &smcd_gid);
+	is_virtdev = smc_ism_is_virtdev(smcd);
+	if (nla_put_u64_64bit(skb, SMC_NLA_LGR_V2_GID_EXT,
+			      is_virtdev ? smcd_gid.gid_ext : 0,
+			      SMC_NLA_LGR_V2_PAD))
+		goto errv2attr;
+	if (nla_put_u64_64bit(skb, SMC_NLA_LGR_V2_PEER_GID_EXT,
+			      is_virtdev ? lgr->peer_gid.gid_ext : 0,
+			      SMC_NLA_LGR_V2_PAD))
+		goto errv2attr;

I ran into a kernel panic which pointed to here, and it turns out the smcd here is NULL. See below:

Quoted 5 lines

But taking a closer look at the code, I'm wondering why those SMCD related attributes are filled in smc_nl_fill_lgr_v2_common() which should only fill the common attributes ?

I guess the crash occurs when running in SMC-R. Due to this improper code, the invalid lgr->smcd was accessed mistakenly in the common helper.

So as mentioned above, this extended GID related code will be moved out of this common helper and fix this issue. Thanks!

Regards,
Wen Gu

Quoted 132 lines
[17567.395214] BUG: kernel NULL pointer dereference, address:
0000000000000000
[17567.395729] #PF: supervisor read access in kernel mode
[17567.396086] #PF: error_code(0x0000) - not-present page
[17567.396442] PGD 0 P4D 0
[17567.396623] Oops: 0000 [#1] SMP NOPTI
[17567.396873] CPU: 7 PID: 34729 Comm: smcr Tainted: G        W   E
6.6.0-rc2-00669-gf4cfa8873d90-dirty #577
[17567.397528] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[17567.398272] RIP: 0010:smc_nl_fill_lgr_v2_common.isra.0+0x1e0/0x2e0
[smc]
[17567.398734] Code: 21 0f 87 0e 01 00 00 0f 84 fc 00 00 00 8d 50 01 48
8d 4c 24 3f be 03 00 00 00 48 89 ef e8 28 c4 1a e1 85 c0 0f 85 72 fe ff
ff <49> 8b 45 00 4c 89 ef 48 8d 74 24 08 48 8b 40 68 ff d0 0f 1f 00 49
[17567.399950] RSP: 0018:ffffc90002adb820 EFLAGS: 00010246
[17567.400297] RAX: 0000000000000000 RBX: ffff8881cad2402c RCX:
0000000000000057
[17567.400769] RDX: 0000000000000000 RSI: ffffc90002adb85f RDI:
ffff8881cad24074
[17567.401237] RBP: ffff888107d4cd00 R08: 0020202020202020 R09:
2020202020202020
[17567.401709] R10: 2020202020202020 R11: 2020202020415955 R12:
ffff8882673e0000
[17567.402176] R13: 0000000000000000 R14: ffff8881cad23fe4 R15:
ffff88813fbe4b60
[17567.402650] FS:  00007f679f655740(0000) GS:ffff88842fdc0000(0000)
knlGS:0000000000000000
[17567.403179] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[17567.403561] CR2: 0000000000000000 CR3: 00000001680d2005 CR4:
0000000000370ee0
[17567.404029] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[17567.404501] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[17567.404968] Call Trace:
[17567.405140]  <TASK>
[17567.405287]  ? __die_body+0x1f/0x70
[17567.405533]  ? page_fault_oops+0x14c/0x440
[17567.405813]  ? exc_page_fault+0x69/0x120
[17567.406079]  ? asm_exc_page_fault+0x26/0x30
[17567.406361]  ? smc_nl_fill_lgr_v2_common.isra.0+0x1e0/0x2e0 [smc]
[17567.406778]  smc_nl_fill_lgr_list.constprop.0+0x368/0x4e0 [smc]
[17567.407180]  smcr_nl_get_link+0x17/0x20 [smc]
[17567.407484]  genl_dumpit+0x32/0x90
[17567.407722]  netlink_dump+0x19d/0x3b0
[17567.407971]  __netlink_dump_start+0x1d3/0x290
[17567.408266]  genl_family_rcv_msg_dumpit.isra.0+0x7d/0xd0
[17567.408625]  ? __pfx_genl_start+0x10/0x10
[17567.408897]  ? __pfx_genl_dumpit+0x10/0x10
[17567.409172]  ? __pfx_genl_done+0x10/0x10
[17567.409437]  genl_rcv_msg+0x113/0x2a0
[17567.409690]  ? __pfx_smcr_nl_get_link+0x10/0x10 [smc]
[17567.410040]  ? __pfx_genl_rcv_msg+0x10/0x10
[17567.410312]  netlink_rcv_skb+0x58/0x110
[17567.410567]  genl_rcv+0x28/0x40
[17567.410777]  netlink_unicast+0x181/0x240
[17567.411033]  netlink_sendmsg+0x240/0x4a0
[17567.411288]  sock_sendmsg+0xb1/0xc0
[17567.411524]  ____sys_sendmsg+0x20f/0x300
[17567.411780]  ? copy_msghdr_from_user+0x62/0x80
[17567.412067]  ___sys_sendmsg+0x81/0xc0
[17567.412306]  ? folio_add_lru+0x2b/0x30
[17567.412555]  ? do_anonymous_page+0x18d/0x4e0
[17567.412837]  ? __handle_mm_fault+0x47f/0x7c0
[17567.413115]  __sys_sendmsg+0x4d/0x80
[17567.413349]  ? exit_to_user_mode_prepare+0x3c/0x190
[17567.413669]  do_syscall_64+0x3c/0x90
[17567.413908]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[17567.414234] RIP: 0033:0x7f679f79c177
[17567.414467] Code: 0c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f
1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 2e 00 00 00 0f
05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 89 54 24 1c 48 89 74 24 10
[17567.415640] RSP: 002b:00007ffe6d08c168 EFLAGS: 00000246 ORIG_RAX:
000000000000002e
[17567.416119] RAX: ffffffffffffffda RBX: 00005619cb470390 RCX:
00007f679f79c177
[17567.416574] RDX: 0000000000000000 RSI: 00007ffe6d08c1a0 RDI:
0000000000000003
[17567.417026] RBP: 00005619cb4702a0 R08: 0000000000000004 R09:
0000000000000300
[17567.417481] R10: 0000000000000004 R11: 0000000000000246 R12:
00005619cb4704b0
[17567.417932] R13: 00007ffe6d08c1a0 R14: 0000000000000000 R15:
0000000000000000
[17567.418385]  </TASK>
[17567.418535] Modules linked in: smc_diag(E) smc(E) rpcrdma(E)
sunrpc(E) ib_srpt(E) ib_isert(E) iscsi_target_mod(E) target_core_mod(E)
ib_ipoib(E) ib_iser(E) libiscsi(E) scsi_transport_iscsi(E) mlx5_ib(E)
rfkill(E)
[17567.419722] CR2: 0000000000000000
[17567.419940] ---[ end trace 0000000000000000 ]---
[17567.420237] RIP: 0010:smc_nl_fill_lgr_v2_common.isra.0+0x1e0/0x2e0
[smc]
[17567.420676] Code: 21 0f 87 0e 01 00 00 0f 84 fc 00 00 00 8d 50 01 48
8d 4c 24 3f be 03 00 00 00 48 89 ef e8 28 c4 1a e1 85 c0 0f 85 72 fe ff
ff <49> 8b 45 00 4c 89 ef 48 8d 74 24 08 48 8b 40 68 ff d0 0f 1f 00 49
[17567.421842] RSP: 0018:ffffc90002adb820 EFLAGS: 00010246
[17567.422176] RAX: 0000000000000000 RBX: ffff8881cad2402c RCX:
0000000000000057
[17567.422631] RDX: 0000000000000000 RSI: ffffc90002adb85f RDI:
ffff8881cad24074
[17567.423082] RBP: ffff888107d4cd00 R08: 0020202020202020 R09:
2020202020202020
[17567.423536] R10: 2020202020202020 R11: 2020202020415955 R12:
ffff8882673e0000
[17567.423987] R13: 0000000000000000 R14: ffff8881cad23fe4 R15:
ffff88813fbe4b60
[17567.424438] FS:  00007f679f655740(0000) GS:ffff88842fdc0000(0000)
knlGS:0000000000000000
[17567.424950] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[17567.425317] CR2: 0000000000000000 CR3: 00000001680d2005 CR4:
0000000000370ee0
[17567.425773] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[17567.426225] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[17567.426677] Kernel panic - not syncing: Fatal exception in interrupt
[17567.427583] Kernel Offset: disabled
[17567.427817] ---[ end Kernel panic - not syncing: Fatal exception in

interrupt ]---

Best regards,
Dust

Quoted 3 lines
net/smc/smc_core.c +0 -0
 	nla_nest_end(skb, v2_attrs);
 	return 0;
author reviewer
#14 lore raw

PATCH v4 7/18 net/smc: disable SEID on non-s390 architecture

In reply to Wen Gu

On s390 architecture SMC-Dv2 SEID is generated by ISMv2 device with partial cpuid. It is predefined and unique to represents the physical machine (CPC), the maximum communication space of SMC-D.

On non-s390 archs like x86/64 or ARM, there is no similar information to identify physical machine, especially in virtualization scenarios like KVM, etc. In such cases, SEID is forcibly disabled and the user- defined UEID will be used to represent the communicable space.

---
 net/smc/smc_clc.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
net/smc/smc_clc.c +14 -0
index aeb9643..deb43db 100644
@@ -155,10 +155,12 @@ static int smc_clc_ueid_remove(char *ueid)
 			rc = 0;
 		}
 	}
+#if IS_ENABLED(CONFIG_S390)
 	if (!rc && !smc_clc_eid_table.ueid_cnt) {
 		smc_clc_eid_table.seid_enabled = 1;
 		rc = -EAGAIN;	/* indicate success and enabling of seid */
 	}
+#endif
 	write_unlock(&smc_clc_eid_table.lock);
 	return rc;
 }
@@ -273,22 +275,30 @@ int smc_nl_dump_seid(struct sk_buff *skb, struct netlink_callback *cb)
 
 int smc_nl_enable_seid(struct sk_buff *skb, struct genl_info *info)
 {
+#if IS_ENABLED(CONFIG_S390)
 	write_lock(&smc_clc_eid_table.lock);
 	smc_clc_eid_table.seid_enabled = 1;
 	write_unlock(&smc_clc_eid_table.lock);
 	return 0;
+#else
+	return -EOPNOTSUPP;
+#endif
 }
 
 int smc_nl_disable_seid(struct sk_buff *skb, struct genl_info *info)
 {
 	int rc = 0;
 
+#if IS_ENABLED(CONFIG_S390)
 	write_lock(&smc_clc_eid_table.lock);
 	if (!smc_clc_eid_table.ueid_cnt)
 		rc = -ENOENT;
 	else
 		smc_clc_eid_table.seid_enabled = 0;
 	write_unlock(&smc_clc_eid_table.lock);
+#else
+	rc = -EOPNOTSUPP;
+#endif
 	return rc;
 }
 
@@ -1292,7 +1302,11 @@ void __init smc_clc_init(void)
 	INIT_LIST_HEAD(&smc_clc_eid_table.list);
 	rwlock_init(&smc_clc_eid_table.lock);
 	smc_clc_eid_table.ueid_cnt = 0;
+#if IS_ENABLED(CONFIG_S390)
 	smc_clc_eid_table.seid_enabled = 1;
+#else
+	smc_clc_eid_table.seid_enabled = 0;
+#endif
 }
 
 void smc_clc_exit(void)
Signature · 1 line
1.8.3.1
author reviewer
#15 lore raw

PATCH v4 8/18 net/smc: enable virtual ISM device feature bit

In reply to Wen Gu

This patch enables the first supplement feature of SMCv2.1, that is the support for virtual ISM devices.

---
 net/smc/af_smc.c | 2 ++
 net/smc/smc.h    | 7 +++++++
 2 files changed, 9 insertions(+)
net/smc/af_smc.c +2 -0
index 5bb41404..c5b7716 100644
@@ -1528,6 +1528,7 @@ static int __smc_connect(struct smc_sock *smc)
 	ini->smcr_version = SMC_V1 | SMC_V2;
 	ini->smc_type_v1 = SMC_TYPE_B;
 	ini->smc_type_v2 = SMC_TYPE_B;
+	ini->feature_mask = SMC_FEATURE_MASK;
 
 	/* get vlan id from IP device */
 	if (smc_vlan_by_tcpsk(smc->clcsock, ini)) {
@@ -1993,6 +1994,7 @@ static int smc_listen_v2_check(struct smc_sock *new_smc,
 	ini->smc_type_v2 = pclc->hdr.typev2;
 	ini->smcd_version = smcd_indicated(ini->smc_type_v1) ? SMC_V1 : 0;
 	ini->smcr_version = smcr_indicated(ini->smc_type_v1) ? SMC_V1 : 0;
+	ini->feature_mask = SMC_FEATURE_MASK;
 	if (pclc->hdr.version > SMC_V1) {
 		if (smcd_indicated(ini->smc_type_v2))
 			ini->smcd_version |= SMC_V2;
net/smc/smc.h +7 -0
index 24745fd..71b9640 100644
@@ -58,6 +58,13 @@ enum smc_state {		/* possible states of an SMC socket */
 	SMC_PROCESSABORT	= 27,
 };
 
+enum smc_supp_feature {	/* supplemental features supported by current version */
+	SMC_SPF_VIRT_ISM_DEV	= 0,
+};
+
+#define SMC_FEATURE_MASK \
+	(BIT(SMC_SPF_VIRT_ISM_DEV))
+
 struct smc_link_group;
 
 struct smc_wr_rx_hdr {	/* common prefix part of LLC and CDC to demultiplex */
Signature · 1 line
1.8.3.1
author reviewer
#16 lore raw

PATCH v4 9/18 net/smc: introduce SMC-D loopback device

In reply to Wen Gu

This patch introduces a kind of loopback device for SMC-D. The device is created when SMC module is loaded and destroyed when the SMC module is unloaded. The loopback device is a kernel device used only by the SMC module and is not restricted by net namespace, so it can be used for local inter-process or inter-container communication.

---
 net/smc/Kconfig        |  13 ++++
 net/smc/Makefile       |   2 +-
 net/smc/af_smc.c       |  12 +++-
 net/smc/smc_loopback.c | 165 +++++++++++++++++++++++++++++++++++++++++++++++++
 net/smc/smc_loopback.h |  33 ++++++++++
 5 files changed, 223 insertions(+), 2 deletions(-)
 create mode 100644 net/smc/smc_loopback.c
 create mode 100644 net/smc/smc_loopback.h
net/smc/Kconfig +13 -0
index 1ab3c5a..3e9e03e 100644
@@ -19,3 +19,16 @@ config SMC_DIAG
 	  smcss.
 
 	  if unsure, say Y.
+
+config SMC_LO
+	bool "SMC_LO: virtual loopback-ism for SMC"
+	depends on SMC
+	default n
+	help
+	  SMC_LO provides a kind of virtual ISM device called loopback-ism
+	  for SMC-D to upgrade AF_INET TCP connections whose ends share the
+	  same kernel.
+	  loopback-ism is a software-implemented simulation device that does
+	  not depend on a specific architecture or hardware.
+
+	  if unsure, say N.
net/smc/Makefile +1 -1
index 875efcd..a8c3711 100644
@@ -4,5 +4,5 @@ obj-$(CONFIG_SMC)	+= smc.o
 obj-$(CONFIG_SMC_DIAG)	+= smc_diag.o
 smc-y := af_smc.o smc_pnet.o smc_ib.o smc_clc.o smc_core.o smc_wr.o smc_llc.o
 smc-y += smc_cdc.o smc_tx.o smc_rx.o smc_close.o smc_ism.o smc_netlink.o smc_stats.o
-smc-y += smc_tracepoint.o
+smc-y += smc_tracepoint.o smc_loopback.o
 smc-$(CONFIG_SYSCTL) += smc_sysctl.o
net/smc/af_smc.c +11 -1
index c5b7716..6435659 100644
@@ -53,6 +53,7 @@
 #include "smc_stats.h"
 #include "smc_tracepoint.h"
 #include "smc_sysctl.h"
+#include "smc_loopback.h"
 
 static DEFINE_MUTEX(smc_server_lgr_pending);	/* serialize link group
 						 * creation on server
@@ -3565,15 +3566,23 @@ static int __init smc_init(void)
 		goto out_sock;
 	}
 
+	rc = smc_loopback_init();
+	if (rc) {
+		pr_err("%s: smc_loopback_init fails with %d\n", __func__, rc);
+		goto out_ib;
+	}
+
 	rc = tcp_register_ulp(&smc_ulp_ops);
 	if (rc) {
 		pr_err("%s: tcp_ulp_register fails with %d\n", __func__, rc);
-		goto out_ib;
+		goto out_lo;
 	}
 
 	static_branch_enable(&tcp_have_smc);
 	return 0;
 
+out_lo:
+	smc_loopback_exit();
 out_ib:
 	smc_ib_unregister_client();
 out_sock:
@@ -3611,6 +3620,7 @@ static void __exit smc_exit(void)
 	tcp_unregister_ulp(&smc_ulp_ops);
 	sock_unregister(PF_SMC);
 	smc_core_exit();
+	smc_loopback_exit();
 	smc_ib_unregister_client();
 	smc_ism_exit();
 	destroy_workqueue(smc_close_wq);
net/smc/smc_loopback.c new file +165 -0

165 lines not on this page · raw message

net/smc/smc_loopback.h new file +33 -0
new file mode 100644
index 0000000..39aa1dc
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ *  Shared Memory Communications Direct over loopback device.
+ *
+ *  Provide a SMC-D loopback dummy device.
+ *
+ *  Copyright (c) 2022, Alibaba Inc.
+ *
+ *  Author: Wen Gu <[email protected]>
+ *          Tony Lu <[email protected]>
+ *
+ */
+
+#ifndef _SMC_LOOPBACK_H
+#define _SMC_LOOPBACK_H
+
+#include <linux/device.h>
+#include <linux/err.h>
+#include <net/smc.h>
+
+#if IS_ENABLED(CONFIG_SMC_LO)
+#define SMC_LODEV_MAX_DMBS 5000
+
+struct smc_lo_dev {
+	struct smcd_dev *smcd;
+	struct device dev;
+};
+#endif
+
+int smc_loopback_init(void);
+void smc_loopback_exit(void);
+
+#endif /* _SMC_LOOPBACK_H */
Signature · 1 line
1.8.3.1
maintainer
#17 lore raw

Re: [PATCH net-next v4 09/18] net/smc: introduce SMC-D loopback device

Quoted 7 lines · Wen Gu

This patch introduces a kind of loopback device for SMC-D. The device is created when SMC module is loaded and destroyed when the SMC module is unloaded. The loopback device is a kernel device used only by the SMC module and is not restricted by net namespace, so it can be used for local inter-process or inter-container communication.

Quoted 9 lines
---
 net/smc/Kconfig        |  13 ++++
 net/smc/Makefile       |   2 +-
 net/smc/af_smc.c       |  12 +++-
 net/smc/smc_loopback.c | 165 +++++++++++++++++++++++++++++++++++++++++++++++++
 net/smc/smc_loopback.h |  33 ++++++++++
 5 files changed, 223 insertions(+), 2 deletions(-)
 create mode 100644 net/smc/smc_loopback.c
 create mode 100644 net/smc/smc_loopback.h

Hello Wen Gu,

thank you for adding the Kconfig, so the distributions can decide when to offer this feature.

I propose you add some kind of runtime switch as well. Not every user who loads the SMC module may want to exploit smcd-loopback. Especially in native environements without containers.

If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC. If loopback is always created unconditionally, there is no way to opt-out.

maintainer
#18 lore raw

Re: [PATCH net-next v4 09/18] net/smc: introduce SMC-D loopback device

Quoted 23 lines · Alexandra Winter
Quoted 7 lines · Wen Gu

This patch introduces a kind of loopback device for SMC-D. The device is created when SMC module is loaded and destroyed when the SMC module is unloaded. The loopback device is a kernel device used only by the SMC module and is not restricted by net namespace, so it can be used for local inter-process or inter-container communication.

Quoted 9 lines
---
 net/smc/Kconfig        |  13 ++++
 net/smc/Makefile       |   2 +-
 net/smc/af_smc.c       |  12 +++-
 net/smc/smc_loopback.c | 165 +++++++++++++++++++++++++++++++++++++++++++++++++
 net/smc/smc_loopback.h |  33 ++++++++++
 5 files changed, 223 insertions(+), 2 deletions(-)
 create mode 100644 net/smc/smc_loopback.c
 create mode 100644 net/smc/smc_loopback.h

Hello Wen Gu,

Quoted 8 lines

thank you for adding the Kconfig, so the distributions can decide when to offer this feature.

I propose you add some kind of runtime switch as well. Not every user who loads the SMC module may want to exploit smcd-loopback. Especially in native environements without containers.

If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC. If loopback is always created unconditionally, there is no way to opt-out.

Another thing came to my mind:

When loopback is created and registered when the SMC module is loaded, it will implicitly always have highest priority, right? That should be stated somewhere.
Also, if you create a runtime switch this will change, so then you need to decide about priority of loopback vs ISM device (and other future smcd-devices).

author reviewer
#19 lore raw

Re: [PATCH net-next v4 09/18] net/smc: introduce SMC-D loopback device

Quoted 35 lines · Alexandra Winter
Quoted 23 lines · Alexandra Winter
Quoted 14 lines · Wen Gu

This patch introduces a kind of loopback device for SMC-D. The device is created when SMC module is loaded and destroyed when the SMC module is unloaded. The loopback device is a kernel device used only by the SMC module and is not restricted by net namespace, so it can be used for local inter-process or inter-container communication.

---
  net/smc/Kconfig        |  13 ++++
  net/smc/Makefile       |   2 +-
  net/smc/af_smc.c       |  12 +++-
  net/smc/smc_loopback.c | 165 +++++++++++++++++++++++++++++++++++++++++++++++++
  net/smc/smc_loopback.h |  33 ++++++++++
  5 files changed, 223 insertions(+), 2 deletions(-)
Quoted 2 lines

create mode 100644 net/smc/smc_loopback.c
create mode 100644 net/smc/smc_loopback.h

Hello Wen Gu,

Quoted 8 lines

thank you for adding the Kconfig, so the distributions can decide when to offer this feature.

I propose you add some kind of runtime switch as well. Not every user who loads the SMC module may want to exploit smcd-loopback. Especially in native environements without containers.

If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC. If loopback is always created unconditionally, there is no way to opt-out.

Quoted 5 lines

Another thing came to my mind:

When loopback is created and registered when the SMC module is loaded, it will implicitly always have highest priority, right? That should be stated somewhere.
Also, if you create a runtime switch this will change, so then you need to decide about priority of loopback vs ISM device (and other future smcd-devices).

Yes. I think the question may become 'How users to define the priority of existing the smcd devices'. In the past,
all the ISMv2 has nearly same performance so priority is not very important. But now there are other virtual ISM devices, they perform differently.

My rough idea is defining a fixed priority, such as whenever loopback-ism is enabled, it is always the first in the slots. If fixed priority is not appropriate, low-priority devices can be prioritized by disabling high-priority devices.

So it seems that the runtime switch of the loopback-ism is even more necessary.

Thanks,
Wen Gu

author reviewer
#20 lore raw

Re: [PATCH net-next v4 09/18] net/smc: introduce SMC-D loopback device

In reply to Alexandra Winter
Quoted 23 lines · Alexandra Winter
Quoted 14 lines · Wen Gu

This patch introduces a kind of loopback device for SMC-D. The device is created when SMC module is loaded and destroyed when the SMC module is unloaded. The loopback device is a kernel device used only by the SMC module and is not restricted by net namespace, so it can be used for local inter-process or inter-container communication.

---
  net/smc/Kconfig        |  13 ++++
  net/smc/Makefile       |   2 +-
  net/smc/af_smc.c       |  12 +++-
  net/smc/smc_loopback.c | 165 +++++++++++++++++++++++++++++++++++++++++++++++++
  net/smc/smc_loopback.h |  33 ++++++++++
  5 files changed, 223 insertions(+), 2 deletions(-)
Quoted 2 lines

create mode 100644 net/smc/smc_loopback.c
create mode 100644 net/smc/smc_loopback.h

Hello Wen Gu,

Quoted 7 lines

thank you for adding the Kconfig, so the distributions can decide when to offer this feature.

I propose you add some kind of runtime switch as well. Not every user who loads the SMC module may want to exploit smcd-loopback. Especially in native environements without containers.

If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC. If loopback is always created unconditionally, there is no way to opt-out.

Yes, I need to think about this. Make a runtime switch to enable/disable the loopback-ism just like ip link up/down. An rough idea is to add an smc-tools command, like 'smcd device disable/enable loopback'.

Thank you very much.

Regards,
Wen Gu

maintainer
#21 lore raw

Re: [PATCH net-next v4 09/18] net/smc: introduce SMC-D loopback device

In reply to Alexandra Winter
Quoted 23 lines · Alexandra Winter
Quoted 7 lines · Wen Gu

This patch introduces a kind of loopback device for SMC-D. The device is created when SMC module is loaded and destroyed when the SMC module is unloaded. The loopback device is a kernel device used only by the SMC module and is not restricted by net namespace, so it can be used for local inter-process or inter-container communication.

Quoted 9 lines
---
 net/smc/Kconfig        |  13 ++++
 net/smc/Makefile       |   2 +-
 net/smc/af_smc.c       |  12 +++-
 net/smc/smc_loopback.c | 165 +++++++++++++++++++++++++++++++++++++++++++++++++
 net/smc/smc_loopback.h |  33 ++++++++++
 5 files changed, 223 insertions(+), 2 deletions(-)
 create mode 100644 net/smc/smc_loopback.c
 create mode 100644 net/smc/smc_loopback.h

Hello Wen Gu,

Quoted 7 lines

thank you for adding the Kconfig, so the distributions can decide when to offer this feature.

I propose you add some kind of runtime switch as well. Not every user who loads the SMC module may want to exploit smcd-loopback. Especially in native environements without containers.

If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC. If loopback is always created unconditionally, there is no way to opt-out.

Hi Sandy,

After talking to Wen Gu offline, I think the real issue here might be we don't have an abstract layer in SMC, something like net/core/dev.c

Without this, we cannot do:

1. Enable/disable those devices dynamically
Currently, If we want to disable a SMC-R device to communicate with others, we need to refer to 'ip link set dev xxx down' to disable the netdevice, then Infiniband subsystem will notify SMC that the state of the IB device has changed. We cannot explicitly choose not to use some specific IB/RoCE devices without disable totally.
If the loopback device need to support enable/disable itself, I think it might be better to enable this feature for all SMC devices.

2. Do statistics per device
Now, we have to relay on IB/RoCE devices' hardware statistics to see how many packets/bytes we have sent through this device.

Both the above issues get worse when the IB/RoCE device is shared by SMC and userspace RDMA applications. If SMC-R and userspace RDMA applications run at the same time, we can't enable the device to run userspace RDMA applications while block it from running SMC. For statistics, we cannot tell how many packets/bytes were sent by SMC and how many were sent by userspace RDMA applications.

So I think those are better to support in the SMC layer.

Best regards!
Dust

maintainer
#22 lore raw

Re: [PATCH net-next v4 09/18] net/smc: introduce SMC-D loopback device

Quoted 36 lines · Dust Li
Quoted 9 lines

Hello Wen Gu,

thank you for adding the Kconfig, so the distributions can decide when to offer this feature.

I propose you add some kind of runtime switch as well. Not every user who loads the SMC module may want to exploit smcd-loopback. Especially in native environements without containers.

If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC. If loopback is always created unconditionally, there is no way to opt-out.

Hi Sandy,

After talking to Wen Gu offline, I think the real issue here might be we don't have an abstract layer in SMC, something like net/core/dev.c

Without this, we cannot do:

1. Enable/disable those devices dynamically
Currently, If we want to disable a SMC-R device to communicate with others, we need to refer to 'ip link set dev xxx down' to disable the netdevice, then Infiniband subsystem will notify SMC that the state of the IB device has changed. We cannot explicitly choose not to use some specific IB/RoCE devices without disable totally.
If the loopback device need to support enable/disable itself, I think it might be better to enable this feature for all SMC devices.

2. Do statistics per device
Now, we have to relay on IB/RoCE devices' hardware statistics to see how many packets/bytes we have sent through this device.

Both the above issues get worse when the IB/RoCE device is shared by SMC and userspace RDMA applications. If SMC-R and userspace RDMA applications run at the same time, we can't enable the device to run userspace RDMA applications while block it from running SMC. For statistics, we cannot tell how many packets/bytes were sent by SMC and how many were sent by userspace RDMA applications.

Quoted 4 lines

So I think those are better to support in the SMC layer.

Best regards!
Dust

Thank you very much for your considerations. I also think a generic handling of these requirements in the smc layer would be best. Especially, if we want to add virtio-ism support soon. There we will face the same issues again. Let's hear what others think about this.

#23 lore raw

Re: [PATCH net-next v4 09/18] net/smc: introduce SMC-D loopback device

Quoted 41 lines · Alexandra Winter
Quoted 38 lines · Dust Li
Quoted 9 lines

Hello Wen Gu,

thank you for adding the Kconfig, so the distributions can decide when to offer this feature.

I propose you add some kind of runtime switch as well. Not every user who loads the SMC module may want to exploit smcd-loopback. Especially in native environements without containers.

If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC. If loopback is always created unconditionally, there is no way to opt-out.

Hi Sandy,

After talking to Wen Gu offline, I think the real issue here might be we don't have an abstract layer in SMC, something like net/core/dev.c

Without this, we cannot do:

1. Enable/disable those devices dynamically
Currently, If we want to disable a SMC-R device to communicate with others, we need to refer to 'ip link set dev xxx down' to disable the netdevice, then Infiniband subsystem will notify SMC that the state of the IB device has changed. We cannot explicitly choose not to use some specific IB/RoCE devices without disable totally.
If the loopback device need to support enable/disable itself, I think it might be better to enable this feature for all SMC devices.

2. Do statistics per device
Now, we have to relay on IB/RoCE devices' hardware statistics to see how many packets/bytes we have sent through this device.

Both the above issues get worse when the IB/RoCE device is shared by SMC and userspace RDMA applications. If SMC-R and userspace RDMA applications run at the same time, we can't enable the device to run userspace RDMA applications while block it from running SMC. For statistics, we cannot tell how many packets/bytes were sent by SMC and how many were sent by userspace RDMA applications.

So I think those are better to support in the SMC layer.

Quoted 9 lines
Quoted 2 lines

Best regards!
Dust

Thank you very much for your considerations. I also think a generic handling of these requirements in the smc layer would be best. Especially, if we want to add virtio-ism support soon. There we will face the same issues again. Let's hear what others think about this.

Thanks you Sandy for bringing it up and Dust Li & Wen Gu for your thoughts.
I agree that such a runtime switch is needed and also that this generic handling would be good in the smc layer.

author reviewer
#24 lore raw

Re: [PATCH net-next v4 09/18] net/smc: introduce SMC-D loopback device

Quoted 54 lines · Jan Karcher
Quoted 41 lines · Alexandra Winter
Quoted 38 lines · Dust Li
Quoted 9 lines

Hello Wen Gu,

thank you for adding the Kconfig, so the distributions can decide when to offer this feature.

I propose you add some kind of runtime switch as well. Not every user who loads the SMC module may want to exploit smcd-loopback. Especially in native environements without containers.

If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC. If loopback is always created unconditionally, there is no way to opt-out.

Hi Sandy,

After talking to Wen Gu offline, I think the real issue here might be we don't have an abstract layer in SMC, something like net/core/dev.c

Without this, we cannot do:

1. Enable/disable those devices dynamically
    Currently, If we want to disable a SMC-R device to communicate with others, we need to refer to 'ip link set dev xxx down' to disable the netdevice, then Infiniband subsystem will notify SMC that the state of the IB device has changed. We cannot explicitly choose not to use some specific IB/RoCE devices without disable totally.
    If the loopback device need to support enable/disable itself, I think it might be better to enable this feature for all SMC devices.

2. Do statistics per device
    Now, we have to relay on IB/RoCE devices' hardware statistics to see how many packets/bytes we have sent through this device.

Both the above issues get worse when the IB/RoCE device is shared by SMC and userspace RDMA applications. If SMC-R and userspace RDMA applications run at the same time, we can't enable the device to run userspace RDMA applications while block it from running SMC. For statistics, we cannot tell how many packets/bytes were sent by SMC and how many were sent by userspace RDMA applications.

So I think those are better to support in the SMC layer.

Quoted 9 lines
Quoted 2 lines

Best regards!
Dust

Thank you very much for your considerations. I also think a generic handling of these requirements in the smc layer would be best. Especially, if we want to add virtio-ism support soon. There we will face the same issues again. Let's hear what others think about this.

Quoted 2 lines

Thanks you Sandy for bringing it up and Dust Li & Wen Gu for your thoughts.
I agree that such a runtime switch is needed and also that this generic handling would be good in the smc layer.

Right. runtime switch is necessary. I'm trying some ways to see which one is more suitable.

As for implementing a abstract layer that capable of handling 1) enable/disable SMC usage of RDMA/ISM devices. 2) count packets/bytes of RDMA/ISM devices that generated/consumed by SMC, I believe it would be helpful, and IMHO its architecture may be:

----------------------------------------------

                  SMC protocol
    (af_smc.c / smc_core.c / smc_clc.c ...)

----------------------------------------------
Abstract layer of SMC device
(define SMC device common operations)
----------------------------------------------
RDMA device | (virt) ISM device
(smc_ib.c) | (smc_ism.c / smc_loopback.c)
----------------------------------------------

But I also believe this may require a lot of works and may be a long-term job.

If only for the virtual ISM device, e.g.loopback-ism, I am considering adding it to the Linux device tree (/sys/devices/virtual/) to make it more 'device-like', and controlling its enable/disable and get the statistics through some files, such as
echo 1 > /sys/devices/virtual/loopback-ism/alive
or
cat /sys/devices/virtual/loopback-ism/statistics/{rx|tx}_{bytes|packets} (similar to what tcp lo have in /sys/devices/virtual/net/lo)

What are your thoughts on it? Thanks.

--
A little off-topic, it's currently China's National Day holiday, which lasts for about a week, so we are now on vacation. As a result, my responses might be a bit slower, but I will still make time to check/reply the mail and prepare for my new version. Thank you all very much!

Regards,
Wen Gu

maintainer
#25 lore raw

Re: [PATCH net-next v4 09/18] net/smc: introduce SMC-D loopback device

Quoted 78 lines · Wen Gu
Quoted 54 lines · Jan Karcher
Quoted 41 lines · Alexandra Winter
Quoted 38 lines · Dust Li
Quoted 9 lines

Hello Wen Gu,

thank you for adding the Kconfig, so the distributions can decide when to offer this feature.

I propose you add some kind of runtime switch as well. Not every user who loads the SMC module may want to exploit smcd-loopback. Especially in native environements without containers.

If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC. If loopback is always created unconditionally, there is no way to opt-out.

Hi Sandy,

After talking to Wen Gu offline, I think the real issue here might be we don't have an abstract layer in SMC, something like net/core/dev.c

Without this, we cannot do:

1. Enable/disable those devices dynamically
    Currently, If we want to disable a SMC-R device to communicate with others, we need to refer to 'ip link set dev xxx down' to disable the netdevice, then Infiniband subsystem will notify SMC that the state of the IB device has changed. We cannot explicitly choose not to use some specific IB/RoCE devices without disable totally.
    If the loopback device need to support enable/disable itself, I think it might be better to enable this feature for all SMC devices.

2. Do statistics per device
    Now, we have to relay on IB/RoCE devices' hardware statistics to see how many packets/bytes we have sent through this device.

Both the above issues get worse when the IB/RoCE device is shared by SMC and userspace RDMA applications. If SMC-R and userspace RDMA applications run at the same time, we can't enable the device to run userspace RDMA applications while block it from running SMC. For statistics, we cannot tell how many packets/bytes were sent by SMC and how many were sent by userspace RDMA applications.

So I think those are better to support in the SMC layer.

Quoted 9 lines
Quoted 2 lines

Best regards!
Dust

Thank you very much for your considerations. I also think a generic handling of these requirements in the smc layer would be best. Especially, if we want to add virtio-ism support soon. There we will face the same issues again. Let's hear what others think about this.

Quoted 2 lines

Thanks you Sandy for bringing it up and Dust Li & Wen Gu for your thoughts.
I agree that such a runtime switch is needed and also that this generic handling would be good in the smc layer.

Right. runtime switch is necessary. I'm trying some ways to see which one is more suitable.

As for implementing a abstract layer that capable of handling 1) enable/disable SMC usage of RDMA/ISM devices. 2) count packets/bytes of RDMA/ISM devices that generated/consumed by SMC, I believe it would be helpful, and IMHO its architecture may be:

----------------------------------------------
                  SMC protocol
    (af_smc.c / smc_core.c / smc_clc.c ...)
----------------------------------------------
          Abstract layer of SMC device
      (define SMC device common operations)
----------------------------------------------
  RDMA device |        (virt) ISM device
  (smc_ib.c)  |   (smc_ism.c / smc_loopback.c)
----------------------------------------------

Quoted 2 lines

But I also believe this may require a lot of works and may be a long-term job.

I like that concept a lot. If we can agree on a direction, we can define meaningful pieces and approach it piece by piece.

Quoted 10 lines

If only for the virtual ISM device, e.g.loopback-ism, I am considering adding it to the Linux device tree (/sys/devices/virtual/) to make it more 'device-like', and controlling its enable/disable and get the statistics through some files, such as
echo 1 > /sys/devices/virtual/loopback-ism/alive
or
cat /sys/devices/virtual/loopback-ism/statistics/{rx|tx}_{bytes|packets} (similar to what tcp lo have in /sys/devices/virtual/net/lo)

What are your thoughts on it? Thanks.

Makes sense to me, but I don't have too much experience in that area.
I have never seen an attribute called 'alive' before.
I think attributes like 'power', 'enable' or 'online' are used for other device types.

Quoted 8 lines
Signature · 7 lines
-- 
A little off-topic, it's currently China's National Day holiday, which lasts for about a week,
so we are now on vacation. As a result, my responses might be a bit slower, but I will still
make time to check/reply the mail and prepare for my new version. Thank you all very much!

Regards,
Wen Gu

Next week is Germany's national holiday, so many of us are out as well.

author reviewer
#26 lore raw

Re: [PATCH net-next v4 09/18] net/smc: introduce SMC-D loopback device

Quoted 81 lines · Alexandra Winter
Quoted 78 lines · Wen Gu
Quoted 54 lines · Jan Karcher
Quoted 41 lines · Alexandra Winter
Quoted 38 lines · Dust Li
Quoted 9 lines

Hello Wen Gu,

thank you for adding the Kconfig, so the distributions can decide when to offer this feature.

I propose you add some kind of runtime switch as well. Not every user who loads the SMC module may want to exploit smcd-loopback. Especially in native environements without containers.

If no RoCE interfaces or no ISM interfaces exist, the respective handling is skipped in SMC. If loopback is always created unconditionally, there is no way to opt-out.

Hi Sandy,

After talking to Wen Gu offline, I think the real issue here might be we don't have an abstract layer in SMC, something like net/core/dev.c

Without this, we cannot do:

1. Enable/disable those devices dynamically
    Currently, If we want to disable a SMC-R device to communicate with others, we need to refer to 'ip link set dev xxx down' to disable the netdevice, then Infiniband subsystem will notify SMC that the state of the IB device has changed. We cannot explicitly choose not to use some specific IB/RoCE devices without disable totally.
    If the loopback device need to support enable/disable itself, I think it might be better to enable this feature for all SMC devices.

2. Do statistics per device
    Now, we have to relay on IB/RoCE devices' hardware statistics to see how many packets/bytes we have sent through this device.

Both the above issues get worse when the IB/RoCE device is shared by SMC and userspace RDMA applications. If SMC-R and userspace RDMA applications run at the same time, we can't enable the device to run userspace RDMA applications while block it from running SMC. For statistics, we cannot tell how many packets/bytes were sent by SMC and how many were sent by userspace RDMA applications.

So I think those are better to support in the SMC layer.

Quoted 9 lines
Quoted 2 lines

Best regards!
Dust

Thank you very much for your considerations. I also think a generic handling of these requirements in the smc layer would be best. Especially, if we want to add virtio-ism support soon. There we will face the same issues again. Let's hear what others think about this.

Quoted 2 lines

Thanks you Sandy for bringing it up and Dust Li & Wen Gu for your thoughts.
I agree that such a runtime switch is needed and also that this generic handling would be good in the smc layer.

Right. runtime switch is necessary. I'm trying some ways to see which one is more suitable.

As for implementing a abstract layer that capable of handling 1) enable/disable SMC usage of RDMA/ISM devices. 2) count packets/bytes of RDMA/ISM devices that generated/consumed by SMC, I believe it would be helpful, and IMHO its architecture may be:

----------------------------------------------
                  SMC protocol
    (af_smc.c / smc_core.c / smc_clc.c ...)
----------------------------------------------
          Abstract layer of SMC device
      (define SMC device common operations)
----------------------------------------------
  RDMA device |        (virt) ISM device
  (smc_ib.c)  |   (smc_ism.c / smc_loopback.c)
----------------------------------------------

Quoted 6 lines
Quoted 2 lines

But I also believe this may require a lot of works and may be a long-term job.

I like that concept a lot. If we can agree on a direction, we can define meaningful pieces and approach it piece by piece.

Yes. It can be added to our interlock's backup list.

Quoted 9 lines
Quoted 9 lines

If only for the virtual ISM device, e.g.loopback-ism, I am considering adding it to the Linux device tree (/sys/devices/virtual/) to make it more 'device-like', and controlling its enable/disable and get the statistics through some files, such as
echo 1 > /sys/devices/virtual/loopback-ism/alive
or
cat /sys/devices/virtual/loopback-ism/statistics/{rx|tx}_{bytes|packets} (similar to what tcp lo have in /sys/devices/virtual/net/lo)

Quoted 7 lines
Quoted 2 lines

What are your thoughts on it? Thanks.

Makes sense to me, but I don't have too much experience in that area.
I have never seen an attribute called 'alive' before.
I think attributes like 'power', 'enable' or 'online' are used for other device types.

Thanks. I will refer to existing devices for reference.

Quoted 10 lines
Quoted 8 lines
Signature · 7 lines
-- 
A little off-topic, it's currently China's National Day holiday, which lasts for about a week,
so we are now on vacation. As a result, my responses might be a bit slower, but I will still
make time to check/reply the mail and prepare for my new version. Thank you all very much!

Regards,
Wen Gu

Next week is Germany's national holiday, so many of us are out as well.

Have a nice holiday! :)

author reviewer
#27 lore raw

PATCH v4 10/18 net/smc: implement ID-related operations of loopback

In reply to Wen Gu

This patch implements GID/CHID/SEID related operations of SMC-D loopback device. In loopback device, GID is generated by UUIDv4 algorithm, CHID is reserved 0xFFFF, SEID is generated using the same algorithm as ISM device under s390 architecture, and is 0 and disabled under non-s390 architecture.

---
 include/net/smc.h      |  6 ++++
 net/smc/smc_ism.h      |  2 ++
 net/smc/smc_loopback.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++----
 net/smc/smc_loopback.h |  3 ++
 4 files changed, 101 insertions(+), 6 deletions(-)
include/net/smc.h +6 -0
index d8db5e1..7c2d35c 100644
@@ -41,6 +41,12 @@ struct smcd_dmb {
 	dma_addr_t dma_addr;
 };
 
+struct smcd_seid {
+	u8 seid_string[24];
+	u8 serial_number[4];
+	u8 type[4];
+};
+
 #define ISM_EVENT_DMB	0
 #define ISM_EVENT_GID	1
 #define ISM_EVENT_SWR	2
net/smc/smc_ism.h +2 -0
index e6ea08c..7ab82dd 100644
@@ -15,6 +15,8 @@
 
 #include "smc.h"
 
+#define S390_ISM_IDENT_MASK		0x00FFFF
+
 #define SMC_VIRT_ISM_CHID_MAX		0xFFFF
 #define SMC_VIRT_ISM_CHID_MIN		0xFF00
 
net/smc/smc_loopback.c +90 -6

124 lines not on this page · raw message

net/smc/smc_loopback.h +3 -0
index 39aa1dc..71a52da 100644
@@ -19,11 +19,14 @@
 #include <net/smc.h>
 
 #if IS_ENABLED(CONFIG_SMC_LO)
+#define SMC_LO_CHID 0xFFFF
 #define SMC_LODEV_MAX_DMBS 5000
 
 struct smc_lo_dev {
 	struct smcd_dev *smcd;
 	struct device dev;
+	u16 chid;
+	struct smcd_gid local_gid;
 };
 #endif
 
Signature · 1 line
1.8.3.1
maintainer
#28 lore raw

Re: [PATCH net-next v4 10/18] net/smc: implement ID-related operations of loopback

Quoted 1 line · Wen Gu

This patch implements GID/CHID/SEID related operations of SMC-D loopback device. In loopback device, GID is generated by UUIDv4 algorithm, CHID is reserved 0xFFFF, SEID is generated using the same algorithm as ISM device under s390 architecture, and is 0 and disabled under non-s390 architecture. Signed-off-by: Wen Gu <[email protected]> ---

IMO, get_system_eid should not be part of smcd_ops. And should not be provided by an smcd device. It is a system_eid is a global value that is valid for all smcd interfaces of this system (os instance). So I think it should be provided by the smc module.

I agree it needs to be architecture dependent and same as today for s390.

author reviewer
#29 lore raw

PATCH v4 11/18 net/smc: implement some unsupported operations of loopback

In reply to Wen Gu

Vlan operations are not supported currently since the need for vlan in loopback does not seem to be strong.

Signal_event operation is not supported since no event now needs to be processed by loopback device.

---
 net/smc/smc_loopback.c | 36 +++++++++++++++++++++++++++++++-----
 1 file changed, 31 insertions(+), 5 deletions(-)
net/smc/smc_loopback.c +31 -5
index 610af99..4341803 100644
@@ -75,6 +75,32 @@ static int smc_lo_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid,
 	return 0;
 }
 
+static int smc_lo_add_vlan_id(struct smcd_dev *smcd, u64 vlan_id)
+{
+	return -EOPNOTSUPP;
+}
+
+static int smc_lo_del_vlan_id(struct smcd_dev *smcd, u64 vlan_id)
+{
+	return -EOPNOTSUPP;
+}
+
+static int smc_lo_set_vlan_required(struct smcd_dev *smcd)
+{
+	return -EOPNOTSUPP;
+}
+
+static int smc_lo_reset_vlan_required(struct smcd_dev *smcd)
+{
+	return -EOPNOTSUPP;
+}
+
+static int smc_lo_signal_event(struct smcd_dev *dev, struct smcd_gid *rgid,
+			       u32 trigger_irq, u32 event_code, u64 info)
+{
+	return 0;
+}
+
 static int smc_lo_supports_v2(void)
 {
 	return SMC_LO_SUPPORTS_V2;
@@ -108,11 +134,11 @@ static struct device *smc_lo_get_dev(struct smcd_dev *smcd)
 	.query_remote_gid = smc_lo_query_rgid,
 	.register_dmb		= NULL,
 	.unregister_dmb		= NULL,
-	.add_vlan_id		= NULL,
-	.del_vlan_id		= NULL,
-	.set_vlan_required	= NULL,
-	.reset_vlan_required	= NULL,
-	.signal_event		= NULL,
+	.add_vlan_id = smc_lo_add_vlan_id,
+	.del_vlan_id = smc_lo_del_vlan_id,
+	.set_vlan_required = smc_lo_set_vlan_required,
+	.reset_vlan_required = smc_lo_reset_vlan_required,
+	.signal_event = smc_lo_signal_event,
 	.move_data		= NULL,
 	.supports_v2 = smc_lo_supports_v2,
 	.get_system_eid = smc_lo_get_system_eid,
Signature · 1 line
1.8.3.1
author reviewer
#30 lore raw

PATCH v4 12/18 net/smc: implement DMB-related operations of loopback

In reply to Wen Gu

This patch implements DMB registration, unregistration and data move operations of SMC-D loopback.

---
 net/smc/smc_cdc.c      |   6 +++
 net/smc/smc_cdc.h      |   1 +
 net/smc/smc_loopback.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++--
 net/smc/smc_loopback.h |  13 +++++
 4 files changed, 144 insertions(+), 3 deletions(-)
net/smc/smc_cdc.c +6 -0
index 89105e9..2641800 100644
@@ -411,6 +411,12 @@ static void smc_cdc_msg_recv(struct smc_sock *smc, struct smc_cdc_msg *cdc)
 static void smcd_cdc_rx_tsklet(struct tasklet_struct *t)
 {
 	struct smc_connection *conn = from_tasklet(conn, t, rx_tsklet);
+
+	smcd_cdc_rx_handler(conn);
+}
+
+void smcd_cdc_rx_handler(struct smc_connection *conn)
+{
 	struct smcd_cdc_msg *data_cdc;
 	struct smcd_cdc_msg cdc;
 	struct smc_sock *smc;
net/smc/smc_cdc.h +1 -0
index 696cc11..11559d4 100644
@@ -301,5 +301,6 @@ int smcr_cdc_msg_send_validation(struct smc_connection *conn,
 				 struct smc_wr_buf *wr_buf);
 int smc_cdc_init(void) __init;
 void smcd_cdc_rx_init(struct smc_connection *conn);
+void smcd_cdc_rx_handler(struct smc_connection *conn);
 
 #endif /* SMC_CDC_H */
net/smc/smc_loopback.c +124 -3

167 lines not on this page · raw message

net/smc/smc_loopback.h +13 -0
index 71a52da..a5b501b 100644
@@ -21,12 +21,25 @@
 #if IS_ENABLED(CONFIG_SMC_LO)
 #define SMC_LO_CHID 0xFFFF
 #define SMC_LODEV_MAX_DMBS 5000
+#define SMC_LODEV_DMBS_HASH_BITS 12
+
+struct smc_lo_dmb_node {
+	struct hlist_node list;
+	u64 token;
+	u32 len;
+	u32 sba_idx;
+	void *cpu_addr;
+	dma_addr_t dma_addr;
+};
 
 struct smc_lo_dev {
 	struct smcd_dev *smcd;
 	struct device dev;
 	u16 chid;
 	struct smcd_gid local_gid;
+	DECLARE_BITMAP(sba_idx_mask, SMC_LODEV_MAX_DMBS);
+	rwlock_t dmb_ht_lock;
+	DECLARE_HASHTABLE(dmb_ht, SMC_LODEV_DMBS_HASH_BITS);
 };
 #endif
 
Signature · 1 line
1.8.3.1
#31 lore raw

Re: [PATCH net-next v4 12/18] net/smc: implement DMB-related operations of loopback

kernel test robot noticed the following build errors:

Hi Wen,

kernel test robot noticed the following build errors:

[auto build test ERROR on net-next/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Wen-Gu/net-smc-decouple-ism_dev-from-SMC-D-device-dump/20230924-231933
base:   net-next/main
patch link:    https://lore.kernel.org/r/1695568613-125057-13-git-send-email-guwen%40linux.alibaba.com
patch subject: [PATCH net-next v4 12/18] net/smc: implement DMB-related operations of loopback
config: mips-allmodconfig (https://download.01.org/0day-ci/archive/20230925/[email protected]/config)
compiler: mips-linux-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20230925/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All error/warnings (new ones prefixed by >>):

net/smc/smc_loopback.c: In function 'smc_lo_register_dmb':

>> net/smc/smc_loopback.c:102:30: error: implicit declaration of function 'vzalloc'; did you mean 'kvzalloc'? [-Werror=implicit-function-declaration]
     102 |         dmb_node->cpu_addr = vzalloc(dmb->dmb_len);
         |                              ^~~~~~~
         |                              kvzalloc
>> net/smc/smc_loopback.c:102:28: warning: assignment to 'void *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
     102 |         dmb_node->cpu_addr = vzalloc(dmb->dmb_len);
         |                            ^
   net/smc/smc_loopback.c: In function 'smc_lo_unregister_dmb':
>> net/smc/smc_loopback.c:159:9: error: implicit declaration of function 'vfree'; did you mean 'kvfree'? [-Werror=implicit-function-declaration]
     159 |         vfree(dmb_node->cpu_addr);
         |         ^~~~~
         |         kvfree

cc1: some warnings being treated as errors

vim +102 net/smc/smc_loopback.c

    79	
    80	static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
    81				       void *client_priv)
    82	{
    83		struct smc_lo_dmb_node *dmb_node, *tmp_node;
    84		struct smc_lo_dev *ldev = smcd->priv;
    85		int sba_idx, rc;
    86	
    87		/* check space for new dmb */
    88		for_each_clear_bit(sba_idx, ldev->sba_idx_mask, SMC_LODEV_MAX_DMBS) {
    89			if (!test_and_set_bit(sba_idx, ldev->sba_idx_mask))
    90				break;
    91		}
    92		if (sba_idx == SMC_LODEV_MAX_DMBS)
    93			return -ENOSPC;
    94	
    95		dmb_node = kzalloc(sizeof(*dmb_node), GFP_KERNEL);
    96		if (!dmb_node) {
    97			rc = -ENOMEM;
    98			goto err_bit;
    99		}
   100	
   101		dmb_node->sba_idx = sba_idx;
 > 102		dmb_node->cpu_addr = vzalloc(dmb->dmb_len);
   103		if (!dmb_node->cpu_addr) {
   104			rc = -ENOMEM;
   105			goto err_node;
   106		}
   107		dmb_node->len = dmb->dmb_len;
   108		dmb_node->dma_addr = SMC_DMA_ADDR_INVALID;
   109	
   110	again:
   111		/* add new dmb into hash table */
   112		get_random_bytes(&dmb_node->token, sizeof(dmb_node->token));
   113		write_lock(&ldev->dmb_ht_lock);
   114		hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_node->token) {
   115			if (tmp_node->token == dmb_node->token) {
   116				write_unlock(&ldev->dmb_ht_lock);
   117				goto again;
   118			}
   119		}
   120		hash_add(ldev->dmb_ht, &dmb_node->list, dmb_node->token);
   121		write_unlock(&ldev->dmb_ht_lock);
   122	
   123		dmb->sba_idx = dmb_node->sba_idx;
   124		dmb->dmb_tok = dmb_node->token;
   125		dmb->cpu_addr = dmb_node->cpu_addr;
   126		dmb->dma_addr = dmb_node->dma_addr;
   127		dmb->dmb_len = dmb_node->len;
   128	
   129		return 0;
   130	
   131	err_node:
   132		kfree(dmb_node);
   133	err_bit:
   134		clear_bit(sba_idx, ldev->sba_idx_mask);
   135		return rc;
   136	}
   137	
   138	static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
   139	{
   140		struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node;
   141		struct smc_lo_dev *ldev = smcd->priv;
   142	
   143		/* remove dmb from hash table */
   144		write_lock(&ldev->dmb_ht_lock);
   145		hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) {
   146			if (tmp_node->token == dmb->dmb_tok) {
   147				dmb_node = tmp_node;
   148				break;
   149			}
   150		}
   151		if (!dmb_node) {
   152			write_unlock(&ldev->dmb_ht_lock);
   153			return -EINVAL;
   154		}
   155		hash_del(&dmb_node->list);
   156		write_unlock(&ldev->dmb_ht_lock);
   157	
   158		clear_bit(dmb_node->sba_idx, ldev->sba_idx_mask);
 > 159		vfree(dmb_node->cpu_addr);

160 kfree(dmb_node);
161
162 return 0;
163 }
164

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author reviewer
#32 lore raw

Re: [PATCH net-next v4 12/18] net/smc: implement DMB-related operations of loopback

Quoted 27 lines · kernel test robot

Hi Wen,

kernel test robot noticed the following build errors:

[auto build test ERROR on net-next/main]

url:    https://github.com/intel-lab-lkp/linux/commits/Wen-Gu/net-smc-decouple-ism_dev-from-SMC-D-device-dump/20230924-231933
base:   net-next/main
patch link:    https://lore.kernel.org/r/1695568613-125057-13-git-send-email-guwen%40linux.alibaba.com
patch subject: [PATCH net-next v4 12/18] net/smc: implement DMB-related operations of loopback
config: mips-allmodconfig (https://download.01.org/0day-ci/archive/20230925/[email protected]/config)
compiler: mips-linux-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20230925/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All error/warnings (new ones prefixed by >>):

net/smc/smc_loopback.c: In function 'smc_lo_register_dmb':

Quoted 1 line
Quoted 1 line

net/smc/smc_loopback.c:102:30: error: implicit declaration of function 'vzalloc'; did you mean 'kvzalloc'? [-Werror=implicit-function-declaration]

      102 |         dmb_node->cpu_addr = vzalloc(dmb->dmb_len);
          |                              ^~~~~~~
          |                              kvzalloc
Quoted 1 line
Quoted 1 line

net/smc/smc_loopback.c:102:28: warning: assignment to 'void *' from 'int' makes pointer from integer without a cast [-Wint-conversion]

Quoted 9 lines

102 | dmb_node->cpu_addr = vzalloc(dmb->dmb_len);
| ^
net/smc/smc_loopback.c: In function 'smc_lo_unregister_dmb':

Quoted 1 line
Quoted 1 line

net/smc/smc_loopback.c:159:9: error: implicit declaration of function 'vfree'; did you mean 'kvfree'? [-Werror=implicit-function-declaration]

      159 |         vfree(dmb_node->cpu_addr);
          |         ^~~~~
          |         kvfree

cc1: some warnings being treated as errors

It can be fixed by including corresponding header file:

#include <linux/vmalloc.h>

Continue to wait for other review comments and will fix this in the next version.

Thanks.

Quoted 90 lines

vim +102 net/smc/smc_loopback.c

     79	
     80	static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
     81				       void *client_priv)
     82	{
     83		struct smc_lo_dmb_node *dmb_node, *tmp_node;
     84		struct smc_lo_dev *ldev = smcd->priv;
     85		int sba_idx, rc;
     86	
     87		/* check space for new dmb */
     88		for_each_clear_bit(sba_idx, ldev->sba_idx_mask, SMC_LODEV_MAX_DMBS) {
     89			if (!test_and_set_bit(sba_idx, ldev->sba_idx_mask))
     90				break;
     91		}
     92		if (sba_idx == SMC_LODEV_MAX_DMBS)
     93			return -ENOSPC;
     94	
     95		dmb_node = kzalloc(sizeof(*dmb_node), GFP_KERNEL);
     96		if (!dmb_node) {
     97			rc = -ENOMEM;
     98			goto err_bit;
     99		}
    100	
    101		dmb_node->sba_idx = sba_idx;
Quoted 1 line

102 dmb_node->cpu_addr = vzalloc(dmb->dmb_len);

    103		if (!dmb_node->cpu_addr) {
    104			rc = -ENOMEM;
    105			goto err_node;
    106		}
    107		dmb_node->len = dmb->dmb_len;
    108		dmb_node->dma_addr = SMC_DMA_ADDR_INVALID;
    109	
    110	again:
    111		/* add new dmb into hash table */
    112		get_random_bytes(&dmb_node->token, sizeof(dmb_node->token));
    113		write_lock(&ldev->dmb_ht_lock);
    114		hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_node->token) {
    115			if (tmp_node->token == dmb_node->token) {
    116				write_unlock(&ldev->dmb_ht_lock);
    117				goto again;
    118			}
    119		}
    120		hash_add(ldev->dmb_ht, &dmb_node->list, dmb_node->token);
    121		write_unlock(&ldev->dmb_ht_lock);
    122	
    123		dmb->sba_idx = dmb_node->sba_idx;
    124		dmb->dmb_tok = dmb_node->token;
    125		dmb->cpu_addr = dmb_node->cpu_addr;
    126		dmb->dma_addr = dmb_node->dma_addr;
    127		dmb->dmb_len = dmb_node->len;
    128	
    129		return 0;
    130	
    131	err_node:
    132		kfree(dmb_node);
    133	err_bit:
    134		clear_bit(sba_idx, ldev->sba_idx_mask);
    135		return rc;
    136	}
    137	
    138	static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
    139	{
    140		struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node;
    141		struct smc_lo_dev *ldev = smcd->priv;
    142	
    143		/* remove dmb from hash table */
    144		write_lock(&ldev->dmb_ht_lock);
    145		hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) {
    146			if (tmp_node->token == dmb->dmb_tok) {
    147				dmb_node = tmp_node;
    148				break;
    149			}
    150		}
    151		if (!dmb_node) {
    152			write_unlock(&ldev->dmb_ht_lock);
    153			return -EINVAL;
    154		}
    155		hash_del(&dmb_node->list);
    156		write_unlock(&ldev->dmb_ht_lock);
    157	
    158		clear_bit(dmb_node->sba_idx, ldev->sba_idx_mask);
Quoted 1 line

159 vfree(dmb_node->cpu_addr);

    160		kfree(dmb_node);
    161	
    162		return 0;
    163	}
    164	
author reviewer
#33 lore raw

PATCH v4 13/18 net/smc: register loopback device as SMC-Dv2 device

In reply to Wen Gu

After loopback device gets ready, add it to the smcd_dev list as an SMC-Dv2 device for use by SMC-D protocol.

---
 net/smc/smc_loopback.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
net/smc/smc_loopback.c +16 -7
index eb13291..8375575 100644
@@ -304,18 +304,27 @@ static int smcd_lo_register_dev(struct smc_lo_dev *ldev)
 
 	ldev->smcd = smcd;
 	smcd->priv = ldev;
-
-	/* TODO:
-	 * register smc_lo to smcd_dev list.
-	 */
+	mutex_lock(&smcd_dev_list.mutex);
+	smc_ism_check_v2_capable(smcd);
+	list_add(&smcd->list, &smcd_dev_list.list);
+	mutex_unlock(&smcd_dev_list.mutex);
+	pr_warn_ratelimited("smc: adding smcd device %s with pnetid %.16s%s\n",
+			    smc_lo_dev_name, smcd->pnetid,
+			    smcd->pnetid_by_user ? " (user defined)" : "");
 	return 0;
 }
 
 static void smcd_lo_unregister_dev(struct smc_lo_dev *ldev)
 {
-	/* TODO:
-	 * unregister smc_lo from smcd_dev list.
-	 */
+	struct smcd_dev *smcd = ldev->smcd;
+
+	pr_warn_ratelimited("smc: removing smcd device %s\n",
+			    smc_lo_dev_name);
+	smcd->going_away = 1;
+	smc_smcd_terminate_all(smcd);
+	mutex_lock(&smcd_dev_list.mutex);
+	list_del_init(&smcd->list);
+	mutex_unlock(&smcd_dev_list.mutex);
 }
 
 static void smc_lo_dev_release(struct device *dev)
Signature · 1 line
1.8.3.1
author reviewer
#34 lore raw

PATCH v4 14/18 net/smc: add operation for getting DMB attribute

In reply to Wen Gu

On s390, ISM devices are used on machine level hypervisors which is a partitioning hypervisor without paging. The sndbufs and peer DMBs in such case can't be mapped to the same physical memory.

However in other cases, such as communication within the same OS instance with loopback, the sndbufs and peer DMBs can be mapped to the same physical memory to avoid memory copy from sndbufs to peer DMBs.

So this patch introduces an operation to smcd_ops to judge whether the sndbufs-DMB map is available. And for reuse, the interface is designed to return DMB attribute, not only the sndbuf-DMB map feature.

---
 include/net/smc.h | 5 +++++
 net/smc/smc_ism.c | 8 ++++++++
 net/smc/smc_ism.h | 1 +
 3 files changed, 14 insertions(+)
include/net/smc.h +5 -0
index 7c2d35c..917572fb 100644
@@ -55,6 +55,10 @@ struct smcd_seid {
 
 #define ISM_ERROR	0xFFFF
 
+enum {
+	ISM_ATTR_DMB_MAP = 0,
+};
+
 struct smcd_dev;
 
 struct smcd_gid {
@@ -82,6 +86,7 @@ struct smcd_ops {
 	void (*get_local_gid)(struct smcd_dev *dev, struct smcd_gid *gid);
 	u16 (*get_chid)(struct smcd_dev *dev);
 	struct device* (*get_dev)(struct smcd_dev *dev);
+	int (*get_dev_attr)(struct smcd_dev *dev);
 };
 
 struct smcd_dev {
net/smc/smc_ism.c +8 -0
index 0922fab..9b31d00 100644
@@ -214,6 +214,14 @@ int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc)
 	return rc;
 }
 
+bool smc_ism_dmb_mappable(struct smcd_dev *smcd)
+{
+	if (smcd->ops->get_dev_attr &&
+	    (smcd->ops->get_dev_attr(smcd) & BIT(ISM_ATTR_DMB_MAP)))
+		return true;
+	return false;
+}
+
 int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len,
 			 struct smc_buf_desc *dmb_desc)
 {
net/smc/smc_ism.h +1 -0
index 7ab82dd..cef212c 100644
@@ -44,6 +44,7 @@ int smc_ism_cantalk(struct smcd_gid *peer_gid, unsigned short vlan_id,
 int smc_ism_register_dmb(struct smc_link_group *lgr, int buf_size,
 			 struct smc_buf_desc *dmb_desc);
 int smc_ism_unregister_dmb(struct smcd_dev *dev, struct smc_buf_desc *dmb_desc);
+bool smc_ism_dmb_mappable(struct smcd_dev *smcd);
 int smc_ism_signal_shutdown(struct smc_link_group *lgr);
 void smc_ism_get_system_eid(u8 **eid);
 u16 smc_ism_get_chid(struct smcd_dev *dev);
Signature · 1 line
1.8.3.1
author reviewer
#35 lore raw

PATCH v4 15/18 net/smc: add operations for DMB attach and detach

In reply to Wen Gu

This patch extends smcd_ops, adding two more semantic for SMC-D DMB:

- attach_dmb:
Attach an already registered DMB to a specific buf_desc, so that we can refer to the DMB through this buf_desc.

- detach_dmb:
Reverse operation of attach_dmb. detach the DMB from the buf_desc.

This interface extension is to prepare for the avoidance of memory copy from sndbuf to RMB with SMC-D device whose DMBs has ISM_ATTR_DMB_MAP attribute.

---
 include/net/smc.h |  2 ++
 net/smc/smc_ism.c | 31 +++++++++++++++++++++++++++++++
 net/smc/smc_ism.h |  2 ++
 3 files changed, 35 insertions(+)
include/net/smc.h +2 -0
index 917572fb..39a21ed 100644
@@ -72,6 +72,8 @@ struct smcd_ops {
 	int (*register_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb,
 			    void *client);
 	int (*unregister_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb);
+	int (*attach_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb);
+	int (*detach_dmb)(struct smcd_dev *dev, u64 token);
 	int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
 	int (*del_vlan_id)(struct smcd_dev *dev, u64 vlan_id);
 	int (*set_vlan_required)(struct smcd_dev *dev);
net/smc/smc_ism.c +31 -0
index 9b31d00..32d5d96 100644
@@ -244,6 +244,37 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len,
 	return rc;
 }
 
+int smc_ism_attach_dmb(struct smcd_dev *dev, u64 token,
+		       struct smc_buf_desc *dmb_desc)
+{
+	struct smcd_dmb dmb;
+	int rc = 0;
+
+	memset(&dmb, 0, sizeof(dmb));
+	dmb.dmb_tok = token;
+
+	if (!dev->ops->attach_dmb)
+		return -EINVAL;
+
+	rc = dev->ops->attach_dmb(dev, &dmb);
+	if (!rc) {
+		dmb_desc->sba_idx = dmb.sba_idx;
+		dmb_desc->token = dmb.dmb_tok;
+		dmb_desc->cpu_addr = dmb.cpu_addr;
+		dmb_desc->dma_addr = dmb.dma_addr;
+		dmb_desc->len = dmb.dmb_len;
+	}
+	return rc;
+}
+
+int smc_ism_detach_dmb(struct smcd_dev *dev, u64 token)
+{
+	if (!dev->ops->detach_dmb)
+		return -EINVAL;
+
+	return dev->ops->detach_dmb(dev, token);
+}
+
 static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd,
 				  struct sk_buff *skb,
 				  struct netlink_callback *cb)
net/smc/smc_ism.h +2 -0
index cef212c..f30aae4 100644
@@ -45,6 +45,8 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int buf_size,
 			 struct smc_buf_desc *dmb_desc);
 int smc_ism_unregister_dmb(struct smcd_dev *dev, struct smc_buf_desc *dmb_desc);
 bool smc_ism_dmb_mappable(struct smcd_dev *smcd);
+int smc_ism_attach_dmb(struct smcd_dev *dev, u64 token, struct smc_buf_desc *dmb_desc);
+int smc_ism_detach_dmb(struct smcd_dev *dev, u64 token);
 int smc_ism_signal_shutdown(struct smc_link_group *lgr);
 void smc_ism_get_system_eid(u8 **eid);
 u16 smc_ism_get_chid(struct smcd_dev *dev);
Signature · 1 line
1.8.3.1
author reviewer
#36 lore raw

PATCH v4 16/18 net/smc: avoid data copy from sndbuf to peer RMB in SMC-D

In reply to Wen Gu

This patch aims to avoid data copy from local sndbuf to peer RMB by mapping local sndbuf to peer RMB when DMBs have ISM_ATTR_DMB_MAP attribute.

After this, local sndbuf and peer RMB share the same physical memory.

 +----------+                     +----------+
 | socket A |                     | socket B |
 +----------+                     +----------+
       |                               ^
       |         +---------+           |
  regard as      |         |     regard as
  local sndbuf   |  B's    |     local RMB
       |         |  RMB    |           |
       |-------> |         |-----------|
                 +---------+

1. From the perspective of RMB:

a. Created or reused when connection is created.
b. Unused and recycled to lgr buffer pool when connection is freed. c. Freed when link group is freed.

2. From the perspective of sndbuf:

a. Mapped to peer RMB by the rtoken exchanged through CLC message.
Then accessing local sndbuf is equivalent to accessing peer RMB.
c. Unmapped from peer RMB and freed when connection is freed. Won't be
recycled to lgr buffer pool.

Therefore, the data written to local sndbuf will directly reach peer RMB.

---
 net/smc/af_smc.c   | 14 +++++++++++
 net/smc/smc_core.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 net/smc/smc_core.h |  1 +
 3 files changed, 84 insertions(+), 1 deletion(-)
net/smc/af_smc.c +14 -0
index 6435659..47556c3 100644
@@ -1442,6 +1442,12 @@ static int smc_connect_ism(struct smc_sock *smc,
 	}
 
 	smc_conn_save_peer_info(smc, aclc);
+
+	if (smc_ism_dmb_mappable(smc->conn.lgr->smcd)) {
+		rc = smcd_buf_attach(smc);
+		if (rc)
+			goto connect_abort;
+	}
 	smc_close_init(smc);
 	smc_rx_init(smc);
 	smc_tx_init(smc);
@@ -2550,6 +2556,14 @@ static void smc_listen_work(struct work_struct *work)
 		mutex_unlock(&smc_server_lgr_pending);
 	}
 	smc_conn_save_peer_info(new_smc, cclc);
+
+	if (ini->is_smcd &&
+	    smc_ism_dmb_mappable(new_smc->conn.lgr->smcd)) {
+		rc = smcd_buf_attach(new_smc);
+		if (rc)
+			goto out_decl;
+	}
+
 	smc_listen_out_connected(new_smc);
 	SMC_STAT_SERV_SUCC_INC(sock_net(newclcsock->sk), ini);
 	goto out_free;
net/smc/smc_core.c +69 -1

102 lines not on this page · raw message

net/smc/smc_core.h +1 -0
index d57eb9b..2cba119 100644
@@ -551,6 +551,7 @@ void smc_smcd_terminate(struct smcd_dev *dev, struct smcd_gid *peer_gid,
 void smc_smcd_terminate_all(struct smcd_dev *dev);
 void smc_smcr_terminate_all(struct smc_ib_device *smcibdev);
 int smc_buf_create(struct smc_sock *smc, bool is_smcd);
+int smcd_buf_attach(struct smc_sock *smc);
 int smc_uncompress_bufsize(u8 compressed);
 int smc_rmb_rtoken_handling(struct smc_connection *conn, struct smc_link *link,
 			    struct smc_clc_msg_accept_confirm *clc);
Signature · 1 line
1.8.3.1
author reviewer
#37 lore raw

PATCH v4 17/18 net/smc: modify cursor update logic when sndbuf mapped to RMB

In reply to Wen Gu

Since local sndbuf shares the same physical memory with peer RMB after mapping, the logic related to cursor update needs to be adapted to ensure that the data written by local won't overwrite the data that has not been consumed by the peer.

So in this case, the fin_curs and sndbuf_space that were originally updated after sending the CDC message should be modified to not be update until the peer updates cons_curs.

---
 net/smc/smc_cdc.c | 50 +++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 39 insertions(+), 11 deletions(-)
net/smc/smc_cdc.c +39 -11
index 2641800..7cc677d 100644
@@ -18,6 +18,7 @@
 #include "smc_tx.h"
 #include "smc_rx.h"
 #include "smc_close.h"
+#include "smc_ism.h"
 
 /********************************** send *************************************/
 
@@ -256,17 +257,24 @@ int smcd_cdc_msg_send(struct smc_connection *conn)
 		return rc;
 	smc_curs_copy(&conn->rx_curs_confirmed, &curs, conn);
 	conn->local_rx_ctrl.prod_flags.cons_curs_upd_req = 0;
-	/* Calculate transmitted data and increment free send buffer space */
-	diff = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin,
-			     &conn->tx_curs_sent);
-	/* increased by confirmed number of bytes */
-	smp_mb__before_atomic();
-	atomic_add(diff, &conn->sndbuf_space);
-	/* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */
-	smp_mb__after_atomic();
-	smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn);
+	if (!smc_ism_dmb_mappable(conn->lgr->smcd)) {
+		/* If local sndbuf has been mapped to peer RMB, then
+		 * don't update the tx_curs_fin and sndbuf_space until
+		 * peer has consumed the data in RMB.
+		 */
 
-	smc_tx_sndbuf_nonfull(smc);
+		/* Calculate transmitted data and increment free send buffer space */
+		diff = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin,
+				     &conn->tx_curs_sent);
+		/* increased by confirmed number of bytes */
+		smp_mb__before_atomic();
+		atomic_add(diff, &conn->sndbuf_space);
+		/* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */
+		smp_mb__after_atomic();
+		smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn);
+
+		smc_tx_sndbuf_nonfull(smc);
+	}
 	return rc;
 }
 
@@ -324,7 +332,7 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc,
 {
 	union smc_host_cursor cons_old, prod_old;
 	struct smc_connection *conn = &smc->conn;
-	int diff_cons, diff_prod;
+	int diff_cons, diff_prod, diff_tx;
 
 	smc_curs_copy(&prod_old, &conn->local_rx_ctrl.prod, conn);
 	smc_curs_copy(&cons_old, &conn->local_rx_ctrl.cons, conn);
@@ -340,6 +348,26 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc,
 		atomic_add(diff_cons, &conn->peer_rmbe_space);
 		/* guarantee 0 <= peer_rmbe_space <= peer_rmbe_size */
 		smp_mb__after_atomic();
+
+		if (conn->lgr->is_smcd &&
+		    smc_ism_dmb_mappable(conn->lgr->smcd)) {
+			/* If local sndbuf has been mapped to peer RMB, then
+			 * update tx_curs_fin and sndbuf_space when peer has
+			 * consumed the data in it's RMB.
+			 */
+
+			/* calculate peer rmb consumed data */
+			diff_tx = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin,
+						&conn->local_rx_ctrl.cons);
+			/* increase local sndbuf space and fin_curs */
+			smp_mb__before_atomic();
+			atomic_add(diff_tx, &conn->sndbuf_space);
+			/* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */
+			smp_mb__after_atomic();
+			smc_curs_copy(&conn->tx_curs_fin, &conn->local_rx_ctrl.cons, conn);
+
+			smc_tx_sndbuf_nonfull(smc);
+		}
 	}
 
 	diff_prod = smc_curs_diff(conn->rmb_desc->len, &prod_old,
Signature · 1 line
1.8.3.1
author reviewer
#38 lore raw

PATCH v4 18/18 net/smc: add interface implementation of loopback device

In reply to Wen Gu

This patch completes the specific implementation of loopback device for the newly added SMC-D DMB-related interface.

The loopback device always provides mappable DMB because the device users are in the same OS instance.

---
 net/smc/smc_loopback.c | 104 ++++++++++++++++++++++++++++++++++++++++++++-----
 net/smc/smc_loopback.h |   5 +++
 2 files changed, 99 insertions(+), 10 deletions(-)
net/smc/smc_loopback.c +94 -10
index 8375575..e9d2d62 100644
@@ -106,6 +106,7 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
 	}
 	dmb_node->len = dmb->dmb_len;
 	dmb_node->dma_addr = SMC_DMA_ADDR_INVALID;
+	refcount_set(&dmb_node->refcnt, 1);
 
 again:
 	/* add new dmb into hash table */
@@ -119,6 +120,7 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
 	}
 	hash_add(ldev->dmb_ht, &dmb_node->list, dmb_node->token);
 	write_unlock(&ldev->dmb_ht_lock);
+	atomic_inc(&ldev->dmb_cnt);
 
 	dmb->sba_idx = dmb_node->sba_idx;
 	dmb->dmb_tok = dmb_node->token;
@@ -140,18 +142,27 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
 	struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node;
 	struct smc_lo_dev *ldev = smcd->priv;
 
-	/* remove dmb from hash table */
-	write_lock(&ldev->dmb_ht_lock);
+	/* find dmb from hash table */
+	read_lock(&ldev->dmb_ht_lock);
 	hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) {
 		if (tmp_node->token == dmb->dmb_tok) {
 			dmb_node = tmp_node;
+			dmb_node->freeing = 1;
 			break;
 		}
 	}
 	if (!dmb_node) {
-		write_unlock(&ldev->dmb_ht_lock);
+		read_unlock(&ldev->dmb_ht_lock);
 		return -EINVAL;
 	}
+	read_unlock(&ldev->dmb_ht_lock);
+
+	/* wait for dmb refcnt to be 0 */
+	if (!refcount_dec_and_test(&dmb_node->refcnt))
+		wait_event(ldev->dmbs_release, !refcount_read(&dmb_node->refcnt));
+
+	/* remove dmb from hash table */
+	write_lock(&ldev->dmb_ht_lock);
 	hash_del(&dmb_node->list);
 	write_unlock(&ldev->dmb_ht_lock);
 
@@ -159,9 +170,69 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
 	vfree(dmb_node->cpu_addr);
 	kfree(dmb_node);
 
+	if (atomic_dec_and_test(&ldev->dmb_cnt))
+		wake_up(&ldev->ldev_release);
 	return 0;
 }
 
+static int smc_lo_attach_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
+{
+	struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node;
+	struct smc_lo_dev *ldev = smcd->priv;
+
+	/* find dmb_node according to dmb->dmb_tok */
+	read_lock(&ldev->dmb_ht_lock);
+	hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) {
+		if (tmp_node->token == dmb->dmb_tok && !tmp_node->freeing) {
+			dmb_node = tmp_node;
+			break;
+		}
+	}
+	if (!dmb_node) {
+		read_unlock(&ldev->dmb_ht_lock);
+		return -EINVAL;
+	}
+	refcount_inc(&dmb_node->refcnt);
+	read_unlock(&ldev->dmb_ht_lock);
+
+	/* provide dmb information */
+	dmb->sba_idx = dmb_node->sba_idx;
+	dmb->dmb_tok = dmb_node->token;
+	dmb->cpu_addr = dmb_node->cpu_addr;
+	dmb->dma_addr = dmb_node->dma_addr;
+	dmb->dmb_len = dmb_node->len;
+	return 0;
+}
+
+static int smc_lo_detach_dmb(struct smcd_dev *smcd, u64 token)
+{
+	struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node;
+	struct smc_lo_dev *ldev = smcd->priv;
+
+	/* find dmb_node according to dmb->dmb_tok */
+	read_lock(&ldev->dmb_ht_lock);
+	hash_for_each_possible(ldev->dmb_ht, tmp_node, list, token) {
+		if (tmp_node->token == token) {
+			dmb_node = tmp_node;
+			break;
+		}
+	}
+	if (!dmb_node) {
+		read_unlock(&ldev->dmb_ht_lock);
+		return -EINVAL;
+	}
+	read_unlock(&ldev->dmb_ht_lock);
+
+	if (refcount_dec_and_test(&dmb_node->refcnt))
+		wake_up_all(&ldev->dmbs_release);
+	return 0;
+}
+
+static int smc_lo_get_dev_attr(struct smcd_dev *smcd)
+{
+	return BIT(ISM_ATTR_DMB_MAP);
+}
+
 static int smc_lo_add_vlan_id(struct smcd_dev *smcd, u64 vlan_id)
 {
 	return -EOPNOTSUPP;
@@ -194,7 +265,15 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok, unsigned int idx
 {
 	struct smc_lo_dmb_node *rmb_node = NULL, *tmp_node;
 	struct smc_lo_dev *ldev = smcd->priv;
-
+	struct smc_connection *conn;
+
+	if (!sf) {
+		/* local sndbuf shares the same physical memory with
+		 * peer RMB, so no need to copy data from local sndbuf
+		 * to peer RMB.
+		 */
+		return 0;
+	}
 	read_lock(&ldev->dmb_ht_lock);
 	hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_tok) {
 		if (tmp_node->token == dmb_tok) {
@@ -210,13 +289,10 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok, unsigned int idx
 
 	memcpy((char *)rmb_node->cpu_addr + offset, data, size);
 
-	if (sf) {
-		struct smc_connection *conn =
-			smcd->conn[rmb_node->sba_idx];
+	conn = smcd->conn[rmb_node->sba_idx];
+	if (conn && !conn->killed)
+		smcd_cdc_rx_handler(conn);
 
-		if (conn && !conn->killed)
-			smcd_cdc_rx_handler(conn);
-	}
 	return 0;
 }
 
@@ -253,6 +329,8 @@ static struct device *smc_lo_get_dev(struct smcd_dev *smcd)
 	.query_remote_gid = smc_lo_query_rgid,
 	.register_dmb = smc_lo_register_dmb,
 	.unregister_dmb = smc_lo_unregister_dmb,
+	.attach_dmb = smc_lo_attach_dmb,
+	.detach_dmb = smc_lo_detach_dmb,
 	.add_vlan_id = smc_lo_add_vlan_id,
 	.del_vlan_id = smc_lo_del_vlan_id,
 	.set_vlan_required = smc_lo_set_vlan_required,
@@ -264,6 +342,7 @@ static struct device *smc_lo_get_dev(struct smcd_dev *smcd)
 	.get_local_gid = smc_lo_get_local_gid,
 	.get_chid = smc_lo_get_chid,
 	.get_dev = smc_lo_get_dev,
+	.get_dev_attr = smc_lo_get_dev_attr,
 };
 
 static struct smcd_dev *smcd_lo_alloc_dev(const struct smcd_ops *ops,
@@ -343,6 +422,9 @@ static int smc_lo_dev_init(struct smc_lo_dev *ldev)
 	smc_lo_generate_id(ldev);
 	rwlock_init(&ldev->dmb_ht_lock);
 	hash_init(ldev->dmb_ht);
+	atomic_set(&ldev->dmb_cnt, 0);
+	init_waitqueue_head(&ldev->dmbs_release);
+	init_waitqueue_head(&ldev->ldev_release);
 
 	return smcd_lo_register_dev(ldev);
 }
@@ -376,6 +458,8 @@ static int smc_lo_dev_probe(void)
 static void smc_lo_dev_exit(struct smc_lo_dev *ldev)
 {
 	smcd_lo_unregister_dev(ldev);
+	if (atomic_read(&ldev->dmb_cnt))
+		wait_event(ldev->ldev_release, !atomic_read(&ldev->dmb_cnt));
 }
 
 static void smc_lo_dev_remove(void)
net/smc/smc_loopback.h +5 -0
index a5b501b..e42c807 100644
@@ -30,6 +30,8 @@ struct smc_lo_dmb_node {
 	u32 sba_idx;
 	void *cpu_addr;
 	dma_addr_t dma_addr;
+	refcount_t refcnt;
+	u8 freeing : 1;
 };
 
 struct smc_lo_dev {
@@ -40,6 +42,9 @@ struct smc_lo_dev {
 	DECLARE_BITMAP(sba_idx_mask, SMC_LODEV_MAX_DMBS);
 	rwlock_t dmb_ht_lock;
 	DECLARE_HASHTABLE(dmb_ht, SMC_LODEV_DMBS_HASH_BITS);
+	atomic_t dmb_cnt;
+	wait_queue_head_t dmbs_release;
+	wait_queue_head_t ldev_release;
 };
 #endif
 
Signature · 1 line
1.8.3.1
maintainer
#39 lore raw
In reply to Wen Gu
Quoted 6 lines · Wen Gu

This patch set includes 4 parts:

- Patch #1-#3: decouple ISM device hard code from SMC-D stack.
- Patch #4-#8: implement virtual ISM extension defined in SMCv2.1.
- Patch #9-#13: implement loopback-ism device.
- Patch #14-#18: memory copy optimization for the case using loopback.

Your cover letter is very well helpful, thanks a lot for that. I really like the way you grouped the patches.
Just a thought:
If it takes too long to get this large patchset reviewed, you could split it up into smaller sets and get them upstream one after the other.

I think it is especially valuable to more crisply define the interface between SMC-D and the smc-d devices, given that virtio-ism may soon be a third kind of smcd device.

maintainer
#40 lore raw
In reply to Wen Gu
Quoted 35 lines · Wen Gu

Wen Gu (18):
net/smc: decouple ism_dev from SMC-D device dump net/smc: decouple ism_dev from SMC-D DMB registration net/smc: extract v2 check helper from SMC-D device registration net/smc: support SMCv2.x supplemental features negotiation net/smc: reserve CHID range for SMC-D virtual device net/smc: extend GID to 128bits only for virtual ISM device net/smc: disable SEID on non-s390 architecture net/smc: enable virtual ISM device feature bit net/smc: introduce SMC-D loopback device
net/smc: implement ID-related operations of loopback net/smc: implement some unsupported operations of loopback net/smc: implement DMB-related operations of loopback net/smc: register loopback device as SMC-Dv2 device net/smc: add operation for getting DMB attribute net/smc: add operations for DMB attach and detach net/smc: avoid data copy from sndbuf to peer RMB in SMC-D net/smc: modify cursor update logic when sndbuf mapped to RMB net/smc: add interface implementation of loopback device

 drivers/s390/net/ism_drv.c    |  20 +-
 include/net/smc.h             |  32 ++-
 include/uapi/linux/smc.h      |   3 +
 include/uapi/linux/smc_diag.h |   2 +
 net/smc/Kconfig               |  13 ++
 net/smc/Makefile              |   2 +-
 net/smc/af_smc.c              |  88 ++++++--
 net/smc/smc.h                 |   7 +
 net/smc/smc_cdc.c             |  56 ++++-
 net/smc/smc_cdc.h             |   1 +
 net/smc/smc_clc.c             |  64 ++++--
 net/smc/smc_clc.h             |  10 +-
 net/smc/smc_core.c            | 111 +++++++++-
 net/smc/smc_core.h            |   9 +-
 net/smc/smc_diag.c            |  11 +-
Quoted 8 lines
 net/smc/smc_ism.c             | 100 ++++++---
 net/smc/smc_ism.h             |  24 ++-
 net/smc/smc_loopback.c        | 489 ++++++++++++++++++++++++++++++++++++++++++
 net/smc/smc_loopback.h        |  54 +++++
 net/smc/smc_pnet.c            |   4 +-
 20 files changed, 996 insertions(+), 104 deletions(-)
 create mode 100644 net/smc/smc_loopback.c
 create mode 100644 net/smc/smc_loopback.h

Hello Wen Gu,

I applied and built your patches and noticed some things that you may want to consider in the next version:

Series should be split up [2]

Several lines exceed 80 columns [1][3]

'git clang-format HEAD~18' finds several formatting issues.
Maybe not all of them need to be fixed.

codespell *.patch 0006-net-smc-extend-GID-to-128bits-only-for-virtual-ISM-d.patch:7: protocal ==> protocol

With your patches applied I get some new warnings [4]:
Seems there are some ntoh conversions missing

CHECK net/smc/af_smc.c
net/smc/af_smc.c:723:32: warning: cast to restricted __be64 net/smc/af_smc.c:1427:52: warning: cast to restricted __be64

  CHECK   net/smc/smc_pnet.c
  CHECK   net/smc/smc_ib.c
  CHECK   net/smc/smc_clc.c

net/smc/smc_clc.c:954:72: warning: incorrect type in argument 1 (different base types)
net/smc/smc_clc.c:954:72: expected unsigned short [usertype] chid
net/smc/smc_clc.c:954:72: got restricted __be16 [usertype] chid
net/smc/smc_clc.c:1050:29: warning: incorrect type in assignment (different base types)
net/smc/smc_clc.c:1050:29: expected unsigned long long [usertype] gid
net/smc/smc_clc.c:1050:29: got restricted __be64 [usertype]
net/smc/smc_clc.c:1051:31: warning: incorrect type in assignment (different base types)
net/smc/smc_clc.c:1051:31: expected unsigned long long [usertype] token
net/smc/smc_clc.c:1051:31: got restricted __be64 [usertype]

[1] linux/Documentation/process/coding-style.rst
[2] https://www.kernel.org/doc/html/v6.3/process/maintainer-netdev.html?highlight=network
[3] scripts/checkpatch.pl --strict --max-line-length=80 --git HEAD-18
[4] make C=2 CF=-D__CHECK_ENDIAN__ M=net/smc -Wunused-function -Wimplicit-fallthrough -Wincompatible-function-pointer-types-strict

When I installed the patches, I noticed that

Quoted 1 line

smcd info

showed an SEID, even though I had no ISM device --> good

Quoted 1 line

smcd device

FID  Type  PCI-ID        PCHID  InUse  #LGs  PNET-ID
0000 0                   0000   Yes       2

This needs some improvements.., but I'm not sure what is the best way to display virtual smcd interfaces in the smc-tools.

I was able to do SMC transfers via the smcd-loopback feature :-D

author reviewer
#42 lore raw
Quoted 5 lines · Alexandra Winter
Quoted 2 lines · Alexandra Winter

Hello Wen Gu,

Quoted 7 lines
Quoted 1 line

I applied and built your patches and noticed some things that you may want to consider in the next version:

FYI, patchwork basically complains about many the same issues:
https://patchwork.kernel.org/project/netdevbpf/list/?series=787037&state=*

In general you should run those check BEFORE you send the patches and not rely on patchwork.

Thank you Sandy. I seem to have not seen the specific content of these checks. May I ask how to run those patchwork check locally? So that I can make sure everything is ok before send them.

maintainer
#43 lore raw
Quoted 11 lines · Wen Gu
Quoted 8 lines · Alexandra Winter
Quoted 3 lines · Alexandra Winter

Hello Wen Gu,

I applied and built your patches and noticed some things that you may want to consider in the next version:

Quoted 7 lines
Quoted 4 lines

FYI, patchwork basically complains about many the same issues:
https://patchwork.kernel.org/project/netdevbpf/list/?series=787037&state=*

In general you should run those check BEFORE you send the patches and not rely on patchwork.

Thank you Sandy. I seem to have not seen the specific content of these checks. May I ask how to run those patchwork check locally? So that I can make sure everything is ok before send them.

Citing from Documentation/process/maintainer-netdev.rst :

"patchwork checks
~~~~~~~~~~~~~~~~

Checks in patchwork are mostly simple wrappers around existing kernel scripts, the sources are available at:

https://github.com/kuba-moo/nipa/tree/master/tests

**Do not** post your patches just to run them through the checks. You must ensure that your patches are ready by testing them locally before posting to the mailing list. The patchwork build bot instance gets overloaded very easily and netdev@vger really doesn't need more traffic if we can help it."

HTH

author reviewer
#44 lore raw
Quoted 32 lines · Alexandra Winter
Quoted 11 lines · Wen Gu
Quoted 8 lines · Alexandra Winter
Quoted 3 lines · Alexandra Winter

Hello Wen Gu,

I applied and built your patches and noticed some things that you may want to consider in the next version:

Quoted 7 lines
Quoted 4 lines

FYI, patchwork basically complains about many the same issues:
https://patchwork.kernel.org/project/netdevbpf/list/?series=787037&state=*

In general you should run those check BEFORE you send the patches and not rely on patchwork.

Thank you Sandy. I seem to have not seen the specific content of these checks. May I ask how to run those patchwork check locally? So that I can make sure everything is ok before send them.

Citing from Documentation/process/maintainer-netdev.rst :

"patchwork checks
~~~~~~~~~~~~~~~~

Checks in patchwork are mostly simple wrappers around existing kernel scripts, the sources are available at:

https://github.com/kuba-moo/nipa/tree/master/tests

Quoted 7 lines

**Do not** post your patches just to run them through the checks. You must ensure that your patches are ready by testing them locally before posting to the mailing list. The patchwork build bot instance gets overloaded very easily and netdev@vger really doesn't need more traffic if we can help it."

HTH

Thank you! Sandy.

author reviewer
#45 lore raw
In reply to Alexandra Winter
Quoted 82 lines · Alexandra Winter
Quoted 41 lines · Wen Gu

Wen Gu (18):
net/smc: decouple ism_dev from SMC-D device dump net/smc: decouple ism_dev from SMC-D DMB registration net/smc: extract v2 check helper from SMC-D device registration net/smc: support SMCv2.x supplemental features negotiation net/smc: reserve CHID range for SMC-D virtual device net/smc: extend GID to 128bits only for virtual ISM device net/smc: disable SEID on non-s390 architecture net/smc: enable virtual ISM device feature bit net/smc: introduce SMC-D loopback device
net/smc: implement ID-related operations of loopback net/smc: implement some unsupported operations of loopback net/smc: implement DMB-related operations of loopback net/smc: register loopback device as SMC-Dv2 device net/smc: add operation for getting DMB attribute net/smc: add operations for DMB attach and detach net/smc: avoid data copy from sndbuf to peer RMB in SMC-D net/smc: modify cursor update logic when sndbuf mapped to RMB net/smc: add interface implementation of loopback device

  drivers/s390/net/ism_drv.c    |  20 +-
  include/net/smc.h             |  32 ++-
  include/uapi/linux/smc.h      |   3 +
  include/uapi/linux/smc_diag.h |   2 +
  net/smc/Kconfig               |  13 ++
  net/smc/Makefile              |   2 +-
  net/smc/af_smc.c              |  88 ++++++--
  net/smc/smc.h                 |   7 +
  net/smc/smc_cdc.c             |  56 ++++-
  net/smc/smc_cdc.h             |   1 +
  net/smc/smc_clc.c             |  64 ++++--
  net/smc/smc_clc.h             |  10 +-
  net/smc/smc_core.c            | 111 +++++++++-
  net/smc/smc_core.h            |   9 +-
  net/smc/smc_diag.c            |  11 +-
  net/smc/smc_ism.c             | 100 ++++++---
  net/smc/smc_ism.h             |  24 ++-
  net/smc/smc_loopback.c        | 489 ++++++++++++++++++++++++++++++++++++++++++
  net/smc/smc_loopback.h        |  54 +++++
  net/smc/smc_pnet.c            |   4 +-
  20 files changed, 996 insertions(+), 104 deletions(-)
Quoted 2 lines

create mode 100644 net/smc/smc_loopback.c
create mode 100644 net/smc/smc_loopback.h

Hello Wen Gu,

I applied and built your patches and noticed some things that you may want to consider in the next version:

Series should be split up [2]

Several lines exceed 80 columns [1][3]

'git clang-format HEAD~18' finds several formatting issues.
Maybe not all of them need to be fixed.

codespell *.patch 0006-net-smc-extend-GID-to-128bits-only-for-virtual-ISM-d.patch:7: protocal ==> protocol

With your patches applied I get some new warnings [4]:
Seems there are some ntoh conversions missing

CHECK net/smc/af_smc.c
net/smc/af_smc.c:723:32: warning: cast to restricted __be64 net/smc/af_smc.c:1427:52: warning: cast to restricted __be64

   CHECK   net/smc/smc_pnet.c
   CHECK   net/smc/smc_ib.c
   CHECK   net/smc/smc_clc.c

net/smc/smc_clc.c:954:72: warning: incorrect type in argument 1 (different base types)
net/smc/smc_clc.c:954:72: expected unsigned short [usertype] chid
net/smc/smc_clc.c:954:72: got restricted __be16 [usertype] chid
net/smc/smc_clc.c:1050:29: warning: incorrect type in assignment (different base types)
net/smc/smc_clc.c:1050:29: expected unsigned long long [usertype] gid
net/smc/smc_clc.c:1050:29: got restricted __be64 [usertype]
net/smc/smc_clc.c:1051:31: warning: incorrect type in assignment (different base types)
net/smc/smc_clc.c:1051:31: expected unsigned long long [usertype] token
net/smc/smc_clc.c:1051:31: got restricted __be64 [usertype]

Quoted 6 lines

[1] linux/Documentation/process/coding-style.rst
[2] https://www.kernel.org/doc/html/v6.3/process/maintainer-netdev.html?highlight=network
[3] scripts/checkpatch.pl --strict --max-line-length=80 --git HEAD-18
[4] make C=2 CF=-D__CHECK_ENDIAN__ M=net/smc -Wunused-function -Wimplicit-fallthrough -Wincompatible-function-pointer-types-strict

Hi Sandy,

Thank you very much for your detailed comments. They are really helpful. I will check and fix them in my next version.

Quoted 5 lines

When I installed the patches, I noticed that

Quoted 1 line

smcd info

showed an SEID, even though I had no ISM device --> good

Yes, virtual ISM device will return the right SEID (same as that returned by ISM device) on s390 arch.

Quoted 7 lines
Quoted 2 lines

smcd device

FID  Type  PCI-ID        PCHID  InUse  #LGs  PNET-ID
0000 0                   0000   Yes       2

This needs some improvements.., but I'm not sure what is the best way to display virtual smcd interfaces in the smc-tools.

Right. My idea is to add the display of the device's name and GID along with the PCI information (all zero) to indicate it's virtual smcd device.

Quoted 3 lines

I was able to do SMC transfers via the smcd-loopback feature :-D

Glad to hear this. I will improve the issues you mentioned in the next version and next version will be split up.

Thank you very much.
Wen Gu

#46 lore raw
In reply to Wen Gu
Quoted 111 lines · Wen Gu

Hi, all

# Background

SMC-D is now used in IBM z with ISM function to optimize network interconnect for intra-CPC communications. Inspired by this, we try to make SMC-D available on the non-s390 architecture through a software-simulated virtual ISM device, such as loopback-ism device here, to accelerate inter-process or inter-containers communication within the same OS.

# Design

This patch set includes 4 parts:

- Patch #1-#3: decouple ISM device hard code from SMC-D stack.
- Patch #4-#8: implement virtual ISM extension defined in SMCv2.1.
- Patch #9-#13: implement loopback-ism device.
- Patch #14-#18: memory copy optimization for the case using loopback.

The loopback-ism device is designed as a kernel device and not be limited to a specific net namespace, ends of both inter-process connection (1/1' in diagram below) or inter-container connection (2/2' in diagram below) will find that peer shares the same loopback-ism device during the CLC handshake. Then loopback-ism device will be chosen.

 Container 1 (ns1)                              Container 2 (ns2)
 +-----------------------------------------+    +-------------------------+
 | +-------+      +-------+      +-------+ |    |        +-------+        |
 | | App A |      | App B |      | App C | |    |        | App D |<-+     |
 | +-------+      +---^---+      +-------+ |    |        +-------+  |(2') |
 |     |127.0.0.1 (1')|             |192.168.0.11       192.168.0.12|     |
 |  (1)|   +--------+ | +--------+  |(2)   |    | +--------+   +--------+ |
 |     `-->|   lo   |-` |  eth0  |<-`      |    | |   lo   |   |  eth0  | |
 +---------+--|---^-+---+-----|--+---------+    +-+--------+---+-^------+-+
              |   |           |                                  |
 Kernel       |   |           |                                  |
 +----+-------v---+-----------v----------------------------------+---+----+
 |    |                            TCP                               |    |
 |    |                                                              |    |
 |    +--------------------------------------------------------------+    |
 |                                                                        |
 |                           +--------------+                             |
 |                           | smc loopback |                             |
 +---------------------------+--------------+-----------------------------+

loopback-ism device allocs RMBs and sndbufs for each connection peer and 'moves' data from sndbuf at one end to RMB at the other end. Since communication occurs within the same kernel, the sndbuf can be mapped to peer RMB so that the data copy in loopback-ism case can be avoided.

 Container 1 (ns1)                              Container 2 (ns2)
 +-----------------------------------------+    +-------------------------+
 | +-------+      +-------+      +-------+ |    |        +-------+        |
 | | App A |      | App B |      | App C | |    |        | App D |        |
 | +-------+      +--^----+      +-------+ |    |        +---^---+        |
 |       |           |               |     |    |            |            |
 |   (1) |      (1') |           (2) |     |    |       (2') |            |
 |       |           |               |     |    |            |            |
 +-------|-----------|---------------|-----+    +------------|------------+
         |           |               |                       |
 Kernel  |           |               |                       |
 +-------|-----------|---------------|-----------------------|------------+
 | +-----v-+      +-------+      +---v---+               +-------+        |
 | | snd A |-+    | RMB B |<--+  | snd C |-+          +->| RMB D |        |
 | +-------+ |    +-------+   |  +-------+ |          |  +-------+        |
 | +-------+ |    +-------+   |  +-------+ |          |  +-------+        |
 | | RMB A | |    | snd B |   |  | RMB C | |          |  | snd D |        |
 | +-------+ |    +-------+   |  +-------+ |          |  +-------+        |
 |           |               +-------------v+         |                   |
 |           +-------------->| smc loopback |---------+                   |
 +---------------------------+--------------+-----------------------------+

# Benchmark Test

* Test environments:
- VM with Intel Xeon Platinum 8 core 2.50GHz, 16 GiB mem.
- SMC sndbuf/RMB size 1MB.

* Test object:
- TCP: run on TCP loopback.
- domain: run on UNIX domain.
- SMC lo: run on SMC loopback device.

1. ipc-benchmark (see [1])

- ./<foo> -c 1000000 -s 100

TCP SMC-lo
Message
rate (msg/s) 81539 151251(+85.50%)

2. sockperf

- serv: <smc_run> taskset -c <cpu> sockperf sr --tcp
- clnt: <smc_run> taskset -c <cpu> sockperf { tp | pp } --tcp --msg-size={ 64000 for tp | 14 for pp } -i 127.0.0.1 -t 30

TCP SMC-lo
Bandwidth(MBps) 5313.66 8270.51(+55.65%)
Latency(us) 5.806 3.207(-44.76%)

3. nginx/wrk

- serv: <smc_run> nginx
- clnt: <smc_run> wrk -t 8 -c 1000 -d 30 http://127.0.0.1:80

TCP SMC-lo
Requests/s 194641.79 258656.13(+32.89%)

4. redis-benchmark

Quoted 9 lines

- serv: <smc_run> redis-server
- clnt: <smc_run> redis-benchmark -h 127.0.0.1 -q -t set,get -n 400000 -c 200 -d 1024

TCP SMC-lo

GET(Requests/s)       85855.34                115640.35(+34.69%)
SET(Requests/s)       86337.15                118203.30(+36.90%)

[1] https://github.com/goldsborough/ipc-bench

Hi Wen Gu,

I've been trying out your series with iperf3, qperf, and uperf on s390x. I'm using network namespaces with a ConnectX VF from the same card in each namespace for the initial TCP/IP connection i.e. initially it goes out to a real NIC even if that can switch internally. All of these look great for streaming workloads both in terms of performance and stability. With a Connect-Request-Response workload and uperf however I've run into issues. The test configuration I use is as follows:

Client Command:

# host=$ip_server ip netns exec client smc_run uperf -m tcp_crr.xml

Server Command:

# ip netns exec server smc_run uperf -s &> /dev/null

Uperf tcp_crr.xml:

<?xml version="1.0"?>
<profile name="TCP_CRR">

        <group nthreads="12">
                <transaction duration="120">
                        <flowop type="connect" options="remotehost=$host protocol=tcp" />
                        <flowop type="write" options="size=200"/>
                        <flowop type="read" options="size=1000"/>
                        <flowop type="disconnect" />
                </transaction>
        </group>

</profile>

The workload first runs fine but then after about 4 GB of data transferred fails with "Connection refused" and "Connection reset by peer" errors. The failure is not permanent however and re-running the streaming workloads run fine again (with both uperf server and client restarted). So I suspect something gets stuck in either the client or server sockets. The same workload runs fine with TCP/IP of course.

Thanks,
Niklas

author reviewer
#47 lore raw
Quoted 42 lines · Niklas Schnelle

Hi Wen Gu,

I've been trying out your series with iperf3, qperf, and uperf on s390x. I'm using network namespaces with a ConnectX VF from the same card in each namespace for the initial TCP/IP connection i.e. initially it goes out to a real NIC even if that can switch internally. All of these look great for streaming workloads both in terms of performance and stability. With a Connect-Request-Response workload and uperf however I've run into issues. The test configuration I use is as follows:

Client Command:

# host=$ip_server ip netns exec client smc_run uperf -m tcp_crr.xml

Server Command:

# ip netns exec server smc_run uperf -s &> /dev/null

Uperf tcp_crr.xml:

<?xml version="1.0"?>
<profile name="TCP_CRR">

         <group nthreads="12">
                 <transaction duration="120">
                         <flowop type="connect" options="remotehost=$host protocol=tcp" />
                         <flowop type="write" options="size=200"/>
                         <flowop type="read" options="size=1000"/>
                         <flowop type="disconnect" />
                 </transaction>
         </group>

</profile>

The workload first runs fine but then after about 4 GB of data transferred fails with "Connection refused" and "Connection reset by peer" errors. The failure is not permanent however and re-running the streaming workloads run fine again (with both uperf server and client restarted). So I suspect something gets stuck in either the client or server sockets. The same workload runs fine with TCP/IP of course.

Quoted 4 lines

Thanks,
Niklas

Hi Niklas,

Thank you very much for the test. With the test example you provided, I've reproduced the issue in my VM. And moreover, sometimes the test complains with 'Error saying goodbye with <ip>'

I'll figure out what's going on here.

Thanks!
Wen Gu

author reviewer
#48 lore raw
Quoted 53 lines · Wen Gu
Quoted 42 lines · Niklas Schnelle

Hi Wen Gu,

I've been trying out your series with iperf3, qperf, and uperf on s390x. I'm using network namespaces with a ConnectX VF from the same card in each namespace for the initial TCP/IP connection i.e. initially it goes out to a real NIC even if that can switch internally. All of these look great for streaming workloads both in terms of performance and stability. With a Connect-Request-Response workload and uperf however I've run into issues. The test configuration I use is as follows:

Client Command:

# host=$ip_server ip netns exec client smc_run uperf -m tcp_crr.xml

Server Command:

# ip netns exec server smc_run uperf -s &> /dev/null

Uperf tcp_crr.xml:

<?xml version="1.0"?>
<profile name="TCP_CRR">
         <group nthreads="12">
                 <transaction duration="120">
                         <flowop type="connect" options="remotehost=$host protocol=tcp" /> <flowop type="write" options="size=200"/>
                         <flowop type="read" options="size=1000"/>
                         <flowop type="disconnect" />
                 </transaction>
         </group>
</profile>

The workload first runs fine but then after about 4 GB of data transferred fails with "Connection refused" and "Connection reset by peer" errors. The failure is not permanent however and re-running the streaming workloads run fine again (with both uperf server and client restarted). So I suspect something gets stuck in either the client or server sockets. The same workload runs fine with TCP/IP of course.

Quoted 4 lines

Thanks,
Niklas

Hi Niklas,

Quoted 8 lines

Thank you very much for the test. With the test example you provided, I've reproduced the issue in my VM. And moreover, sometimes the test complains with 'Error saying goodbye with <ip>'

I'll figure out what's going on here.

Thanks!
Wen Gu

I think that there is a common issue for SMC-R and SMC-D. I also reproduce 'connection reset by peer' and 'Error saying goodbye with <ip>' when using SMC-R under the same test condition. They occur at the end of the test.

When the uperf test time ends, some signals are sent. At this point there are usually some SMC connections doing CLC handshake. I catch some -EINTR(-4) in client and -ECONNRESET(-104) in server returned from smc_clc_wait_msg, (correspondingly handshake error counts also increase) and TCP RST packets sent to terminate the CLC TCP connection(clcsock).

I am not sure if this should be considered as a bydesign or a bug of SMC. From an application perspective, the conn reset behavior only happens when using SMC.

@Wenjia, could you please take a look at this?

Thanks,
Wen Gu

#49 lore raw
Quoted 76 lines · Wen Gu
Quoted 53 lines · Wen Gu
Quoted 42 lines · Niklas Schnelle

Hi Wen Gu,

I've been trying out your series with iperf3, qperf, and uperf on s390x. I'm using network namespaces with a ConnectX VF from the same card in each namespace for the initial TCP/IP connection i.e. initially it goes out to a real NIC even if that can switch internally. All of these look great for streaming workloads both in terms of performance and stability. With a Connect-Request-Response workload and uperf however I've run into issues. The test configuration I use is as follows:

Client Command:

# host=$ip_server ip netns exec client smc_run uperf -m tcp_crr.xml

Server Command:

# ip netns exec server smc_run uperf -s &> /dev/null

Uperf tcp_crr.xml:

<?xml version="1.0"?>
<profile name="TCP_CRR">
         <group nthreads="12">
                 <transaction duration="120">
                         <flowop type="connect" options="remotehost=$host protocol=tcp" /> <flowop type="write" options="size=200"/>
                         <flowop type="read" options="size=1000"/>
                         <flowop type="disconnect" />
                 </transaction>
         </group>
</profile>

The workload first runs fine but then after about 4 GB of data transferred fails with "Connection refused" and "Connection reset by peer" errors. The failure is not permanent however and re-running the streaming workloads run fine again (with both uperf server and client restarted). So I suspect something gets stuck in either the client or server sockets. The same workload runs fine with TCP/IP of course.

Quoted 4 lines

Thanks,
Niklas

Hi Niklas,

Quoted 8 lines

Thank you very much for the test. With the test example you provided, I've reproduced the issue in my VM. And moreover, sometimes the test complains with 'Error saying goodbye with <ip>'

I'll figure out what's going on here.

Thanks!
Wen Gu

I think that there is a common issue for SMC-R and SMC-D. I also reproduce 'connection reset by peer' and 'Error saying goodbye with <ip>' when using SMC-R under the same test condition. They occur at the end of the test.

When the uperf test time ends, some signals are sent. At this point there are usually some SMC connections doing CLC handshake. I catch some -EINTR(-4)
in client and -ECONNRESET(-104) in server returned from smc_clc_wait_msg, (correspondingly handshake error counts also increase) and TCP RST packets sent to terminate the CLC TCP connection(clcsock).

Quoted 8 lines

I am not sure if this should be considered as a bydesign or a bug of SMC.
From an application perspective, the conn reset behavior only happens when
using SMC.

@Wenjia, could you please take a look at this?

Thanks,
Wen Gu

Hi Wen,

Do you mean the bug in smc_clc_wait_msg()?
If yes, I can not see any problem in the smc_clc_wait_msg(). From your description, it looks to me like the server should get the CLC_PROPOSAL message, but nothing in it while the client is waiting for the accept CLC_ACCEPT message from the server until the wait loops is broken out.

Thanks,
Wenjia