Skip to contentSkip to conversation

net/smc: introduce ringbufs usage statistics

PATCH Superseded Wen Gu 3 participants 12 messages Aug 7, 2024 – Aug 15, 2024

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Wen Gu

author reviewer
#0 lore raw

v2 cover net/smc: introduce ringbufs usage statistics

Currently, we have histograms that show the sizes of ringbufs that ever used by SMC connections. However, they are always incremental and since SMC allows the reuse of ringbufs, we cannot know the actual amount of ringbufs being allocated or actively used.

So this patch set introduces statistics for the amount of ringbufs that actually allocated by link group and actively used by connections of a certain net namespace, so that we can react based on these memory usage information, e.g. active fallback to TCP.

With appropriate adaptations of smc-tools, we can obtain these ringbufs usage information:

$ smcr -d linkgroup
LG-ID    : 00000500
LG-Role  : SERV
LG-Type  : ASYML
VLAN     : 0
PNET-ID  :
Version  : 1
Conns    : 0
Sndbuf   : 12910592 B    <-
RMB      : 12910592 B    <-

or

$ smcr -d stats
[...]
RX Stats
  Data transmitted (Bytes)      869225943 (869.2M)
  Total requests                 18494479
  Buffer usage  (Bytes)          12910592 (12.31M)  <-
  [...]

TX Stats
Data transmitted (Bytes) 12760884405 (12.76G)
Total requests 36988338
Buffer usage (Bytes) 12910592 (12.31M) <-
[...]
[...]

Wen Gu (2):
net/smc: introduce statistics for allocated ringbufs of link group net/smc: introduce statistics for ringbufs usage of net namespace

 include/uapi/linux/smc.h |  6 ++++
 net/smc/smc_core.c       | 72 ++++++++++++++++++++++++++++++++++------
 net/smc/smc_core.h       |  2 ++
 net/smc/smc_stats.c      |  8 +++++
 net/smc/smc_stats.h      | 28 +++++++++++-----
 5 files changed, 96 insertions(+), 20 deletions(-)
Signature · 1 line
2.32.0.3.g01195cf9f
author reviewer
#1 lore raw

PATCH v2 1/2 net/smc: introduce statistics for allocated ringbufs of link group

Currently we have the statistics on sndbuf/RMB sizes of all connections that have ever been on the link group, namely smc_stats_memsize. However these statistics are incremental and since the ringbufs of link group are allowed to be reused, we cannot know the actual allocated buffers through these. So here introduces the statistic on actual allocated ringbufs of the link group, it will be incremented when a new ringbuf is added into buf_list and decremented when it is deleted from buf_list.

---
 include/uapi/linux/smc.h |  4 ++++
 net/smc/smc_core.c       | 50 ++++++++++++++++++++++++++++++++++++----
 net/smc/smc_core.h       |  2 ++
 3 files changed, 52 insertions(+), 4 deletions(-)
include/uapi/linux/smc.h +4 -0
index b531e3ef011a..d27b8dc50f90 100644
@@ -127,6 +127,8 @@ enum {
 	SMC_NLA_LGR_R_NET_COOKIE,	/* u64 */
 	SMC_NLA_LGR_R_PAD,		/* flag */
 	SMC_NLA_LGR_R_BUF_TYPE,		/* u8 */
+	SMC_NLA_LGR_R_SNDBUF_ALLOC,	/* u64 */
+	SMC_NLA_LGR_R_RMB_ALLOC,	/* u64 */
 	__SMC_NLA_LGR_R_MAX,
 	SMC_NLA_LGR_R_MAX = __SMC_NLA_LGR_R_MAX - 1
 };
@@ -162,6 +164,8 @@ enum {
 	SMC_NLA_LGR_D_V2_COMMON,	/* nest */
 	SMC_NLA_LGR_D_EXT_GID,		/* u64 */
 	SMC_NLA_LGR_D_PEER_EXT_GID,	/* u64 */
+	SMC_NLA_LGR_D_SNDBUF_ALLOC,	/* u64 */
+	SMC_NLA_LGR_D_DMB_ALLOC,	/* u64 */
 	__SMC_NLA_LGR_D_MAX,
 	SMC_NLA_LGR_D_MAX = __SMC_NLA_LGR_D_MAX - 1
 };
net/smc/smc_core.c +46 -4

92 lines not on this page · raw message

net/smc/smc_core.h +2 -0
index d93cf51dbd7c..0db4e5f79ac4 100644
@@ -281,6 +281,8 @@ struct smc_link_group {
 	struct rw_semaphore	sndbufs_lock;	/* protects tx buffers */
 	struct list_head	rmbs[SMC_RMBE_SIZES];	/* rx buffers */
 	struct rw_semaphore	rmbs_lock;	/* protects rx buffers */
+	u64			alloc_sndbufs;	/* stats of tx buffers */
+	u64			alloc_rmbs;	/* stats of rx buffers */
 
 	u8			id[SMC_LGR_ID_SIZE];	/* unique lgr id */
 	struct delayed_work	free_work;	/* delayed freeing of an lgr */
Signature · 1 line
2.32.0.3.g01195cf9f
maintainer
#2 lore raw

Re: [PATCH net-next v2 1/2] net/smc: introduce statistics for allocated ringbufs of link group

net/smc/smc_core.c static int smc_nl_fill_lgr(struct smc_link_group *lgr, · line 396

Quoted 2 lines · On Wed, 7 Aug 2024 15:59:38 +0800 Wen Gu wrote:
net/smc/smc_core.c +2 -0
+	if (nla_put_u64_64bit(skb, SMC_NLA_LGR_R_SNDBUF_ALLOC,
+			      lgr->alloc_sndbufs, SMC_NLA_LGR_R_PAD))

nla_put_uint()

Signature · 1 line
pw-bot: cr
author reviewer
#3 lore raw

Re: [PATCH net-next v2 1/2] net/smc: introduce statistics for allocated ringbufs of link group

Quoted 5 lines · Jakub Kicinski

net/smc/smc_core.c static int smc_nl_fill_lgr(struct smc_link_group *lgr, · line 396

Quoted 2 lines · On Wed, 7 Aug 2024 15:59:38 +0800 Wen Gu wrote:
net/smc/smc_core.c +2 -0
+	if (nla_put_u64_64bit(skb, SMC_NLA_LGR_R_SNDBUF_ALLOC,
+			      lgr->alloc_sndbufs, SMC_NLA_LGR_R_PAD))

nla_put_uint()

Hi, Jakub. Thank you for reminder.

I read the commit log and learned the advantages of this helper. But it seems that the support for corresponding user-space helpers hasn't kept up yet, e.g. can't find a helper like nla_get_uint in latest libnl.

maintainer
#4 lore raw

Re: [PATCH net-next v2 1/2] net/smc: introduce statistics for allocated ringbufs of link group

Quoted 7 lines · On Tue, 13 Aug 2024 17:55:17 +0800 Wen Gu wrote:
Quoted 5 lines · Jakub Kicinski

net/smc/smc_core.c static int smc_nl_fill_lgr(struct smc_link_group *lgr, · line 396

Quoted 2 lines · On Wed, 7 Aug 2024 15:59:38 +0800 Wen Gu wrote:
net/smc/smc_core.c +2 -0
+	if (nla_put_u64_64bit(skb, SMC_NLA_LGR_R_SNDBUF_ALLOC,
+			      lgr->alloc_sndbufs, SMC_NLA_LGR_R_PAD))  

nla_put_uint()

Quoted 6 lines

Hi, Jakub. Thank you for reminder.

I read the commit log and learned the advantages of this helper. But it seems that the support for corresponding user-space helpers hasn't kept up yet, e.g. can't find a helper like nla_get_uint in latest libnl.

Add it, then.

#5 lore raw

Re: [PATCH net-next v2 1/2] net/smc: introduce statistics for allocated ringbufs of link group

Quoted 8 lines · Jakub Kicinski
Quoted 7 lines · On Tue, 13 Aug 2024 17:55:17 +0800 Wen Gu wrote:
Quoted 5 lines · Jakub Kicinski

net/smc/smc_core.c static int smc_nl_fill_lgr(struct smc_link_group *lgr, · line 396

Quoted 2 lines · On Wed, 7 Aug 2024 15:59:38 +0800 Wen Gu wrote:
net/smc/smc_core.c +2 -0
+	if (nla_put_u64_64bit(skb, SMC_NLA_LGR_R_SNDBUF_ALLOC,
+			      lgr->alloc_sndbufs, SMC_NLA_LGR_R_PAD))  

nla_put_uint()

Quoted 8 lines
Quoted 6 lines

Hi, Jakub. Thank you for reminder.

I read the commit log and learned the advantages of this helper. But it seems that the support for corresponding user-space helpers hasn't kept up yet, e.g. can't find a helper like nla_get_uint in latest libnl.

Add it, then.

Danielle added one to libmnl:

https://git.netfilter.org/libmnl/commit/?id=102942be401a99943b2c68981b238dadfa788f2d

Intention is to use it in ethtool once it appears in a released version of libmnl.

author reviewer
#6 lore raw

Re: [PATCH net-next v2 1/2] net/smc: introduce statistics for allocated ringbufs of link group

Quoted 16 lines · Ido Schimmel
Quoted 15 lines · Jakub Kicinski
Quoted 13 lines · On Tue, 13 Aug 2024 17:55:17 +0800 Wen Gu wrote:
Quoted 5 lines · Jakub Kicinski

net/smc/smc_core.c static int smc_nl_fill_lgr(struct smc_link_group *lgr, · line 396

Quoted 2 lines · On Wed, 7 Aug 2024 15:59:38 +0800 Wen Gu wrote:
net/smc/smc_core.c +2 -0
+	if (nla_put_u64_64bit(skb, SMC_NLA_LGR_R_SNDBUF_ALLOC,
+			      lgr->alloc_sndbufs, SMC_NLA_LGR_R_PAD))

nla_put_uint()

Hi, Jakub. Thank you for reminder.

I read the commit log and learned the advantages of this helper. But it seems that the support for corresponding user-space helpers hasn't kept up yet, e.g. can't find a helper like nla_get_uint in latest libnl.

Quoted 8 lines
Quoted 1 line

Add it, then.

Danielle added one to libmnl:

https://git.netfilter.org/libmnl/commit/?id=102942be401a99943b2c68981b238dadfa788f2d

Intention is to use it in ethtool once it appears in a released version of libmnl.

Thanks, that is a good example.

maintainer
#7 lore raw

Re: [PATCH net-next v2 1/2] net/smc: introduce statistics for allocated ringbufs of link group

Quoted 8 lines · On Wed, 14 Aug 2024 11:12:41 +0800 Wen Gu wrote:
Quoted 6 lines

Danielle added one to libmnl:

https://git.netfilter.org/libmnl/commit/?id=102942be401a99943b2c68981b238dadfa788f2d

Intention is to use it in ethtool once it appears in a released version of libmnl.

Thanks, that is a good example.

FWIW - technically the kernel version of a uint / sint are only either 32b or 64b, because smaller types get padded to 4bytes in netlink messages, anyway. But doesn't hurt for the get. For the put make sure you don't narrow down too much.

author reviewer
#8 lore raw

Re: [PATCH net-next v2 1/2] net/smc: introduce statistics for allocated ringbufs of link group

Quoted 8 lines · Jakub Kicinski
Quoted 7 lines · On Wed, 14 Aug 2024 11:12:41 +0800 Wen Gu wrote:
Quoted 6 lines

Danielle added one to libmnl:

https://git.netfilter.org/libmnl/commit/?id=102942be401a99943b2c68981b238dadfa788f2d

Intention is to use it in ethtool once it appears in a released version of libmnl.

Quoted 6 lines
Quoted 1 line

Thanks, that is a good example.

FWIW - technically the kernel version of a uint / sint are only either 32b or 64b, because smaller types get padded to 4bytes in netlink messages, anyway. But doesn't hurt for the get. For the put make sure you don't narrow down too much.

Understand. Thanks.

author reviewer
#9 lore raw

Re: [PATCH net-next v2 1/2] net/smc: introduce statistics for allocated ringbufs of link group

In reply to Jakub Kicinski
Quoted 8 lines · Jakub Kicinski
Quoted 7 lines · On Tue, 13 Aug 2024 17:55:17 +0800 Wen Gu wrote:
Quoted 5 lines · Jakub Kicinski

net/smc/smc_core.c static int smc_nl_fill_lgr(struct smc_link_group *lgr, · line 396

Quoted 2 lines · On Wed, 7 Aug 2024 15:59:38 +0800 Wen Gu wrote:
net/smc/smc_core.c +2 -0
+	if (nla_put_u64_64bit(skb, SMC_NLA_LGR_R_SNDBUF_ALLOC,
+			      lgr->alloc_sndbufs, SMC_NLA_LGR_R_PAD))

nla_put_uint()

Quoted 8 lines
Quoted 6 lines

Hi, Jakub. Thank you for reminder.

I read the commit log and learned the advantages of this helper. But it seems that the support for corresponding user-space helpers hasn't kept up yet, e.g. can't find a helper like nla_get_uint in latest libnl.

Add it, then.

OK. So I guess we should use nla_put_uint for all 64bit cases from now on?

maintainer
#10 lore raw

Re: [PATCH net-next v2 1/2] net/smc: introduce statistics for allocated ringbufs of link group

Quoted 10 lines · On Wed, 14 Aug 2024 11:10:33 +0800 Wen Gu wrote:
Quoted 8 lines
Quoted 6 lines

Hi, Jakub. Thank you for reminder.

I read the commit log and learned the advantages of this helper. But it seems that the support for corresponding user-space helpers hasn't kept up yet, e.g. can't find a helper like nla_get_uint in latest libnl.

Add it, then.

OK. So I guess we should use nla_put_uint for all 64bit cases from now on?

Yes, actually I'd go further and say all ints should use the variable size types, unless we are guaranteed they will never be larger than 32b (e.g. ifindex is used in many places in uAPI as 31b already, we can't grow it), or the field is explicitly fixed size (e.g. protocol header fields defined by standards).

author reviewer
#11 lore raw

PATCH v2 2/2 net/smc: introduce statistics for ringbufs usage of net namespace

In reply to Wen Gu

The buffer size histograms in smc_stats, namely rx/tx_rmbsize, record the sizes of ringbufs for all connections that have ever appeared in the net namespace. They are incremental and we cannot know the actual ringbufs usage from these. So here introduces statistics for current ringbufs usage of existing smc connections in the net namespace into smc_stats, it will be incremented when new connection uses a ringbuf and decremented when the ringbuf is unused.

---
 include/uapi/linux/smc.h |  2 ++
 net/smc/smc_core.c       | 22 +++++++++++++++-------
 net/smc/smc_stats.c      |  8 ++++++++
 net/smc/smc_stats.h      | 28 +++++++++++++++++++---------
 4 files changed, 44 insertions(+), 16 deletions(-)
include/uapi/linux/smc.h +2 -0
index d27b8dc50f90..304e202c03bb 100644
@@ -253,6 +253,8 @@ enum {
 	SMC_NLA_STATS_T_TX_BYTES,	/* u64 */
 	SMC_NLA_STATS_T_RX_CNT,		/* u64 */
 	SMC_NLA_STATS_T_TX_CNT,		/* u64 */
+	SMC_NLA_STATS_T_RX_RMB_USAGE,	/* u64 */
+	SMC_NLA_STATS_T_TX_RMB_USAGE,	/* u64 */
 	__SMC_NLA_STATS_T_MAX,
 	SMC_NLA_STATS_T_MAX = __SMC_NLA_STATS_T_MAX - 1
 };
net/smc/smc_core.c +15 -7
index b1ff44d72fe4..72c6c7363f93 100644
@@ -1207,22 +1207,30 @@ static void smcd_buf_detach(struct smc_connection *conn)
 static void smc_buf_unuse(struct smc_connection *conn,
 			  struct smc_link_group *lgr)
 {
+	struct smc_sock *smc = container_of(conn, struct smc_sock, conn);
+	bool is_smcd = lgr->is_smcd;
+	int bufsize;
+
 	if (conn->sndbuf_desc) {
-		if (!lgr->is_smcd && conn->sndbuf_desc->is_vm) {
+		bufsize = conn->sndbuf_desc->len;
+		if (!is_smcd && conn->sndbuf_desc->is_vm) {
 			smcr_buf_unuse(conn->sndbuf_desc, false, lgr);
 		} else {
-			memzero_explicit(conn->sndbuf_desc->cpu_addr, conn->sndbuf_desc->len);
+			memzero_explicit(conn->sndbuf_desc->cpu_addr, bufsize);
 			WRITE_ONCE(conn->sndbuf_desc->used, 0);
 		}
+		SMC_STAT_RMB_SIZE(smc, is_smcd, false, false, bufsize);
 	}
 	if (conn->rmb_desc) {
-		if (!lgr->is_smcd) {
+		bufsize = conn->rmb_desc->len;
+		if (!is_smcd) {
 			smcr_buf_unuse(conn->rmb_desc, true, lgr);
 		} else {
-			memzero_explicit(conn->rmb_desc->cpu_addr,
-					 conn->rmb_desc->len + sizeof(struct smcd_cdc_msg));
+			bufsize += sizeof(struct smcd_cdc_msg);
+			memzero_explicit(conn->rmb_desc->cpu_addr, bufsize);
 			WRITE_ONCE(conn->rmb_desc->used, 0);
 		}
+		SMC_STAT_RMB_SIZE(smc, is_smcd, true, false, bufsize);
 	}
 }
 
@@ -2431,7 +2439,7 @@ static int __smc_buf_create(struct smc_sock *smc, bool is_smcd, bool is_rmb)
 		buf_desc = smc_buf_get_slot(bufsize_comp, lock, buf_list);
 		if (buf_desc) {
 			buf_desc->is_dma_need_sync = 0;
-			SMC_STAT_RMB_SIZE(smc, is_smcd, is_rmb, bufsize);
+			SMC_STAT_RMB_SIZE(smc, is_smcd, is_rmb, true, bufsize);
 			SMC_STAT_BUF_REUSE(smc, is_smcd, is_rmb);
 			break; /* found reusable slot */
 		}
@@ -2452,7 +2460,7 @@ static int __smc_buf_create(struct smc_sock *smc, bool is_smcd, bool is_rmb)
 		}
 
 		SMC_STAT_RMB_ALLOC(smc, is_smcd, is_rmb);
-		SMC_STAT_RMB_SIZE(smc, is_smcd, is_rmb, bufsize);
+		SMC_STAT_RMB_SIZE(smc, is_smcd, is_rmb, true, bufsize);
 		buf_desc->used = 1;
 		down_write(lock);
 		smc_lgr_buf_list_add(lgr, is_rmb, buf_list, buf_desc);
net/smc/smc_stats.c +8 -0
index ca14c0f3a07d..3f2ebc6c06ba 100644
@@ -218,6 +218,14 @@ static int smc_nl_fill_stats_tech_data(struct sk_buff *skb,
 			      smc_tech->tx_bytes,
 			      SMC_NLA_STATS_PAD))
 		goto errattr;
+	if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_RX_RMB_USAGE,
+			      smc_tech->rx_rmbuse,
+			      SMC_NLA_STATS_PAD))
+		goto errattr;
+	if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_TX_RMB_USAGE,
+			      smc_tech->tx_rmbuse,
+			      SMC_NLA_STATS_PAD))
+		goto errattr;
 	if (nla_put_u64_64bit(skb, SMC_NLA_STATS_T_RX_CNT,
 			      smc_tech->rx_cnt,
 			      SMC_NLA_STATS_PAD))
net/smc/smc_stats.h +19 -9
index 9d32058db2b5..6ac465380431 100644
@@ -79,6 +79,8 @@ struct smc_stats_tech {
 	u64			tx_bytes;
 	u64			rx_cnt;
 	u64			tx_cnt;
+	u64			rx_rmbuse;
+	u64			tx_rmbuse;
 };
 
 struct smc_stats {
@@ -135,38 +137,46 @@ do { \
 } \
 while (0)
 
-#define SMC_STAT_RMB_SIZE_SUB(_smc_stats, _tech, k, _len) \
+#define SMC_STAT_RMB_SIZE_SUB(_smc_stats, _tech, k, _is_add, _len) \
 do { \
+	typeof(_smc_stats) stats = (_smc_stats); \
+	typeof(_is_add) is_a = (_is_add); \
 	typeof(_len) _l = (_len); \
 	typeof(_tech) t = (_tech); \
 	int _pos; \
 	int m = SMC_BUF_MAX - 1; \
 	if (_l <= 0) \
 		break; \
-	_pos = fls((_l - 1) >> 13); \
-	_pos = (_pos <= m) ? _pos : m; \
-	this_cpu_inc((*(_smc_stats)).smc[t].k ## _rmbsize.buf[_pos]); \
+	if (is_a) { \
+		_pos = fls((_l - 1) >> 13); \
+		_pos = (_pos <= m) ? _pos : m; \
+		this_cpu_inc((*stats).smc[t].k ## _rmbsize.buf[_pos]); \
+		this_cpu_add((*stats).smc[t].k ## _rmbuse, _l); \
+	} else { \
+		this_cpu_sub((*stats).smc[t].k ## _rmbuse, _l); \
+	} \
 } \
 while (0)
 
 #define SMC_STAT_RMB_SUB(_smc_stats, type, t, key) \
 	this_cpu_inc((*(_smc_stats)).smc[t].rmb ## _ ## key.type ## _cnt)
 
-#define SMC_STAT_RMB_SIZE(_smc, _is_smcd, _is_rx, _len) \
+#define SMC_STAT_RMB_SIZE(_smc, _is_smcd, _is_rx, _is_add, _len) \
 do { \
 	struct net *_net = sock_net(&(_smc)->sk); \
 	struct smc_stats __percpu *_smc_stats = _net->smc.smc_stats; \
+	typeof(_is_add) is_add = (_is_add); \
 	typeof(_is_smcd) is_d = (_is_smcd); \
 	typeof(_is_rx) is_r = (_is_rx); \
 	typeof(_len) l = (_len); \
 	if ((is_d) && (is_r)) \
-		SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_D, rx, l); \
+		SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_D, rx, is_add, l); \
 	if ((is_d) && !(is_r)) \
-		SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_D, tx, l); \
+		SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_D, tx, is_add, l); \
 	if (!(is_d) && (is_r)) \
-		SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_R, rx, l); \
+		SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_R, rx, is_add, l); \
 	if (!(is_d) && !(is_r)) \
-		SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_R, tx, l); \
+		SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_R, tx, is_add, l); \
 } \
 while (0)
 
Signature · 1 line
2.32.0.3.g01195cf9f