net/smc: introduce ringbufs usage statistics
Wen Gu
reviewerv3 cover net/smc: introduce ringbufs usage statistics
Currently, we have histograms that show the sizes of ringbufs that ever used by SMC connections. However, they are always incremental and since SMC allows the reuse of ringbufs, we cannot know the actual amount of ringbufs being allocated or actively used.
So this patch set introduces statistics for the amount of ringbufs that actually allocated by link group and actively used by connections of a certain net namespace, so that we can react based on these memory usage information, e.g. active fallback to TCP.
With appropriate adaptations of smc-tools, we can obtain these ringbufs usage information:
$ smcr -d linkgroup LG-ID : 00000500 LG-Role : SERV LG-Type : ASYML VLAN : 0 PNET-ID : Version : 1 Conns : 0 Sndbuf : 12910592 B <- RMB : 12910592 B <-
or
$ smcr -d stats [...] RX Stats Data transmitted (Bytes) 869225943 (869.2M) Total requests 18494479 Buffer usage (Bytes) 12910592 (12.31M) <- [...]
TX Stats
Data transmitted (Bytes) 12760884405 (12.76G)
Total requests 36988338
Buffer usage (Bytes) 12910592 (12.31M) <-
[...]
[...]
Change log:v3->v2
- use new helper nla_put_uint() instead of nla_put_u64_64bit().
v2->v1
https://lore.kernel.org/r/[email protected]/
- remove inline keyword in .c files.
- use local variable in macros to avoid potential side effects.
v1
https://lore.kernel.org/r/[email protected]/
Wen Gu (2):
net/smc: introduce statistics for allocated ringbufs of link group net/smc: introduce statistics for ringbufs usage of net namespace
include/uapi/linux/smc.h | 6 ++++ net/smc/smc_core.c | 68 +++++++++++++++++++++++++++++++++------- net/smc/smc_core.h | 2 ++ net/smc/smc_stats.c | 6 ++++ net/smc/smc_stats.h | 28 +++++++++++------ 5 files changed, 90 insertions(+), 20 deletions(-)
Signature · 1 line
2.32.0.3.g01195cf9f
Wen Gu
reviewerPATCH v3 1/2 net/smc: introduce statistics for allocated ringbufs of link group
Currently we have the statistics on sndbuf/RMB sizes of all connections that have ever been on the link group, namely smc_stats_memsize. However these statistics are incremental and since the ringbufs of link group are allowed to be reused, we cannot know the actual allocated buffers through these. So here introduces the statistic on actual allocated ringbufs of the link group, it will be incremented when a new ringbuf is added into buf_list and decremented when it is deleted from buf_list.
--- include/uapi/linux/smc.h | 4 ++++ net/smc/smc_core.c | 46 ++++++++++++++++++++++++++++++++++++---- net/smc/smc_core.h | 2 ++ 3 files changed, 48 insertions(+), 4 deletions(-)
include/uapi/linux/smc.h +4 -0
@@ -127,6 +127,8 @@ enum {SMC_NLA_LGR_R_NET_COOKIE ,/* u64 */ SMC_NLA_LGR_R_PAD ,/* flag */ SMC_NLA_LGR_R_BUF_TYPE ,/* u8 */ +SMC_NLA_LGR_R_SNDBUF_ALLOC ,/* uint */ +SMC_NLA_LGR_R_RMB_ALLOC ,/* uint */ __SMC_NLA_LGR_R_MAX, SMC_NLA_LGR_R_MAX = __SMC_NLA_LGR_R_MAX - 1 };
@@ -162,6 +164,8 @@ enum {SMC_NLA_LGR_D_V2_COMMON ,/* nest */ SMC_NLA_LGR_D_EXT_GID ,/* u64 */ SMC_NLA_LGR_D_PEER_EXT_GID ,/* u64 */ +SMC_NLA_LGR_D_SNDBUF_ALLOC ,/* uint */ +SMC_NLA_LGR_D_DMB_ALLOC ,/* uint */ __SMC_NLA_LGR_D_MAX, SMC_NLA_LGR_D_MAX = __SMC_NLA_LGR_D_MAX - 1 };
net/smc/smc_core.c +42 -4
88 lines not on this page · raw message
net/smc/smc_core.h +2 -0
@@ -281,6 +281,8 @@ struct smc_link_group {struct rw_semaphore sndbufs_lock;/* protects tx buffers */ struct list_head rmbs[SMC_RMBE_SIZES ];/* rx buffers */ struct rw_semaphore rmbs_lock;/* protects rx buffers */ +u64 alloc_sndbufs;/* stats of tx buffers */ +u64 alloc_rmbs;/* stats of rx buffers */ u8 id[SMC_LGR_ID_SIZE ];/* unique lgr id */ struct delayed_work free_work;/* delayed freeing of an lgr */
Signature · 1 line
2.32.0.3.g01195cf9f
Wen Gu
reviewerPATCH v3 2/2 net/smc: introduce statistics for ringbufs usage of net namespace
In reply to Wen GuThe buffer size histograms in smc_stats, namely rx/tx_rmbsize, record the sizes of ringbufs for all connections that have ever appeared in the net namespace. They are incremental and we cannot know the actual ringbufs usage from these. So here introduces statistics for current ringbufs usage of existing smc connections in the net namespace into smc_stats, it will be incremented when new connection uses a ringbuf and decremented when the ringbuf is unused.
--- include/uapi/linux/smc.h | 2 ++ net/smc/smc_core.c | 22 +++++++++++++++------- net/smc/smc_stats.c | 6 ++++++ net/smc/smc_stats.h | 28 +++++++++++++++++++--------- 4 files changed, 42 insertions(+), 16 deletions(-)
include/uapi/linux/smc.h +2 -0
@@ -253,6 +253,8 @@ enum {SMC_NLA_STATS_T_TX_BYTES ,/* u64 */ SMC_NLA_STATS_T_RX_CNT ,/* u64 */ SMC_NLA_STATS_T_TX_CNT ,/* u64 */ +SMC_NLA_STATS_T_RX_RMB_USAGE ,/* uint */ +SMC_NLA_STATS_T_TX_RMB_USAGE ,/* uint */ __SMC_NLA_STATS_T_MAX, SMC_NLA_STATS_T_MAX = __SMC_NLA_STATS_T_MAX - 1 };
net/smc/smc_core.c +15 -7
@@ -1203,22 +1203,30 @@ static void smcd_buf_detach(struct smc_connection *conn)static void smc_buf_unuse (struct smc_connection *conn,struct smc_link_group *lgr) { + struct smc_sock *smc = container_of(conn, struct smc_sock, conn); +bool is_smcd = lgr->is_smcd; +int bufsize; +if (conn->sndbuf_desc) { -if (!lgr->is_smcd && conn->sndbuf_desc->is_vm) { + bufsize = conn->sndbuf_desc->len; +if (!is_smcd && conn->sndbuf_desc->is_vm) {smcr_buf_unuse (conn->sndbuf_desc, false, lgr); }else { -memzero_explicit (conn->sndbuf_desc->cpu_addr, conn->sndbuf_desc->len); +memzero_explicit (conn->sndbuf_desc->cpu_addr, bufsize);WRITE_ONCE (conn->sndbuf_desc->used,0 ); } +SMC_STAT_RMB_SIZE (smc, is_smcd, false, false, bufsize); }if (conn->rmb_desc) { -if (!lgr->is_smcd) { + bufsize = conn->rmb_desc->len; +if (!is_smcd) {smcr_buf_unuse (conn->rmb_desc, true, lgr); }else { -memzero_explicit (conn->rmb_desc->cpu_addr, - conn->rmb_desc->len +sizeof (struct smcd_cdc_msg )); + bufsize +=sizeof (struct smcd_cdc_msg ); +memzero_explicit (conn->rmb_desc->cpu_addr, bufsize);WRITE_ONCE (conn->rmb_desc->used,0 ); } +SMC_STAT_RMB_SIZE (smc, is_smcd, true, false, bufsize); } }
@@ -2427,7 +2435,7 @@ static int __smc_buf_create(struct smc_sock *smc, bool is_smcd, bool is_rmb) buf_desc =smc_buf_get_slot (bufsize_comp, lock, buf_list);if (buf_desc) { buf_desc->is_dma_need_sync =0 ; -SMC_STAT_RMB_SIZE (smc, is_smcd, is_rmb, bufsize); +SMC_STAT_RMB_SIZE (smc, is_smcd, is_rmb, true, bufsize);SMC_STAT_BUF_REUSE (smc, is_smcd, is_rmb);break ;/* found reusable slot */ }
@@ -2448,7 +2456,7 @@ static int __smc_buf_create(struct smc_sock *smc, bool is_smcd, bool is_rmb) }SMC_STAT_RMB_ALLOC (smc, is_smcd, is_rmb); -SMC_STAT_RMB_SIZE (smc, is_smcd, is_rmb, bufsize); +SMC_STAT_RMB_SIZE (smc, is_smcd, is_rmb, true, bufsize); buf_desc->used =1 ;down_write (lock);smc_lgr_buf_list_add (lgr, is_rmb, buf_list, buf_desc);
net/smc/smc_stats.c +6 -0
@@ -218,6 +218,12 @@ static int smc_nl_fill_stats_tech_data(struct sk_buff *skb, smc_tech->tx_bytes, SMC_NLA_STATS_PAD)) goto errattr; +if (nla_put_uint (skb,SMC_NLA_STATS_T_RX_RMB_USAGE , + smc_tech->rx_rmbuse)) + goto errattr; +if (nla_put_uint (skb,SMC_NLA_STATS_T_TX_RMB_USAGE , + smc_tech->tx_rmbuse)) + goto errattr;if (nla_put_u64_64bit (skb,SMC_NLA_STATS_T_RX_CNT , smc_tech->rx_cnt,SMC_NLA_STATS_PAD ))
net/smc/smc_stats.h +19 -9
@@ -79,6 +79,8 @@ struct smc_stats_tech {u64 tx_bytes;u64 rx_cnt;u64 tx_cnt; +u64 rx_rmbuse; +u64 tx_rmbuse; };struct smc_stats {
@@ -135,38 +137,46 @@ do { \ } \ while (0) -#define SMC_STAT_RMB_SIZE_SUB (_smc_stats, _tech, k, _len) \ +#define SMC_STAT_RMB_SIZE_SUB (_smc_stats, _tech, k, _is_add, _len) \ do { \ + typeof(_smc_stats) stats = (_smc_stats); \ + typeof(_is_add) is_a = (_is_add); \ typeof(_len) _l = (_len); \ typeof(_tech) t = (_tech); \ int _pos; \ int m = SMC_BUF_MAX - 1; \ if (_l <= 0) \ break; \ - _pos = fls((_l - 1) >> 13); \ - _pos = (_pos <= m) ? _pos : m; \ - this_cpu_inc((*(_smc_stats)).smc[t].k ## _rmbsize.buf[_pos]); \ + if (is_a) { \ + _pos = fls((_l - 1) >> 13); \ + _pos = (_pos <= m) ? _pos : m; \ + this_cpu_inc((*stats).smc[t].k ## _rmbsize.buf[_pos]); \ + this_cpu_add((*stats).smc[t].k ## _rmbuse, _l); \ + } else { \ + this_cpu_sub((*stats).smc[t].k ## _rmbuse, _l); \ + } \ } \ while (0)#define SMC_STAT_RMB_SUB (_smc_stats, type, t, key) \ this_cpu_inc((*(_smc_stats)).smc[t].rmb ## _ ## key.type ## _cnt) -#define SMC_STAT_RMB_SIZE (_smc, _is_smcd, _is_rx, _len) \ +#define SMC_STAT_RMB_SIZE (_smc, _is_smcd, _is_rx, _is_add, _len) \ do { \ struct net *_net = sock_net(&(_smc)->sk); \ struct smc_stats __percpu *_smc_stats = _net->smc.smc_stats; \ + typeof(_is_add) is_add = (_is_add); \ typeof(_is_smcd) is_d = (_is_smcd); \ typeof(_is_rx) is_r = (_is_rx); \ typeof(_len) l = (_len); \ if ((is_d) && (is_r)) \ - SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_D, rx, l); \ + SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_D, rx, is_add, l); \ if ((is_d) && !(is_r)) \ - SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_D, tx, l); \ + SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_D, tx, is_add, l); \ if (!(is_d) && (is_r)) \ - SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_R, rx, l); \ + SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_R, rx, is_add, l); \ if (!(is_d) && !(is_r)) \ - SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_R, tx, l); \ + SMC_STAT_RMB_SIZE_SUB(_smc_stats, SMC_TYPE_R, tx, is_add, l); \ } \ while (0)
Signature · 1 line
2.32.0.3.g01195cf9f
Wen Gu
reviewerQuoted 34 lines · Wen Gu
Currently, we have histograms that show the sizes of ringbufs that ever used by SMC connections. However, they are always incremental and since SMC allows the reuse of ringbufs, we cannot know the actual amount of ringbufs being allocated or actively used.
So this patch set introduces statistics for the amount of ringbufs that actually allocated by link group and actively used by connections of a certain net namespace, so that we can react based on these memory usage information, e.g. active fallback to TCP.
With appropriate adaptations of smc-tools, we can obtain these ringbufs usage information:
$ smcr -d linkgroup LG-ID : 00000500 LG-Role : SERV LG-Type : ASYML VLAN : 0 PNET-ID : Version : 1 Conns : 0 Sndbuf : 12910592 B <- RMB : 12910592 B <-or
$ smcr -d stats [...] RX Stats Data transmitted (Bytes) 869225943 (869.2M) Total requests 18494479 Buffer usage (Bytes) 12910592 (12.31M) <- [...]
Quoted 7 lines
TX Stats
Data transmitted (Bytes) 12760884405 (12.76G)
Total requests 36988338
Buffer usage (Bytes) 12910592 (12.31M) <-
[...]
[...]
FYI, The corresponding smc-tools modification has been submitted.
https://github.com/ibm-s390-linux/smc-tools/pull/11
Thanks!
This series was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Quoted 11 lines · On Wed, 14 Aug 2024 21:08:25 +0800 you wrote:
Currently, we have histograms that show the sizes of ringbufs that ever used by SMC connections. However, they are always incremental and since SMC allows the reuse of ringbufs, we cannot know the actual amount of ringbufs being allocated or actively used.
So this patch set introduces statistics for the amount of ringbufs that actually allocated by link group and actively used by connections of a certain net namespace, so that we can react based on these memory usage information, e.g. active fallback to TCP.
[...]
Here is the summary with links:
- [net-next,v3,1/2] net/smc: introduce statistics for allocated ringbufs of link group
https://git.kernel.org/netdev/net-next/c/d386d59b7c1a - [net-next,v3,2/2] net/smc: introduce statistics for ringbufs usage of net namespace https://git.kernel.org/netdev/net-next/c/e0d103542b06
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html