Skip to contentSkip to conversation

net/smc: fix application data exception

PATCH In review D.Wythe 3 participants 5 messages Nov 26, 2022 – Feb 16, 2023

Subsystems (5)
lore
Share & tools
lore

D.Wythe

author maintainer
#0 lore raw

PATCH 1/1 net/smc: fix application data exception

From: "D. Wythe" <[email protected]>

There is a certain probability that following
exceptions will occur in the wrk benchmark test:

Running 10s test @ http://11.213.45.6:80
8 threads and 64 connections
Thread Stats Avg Stdev Max +/- Stdev

    Latency     3.72ms   13.94ms 245.33ms   94.17%
    Req/Sec     1.96k   713.67     5.41k    75.16%

155262 requests in 10.10s, 23.10MB read
Non-2xx or 3xx responses: 3

We will find that the error is HTTP 400 error, which is a serious exception in our test, which means the application data was corrupted.

Consider the following scenarios:

CPU0 CPU1

buf_desc->used = 0;
                                cmpxchg(buf_desc->used, 0, 1)
                                deal_with(buf_desc)

memset(buf_desc->cpu_addr,0);

This will cause the data received by a victim connection to be cleared, thus triggering an HTTP 400 error in the server.

This patch exchange the order between clear used and memset, add barrier to ensure memory consistency.

Fixes 1c5526968e27 ("net/smc: Clear memory when release and reuse buffer")
---
 net/smc/smc_core.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)
net/smc/smc_core.c +8 -9
index c305d8d..c19d4b7 100644
@@ -1120,8 +1120,9 @@ static void smcr_buf_unuse(struct smc_buf_desc *buf_desc, bool is_rmb,
 
 		smc_buf_free(lgr, is_rmb, buf_desc);
 	} else {
-		buf_desc->used = 0;
-		memset(buf_desc->cpu_addr, 0, buf_desc->len);
+		/* memzero_explicit provides potential memory barrier semantics */
+		memzero_explicit(buf_desc->cpu_addr, buf_desc->len);
+		WRITE_ONCE(buf_desc->used, 0);
 	}
 }
 
@@ -1132,19 +1133,17 @@ static void smc_buf_unuse(struct smc_connection *conn,
 		if (!lgr->is_smcd && conn->sndbuf_desc->is_vm) {
 			smcr_buf_unuse(conn->sndbuf_desc, false, lgr);
 		} else {
-			conn->sndbuf_desc->used = 0;
-			memset(conn->sndbuf_desc->cpu_addr, 0,
-			       conn->sndbuf_desc->len);
+			memzero_explicit(conn->sndbuf_desc->cpu_addr, conn->sndbuf_desc->len);
+			WRITE_ONCE(conn->sndbuf_desc->used, 0);
 		}
 	}
 	if (conn->rmb_desc) {
 		if (!lgr->is_smcd) {
 			smcr_buf_unuse(conn->rmb_desc, true, lgr);
 		} else {
-			conn->rmb_desc->used = 0;
-			memset(conn->rmb_desc->cpu_addr, 0,
-			       conn->rmb_desc->len +
-			       sizeof(struct smcd_cdc_msg));
+			memzero_explicit(conn->rmb_desc->cpu_addr,
+					 conn->rmb_desc->len + sizeof(struct smcd_cdc_msg));
+			WRITE_ONCE(conn->rmb_desc->used, 0);
 		}
 	}
 }
Signature · 1 line
1.8.3.1
author maintainer
#1 lore raw

Hi, wenjia

This patch of bugfix seems to have been hanging for a long time. If you have any concerns, please let us know.

Best wishes.
D. Wythe

Quoted 80 lines · D.Wythe

From: "D. Wythe" <[email protected]>

There is a certain probability that following
exceptions will occur in the wrk benchmark test:

Running 10s test @ http://11.213.45.6:80
8 threads and 64 connections
Thread Stats Avg Stdev Max +/- Stdev

     Latency     3.72ms   13.94ms 245.33ms   94.17%
     Req/Sec     1.96k   713.67     5.41k    75.16%

155262 requests in 10.10s, 23.10MB read
Non-2xx or 3xx responses: 3

We will find that the error is HTTP 400 error, which is a serious exception in our test, which means the application data was corrupted.

Consider the following scenarios:

CPU0 CPU1

buf_desc->used = 0;
                                 cmpxchg(buf_desc->used, 0, 1)
                                 deal_with(buf_desc)

memset(buf_desc->cpu_addr,0);

This will cause the data received by a victim connection to be cleared, thus triggering an HTTP 400 error in the server.

This patch exchange the order between clear used and memset, add barrier to ensure memory consistency.

Fixes 1c5526968e27 ("net/smc: Clear memory when release and reuse buffer")
---
  net/smc/smc_core.c | 17 ++++++++---------
  1 file changed, 8 insertions(+), 9 deletions(-)
net/smc/smc_core.c +8 -9
index c305d8d..c19d4b7 100644
@@ -1120,8 +1120,9 @@ static void smcr_buf_unuse(struct smc_buf_desc *buf_desc, bool is_rmb,
 
  		smc_buf_free(lgr, is_rmb, buf_desc);
  	} else {
-		buf_desc->used = 0;
-		memset(buf_desc->cpu_addr, 0, buf_desc->len);
+		/* memzero_explicit provides potential memory barrier semantics */
+		memzero_explicit(buf_desc->cpu_addr, buf_desc->len);
+		WRITE_ONCE(buf_desc->used, 0);
  	}
  }
 
@@ -1132,19 +1133,17 @@ static void smc_buf_unuse(struct smc_connection *conn,
  		if (!lgr->is_smcd && conn->sndbuf_desc->is_vm) {
  			smcr_buf_unuse(conn->sndbuf_desc, false, lgr);
  		} else {
-			conn->sndbuf_desc->used = 0;
-			memset(conn->sndbuf_desc->cpu_addr, 0,
-			       conn->sndbuf_desc->len);
+			memzero_explicit(conn->sndbuf_desc->cpu_addr, conn->sndbuf_desc->len);
+			WRITE_ONCE(conn->sndbuf_desc->used, 0);
  		}
  	}
  	if (conn->rmb_desc) {
  		if (!lgr->is_smcd) {
  			smcr_buf_unuse(conn->rmb_desc, true, lgr);
  		} else {
-			conn->rmb_desc->used = 0;
-			memset(conn->rmb_desc->cpu_addr, 0,
-			       conn->rmb_desc->len +
-			       sizeof(struct smcd_cdc_msg));
+			memzero_explicit(conn->rmb_desc->cpu_addr,
+					 conn->rmb_desc->len + sizeof(struct smcd_cdc_msg));
+			WRITE_ONCE(conn->rmb_desc->used, 0);
  		}
  	}
  }
Reviewed-by on v1 1/1
#2 lore raw
In reply to D.Wythe
Quoted 70 lines · D.Wythe

From: "D. Wythe" <[email protected]>

There is a certain probability that following
exceptions will occur in the wrk benchmark test:

Running 10s test @ http://11.213.45.6:80
8 threads and 64 connections
Thread Stats Avg Stdev Max +/- Stdev

     Latency     3.72ms   13.94ms 245.33ms   94.17%
     Req/Sec     1.96k   713.67     5.41k    75.16%

155262 requests in 10.10s, 23.10MB read
Non-2xx or 3xx responses: 3

We will find that the error is HTTP 400 error, which is a serious exception in our test, which means the application data was corrupted.

Consider the following scenarios:

CPU0 CPU1

buf_desc->used = 0;
                                 cmpxchg(buf_desc->used, 0, 1)
                                 deal_with(buf_desc)

memset(buf_desc->cpu_addr,0);

This will cause the data received by a victim connection to be cleared, thus triggering an HTTP 400 error in the server.

This patch exchange the order between clear used and memset, add barrier to ensure memory consistency.

Fixes 1c5526968e27 ("net/smc: Clear memory when release and reuse buffer")
---
  net/smc/smc_core.c | 17 ++++++++---------
  1 file changed, 8 insertions(+), 9 deletions(-)
net/smc/smc_core.c +5 -5
index c305d8d..c19d4b7 100644
@@ -1120,8 +1120,9 @@ static void smcr_buf_unuse(struct smc_buf_desc *buf_desc, bool is_rmb,
 
  		smc_buf_free(lgr, is_rmb, buf_desc);
  	} else {
-		buf_desc->used = 0;
-		memset(buf_desc->cpu_addr, 0, buf_desc->len);
+		/* memzero_explicit provides potential memory barrier semantics */
+		memzero_explicit(buf_desc->cpu_addr, buf_desc->len);
+		WRITE_ONCE(buf_desc->used, 0);
  	}
  }
 
@@ -1132,19 +1133,17 @@ static void smc_buf_unuse(struct smc_connection *conn,
  		if (!lgr->is_smcd && conn->sndbuf_desc->is_vm) {
  			smcr_buf_unuse(conn->sndbuf_desc, false, lgr);
  		} else {
-			conn->sndbuf_desc->used = 0;
-			memset(conn->sndbuf_desc->cpu_addr, 0,
-			       conn->sndbuf_desc->len);
+			memzero_explicit(conn->sndbuf_desc->cpu_addr, conn->sndbuf_desc->len);
+			WRITE_ONCE(conn->sndbuf_desc->used, 0);
  		}
  	}
  	if (conn->rmb_desc) {
  		if (!lgr->is_smcd) {
  			smcr_buf_unuse(conn->rmb_desc, true, lgr);
  		} else {

net/smc/smc_core.c static void smc_buf_unuse(struct smc_connection *conn, · line 1149

Quoted 10 lines
net/smc/smc_core.c +3 -4
-			conn->rmb_desc->used = 0;
-			memset(conn->rmb_desc->cpu_addr, 0,
-			       conn->rmb_desc->len +
-			       sizeof(struct smcd_cdc_msg));
+			memzero_explicit(conn->rmb_desc->cpu_addr,
+					 conn->rmb_desc->len + sizeof(struct smcd_cdc_msg));
+			WRITE_ONCE(conn->rmb_desc->used, 0);
  		}
  	}
  }

Hi David,

Thank you for remembering me again about this patch. I did forget to answer you, sorry!

My consideration was if memzero_explicit() is necessary in this case. But sure, it makes sense, especiall when the dereferencing is in somewhere else.

Thank you for the fix!

maintainer
#3 lore raw
Quoted 5 lines · On Wed, 15 Feb 2023 10:27:55 +0100 Wenjia Zhang wrote:

Hi David,

Thank you for remembering me again about this patch. I did forget to answer you, sorry!

Quoted 7 lines

My consideration was if memzero_explicit() is necessary in this case. But sure, it makes sense, especiall when the dereferencing is in somewhere else.

Thank you for the fix!

Thanks! David, please repost if you'd like the patch to be applied to the networking tree. The original posting is too old to use.

author maintainer
#4 lore raw
Quoted 10 lines · Jakub Kicinski
Quoted 9 lines · On Wed, 15 Feb 2023 10:27:55 +0100 Wenjia Zhang wrote:

Hi David,

Thank you for remembering me again about this patch. I did forget to answer you, sorry!

My consideration was if memzero_explicit() is necessary in this case. But sure, it makes sense, especiall when the dereferencing is in somewhere else.

Quoted 6 lines
Quoted 3 lines

Thank you for the fix!

Thanks! David, please repost if you'd like the patch to be applied to the networking tree. The original posting is too old to use.

Thank you for your reminder. I will repost it after rebasing. D. Wythe