Skip to contentSkip to conversation

tls: fix race between tx work scheduling and socket close

PATCH In review Lee Jones 4 participants 9 messages Mar 7, 2024 – Aug 13, 2024 Stable branch

lore
Share & tools
lore

Lee Jones

author
#0 lore raw

PATCH 1/1 tls: fix race between tx work scheduling and socket close

From: Jakub Kicinski <[email protected]>

[ Upstream commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb ]

Similarly to previous commit, the submitting thread (recvmsg/sendmsg) may exit as soon as the async crypto handler calls complete(). Reorder scheduling the work before calling complete(). This seems more logical in the first place, as it's
the inverse order of what the submitting thread will do.

Fixes a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
(cherry picked from commit 6db22d6c7a6dc914b12c0469b94eb639b6a8a146)
[Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
---
 net/tls/tls_sw.c | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)
net/tls/tls_sw.c +6 -10
index 2bd27b77769cb..d53587ff9ddea 100644
@@ -449,7 +449,6 @@ static void tls_encrypt_done(crypto_completion_data_t *data, int err)
 	struct scatterlist *sge;
 	struct sk_msg *msg_en;
 	struct tls_rec *rec;
-	bool ready = false;
 	struct sock *sk;
 
 	rec = container_of(aead_req, struct tls_rec, aead_req);
@@ -486,19 +485,16 @@ static void tls_encrypt_done(crypto_completion_data_t *data, int err)
 		/* If received record is at head of tx_list, schedule tx */
 		first_rec = list_first_entry(&ctx->tx_list,
 					     struct tls_rec, list);
-		if (rec == first_rec)
-			ready = true;
+		if (rec == first_rec) {
+			/* Schedule the transmission */
+			if (!test_and_set_bit(BIT_TX_SCHEDULED,
+					      &ctx->tx_bitmask))
+				schedule_delayed_work(&ctx->tx_work.work, 1);
+		}
 	}
 
 	if (atomic_dec_and_test(&ctx->encrypt_pending))
 		complete(&ctx->async_wait.completion);
-
-	if (!ready)
-		return;
-
-	/* Schedule the transmission */
-	if (!test_and_set_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask))
-		schedule_delayed_work(&ctx->tx_work.work, 1);
 }
 
 static int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)
Signature · 1 line
2.44.0.278.ge034bb2e1d-goog
#1 lore raw
Quoted 10 lines · On Thu, 7 Mar 2024 15:59:29 +0000 Lee Jones wrote:

From: Jakub Kicinski <[email protected]>

[ Upstream commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb ]

Similarly to previous commit, the submitting thread (recvmsg/sendmsg) may exit as soon as the async crypto handler calls complete(). Reorder scheduling the work before calling complete(). This seems more logical in the first place, as it's
the inverse order of what the submitting thread will do.

Quoted 10 lines
Fixes a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
(cherry picked from commit 6db22d6c7a6dc914b12c0469b94eb639b6a8a146)
[Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]

LG, thank you!

The 5.15 / 5.10 / 5.4 fixes won't be effective, tho. I don't see commit aec7961916f3f9e88766 in the other LTS branches. Without that (it's still correct but) it doesn't fix the problem, because we still touch the context after releasing the reference (unlocking the spin lock).

author
#2 lore raw
Quoted 22 lines · Jakub Kicinski
Quoted 10 lines · On Thu, 7 Mar 2024 15:59:29 +0000 Lee Jones wrote:

From: Jakub Kicinski <[email protected]>

[ Upstream commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb ]

Similarly to previous commit, the submitting thread (recvmsg/sendmsg) may exit as soon as the async crypto handler calls complete(). Reorder scheduling the work before calling complete(). This seems more logical in the first place, as it's
the inverse order of what the submitting thread will do.

Quoted 10 lines
Fixes a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
(cherry picked from commit 6db22d6c7a6dc914b12c0469b94eb639b6a8a146)
[Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
Quoted 7 lines

LG, thank you!

The 5.15 / 5.10 / 5.4 fixes won't be effective, tho. I don't see commit aec7961916f3f9e88766 in the other LTS branches. Without that (it's still correct but) it doesn't fix the problem, because we still touch the context after releasing the reference (unlocking the spin lock).

No problem.

Should I accompany aec7961916f3 with this fix into the aforementioned branches then? Would that then be effective?

Signature · 1 line
Lee Jones [李琼斯]
#3 lore raw
Quoted 10 lines · On Thu, 7 Mar 2024 17:09:59 +0000 Lee Jones wrote:
Quoted 5 lines

The 5.15 / 5.10 / 5.4 fixes won't be effective, tho. I don't see commit aec7961916f3f9e88766 in the other LTS branches. Without that (it's still correct but) it doesn't fix the problem, because we still touch the context after releasing the reference (unlocking the spin lock).

No problem.

Should I accompany aec7961916f3 with this fix into the aforementioned branches then? Would that then be effective?

Yes, (and c57ca512f3b68d, that's all the pre-req, I think.) Tho, it may be a more tedious backport.
FWIW tools/testing/selftests/net/tls.c should be relatively solid.

maintainer
#7 lore raw
In reply to Lee Jones
Quoted 20 lines · Lee Jones

From: Jakub Kicinski <[email protected]>

[ Upstream commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb ]

Similarly to previous commit, the submitting thread (recvmsg/sendmsg) may exit as soon as the async crypto handler calls complete(). Reorder scheduling the work before calling complete(). This seems more logical in the first place, as it's
the inverse order of what the submitting thread will do.

Fixes a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
(cherry picked from commit 6db22d6c7a6dc914b12c0469b94eb639b6a8a146)
[Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
Quoted 4 lines
---
 net/tls/tls_sw.c | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)

Now qeueued up, but only this version, the older ones I've dropped from my review queue based on the review from Jakub. If they are still needed, can you provide backported versions?

thanks,

greg k-h

author
#8 lore raw
Quoted 26 lines · Greg KH
Quoted 20 lines · Lee Jones

From: Jakub Kicinski <[email protected]>

[ Upstream commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb ]

Similarly to previous commit, the submitting thread (recvmsg/sendmsg) may exit as soon as the async crypto handler calls complete(). Reorder scheduling the work before calling complete(). This seems more logical in the first place, as it's
the inverse order of what the submitting thread will do.

Fixes a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
(cherry picked from commit 6db22d6c7a6dc914b12c0469b94eb639b6a8a146)
[Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
Quoted 4 lines
---
 net/tls/tls_sw.c | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)
Quoted 3 lines

Now qeueued up, but only this version, the older ones I've dropped from my review queue based on the review from Jakub. If they are still needed, can you provide backported versions?

Thanks.

Full disclosure, I have no plans to backport the remainder given Jakub's comments.

Signature · 1 line
Lee Jones [李琼斯]
#6 lore raw
In reply to Jakub Kicinski

Hi Jakub,

Quoted 22 lines · Jakub Kicinski
Quoted 10 lines · On Thu, 7 Mar 2024 15:59:29 +0000 Lee Jones wrote:

From: Jakub Kicinski <[email protected]>

[ Upstream commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb ]

Similarly to previous commit, the submitting thread (recvmsg/sendmsg) may exit as soon as the async crypto handler calls complete(). Reorder scheduling the work before calling complete(). This seems more logical in the first place, as it's
the inverse order of what the submitting thread will do.

Quoted 10 lines
Fixes a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
(cherry picked from commit 6db22d6c7a6dc914b12c0469b94eb639b6a8a146)
[Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]
Quoted 8 lines

LG, thank you!

The 5.15 / 5.10 / 5.4 fixes won't be effective, tho. I don't see commit aec7961916f3f9e88766 in the other LTS branches. Without that (it's still correct but) it doesn't fix the problem, because we still touch the context after releasing the reference (unlocking the spin lock).

commit: aec7961916f3 ("tls: fix race between async notify and socket close") is backported to 5.15.y as commit f17d21ea7391 ("tls: fix race between async notify and socket close") in v5.15.160

So I think we should now backport this fix e01e3934a1b2 ("tls: fix race between tx work scheduling and socket close") to 5.15.y ( this also fixes CVE-2024-26585)

Will send the backport to stable kernel list.

Thanks,
Harshit

#4 lore raw

PATCH 1/1 tls: fix race between tx work scheduling and socket close

In reply to Jakub Kicinski

From: Jakub Kicinski <[email protected]>

commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb upstream.

Similarly to previous commit, the submitting thread (recvmsg/sendmsg) may exit as soon as the async crypto handler calls complete(). Reorder scheduling the work before calling complete(). This seems more logical in the first place, as it's
the inverse order of what the submitting thread will do.

Fixes a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
[Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]

[Harshit: bp to 5.15.y, minor conflict resolutin due to missing commit:
8ae187386420 ("tls: Only use data field in crypto completion function") in 5.15.y]

---
This is a fix for CVE-2024-26585, minor conflict resolution involved Ran the tls self tests:

ok 183 tls.13_chacha.shutdown_reuse
# PASSED: 183 / 183 tests passed.
# Totals: pass:183 fail:0 xfail:0 xpass:0 skip:0 error:0

 net/tls/tls_sw.c | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)
net/tls/tls_sw.c +6 -10
index 90f6cbe5cd5d..c17c3a14b9c1 100644
@@ -468,7 +468,6 @@ static void tls_encrypt_done(struct crypto_async_request *req, int err)
 	struct scatterlist *sge;
 	struct sk_msg *msg_en;
 	struct tls_rec *rec;
-	bool ready = false;
 
 	if (err == -EINPROGRESS) /* see the comment in tls_decrypt_done() */
 		return;
@@ -502,19 +501,16 @@ static void tls_encrypt_done(struct crypto_async_request *req, int err)
 		/* If received record is at head of tx_list, schedule tx */
 		first_rec = list_first_entry(&ctx->tx_list,
 					     struct tls_rec, list);
-		if (rec == first_rec)
-			ready = true;
+		if (rec == first_rec) {
+			/* Schedule the transmission */
+			if (!test_and_set_bit(BIT_TX_SCHEDULED,
+					      &ctx->tx_bitmask))
+				schedule_delayed_work(&ctx->tx_work.work, 1);
+		}
 	}
 
 	if (atomic_dec_and_test(&ctx->encrypt_pending))
 		complete(&ctx->async_wait.completion);
-
-	if (!ready)
-		return;
-
-	/* Schedule the transmission */
-	if (!test_and_set_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask))
-		schedule_delayed_work(&ctx->tx_work.work, 1);
 }
 
 static int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)
Signature · 1 line
2.45.2
#5 lore raw
Quoted 27 lines · Harshit Mogalapalli

From: Jakub Kicinski <[email protected]>

commit e01e3934a1b2d122919f73bc6ddbe1cdafc4bbdb upstream.

Similarly to previous commit, the submitting thread (recvmsg/sendmsg) may exit as soon as the async crypto handler calls complete(). Reorder scheduling the work before calling complete(). This seems more logical in the first place, as it's
the inverse order of what the submitting thread will do.

Fixes a42055e8d2c3 ("net/tls: Add support for async encryption of records for performance")
[Lee: Fixed merge-conflict in Stable branches linux-6.1.y and older]

[Harshit: bp to 5.15.y, minor conflict resolutin due to missing commit:
8ae187386420 ("tls: Only use data field in crypto completion function") in 5.15.y]

---
This is a fix for CVE-2024-26585, minor conflict resolution involved Ran the tls self tests:

Quoted 6 lines

ok 183 tls.13_chacha.shutdown_reuse
# PASSED: 183 / 183 tests passed.
# Totals: pass:183 fail:0 xfail:0 xpass:0 skip:0 error:0

 net/tls/tls_sw.c | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)

Now queued up, thanks.

greg k-h