fix wrong http response for sstate cache checkstatus
PATCH 1/1 fix wrong http response for sstate cache checkstatus
Hi
It's about bug and the patch to fix it.
I tried 'git send-email' but unable to successfully send email by this command. so I try to send it by office client
the issue is in sstate-cache checkstatus() for http mirrors
in http checkstatus it use HTTPConnectionCache to share connections
A request expired 30s timeout and released the connection,
B request took the connection into use and get the wrong response(meant to response A request and late than 30s) details can be found in attached image and patch
Yang zhang
Central Build SCM Team
From 76bc06e823d67e4eecaf7f2c7be0b67e405b1f76 Mon Sep 17 00:00:00 2001
From: y75zhang <[email protected]>
Date: Wed, 3 Jul 2024 08:20:34 +0300
Subject: [PATCH] bitbake: checkstatus: drop shared connecton when catch
Timeout error
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
* to avoid wrong http response in checkstatus function:
in wget checkstatus() we are using 'HTTPConnectionCache' to share connections
1. state_file1(exists on http server) use shared connection <shared1> to send request
2. http_server recieved request of state_file1, but delayed by some reason to sent respone
3. state_file1 checkstatus() failed by timeout and drop shared connection <shared1>
4. state_file2(not exists on http server) get shared connection <shared1> and send request
5. http_server finally send 200 response for state_file1
6. state_file2 recived 200 response and thought it was exists on http_server
* how to reproduce:
modify /usr/lib64/python3.10/http/client.py like this to fake timeout event
you will find that some sstate files with 200 response in fact do not exists in remote http server
— a/client.py
+++ b/client.py
@@ -68,6 +68,7 @@ Req-started-unread-response _CS_REQ_STARTED <response_class>
Req-sent-unread-response _CS_REQ_SENT <response_class>
"""
+import random
import email.parser
import email.message
import errno
@@ -1370,6 +1371,8 @@ class HTTPConnection:
else:
response = self.response_class(self.sock, method=self._method)
+ if random.choice([False, False, False, False, False, False, False, True]):
+ raise TimeoutError('timed out test')
try:
try:
response.begin()
--- bitbake/lib/bb/fetch2/wget.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
bitbake/lib/bb/fetch2/wget.py +7 -2
@@ -244,7 +244,12 @@ class Wget(FetchMethod): fetch.connection_cache.remove_connection(h.host, h.port) raise urllib.error.URLError(err) else: - r = h.getresponse() + try: + r = h.getresponse() + except TimeoutError as e: + if fetch.connection_cache: + fetch.connection_cache.remove_connection(h.host, h.port) + raise TimeoutError(e) # Pick apart the HTTPResponse object to get the addinfourl # object initialized properly.
@@ -496,7 +501,7 @@ class Wget(FetchMethod): valid = 1 elif self._vercmp(version, newver) < 0: version = newver - + pupver = re.sub('_', '.', version[1]) bb.debug(3, "*** %s -> UpstreamVersion = %s (CurrentVersion = %s)" %
Signature · 1 line
2.9.5
Note that reviewing patches sent as attachments is very difficult, as we can't comment on what is specifically in the patch. Can you make an account on gmail perhaps and use that to send them? 'git send-email' will ensure the patch credentials will be the same, it's only for the patch transport.
Alex
Quoted 39 lines · Yang-Mark Zhang (NSB) via lists.openembedded.org
Hi
It’s about bug and the patch to fix it.
I tried ‘git send-email’ but unable to successfully send email by this command. so I try to send it by office clientthe issue is in sstate-cache
checkstatus()for http mirrors
in http checkstatus it use HTTPConnectionCache to share connections
A request expired 30s timeout and released the connection,
B request took the connection into use and get the wrong response(meant to response A request and late than 30s) details can be found in attached image and patchYang zhang
Central Build SCM Team
--
Yang zhang
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16386): https://lists.openembedded.org/g/bitbake-devel/message/16386
Mute This Topic: https://lists.openembedded.org/mt/107033227/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 6 lines · Alexander Kanavin via lists.openembedded.org
Note that reviewing patches sent as attachments is very difficult, as we can't comment on what is specifically in the patch. Can you make an
account on gmail perhaps and use that to send them? 'git send-email' will ensure the patch credentials will be the same, it's only for the patch transport.
The issue in this one looks quite serious so we've extracted the patch and queued for testing. In general we can't always commit to doing that though.
Cheers,
Richard
Hi Richard,
after team member's help
we finally can send out the email through 'git send-email'
https://lists.openembedded.org/g/bitbake-devel/message/16391
thank you very much for your nice helping
Yang zhang
-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: 2024年7月4日 23:27
To: [email protected]; Yang-Mark Zhang (NSB) <[email protected]>
Cc: [email protected]
Subject: [External] Re: [bitbake-devel] [PATCH] fix wrong http response for sstate cache checkstatus
Caution: This is an external email. Please be very careful when clicking links or opening attachments. See http://nok.it/nsb for additional information.
Quoted 5 lines · Alexander Kanavin via lists.openembedded.org
Note that reviewing patches sent as attachments is very difficult, as we can't comment on what is specifically in the patch. Can you make an account on gmail perhaps and use that to send them? 'git send-email' will ensure the patch credentials will be the same, it's only for the patch transport.
The issue in this one looks quite serious so we've extracted the patch and queued for testing. In general we can't always commit to doing that though.
Cheers,
Richard