Connectivity check uris
Hi folks,
I have noticed some hang-ups at the beginning of my builds on the master branch. I have search a little and discovered that the connectivity check made by poky was the root cause. In fact connectivity check use the CONNECTIVITY_CHECK_URIS variable which is by default set to "https://www.example.com/".
So I have made some tests with a little script which gets the URI 100 times:
$ cat connectivity_check.sh
#!/bin/sh
connectivity_check()
{
local ok=0
for i in $(seq 100)
do
$@ > /dev/null && ok=$(($ok + 1))
done
echo $ok $@
}
connectivity_check wget -qO- --timeout=1 --tries=1 $1
connectivity_check curl -s --connect-timeout 1 $1Note that in the script, I use wget and curl to see if there are any differences but I don't think that's a thing.
Here are the results:
$ time ./connectivity_check.sh https://example.com 45 wget -qO- --timeout=1 --tries=1 https://example.com 45 curl -s --connect-timeout 1 https://example.com
real 2m23.595s user 0m3.352s sys 0m1.834s $ time ./connectivity_check.sh https://openembedded.org 100 wget -qO- --timeout=1 --tries=1 https://openembedded.org 100 curl -s --connect-timeout 1 https://openembedded.org real 4m36.188s user 0m3.853s sys 0m1.761s $ time ./connectivity_check.sh https://google.com 100 wget -qO- --timeout=1 --tries=1 https://google.com 100 curl -s --connect-timeout 1 https://google.com real 0m36.328s user 0m4.265s sys 0m2.028s
So "example.com" is clearly not a good domain to do connectivity checks.
The "openembedded.org" domain is good but have a slow response time.
And finally "google.com" which have all sort of speedy networking stuff is very efficient.
---
I think the CONNECTIVITY_CHECK_URIS variable must be updated and not use the "example.com" domain anymore. Using "google.com" may be the best option. But eetting up a custom domain like "connectivitycheck.openembedded.org" could be another idea.
Any ideas ?
Jean-Marie LEMETAYER
Passionate embedded systems engineer
Savoir-faire Linux
Quoted 59 lines · Jean-Marie LEMETAYER
Hi folks,
I have noticed some hang-ups at the beginning of my builds on the master branch. I have search a little and discovered that the connectivity check made by poky was the root cause. In fact connectivity check use the
CONNECTIVITY_CHECK_URISvariable which is by default set to "https://www.example.com/".So I have made some tests with a little script which gets the URI 100 times:
$ cat connectivity_check.sh #!/bin/sh connectivity_check() { local ok=0 for i in $(seq 100) do $@ > /dev/null && ok=$(($ok + 1)) done echo $ok $@ } connectivity_check wget -qO- --timeout=1 --tries=1 $1 connectivity_check curl -s --connect-timeout 1 $1Note that in the script, I use wget and curl to see if there are any differences but I don't think that's a thing.
Here are the results:
$ time ./connectivity_check.sh https://example.com 45 wget -qO- --timeout=1 --tries=1 https://example.com 45 curl -s --connect-timeout 1 https://example.comreal 2m23.595s user 0m3.352s sys 0m1.834s$ time ./connectivity_check.sh https://openembedded.org 100 wget -qO- --timeout=1 --tries=1 https://openembedded.org 100 curl -s --connect-timeout 1 https://openembedded.orgreal 4m36.188s user 0m3.853s sys 0m1.761s$ time ./connectivity_check.sh https://google.com 100 wget -qO- --timeout=1 --tries=1 https://google.com 100 curl -s --connect-timeout 1 https://google.comreal 0m36.328s user 0m4.265s sys 0m2.028sSo "example.com" is clearly not a good domain to do connectivity checks.
The "openembedded.org" domain is good but have a slow response time.
And finally "google.com" which have all sort of speedy networking stuff is very efficient.
Quoted 6 lines
---
I think the
CONNECTIVITY_CHECK_URISvariable must be updated and not use the "example.com" domain anymore. Using "google.com" may be the best option. But eetting up a custom domain like "connectivitycheck.openembedded.org" could be another idea.
Had this idea long time in the past... Unfortunately, google.com is not available in all countries worldwide (Say "Nihao" to chinese Great firewall ).
Quoted 11 lines
Any ideas ?
Jean-Marie LEMETAYER
Passionate embedded systems engineer
Savoir-faire Linux
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 2 lines
Regards, Andrey.
Quoted 75 lines · Jean-Marie LEMETAYER
Hi folks,
I have noticed some hang-ups at the beginning of my builds on the master branch.
I have search a little and discovered that the connectivity check made by poky
was the root cause. In fact connectivity check use theCONNECTIVITY_CHECK_URIS
variable which is by default set to "https://www.example.com/".So I have made some tests with a little script which gets the URI 100 times:
$ cat connectivity_check.sh #!/bin/sh connectivity_check() { local ok=0 for i in $(seq 100) do $@ > /dev/null && ok=$(($ok + 1)) done echo $ok $@ } connectivity_check wget -qO- --timeout=1 --tries=1 $1 connectivity_check curl -s --connect-timeout 1 $1Note that in the script, I use wget and curl to see if there are any differences
but I don't think that's a thing.Here are the results:
$ time ./connectivity_check.sh https://example.com 45 wget -qO- --timeout=1 --tries=1 https://example.com 45 curl -s --connect-timeout 1 https://example.comreal 2m23.595s user 0m3.352s sys 0m1.834s $ time ./connectivity_check.sh https://openembedded.org 100 wget -qO- --timeout=1 --tries=1 https://openembedded.org 100 curl -s --connect-timeout 1 https://openembedded.org real 4m36.188s user 0m3.853s sys 0m1.761s $ time ./connectivity_check.sh https://google.com 100 wget -qO- --timeout=1 --tries=1 https://google.com 100 curl -s --connect-timeout 1 https://google.com real 0m36.328s user 0m4.265s sys 0m2.028sSo "example.com" is clearly not a good domain to do connectivity checks.
The "openembedded.org" domain is good but have a slow response time.
And finally "google.com" which have all sort of speedy networking stuff is very
efficient.---
I think the
CONNECTIVITY_CHECK_URISvariable must be updated and not use the
"example.com" domain anymore. Using "google.com" may be the best option. But
eetting up a custom domain like "connectivitycheck.openembedded.org" could be
another idea.
Quoted 1 line
Any ideas
I think if you look at the history we did use google.com however that caused problems for people in geos where that is blocked (e.g. PRC).
We may want to move the default back given the performance isse, not sure...
Cheers,
Richard
Quoted 76 lines · On Jan 23, 2020, at 2:26 PM, Richard Purdie [email protected]…
Quoted 75 lines · Jean-Marie LEMETAYER
Hi folks,
I have noticed some hang-ups at the beginning of my builds on the master branch.
I have search a little and discovered that the connectivity check made by poky
was the root cause. In fact connectivity check use theCONNECTIVITY_CHECK_URIS
variable which is by default set to "https://www.example.com/".So I have made some tests with a little script which gets the URI 100 times:
$ cat connectivity_check.sh #!/bin/sh connectivity_check() { local ok=0 for i in $(seq 100) do $@ > /dev/null && ok=$(($ok + 1)) done echo $ok $@ } connectivity_check wget -qO- --timeout=1 --tries=1 $1 connectivity_check curl -s --connect-timeout 1 $1Note that in the script, I use wget and curl to see if there are any differences
but I don't think that's a thing.Here are the results:
$ time ./connectivity_check.sh https://example.com 45 wget -qO- --timeout=1 --tries=1 https://example.com 45 curl -s --connect-timeout 1 https://example.comreal 2m23.595s user 0m3.352s sys 0m1.834s $ time ./connectivity_check.sh https://openembedded.org 100 wget -qO- --timeout=1 --tries=1 https://openembedded.org 100 curl -s --connect-timeout 1 https://openembedded.org real 4m36.188s user 0m3.853s sys 0m1.761s $ time ./connectivity_check.sh https://google.com 100 wget -qO- --timeout=1 --tries=1 https://google.com 100 curl -s --connect-timeout 1 https://google.com real 0m36.328s user 0m4.265s sys 0m2.028sSo "example.com" is clearly not a good domain to do connectivity checks.
The "openembedded.org" domain is good but have a slow response time.
And finally "google.com" which have all sort of speedy networking stuff is very
efficient.---
I think the
CONNECTIVITY_CHECK_URISvariable must be updated and not use the
"example.com" domain anymore. Using "google.com" may be the best option. But
eetting up a custom domain like "connectivitycheck.openembedded.org" could be
another idea.
Quoted 7 lines
Quoted 1 line
Any ideas
I think if you look at the history we did use google.com however that caused problems for people in geos where that is blocked (e.g. PRC).
We may want to move the default back given the performance isse, not sure...
I understand the issue with blocked domains this is why I suggest the custom domain "connectivitycheck.openembedded.org".
But I dont know if it could be an option. Can OE have a sub-domain like that? maybe using some cdn to handle the load?
Another option will be to provide multiple URIs:
CONNECTIVITY_CHECK_URIS ?= " \
https://google.com \
https://www.bing.com \
https://www.baidu.com \
"Best regards,
Jean-Marie
Quoted 5 lines · Jean-Marie LEMETAYER
I understand the issue with blocked domains this is why I suggest the custom domain "connectivitycheck.openembedded.org".
But I dont know if it could be an option. Can OE have a sub-domain like that? maybe using some cdn to handle the load?
We can certainly setup domains but we can't easily get a cdn. I'd be worried about OE's response time too.
Quoted 6 lines
Another option will be to provide multiple URIs:
CONNECTIVITY_CHECK_URIS ?= " \ https://google.com \ https://www.bing.com \ https://www.baidu.com \ "
Looking at the code, I wonder if a CONNECTIVITY_CHECK_MIRRORS, injected into MIRRORS internally with google as the default and baidu as the mirror would work?
Cheers,
Richard
Quoted 17 lines · On Jan 23, 2020, at 3:37 PM, Richard Purdie [email protected]…
Quoted 5 lines · Jean-Marie LEMETAYER
I understand the issue with blocked domains this is why I suggest the custom domain "connectivitycheck.openembedded.org".
But I dont know if it could be an option. Can OE have a sub-domain like that? maybe using some cdn to handle the load?
We can certainly setup domains but we can't easily get a cdn. I'd be worried about OE's response time too.
Quoted 6 lines
Another option will be to provide multiple URIs:
CONNECTIVITY_CHECK_URIS ?= " \ https://google.com \ https://www.bing.com \ https://www.baidu.com \ "
Quoted 3 lines
Looking at the code, I wonder if a
CONNECTIVITY_CHECK_MIRRORS, injected into MIRRORS internally with google as the default and baidu as the mirror would work?
It should work but it will not be efficient as the tries are serially executed. So someone in China should have to wait that the former domains failed to be able to start the build.
I am still thinking that the OE subdomain with CDN could be the best option.
Regards,
Jean-Marie
Quoted 22 lines · Jean-Marie LEMETAYER
Quoted 20 lines · On Jan 23, 2020, at 3:37 PM, Richard Purdie [email protected]…
Quoted 7 lines · Jean-Marie LEMETAYER
I understand the issue with blocked domains this is why I suggest the
custom domain "connectivitycheck.openembedded.org".But I dont know if it could be an option. Can OE have a sub- domain
like that? maybe using some cdn to handle the load?We can certainly setup domains but we can't easily get a cdn. I'd be
worried about OE's response time too.Quoted 6 lines
Another option will be to provide multiple URIs:
CONNECTIVITY_CHECK_URIS ?= " \ https://google.com \ https://www.bing.com \ https://www.baidu.com \ "
Quoted 8 lines
Quoted 4 lines
Looking at the code, I wonder if a
CONNECTIVITY_CHECK_MIRRORS, injected
into MIRRORS internally with google as the default and baidu as the mirror would work?It should work but it will not be efficient as the tries are serially executed. So someone in China should have to wait that the former domains failed to be able to start the build.
I'm guessing those wouldn't resolve so it should be pretty fast?
Maybe we make the configuration more obvious in local.conf?
Quoted 2 lines
I am still thinking that the OE subdomain with CDN could be the best option.
Sure, but who is going to pay for and maintain a CDN setup?
Cheers,
Richard
Quoted 23 lines · Richard Purdie
Quoted 22 lines · Jean-Marie LEMETAYER
Quoted 20 lines · On Jan 23, 2020, at 3:37 PM, Richard Purdie [email protected]…
Quoted 7 lines · Jean-Marie LEMETAYER
I understand the issue with blocked domains this is why I suggest the
custom domain "connectivitycheck.openembedded.org".But I dont know if it could be an option. Can OE have a sub- domain
like that? maybe using some cdn to handle the load?We can certainly setup domains but we can't easily get a cdn. I'd be
worried about OE's response time too.Quoted 6 lines
Another option will be to provide multiple URIs:
CONNECTIVITY_CHECK_URIS ?= " \ https://google.com \ https://www.bing.com \ https://www.baidu.com \ "
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Looking at the code, I wonder if a
CONNECTIVITY_CHECK_MIRRORS, injected
into MIRRORS internally with google as the default and baidu as the mirror would work?
How about checking github.com? That's probably the host that a network dependent build needs to access most often.
Quoted 25 lines · On Jan 23, 2020, at 8:25 PM, Andre McCurdy [email protected] wrote:
Quoted 23 lines · Richard Purdie
Quoted 22 lines · Jean-Marie LEMETAYER
Quoted 20 lines · On Jan 23, 2020, at 3:37 PM, Richard Purdie [email protected]…
Quoted 7 lines · Jean-Marie LEMETAYER
I understand the issue with blocked domains this is why I suggest the
custom domain "connectivitycheck.openembedded.org".But I dont know if it could be an option. Can OE have a sub- domain
like that? maybe using some cdn to handle the load?We can certainly setup domains but we can't easily get a cdn. I'd be
worried about OE's response time too.Quoted 6 lines
Another option will be to provide multiple URIs:
CONNECTIVITY_CHECK_URIS ?= " \ https://google.com \ https://www.bing.com \ https://www.baidu.com \ "
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Looking at the code, I wonder if a
CONNECTIVITY_CHECK_MIRRORS, injected
into MIRRORS internally with google as the default and baidu as the mirror would work?How about checking github.com? That's probably the host that a network dependent build needs to access most often.
It is a good idea!
To go further I have created a new GitHub repository to host a simple index.html file using the GitHub Pages:
https://github.com/connectivitycheck/connectivitycheck.github.io
The URL of the page is here:
https://connectivitycheck.github.io
The results are good for me:
$ time ./connectivity_check.sh https://connectivitycheck.github.io 100 wget -qO- --timeout=1 --tries=1 https://connectivitycheck.github.io 100 curl -s --connect-timeout 1 https://connectivitycheck.github.io
real 0m17.202s
user 0m0.944s
sys 0m0.241s
Moreover github.com seems to be accessible in china. (checked with: https://www.websitepulse.com/tools/china-firewall-test)
Finally the same thing can be done using GitLab. This could make it possible to have mirrors.
Best regards,
Jean-Marie
Quoted 6 lines · Jean-Marie LEMETAYER
So "example.com" is clearly not a good domain to do connectivity checks.
The "openembedded.org" domain is good but have a slow response time.
And finally "google.com" which have all sort of speedy networking stuff is very efficient.
Oh my domestic ADSL in the UK I ran your script, example.com finished in 1 minute with 100/100. What internet connectivity to you have? The performance hit with the OE servers isn't ideal, and Google isn't suitable as has been said.
Note that if you're having problems with the check, a quick fix would be to just set CONNECTIVITY_CHECK_URIS="" in your distro to turn the check off entirely.
Ross
Quoted 5 lines · On Jan 23, 2020, at 2:56 PM, Ross Burton [email protected] wrote:
Quoted 4 lines · Jean-Marie LEMETAYER
So "example.com" is clearly not a good domain to do connectivity checks.
The "openembedded.org" domain is good but have a slow response time.
Quoted 7 lines
Quoted 2 lines
And finally "google.com" which have all sort of speedy networking stuff is very efficient.
Oh my domestic ADSL in the UK I ran your script, example.com finished in 1 minute with 100/100. What internet connectivity to you have? The performance hit with the OE servers isn't ideal, and Google isn't suitable as has been said.
I have a fiber connection in france ... and the example.com server seems to be in the US.
But my best score is 56% :(