Skip to contentSkip to conversation

Connectivity check uris

DISCUSSION Jean-Marie LEMETAYER 5 participants 11 messages Jan 23, 2020 – Jan 24, 2020

lore
Share & tools
lore
#0 lore raw

Hi folks,

I have noticed some hang-ups at the beginning of my builds on the master branch. I have search a little and discovered that the connectivity check made by poky was the root cause. In fact connectivity check use the CONNECTIVITY_CHECK_URIS variable which is by default set to "https://www.example.com/".

So I have made some tests with a little script which gets the URI 100 times:

	$ cat connectivity_check.sh
	#!/bin/sh
	connectivity_check()
	{
		local ok=0
		for i in $(seq 100)
		do
			$@ > /dev/null && ok=$(($ok + 1))
		done
		echo $ok $@
	}
	connectivity_check wget -qO- --timeout=1 --tries=1 $1
	connectivity_check curl -s --connect-timeout 1 $1

Note that in the script, I use wget and curl to see if there are any differences but I don't think that's a thing.

Here are the results:

	$ time ./connectivity_check.sh https://example.com
	45 wget -qO- --timeout=1 --tries=1 https://example.com
	45 curl -s --connect-timeout 1 https://example.com
	real	2m23.595s
	user	0m3.352s
	sys	0m1.834s

	$ time ./connectivity_check.sh https://openembedded.org
	100 wget -qO- --timeout=1 --tries=1 https://openembedded.org
	100 curl -s --connect-timeout 1 https://openembedded.org

	real	4m36.188s
	user	0m3.853s
	sys	0m1.761s

	$ time ./connectivity_check.sh https://google.com
	100 wget -qO- --timeout=1 --tries=1 https://google.com
	100 curl -s --connect-timeout 1 https://google.com

	real	0m36.328s
	user	0m4.265s
	sys	0m2.028s

So "example.com" is clearly not a good domain to do connectivity checks.

The "openembedded.org" domain is good but have a slow response time.

And finally "google.com" which have all sort of speedy networking stuff is very efficient.

---

I think the CONNECTIVITY_CHECK_URIS variable must be updated and not use the "example.com" domain anymore. Using "google.com" may be the best option. But eetting up a custom domain like "connectivitycheck.openembedded.org" could be another idea.

Any ideas ?

Jean-Marie LEMETAYER
Passionate embedded systems engineer
Savoir-faire Linux

#1 lore raw
Quoted 59 lines · Jean-Marie LEMETAYER

Hi folks,

I have noticed some hang-ups at the beginning of my builds on the master branch. I have search a little and discovered that the connectivity check made by poky was the root cause. In fact connectivity check use the CONNECTIVITY_CHECK_URIS variable which is by default set to "https://www.example.com/".

So I have made some tests with a little script which gets the URI 100 times:

        $ cat connectivity_check.sh
        #!/bin/sh
        connectivity_check()
        {
                local ok=0
                for i in $(seq 100)
                do
                        $@ > /dev/null && ok=$(($ok + 1))
                done
                echo $ok $@
        }
        connectivity_check wget -qO- --timeout=1 --tries=1 $1
        connectivity_check curl -s --connect-timeout 1 $1

Note that in the script, I use wget and curl to see if there are any differences but I don't think that's a thing.

Here are the results:

        $ time ./connectivity_check.sh https://example.com
        45 wget -qO- --timeout=1 --tries=1 https://example.com
        45 curl -s --connect-timeout 1 https://example.com
        real    2m23.595s
        user    0m3.352s
        sys     0m1.834s
        $ time ./connectivity_check.sh https://openembedded.org
        100 wget -qO- --timeout=1 --tries=1 https://openembedded.org
        100 curl -s --connect-timeout 1 https://openembedded.org
        real    4m36.188s
        user    0m3.853s
        sys     0m1.761s
        $ time ./connectivity_check.sh https://google.com
        100 wget -qO- --timeout=1 --tries=1 https://google.com
        100 curl -s --connect-timeout 1 https://google.com
        real    0m36.328s
        user    0m4.265s
        sys     0m2.028s

So "example.com" is clearly not a good domain to do connectivity checks.

The "openembedded.org" domain is good but have a slow response time.

And finally "google.com" which have all sort of speedy networking stuff is very efficient.

Quoted 6 lines

---

I think the CONNECTIVITY_CHECK_URIS variable must be updated and not use the "example.com" domain anymore. Using "google.com" may be the best option. But eetting up a custom domain like "connectivitycheck.openembedded.org" could be another idea.

Had this idea long time in the past... Unfortunately, google.com is not available in all countries worldwide (Say "Nihao" to chinese Great firewall ).

Quoted 11 lines

Any ideas ?

Jean-Marie LEMETAYER
Passionate embedded systems engineer
Savoir-faire Linux
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Signature · 2 lines
Regards,
Andrey.
#2 lore raw
In reply to Jean-Marie LEMETAYER
Quoted 75 lines · Jean-Marie LEMETAYER

Hi folks,

I have noticed some hang-ups at the beginning of my builds on the master branch.
I have search a little and discovered that the connectivity check made by poky
was the root cause. In fact connectivity check use the CONNECTIVITY_CHECK_URIS
variable which is by default set to "https://www.example.com/".

So I have made some tests with a little script which gets the URI 100 times:

	$ cat connectivity_check.sh
	#!/bin/sh
	connectivity_check()
	{
		local ok=0
		for i in $(seq 100)
		do
			$@ > /dev/null && ok=$(($ok + 1))
		done
		echo $ok $@
	}
	connectivity_check wget -qO- --timeout=1 --tries=1 $1
	connectivity_check curl -s --connect-timeout 1 $1

Note that in the script, I use wget and curl to see if there are any differences
but I don't think that's a thing.

Here are the results:

	$ time ./connectivity_check.sh https://example.com
	45 wget -qO- --timeout=1 --tries=1 https://example.com
	45 curl -s --connect-timeout 1 https://example.com
	real	2m23.595s
	user	0m3.352s
	sys	0m1.834s

	$ time ./connectivity_check.sh https://openembedded.org
	100 wget -qO- --timeout=1 --tries=1 https://openembedded.org
	100 curl -s --connect-timeout 1 https://openembedded.org

	real	4m36.188s
	user	0m3.853s
	sys	0m1.761s

	$ time ./connectivity_check.sh https://google.com
	100 wget -qO- --timeout=1 --tries=1 https://google.com
	100 curl -s --connect-timeout 1 https://google.com

	real	0m36.328s
	user	0m4.265s
	sys	0m2.028s

So "example.com" is clearly not a good domain to do connectivity checks.

The "openembedded.org" domain is good but have a slow response time.

And finally "google.com" which have all sort of speedy networking stuff is very
efficient.

---

I think the CONNECTIVITY_CHECK_URIS variable must be updated and not use the
"example.com" domain anymore. Using "google.com" may be the best option. But
eetting up a custom domain like "connectivitycheck.openembedded.org" could be
another idea.

Quoted 1 line

Any ideas

I think if you look at the history we did use google.com however that caused problems for people in geos where that is blocked (e.g. PRC).

We may want to move the default back given the performance isse, not sure...

Cheers,

Richard

#3 lore raw
Quoted 76 lines · On Jan 23, 2020, at 2:26 PM, Richard Purdie [email protected]…
Quoted 75 lines · Jean-Marie LEMETAYER

Hi folks,

I have noticed some hang-ups at the beginning of my builds on the master branch.
I have search a little and discovered that the connectivity check made by poky
was the root cause. In fact connectivity check use the CONNECTIVITY_CHECK_URIS
variable which is by default set to "https://www.example.com/".

So I have made some tests with a little script which gets the URI 100 times:

	$ cat connectivity_check.sh
	#!/bin/sh
	connectivity_check()
	{
		local ok=0
		for i in $(seq 100)
		do
			$@ > /dev/null && ok=$(($ok + 1))
		done
		echo $ok $@
	}
	connectivity_check wget -qO- --timeout=1 --tries=1 $1
	connectivity_check curl -s --connect-timeout 1 $1

Note that in the script, I use wget and curl to see if there are any differences
but I don't think that's a thing.

Here are the results:

	$ time ./connectivity_check.sh https://example.com
	45 wget -qO- --timeout=1 --tries=1 https://example.com
	45 curl -s --connect-timeout 1 https://example.com
	real	2m23.595s
	user	0m3.352s
	sys	0m1.834s

	$ time ./connectivity_check.sh https://openembedded.org
	100 wget -qO- --timeout=1 --tries=1 https://openembedded.org
	100 curl -s --connect-timeout 1 https://openembedded.org

	real	4m36.188s
	user	0m3.853s
	sys	0m1.761s

	$ time ./connectivity_check.sh https://google.com
	100 wget -qO- --timeout=1 --tries=1 https://google.com
	100 curl -s --connect-timeout 1 https://google.com

	real	0m36.328s
	user	0m4.265s
	sys	0m2.028s

So "example.com" is clearly not a good domain to do connectivity checks.

The "openembedded.org" domain is good but have a slow response time.

And finally "google.com" which have all sort of speedy networking stuff is very
efficient.

---

I think the CONNECTIVITY_CHECK_URIS variable must be updated and not use the
"example.com" domain anymore. Using "google.com" may be the best option. But
eetting up a custom domain like "connectivitycheck.openembedded.org" could be
another idea.

Quoted 7 lines
Quoted 1 line

Any ideas

I think if you look at the history we did use google.com however that caused problems for people in geos where that is blocked (e.g. PRC).

We may want to move the default back given the performance isse, not sure...

I understand the issue with blocked domains this is why I suggest the custom domain "connectivitycheck.openembedded.org".

But I dont know if it could be an option. Can OE have a sub-domain like that? maybe using some cdn to handle the load?

Another option will be to provide multiple URIs:

    CONNECTIVITY_CHECK_URIS ?= " \
        https://google.com \
        https://www.bing.com \
        https://www.baidu.com \
    "

Best regards,
Jean-Marie

#4 lore raw
Quoted 5 lines · Jean-Marie LEMETAYER

I understand the issue with blocked domains this is why I suggest the custom domain "connectivitycheck.openembedded.org".

But I dont know if it could be an option. Can OE have a sub-domain like that? maybe using some cdn to handle the load?

We can certainly setup domains but we can't easily get a cdn. I'd be worried about OE's response time too.

Quoted 6 lines

Another option will be to provide multiple URIs:

    CONNECTIVITY_CHECK_URIS ?= " \
        https://google.com \
        https://www.bing.com \
        https://www.baidu.com \
    "

Looking at the code, I wonder if a CONNECTIVITY_CHECK_MIRRORS, injected into MIRRORS internally with google as the default and baidu as the mirror would work?

Cheers,

Richard

#5 lore raw
Quoted 17 lines · On Jan 23, 2020, at 3:37 PM, Richard Purdie [email protected]…
Quoted 5 lines · Jean-Marie LEMETAYER

I understand the issue with blocked domains this is why I suggest the custom domain "connectivitycheck.openembedded.org".

But I dont know if it could be an option. Can OE have a sub-domain like that? maybe using some cdn to handle the load?

We can certainly setup domains but we can't easily get a cdn. I'd be worried about OE's response time too.

Quoted 6 lines

Another option will be to provide multiple URIs:

    CONNECTIVITY_CHECK_URIS ?= " \
        https://google.com \
        https://www.bing.com \
        https://www.baidu.com \
    "
Quoted 3 lines

Looking at the code, I wonder if a CONNECTIVITY_CHECK_MIRRORS, injected into MIRRORS internally with google as the default and baidu as the mirror would work?

It should work but it will not be efficient as the tries are serially executed. So someone in China should have to wait that the former domains failed to be able to start the build.

I am still thinking that the OE subdomain with CDN could be the best option.

Regards,
Jean-Marie

#6 lore raw
Quoted 22 lines · Jean-Marie LEMETAYER
Quoted 20 lines · On Jan 23, 2020, at 3:37 PM, Richard Purdie [email protected]…
Quoted 7 lines · Jean-Marie LEMETAYER

I understand the issue with blocked domains this is why I suggest the
custom domain "connectivitycheck.openembedded.org".

But I dont know if it could be an option. Can OE have a sub- domain
like that? maybe using some cdn to handle the load?

We can certainly setup domains but we can't easily get a cdn. I'd be
worried about OE's response time too.

Quoted 6 lines

Another option will be to provide multiple URIs:

    CONNECTIVITY_CHECK_URIS ?= " \
        https://google.com \
        https://www.bing.com \
        https://www.baidu.com \
    "
Quoted 8 lines
Quoted 4 lines

Looking at the code, I wonder if a CONNECTIVITY_CHECK_MIRRORS, injected
into MIRRORS internally with google as the default and baidu as the mirror would work?

It should work but it will not be efficient as the tries are serially executed. So someone in China should have to wait that the former domains failed to be able to start the build.

I'm guessing those wouldn't resolve so it should be pretty fast?

Maybe we make the configuration more obvious in local.conf?

Quoted 2 lines

I am still thinking that the OE subdomain with CDN could be the best option.

Sure, but who is going to pay for and maintain a CDN setup?

Cheers,

Richard

#7 lore raw
Quoted 23 lines · Richard Purdie
Quoted 22 lines · Jean-Marie LEMETAYER
Quoted 20 lines · On Jan 23, 2020, at 3:37 PM, Richard Purdie [email protected]…
Quoted 7 lines · Jean-Marie LEMETAYER

I understand the issue with blocked domains this is why I suggest the
custom domain "connectivitycheck.openembedded.org".

But I dont know if it could be an option. Can OE have a sub- domain
like that? maybe using some cdn to handle the load?

We can certainly setup domains but we can't easily get a cdn. I'd be
worried about OE's response time too.

Quoted 6 lines

Another option will be to provide multiple URIs:

    CONNECTIVITY_CHECK_URIS ?= " \
        https://google.com \
        https://www.bing.com \
        https://www.baidu.com \
    "
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Looking at the code, I wonder if a CONNECTIVITY_CHECK_MIRRORS, injected
into MIRRORS internally with google as the default and baidu as the mirror would work?

How about checking github.com? That's probably the host that a network dependent build needs to access most often.

#8 lore raw
Quoted 25 lines · On Jan 23, 2020, at 8:25 PM, Andre McCurdy [email protected] wrote:
Quoted 23 lines · Richard Purdie
Quoted 22 lines · Jean-Marie LEMETAYER
Quoted 20 lines · On Jan 23, 2020, at 3:37 PM, Richard Purdie [email protected]…
Quoted 7 lines · Jean-Marie LEMETAYER

I understand the issue with blocked domains this is why I suggest the
custom domain "connectivitycheck.openembedded.org".

But I dont know if it could be an option. Can OE have a sub- domain
like that? maybe using some cdn to handle the load?

We can certainly setup domains but we can't easily get a cdn. I'd be
worried about OE's response time too.

Quoted 6 lines

Another option will be to provide multiple URIs:

    CONNECTIVITY_CHECK_URIS ?= " \
        https://google.com \
        https://www.bing.com \
        https://www.baidu.com \
    "
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Looking at the code, I wonder if a CONNECTIVITY_CHECK_MIRRORS, injected
into MIRRORS internally with google as the default and baidu as the mirror would work?

How about checking github.com? That's probably the host that a network dependent build needs to access most often.

It is a good idea!

To go further I have created a new GitHub repository to host a simple index.html file using the GitHub Pages:
https://github.com/connectivitycheck/connectivitycheck.github.io

The URL of the page is here:
https://connectivitycheck.github.io

The results are good for me:

$ time ./connectivity_check.sh https://connectivitycheck.github.io
100 wget -qO- --timeout=1 --tries=1 https://connectivitycheck.github.io
100 curl -s --connect-timeout 1 https://connectivitycheck.github.io

real 0m17.202s
user 0m0.944s
sys 0m0.241s

Moreover github.com seems to be accessible in china. (checked with: https://www.websitepulse.com/tools/china-firewall-test)

Finally the same thing can be done using GitLab. This could make it possible to have mirrors.

Best regards,
Jean-Marie

#9 lore raw
In reply to Jean-Marie LEMETAYER
Quoted 6 lines · Jean-Marie LEMETAYER

So "example.com" is clearly not a good domain to do connectivity checks.

The "openembedded.org" domain is good but have a slow response time.

And finally "google.com" which have all sort of speedy networking stuff is very efficient.

Oh my domestic ADSL in the UK I ran your script, example.com finished in 1 minute with 100/100. What internet connectivity to you have? The performance hit with the OE servers isn't ideal, and Google isn't suitable as has been said.

Note that if you're having problems with the check, a quick fix would be to just set CONNECTIVITY_CHECK_URIS="" in your distro to turn the check off entirely.

Ross

#10 lore raw
Quoted 5 lines · On Jan 23, 2020, at 2:56 PM, Ross Burton [email protected] wrote:
Quoted 4 lines · Jean-Marie LEMETAYER

So "example.com" is clearly not a good domain to do connectivity checks.

The "openembedded.org" domain is good but have a slow response time.

Quoted 7 lines
Quoted 2 lines

And finally "google.com" which have all sort of speedy networking stuff is very efficient.

Oh my domestic ADSL in the UK I ran your script, example.com finished in 1 minute with 100/100. What internet connectivity to you have? The performance hit with the OE servers isn't ideal, and Google isn't suitable as has been said.

I have a fiber connection in france ... and the example.com server seems to be in the US.

But my best score is 56% :(