Missing certificates
I was trying to run a simple fetch from python using
url = 'https://raw.github.com/Itseez/opencv/master/samples/c/fruits.jpg' filedata = urllib2.urlopen(url).read()
This failed:
Traceback (most recent call last):
File "./edge.py", line 36, in <module>
filedata = urllib2.urlopen(url).read()
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
context=self._context)
File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
I can see that it was looking for some certificates in /usr/lib/ssl/certs but that directory is missing.
Anyone know what I might be missing (or have misconfigured)?
Thanks
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Hi Gary,
What version of python do you use?.
Since 2.7.9 cert checking is enabled by default causing this kind of errors. [1]
[1] https://www.python.org/dev/peps/pep-0476/
Kind regards,
alimon
Quoted 30 lines · Gary Thomas
I was trying to run a simple fetch from python using
url =
'https://raw.github.com/Itseez/opencv/master/samples/c/fruits.jpg'
filedata = urllib2.urlopen(url).read()This failed:
Traceback (most recent call last):
File "./edge.py", line 36, in <module>
filedata = urllib2.urlopen(url).read()
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
returnopener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 431, in open
response =self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 449, in_open'_open', req)File "
/usr/lib/python2.7/urllib2.py", line 409, in_call_chain
result =func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1240, inhttps_open
context=self._context)
File "/usr/lib/python2.7/urllib2.py", line 1197, indo_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL:CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>I can see that it was looking for some certificates in
/usr/lib/ssl/certsbut that directory is missing.Anyone know what I might be missing (or have misconfigured)?
Thanks
Quoted 10 lines · Aníbal Limón
Hi Gary,
What version of python do you use?.
Since 2.7.9 cert checking is enabled by default causing this kind of errors. [1]
[1] https://www.python.org/dev/peps/pep-0476/
Kind regards,
alimon
I'm using the stock python 2.7.9 from Poky/Yocto master:901be2cb69892595443ed41ab4be285932db15eb
Is there an answer for this that's a bit less intrusive? Perhaps there could be a DISTRO or even IMAGE feature to enable/disable this checking?
The pep you referenced mostly talks about why this was changed and how to disable it - manually within the python code itself. What I don't see is where/how/what to change/import to actually let the full certificate checking happen.
Quoted 31 lines · Gary Thomas
Quoted 29 lines
I was trying to run a simple fetch from python using
url = 'https://raw.github.com/Itseez/opencv/master/samples/c/fruits.jpg' filedata = urllib2.urlopen(url).read()This failed:
Traceback (most recent call last):
File "./edge.py", line 36, in <module>
filedata = urllib2.urlopen(url).read()
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
returnopener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 431, in open
response =self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 449, in_open'_open', req)File "
/usr/lib/python2.7/urllib2.py", line 409, in_call_chain
result =func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1240, inhttps_open
context=self._context)
File "/usr/lib/python2.7/urllib2.py", line 1197, indo_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL:CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>I can see that it was looking for some certificates in
/usr/lib/ssl/certsbut that directory is missing.Anyone know what I might be missing (or have misconfigured)?
Thanks
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 9 lines · Gary Thomas
Quoted 8 lines · Aníbal Limón
Hi Gary,
What version of python do you use?.
Since 2.7.9 cert checking is enabled by default causing this kind of errors. [1]
Quoted 8 lines
Quoted 2 lines
Kind regards,
alimonI'm using the stock python 2.7.9 from Poky/Yocto master:901be2cb69892595443ed41ab4be285932db15eb
Is there an answer for this that's a bit less intrusive? Perhaps there could be a DISTRO or even IMAGE feature to enable/disable this checking?
I don't think that Python guys include a configuration flags to disable this behavior because it's the default now due to security issues.
Quoted 5 lines
The pep you referenced mostly talks about why this was changed and how to disable it - manually within the python code itself. What I don't see is where/how/what to change/import to actually let the full certificate checking happen.
You can use this code for disable per urlopen call or globally [1].
Regards,
alimon
[1] https://www.python.org/dev/peps/pep-0476/#opting-out
Quoted 34 lines
Quoted 33 lines · Gary Thomas
Quoted 30 lines
I was trying to run a simple fetch from python using
url =
'https://raw.github.com/Itseez/opencv/master/samples/c/fruits.jpg'
filedata = urllib2.urlopen(url).read()This failed:
Traceback (most recent call last):
File "./edge.py", line 36, in <module>
filedata = urllib2.urlopen(url).read()
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
returnopener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 431, in open
response =self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 449, in_open'_open', req)File "
/usr/lib/python2.7/urllib2.py", line 409, in_call_chain
result =func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1240, inhttps_open
context=self._context)
File "/usr/lib/python2.7/urllib2.py", line 1197, indo_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL:CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>I can see that it was looking for some certificates in
/usr/lib/ssl/certsbut that directory is missing.Anyone know what I might be missing (or have misconfigured)?
Thanks
Quoted 18 lines · Gary Thomas
Quoted 4 lines · Aníbal Limón
Hi Gary,
What version of python do you use?.
Quoted 8 lines
Since 2.7.9 cert checking is enabled by default causing this kind of errors. [1]
[1] https://www.python.org/dev/peps/pep-0476/
Kind regards,
alimonI'm using the stock python 2.7.9 from Poky/Yocto
master:901be2cb69892595443ed41ab4be285932db15eb
Quoted 8 lines
Is there an answer for this that's a bit less intrusive? Perhaps there could be a DISTRO or even IMAGE feature to enable/disable this checking?
The pep you referenced mostly talks about why this was changed and how to disable it - manually within the python code itself. What I don't see is where/how/what to change/import to actually let the full certificate checking happen.
I think the better bet is to fix it so it actually finds the certs from ca-certificates rather than bypassing certificate checking, personally, but I can see how that would be a useful workaround. :)
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 23 lines · Christopher Larson
On Fri, Jul 24, 2015 at 12:49 PM, Gary Thomas <[email protected] <mailto:[email protected]>> wrote:
On 2015-07-24 13:30, Aníbal Limón wrote:
Hi Gary,
What version of python do you use?.
Since 2.7.9 cert checking is enabled by default causing this kind of errors. [1]
[1] https://www.python.org/dev/peps/pep-0476/
Kind regards,
alimonI'm using the stock python 2.7.9 from Poky/Yocto master:901be2cb69892595443ed41ab4be285932db15eb
Is there an answer for this that's a bit less intrusive? Perhaps there could be a DISTRO or even IMAGE feature to enable/disable this checking?
Quoted 8 lines
The pep you referenced mostly talks about why this was changed and how to disable it - manually within the python code itself. What I don't see is where/how/what to change/import to actually let the full certificate checking happen.
I think the better bet is to fix it so it actually finds the certs from ca-certificates rather than bypassing certificate checking, personally, but I can see how that would be a useful workaround. :)
I tried this same code on my Ubuntu 15.04 desktop and it looks like they've disabled it in the main python http[s] code - there were no certificates examined during the transaction as far as I could tell (strace is my friend)
I'll see if I can figure out how to stitch this together with our [Poky/Yocto/OE-core] setup for OpenSSL and ca-certificates.
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 24 lines · Gary Thomas
I was trying to run a simple fetch from python using
url = 'https://raw.github.com/Itseez/opencv/master/samples/c/fruits.jpg'
filedata = urllib2.urlopen(url).read()This failed:
Traceback (most recent call last):
File "./edge.py", line 36, in <module>
filedata = urllib2.urlopen(url).read()
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
returnopener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 431, in open
response =self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 449, in_open'_open', req)File "
/usr/lib/python2.7/urllib2.py", line 409, in_call_chain
result =func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1240, inhttps_open
context=self._context)
File "/usr/lib/python2.7/urllib2.py", line 1197, indo_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL:CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
Quoted 5 lines
I can see that it was looking for some certificates in
/usr/lib/ssl/certsbut that directory is missing.Anyone know what I might be missing (or have misconfigured)?
At least in the past, we’ve been highly inconsistent in certificate store configuration in various recipes, some pointing at a dir, some a file, and the paths vary. I don’t know if anyone ever fixed that, however — this is from memory.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 20 lines · Gary Thomas
I was trying to run a simple fetch from python using
url = 'https://raw.github.com/Itseez/opencv/master/samples/c/fruits.jpg' filedata = urllib2.urlopen(url).read()This failed:
Traceback (most recent call last): File "./edge.py", line 36, in <module> filedata = urllib2.urlopen(url).read() File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 431, in open response = self._open(req, data) File "/usr/lib/python2.7/urllib2.py", line 449, in _open '_open', req) File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open context=self._context) File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open raise URLError(err)
Quoted 9 lines
urllib2.URLError: <urlopen error [SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>I can see that it was looking for some certificates in
/usr/lib/ssl/certsbut that directory is missing.Anyone know what I might be missing (or have misconfigured)?
Thanks
I've found a discussion about this problem on the OpenEmbedded development list:
http://lists.openembedded.org/pipermail/openembedded-devel/2015-July/102160.html
So the problem that this has uncovered is twofold:
1) Python (and OpenSSL) are not using the certificates that are installed by the ca-certificates package OpenSSL expects the certificates in /usr/lib/ssl/certs and ca-certificates uses /etc/ssl/certs
2) The certificates from ca-certificates are not immediately usable by OpenSSL since they are not hashed. This is done by the 'c_rehash' program but has been explicitly disabled by a patch. Further exploration implies that this was disabled because not all targets will have c_rehash available and since the hashing is expected to be done on the target when the certificates are loaded/updated. Finally, c_rehash, may or may not exist in the OpenSSL packages, depending on whether or not perl is available on the target (it's a perl script)
How best to solve this? As is, python htts:// support is broken in OE-core, so I think an off-the-shelf solution is warranted.
Perhaps the PACKAGECONFIG for openssl should default to supporting perl on the target, and hence the c_rehash utility would be available? Certainly the choice of where the certificates live, etc, should be standardized.
Maybe the c_rehash can be run at package build time for ca-certificates? This would make things work, at least for the real CA certificates.
Ideas?
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------