BMCWeb policy for HTTPS site identity certificate
This is a followup to the OpenBMC security working group meeting discussion on 2020-07-22 (https://docs.google.com/document/d/1b7x9BaxsfcukQDqbvZsU2ehMq4xoJRQvLxxsDUWmAOI).
Background:
Per [BMCWeb configuration policy](https://github.com/openbmc/bmcweb#configuration), BMCWeb generates a new HTTPS site identity certificate if a usable one cannot be found. You can upload one via APIs described here: https://github.com/openbmc/phosphor-dbus-interfaces/blob/master/xyz/openbmc_project/Certs/README.md#redfish-certificate-support
Problem:
BMCWeb apparently treats certificates that are either expired or not valid until a future date as unusable (investigation needed). And BMCWeb deletes unusable certificates. This can confuse the administrator, especially considering the BMC's time-of-day clock may not be set as expected.
Proposal:
What certificate management policy should BMCWeb use? Here is an initial proposal:
1. certificate is perfectly good - Use the certificate.
2. certificate is good but expired or not yet valid - Use the certificate and log a warning.
3. certificate is missing or bad format or algorithm too old - Use another certificate or self-generate a certificate (and log that action). In no case should BMCWeb should delete any certificate.
Discussion?
Quoted 10 lines · Joseph Reynolds
Problem:
BMCWeb apparently treats certificates that are either expired or not valid until a future date as unusable (investigation needed). And BMCWeb deletes unusable certificates. This can confuse the administrator, especially considering the BMC's time-of-day clock may not be set as expected.Proposal:
What certificate management policy should BMCWeb use? Here is an initial proposal:
1. certificate is perfectly good - Use the certificate.
okay.
Quoted 2 lines
2. certificate is good but expired or not yet valid - Use the certificate and log a warning.
very good.
Quoted 3 lines
3. certificate is missing or bad format or algorithm too old - Use another certificate or self-generate a certificate (and log that action). In no case should BMCWeb should delete any certificate.
I think that there is a problem in 3.
"certificate is missing" is pretty much unambiguous.
"bad format" depends a bit upon evolution of libraries.
In particular, a new version of libssl might support some new algorithm, and then should the firmware be rolled back, it will "bad format".
So I suggest that the certificate+keypair is never deleted, but may be renamed. I think that we could have a debate about getting telemetry about bad certificates back via HTTP.
I think that there are some operational considerations relating to determining root cause that may trump some security issues relating to telling bad actors whether they have succeeded in damaging a certificate.
--
] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works | IoT architect [ ] [email protected] http://www.sandelman.ca/ | ruby on rails [
Quoted 46 lines
-----Original Message-----
From: openbmc [mailto:openbmc- [email protected]] On Behalf Of Michael Richardson
Sent: Sunday, July 26, 2020 13:35
To: openbmc
Subject: Re: BMCWeb policy for HTTPS site identity certificateQuoted 2 lines · Joseph Reynolds
Problem:
BMCWeb apparently treats certificates that are either expired or notvalid
Quoted 1 line
until a future date as unusable (investigation needed). And BMCWeb
deletes
Quoted 5 lines
unusable certificates. This can confuse the administrator, especially considering the BMC's time-of-day clock may not be set as expected.
Proposal:
What certificate management policy should BMCWeb use? Here is aninitial
Quoted 2 lines
proposal:
1. certificate is perfectly good - Use the certificate.okay.
Quoted 1 line
2. certificate is good but expired or not yet valid - Use the certificate
and
Quoted 1 line
log a warning.
very good.
Quoted 1 line
3. certificate is missing or bad format or algorithm too old - Use
another
Quoted 2 lines
certificate or self-generate a certificate (and log that action). In no case should BMCWeb should delete any certificate.
I think that there is a problem in 3.
"certificate is missing" is pretty much unambiguous.
"bad format" depends a bit upon evolution of libraries.
In particular, a new version of libssl might support some new algorithm, and then should the firmware be rolled back, it will "bad format".
Quoted 8 lines
So I suggest that the certificate+keypair is never deleted, but may be renamed.
I think that we could have a debate about getting telemetry about bad certificates back via HTTP.I think that there are some operational considerations relating to determining root cause that may trump some security issues relating to telling bad actors whether they have succeeded in damaging a certificate.
One more thing is for 3 is that the incident must be logged.
Quoted 11 lines
--
] Never tell me the odds! | ipv6 mesh
networks [
] Michael Richardson, Sandelman Software Works | IoT
architect [
] [email protected] http://www.sandelman.ca/ | ruby on
rails [
Like I said in the other thread. The current behavior is a regression on what the bmcweb behavior was (and was designed to be).
Quoted 21 lines · Michael Richardson
Quoted 10 lines · Joseph Reynolds
Problem:
BMCWeb apparently treats certificates that are either expired or not valid until a future date as unusable (investigation needed). And BMCWeb deletes unusable certificates. This can confuse the administrator, especially considering the BMC's time-of-day clock may not be set as expected.Proposal:
What certificate management policy should BMCWeb use? Here is an initial proposal:
1. certificate is perfectly good - Use the certificate.okay.
Quoted 2 lines
2. certificate is good but expired or not yet valid - Use the certificate and log a warning.
very good.
Quoted 7 lines
Quoted 3 lines
3. certificate is missing or bad format or algorithm too old - Use another certificate or self-generate a certificate (and log that action). In no case should BMCWeb should delete any certificate.
I think that there is a problem in 3.
"certificate is missing" is pretty much unambiguous.
Unfortunately, this ambiguity comes with the territory. On first boot, bmcweb has no certificate, and doesn't know the difference between "missing" and "was never there". Regardless, to bring up TLS it needs _some_ certificate, so the original behavior was that it generated a new one in all cases where the existing one either didn't exist or couldn't be used. This also allows people to start bmcweb up in "developer" mode, by only sending the binary over, and is useful for doing A/B compares.
(note, an SSLContext can still be created with a certificate with bad dates or an unknown certificate chain)
Quoted 1 line
"bad format" depends a bit upon evolution of libraries.
Today this is defined as the above. "Can we use this certificate file for starting up an SSL context?" If the answer is no, we regenerate. In theory, the only library we rely on for this is OpenSSL, which I would hope doesn't have a backward incompatible evolution in this area.
Quoted 2 lines
In particular, a new version of libssl might support some new algorithm, and then should the firmware be rolled back, it will "bad format".
In this hypothetical, you're thinking about a new, non x509 certificate file format? I vote let's cross that bridge when we get there, as it seems like there's a lot more discussion that would need to happen around upgrades and downgrades. Today the assumption we make is that x509 certificate reading is backward and forward compatible since the begining of openbmc, which, to my knowledge, it is.
In this hypothetical, if x509 instituted a backward incompatible change AND previous OpenBMC instances were unable to read it, bmcweb would simply generate a new default certificate. I don't know if we've instituted a firmware rollback policy for that case, but I'm guessing it would be possible (but difficult to maintain).
Quoted 4 lines
So I suggest that the certificate+keypair is never deleted, but may be renamed. I think that we could have a debate about getting telemetry about bad certificates back via HTTP.
We can have a discussion, but I suspect a lot of people would be very against using unencrypted HTTP for this purpose.
Quoted 12 lines
I think that there are some operational considerations relating to determining root cause that may trump some security issues relating to telling bad actors whether they have succeeded in damaging a certificate.
--
] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works | IoT architect [ ] [email protected] http://www.sandelman.ca/ | ruby on rails [
Quoted 6 lines · Ed Tanous
Quoted 1 line
"certificate is missing" is pretty much unambiguous.
Unfortunately, this ambiguity comes with the territory. On first boot, bmcweb has no certificate, and doesn't know the difference between "missing" and "was never there". Regardless, to bring up TLS it needs _some_ certificate, so the original behavior was that it
This is reasonable behaviour, but given that browsers are trying very hard to make the certificate exception box go away, this does not really help long-term in my opinion.
Missing means: "ENOFILE", not "Can we use this certificate file for starting up an SSL Connect".
Quoted 7 lines
Quoted 1 line
"bad format" depends a bit upon evolution of libraries.
Today this is defined as the above. "Can we use this certificate file for starting up an SSL context?" If the answer is no, we regenerate. In theory, the only library we rely on for this is OpenSSL, which I would hope doesn't have a backward incompatible evolution in this area.
Yes, it does.
For instance, you can't load 1024-bit RSA keys with 1.1.1.
It refuses to start.
Meanwhile, 1.0.x does not have any ECDSA support, and you won't find this out until the TLS session actually tries to start, at which point, it logs an obsure message to stderr, and returns an error that most programs don't know what to do with.
(And the TCP connection just ends)
Quoted 5 lines
Quoted 2 lines
In particular, a new version of libssl might support some new algorithm, and then should the firmware be rolled back, it will "bad format".
In this hypothetical, you're thinking about a new, non x509 certificate file format? I vote let's cross that bridge when we get
Nope, not about non-X.509.
Algorithms and keysize changes.
Quoted 5 lines
there, as it seems like there's a lot more discussion that would need to happen around upgrades and downgrades. Today the assumption we make is that x509 certificate reading is backward and forward compatible since the begining of openbmc, which, to my knowledge, it is.
Until... it isn't.
But, the proposal would have considered a certificate with an invalid date as being invalid, and generated a new one.
Quoted 6 lines
Quoted 3 lines
So I suggest that the certificate+keypair is never deleted, but may be renamed. I think that we could have a debate about getting telemetry about bad certificates back via HTTP.
We can have a discussion, but I suspect a lot of people would be very against using unencrypted HTTP for this purpose.
I agree.
So, how do you get information at this point?
--
] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works | IoT architect [ ] [email protected] http://www.sandelman.ca/ | ruby on rails [
Quoted 5 lines · Michael Richardson
Quoted 1 line · Ed Tanous
Quoted 1 line
"certificate is missing" is pretty much unambiguous.
Quoted 8 lines
Quoted 4 lines
Unfortunately, this ambiguity comes with the territory. On first boot, bmcweb has no certificate, and doesn't know the difference between "missing" and "was never there". Regardless, to bring up TLS it needs _some_ certificate, so the original behavior was that it
This is reasonable behaviour, but given that browsers are trying very hard to make the certificate exception box go away, this does not really help long-term in my opinion.
I'd still be very surprised if this ever happened to browsers in the long term without any control server side. I get where their motivation is, and I agree with it in principle, but without some mechanism for initial embedded system provisioning, I don't know how you completely disable that bypass. With that said, I'm not a browser developer, so if it does happen, we'll need to figure out another way to handle initial boot and provisioning. If you have a proposal for how to handle this without self signed certs, it would be an interesting discussion to have.
Quoted 3 lines
Missing means: "ENOFILE", not "Can we use this certificate file for starting up an SSL Connect".
Today, that's not the definition bmcweb uses to determine whether to generate a new cert. I can't tell if you're proposing a different behavior here, or making a statement about current behavior. If the file is present but corrupt, or inaccessible due to permissions, what are you proposing the behavior should be?
Flash corruption does happen, and in that case, we need a way to bring up the (sometimes only) configuration interface in a way that is usable to exchange the certs with valid ones, even if it's sub-optimal for security.
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
"bad format" depends a bit upon evolution of libraries.
Today this is defined as the above. "Can we use this certificate file for starting up an SSL context?" If the answer is no, we regenerate. In theory, the only library we rely on for this is OpenSSL, which I would hope doesn't have a backward incompatible evolution in this area.
Quoted 3 lines
Yes, it does.
For instance, you can't load 1024-bit RSA keys with 1.1.1. It refuses to start.
When I get a free second, I'll look up where we landed on "should we allow 1k keys" discussion we had a long time back. I know we had talked about disallowing them and I think the conclusion was that we disallow them at upload time. With that said, maybe 2k keys fail to load in the future?
Quoted 1 line
Meanwhile, 1.0.x does not have any ECDSA support,
bmcweb has never targeted OpenSSL 1.0, and has always generated self signed EC keys so this shouldn't be an issue in practice, but your point about "could've broken us if" is well taken.
Quoted 5 lines
and you won't find this out
until the TLS session actually tries to start, at which point, it logs an obsure message to stderr, and returns an error that most programs don't know what to do with.
(And the TCP connection just ends)
I could've sworn that EVP_PKEY_get1_RSA returns NULL if it's an EC key (which is a call that bmcweb explicitly checks). That call is one of our "can we build an SSL context" checks today. Maybe OpenSSL 1.0 is different? Regardless, it's really hard to talk about backward compatibility with hypothetical openbmc + openssl 1.0 builds that to my knowledge have never existed. If this situation presents itself in the future on another OpenSSL upgrade, I suspect that is the best time to discuss it.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
In particular, a new version of libssl might support some new algorithm, and then should the firmware be rolled back, it will "bad format".
In this hypothetical, you're thinking about a new, non x509 certificate file format? I vote let's cross that bridge when we get
Nope, not about non-X.509.
Algorithms and keysize changes.
Agreed, there are possible changes that could break us in the future (if openssl stops accepting 2k keys for example).
Quoted 10 lines
Quoted 6 lines
there, as it seems like there's a lot more discussion that would need to happen around upgrades and downgrades. Today the assumption we make is that x509 certificate reading is backward and forward compatible since the begining of openbmc, which, to my knowledge, it is.
Until... it isn't.
But, the proposal would have considered a certificate with an invalid date as being invalid, and generated a new one.
Yes, I do not believe date, nor cert chain should be used under the definition of "valid"; "Can we use this certificate file for starting up an SSL context?" answers yes, even if the date and/or cert chain is invalid, so I think the definition still works.
With that said, I think all of the above is covered by general idea of "upgrades are guaranteed, downgrades are best effort" that most BMC implementations (including OpenBMC at this point) tend to take. (yes, sometimes we break the upgrade path and have to fix it). I don't think I've seen anywhere in the project where we include both a forward and backward path for nonvolatile schema migrations. Are you proposing something different we should do to handle these types of situations?
With all of this said, I'm open to the possibility that we have a backward incompatible openssl change that invalidates the cert. Do you think you could code up a patch with what you're hoping the behavior to be? It might be easier to approach it from a patchset.
Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
So I suggest that the certificate+keypair is never deleted, but may be renamed. I think that we could have a debate about getting telemetry about bad certificates back via HTTP.
We can have a discussion, but I suspect a lot of people would be very against using unencrypted HTTP for this purpose.
I agree.
So, how do you get information at this point?
I'm not following. At the point where you've downgraded, and your key is no longer valid? bmcweb will regenerate a self-signed one, and a user can connect to the HTTPS port insecurely. Hopefully their next step is to set up a valid key again, but I don't know how to force that on people. Is there a better behavior you'd like? Brick the system until it's factory reset?
Quoted 2 lines · Joseph Reynolds
2. certificate is good but expired or not yet valid - Use the certificate and log a warning.
I suspect that "not yet valid" is a more common case than might be assumed on the surface. I agree with the recommended action.
Many of the Facebook server designs do not have a hardware RTC available to the BMC. We have an RTC accessible by the BIOS and we also sync with NTP. That means there is always a period of time after we first plug in the rack where the servers in the rack have a date that is way wrong.
It is reasonable to assume the date is just wrong and the certificate is valid. The clients can validate a certificate which is actually out of date.
I'm less settled on using a certificate which is clearly expired, but it is still likely better than using a newly-generated self-signed certificate.
Signature · 1 line
Patrick Williams
Quoted 7 lines · Patrick Williams
Quoted 2 lines · Joseph Reynolds
2. certificate is good but expired or not yet valid - Use the certificate and log a warning.
I suspect that "not yet valid" is a more common case than might be assumed on the surface. I agree with the recommended action.
Quoted 8 lines
Many of the Facebook server designs do not have a hardware RTC available to the BMC. We have an RTC accessible by the BIOS and we also sync with NTP. That means there is always a period of time after we first plug in the rack where the servers in the rack have a date that is way wrong.
It is reasonable to assume the date is just wrong and the certificate is valid. The clients can validate a certificate which is actually out of date.
An additional design idea if you think you have no valid time, is to set the time to be the notBefore of the certificate you have. It's probably at least that date :-)
Quoted 3 lines
I'm less settled on using a certificate which is clearly expired, but it is still likely better than using a newly-generated self-signed certificate.
+1.
--
] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works | IoT architect [ ] [email protected] http://www.sandelman.ca/ | ruby on rails [
Quoted 4 lines · Michael Richardson
Quoted 4 lines
I'm less settled on using a certificate which is clearly expired, but it is still likely better than using a newly-generated self-signed certificate.
The original implementation just caught the X509_V_ERR_CERT_NOT_YET_VALID error and ignored it, but your idea would work as well.
One thing we had considered is requiring that the CERT date be at minimum AFTER the firmware build date, under the assumption that the build machine had a good grasp on what time it was at the time. We could use this for gating the upload of a new cert, but can't use it for invalidating a cert that already exists, as we run into the "upgrade causes denial of service" problem.