Skip to contentSkip to conversation

OpenBMC 2.8 security audit results

DISCUSSION Joseph Reynolds 2 participants 4 messages Jun 20, 2020 – Jun 23, 2020

lore
Share & tools
lore
#0 lore raw

Here are the results from my "security audit" on the planned OpenBMC 2.8 release.  The results are not intended as a complete analysis, but only offer highlights into the BMC's security configuration. Contributions are appreciated. The script to perform these tests was presented here: https://lists.ozlabs.org/pipermail/openbmc/2020-April/021186.html and was followed more-or-less.

- Joseph

__________

The image under test was built from 2.8.0-rc1 and necessarily customized for its target machine. This customization may change some behaviors away from project defaults.
The BMC had its out of box setup, with the password as 0penBmc.
The bmcweb HTTPS certificate was the self-signed cert generated by BMCWeb.
Commands were run one-by-one, and results cut&pasted; any errors in that process are mine.
Not all results are shown here.  Some of the results contain sensitive data; these are removed and marked as REDACTED.

$ echo "Collecting test host basic data"
$ echo "Checking test host openssl version"
Checking test host openssl version
$ openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017
$ echo "Testing: Ping the BMC"
observed: ping works
$ echo "Testing: Check for mDNS discovery service"
$ avahi-browse --all --ignore-local --resolve --terminate
note: the avahi client was not available
result: did not test the BMC's discovery service
$ echo "Testing: network (out of band) IPMI"
$ ipmitool_args="-H ${bmc} -I lanplus -C 17 -U ${bmcadminuser} -P ${bmcadminpassword}"
note: parameter is newly required `-C 17` because cipher suite 3 was removed
$ echo "Testing: List IPMI users"
$ ipmitool ${ipmitool_args} user list 1
ID  Name         Callin  Link Auth    IPMI Msg   Channel Priv Limit
1   root             false   true       true       ADMINISTRATOR
2   readonly         false   true       true       USER
3                    true    false      false      NO ACCESS
4                    true    false      false      NO ACCESS
5                    true    false      false      NO ACCESS
6                    true    false      false      NO ACCESS
7                    true    false      false      NO ACCESS
8                    true    false      false      NO ACCESS
9                    true    false      false      NO ACCESS
10                   true    false      false      NO ACCESS
11                   true    false      false      NO ACCESS
12                   true    false      false      NO ACCESS
13                   true    false      false      NO ACCESS
14                   true    false      false      NO ACCESS
15                   true    false      false      NO ACCESS

interpretation: The default root admin user and an additional readonly user are configured.

$ echo "Testing: Print IPMI network settings"
Testing: Print IPMI network settings
$ ipmitool ${ipmitool_args} lan print 1
Set in Progress         : Set Complete
Auth Type Support       :
Auth Type Enable        : Callback :
                        : User     :
                        : Operator :
                        : Admin    :
                        : OEM      :
IP Address Source       : Static Address
IP Address              : REDACTED
Subnet Mask             : REDACTED
MAC Address             : REDACTED
Default Gateway IP      : REDACTED
Default Gateway MAC     : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
RMCP+ Cipher Suites     : 17
Cipher Suite Priv Max   : aaaaaaaaaaaaaaa
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
Bad Password Threshold  : Not Available

observation: Only suite 17 is available; that is as intended

$ echo "Testing: Show IPMI supported cipher suites"
Testing: Show IPMI supported cipher suites
$ ipmitool ${ipmitool_args} channel getciphers ipmi 0x1
ID   IANA    Auth Alg        Integrity Alg   Confidentiality Alg
17   N/A     hmac_sha256     sha256_128      aes_cbc_128
$ ipmitool ${ipmitool_args} channel getciphers sol 0x1
ID   IANA    Auth Alg        Integrity Alg   Confidentiality Alg
17   N/A     hmac_sha256     sha256_128      aes_cbc_128

Access to the BMC via SSH

jrey@gfwa180:~ $ echo "Testing: security aspects of the SSH connection"
Testing: security aspects of the SSH connection

$ ssh -p ${bmcsshport} -vv ${bmcadminuser}@${bmc} echo "Hello OpenBMC"
OpenSSH_7.8p1, OpenSSL 1.0.1u  22 Sep 2016
...excerpted...snip...
debug1: Local version string SSH-2.0-OpenSSH_7.8
debug1: Remote protocol version 2.0, remote software version dropbear_2019.78
...snip...
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group14-sha256,[email protected]
debug2: host key algorithms: ssh-rsa
debug2: ciphers ctos: aes128-ctr,aes256-ctr
debug2: ciphers stoc: aes128-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha1,hmac-sha2-256
debug2: MACs stoc: hmac-sha1,hmac-sha2-256
debug2: compression ctos: [email protected],none
debug2: compression stoc: [email protected],none
...snip...

observation: The BMC offers its exact SSH server version.  That is not recommended.
observation: The BMC SSH server offers the algorithms shown above. Should we remove HMAC-SHA1?

$ echo "Testing: SSH SoL connection (port 2200)"
jrey@gfwa180:~ $ timeout 20s ssh -p 2200 ${bmcadminuser}@${bmc}
root@REDACTED's password:   <-- entered correct password for BMC root account
result: Access was granted when the correct BMC root account password was used.
result: Access was denied when an incorrect BMC root account password was used.
$ echo "Testing: HTTP response headers"
Testing: HTTP response headers
$ curl -k -D ${workdir}/http-response-headers -X GET https://${bmc}/redfish/v1
...the HTTP response body is not shown...
jrey@gfwa180:~ $ cat ${workdir}/http-response-headers
HTTP/1.1 200 OK
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: DENY
Pragma: no-cache
Cache-Control: no-Store,no-Cache
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'none'; img-src 'self' data:; font-src 'self'; style-src 'self'; script-src 'self'; connect-src 'self' wss:
Content-Type: application/json
Server: iBMC
Date: Fri, 19 Jun 2020 22:00:00 GMT
Content-Length: 1050

observation: the http response appear correct per https://github.com/openbmc/bmcweb/blob/master/include/security_headers_middleware.hpp

$ curl -k -v --tlsv1.1 -X GET https://${bmc}/redfish
* About to connect() to 9.41.167.159 port 443 (#0)
*   Trying 9.41.167.159...
* Connected to 9.41.167.159 (9.41.167.159) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* NSS error -12190 (SSL_ERROR_PROTOCOL_VERSION_ALERT)
* Peer reports incompatible or unsupported protocol version.
* Closing connection 0
curl: (35) Peer reports incompatible or unsupported protocol version.

observation: TLSv1.1 is disabled as intended

$ curl -k --tlsv1.2 -X GET https://${bmc}/redfish
{
  "v1": "/redfish/v1/"
}

observation: TLSv1.2 works as intended

echo "Testing: SSL renegotiation"

$ openssl s_client -connect ${bmc}:443
CONNECTED(00000003)
...snip...
---
R
RENEGOTIATING
140737353955216:error:14094153:SSL routines:ssl3_read_bytes:no renegotiation:s3_pkt.c:1481:

observation: Requesting renegotiation (the "R" above) failed.  See https://github.com/openbmc/openbmc/issues/3624

# This probes unauthenticated REST API access

$ curl -k -X GET https://${bmc}/ | gunzip
<!doctype html><html ng-app="app" ng-csp lang="en"><head><meta http-equiv="Content-Security-Policy"><meta charset="UTF-8"><title>OpenBMC</title><meta name="viewport" content="width=device-width,initial-scale=1"><base href="/"><link rel="shortcut icon" href="/favicon.ico"><link href="/app.css" rel="stylesheet"></head><body ng-style="dataService.bodyStyle" ng-attr-id="{{!dataService.showNavigation ? 'login': ''}}"><app-header ng-if="dataService.showNavigation" path="dataService.path"></app-header><app-navigation ng-if="dataService.showNavigation" path="dataService.path" show-navigation="dataService.showNavigation"></app-navigation><toast ng-if="dataService.showNavigation"></toast><main ng-view ng-class="{'content__container': dataService.showNavigation, 'login__wrapper': !dataService.showNavigation}"></main><script src="/app.bundle.js"></script></body></html>

curl -k -X GET https://${bmc}/redfish/
curl -k -X GET https://${bmc}/redfish/v1/
curl -k -X GET https://${bmc}/redfish/v1/JsonSchemas
note: results omitted

observation: Unauthenticated access to the URIs shown above is allowed, as intended

$ curl -k -X GET https://${bmc}/redfish/v1/Registries
Unauthorized
$ curl -k -X GET https://${bmc}/redfish/v1/SessionService
Unauthorized
$ curl -k -X GET https://${bmc}/redfish/v1/SessionService/Sessions
Unauthorized
$ curl -k -X GET https://${bmc}/redfish/v1/AccountService
Unauthorized
$ curl -k -X GET https://${bmc}/redfish/v1/AccountService/Accounts
Unauthorized

observation: Unauthenticated access to the URIs shown above is rejected, as intended

# Probe BMC security settings via Administrator REST API access : First create a login session:

$ curl --insecure -X POST -D headers.txt \
Quoted 2 lines

https://${bmc}:${bmc_https_port}/redfish/v1/SessionService/Sessions \
      -d '{"UserName":"'${bmcadminuser}'", "Password":"'${bmcadminpassword}'"}' | tee ${workdir}/results.txt

{
  "@odata.id": "/redfish/v1/SessionService/Sessions/lwpuEVQ1th", "@odata.type": "#Session.v1_0_2.Session", "Description": "Manager User Session",
  "Id": "REDACTED",
  "Name": "User Session",
  "UserName": "root"
}

$ authtok=$(grep "^X-Auth-Token: " headers.txt | cut -d' ' -f2 | tr -d '\r')
$ test -n "${authtok}" && echo "Got X-Auth-Token okay (in shell variable authtok)" || { echo "Failed to get X-Auth-Token" && false; }
Got X-Auth-Token okay (in shell variable authtok)
$ sessid=$(grep -e '"Id":' ${workdir}/results.txt | cut -d\" -f4)
$ curl -k -H "X-Auth-Token: ${authtok}" -X GET https://${bmc}/redfish/v1/AccountService
{
  "@odata.id": "/redfish/v1/AccountService",
  "@odata.type": "#AccountService.v1_5_0.AccountService",
  "AccountLockoutDuration": 300,
  "AccountLockoutThreshold": 5,
  "Accounts": {
    "@odata.id": "/redfish/v1/AccountService/Accounts"
  },
  "MaxPasswordLength": 20,
  "MinPasswordLength": 7,
  "Name": "Account Service",
  "Oem": {
    "OpenBMC": {
      "@odata.type": "#OemAccountService.v1_0_0.AccountService",
      "AuthMethods": {
        "BasicAuth": true,
        "Cookie": true,
        "SessionToken": true,
        "TLS": false,
        "XToken": true
      }
    }
  },
  "ServiceEnabled": true
  ...snip...not all fields are shown...
}

note: The AccountLockoutDuration and AccountLockoutThreshold values used for this test may be customized differently than the base OpenBMC image.  They appear correct.
observation: the value for MinPasswordLength does not seem to match the content of the BMC's Linux-PAM config file /etc/pam.d/common-password.  This should be investigated.

Probe BMC settings when logged in as root

# cat /etc/os-release
ID=openbmc-witherspoon
NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)"
VERSION="2.8.0-rc1"
VERSION_ID=2.8.0-rc1-0-g35a774200
PRETTY_NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) 2.8.0-rc1"
BUILD_ID="2.8.0-rc1"
...snip...

The intended release appears to appears to be installed on the BMC.

# cat /etc/shadow
root:$6$REDACTED:18425:0:99999:7:::
...snip...
readonly:$6$REDACTED:18431:0:99999:7:::

A user account called "readonly" is present.

# groups root
priv-admin web redfish ipmi root

note: This denotes the root user has Privilege Role "admin" and Group Roles as shown; these appear correct modulo https://github.com/openbmc/openbmc/issues/3643

observed: User account "readonly" cannot authenticate via ssh (because that requires the Administrator role).

observed: User "testuser" with Role=Administrator can authenticate via SSH.

When logged into the BMC via SSH as testuser:
testuser$ ls -la /home/root
drwxr-xr-x    2 root     root           384 Jun 18 00:00 .
drwxr-xr-x    5 root     root           368 Jun 20 00:23 ..
-rw-------    1 root     root         12437 Jun 20 00:24 .bash_history -rw-r-----    1 root     root           459 Jun 19 20:19 bmcweb_persistent_data.json

observation: Non-root user can list files in /home/root; that seems undesireable.  The files themselves are not readable.

testuser$ ls -l /etc
...snip...
-rw-r--r--    1 root     root           898 Jun 20 00:23 group ...snip...
-rw-------    1 root     root           208 Jun 20 00:23 ipmi_pass ...snip...
-rw-------    1 root     root             9 May 26 19:30 key_file ...snip...
drwxr-xr-x    1 root     root           248 Jun 19 22:42 pam.d -rw-r--r--    1 root     root          1164 Jun 20 00:23 passwd ...snip...
-r--------    1 root     root          1070 Jun 20 00:23 shadow ...snip...

observation: Files in /etc seem to be protected properly.  See CVE-2020-14156 here: https://github.com/openbmc/openbmc/issues/3670

testuser$ ls -lR /etc/ssl
/etc/ssl:
drwxr-xr-x    4 root     root           296 Jun 10 06:22 certs -rw-r--r--    1 root     root         10909 May 26 18:48 openssl.cnf

/etc/ssl/certs:
drwx------    2 root     root           160 Jun 10 06:22 authority drwx------    2 root     root           304 Jun 10 06:23 https

observation: Certificates under /etc/ssl are protected from reading

#1 lore raw
Quoted 8 lines · Joseph Reynolds

Here are the results from my "security audit" on the planned OpenBMC 2.8 release.  The results are not intended as a complete analysis, but only offer highlights into the BMC's security configuration. Contributions are appreciated. The script to perform these tests was presented here: https://lists.ozlabs.org/pipermail/openbmc/2020-April/021186.html and was followed more-or-less.

$ echo "Checking test host openssl version"
Checking test host openssl version
$ openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

I'm not sure I get this one - is "test host" a BMC or the one where the test script is being run? If the former, this is really old, no longer publicly supported by the OpenSSL team and has multiple CVEs against it [1], so should be upgraded.

[1] https://www.openssl.org/news/openssl-1.0.2-notes.html

Quoted 1 line

debug1: Remote protocol version 2.0, remote software version dropbear_2019.78

There's a very recent 2020.79, which has one CVE fix and some useful changes (e.g. using getrandom(), AES-GCM and so on), would be good to update it for the next release.

Quoted 1 line

observation: The BMC SSH server offers the algorithms shown above. Should we remove HMAC-SHA1?

While it's not [yet] formally broken in the HMAC setting, IMO SHA1 is "broken enough" to remove it everywhere, so yes, I'd vote for that.

Quoted 10 lines

When logged into the BMC via SSH as testuser:
testuser$ ls -la /home/root
drwxr-xr-x    2 root     root           384 Jun 18 00:00 . drwxr-xr-x    5 root     root           368 Jun 20 00:23 ..

(quoted diff) +0 -2
-rw-------    1 root     root         12437 Jun 20 00:24 .bash_history
-rw-r-----    1 root     root           459 Jun 19 20:19 

bmcweb_persistent_data.json

observation: Non-root user can list files in /home/root; that seems undesireable.  The files themselves are not readable.

Agree, it doesn't seem necessary, so should be restricted

Quoted 5 lines

/etc/ssl/certs:
drwx------    2 root     root           160 Jun 10 06:22 authority drwx------    2 root     root           304 Jun 10 06:23 https

observation: Certificates under /etc/ssl are protected from reading

This actually seems to be surplus - *certificates* are public by definition, it's the private parts of them (private keys corresponding to public ones in certificates) that need protection like that.

regards,
Alexander

#2 lore raw
Quoted 4 lines · Alexander Tereschenko
Quoted 3 lines · Joseph Reynolds

Here are the results from my "security audit" on the planned OpenBMC 2.8 release.  The results are not intended as a complete analysis, but only offer highlights into the BMC's security configuration. Contributions are appreciated. The script to perform these tests was presented here: https://lists.ozlabs.org/pipermail/openbmc/2020-April/021186.html and was followed more-or-less.

Quoted 8 lines
Quoted 5 lines
$ echo "Checking test host openssl version"
Checking test host openssl version
$ openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

I'm not sure I get this one - is "test host" a BMC or the one where the test script is being run? If the former, this is really old, no longer publicly supported by the OpenSSL team and has multiple CVEs against it [1], so should be upgraded.

[1] https://www.openssl.org/news/openssl-1.0.2-notes.html

Alexander, thanks for your reply.  The "test host" is a Linux system used to probe the BMC via network interfaces such as SSH, HTTPS, and REST APIs.  To reflect actual customer use, I used a test host that has an older operating system.  I've included the test host's software versions to help answer questions about the results below, when the host version factors into the results.  [I'll update my preamble with this information.]

Quoted 5 lines
Quoted 3 lines

debug1: Remote protocol version 2.0, remote software version dropbear_2019.78

There's a very recent 2020.79, which has one CVE fix and some useful changes (e.g. using getrandom(), AES-GCM and so on), would be good to update it for the next release.

Agreed.  Yocto will pull in the new dropbear release, and it will downstream to OpenBMC.  We may want to update our dropbear/options.patch

https://github.com/openbmc/openbmc/blob/master/meta-phosphor/recipes-core/dropbear/dropbear/options.patch

Quoted 21 lines
Quoted 3 lines

observation: The BMC SSH server offers the algorithms shown above. Should we remove HMAC-SHA1?

While it's not [yet] formally broken in the HMAC setting, IMO SHA1 is "broken enough" to remove it everywhere, so yes, I'd vote for that.

Quoted 10 lines

When logged into the BMC via SSH as testuser:
testuser$ ls -la /home/root
drwxr-xr-x    2 root     root           384 Jun 18 00:00 . drwxr-xr-x    5 root     root           368 Jun 20 00:23 ..

(quoted diff) +0 -2
-rw-------    1 root     root         12437 Jun 20 00:24 .bash_history
-rw-r-----    1 root     root           459 Jun 19 20:19 

bmcweb_persistent_data.json

observation: Non-root user can list files in /home/root; that seems undesireable.  The files themselves are not readable.

Agree, it doesn't seem necessary, so should be restricted

Quoted 7 lines
Quoted 5 lines

/etc/ssl/certs:
drwx------    2 root     root           160 Jun 10 06:22 authority drwx------    2 root     root           304 Jun 10 06:23 https

observation: Certificates under /etc/ssl are protected from reading

This actually seems to be surplus - *certificates* are public by definition, it's the private parts of them (private keys corresponding to public ones in certificates) that need protection like that.

Thanks for the clarification.  I've heard a private certificate is improperly being stored along with its public cert in there somewhere, but I don't really know.

Quoted 5 lines

regards,
Alexander

#3 lore raw
Quoted 10 lines · Joseph Reynolds
Quoted 9 lines · Alexander Tereschenko
Quoted 8 lines · Joseph Reynolds

Here are the results from my "security audit" on the planned OpenBMC 2.8 release.  The results are not intended as a complete analysis, but only offer highlights into the BMC's security configuration. Contributions are appreciated. The script to perform these tests was presented here: https://lists.ozlabs.org/pipermail/openbmc/2020-April/021186.html and was followed more-or-less.

$ echo "Checking test host openssl version"
Checking test host openssl version
$ openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017
Quoted 5 lines
Quoted 3 lines

I'm not sure I get this one - is "test host" a BMC or the one where the test script is being run? If the former, this is really old, no longer publicly supported by the OpenSSL team and has multiple CVEs against it [1], so should be upgraded.

[1] https://www.openssl.org/news/openssl-1.0.2-notes.html

Alexander, thanks for your reply.  The "test host" is a Linux system used to probe the BMC via network interfaces such as SSH, HTTPS, and REST APIs.  To reflect actual customer use, I used a test host that has an older operating system.  I've included the test host's software versions to help answer questions about the results below, when the host version factors into the results. [I'll update my preamble with this information.]

Got it, in this case that comment is not important, the text host may have whatever versions :)

Quoted 9 lines
Quoted 7 lines
Quoted 5 lines

/etc/ssl/certs:
drwx------    2 root     root           160 Jun 10 06:22 authority drwx------    2 root     root           304 Jun 10 06:23 https

observation: Certificates under /etc/ssl are protected from reading

This actually seems to be surplus - *certificates* are public by definition, it's the private parts of them (private keys corresponding to public ones in certificates) that need protection like that.

Thanks for the clarification.  I've heard a private certificate is improperly being stored along with its public cert in there somewhere, but I don't really know.

Ah, indeed, that reminds me - OpenBMC stores both private key and the cert (with the public key) in one file, concatenated. That's how bmcweb consumes that (unlike e.g. Apache or nginx, which have separate settings/paths for private and public keys, though nginx also allows for combining) and indeed that in turn necessitates closing down the respective files/dirs - so your check is correct and my comment is not applicable.

regards,
Alexander