cxl: Fix endpoint access issues with CXL MCE notifier handler
Subsystems (2)
Dave Jiang
maintainercover cxl: Fix endpoint access issues with CXL MCE notifier handler
Fix the issues reported by Flavien to the Linux security mailing list.
First patch addresses accessing possible invalid pointers for cxlmd and cxmd->endpoint.
The second patch adddresses the endpoint lifetime that is shorter than mds and needs appropriate guardrails to access.
Dave Jiang (2):
cxl/mce: Validate memdev and endpoint before dereference in
cxl_handle_mce()
cxl/mce: Serialize the MCE handler against endpoint teardown
drivers/cxl/core/mce.c | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-)
base-commit: 8cd9520d35a6c38db6567e97dd93b1f11f185dc6
Signature · 1 line
2.54.0
Dave Jiang
maintainerPATCH 1/2 cxl/mce: Validate memdev and endpoint before dereference in cxl_handle_mce()
cxlmd and endpoint are both used in cxl_handle_mce() without proper validation, which can lead to NULL pointer dereference or invalid pointer dereference. The notifier is registered in cxl_memdev_state_create() when the CXL PCI driver first binds, before the memdev is published and before it is attached to a CXL topology.
Add checks to cxlmd and endpoint to ensure they are valid before usage.
--- drivers/cxl/core/mce.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/cxl/core/mce.c +6 -2
@@ -13,7 +13,7 @@ static int cxl_handle_mce(struct notifier_block *nb, unsigned long val, struct cxl_memdev_state *mds = container_of(nb, struct cxl_memdev_state, mce_notifier);struct cxl_memdev *cxlmd = mds->cxlds.cxlmd; -struct cxl_port *endpoint = cxlmd->endpoint; +struct cxl_port *endpoint;struct mce *mce = data;u64 spa, spa_alias;unsigned long pfn;
@@ -21,7 +21,11 @@ static int cxl_handle_mce(struct notifier_block *nb, unsigned long val,if (!mce || !mce_usable_address (mce))return NOTIFY_DONE ; -if (!endpoint) +if (!cxlmd) +return NOTIFY_DONE ; + + endpoint = cxlmd->endpoint; +if (IS_ERR_OR_NULL (endpoint))return NOTIFY_DONE ; spa = mce->addr &MCI_ADDR_PHYSADDR ;
Signature · 1 line
2.54.0
Dave Jiang
maintainerPATCH 2/2 cxl/mce: Serialize the MCE handler against endpoint teardown
In reply to Dave JiangCXL endpoint has a shorter lifetime than CXL memdev state (mds) and the MCE notifier is part of the mds. The MCE handler needs to take a reference on the endpoint in order to keep it alive while operating on it. Take the cxlmd lock to verify the endpoint is still valid and take a reference on it before accessing it.
--- drivers/cxl/core/mce.c | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-)
drivers/cxl/core/mce.c +23 -4
@@ -7,13 +7,27 @@#include <cxlmem.h> #include "mce.h" +static struct device *cxlmd_get_endpoint_dev (struct cxl_memdev *cxlmd) +{ +struct cxl_port *endpoint; + +if (!cxlmd) +return NULL ; + +guard (device)(&cxlmd->dev); + endpoint = cxlmd->endpoint; +if (IS_ERR_OR_NULL (endpoint)) +return NULL ; + +return get_device (&endpoint->dev); +} +static int cxl_handle_mce (struct notifier_block *nb,unsigned long val,void *data) { struct cxl_memdev_state *mds = container_of(nb, struct cxl_memdev_state, mce_notifier);struct cxl_memdev *cxlmd = mds->cxlds.cxlmd; -struct cxl_port *endpoint;struct mce *mce = data;u64 spa, spa_alias;unsigned long pfn;
@@ -24,8 +38,13 @@ static int cxl_handle_mce(struct notifier_block *nb, unsigned long val,if (!cxlmd)return NOTIFY_DONE ; - endpoint = cxlmd->endpoint; -if (IS_ERR_OR_NULL (endpoint)) +/* +* With the cxlmd device lock held, check the cxlmd->endpoint pointer +* and then take a reference of the device in order to keep it alive +* while accessing it. +*/ + struct device *dev __free(put_device) = cxlmd_get_endpoint_dev(cxlmd); +if (!dev)return NOTIFY_DONE ; spa = mce->addr &MCI_ADDR_PHYSADDR ;
@@ -34,7 +53,7 @@ static int cxl_handle_mce(struct notifier_block *nb, unsigned long val,if (!pfn_valid (pfn))return NOTIFY_DONE ; - spa_alias =cxl_port_get_spa_cache_alias (endpoint, spa); + spa_alias =cxl_port_get_spa_cache_alias (to_cxl_port (dev), spa);if (spa_alias == ~0ULL )return NOTIFY_DONE ;
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH 1/2] cxl/mce: Validate memdev and endpoint before dereference in cxl_handle_mce()
In reply to Dave JiangThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] The validation added for cxlmd fails to account for device teardown, where cxlds->cxlmd becomes a dangling pointer, leading to an incomplete fix and Use-After-Free. --
commit d4d366d587ddef82637af37fbf8cd28754d28024
Author: Dave Jiang <[email protected]>
cxl/mce: Validate memdev and endpoint before dereference in cxl_handle_mce()
cxlmd and endpoint are both used in cxl_handle_mce() without proper validation, which can lead to NULL pointer dereference or invalid pointer dereference. The notifier is registered in cxl_memdev_state_create() when the CXL PCI driver first binds.
Quoted 4 lines
drivers/cxl/core/mce.c +0
-0
[ ... ]
drivers/cxl/core/mce.c static int cxl_handle_mce(struct notifier_block *nb, unsigned long val, · line 25
Quoted 7 lines
drivers/cxl/core/mce.c +2
-1@@ -21,7 +21,11 @@ static int cxl_handle_mce(struct notifier_block *nb, unsigned long val,if (!mce || !mce_usable_address (mce))return NOTIFY_DONE ; -if (!endpoint) +if (!cxlmd) +return NOTIFY_DONE ;
[Severity: High]
This isn't a bug introduced by this patch, but does this check adequately protect against a use-after-free during device teardown?
The MCE notifier is registered via devm_cxl_register_mce_notifier() early in the PCI probe sequence. Later, cxl_memdev_autoremove() registers the cxl_memdev_unregister() teardown action via devm_add_action_or_reset(). Because devm actions execute in LIFO order, cxl_memdev_unregister() runs before the MCE notifier is unregistered.
During cxl_memdev_unregister(), put_device(&cxlmd->dev) is called, which can free the cxlmd structure. However, neither cxl_memdev_unregister() nor cxl_memdev_shutdown() clears the cxlds->cxlmd pointer.
This leaves mds->cxlds.cxlmd as a dangling pointer. If an MCE occurs during this teardown window, cxl_handle_mce() will find a non-NULL cxlmd and pass it to cxlmd_get_endpoint_dev(), which calls guard(device)(&cxlmd->dev), resulting in a use-after-free.
Quoted 6 lines
drivers/cxl/core/mce.c +3
-0+ + endpoint = cxlmd->endpoint; +if (IS_ERR_OR_NULL (endpoint))return NOTIFY_DONE ; spa = mce->addr &MCI_ADDR_PHYSADDR ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewRe: [PATCH 2/2] cxl/mce: Serialize the MCE handler against endpoint teardown
In reply to Dave JiangThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [Critical] The MCE notifier accesses a freed cxlmd structure and attempts to lock its mutex if an MCE occurs during device teardown, leading to a Use-After-Free and kernel crash. --
commit a9bb7894eaa94a0fe2d0d4c6dd69b86cf2d1fe0f
Author: Dave Jiang <[email protected]>
cxl/mce: Serialize the MCE handler against endpoint teardown
This commit modifies the MCE handler to take the cxlmd lock before verifying the endpoint pointer. It then takes a reference on the endpoint device to ensure it remains alive during access.
Quoted 6 lines
drivers/cxl/core/mce.c +0
-0@@ -7,13 +7,27 @@#include <cxlmem.h>
drivers/cxl/core/mce.c line 17
Quoted 10 lines
drivers/cxl/core/mce.c +8
-0#include "mce.h" +static struct device *cxlmd_get_endpoint_dev (struct cxl_memdev *cxlmd) +{ +struct cxl_port *endpoint; + +if (!cxlmd) +return NULL ; + +guard (device)(&cxlmd->dev);
[Severity: Critical]
This is a pre-existing issue, but does accessing cxlmd here lead to a use-after-free if a Machine Check Exception occurs during device teardown?
During device teardown or driver unbind, devm actions execute in reverse order. The MCE notifier is registered before the cxl_memdev structure is allocated, which means cxl_memdev is unregistered and its memory freed before the MCE notifier is deactivated.
While cxl_memdev_shutdown() clears cxlmd->cxlds, it does not clear mds->cxlds.cxlmd. This leaves mds->cxlds.cxlmd as a dangling pointer.
If a Machine Check Exception happens during this window, cxl_handle_mce() will read the freed cxlmd pointer and pass it here. Attempting to lock a mutex inside freed memory will cause memory corruption or a crash.
Quoted 1 line
drivers/cxl/core/mce.c +1
-0+ endpoint = cxlmd->endpoint;
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Dan Williams (nvidia)
maintainerRe: [PATCH 1/2] cxl/mce: Validate memdev and endpoint before dereference in cxl_handle_mce()
In reply to Dave JiangQuoted 7 lines · Dave Jiang
cxlmd and endpoint are both used in
cxl_handle_mce()without proper validation, which can lead to NULL pointer dereference or invalid pointer dereference. The notifier is registered incxl_memdev_state_create()when the CXL PCI driver first binds, before the memdev is published and before it is attached to a CXL topology.Add checks to cxlmd and endpoint to ensure they are valid before usage.
This looks to be trying to band-aid the original mistake of having cxl_memdev_state_create() register a region-relative callback.
Move the mce notifier registration to be per-region and all the lookup lifetime problems disappear.
Dave Jiang
maintainerRe: [PATCH 1/2] cxl/mce: Validate memdev and endpoint before dereference in cxl_handle_mce()
Quoted 7 lines · Dan Williams (nvidia)
Quoted 6 lines · Dave Jiang
cxlmd and endpoint are both used in
cxl_handle_mce()without proper validation, which can lead to NULL pointer dereference or invalid pointer dereference. The notifier is registered incxl_memdev_state_create()when the CXL PCI driver first binds, before the memdev is published and before it is attached to a CXL topology.
Quoted 7 lines
Quoted 1 line
Add checks to cxlmd and endpoint to ensure they are valid before usage.
This looks to be trying to band-aid the original mistake of having
cxl_memdev_state_create()register a region-relative callback.Move the mce notifier registration to be per-region and all the lookup lifetime problems disappear.
ok that makes sense.
Dan Williams (nvidia)
maintainerRe: [PATCH 2/2] cxl/mce: Serialize the MCE handler against endpoint teardown
In reply to Dave JiangQuoted 5 lines · Dave Jiang
CXL endpoint has a shorter lifetime than CXL memdev state (mds) and the MCE notifier is part of the mds. The MCE handler needs to take a reference on the endpoint in order to keep it alive while operating on it. Take the cxlmd lock to verify the endpoint is still valid and take a reference on it before accessing it.
The only way to synchronize against the removal of cxlmd would be to lock its parent device which is moving in the wrong direction.
This lifetime problem disappears with a region-relative mce notification.