Skip to contentSkip to conversation

hwmon: (peci/dimmtemp) Do not provide fake thresholds data

PATCH Applied Paul Fertser 3 participants 8 messages Jan 23, 2025 – Jan 28, 2025

Subsystems (3)
Stable branch Hardware monitoring PECI hardware monitoring drivers
lore
Share & tools
lore

Paul Fertser

author
#0 lore raw

PATCH 1/1 hwmon: (peci/dimmtemp) Do not provide fake thresholds data

When an Icelake or Sapphire Rapids CPU isn't providing the maximum and critical thresholds for particular DIMM the driver should return an error to the userspace instead of giving it stale (best case) or wrong (the structure contains all zeros after kzalloc() call) data.

The issue can be reproduced by binding the peci driver while the host is fully booted and idle, this makes PECI interaction unreliable enough.

Fixes 73bc1b885dae ("hwmon: peci: Add dimmtemp driver")
Fixes 621995b6d795 ("hwmon: (peci/dimmtemp) Add Sapphire Rapids support")
Cc [email protected]
Signed-off-by Paul Fertser <[email protected]>
---
 drivers/hwmon/peci/dimmtemp.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
drivers/hwmon/peci/dimmtemp.c +4 -6
index d6762259dd69..fbe82d9852e0 100644
@@ -127,8 +127,6 @@ static int update_thresholds(struct peci_dimmtemp *priv, int dimm_no)
 		return 0;
 
 	ret = priv->gen_info->read_thresholds(priv, dimm_order, chan_rank, &data);
-	if (ret == -ENODATA) /* Use default or previous value */
-		return 0;
 	if (ret)
 		return ret;
 

@@ -509,11 +507,11 @@ read_thresholds_icx(struct peci_dimmtemp *priv, int dimm_order, int chan_rank, u
 
 	ret = peci_ep_pci_local_read(priv->peci_dev, 0, 13, 0, 2, 0xd4, &reg_val);
 	if (ret || !(reg_val & BIT(31)))
-		return -ENODATA; /* Use default or previous value */
+		return -ENODATA;
 
 	ret = peci_ep_pci_local_read(priv->peci_dev, 0, 13, 0, 2, 0xd0, &reg_val);
 	if (ret)
-		return -ENODATA; /* Use default or previous value */
+		return -ENODATA;
 
 	/*
 	 * Device 26, Offset 224e0: IMC 0 channel 0 -> rank 0

@@ -546,11 +544,11 @@ read_thresholds_spr(struct peci_dimmtemp *priv, int dimm_order, int chan_rank, u
 
 	ret = peci_ep_pci_local_read(priv->peci_dev, 0, 30, 0, 2, 0xd4, &reg_val);
 	if (ret || !(reg_val & BIT(31)))
-		return -ENODATA; /* Use default or previous value */
+		return -ENODATA;
 
 	ret = peci_ep_pci_local_read(priv->peci_dev, 0, 30, 0, 2, 0xd0, &reg_val);
 	if (ret)
-		return -ENODATA; /* Use default or previous value */
+		return -ENODATA;
 
 	/*
 	 * Device 26, Offset 219a8: IMC 0 channel 0 -> rank 0
Signature · 1 line
2.34.1
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 5 lines · Paul Fertser

When an Icelake or Sapphire Rapids CPU isn't providing the maximum and critical thresholds for particular DIMM the driver should return an error to the userspace instead of giving it stale (best case) or wrong (the structure contains all zeros after kzalloc() call) data.

Quoted 7 lines

The issue can be reproduced by binding the peci driver while the host is fully booted and idle, this makes PECI interaction unreliable enough.

Fixes 73bc1b885dae ("hwmon: peci: Add dimmtemp driver")
Fixes 621995b6d795 ("hwmon: (peci/dimmtemp) Add Sapphire Rapids support")
Cc [email protected]
Signed-off-by Paul Fertser <[email protected]>

Hi!

Thank you for the patch.
Did you have a chance to test it with OpenBMC dbus-sensors?
In general, the change looks okay to me, but since it modifies the behavior (applications will need to handle this, and returning an error will happen more often) we need to confirm that it does not cause any regressions for userspace.

Once we are able to confirm that:

Reviewed-by Iwona Winiarska <[email protected]>

Thanks
-Iwona

Quoted 53 lines

---
 drivers/hwmon/peci/dimmtemp.c | 10 ++++------

 1 file changed, 4 insertions(+), 6 deletions(-)
drivers/hwmon/peci/dimmtemp.c +0 -0
index d6762259dd69..fbe82d9852e0 100644
@@ -127,8 +127,6 @@ static int update_thresholds(struct peci_dimmtemp *priv,
 int dimm_no)
  		return 0;


  ret = priv->gen_info->read_thresholds(priv, dimm_order, chan_rank, &data);

drivers/hwmon/peci/dimmtemp.c +0 -2
-	if (ret == -ENODATA) /* Use default or previous value */
-		return 0;

  if (ret)
  return ret;

drivers/hwmon/peci/dimmtemp.c +0 -0
@@ -509,11 +507,11 @@ read_thresholds_icx(struct peci_dimmtemp *priv, int

dimm_order, int chan_rank, u

  ret = peci_ep_pci_local_read(priv->peci_dev, 0, 13, 0, 2, 0xd4, &reg_val);
  if (ret || !(reg_val & BIT(31)))

drivers/hwmon/peci/dimmtemp.c +1 -1
-		return -ENODATA; /* Use default or previous value */
+		return -ENODATA;


  ret = peci_ep_pci_local_read(priv->peci_dev, 0, 13, 0, 2, 0xd0, &reg_val);
  if (ret)

drivers/hwmon/peci/dimmtemp.c +1 -1
-		return -ENODATA; /* Use default or previous value */
+		return -ENODATA;


  /*
  * Device 26, Offset 224e0: IMC 0 channel 0 -> rank 0

drivers/hwmon/peci/dimmtemp.c +0 -0
@@ -546,11 +544,11 @@ read_thresholds_spr(struct peci_dimmtemp *priv, int

dimm_order, int chan_rank, u

  ret = peci_ep_pci_local_read(priv->peci_dev, 0, 30, 0, 2, 0xd4, &reg_val);
  if (ret || !(reg_val & BIT(31)))

drivers/hwmon/peci/dimmtemp.c +1 -1
-		return -ENODATA; /* Use default or previous value */
+		return -ENODATA;


  ret = peci_ep_pci_local_read(priv->peci_dev, 0, 30, 0, 2, 0xd0, &reg_val);
  if (ret)

drivers/hwmon/peci/dimmtemp.c +1 -1
-		return -ENODATA; /* Use default or previous value */
+		return -ENODATA;


  /*
  * Device 26, Offset 219a8: IMC 0 channel 0 -> rank 0

maintainer
#2 lore raw
Quoted 14 lines · Iwona
Quoted 5 lines · Paul Fertser

When an Icelake or Sapphire Rapids CPU isn't providing the maximum and critical thresholds for particular DIMM the driver should return an error to the userspace instead of giving it stale (best case) or wrong (the structure contains all zeros after kzalloc() call) data.

Quoted 7 lines

The issue can be reproduced by binding the peci driver while the host is fully booted and idle, this makes PECI interaction unreliable enough.

Fixes 73bc1b885dae ("hwmon: peci: Add dimmtemp driver")
Fixes 621995b6d795 ("hwmon: (peci/dimmtemp) Add Sapphire Rapids support")
Cc [email protected]
Signed-off-by Paul Fertser <[email protected]>
Quoted 8 lines

Hi!

Thank you for the patch.
Did you have a chance to test it with OpenBMC dbus-sensors?
In general, the change looks okay to me, but since it modifies the behavior (applications will need to handle this, and returning an error will happen more often) we need to confirm that it does not cause any regressions for userspace.

I would also like to understand if the error is temporary or permanent. If it is permanent, the attributes should not be created in the first place. It does not make sense to have limit attributes which always report -ENODATA.

Guenter

author
#3 lore raw

Hi Guenter,

Quoted 24 lines · Guenter Roeck
Quoted 14 lines · Iwona
Quoted 5 lines · Paul Fertser

When an Icelake or Sapphire Rapids CPU isn't providing the maximum and critical thresholds for particular DIMM the driver should return an error to the userspace instead of giving it stale (best case) or wrong (the structure contains all zeros after kzalloc() call) data.

Quoted 7 lines

The issue can be reproduced by binding the peci driver while the host is fully booted and idle, this makes PECI interaction unreliable enough.

Fixes 73bc1b885dae ("hwmon: peci: Add dimmtemp driver")
Fixes 621995b6d795 ("hwmon: (peci/dimmtemp) Add Sapphire Rapids support")
Cc [email protected]
Signed-off-by Paul Fertser <[email protected]>
Quoted 8 lines

Hi!

Thank you for the patch.
Did you have a chance to test it with OpenBMC dbus-sensors?
In general, the change looks okay to me, but since it modifies the behavior (applications will need to handle this, and returning an error will happen more often) we need to confirm that it does not cause any regressions for userspace.

Quoted 4 lines

I would also like to understand if the error is temporary or permanent. If it is permanent, the attributes should not be created in the first place. It does not make sense to have limit attributes which always report -ENODATA.

The error is temporary. The underlying reason is that when host CPUs go to deep enough idle sleep state (probably C6) they stop responding to PECI requests from BMC. Once something starts running the CPU leaves C6 and starts responding and all the temperature data (including the thresholds) becomes available again.

Signature · 2 lines
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:[email protected]
maintainer
#4 lore raw
Quoted 32 lines · Paul Fertser

Hi Guenter,

Quoted 24 lines · Guenter Roeck
Quoted 14 lines · Iwona
Quoted 5 lines · Paul Fertser

When an Icelake or Sapphire Rapids CPU isn't providing the maximum and critical thresholds for particular DIMM the driver should return an error to the userspace instead of giving it stale (best case) or wrong (the structure contains all zeros after kzalloc() call) data.

Quoted 7 lines

The issue can be reproduced by binding the peci driver while the host is fully booted and idle, this makes PECI interaction unreliable enough.

Fixes 73bc1b885dae ("hwmon: peci: Add dimmtemp driver")
Fixes 621995b6d795 ("hwmon: (peci/dimmtemp) Add Sapphire Rapids support")
Cc [email protected]
Signed-off-by Paul Fertser <[email protected]>
Quoted 8 lines

Hi!

Thank you for the patch.
Did you have a chance to test it with OpenBMC dbus-sensors?
In general, the change looks okay to me, but since it modifies the behavior (applications will need to handle this, and returning an error will happen more often) we need to confirm that it does not cause any regressions for userspace.

Quoted 4 lines

I would also like to understand if the error is temporary or permanent. If it is permanent, the attributes should not be created in the first place. It does not make sense to have limit attributes which always report -ENODATA.

Quoted 6 lines

The error is temporary. The underlying reason is that when host CPUs go to deep enough idle sleep state (probably C6) they stop responding to PECI requests from BMC. Once something starts running the CPU leaves C6 and starts responding and all the temperature data (including the thresholds) becomes available again.

Thanks.

Next question: Is there evidence that the thresholds change while the CPU is in a deep sleep state (or, in other words, that they are indeed stale) ? Because if not it would be (much) better to only report -ENODATA if the thresholds are uninitialized, and it would be even better than that if the limits are read during initialization (and not updated at all) if they do not change dynamically.

Guenter

author
#5 lore raw
Quoted 40 lines · Guenter Roeck
Quoted 32 lines · Paul Fertser

Hi Guenter,

Quoted 24 lines · Guenter Roeck
Quoted 14 lines · Iwona
Quoted 5 lines · Paul Fertser

When an Icelake or Sapphire Rapids CPU isn't providing the maximum and critical thresholds for particular DIMM the driver should return an error to the userspace instead of giving it stale (best case) or wrong (the structure contains all zeros after kzalloc() call) data.

Quoted 7 lines

The issue can be reproduced by binding the peci driver while the host is fully booted and idle, this makes PECI interaction unreliable enough.

Fixes 73bc1b885dae ("hwmon: peci: Add dimmtemp driver")
Fixes 621995b6d795 ("hwmon: (peci/dimmtemp) Add Sapphire Rapids support")
Cc [email protected]
Signed-off-by Paul Fertser <[email protected]>
Quoted 8 lines

Hi!

Thank you for the patch.
Did you have a chance to test it with OpenBMC dbus-sensors?
In general, the change looks okay to me, but since it modifies the behavior (applications will need to handle this, and returning an error will happen more often) we need to confirm that it does not cause any regressions for userspace.

Quoted 4 lines

I would also like to understand if the error is temporary or permanent. If it is permanent, the attributes should not be created in the first place. It does not make sense to have limit attributes which always report -ENODATA.

Quoted 6 lines

The error is temporary. The underlying reason is that when host CPUs go to deep enough idle sleep state (probably C6) they stop responding to PECI requests from BMC. Once something starts running the CPU leaves C6 and starts responding and all the temperature data (including the thresholds) becomes available again.

Quoted 8 lines

Thanks.

Next question: Is there evidence that the thresholds change while the CPU is in a deep sleep state (or, in other words, that they are indeed stale) ? Because if not it would be (much) better to only report -ENODATA if the thresholds are uninitialized, and it would be even better than that if the limits are read during initialization (and not updated at all) if they do not change dynamically.

From BMC point of view when getting a timeout there is little difference between the host not answering being in idle deep sleep state and between host being completely powered off. Now I can imagine a server system where BMC keeps running and the server has its DIMMs physically changed to a different model with different threshold.

Whether it's realistic scenario and whether it's worth caching the thresholds in the kernel I hope Iwona can clarify. In my current opinion the added complexity isn't worth it, the PECI operation needs to be reliable enough anyway for BMC to monitor at least the CPU temperatures once a second to feed this essential data to the cooling fans control loop. And if we can read CPU temperatures we can also read DIMM thresholds when we need them and worse case retry a few times while starting up the daemon.

Signature · 2 lines
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:[email protected]
maintainer Applied
#6 lore raw
Quoted 56 lines · Paul Fertser
Quoted 42 lines · Guenter Roeck
Quoted 32 lines · Paul Fertser

Hi Guenter,

Quoted 24 lines · Guenter Roeck
Quoted 14 lines · Iwona
Quoted 5 lines · Paul Fertser

When an Icelake or Sapphire Rapids CPU isn't providing the maximum and critical thresholds for particular DIMM the driver should return an error to the userspace instead of giving it stale (best case) or wrong (the structure contains all zeros after kzalloc() call) data.

Quoted 7 lines

The issue can be reproduced by binding the peci driver while the host is fully booted and idle, this makes PECI interaction unreliable enough.

Fixes 73bc1b885dae ("hwmon: peci: Add dimmtemp driver")
Fixes 621995b6d795 ("hwmon: (peci/dimmtemp) Add Sapphire Rapids support")
Cc [email protected]
Signed-off-by Paul Fertser <[email protected]>
Quoted 8 lines

Hi!

Thank you for the patch.
Did you have a chance to test it with OpenBMC dbus-sensors?
In general, the change looks okay to me, but since it modifies the behavior (applications will need to handle this, and returning an error will happen more often) we need to confirm that it does not cause any regressions for userspace.

Quoted 4 lines

I would also like to understand if the error is temporary or permanent. If it is permanent, the attributes should not be created in the first place. It does not make sense to have limit attributes which always report -ENODATA.

Quoted 6 lines

The error is temporary. The underlying reason is that when host CPUs go to deep enough idle sleep state (probably C6) they stop responding to PECI requests from BMC. Once something starts running the CPU leaves C6 and starts responding and all the temperature data (including the thresholds) becomes available again.

Thanks.

Quoted 8 lines

Next question: Is there evidence that the thresholds change while the CPU is in a deep sleep state (or, in other words, that they are indeed stale) ? Because if not it would be (much) better to only report -ENODATA if the thresholds are uninitialized, and it would be even better than that if the limits are read during initialization (and not updated at all) if they do not change dynamically.

From BMC point of view when getting a timeout there is little

difference between the host not answering being in idle deep sleep state and between host being completely powered off. Now I can imagine a server system where BMC keeps running and the server has its DIMMs physically changed to a different model with different threshold.

Quoted 9 lines

Whether it's realistic scenario and whether it's worth caching the thresholds in the kernel I hope Iwona can clarify. In my current opinion the added complexity isn't worth it, the PECI operation needs to be reliable enough anyway for BMC to monitor at least the CPU temperatures once a second to feed this essential data to the cooling fans control loop. And if we can read CPU temperatures we can also read DIMM thresholds when we need them and worse case retry a few times while starting up the daemon.

Makes sense.

Applied.

Thanks,
Guenter

author
#7 lore raw
In reply to Winiarska, Iwona

Hi Iwona,

Thank you for the review. Please see inline.

Quoted 9 lines · Iwona
Quoted 8 lines · Paul Fertser

When an Icelake or Sapphire Rapids CPU isn't providing the maximum and critical thresholds for particular DIMM the driver should return an error to the userspace instead of giving it stale (best case) or wrong (the structure contains all zeros after kzalloc() call) data.

The issue can be reproduced by binding the peci driver while the host is fully booted and idle, this makes PECI interaction unreliable enough.

Quoted 6 lines
Quoted 4 lines
Fixes 73bc1b885dae ("hwmon: peci: Add dimmtemp driver")
Fixes 621995b6d795 ("hwmon: (peci/dimmtemp) Add Sapphire Rapids support")
Cc [email protected]
Signed-off-by Paul Fertser <[email protected]>

Did you have a chance to test it with OpenBMC dbus-sensors?

Using OpenBMC dbus-sensors is exactly the reason why I'm sending this patch, so yes, I tested it before and after the change.

Quoted 3 lines

In general, the change looks okay to me, but since it modifies the behavior (applications will need to handle this, and returning an error will happen more often) we need to confirm that it does not cause any regressions for userspace.

The change is prompted by the current behaviour which is unacceptably
bad: every now and then while powering on the host for the first time BMC happens to request one of the memory thresholds at a wrong time (e.g. when UEFI is busy doing something which prevents normal PECI operation); this leads to the unfixed kernel code returning zero and dbus-sensors happily using that as a threshold value which later results in bogus critical over temperature events for the affected DIMM (as their normal temperatures are always above zero). It was relatively easy to reproduce on an IceLake-based system.

I consider the current behaviour (in case of PECI timeouts when requesting DIMM temperature thresholds) to be so broken that changing it to do the right thing can only do good. The non-failure case is not affected by this patch.

That said, for sensible operation a dbus-sensors change is indeed needed and I now have a patch pending upstream review[0] to handle those errors by retrying until success. Without the patch the daemon would just load with those thresholds missing but it's better to have thresholds missing than to have them at zero producing a critical error right away I think.

[0] https://gerrit.openbmc.org/c/openbmc/dbus-sensors/+/77500/

Signature · 2 lines
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:[email protected]