Skip to contentSkip to conversation

thermal: core: Call monitor_thermal_zone() if zone temperature is invalid

PATCH Superseded Rafael J. Wysocki 4 participants 15 messages Jun 28, 2024 – Jul 4, 2024

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 thermal: core: Call monitor_thermal_zone() if zone temperature is invalid

From: Rafael J. Wysocki <[email protected]>

Commit 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid") caused __thermal_zone_device_update() to return early if the current thermal zone temperature was invalid.

This was done to avoid running handle_thermal_trip() and governor callbacks in that case which led to confusion. However, it went too far because monitor_thermal_zone() still needs to be called even when the zone temperature is invalid to ensure that it will be updated eventually in case thermal polling is enabled and the driver has no other means to notify the core of zone temperature changes (for example, it does not register an interrupt handler or ACPI notifier).

Also if the .set_trips() zone callback is expected to set up monitoring interrupts for a thermal zone, it has to be provided with valid boundaries and that can only happen if the zone temperature is known.

Accordingly, to ensure that __thermal_zone_device_update() will run again after a failing zone temperature check, make it call monitor_thermal_zone() regardless of whether or not the zone temperature is valid and make the latter schedule a thermal zone temperature update if the zone temperature is invalid even if polling is not enabled for the thermal zone.

Fixes 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid")
---
 drivers/thermal/thermal_core.c |    5 ++++-
 drivers/thermal/thermal_core.h |    6 ++++++
 2 files changed, 10 insertions(+), 1 deletion(-)
drivers/thermal/thermal_core.c +4 -1
Index: linux-pm/drivers/thermal/thermal_core.c
===================================================================
@@ -300,6 +300,8 @@ static void monitor_thermal_zone(struct
 		thermal_zone_device_set_polling(tz, tz->passive_delay_jiffies);
 	else if (tz->polling_delay_jiffies)
 		thermal_zone_device_set_polling(tz, tz->polling_delay_jiffies);
+	else if (tz->temperature == THERMAL_TEMP_INVALID)
+		thermal_zone_device_set_polling(tz, msecs_to_jiffies(THERMAL_RECHECK_DELAY_MS));
 }
 
 static struct thermal_governor *thermal_get_tz_governor(struct thermal_zone_device *tz)
@@ -514,7 +516,7 @@ void __thermal_zone_device_update(struct
 	update_temperature(tz);
 
 	if (tz->temperature == THERMAL_TEMP_INVALID)
-		return;
+		goto monitor;
 
 	tz->notify_event = event;
 
@@ -536,6 +538,7 @@ void __thermal_zone_device_update(struct
 
 	thermal_debug_update_trip_stats(tz);
 
+monitor:
 	monitor_thermal_zone(tz);
 }
 
drivers/thermal/thermal_core.h +6 -0
Index: linux-pm/drivers/thermal/thermal_core.h
===================================================================
@@ -133,6 +133,12 @@ struct thermal_zone_device {
 	struct thermal_trip_desc trips[] __counted_by(num_trips);
 };
 
+/*
+ * Default delay after a failing thermal zone temperature check before
+ * attempting to check it again.
+ */
+#define THERMAL_RECHECK_DELAY_MS	100
+
 /* Default Thermal Governor */
 #if defined(CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE)
 #define DEFAULT_THERMAL_GOVERNOR       "step_wise"
Tested-by on v2 1/1
#1 lore raw
Quoted 25 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Commit 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid") caused __thermal_zone_device_update() to return early if the current thermal zone temperature was invalid.

This was done to avoid running handle_thermal_trip() and governor callbacks in that case which led to confusion. However, it went too far because monitor_thermal_zone() still needs to be called even when the zone temperature is invalid to ensure that it will be updated eventually in case thermal polling is enabled and the driver has no other means to notify the core of zone temperature changes (for example, it does not register an interrupt handler or ACPI notifier).

Also if the .set_trips() zone callback is expected to set up monitoring interrupts for a thermal zone, it has to be provided with valid boundaries and that can only happen if the zone temperature is known.

Accordingly, to ensure that __thermal_zone_device_update() will run again after a failing zone temperature check, make it call monitor_thermal_zone() regardless of whether or not the zone temperature is valid and make the latter schedule a thermal zone temperature update if the zone temperature is invalid even if polling is not enabled for the thermal zone.

Quoted 3 lines
Fixes 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid")
Signature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog
#2 lore raw
In reply to Rafael J. Wysocki

Hi,

Quoted 74 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Commit 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid") caused __thermal_zone_device_update() to return early if the current thermal zone temperature was invalid.

This was done to avoid running handle_thermal_trip() and governor callbacks in that case which led to confusion. However, it went too far because monitor_thermal_zone() still needs to be called even when the zone temperature is invalid to ensure that it will be updated eventually in case thermal polling is enabled and the driver has no other means to notify the core of zone temperature changes (for example, it does not register an interrupt handler or ACPI notifier).

Also if the .set_trips() zone callback is expected to set up monitoring interrupts for a thermal zone, it has to be provided with valid boundaries and that can only happen if the zone temperature is known.

Accordingly, to ensure that __thermal_zone_device_update() will run again after a failing zone temperature check, make it call monitor_thermal_zone() regardless of whether or not the zone temperature is valid and make the latter schedule a thermal zone temperature update if the zone temperature is invalid even if polling is not enabled for the thermal zone.

Fixes 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid")
---
  drivers/thermal/thermal_core.c |    5 ++++-
  drivers/thermal/thermal_core.h |    6 ++++++
  2 files changed, 10 insertions(+), 1 deletion(-)

Index: linux-pm/drivers/thermal/thermal_core.c
===================================================================

drivers/thermal/thermal_core.c +4 -1
@@ -300,6 +300,8 @@ static void monitor_thermal_zone(struct
  		thermal_zone_device_set_polling(tz, tz->passive_delay_jiffies);
  	else if (tz->polling_delay_jiffies)
  		thermal_zone_device_set_polling(tz, tz->polling_delay_jiffies);
+	else if (tz->temperature == THERMAL_TEMP_INVALID)
+		thermal_zone_device_set_polling(tz, msecs_to_jiffies(THERMAL_RECHECK_DELAY_MS));
  }
 
  static struct thermal_governor *thermal_get_tz_governor(struct thermal_zone_device *tz)
@@ -514,7 +516,7 @@ void __thermal_zone_device_update(struct
  	update_temperature(tz);
 
  	if (tz->temperature == THERMAL_TEMP_INVALID)
-		return;
+		goto monitor;
 
  	tz->notify_event = event;
 
@@ -536,6 +538,7 @@ void __thermal_zone_device_update(struct
 
  	thermal_debug_update_trip_stats(tz);
 
+monitor:
  	monitor_thermal_zone(tz);
  }
 

Index: linux-pm/drivers/thermal/thermal_core.h
===================================================================

drivers/thermal/thermal_core.h +0 -0
@@ -133,6 +133,12 @@ struct thermal_zone_device {
  	struct thermal_trip_desc trips[] __counted_by(num_trips);
  };

drivers/thermal/thermal_core.h struct thermal_zone_device { · line 144

Quoted 10 lines
drivers/thermal/thermal_core.h +6 -0
 
+/*
+ * Default delay after a failing thermal zone temperature check before
+ * attempting to check it again.
+ */
+#define THERMAL_RECHECK_DELAY_MS	100
+
  /* Default Thermal Governor */
  #if defined(CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE)
  #define DEFAULT_THERMAL_GOVERNOR       "step_wise"

This patch on next-20240702 makes Qualcomm HDK8350, HDK8450, QRD8550, HDK8560, QRD8650 & HDK8650 output in loop:

thermal thermal_zoneXX: failed to read out thermal zone (-19)

Boot logs or ARM64 defconfig:
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152439#L1393
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152440#L2200
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152442#L2828
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152441#L1862
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152443#L1776
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152444#L1723

Result of git bisect:
# bad: [82e4255305c554b0bb18b7ccf2db86041b4c8b6e] Add linux-next specific files for 20240702
# good: [22a40d14b572deb80c0648557f4bd502d7e83826] Linux 6.10-rc6
git bisect start 'FETCH_HEAD' 'v6.10-rc6'
# bad: [f6dfcf0e9567b57b93f2564966d9177f0d8dbe05] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git
git bisect bad f6dfcf0e9567b57b93f2564966d9177f0d8dbe05
# good: [7f86ae0c2dc19fea7be1da29b2bf03f085463ae7] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux.git
git bisect good 7f86ae0c2dc19fea7be1da29b2bf03f085463ae7
# bad: [077d5bbd75dd12af2096c96846ffc78ab5dd65b1] Merge branch 'devfreq-next' of git://git.kernel.org/pub/scm/linux/kernel/git/chanwoo/linux.git
git bisect bad 077d5bbd75dd12af2096c96846ffc78ab5dd65b1
# good: [271bcaf753d0afe2bd0386ab1e98132ee65b61ca] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux.git
git bisect good 271bcaf753d0afe2bd0386ab1e98132ee65b61ca
# good: [9758a2ee5316a6f8736ab4fd39a6f6176aa057ec] Merge branch 'hwmon-next' of git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging.git
git bisect good 9758a2ee5316a6f8736ab4fd39a6f6176aa057ec
# good: [e6bd69ea345045520bd63487b85a4b5676aff76b] Merge branch 'master' of git://linuxtv.org/mchehab/media-next.git
git bisect good e6bd69ea345045520bd63487b85a4b5676aff76b
# good: [46398edfb36e2882be5e86ea563b2db9138ae499] Merge branches 'pm-cpuidle' and 'pm-powercap' into linux-next
git bisect good 46398edfb36e2882be5e86ea563b2db9138ae499
# good: [d3927cbc52eed166f74ea7e031ed6384cc3d4d5f] Merge branch 'thermal-intel' into linux-next
git bisect good d3927cbc52eed166f74ea7e031ed6384cc3d4d5f
# good: [ce84b7beeb524e7b20983838687862454ba54df7] cpufreq: sti: add missing MODULE_DEVICE_TABLE entry for stih418
git bisect good ce84b7beeb524e7b20983838687862454ba54df7
# bad: [fcf61315d38d41f4e55856b179f9e5538e299ef4] Merge branch 'thermal-fixes' into linux-next
git bisect bad fcf61315d38d41f4e55856b179f9e5538e299ef4
# good: [4262b8d782a74c7cf7b8b94ed9e4fcb94e856d1e] dt-bindings: thermal: mediatek: Fix thermal zone definition for MT8186
git bisect good 4262b8d782a74c7cf7b8b94ed9e4fcb94e856d1e
# good: [7eeb114a635a04bea2fa7d57cedbf374c714d29e] dt-bindings: thermal: convert hisilicon-thermal.txt to dt-schema
git bisect good 7eeb114a635a04bea2fa7d57cedbf374c714d29e
# good: [107ac0d49ae6a86b4986146b9a612294f7e34406] Merge branch 'thermal/linux-next' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/thermal/linux into linux-next git bisect good 107ac0d49ae6a86b4986146b9a612294f7e34406
# bad: [5725f40698b9ba7f84fbfee25b9059ba044c4b86] thermal: core: Call monitor_thermal_zone() if zone temperature is invalid
git bisect bad 5725f40698b9ba7f84fbfee25b9059ba044c4b86

# first bad commit: [5725f40698b9ba7f84fbfee25b9059ba044c4b86] thermal: core: Call monitor_thermal_zone() if zone temperature is invalid

#regzbot introduced: 5725f40698b9ba7f84fbfee25b9059ba044c4b86

Thanks,
Neil

#3 lore raw

Hi,

Quoted 91 lines · Neil Armstrong

Hi,

Quoted 74 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Commit 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid") caused __thermal_zone_device_update() to return early if the current thermal zone temperature was invalid.

This was done to avoid running handle_thermal_trip() and governor callbacks in that case which led to confusion. However, it went too far because monitor_thermal_zone() still needs to be called even when the zone temperature is invalid to ensure that it will be updated eventually in case thermal polling is enabled and the driver has no other means to notify the core of zone temperature changes (for example, it does not register an interrupt handler or ACPI notifier).

Also if the .set_trips() zone callback is expected to set up monitoring interrupts for a thermal zone, it has to be provided with valid boundaries and that can only happen if the zone temperature is known.

Accordingly, to ensure that __thermal_zone_device_update() will run again after a failing zone temperature check, make it call monitor_thermal_zone() regardless of whether or not the zone temperature is valid and make the latter schedule a thermal zone temperature update if the zone temperature is invalid even if polling is not enabled for the thermal zone.

Fixes 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid")
---
  drivers/thermal/thermal_core.c |    5 ++++-
  drivers/thermal/thermal_core.h |    6 ++++++
  2 files changed, 10 insertions(+), 1 deletion(-)

Index: linux-pm/drivers/thermal/thermal_core.c
===================================================================

drivers/thermal/thermal_core.c +4 -1
@@ -300,6 +300,8 @@ static void monitor_thermal_zone(struct
              thermal_zone_device_set_polling(tz, tz->passive_delay_jiffies);
      else if (tz->polling_delay_jiffies)
              thermal_zone_device_set_polling(tz, tz->polling_delay_jiffies);
+     else if (tz->temperature == THERMAL_TEMP_INVALID)
+             thermal_zone_device_set_polling(tz, msecs_to_jiffies(THERMAL_RECHECK_DELAY_MS));
  }
 
  static struct thermal_governor *thermal_get_tz_governor(struct thermal_zone_device *tz)
@@ -514,7 +516,7 @@ void __thermal_zone_device_update(struct
      update_temperature(tz);
 
      if (tz->temperature == THERMAL_TEMP_INVALID)
-             return;
+             goto monitor;
 
      tz->notify_event = event;
 
@@ -536,6 +538,7 @@ void __thermal_zone_device_update(struct
 
      thermal_debug_update_trip_stats(tz);
 
+monitor:
      monitor_thermal_zone(tz);
  }
 

Index: linux-pm/drivers/thermal/thermal_core.h
===================================================================

drivers/thermal/thermal_core.h +0 -0
@@ -133,6 +133,12 @@ struct thermal_zone_device {
      struct thermal_trip_desc trips[] __counted_by(num_trips);
  };

drivers/thermal/thermal_core.h struct thermal_zone_device { · line 144

Quoted 10 lines
drivers/thermal/thermal_core.h +6 -0
 
+/*
+ * Default delay after a failing thermal zone temperature check before
+ * attempting to check it again.
+ */
+#define THERMAL_RECHECK_DELAY_MS     100
+
  /* Default Thermal Governor */
  #if defined(CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE)
  #define DEFAULT_THERMAL_GOVERNOR       "step_wise"

This patch on next-20240702 makes Qualcomm HDK8350, HDK8450, QRD8550, HDK8560, QRD8650 & HDK8650 output in loop:

Quoted 1 line

thermal thermal_zoneXX: failed to read out thermal zone (-19)

Is the loop endless? If not, how many times does the message get printed?

If I'm not mistaken, it would be printed at least once without the commit in question. Can you please check that?

Also, can you check the previous version of the patch in question:

https://lore.kernel.org/linux-pm/[email protected]/

and see if it has the same problem (just apply it instead of the $subject one).

Thanks!

#4 lore raw

Hi,

Quoted 94 lines · Rafael J. Wysocki

Hi,

Quoted 91 lines · Neil Armstrong

Hi,

Quoted 74 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Commit 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid") caused __thermal_zone_device_update() to return early if the current thermal zone temperature was invalid.

This was done to avoid running handle_thermal_trip() and governor callbacks in that case which led to confusion. However, it went too far because monitor_thermal_zone() still needs to be called even when the zone temperature is invalid to ensure that it will be updated eventually in case thermal polling is enabled and the driver has no other means to notify the core of zone temperature changes (for example, it does not register an interrupt handler or ACPI notifier).

Also if the .set_trips() zone callback is expected to set up monitoring interrupts for a thermal zone, it has to be provided with valid boundaries and that can only happen if the zone temperature is known.

Accordingly, to ensure that __thermal_zone_device_update() will run again after a failing zone temperature check, make it call monitor_thermal_zone() regardless of whether or not the zone temperature is valid and make the latter schedule a thermal zone temperature update if the zone temperature is invalid even if polling is not enabled for the thermal zone.

Fixes 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid")
---
   drivers/thermal/thermal_core.c |    5 ++++-
   drivers/thermal/thermal_core.h |    6 ++++++
   2 files changed, 10 insertions(+), 1 deletion(-)

Index: linux-pm/drivers/thermal/thermal_core.c
===================================================================

drivers/thermal/thermal_core.c +4 -1
@@ -300,6 +300,8 @@ static void monitor_thermal_zone(struct
               thermal_zone_device_set_polling(tz, tz->passive_delay_jiffies);
       else if (tz->polling_delay_jiffies)
               thermal_zone_device_set_polling(tz, tz->polling_delay_jiffies);
+     else if (tz->temperature == THERMAL_TEMP_INVALID)
+             thermal_zone_device_set_polling(tz, msecs_to_jiffies(THERMAL_RECHECK_DELAY_MS));
   }
 
   static struct thermal_governor *thermal_get_tz_governor(struct thermal_zone_device *tz)
@@ -514,7 +516,7 @@ void __thermal_zone_device_update(struct
       update_temperature(tz);
 
       if (tz->temperature == THERMAL_TEMP_INVALID)
-             return;
+             goto monitor;
 
       tz->notify_event = event;
 
@@ -536,6 +538,7 @@ void __thermal_zone_device_update(struct
 
       thermal_debug_update_trip_stats(tz);
 
+monitor:
       monitor_thermal_zone(tz);
   }
 

Index: linux-pm/drivers/thermal/thermal_core.h
===================================================================

drivers/thermal/thermal_core.h +0 -0
@@ -133,6 +133,12 @@ struct thermal_zone_device {
       struct thermal_trip_desc trips[] __counted_by(num_trips);
   };

drivers/thermal/thermal_core.h struct thermal_zone_device { · line 144

Quoted 10 lines
drivers/thermal/thermal_core.h +6 -0
 
+/*
+ * Default delay after a failing thermal zone temperature check before
+ * attempting to check it again.
+ */
+#define THERMAL_RECHECK_DELAY_MS     100
+
   /* Default Thermal Governor */
   #if defined(CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE)
   #define DEFAULT_THERMAL_GOVERNOR       "step_wise"

This patch on next-20240702 makes Qualcomm HDK8350, HDK8450, QRD8550, HDK8560, QRD8650 & HDK8650 output in loop:

Quoted 3 lines
Quoted 1 line

thermal thermal_zoneXX: failed to read out thermal zone (-19)

Is the loop endless? If not, how many times does the message get printed?

It get printed indefinitely

Quoted 9 lines

If I'm not mistaken, it would be printed at least once without the commit in question. Can you please check that?

Also, can you check the previous version of the patch in question:

https://lore.kernel.org/linux-pm/[email protected]/

and see if it has the same problem (just apply it instead of the $subject one).

I reverted this one a applied v1 and the message disappeared completely.

Neil

Quoted 2 lines

Thanks!

#5 lore raw
In reply to Neil Armstrong

Hi Neil,

it seems there is something wrong with the driver actually.

There can be a moment where the sensor is not yet initialized for different reason, so reading the temperature fails. The routine will just retry until the sensor gets ready.

Having these errors seem to me that the sensor for this specific thermal zone is never ready which may be the root cause of your issue. The change is spotting this problem IMO.

Quoted 131 lines · Neil Armstrong

Hi,

Quoted 67 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Commit 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid") caused __thermal_zone_device_update() to return early if the current thermal zone temperature was invalid.

This was done to avoid running handle_thermal_trip() and governor callbacks in that case which led to confusion.  However, it went too far because monitor_thermal_zone() still needs to be called even when the zone temperature is invalid to ensure that it will be updated eventually in case thermal polling is enabled and the driver has no other means to notify the core of zone temperature changes (for example, it does not register an interrupt handler or ACPI notifier).

Also if the .set_trips() zone callback is expected to set up monitoring interrupts for a thermal zone, it has to be provided with valid boundaries and that can only happen if the zone temperature is known.

Accordingly, to ensure that __thermal_zone_device_update() will run again after a failing zone temperature check, make it call monitor_thermal_zone() regardless of whether or not the zone temperature is valid and make the latter schedule a thermal zone temperature update if the zone temperature is invalid even if polling is not enabled for the thermal zone.

Fixes 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid")

---
  drivers/thermal/thermal_core.c |    5 ++++-
  drivers/thermal/thermal_core.h |    6 ++++++

  2 files changed, 10 insertions(+), 1 deletion(-)

Index: linux-pm/drivers/thermal/thermal_core.c
===================================================================

drivers/thermal/thermal_core.c +4 -1
@@ -300,6 +300,8 @@ static void monitor_thermal_zone(struct
           thermal_zone_device_set_polling(tz, tz->passive_delay_jiffies);
       else if (tz->polling_delay_jiffies)
           thermal_zone_device_set_polling(tz, tz->polling_delay_jiffies);
+    else if (tz->temperature == THERMAL_TEMP_INVALID)
+        thermal_zone_device_set_polling(tz, 
 msecs_to_jiffies(THERMAL_RECHECK_DELAY_MS));
   }
   static struct thermal_governor *thermal_get_tz_governor(struct thermal_zone_device *tz)
@@ -514,7 +516,7 @@ void __thermal_zone_device_update(struct
       update_temperature(tz);
       if (tz->temperature == THERMAL_TEMP_INVALID)
-        return;
+        goto monitor;
       tz->notify_event = event;
@@ -536,6 +538,7 @@ void __thermal_zone_device_update(struct
       thermal_debug_update_trip_stats(tz);
+monitor:
       monitor_thermal_zone(tz);
   }

Index: linux-pm/drivers/thermal/thermal_core.h
===================================================================

drivers/thermal/thermal_core.h +0 -0
@@ -133,6 +133,12 @@ struct thermal_zone_device {
       struct thermal_trip_desc trips[] __counted_by(num_trips);
   };

drivers/thermal/thermal_core.h struct thermal_zone_device { · line 144

Quoted 10 lines
drivers/thermal/thermal_core.h +6 -0
+/*
+ * Default delay after a failing thermal zone temperature check before
+ * attempting to check it again.
+ */
+#define THERMAL_RECHECK_DELAY_MS    100
+
   /* Default Thermal Governor */
   #if defined(CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE)
   #define DEFAULT_THERMAL_GOVERNOR       "step_wise"
 

This patch on next-20240702 makes Qualcomm HDK8350, HDK8450, QRD8550, HDK8560, QRD8650 & HDK8650 output in loop:

thermal thermal_zoneXX: failed to read out thermal zone (-19)

Boot logs or ARM64 defconfig:
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152439#L1393
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152440#L2200
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152442#L2828
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152441#L1862
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152443#L1776
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152444#L1723

Result of git bisect:
# bad: [82e4255305c554b0bb18b7ccf2db86041b4c8b6e] Add linux-next specific files for 20240702
# good: [22a40d14b572deb80c0648557f4bd502d7e83826] Linux 6.10-rc6
git bisect start 'FETCH_HEAD' 'v6.10-rc6'
# bad: [f6dfcf0e9567b57b93f2564966d9177f0d8dbe05] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git
git bisect bad f6dfcf0e9567b57b93f2564966d9177f0d8dbe05
# good: [7f86ae0c2dc19fea7be1da29b2bf03f085463ae7] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux.git
git bisect good 7f86ae0c2dc19fea7be1da29b2bf03f085463ae7
# bad: [077d5bbd75dd12af2096c96846ffc78ab5dd65b1] Merge branch 'devfreq-next' of git://git.kernel.org/pub/scm/linux/kernel/git/chanwoo/linux.git
git bisect bad 077d5bbd75dd12af2096c96846ffc78ab5dd65b1
# good: [271bcaf753d0afe2bd0386ab1e98132ee65b61ca] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux.git
git bisect good 271bcaf753d0afe2bd0386ab1e98132ee65b61ca
# good: [9758a2ee5316a6f8736ab4fd39a6f6176aa057ec] Merge branch 'hwmon-next' of git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging.git
git bisect good 9758a2ee5316a6f8736ab4fd39a6f6176aa057ec
# good: [e6bd69ea345045520bd63487b85a4b5676aff76b] Merge branch 'master' of git://linuxtv.org/mchehab/media-next.git
git bisect good e6bd69ea345045520bd63487b85a4b5676aff76b
# good: [46398edfb36e2882be5e86ea563b2db9138ae499] Merge branches 'pm-cpuidle' and 'pm-powercap' into linux-next
git bisect good 46398edfb36e2882be5e86ea563b2db9138ae499
# good: [d3927cbc52eed166f74ea7e031ed6384cc3d4d5f] Merge branch 'thermal-intel' into linux-next
git bisect good d3927cbc52eed166f74ea7e031ed6384cc3d4d5f
# good: [ce84b7beeb524e7b20983838687862454ba54df7] cpufreq: sti: add missing MODULE_DEVICE_TABLE entry for stih418
git bisect good ce84b7beeb524e7b20983838687862454ba54df7
# bad: [fcf61315d38d41f4e55856b179f9e5538e299ef4] Merge branch 'thermal-fixes' into linux-next
git bisect bad fcf61315d38d41f4e55856b179f9e5538e299ef4
# good: [4262b8d782a74c7cf7b8b94ed9e4fcb94e856d1e] dt-bindings: thermal: mediatek: Fix thermal zone definition for MT8186
git bisect good 4262b8d782a74c7cf7b8b94ed9e4fcb94e856d1e
# good: [7eeb114a635a04bea2fa7d57cedbf374c714d29e] dt-bindings: thermal: convert hisilicon-thermal.txt to dt-schema
git bisect good 7eeb114a635a04bea2fa7d57cedbf374c714d29e
# good: [107ac0d49ae6a86b4986146b9a612294f7e34406] Merge branch 'thermal/linux-next' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/thermal/linux into linux-next git bisect good 107ac0d49ae6a86b4986146b9a612294f7e34406
# bad: [5725f40698b9ba7f84fbfee25b9059ba044c4b86] thermal: core: Call monitor_thermal_zone() if zone temperature is invalid
git bisect bad 5725f40698b9ba7f84fbfee25b9059ba044c4b86

# first bad commit: [5725f40698b9ba7f84fbfee25b9059ba044c4b86] thermal: core: Call monitor_thermal_zone() if zone temperature is invalid

#regzbot introduced: 5725f40698b9ba7f84fbfee25b9059ba044c4b86

Thanks,
Neil

Signature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog
#6 lore raw

Hi,

Quoted 8 lines · Daniel Lezcano

Hi Neil,

it seems there is something wrong with the driver actually.

There can be a moment where the sensor is not yet initialized for different reason, so reading the temperature fails. The routine will just retry until the sensor gets ready.

Having these errors seem to me that the sensor for this specific thermal zone is never ready which may be the root cause of your issue. The change is spotting this problem IMO.

Probably, but it gets printed every second until system shutdown, but only for a single thermal_zone.

Using v1 of Rafael's patch makes the message disappear completely.

Neil

Quoted 134 lines · Neil Armstrong
Quoted 130 lines

Hi,

Quoted 66 lines · Rafael J. Wysocki

From: Rafael J. Wysocki <[email protected]>

Commit 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid") caused __thermal_zone_device_update() to return early if the current thermal zone temperature was invalid.

This was done to avoid running handle_thermal_trip() and governor callbacks in that case which led to confusion.  However, it went too far because monitor_thermal_zone() still needs to be called even when the zone temperature is invalid to ensure that it will be updated eventually in case thermal polling is enabled and the driver has no other means to notify the core of zone temperature changes (for example, it does not register an interrupt handler or ACPI notifier).

Also if the .set_trips() zone callback is expected to set up monitoring interrupts for a thermal zone, it has to be provided with valid boundaries and that can only happen if the zone temperature is known.

Accordingly, to ensure that __thermal_zone_device_update() will run again after a failing zone temperature check, make it call monitor_thermal_zone() regardless of whether or not the zone temperature is valid and make the latter schedule a thermal zone temperature update if the zone temperature is invalid even if polling is not enabled for the thermal zone.

Fixes 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid")

---
  drivers/thermal/thermal_core.c |    5 ++++-
  drivers/thermal/thermal_core.h |    6 ++++++

  2 files changed, 10 insertions(+), 1 deletion(-)

Index: linux-pm/drivers/thermal/thermal_core.c
===================================================================

drivers/thermal/thermal_core.c +4 -1
@@ -300,6 +300,8 @@ static void monitor_thermal_zone(struct
           thermal_zone_device_set_polling(tz, tz->passive_delay_jiffies);
       else if (tz->polling_delay_jiffies)
           thermal_zone_device_set_polling(tz, tz->polling_delay_jiffies);
+    else if (tz->temperature == THERMAL_TEMP_INVALID)
+        thermal_zone_device_set_polling(tz, msecs_to_jiffies(THERMAL_RECHECK_DELAY_MS));
   }
   static struct thermal_governor *thermal_get_tz_governor(struct thermal_zone_device *tz)
@@ -514,7 +516,7 @@ void __thermal_zone_device_update(struct
       update_temperature(tz);
       if (tz->temperature == THERMAL_TEMP_INVALID)
-        return;
+        goto monitor;
       tz->notify_event = event;
@@ -536,6 +538,7 @@ void __thermal_zone_device_update(struct
       thermal_debug_update_trip_stats(tz);
+monitor:
       monitor_thermal_zone(tz);
   }

Index: linux-pm/drivers/thermal/thermal_core.h
===================================================================

drivers/thermal/thermal_core.h +0 -0
@@ -133,6 +133,12 @@ struct thermal_zone_device {
       struct thermal_trip_desc trips[] __counted_by(num_trips);
   };

drivers/thermal/thermal_core.h struct thermal_zone_device { · line 144

Quoted 10 lines
drivers/thermal/thermal_core.h +6 -0
+/*
+ * Default delay after a failing thermal zone temperature check before
+ * attempting to check it again.
+ */
+#define THERMAL_RECHECK_DELAY_MS    100
+
   /* Default Thermal Governor */
   #if defined(CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE)
   #define DEFAULT_THERMAL_GOVERNOR       "step_wise"
 

This patch on next-20240702 makes Qualcomm HDK8350, HDK8450, QRD8550, HDK8560, QRD8650 & HDK8650 output in loop:

thermal thermal_zoneXX: failed to read out thermal zone (-19)

Boot logs or ARM64 defconfig:
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152439#L1393
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152440#L2200
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152442#L2828
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152441#L1862
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152443#L1776
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/152444#L1723

Result of git bisect:
# bad: [82e4255305c554b0bb18b7ccf2db86041b4c8b6e] Add linux-next specific files for 20240702
# good: [22a40d14b572deb80c0648557f4bd502d7e83826] Linux 6.10-rc6
git bisect start 'FETCH_HEAD' 'v6.10-rc6'
# bad: [f6dfcf0e9567b57b93f2564966d9177f0d8dbe05] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git
git bisect bad f6dfcf0e9567b57b93f2564966d9177f0d8dbe05
# good: [7f86ae0c2dc19fea7be1da29b2bf03f085463ae7] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux.git
git bisect good 7f86ae0c2dc19fea7be1da29b2bf03f085463ae7
# bad: [077d5bbd75dd12af2096c96846ffc78ab5dd65b1] Merge branch 'devfreq-next' of git://git.kernel.org/pub/scm/linux/kernel/git/chanwoo/linux.git
git bisect bad 077d5bbd75dd12af2096c96846ffc78ab5dd65b1
# good: [271bcaf753d0afe2bd0386ab1e98132ee65b61ca] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux.git
git bisect good 271bcaf753d0afe2bd0386ab1e98132ee65b61ca
# good: [9758a2ee5316a6f8736ab4fd39a6f6176aa057ec] Merge branch 'hwmon-next' of git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging.git
git bisect good 9758a2ee5316a6f8736ab4fd39a6f6176aa057ec
# good: [e6bd69ea345045520bd63487b85a4b5676aff76b] Merge branch 'master' of git://linuxtv.org/mchehab/media-next.git
git bisect good e6bd69ea345045520bd63487b85a4b5676aff76b
# good: [46398edfb36e2882be5e86ea563b2db9138ae499] Merge branches 'pm-cpuidle' and 'pm-powercap' into linux-next
git bisect good 46398edfb36e2882be5e86ea563b2db9138ae499
# good: [d3927cbc52eed166f74ea7e031ed6384cc3d4d5f] Merge branch 'thermal-intel' into linux-next
git bisect good d3927cbc52eed166f74ea7e031ed6384cc3d4d5f
# good: [ce84b7beeb524e7b20983838687862454ba54df7] cpufreq: sti: add missing MODULE_DEVICE_TABLE entry for stih418
git bisect good ce84b7beeb524e7b20983838687862454ba54df7
# bad: [fcf61315d38d41f4e55856b179f9e5538e299ef4] Merge branch 'thermal-fixes' into linux-next
git bisect bad fcf61315d38d41f4e55856b179f9e5538e299ef4
# good: [4262b8d782a74c7cf7b8b94ed9e4fcb94e856d1e] dt-bindings: thermal: mediatek: Fix thermal zone definition for MT8186
git bisect good 4262b8d782a74c7cf7b8b94ed9e4fcb94e856d1e
# good: [7eeb114a635a04bea2fa7d57cedbf374c714d29e] dt-bindings: thermal: convert hisilicon-thermal.txt to dt-schema
git bisect good 7eeb114a635a04bea2fa7d57cedbf374c714d29e
# good: [107ac0d49ae6a86b4986146b9a612294f7e34406] Merge branch 'thermal/linux-next' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/thermal/linux into linux-next git bisect good 107ac0d49ae6a86b4986146b9a612294f7e34406
# bad: [5725f40698b9ba7f84fbfee25b9059ba044c4b86] thermal: core: Call monitor_thermal_zone() if zone temperature is invalid
git bisect bad 5725f40698b9ba7f84fbfee25b9059ba044c4b86

# first bad commit: [5725f40698b9ba7f84fbfee25b9059ba044c4b86] thermal: core: Call monitor_thermal_zone() if zone temperature is invalid

#regzbot introduced: 5725f40698b9ba7f84fbfee25b9059ba044c4b86

Thanks,
Neil

#7 lore raw
Quoted 8 lines · On 03/07/2024 14:43, [email protected] wrote:

Hi,

Quoted 5 lines · Daniel Lezcano

Hi Neil,

it seems there is something wrong with the driver actually.

Quoted 7 lines
Quoted 3 lines

There can be a moment where the sensor is not yet initialized for different reason, so reading the temperature fails. The routine will just retry until the sensor gets ready.

Having these errors seem to me that the sensor for this specific thermal zone is never ready which may be the root cause of your issue. The change is spotting this problem IMO.

Probably, but it gets printed every second until system shutdown, but only for a single thermal_zone.

Using v1 of Rafael's patch makes the message disappear completely.

Yes, because you have probably the thermal zone polling delay set to zero, thus it fails the first time and does no longer try to set it up again. The V1 is an incomplete fix.

Very likely the problem is in the sensor platform driver, or in the thermal zone description in the device tree which describes a non functional thermal zone.

Signature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog
#8 lore raw
Quoted 13 lines · Daniel Lezcano
Quoted 12 lines · On 03/07/2024 14:43, [email protected] wrote:

Hi,

Quoted 8 lines · Daniel Lezcano

Hi Neil,

it seems there is something wrong with the driver actually.

There can be a moment where the sensor is not yet initialized for different reason, so reading the temperature fails. The routine will just retry until the sensor gets ready.

Having these errors seem to me that the sensor for this specific thermal zone is never ready which may be the root cause of your issue. The change is spotting this problem IMO.

Quoted 8 lines
Quoted 3 lines

Probably, but it gets printed every second until system shutdown, but only for a single thermal_zone.

Using v1 of Rafael's patch makes the message disappear completely.

Yes, because you have probably the thermal zone polling delay set to zero, thus it fails the first time and does no longer try to set it up again. The V1 is an incomplete fix.

Very likely the problem is in the sensor platform driver, or in the thermal zone description in the device tree which describes a non functional thermal zone.

It was at 0 but the delay was removed recently:
https://lore.kernel.org/all/[email protected]/

That doesn't explain it because only the last platforms have this error message printed.

Neil

#9 lore raw
Quoted 18 lines · On 03/07/2024 16:42, [email protected] wrote:
Quoted 17 lines · Daniel Lezcano
Quoted 15 lines · On 03/07/2024 14:43, [email protected] wrote:

Hi,

Quoted 8 lines · Daniel Lezcano

Hi Neil,

it seems there is something wrong with the driver actually.

There can be a moment where the sensor is not yet initialized for different reason, so reading the temperature fails. The routine will just retry until the sensor gets ready.

Having these errors seem to me that the sensor for this specific thermal zone is never ready which may be the root cause of your issue. The change is spotting this problem IMO.

Probably, but it gets printed every second until system shutdown, but only for a single thermal_zone.

Using v1 of Rafael's patch makes the message disappear completely.

Quoted 7 lines
Quoted 4 lines

Yes, because you have probably the thermal zone polling delay set to zero, thus it fails the first time and does no longer try to set it up again. The V1 is an incomplete fix.

Very likely the problem is in the sensor platform driver, or in the thermal zone description in the device tree which describes a non functional thermal zone.

It was at 0 but the delay was removed recently:
https://lore.kernel.org/all/[email protected]/

Yes, these changes are because another change did:

commit 488164006a281986d95abbc4b26e340c19c4c85b
Author: Konrad Dybcio <[email protected]>

thermal/of: Assume polling-delay(-passive) 0 when absent

drivers/thermal/thermal_of.c +0 -0
Quoted 1 line

That doesn't explain it because only the last platforms have this error message printed.

Let me recap.

It has been reported if a thermal-zone with zero delay fails to initialize because the sensor returns an error, then there is no more attempt to initialize it and the thermal zone won't be functional.

The provided fix will periodically read the sensor temperature until there is a valid temperature. When there is a valid temperature, then the interrupts are set for the previous and the next temperature thresholds. That leads to the end of the routine of initializing the thermal zone and cancels the timer.

The platforms you reported, the delay is zero (before and after the 'polling cleanup').

My hypothesis is the following:

The thermal-zone29 describes a sensor which does not operate.

Before the patch:

First attempt to initialize it, the temperature is invalid, then because the delay is zero, the routine stops, and there is no more attempts to initialize it. Nothing will happen to this thermal zone and it will stay stuck silently. So at this point, the thermal zone is broken and you don't notice it.

After the patch:

The initialization routine is constantly retrying to init the thermal zone.

-------------------

If you revert the fix and you try to read the thermal zone 29, it should always fail to return an error.

If I'm correct, then I suggest to identify what thermal zone is 29 (type file), identify the node name in the DT, find the tsens channel and double check if it really describes an existing sensor

Signature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog
#10 lore raw
Quoted 56 lines · Daniel Lezcano
Quoted 18 lines · On 03/07/2024 16:42, [email protected] wrote:
Quoted 17 lines · Daniel Lezcano
Quoted 15 lines · On 03/07/2024 14:43, [email protected] wrote:

Hi,

Quoted 8 lines · Daniel Lezcano

Hi Neil,

it seems there is something wrong with the driver actually.

There can be a moment where the sensor is not yet initialized for different reason, so reading the temperature fails. The routine will just retry until the sensor gets ready.

Having these errors seem to me that the sensor for this specific thermal zone is never ready which may be the root cause of your issue. The change is spotting this problem IMO.

Probably, but it gets printed every second until system shutdown, but only for a single thermal_zone.

Using v1 of Rafael's patch makes the message disappear completely.

Quoted 7 lines
Quoted 4 lines

Yes, because you have probably the thermal zone polling delay set to zero, thus it fails the first time and does no longer try to set it up again. The V1 is an incomplete fix.

Very likely the problem is in the sensor platform driver, or in the thermal zone description in the device tree which describes a non functional thermal zone.

It was at 0 but the delay was removed recently:
https://lore.kernel.org/all/[email protected]/

Yes, these changes are because another change did:

commit 488164006a281986d95abbc4b26e340c19c4c85b
Author: Konrad Dybcio <[email protected]>

    thermal/of: Assume polling-delay(-passive) 0 when absent

drivers/thermal/thermal_of.c +0 -0
Quoted 2 lines

That doesn't explain it because only the last platforms have this error message printed.

Let me recap.

It has been reported if a thermal-zone with zero delay fails to initialize because the sensor returns an error, then there is no more attempt to initialize it and the thermal zone won't be functional.

The provided fix will periodically read the sensor temperature until there is a valid temperature. When there is a valid temperature, then the interrupts are set for the previous and the next temperature thresholds. That leads to the end of the routine of initializing the thermal zone and cancels the timer.

The platforms you reported, the delay is zero (before and after the 'polling cleanup').

My hypothesis is the following:

The thermal-zone29 describes a sensor which does not operate.

Before the patch:

First attempt to initialize it, the temperature is invalid, then because the delay is zero, the routine stops, and there is no more attempts to initialize it. Nothing will happen to this thermal zone and it will stay stuck silently. So at this point, the thermal zone is broken and you don't notice it.

After the patch:

Quoted 10 lines

The initialization routine is constantly retrying to init the thermal zone.

-------------------

If you revert the fix and you try to read the thermal zone 29, it should always fail to return an error.

If I'm correct, then I suggest to identify what thermal zone is 29 (type file), identify the node name in the DT, find the tsens channel and double check if it really describes an existing sensor

OK I just found out, it's the qcom-battmgr-bat thermal zone, and in CI we do not have the firmwares so the temperature is never available, this is why it fails in a loop.

Before this patch it would fail silently, but would be useless if we start the firmware too late.

So since it's firmware based, valid data could arrive very late in the boot stage, and sending an error message in a loop until the firmware isn't started doesn't seem right.

I think Rafael's new patch is good, but perhaps it should send an error when it finally stops monitoring.

Neil

#11 lore raw

On 04/07/2024 09:39, [email protected] wrote:

[ ... ]

Quoted 7 lines

OK I just found out, it's the qcom-battmgr-bat thermal zone, and in CI we do not have the firmwares so the temperature is never available, this is why it fails in a loop.

Before this patch it would fail silently, but would be useless if we start the firmware too late.

So since it's firmware based, valid data could arrive very late in the boot stage, and sending an error message in a loop until the firmware isn't started doesn't seem right.

Yeah, there was a similar bug with iwlwifi. They fixed it by registering the thermal zone after the firmware was successfully loaded.

Is that possible to do the same ?

Quoted 1 line

I think Rafael's new patch is good, but perhaps it should send an error when it finally stops monitoring.

Signature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog
#12 lore raw
Quoted 7 lines · Daniel Lezcano

On 04/07/2024 09:39, [email protected] wrote:

[ ... ]

Quoted 3 lines

OK I just found out, it's the qcom-battmgr-bat thermal zone, and in CI we do not have the firmwares so the temperature is never available, this is why it fails in a loop.

Quoted 8 lines
Quoted 4 lines

Before this patch it would fail silently, but would be useless if we start the firmware too late.

So since it's firmware based, valid data could arrive very late in the boot stage, and sending an error message in a loop until the firmware isn't started doesn't seem right.

Yeah, there was a similar bug with iwlwifi. They fixed it by registering the thermal zone after the firmware was successfully loaded.

Is that possible to do the same ?

The thermal zone is indirect, it's registered via power_supply_core.

A tentative was done to delay registering the power supply , since it caused issues in suspend/resume, but it was reverted because it would require much more work:
https://lore.kernel.org/all/[email protected]/

Seems we should instead return -EAGAIN instead of -ENODEV in qcom_battmgr_bat_get_property(),

But I think power_supply_read_temp() should return -EAGAIN on -ENODEV, since it's the return code for when a power supply isn't initialized.

Neil

Quoted 5 lines
Quoted 2 lines

I think Rafael's new patch is good, but perhaps it should send an error when it finally stops monitoring.

#13 lore raw
In reply to [email protected]
Quoted 71 lines · On Thu, Jul 4, 2024 at 9:39 AM <[email protected]> wrote:
Quoted 56 lines · Daniel Lezcano
Quoted 18 lines · On 03/07/2024 16:42, [email protected] wrote:
Quoted 17 lines · Daniel Lezcano
Quoted 15 lines · On 03/07/2024 14:43, [email protected] wrote:

Hi,

Quoted 8 lines · Daniel Lezcano

Hi Neil,

it seems there is something wrong with the driver actually.

There can be a moment where the sensor is not yet initialized for different reason, so reading the temperature fails. The routine will just retry until the sensor gets ready.

Having these errors seem to me that the sensor for this specific thermal zone is never ready which may be the root cause of your issue. The change is spotting this problem IMO.

Probably, but it gets printed every second until system shutdown, but only for a single thermal_zone.

Using v1 of Rafael's patch makes the message disappear completely.

Quoted 7 lines
Quoted 4 lines

Yes, because you have probably the thermal zone polling delay set to zero, thus it fails the first time and does no longer try to set it up again. The V1 is an incomplete fix.

Very likely the problem is in the sensor platform driver, or in the thermal zone description in the device tree which describes a non functional thermal zone.

It was at 0 but the delay was removed recently:
https://lore.kernel.org/all/[email protected]/

Yes, these changes are because another change did:

commit 488164006a281986d95abbc4b26e340c19c4c85b
Author: Konrad Dybcio <[email protected]>

thermal/of: Assume polling-delay(-passive) 0 when absent

drivers/thermal/thermal_of.c +0 -0
Quoted 2 lines

That doesn't explain it because only the last platforms have this error message printed.

Let me recap.

It has been reported if a thermal-zone with zero delay fails to initialize because the sensor returns an error, then there is no more attempt to initialize it and the thermal zone won't be functional.

The provided fix will periodically read the sensor temperature until there is a valid temperature. When there is a valid temperature, then the interrupts are set for the previous and the next temperature thresholds. That leads to the end of the routine of initializing the thermal zone and cancels the timer.

The platforms you reported, the delay is zero (before and after the 'polling cleanup').

My hypothesis is the following:

The thermal-zone29 describes a sensor which does not operate.

Before the patch:

First attempt to initialize it, the temperature is invalid, then because the delay is zero, the routine stops, and there is no more attempts to initialize it. Nothing will happen to this thermal zone and it will stay stuck silently. So at this point, the thermal zone is broken and you don't notice it.

After the patch:

Quoted 10 lines

The initialization routine is constantly retrying to init the thermal zone.

-------------------

If you revert the fix and you try to read the thermal zone 29, it should always fail to return an error.

If I'm correct, then I suggest to identify what thermal zone is 29 (type file), identify the node name in the DT, find the tsens channel and double check if it really describes an existing sensor

OK I just found out, it's the qcom-battmgr-bat thermal zone, and in CI we do not have the firmwares so the temperature is never available, this is why it fails in a loop.

Quoted 6 lines

Before this patch it would fail silently, but would be useless if we start the firmware too late.

So since it's firmware based, valid data could arrive very late in the boot stage, and sending an error message in a loop until the firmware isn't started doesn't seem right.

I think Rafael's new patch is good, but perhaps it should send an error when it finally stops monitoring.

Do you mean do something in addition to printing the message? It can do a couple of things. For instance, it could disable the thermal zone which would also cause a netlink message to be sent. However, I'd rather send another patch for this for the next cycle because we are late in the current one and I'd rather stay on the conservative side of things ATM.

Or do you mean the pr_info() log level is too low for this message?

Anyway, I'm going to submit the patch officially as is and please feel free to send comments on that submission.

#14 lore raw
In reply to Daniel Lezcano
Quoted 23 lines · Daniel Lezcano
Quoted 12 lines · On 03/07/2024 14:43, [email protected] wrote:

Hi,

Quoted 9 lines · Daniel Lezcano

Hi Neil,

it seems there is something wrong with the driver actually.

There can be a moment where the sensor is not yet initialized for different reason, so reading the temperature fails. The routine will just retry until the sensor gets ready.

Quoted 8 lines
Quoted 3 lines

Having these errors seem to me that the sensor for this specific thermal zone is never ready which may be the root cause of your issue. The change is spotting this problem IMO.

Probably, but it gets printed every second until system shutdown, but only for a single thermal_zone.

Using v1 of Rafael's patch makes the message disappear completely.

Quoted 7 lines

Yes, because you have probably the thermal zone polling delay set to zero, thus it fails the first time and does no longer try to set it up again. The V1 is an incomplete fix.

Very likely the problem is in the sensor platform driver, or in the thermal zone description in the device tree which describes a non functional thermal zone.

I agree, but polling this useless thermal zone forever is not particularly useful.

I was kind of afraid that something like this would happen, but then I didn't want to complicate the patch unnecessarily until I knew that it really would happen.

So attached is a modification of the $subject patch that will double the temperature recheck delay after every failed attempt to get the zone temperature and it will give up eventually (in this particular version, after the recheck delay exceeds 30 s).

I would appreciate giving it a go (obviously, by replacing the $subject one with it).

From: Rafael J. Wysocki <[email protected]>
Subject: [PATCH v3] thermal: core: Call monitor_thermal_zone() if zone temperature is invalid

Commit 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid") caused __thermal_zone_device_update() to return early if the current thermal zone temperature was invalid.

This was done to avoid running handle_thermal_trip() and governor callbacks in that case which led to confusion. However, it went too far because monitor_thermal_zone() still needs to be called even when the zone temperature is invalid to ensure that it will be updated eventually in case thermal polling is enabled and the driver has no other means to notify the core of zone temperature changes (for example, it does not register an interrupt handler or ACPI notifier).

Also if the .set_trips() zone callback is expected to set up monitoring interrupts for a thermal zone, it needs to be provided with valid boundaries and that can only be done if the zone temperature is known.

Accordingly, to ensure that __thermal_zone_device_update() will run again after a failing zone temperature check, make it call monitor_thermal_zone() regardless of whether or not the zone temperature is valid and make the latter schedule a thermal zone temperature update if the zone temperature is invalid even if polling is not enabled for the thermal zone (however, if this continues to fail, give up after some time).

Fixes 202aa0d4bb53 ("thermal: core: Do not call handle_thermal_trip() if zone temperature is invalid")
---
 drivers/thermal/thermal_core.c |   13 ++++++++++++-
 drivers/thermal/thermal_core.h |    9 +++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)
drivers/thermal/thermal_core.c +12 -1
Index: linux-pm/drivers/thermal/thermal_core.c
===================================================================
@@ -300,6 +300,14 @@ static void monitor_thermal_zone(struct
 		thermal_zone_device_set_polling(tz, tz->passive_delay_jiffies);
 	else if (tz->polling_delay_jiffies)
 		thermal_zone_device_set_polling(tz, tz->polling_delay_jiffies);
+	else if (tz->temperature == THERMAL_TEMP_INVALID &&
+		 tz->recheck_delay_jiffies <= THERMAL_MAX_RECHECK_DELAY) {
+		thermal_zone_device_set_polling(tz, tz->recheck_delay_jiffies);
+		/* Double the recheck delay for the next attempt. */
+		tz->recheck_delay_jiffies += tz->recheck_delay_jiffies;
+		if (tz->recheck_delay_jiffies > THERMAL_MAX_RECHECK_DELAY)
+			dev_info(&tz->device, "Temperature unknown, giving up\n");
+	}
 }
 
 static struct thermal_governor *thermal_get_tz_governor(struct thermal_zone_device *tz)
@@ -430,6 +438,7 @@ static void update_temperature(struct th
 
 	tz->last_temperature = tz->temperature;
 	tz->temperature = temp;
+	tz->recheck_delay_jiffies = 1;
 
 	trace_thermal_temperature(tz);
 
@@ -514,7 +523,7 @@ void __thermal_zone_device_update(struct
 	update_temperature(tz);
 
 	if (tz->temperature == THERMAL_TEMP_INVALID)
-		return;
+		goto monitor;
 
 	tz->notify_event = event;
 
@@ -536,6 +545,7 @@ void __thermal_zone_device_update(struct
 
 	thermal_debug_update_trip_stats(tz);
 
+monitor:
 	monitor_thermal_zone(tz);
 }
 
@@ -1438,6 +1448,7 @@ thermal_zone_device_register_with_trips(
 
 	thermal_set_delay_jiffies(&tz->passive_delay_jiffies, passive_delay);
 	thermal_set_delay_jiffies(&tz->polling_delay_jiffies, polling_delay);
+	tz->recheck_delay_jiffies = 1;
 
 	/* sys I/F */
 	/* Add nodes that are always present via .groups */
drivers/thermal/thermal_core.h +9 -0
Index: linux-pm/drivers/thermal/thermal_core.h
===================================================================
@@ -67,6 +67,8 @@ struct thermal_governor {
  * @polling_delay_jiffies: number of jiffies to wait between polls when
  *			checking whether trip points have been crossed (0 for
  *			interrupt driven systems)
+ * @recheck_delay_jiffies: delay after a failed thermal zone temperature check
+ * 			before attempting to check it again
  * @temperature:	current temperature.  This is only for core code,
  *			drivers should use thermal_zone_get_temp() to get the
  *			current temperature
@@ -108,6 +110,7 @@ struct thermal_zone_device {
 	int num_trips;
 	unsigned long passive_delay_jiffies;
 	unsigned long polling_delay_jiffies;
+	unsigned long recheck_delay_jiffies;
 	int temperature;
 	int last_temperature;
 	int emul_temperature;
@@ -133,6 +136,12 @@ struct thermal_zone_device {
 	struct thermal_trip_desc trips[] __counted_by(num_trips);
 };
 
+/*
+ * Maximum delay after a failing thermal zone temperature check before
+ * attempting to check it again (in jiffies).
+ */
+#define THERMAL_MAX_RECHECK_DELAY	(30 * HZ)
+
 /* Default Thermal Governor */
 #if defined(CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE)
 #define DEFAULT_THERMAL_GOVERNOR       "step_wise"