Add driver for NVIDIA Tegra30 SoC Thermal sensor

All NVIDIA Tegra30 SoCs have a two-channel on-chip sensor unit which monitors temperature and voltage of the SoC. Sensors control CPU frequency throttling, which is activated by hardware once preprogrammed temperature level is breached, they also send signal to Power Management controller to perform emergency shutdown on a critical overheat of the SoC die. Add driver for the Tegra30 TSENSOR module, exposing it as a thermal sensor and a cooling device.

Tested-by Andreas Westman Dorcsak <[email protected]> # Asus TF700T

Tested-by Maxim Schwalm <[email protected]> # Asus TF700T

Tested-by Svyatoslav Ryhel <[email protected]> # Asus TF201T

Tested-by Ihor Didenko <[email protected]> # Asus TF300T

Tested-by Ion Agorria <[email protected]> # Asus TF201T

Tested-by Matt Merhar <[email protected]> # Ouya

Tested-by Peter Geis <[email protected]> # Ouya

Signed-off-by Dmitry Osipenko <[email protected]>

---
 drivers/thermal/tegra/Kconfig           |   7 +
 drivers/thermal/tegra/Makefile          |   1 +
 drivers/thermal/tegra/tegra30-tsensor.c | 736 ++++++++++++++++++++++++
 3 files changed, 744 insertions(+)
 create mode 100644 drivers/thermal/tegra/tegra30-tsensor.c

All NVIDIA Tegra30 SoCs have a two-channel on-chip sensor unit which monitors temperature and voltage of the SoC. Sensors control CPU frequency throttling, which is activated by hardware once preprogrammed temperature level is breached, they also send signal to Power Management controller to perform emergency shutdown on a critical overheat of the SoC die. Add driver for the Tegra30 TSENSOR module, exposing it as a thermal sensor and a cooling device.

[Cc Viresh]

Quoted 7 lines · Dmitry Osipenko

All NVIDIA Tegra30 SoCs have a two-channel on-chip sensor unit which monitors temperature and voltage of the SoC. Sensors control CPU frequency throttling, which is activated by hardware once preprogrammed temperature level is breached, they also send signal to Power Management controller to perform emergency shutdown on a critical overheat of the SoC die. Add driver for the Tegra30 TSENSOR module, exposing it as a thermal sensor and a cooling device.

IMO it does not make sense to expose the hardware throttling mechanism as a cooling device because it is not usable anywhere from the thermal framework.

Moreover, that will collide with the thermal / cpufreq framework mitigation (hardware sets the frequency but the software thinks the freq is different), right ?

The hardware limiter should let know the cpufreq framework about the frequency change.

	https://lkml.org/lkml/2021/6/8/1792

May be post the sensor without the hw limiter for now and address that in a separate series ?

Quoted 12 lines

[Cc Viresh]

Quoted 7 lines · Dmitry Osipenko

All NVIDIA Tegra30 SoCs have a two-channel on-chip sensor unit which monitors temperature and voltage of the SoC. Sensors control CPU frequency throttling, which is activated by hardware once preprogrammed temperature level is breached, they also send signal to Power Management controller to perform emergency shutdown on a critical overheat of the SoC die. Add driver for the Tegra30 TSENSOR module, exposing it as a thermal sensor and a cooling device.

Quoted 7 lines

IMO it does not make sense to expose the hardware throttling mechanism as a cooling device because it is not usable anywhere from the thermal framework.

Moreover, that will collide with the thermal / cpufreq framework mitigation (hardware sets the frequency but the software thinks the freq is different), right ?

I am not even sure what the cooling device is doing here:

tegra_tsensor_set_cur_state() is not implemented and it says hardware changed it by itself. What is the benefit you are getting out of the cooling device here ?

Quoted 7 lines

The hardware limiter should let know the cpufreq framework about the frequency change.

	https://lkml.org/lkml/2021/6/8/1792

May be post the sensor without the hw limiter for now and address that in a separate series ?

15.06.2021 13:26, Viresh Kumar пишет:

Quoted 13 lines · Daniel Lezcano

Quoted 12 lines

[Cc Viresh]

Quoted 7 lines · Dmitry Osipenko

All NVIDIA Tegra30 SoCs have a two-channel on-chip sensor unit which monitors temperature and voltage of the SoC. Sensors control CPU frequency throttling, which is activated by hardware once preprogrammed temperature level is breached, they also send signal to Power Management controller to perform emergency shutdown on a critical overheat of the SoC die. Add driver for the Tegra30 TSENSOR module, exposing it as a thermal sensor and a cooling device.

Quoted 7 lines

Quoted 7 lines

IMO it does not make sense to expose the hardware throttling mechanism as a cooling device because it is not usable anywhere from the thermal framework.

Moreover, that will collide with the thermal / cpufreq framework mitigation (hardware sets the frequency but the software thinks the freq is different), right ?

H/w mitigation is additional and should be transparent to the software mitigation. The software mitigation is much more flexible, but it has latency. Software also could crash and hang.

Hardware mitigation doesn't have latency and it will continue to work regardless of the software state.

The CCF driver is aware about the h/w cooling status [1], hence software sees the actual frequency.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit?id=344d5df34f5abd468267daa98f041abf90b2f660

Quoted 5 lines

I am not even sure what the cooling device is doing here:

tegra_tsensor_set_cur_state() is not implemented and it says hardware changed it by itself. What is the benefit you are getting out of the cooling device here ?

It allows userspace to check whether hardware cooling is active via the cooling_device sysfs. Otherwise we don't have ability to check whether h/w cooling is active, I think it's a useful information. It's also interesting to see the cooling_device stats, showing how many times h/w mitigation was active.

Quoted 8 lines

Quoted 7 lines

The hardware limiter should let know the cpufreq framework about the frequency change.

	https://lkml.org/lkml/2021/6/8/1792

May be post the sensor without the hw limiter for now and address that in a separate series ?

I wasn't aware about existence of the thermal pressure, thank you for pointing at it. At a quick glance it should be possible to benefit from the information about the additional pressure.

Seems the current thermal pressure API assumes that there is only one user of the API. So it's impossible to aggregate the pressure from different sources, like software cpufreq pressure + h/w freq pressure. Correct? If yes, then please let me know yours thoughts about the best approach of supporting the aggregation.

I'll factor out the h/w limiter from this patchset and prepare the v4. Thank you all for taking a look at the patches.

Quoted 22 lines

15.06.2021 13:26, Viresh Kumar пишет:

Quoted 13 lines · Daniel Lezcano

Quoted 12 lines

[Cc Viresh]

Quoted 7 lines · Dmitry Osipenko

All NVIDIA Tegra30 SoCs have a two-channel on-chip sensor unit which monitors temperature and voltage of the SoC. Sensors control CPU frequency throttling, which is activated by hardware once preprogrammed temperature level is breached, they also send signal to Power Management controller to perform emergency shutdown on a critical overheat of the SoC die. Add driver for the Tegra30 TSENSOR module, exposing it as a thermal sensor and a cooling device.

Quoted 7 lines

Quoted 7 lines

IMO it does not make sense to expose the hardware throttling mechanism as a cooling device because it is not usable anywhere from the thermal framework.

Moreover, that will collide with the thermal / cpufreq framework mitigation (hardware sets the frequency but the software thinks the freq is different), right ?

Quoted 6 lines

H/w mitigation is additional and should be transparent to the software mitigation. The software mitigation is much more flexible, but it has latency. Software also could crash and hang.

Hardware mitigation doesn't have latency and it will continue to work regardless of the software state.

Yes, I agree. Both solutions have their pros and cons. However, I don't think they can co-exist sanely.

Quoted 5 lines

The CCF driver is aware about the h/w cooling status [1], hence software sees the actual frequency.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit?id=344d5df34f5abd468267daa98f041abf90b2f660

Ah interesting, thanks for the pointer.

What I'm worried about is the consistency with cpufreq.

Probably cpufreq_update_limits() should be called from the interrupt handler.

Quoted 11 lines

Quoted 5 lines

I am not even sure what the cooling device is doing here:

tegra_tsensor_set_cur_state() is not implemented and it says hardware changed it by itself. What is the benefit you are getting out of the cooling device here ?

It allows userspace to check whether hardware cooling is active via the cooling_device sysfs. Otherwise we don't have ability to check whether h/w cooling is active, I think it's a useful information. It's also interesting to see the cooling_device stats, showing how many times h/w mitigation was active.

Actually the stats are for software mitigation. For the driver, create a debugfs entry like what do the other drivers or a module parameter with the stats.

Quoted 13 lines

Quoted 8 lines

Quoted 7 lines

The hardware limiter should let know the cpufreq framework about the frequency change.

	https://lkml.org/lkml/2021/6/8/1792

May be post the sensor without the hw limiter for now and address that in a separate series ?

I wasn't aware about existence of the thermal pressure, thank you for pointing at it. At a quick glance it should be possible to benefit from the information about the additional pressure.

Quoted 5 lines

Seems the current thermal pressure API assumes that there is only one user of the API. So it's impossible to aggregate the pressure from different sources, like software cpufreq pressure + h/w freq pressure. Correct? If yes, then please let me know yours thoughts about the best approach of supporting the aggregation.

That is a good question. IMO, first step would be to call cpufreq_update_limits().

[ Cc Thara who implemented the thermal pressure ]

May be Thara has an idea about how to aggregate both? There is another series floating around with hardware limiter [1] and the same problematic.

[1] https://lkml.org/lkml/2021/6/8/1791

15.06.2021 19:18, Daniel Lezcano пишет:

Quoted 39 lines · Dmitry Osipenko

Quoted 22 lines

15.06.2021 13:26, Viresh Kumar пишет:

Quoted 13 lines · Daniel Lezcano

Quoted 12 lines

[Cc Viresh]

Quoted 7 lines · Dmitry Osipenko

All NVIDIA Tegra30 SoCs have a two-channel on-chip sensor unit which monitors temperature and voltage of the SoC. Sensors control CPU frequency throttling, which is activated by hardware once preprogrammed temperature level is breached, they also send signal to Power Management controller to perform emergency shutdown on a critical overheat of the SoC die. Add driver for the Tegra30 TSENSOR module, exposing it as a thermal sensor and a cooling device.

Quoted 7 lines

Quoted 7 lines

IMO it does not make sense to expose the hardware throttling mechanism as a cooling device because it is not usable anywhere from the thermal framework.

Moreover, that will collide with the thermal / cpufreq framework mitigation (hardware sets the frequency but the software thinks the freq is different), right ?

Quoted 6 lines

H/w mitigation is additional and should be transparent to the software mitigation. The software mitigation is much more flexible, but it has latency. Software also could crash and hang.

Hardware mitigation doesn't have latency and it will continue to work regardless of the software state.

Yes, I agree. Both solutions have their pros and cons. However, I don't think they can co-exist sanely.

Quoted 5 lines

The CCF driver is aware about the h/w cooling status [1], hence software sees the actual frequency.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit?id=344d5df34f5abd468267daa98f041abf90b2f660

Quoted 6 lines

Ah interesting, thanks for the pointer.

What I'm worried about is the consistency with cpufreq.

Probably cpufreq_update_limits() should be called from the interrupt handler.

IIUC, the cpufreq already should be prepared for the case where firmware may override frequency. Viresh, could you please clarify what are the possible implications of the frequency overriding?

Quoted 12 lines

Quoted 11 lines

Quoted 5 lines

I am not even sure what the cooling device is doing here:

tegra_tsensor_set_cur_state() is not implemented and it says hardware changed it by itself. What is the benefit you are getting out of the cooling device here ?

It allows userspace to check whether hardware cooling is active via the cooling_device sysfs. Otherwise we don't have ability to check whether h/w cooling is active, I think it's a useful information. It's also interesting to see the cooling_device stats, showing how many times h/w mitigation was active.

Quoted 3 lines

Actually the stats are for software mitigation. For the driver, create a debugfs entry like what do the other drivers or a module parameter with the stats.

Okay

Quoted 18 lines

Quoted 18 lines

Quoted 8 lines

Quoted 7 lines

The hardware limiter should let know the cpufreq framework about the frequency change.

	https://lkml.org/lkml/2021/6/8/1792

May be post the sensor without the hw limiter for now and address that in a separate series ?

I wasn't aware about existence of the thermal pressure, thank you for pointing at it. At a quick glance it should be possible to benefit from the information about the additional pressure.

Seems the current thermal pressure API assumes that there is only one user of the API. So it's impossible to aggregate the pressure from different sources, like software cpufreq pressure + h/w freq pressure. Correct? If yes, then please let me know yours thoughts about the best approach of supporting the aggregation.

Quoted 3 lines

That is a good question. IMO, first step would be to call cpufreq_update_limits().

Right

Quoted 5 lines

[ Cc Thara who implemented the thermal pressure ]

May be Thara has an idea about how to aggregate both? There is another series floating around with hardware limiter [1] and the same problematic.

Quoted 1 line

[1] https://lkml.org/lkml/2021/6/8/1791

Thanks, it indeed looks similar.

I guess the common thermal pressure update code could be moved out into a new special cpufreq thermal QoS handler (policy->thermal_constraints), where handler will select the frequency constraint and set up the pressure accordingly. So there won't be any races in the code.

Hi,

Thermal pressure is letting scheduler know that the max capacity available for a cpu to schedule tasks is reduced due to a thermal event. So you cannot have a h/w thermal pressure and s/w thermal pressure. There is eventually only one capping applied at h/w level and the frequency corresponding to this capping should be used for thermal pressure.

Ideally you should not be having both s/w and h/w trying to throttle at the same time. Why is this a scenario and what prevents you from disabling s/w throttling when h/w throttling is enabled. Now if there has to a aggregation for whatever reason this should be done at the thermal driver level and passed to scheduler.

Quoted 12 lines

Quoted 3 lines

That is a good question. IMO, first step would be to call cpufreq_update_limits().

Right

Quoted 6 lines

[ Cc Thara who implemented the thermal pressure ]

May be Thara has an idea about how to aggregate both? There is another series floating around with hardware limiter [1] and the same problematic.

Quoted 9 lines

Quoted 1 line

  [1] https://lkml.org/lkml/2021/6/8/1791

Thanks, it indeed looks similar.

I guess the common thermal pressure update code could be moved out into a new special cpufreq thermal QoS handler (policy->thermal_constraints), where handler will select the frequency constraint and set up the pressure accordingly. So there won't be any races in the code.

It was a conscious decision to keep thermal pressure update out of qos max freq update because there are platforms that don't use the qos framework. For eg acpi uses cpufreq_update_policy.
But you are right. We have two platforms now applying h/w throttling and cpufreq_cooling applying s/w throttling. So it does make sense to have one api doing all the computation to update thermal pressure. I am not sure how exactly/where exactly this will reside.

So for starters, I think you should replicate the update of thermal pressure in your h/w driver when you know that h/w is throttling/throttled the frequency. You can refer to cpufreq_cooling.c to see how it is done.

Moving to a common api can be done as a separate patch series.

16.06.2021 05:50, Thara Gopinath пишет:
...

Quoted 10 lines

Hi,

Thermal pressure is letting scheduler know that the max capacity available for a cpu to schedule tasks is reduced due to a thermal event. So you cannot have a h/w thermal pressure and s/w thermal pressure. There is eventually only one capping applied at h/w level and the frequency corresponding to this capping should be used for thermal pressure.

Quoted 5 lines

Ideally you should not be having both s/w and h/w trying to throttle at the same time. Why is this a scenario and what prevents you from disabling s/w throttling when h/w throttling is enabled. Now if there has to a aggregation for whatever reason this should be done at the thermal driver level and passed to scheduler.

Hello,

The h/w mitigation is much more reactive than software, in the same time it's much less flexible than software. It should provide additional protection in a cases where software isn't doing a good job. Ideally h/w mitigation should stay inactive all the time, nevertheless it should be modeled properly by the driver.

Quoted 22 lines

Quoted 13 lines

Quoted 4 lines

That is a good question. IMO, first step would be to call cpufreq_update_limits().

Right

Quoted 6 lines

[ Cc Thara who implemented the thermal pressure ]

May be Thara has an idea about how to aggregate both? There is another series floating around with hardware limiter [1] and the same problematic.

Quoted 9 lines

Quoted 1 line

  [1] https://lkml.org/lkml/2021/6/8/1791

Thanks, it indeed looks similar.

I guess the common thermal pressure update code could be moved out into a new special cpufreq thermal QoS handler (policy->thermal_constraints), where handler will select the frequency constraint and set up the pressure accordingly. So there won't be any races in the code.

Quoted 7 lines

It was a conscious decision to keep thermal pressure update out of qos max freq update because there are platforms that don't use the qos framework. For eg acpi uses cpufreq_update_policy.
But you are right. We have two platforms now applying h/w throttling and cpufreq_cooling applying s/w throttling. So it does make sense to have one api doing all the computation to update thermal pressure. I am not sure how exactly/where exactly this will reside.

The generic cpufreq_cooling already uses QoS for limiting the CPU frequency. It could be okay to use QoS for the OF drivers, this needs a closer look.

We have the case where CPU frequency is changed by the thermal event and the thermal pressure equation is the same for both s/w cpufreq_cooling and h/w thermal driver. The pressure is calculated based on the QoS cpufreq constraint that is already aggregated.

Hence what we may need to do on the thermal event is:

1. Update the QoS request
2. Update the thermal pressure
3. Ensure that updates are not racing

Quoted 8 lines

So for starters, I think you should replicate the update of thermal pressure in your h/w driver when you know that h/w is throttling/throttled the frequency. You can refer to cpufreq_cooling.c to see how it is done.

Moving to a common api can be done as a separate patch series.

Thank you for the clarification and suggestion.

IIUC, the cpufreq already should be prepared for the case where firmware may override frequency. Viresh, could you please clarify what are the possible implications of the frequency overriding?

+Vincent.

Quoted 3 lines · Dmitry Osipenko

IIUC, the cpufreq already should be prepared for the case where firmware may override frequency. Viresh, could you please clarify what are the possible implications of the frequency overriding?

The only implication is software would think hardware is running at some other frequency, while it is not. Not sure if something may break as a result of this.

The scheduler's view of CPUs will not be same though, i.e. scheduler will see capacity as X, while in reality it has changed to Y.

--
viresh

Quoted 12 lines

+Vincent.

Quoted 3 lines · Dmitry Osipenko

IIUC, the cpufreq already should be prepared for the case where firmware may override frequency. Viresh, could you please clarify what are the possible implications of the frequency overriding?

The only implication is software would think hardware is running at some other frequency, while it is not. Not sure if something may break as a result of this.

Quoted 2 lines

The scheduler's view of CPUs will not be same though, i.e. scheduler will see capacity as X, while in reality it has changed to Y.

thermal_pressure is used by scheduler to balance the load between CPUs according to the actual max frequency. If the thermal pressure doesn't reflect reality, scheduler will end up enqueuing too many tasks on a throttle CPU.

16.06.2021 11:30, Vincent Guittot пишет:

Quoted 16 lines · Viresh Kumar

Quoted 14 lines

+Vincent.

Quoted 3 lines · Dmitry Osipenko

IIUC, the cpufreq already should be prepared for the case where firmware may override frequency. Viresh, could you please clarify what are the possible implications of the frequency overriding?

The only implication is software would think hardware is running at some other frequency, while it is not. Not sure if something may break as a result of this.

The scheduler's view of CPUs will not be same though, i.e. scheduler will see capacity as X, while in reality it has changed to Y.

Quoted 4 lines

thermal_pressure is used by scheduler to balance the load between CPUs according to the actual max frequency. If the thermal pressure doesn't reflect reality, scheduler will end up enqueuing too many tasks on a throttle CPU.

What if all CPUs are throttled equally and running on the same frequency, will throttling have any effect on the scheduler decisions?

Quoted 23 lines

16.06.2021 11:30, Vincent Guittot пишет:

Quoted 20 lines · Viresh Kumar

Quoted 8 lines

+Vincent.

Quoted 3 lines · Dmitry Osipenko

IIUC, the cpufreq already should be prepared for the case where firmware may override frequency. Viresh, could you please clarify what are the possible implications of the frequency overriding?

Quoted 6 lines

The only implication is software would think hardware is running at some other frequency, while it is not. Not sure if something may break as a result of this.

The scheduler's view of CPUs will not be same though, i.e. scheduler will see capacity as X, while in reality it has changed to Y.

thermal_pressure is used by scheduler to balance the load between CPUs according to the actual max frequency. If the thermal pressure doesn't reflect reality, scheduler will end up enqueuing too many tasks on a throttle CPU.

Quoted 2 lines

What if all CPUs are throttled equally and running on the same frequency, will throttling have any effect on the scheduler decisions?

Yes, the capacity is also used to detect when CPUs have spare capacity or are already overloaded. We usually try to fill the spare capacity of a CPU (CPU's max capacity - current utilization) but he max capacity is reduced when the CPU is throttled, and the spare capacity doesn't exist but scheduler could try to it