Skip to contentSkip to conversation

cpufreq: Fix race between suspend/resume and CPU hotplug

PATCH In review Tianxiang Chen 3 participants 9 messages Apr 7 – May 21 CPU frequency scaling framework

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cpufreq: Fix race between suspend/resume and CPU hotplug

CPU hotplug operations can race with cpufreq_suspend() and cpufreq_resume(), leading to null pointer dereferences when accessing governor data. This occurs because there is no synchronization between suspend/resume operations and CPU hotplug, allowing concurrent access to policy->governor_data while it is being freed or initialized.

Detailed race condition scenario:

1. Thread A (cpufreq_suspend) starts execution:
- Iterates through active policies
- Calls cpufreq_stop_governor(policy) for each policy
- Sets cpufreq_suspended = true

2. Thread B (CPU hotplug) executes concurrently:
- Calls cpu_down(cpu)
- Calls cpuhp_cpufreq_offline(cpu)
- Calls cpufreq_offline(cpu)
- Inside cpufreq_offline():
* Stops governor: policy->governor->stop(policy)
* Exits governor: policy->governor->exit(policy)
* Frees governor_data: kfree(policy->governor_data)
* Sets policy->governor_data = NULL

3. Race window between step 1 and step 2:
- Thread A is iterating policies and stopping governors
- Thread B is concurrently executing CPU offline
- Both threads may access the same policy->governor_data
- Thread B frees governor_data while Thread A is still using it
- Thread A accesses freed governor_data → null pointer dereference

Similarly, cpufreq_resume() can race with CPU hotplug where governor_data is being initialized while hotplug is trying to access it, leading to accessing uninitialized data.

---
 drivers/cpufreq/cpufreq.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
drivers/cpufreq/cpufreq.c +7 -1
index 1f794524a1d9..8b03785764fa 100644
@@ -1979,6 +1979,7 @@ void cpufreq_suspend(void)
        if (!cpufreq_driver)
                return;
 
+       cpus_read_lock();
        if (!has_target() && !cpufreq_driver->suspend)
                goto suspend;
 
@@ -1998,6 +1999,7 @@ void cpufreq_suspend(void)
 
 suspend:
        cpufreq_suspended = true;
+       cpus_read_unlock();
 }
 
 /**
@@ -2017,10 +2019,11 @@ void cpufreq_resume(void)
        if (unlikely(!cpufreq_suspended))
                return;
 
+       cpus_read_lock();
        cpufreq_suspended = false;
 
        if (!has_target() && !cpufreq_driver->resume)
-               return;
+               goto out;
 
        pr_debug("%s: Resuming Governors\n", __func__);
 
@@ -2038,6 +2041,9 @@ void cpufreq_resume(void)
                                       __func__, policy->cpu);
                }
        }
+
+out:
+       cpus_read_unlock();
 }
 
 /**

--
2.34.1

#/******本邮件及其附件含有小米公司的保密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件! This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#

#1 lore raw
Quoted 4 lines · Tianxiang Chen

CPU hotplug operations can race with cpufreq_suspend() and cpufreq_resume(), leading to null pointer dereferences when accessing governor data.

So how exactly would CPU hotplug be started during a system suspend or resume?

Quoted 86 lines

This occurs because there is
no synchronization between suspend/resume operations and CPU hotplug, allowing concurrent access to policy->governor_data while it is being freed or initialized.

Detailed race condition scenario:

1. Thread A (cpufreq_suspend) starts execution:
- Iterates through active policies
- Calls cpufreq_stop_governor(policy) for each policy
- Sets cpufreq_suspended = true

2. Thread B (CPU hotplug) executes concurrently:
- Calls cpu_down(cpu)
- Calls cpuhp_cpufreq_offline(cpu)
- Calls cpufreq_offline(cpu)
- Inside cpufreq_offline():
* Stops governor: policy->governor->stop(policy)
* Exits governor: policy->governor->exit(policy)
* Frees governor_data: kfree(policy->governor_data)
* Sets policy->governor_data = NULL

3. Race window between step 1 and step 2:
- Thread A is iterating policies and stopping governors
- Thread B is concurrently executing CPU offline
- Both threads may access the same policy->governor_data
- Thread B frees governor_data while Thread A is still using it
- Thread A accesses freed governor_data → null pointer dereference

Similarly, cpufreq_resume() can race with CPU hotplug where governor_data is being initialized while hotplug is trying to access it, leading to accessing uninitialized data.

---
 drivers/cpufreq/cpufreq.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
drivers/cpufreq/cpufreq.c +7 -2
index 1f794524a1d9..8b03785764fa 100644
@@ -1979,6 +1979,7 @@ void cpufreq_suspend(void)
        if (!cpufreq_driver)
                return;
 
+       cpus_read_lock();
        if (!has_target() && !cpufreq_driver->suspend)
                goto suspend;
 
@@ -1998,6 +1999,7 @@ void cpufreq_suspend(void)
 
 suspend:
        cpufreq_suspended = true;
+       cpus_read_unlock();
 }
 
 /**
@@ -2017,10 +2019,11 @@ void cpufreq_resume(void)
        if (unlikely(!cpufreq_suspended))
                return;
 
+       cpus_read_lock();
        cpufreq_suspended = false;
 
        if (!has_target() && !cpufreq_driver->resume)
-               return;
+               goto out;
 
        pr_debug("%s: Resuming Governors\n", __func__);
 
@@ -2038,6 +2041,9 @@ void cpufreq_resume(void)
                                       __func__, policy->cpu);
                }
        }
+
+out:
+       cpus_read_unlock();
 }
 
 /**
--

2.34.1

#/******本邮件及其附件含有小米公司的保密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件! This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#

#2 lore raw

Re: [PATCH] cpufreq: fix race between hotplug and suspend

Quoted 1 line · Rafael J. Wysocki

So how exactly would CPU hotplug be started during a system suspend or resume?

Hi Rafael,

Thank you for your question. Let me explain the two scenarios:

1. cpufreq_suspend() During Reboot (Confirmed Issue)

The real and reproducible race I encountered occurs during system reboot.

Call chain:
kernel_restart() -> kernel_restart_prepare()

            -> device_shutdown() -> cpufreq_suspend()

Different from the regular suspend path, the reboot path does NOT call freeze_processes() at all.

All userspace processes, drivers and kernel threads are still running when cpufreq_suspend() executes. This allows CPU hotplug (offline/online) operations to run concurrently with cpufreq_suspend().

2. System suspend/resume (Less Likely but Possible)

CPU hotplug is less likely during system suspend/resume. However, non-freezable kernel threads may keep running throughout the entire process, which may still trigger CPU hotplug in theory.

So I added cpus_read_lock()/cpus_read_unlock() to block CPU hotplug while resume is in progress.

Signature · 2 lines
Thx and BRs,
Tianxiang Chen
#3 lore raw

Re: [PATCH] cpufreq: fix race between hotplug and suspend

Quoted 9 lines · Tianxiang Chen
Quoted 1 line · Rafael J. Wysocki

So how exactly would CPU hotplug be started during a system suspend or resume?

Hi Rafael,

Thank you for your question. Let me explain the two scenarios:

1. cpufreq_suspend() During Reboot (Confirmed Issue)

Which needs to be mentioned in the patch changelog.

Quoted 8 lines

The real and reproducible race I encountered occurs during system reboot.

Call chain:
kernel_restart() -> kernel_restart_prepare()

            -> device_shutdown() -> cpufreq_suspend()

Different from the regular suspend path, the reboot path does NOT call freeze_processes() at all.

That's correct.

Quoted 9 lines

All userspace processes, drivers and kernel threads are still running when cpufreq_suspend() executes. This allows CPU hotplug (offline/online) operations to run concurrently with cpufreq_suspend().

2. System suspend/resume (Less Likely but Possible)

CPU hotplug is less likely during system suspend/resume. However, non-freezable kernel threads may keep running throughout the entire process, which may still trigger CPU hotplug in theory.

Which would be a bug in the kernel thread in question. So not really.

Quoted 2 lines

So I added cpus_read_lock()/cpus_read_unlock() to block CPU hotplug while resume is in progress.

Please resend the patch with a changelog actually mentioning the failure that you have observed.

Thanks!

#4 lore raw

PATCH v2 1/1 cpufreq: Fix hotplug-suspend race during reboot

During system reboot, cpufreq_suspend() is called via the kernel_restart() -> device_shutdown() -> pm_notifier_call_chain() path. Unlike the normal system suspend path, the reboot path does not call freeze_processes(), so userspace processes and kernel threads remain active.

This allows CPU hotplug operations to run concurrently with cpufreq_suspend(). The original code has no synchronization with CPU hotplug, leading to a race condition where governor_data can be freed by the hotplug path while cpufreq_suspend() is still accessing it, resulting in a null pointer dereference:

  Unable to handle kernel NULL pointer dereference
  Call Trace:
   do_kernel_fault+0x28/0x3c
   cpufreq_suspend+0xdc/0x160
   device_shutdown+0x18/0x200
   kernel_restart+0x40/0x80
   arm64_sys_reboot+0x1b0/0x200

Fix this by adding cpus_read_lock()/cpus_read_unlock() to cpufreq_suspend() to block CPU hotplug operations while suspend is in progress.

---
v2:
- Update changelog to explicitly mention reboot scenario
- Add observed crash trace

---
 drivers/cpufreq/cpufreq.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/cpufreq/cpufreq.c +2 -0
index 1f794524a1d9..6f1d264c378b 100644
@@ -1979,6 +1979,7 @@ void cpufreq_suspend(void)
 	if (!cpufreq_driver)
 		return;
 
+	cpus_read_lock();
 	if (!has_target() && !cpufreq_driver->suspend)
 		goto suspend;
 
@@ -1998,6 +1999,7 @@ void cpufreq_suspend(void)
 
 suspend:
 	cpufreq_suspended = true;
+	cpus_read_unlock();
 }
 
 /**
Signature · 1 line
2.34.1
#5 lore raw

Re: [PATCH v2] cpufreq: Fix hotplug-suspend race during reboot

Quoted 46 lines · Tianxiang Chen

During system reboot, cpufreq_suspend() is called via the kernel_restart() -> device_shutdown() -> pm_notifier_call_chain() path. Unlike the normal system suspend path, the reboot path does not call freeze_processes(), so userspace processes and kernel threads remain active.

This allows CPU hotplug operations to run concurrently with cpufreq_suspend(). The original code has no synchronization with CPU hotplug, leading to a race condition where governor_data can be freed by the hotplug path while cpufreq_suspend() is still accessing it, resulting in a null pointer dereference:

   Unable to handle kernel NULL pointer dereference
   Call Trace:
    do_kernel_fault+0x28/0x3c
    cpufreq_suspend+0xdc/0x160
    device_shutdown+0x18/0x200
    kernel_restart+0x40/0x80
    arm64_sys_reboot+0x1b0/0x200

Fix this by adding cpus_read_lock()/cpus_read_unlock() to cpufreq_suspend() to block CPU hotplug operations while suspend is in progress.

---
v2:
- Update changelog to explicitly mention reboot scenario
- Add observed crash trace

---
  drivers/cpufreq/cpufreq.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/cpufreq/cpufreq.c +1 -0
index 1f794524a1d9..6f1d264c378b 100644
@@ -1979,6 +1979,7 @@ void cpufreq_suspend(void)
  	if (!cpufreq_driver)
  		return;
 
+	cpus_read_lock();
  	if (!has_target() && !cpufreq_driver->suspend)
  		goto suspend;
 

drivers/cpufreq/cpufreq.c void cpufreq_suspend(void) · line 2005

Quoted 7 lines
drivers/cpufreq/cpufreq.c +1 -0
@@ -1998,6 +1999,7 @@ void cpufreq_suspend(void)
 
  suspend:
  	cpufreq_suspended = true;
+	cpus_read_unlock();
  }
 
  /**

Hi Tianxiang,

May I know did you test this with lockdep enabled? Specifically, does the new cpus_read_lock() → policy->rwsem ordering in cpufreq_suspend() trigger any lockdep warnings? Thanks

Signature · 2 lines
Thx and BRs,
Zhongqiu Han
#6 lore raw

Re: [PATCH v2] cpufreq: Fix hotplug-suspend race during reboot

Quoted 3 lines · Zhongqiu Han

May I know did you test this with lockdep enabled? Specifically, does the new cpus_read_lock() -> policy->rwsem ordering in cpufreq_suspend() trigger any lockdep warnings? Thanks

Hi Zhongqiu,

Thanks for the review.

I did test v2 with lockdep enabled and was NOT able to reproduce any warning.

No circular-locking splat or "possible deadlock" report was observed in dmesg across multiple runs.

My reasoning for why the new order should be safe:

* The patch establishes cpus_read_lock() -> policy->rwsem.
* The hotplug path already holds cpu_hotplug_lock (write side, via the hotplug core) before taking policy->rwsem inside cpufreq_offline()/cpufreq_online(), i.e. the same direction.
* I grep'd cpufreq and did not find an existing path that takes policy->rwsem first and then acquires cpus_read_lock() underneath. If I missed one, please point me at it.
* cpus_read_lock() is a percpu-rwsem read side and is re-entrant, so even if an outer caller already holds it (e.g. via a pm notifier running inside a hotplug callback) this is safe.

May I ask whether you have actually observed a lockdep splat on this change on any downstream tree, or is this a precautionary question? If you have a specific call chain in mind, I would like to add targeted coverage before v3 so we can nail it down definitively.

Thanks,
Tianxiang

Reviewed-by
#7 lore raw

Re: [PATCH v2] cpufreq: Fix hotplug-suspend race during reboot

Quoted 7 lines · Tianxiang Chen
Quoted 3 lines · Zhongqiu Han

May I know did you test this with lockdep enabled? Specifically, does the new cpus_read_lock() -> policy->rwsem ordering in cpufreq_suspend() trigger any lockdep warnings? Thanks

Hi Zhongqiu,

Quoted 7 lines

Thanks for the review.

I did test v2 with lockdep enabled and was NOT able to reproduce any warning.

No circular-locking splat or "possible deadlock" report was observed in dmesg across multiple runs.

Thanks for the confirmation.

Quoted 14 lines

My reasoning for why the new order should be safe:

* The patch establishes cpus_read_lock() -> policy->rwsem.
* The hotplug path already holds cpu_hotplug_lock (write side, via the hotplug core) before taking policy->rwsem inside cpufreq_offline()/cpufreq_online(), i.e. the same direction.
* I grep'd cpufreq and did not find an existing path that takes policy->rwsem first and then acquires cpus_read_lock() underneath. If I missed one, please point me at it.
* cpus_read_lock() is a percpu-rwsem read side and is re-entrant, so even if an outer caller already holds it (e.g. via a pm notifier running inside a hotplug callback) this is safe.

Quoted 4 lines

May I ask whether you have actually observed a lockdep splat on this change on any downstream tree, or is this a precautionary question? If you have a specific call chain in mind, I would like to add targeted coverage before v3 so we can nail it down definitively.

This was mainly a precautionary question on my side, to make sure there aren't any hidden locking concerns. In my experience, running new locking changes under lockdep can be a useful sanity check, so I wanted to double‑check.

For context, none of the existing cpufreq driver .suspend callbacks currently invoke cpus_read_lock() so this does not affect any existing suspend paths.

Looks okay to me.

Quoted 3 lines

Thanks,
Tianxiang

Signature · 2 lines
Thx and BRs,
Zhongqiu Han
maintainer Applied
#8 lore raw

Re: [PATCH v2] cpufreq: Fix hotplug-suspend race during reboot

Quoted 46 lines · Zhongqiu Han
Quoted 7 lines · Tianxiang Chen
Quoted 3 lines · Zhongqiu Han

May I know did you test this with lockdep enabled? Specifically, does the new cpus_read_lock() -> policy->rwsem ordering in cpufreq_suspend() trigger any lockdep warnings? Thanks

Hi Zhongqiu,

Quoted 7 lines

Thanks for the review.

I did test v2 with lockdep enabled and was NOT able to reproduce any warning.

No circular-locking splat or "possible deadlock" report was observed in dmesg across multiple runs.

Thanks for the confirmation.

Quoted 14 lines

My reasoning for why the new order should be safe:

* The patch establishes cpus_read_lock() -> policy->rwsem.
* The hotplug path already holds cpu_hotplug_lock (write side, via the hotplug core) before taking policy->rwsem inside cpufreq_offline()/cpufreq_online(), i.e. the same direction.
* I grep'd cpufreq and did not find an existing path that takes policy->rwsem first and then acquires cpus_read_lock() underneath. If I missed one, please point me at it.
* cpus_read_lock() is a percpu-rwsem read side and is re-entrant, so even if an outer caller already holds it (e.g. via a pm notifier running inside a hotplug callback) this is safe.

Quoted 4 lines

May I ask whether you have actually observed a lockdep splat on this change on any downstream tree, or is this a precautionary question? If you have a specific call chain in mind, I would like to add targeted coverage before v3 so we can nail it down definitively.

This was mainly a precautionary question on my side, to make sure there aren't any hidden locking concerns. In my experience, running new locking changes under lockdep can be a useful sanity check, so I wanted to double‑check.

Quoted 8 lines

For context, none of the existing cpufreq driver .suspend callbacks currently invoke cpus_read_lock() so this does not affect any existing suspend paths.

Looks okay to me.

Applied as 7.1-rc material, thanks!