platform/chrome: cros_ec_debugfs: detach log reader wq from devm
Subsystems (2)
PATCH RESEND 1/1 platform/chrome: cros_ec_debugfs: detach log reader wq from devm
Debugfs console_log uses devm memory (see struct cros_ec_debugfs in cros_ec_console_log fops). However, lifecycles of device and debugfs are independent. An use-after-free issue is observed if userland program operates the debugfs after the memory has been freed.
An userland program example in Python code:
... import select
... p = select.epoll()
... f = open('/sys/kernel/debug/cros_scp/console_log') ... p.register(f, select.POLLIN)
... p.poll(1)
[(4, 1)] # 4=fd, 1=select.POLLIN
[ shutdown cros_scp at the point ]
... p.poll(1)
[(4, 16)] # 4=fd, 16=select.POLLHUP
... p.unregister(f)
An use-after-free issue raises here. It called epoll_ctl with EPOLL_CTL_DEL which in turn to use the workqueue in the devm (i.e. log_wq).
The calling stack:do_raw_spin_lock_raw_spin_lock_irqsaveremove_wait_queueep_unregister_pollwaitep_removedo_epoll_ctl
Detaches log reader's workqueue from devm to make sure it is persistent even if the device has been removed.
---
As for 2 other cases:
Case 1. userland program opens the debugfs after the device has been removed
ENOENT. cros_ec_debugfs_remove() calls debugfs_remove_recursive().
Case 2. userland program is reading when the device is removing
If the userland program is waiting in cros_ec_console_log_read(), the device removal will wait for it.
See the calling stack for the case:wait_for_completion__debugfs_file_removedremove_onesimple_recursive_removaldebugfs_removecros_ec_debugfs_removeplatform_drv_removedevice_release_driver_internaldevice_release_driverbus_remove_devicedevice_delplatform_device_delplatform_device_unregister
drivers/platform/chrome/cros_ec_debugfs.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/platform/chrome/cros_ec_debugfs.c +6 -6
@@ -25,6 +25,9 @@#define CIRC_ADD (idx, size, value) (((idx) + (value)) & ((size) - 1)) +/* waitqueue for log readers */ +static DECLARE_WAIT_QUEUE_HEAD (cros_ec_debugfs_log_wq); +/** * struct cros_ec_debugfs - EC debugging information. *
@@ -33,7 +36,6 @@* @log_buffer: circular buffer for console log information * @read_msg: preallocated EC command and buffer to read console log * @log_mutex: mutex to protect circular buffer -* @log_wq: waitqueue for log readers * @log_poll_work: recurring task to poll EC for new console log data * @panicinfo_blob: panicinfo debugfs blob */
@@ -44,7 +46,6 @@ struct cros_ec_debugfs {struct circ_buf log_buffer;struct cros_ec_command *read_msg;struct mutex log_mutex; -wait_queue_head_t log_wq;struct delayed_work log_poll_work;/* EC panicinfo */ struct debugfs_blob_wrapper panicinfo_blob;
@@ -107,7 +108,7 @@ static void cros_ec_console_log_work(struct work_struct *__work) buf_space--; } -wake_up (&debug_info->log_wq); +wake_up (&cros_ec_debugfs_log_wq); }mutex_unlock (&debug_info->log_mutex);
@@ -141,7 +142,7 @@ static ssize_t cros_ec_console_log_read(struct file *file, char __user *buf,mutex_unlock (&debug_info->log_mutex); - ret =wait_event_interruptible (debug_info->log_wq, + ret =wait_event_interruptible (cros_ec_debugfs_log_wq,CIRC_CNT (cb->head, cb->tail,LOG_SIZE ));if (ret <0 )return ret;
@@ -173,7 +174,7 @@ static __poll_t cros_ec_console_log_poll(struct file *file,struct cros_ec_debugfs *debug_info = file->private_data;__poll_t mask =0 ; -poll_wait (file, &debug_info->log_wq, wait); +poll_wait (file, &cros_ec_debugfs_log_wq, wait);mutex_lock (&debug_info->log_mutex); if (CIRC_CNT(debug_info->log_buffer.head,
@@ -377,7 +378,6 @@ static int cros_ec_create_console_log(struct cros_ec_debugfs *debug_info) debug_info->log_buffer.tail =0 ;mutex_init (&debug_info->log_mutex); -init_waitqueue_head (&debug_info->log_wq);debugfs_create_file ("console_log" ,S_IFREG |0444 , debug_info->dir, debug_info, &cros_ec_console_log_fops);
Signature · 1 line
2.33.0.1079.g6e70778dc9-goog
Quoted 6 lines · Tzung-Bi Shih
Debugfs
console_loguses devm memory (see structcros_ec_debugfsincros_ec_console_logfops). However, lifecycles of device and debugfs are independent. An use-after-free issue is observed if userland program operates the debugfs after the memory has been freed.
It would help to see the backtrace. Without it, it is difficult to determine where the UAF is observed. Also, most if not all of the touched functions access struct cros_ec_debugfs all over the place, not only for the wait queue, so I am not sure if moving the wait queue out of the data structure is the correct fix. It might instead be necessary to disconnect memory allocations from the ec device.
Guenter
Guenter
Quoted 128 lines
An userland program example in Python code:
... import select
... p =select.epoll()
... f =open('/sys/kernel/debug/cros_scp/console_log')...p.register(f, select.POLLIN)
...p.poll(1)
[(4, 1)] # 4=fd, 1=select.POLLIN[ shutdown
cros_scpat the point ]...
p.poll(1)
[(4, 16)] # 4=fd, 16=select.POLLHUP
...p.unregister(f)An use-after-free issue raises here. It called
epoll_ctlwithEPOLL_CTL_DELwhich in turn to use the workqueue in the devm (i.e.log_wq).The calling stack:
do_raw_spin_lock_raw_spin_lock_irqsaveremove_wait_queueep_unregister_pollwaitep_removedo_epoll_ctlDetaches log reader's workqueue from devm to make sure it is persistent even if the device has been removed.
Signed-off-by Tzung-Bi Shih <[email protected]>---
As for 2 other cases:Case 1. userland program opens the debugfs after the device has been removed
ENOENT.
cros_ec_debugfs_remove()callsdebugfs_remove_recursive().Case 2. userland program is reading when the device is removing
If the userland program is waiting in
cros_ec_console_log_read(), the device removal will wait for it.See the calling stack for the case:
wait_for_completion__debugfs_file_removedremove_onesimple_recursive_removaldebugfs_removecros_ec_debugfs_removeplatform_drv_removedevice_release_driver_internaldevice_release_driverbus_remove_devicedevice_delplatform_device_delplatform_device_unregisterdrivers/platform/chrome/cros_ec_debugfs.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)drivers/platform/chrome/cros_ec_debugfs.c +6
-7@@ -25,6 +25,9 @@#define CIRC_ADD (idx, size, value) (((idx) + (value)) & ((size) - 1)) +/* waitqueue for log readers */ +static DECLARE_WAIT_QUEUE_HEAD (cros_ec_debugfs_log_wq); +/** * struct cros_ec_debugfs - EC debugging information. * @@ -33,7 +36,6 @@* @log_buffer: circular buffer for console log information * @read_msg: preallocated EC command and buffer to read console log * @log_mutex: mutex to protect circular buffer -* @log_wq: waitqueue for log readers * @log_poll_work: recurring task to poll EC for new console log data * @panicinfo_blob: panicinfo debugfs blob */ @@ -44,7 +46,6 @@ struct cros_ec_debugfs {struct circ_buf log_buffer;struct cros_ec_command *read_msg;struct mutex log_mutex; -wait_queue_head_t log_wq;struct delayed_work log_poll_work;/* EC panicinfo */ struct debugfs_blob_wrapper panicinfo_blob;@@ -107,7 +108,7 @@ static void cros_ec_console_log_work(struct work_struct *__work) buf_space--; } -wake_up (&debug_info->log_wq); +wake_up (&cros_ec_debugfs_log_wq); }mutex_unlock (&debug_info->log_mutex);@@ -141,7 +142,7 @@ static ssize_t cros_ec_console_log_read(struct file *file, char __user *buf,mutex_unlock (&debug_info->log_mutex); - ret =wait_event_interruptible (debug_info->log_wq, + ret =wait_event_interruptible (cros_ec_debugfs_log_wq,CIRC_CNT (cb->head, cb->tail,LOG_SIZE ));if (ret <0 )return ret;@@ -173,7 +174,7 @@ static __poll_t cros_ec_console_log_poll(struct file *file,struct cros_ec_debugfs *debug_info = file->private_data;__poll_t mask =0 ; -poll_wait (file, &debug_info->log_wq, wait); +poll_wait (file, &cros_ec_debugfs_log_wq, wait);mutex_lock (&debug_info->log_mutex); if (CIRC_CNT(debug_info->log_buffer.head,@@ -377,7 +378,6 @@ static int cros_ec_create_console_log(struct cros_ec_debugfs *debug_info) debug_info->log_buffer.tail =0 ;mutex_init (&debug_info->log_mutex); -init_waitqueue_head (&debug_info->log_wq);debugfs_create_file ("console_log" ,S_IFREG |0444 , debug_info->dir, debug_info, &cros_ec_console_log_fops); --2.33.0.1079.g6e70778dc9-goog
Quoted 9 lines · Guenter Roeck
Quoted 6 lines · Tzung-Bi Shih
Debugfs
console_loguses devm memory (see structcros_ec_debugfsincros_ec_console_logfops). However, lifecycles of device and debugfs are independent. An use-after-free issue is observed if userland program operates the debugfs after the memory has been freed.
Quoted 6 lines
It would help to see the backtrace. Without it, it is difficult to determine where the UAF is observed. Also, most if not all of the touched functions access struct
cros_ec_debugfsall over the place, not only for the wait queue, so I am not sure if moving the wait queue out of the data structure is the correct fix. It might instead be necessary to disconnect memory allocations from the ec device.
A trimmed backtrace is in the commit message, but the more verbose one:
[ 426.174308] Call trace: [ 426.174314] dump_backtrace+0x0/0x3ec [ 426.174318] show_stack+0x20/0x2c [ 426.174324] dump_stack+0x11c/0x1ac [ 426.174329] print_address_description+0x7c/0x510 [ 426.174333] kasan_report+0x134/0x174 [ 426.174337] __asan_report_load4_noabort+0x44/0x50 [ 426.174341] do_raw_spin_lock+0x214/0x308 [ 426.174345] _raw_spin_lock_irqsave+0x68/0xf0 [ 426.174350] remove_wait_queue+0x3c/0x10c [ 426.174355] ep_unregister_pollwait+0x120/0x170 [ 426.174358] ep_remove+0x60/0x2a0 [ 426.174362] do_epoll_ctl+0x590/0x7f4
I guess only the wait queue in the struct cros_ec_debugfs has deep-coupled to console_log debugfs. There are 2 more file operation scenarios appended after the "--".
Quoted 18 lines
Quoted 18 lines
An userland program example in Python code:
... import select
... p =select.epoll()
... f =open('/sys/kernel/debug/cros_scp/console_log')...p.register(f, select.POLLIN)
...p.poll(1)
[(4, 1)] # 4=fd, 1=select.POLLIN[ shutdown
cros_scpat the point ]...
p.poll(1)
[(4, 16)] # 4=fd, 16=select.POLLHUP
...p.unregister(f)An use-after-free issue raises here. It called
epoll_ctlwithEPOLL_CTL_DELwhich in turn to use the workqueue in the devm (i.e.log_wq).
Quoted 7 lines
Quoted 7 lines
The calling stack:
do_raw_spin_lock_raw_spin_lock_irqsaveremove_wait_queueep_unregister_pollwaitep_removedo_epoll_ctl
Here is the trimmed backtrace.
[...]
Quoted 12 lines
Quoted 12 lines
---
As for 2 other cases:Case 1. userland program opens the debugfs after the device has been removed
ENOENT.
cros_ec_debugfs_remove()callsdebugfs_remove_recursive().Case 2. userland program is reading when the device is removing
If the userland program is waiting in
cros_ec_console_log_read(), the device removal will wait for it.
Quoted 14 lines
Quoted 6 lines
See the calling stack for the case:
wait_for_completion__debugfs_file_removedremove_onesimple_recursive_removaldebugfs_removeQuoted 8 lines
cros_ec_debugfs_removeplatform_drv_removedevice_release_driver_internaldevice_release_driverbus_remove_devicedevice_delplatform_device_delplatform_device_unregister
Here are the 2 other cases.
Hi Guenter,
Quoted 33 lines · Tzung-Bi Shih
Quoted 9 lines · Guenter Roeck
Quoted 6 lines · Tzung-Bi Shih
Debugfs
console_loguses devm memory (see structcros_ec_debugfsincros_ec_console_logfops). However, lifecycles of device and debugfs are independent. An use-after-free issue is observed if userland program operates the debugfs after the memory has been freed.Quoted 6 lines
It would help to see the backtrace. Without it, it is difficult to determine where the UAF is observed. Also, most if not all of the touched functions access struct
cros_ec_debugfsall over the place, not only for the wait queue, so I am not sure if moving the wait queue out of the data structure is the correct fix. It might instead be necessary to disconnect memory allocations from the ec device.A trimmed backtrace is in the commit message, but the more verbose one:
[ 426.174308] Call trace: [ 426.174314] dump_backtrace+0x0/0x3ec [ 426.174318] show_stack+0x20/0x2c [ 426.174324] dump_stack+0x11c/0x1ac [ 426.174329] print_address_description+0x7c/0x510 [ 426.174333] kasan_report+0x134/0x174 [ 426.174337] __asan_report_load4_noabort+0x44/0x50 [ 426.174341] do_raw_spin_lock+0x214/0x308 [ 426.174345] _raw_spin_lock_irqsave+0x68/0xf0 [ 426.174350] remove_wait_queue+0x3c/0x10c [ 426.174355] ep_unregister_pollwait+0x120/0x170 [ 426.174358] ep_remove+0x60/0x2a0 [ 426.174362] do_epoll_ctl+0x590/0x7f4
Quoted 3 lines
I guess only the wait queue in the struct
cros_ec_debugfshas deep-coupled toconsole_logdebugfs. There are 2 more file operation scenarios appended after the "--".
Do you think the backtrace is sufficient to determine the UAF happens on the wait queue?
How about we keep the fix as is since we have a constantly reproducing step for the UAF. And look forward to the approach "disconnect memory allocations from the ec device" if we could discover more UAFs?