gpio: cdev: fix a crash on line-request release
PATCH 1/1 gpio: cdev: fix a crash on line-request release
When a GPIO device is forcefully unregistered, we are left with an inactive object. If user-space kept an open file descriptor to a line request associated with such a structure, upon closing it, we'll see the kernel crash due to freeing unexistent GPIO descriptors.
Fix it by checking if chip is still alive before calling gpiod_free() in release callbacks for both v2 and v1 ABI.
--- drivers/gpio/gpiolib-cdev.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-)
drivers/gpio/gpiolib-cdev.c +16 -6
@@ -315,13 +315,19 @@ static long linehandle_ioctl_compat(struct file *file, unsigned int cmd,static void linehandle_free (struct linehandle_state *lh) { +struct gpio_device *gdev = lh->gdev;int i; -for (i =0 ; i < lh->num_descs; i++) -if (lh->descs[i]) -gpiod_free (lh->descs[i]); +for (i =0 ; i < lh->num_descs; i++) { +if (lh->descs[i]) { +down_write (&gdev->sem); +if (gdev->chip) +gpiod_free (lh->descs[i]); +up_write (&gdev->sem); + } + }kfree (lh->label); -gpio_device_put (lh->gdev); +gpio_device_put (gdev);kfree (lh); }
@@ -1565,17 +1571,21 @@ static ssize_t linereq_read(struct file *file, char __user *buf,static void linereq_free (struct linereq *lr) { +struct gpio_device *gdev = lr->gdev;unsigned int i;for (i =0 ; i < lr->num_lines; i++) {if (lr->lines[i].desc) {edge_detector_stop (&lr->lines[i]); -gpiod_free (lr->lines[i].desc); +down_write (&gdev->sem); +if (gdev->chip) +gpiod_free (lr->lines[i].desc); +up_write (&gdev->sem); } }kfifo_free (&lr->events);kfree (lr->label); -gpio_device_put (lr->gdev); +gpio_device_put (gdev);kfree (lr); }
Signature · 1 line
2.39.2
Kent Gibson
reviewerQuoted 5 lines · Bartosz Golaszewski
When a GPIO device is forcefully unregistered, we are left with an inactive object. If user-space kept an open file descriptor to a line request associated with such a structure, upon closing it, we'll see the kernel crash due to freeing unexistent GPIO descriptors.
nonexistent
But I'm not sure that works - gpiod_free() is null aware, so strictly speaking "freeing nonexistent GPIO descriptors" isn't the problem. You mean orphaned GPIO descriptors?
Quoted 4 lines
Fix it by checking if chip is still alive before calling
gpiod_free()in release callbacks for both v2 and v1 ABI.Fixes 3c0d9c635ae2 ("gpiolib: cdev: support GPIO_V2_GET_LINE_IOCTL and GPIO_V2_LINE_GET_VALUES_IOCTL")
The problem is also in v1, so do we want to consider backporting a fix for that too?
Quoted 11 lines
Reported-by Kent Gibson <[email protected]>Signed-off-by Bartosz Golaszewski <[email protected]>--- drivers/gpio/gpiolib-cdev.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-)drivers/gpio/gpiolib-cdev.c +0
-0
drivers/gpio/gpiolib-cdev.c static long linehandle_ioctl_compat(struct file *file, unsigned int cmd, · line 318
Quoted 4 lines
drivers/gpio/gpiolib-cdev.c +1
-0@@ -315,13 +315,19 @@ static long linehandle_ioctl_compat(struct file *file, unsigned int cmd,static void linehandle_free (struct linehandle_state *lh) { +struct gpio_device *gdev = lh->gdev;
It isn't clear to me what this is for.
The flow below now calls gpiod_free() less often, so not that. It is there for the normal case??
Quoted 9 lines
drivers/gpio/gpiolib-cdev.c +4
-3int i; -for (i =0 ; i < lh->num_descs; i++) -if (lh->descs[i]) -gpiod_free (lh->descs[i]); +for (i =0 ; i < lh->num_descs; i++) { +if (lh->descs[i]) { +down_write (&gdev->sem); +if (gdev->chip)
drivers/gpio/gpiolib-cdev.c static long linehandle_ioctl_compat(struct file *file, unsigned int cmd, · line 333
Quoted 10 lines
drivers/gpio/gpiolib-cdev.c +5
-1+gpiod_free (lh->descs[i]); +up_write (&gdev->sem); + } + }kfree (lh->label); -gpio_device_put (lh->gdev); +gpio_device_put (gdev);kfree (lh); }
lineevent_free() needs the fix too?
Quoted 14 lines
drivers/gpio/gpiolib-cdev.c +4
-1@@ -1565,17 +1571,21 @@ static ssize_t linereq_read(struct file *file, char __user *buf,static void linereq_free (struct linereq *lr) { +struct gpio_device *gdev = lr->gdev;unsigned int i;for (i =0 ; i < lr->num_lines; i++) {if (lr->lines[i].desc) {edge_detector_stop (&lr->lines[i]); -gpiod_free (lr->lines[i].desc); +down_write (&gdev->sem); +if (gdev->chip) +gpiod_free (lr->lines[i].desc);
drivers/gpio/gpiolib-cdev.c static ssize_t linereq_read(struct file *file, char __user *buf, · line 1591
Quoted 10 lines
drivers/gpio/gpiolib-cdev.c +2
-1+up_write (&gdev->sem); } }kfifo_free (&lr->events);kfree (lr->label); -gpio_device_put (lr->gdev); +gpio_device_put (gdev);kfree (lr); }
TBH the fact you have to mess with sems here indicates to me the problem lies in gpiolib itself. As a gpiolib client, cdev should just be able to release the desc back to gpiolib and have it cleanup the mess.
Not that I ever got my head around the whole gpiolib object lifecycle here
- for v2 I just followed what v1 did.
Also, gpiolib still reports an error when forceably removing chips that have open requests:
dev_crit(&gdev->dev, "REMOVING GPIOCHIP WITH GPIOS STILL REQUESTED\n");
Any other gpiolib clients out there that this might impact? Else why report that crit error if you expect it is dealt with?
So while this may fix the crash, I can't say I'm happy with it.
Cheers,
Kent.
Kent Gibson
reviewerQuoted 14 lines · Kent Gibson
Quoted 13 lines · Bartosz Golaszewski
When a GPIO device is forcefully unregistered, we are left with an inactive object. If user-space kept an open file descriptor to a line request associated with such a structure, upon closing it, we'll see the kernel crash due to freeing unexistent GPIO descriptors.
drivers/gpio/gpiolib-cdev.c +1
-0@@ -1565,17 +1571,21 @@ static ssize_t linereq_read(struct file *file, char __user *buf,static void linereq_free (struct linereq *lr) { +struct gpio_device *gdev = lr->gdev;unsigned int i;
Quoted 8 lines
drivers/gpio/gpiolib-cdev.c static ssize_t linereq_read(struct file *file, char __user *buf, · line 1583
Quoted 8 lines
drivers/gpio/gpiolib-cdev.c +4
-1for (i =0 ; i < lr->num_lines; i++) {if (lr->lines[i].desc) {edge_detector_stop (&lr->lines[i]); -gpiod_free (lr->lines[i].desc); +down_write (&gdev->sem); +if (gdev->chip) +gpiod_free (lr->lines[i].desc); +up_write (&gdev->sem);
Ummm, taking another look at the oops I sent you, the crash actually occurs in edge_detector_stop():
May 23 11:47:06 firefly kernel: [ 4216.877056] Call Trace: May 23 11:47:06 firefly kernel: [ 4216.877512] <TASK> May 23 11:47:06 firefly kernel: [ 4216.877924] irq_domain_deactivate_irq+0x19/0x30 May 23 11:47:06 firefly kernel: [ 4216.878543] free_irq+0x257/0x360 May 23 11:47:06 firefly kernel: [ 4216.879056] linereq_free+0x9b/0xe0 May 23 11:47:06 firefly kernel: [ 4216.879608] linereq_release+0xc/0x20 May 23 11:47:06 firefly kernel: [ 4216.880230] __fput+0x87/0x240 May 23 11:47:06 firefly kernel: [ 4216.880744] task_work_run+0x54/0x80
That free_irq() call is in edge_detector_stop() (which apparently is inlined), not in gpiod_free().
So pretty sure this patch doesn't even solve my problem, but I will test it to confirm.
Cheers,
Kent.
Kent Gibson
reviewerQuoted 36 lines · Kent Gibson
Quoted 14 lines · Kent Gibson
Quoted 13 lines · Bartosz Golaszewski
When a GPIO device is forcefully unregistered, we are left with an inactive object. If user-space kept an open file descriptor to a line request associated with such a structure, upon closing it, we'll see the kernel crash due to freeing unexistent GPIO descriptors.
drivers/gpio/gpiolib-cdev.c +1
-0@@ -1565,17 +1571,21 @@ static ssize_t linereq_read(struct file *file, char __user *buf,static void linereq_free (struct linereq *lr) { +struct gpio_device *gdev = lr->gdev;unsigned int i;Quoted 8 lines
drivers/gpio/gpiolib-cdev.c static ssize_t linereq_read(struct file *file, char __user *buf, · line 1583
Quoted 8 lines
drivers/gpio/gpiolib-cdev.c +4
-1for (i =0 ; i < lr->num_lines; i++) {if (lr->lines[i].desc) {edge_detector_stop (&lr->lines[i]); -gpiod_free (lr->lines[i].desc); +down_write (&gdev->sem); +if (gdev->chip) +gpiod_free (lr->lines[i].desc); +up_write (&gdev->sem);Ummm, taking another look at the oops I sent you, the crash actually occurs in
edge_detector_stop():May 23 11:47:06 firefly kernel: [ 4216.877056] Call Trace: May 23 11:47:06 firefly kernel: [ 4216.877512] <TASK> May 23 11:47:06 firefly kernel: [ 4216.877924] irq_domain_deactivate_irq+0x19/0x30 May 23 11:47:06 firefly kernel: [ 4216.878543] free_irq+0x257/0x360 May 23 11:47:06 firefly kernel: [ 4216.879056] linereq_free+0x9b/0xe0 May 23 11:47:06 firefly kernel: [ 4216.879608] linereq_release+0xc/0x20 May 23 11:47:06 firefly kernel: [ 4216.880230] __fput+0x87/0x240 May 23 11:47:06 firefly kernel: [ 4216.880744] task_work_run+0x54/0x80
Quoted 6 lines
That
free_irq()call is inedge_detector_stop()(which apparently is inlined), not ingpiod_free().So pretty sure this patch doesn't even solve my problem, but I will test it to confirm.
Yeah, doesn't fix my problem still crashes.
If the line request doesn't have edge detection enabled (so no irq) then I don't get a crash.
i.e. use gpioset to request the line, rather than gpiomon.
(To provide background for anyone else trying to follow along, the scenario is:
1. create a gpio-sim
2. request a line
3. destroy the gpio-sim
4. release the line.
3 triggers this error:
May 24 11:11:12 firefly kernel: [ 200.027280] gpio_stub_drv gpiochip0: REMOVING GPIOCHIP WITH GPIOS STILL REQUESTED
and 4 triggers a crash - if the requested line holds an irq.)
I would point out:
/**
* gpiochip_remove() - unregister a gpio_chip
* @gc: the chip to unregister
*
* A gpio_chip with any GPIOs still requested may not be removed. */
void gpiochip_remove(struct gpio_chip *gc)
which is where that dev_crit() is, so destroying the gpio-sim has already invalidated that contract.
Anyway, it seems my problem is the forced removal of the gpiochip invalidates the irq that the line request is holding.
Not sure how best to deal with that.
Moving the edge_detector_stop() inside the "if (gdev->chip)" check of your patch does prevent crash.
But in that case edge_detector_stop() does other cleanup that is no longer getting done.
Perhaps if the chip is gone then zero line->irq prior to calling edge_detector_stop()?
Again, this is starting to feel like a hack for gpiolib not being good at telling the client that it has to pull the rug.
Though according the the gpiochip_remove() docs, it WONT pull the rug, so you get that.
Cheers,
Kent.
Quoted 91 lines · Kent Gibson
Quoted 36 lines · Kent Gibson
Quoted 14 lines · Kent Gibson
Quoted 13 lines · Bartosz Golaszewski
When a GPIO device is forcefully unregistered, we are left with an inactive object. If user-space kept an open file descriptor to a line request associated with such a structure, upon closing it, we'll see the kernel crash due to freeing unexistent GPIO descriptors.
drivers/gpio/gpiolib-cdev.c +1
-0@@ -1565,17 +1571,21 @@ static ssize_t linereq_read(struct file *file, char __user *buf,static void linereq_free (struct linereq *lr) { +struct gpio_device *gdev = lr->gdev;unsigned int i;Quoted 8 lines
drivers/gpio/gpiolib-cdev.c static ssize_t linereq_read(struct file *file, char __user *buf, · line 1583
Quoted 8 lines
drivers/gpio/gpiolib-cdev.c +4
-1for (i =0 ; i < lr->num_lines; i++) {if (lr->lines[i].desc) {edge_detector_stop (&lr->lines[i]); -gpiod_free (lr->lines[i].desc); +down_write (&gdev->sem); +if (gdev->chip) +gpiod_free (lr->lines[i].desc); +up_write (&gdev->sem);Ummm, taking another look at the oops I sent you, the crash actually occurs in
edge_detector_stop():May 23 11:47:06 firefly kernel: [ 4216.877056] Call Trace: May 23 11:47:06 firefly kernel: [ 4216.877512] <TASK> May 23 11:47:06 firefly kernel: [ 4216.877924] irq_domain_deactivate_irq+0x19/0x30 May 23 11:47:06 firefly kernel: [ 4216.878543] free_irq+0x257/0x360 May 23 11:47:06 firefly kernel: [ 4216.879056] linereq_free+0x9b/0xe0 May 23 11:47:06 firefly kernel: [ 4216.879608] linereq_release+0xc/0x20 May 23 11:47:06 firefly kernel: [ 4216.880230] __fput+0x87/0x240 May 23 11:47:06 firefly kernel: [ 4216.880744] task_work_run+0x54/0x80Quoted 6 lines
That
free_irq()call is inedge_detector_stop()(which apparently is inlined), not ingpiod_free().So pretty sure this patch doesn't even solve my problem, but I will test it to confirm.
Yeah, doesn't fix my problem still crashes.
If the line request doesn't have edge detection enabled (so no irq) then I don't get a crash.
i.e. use gpioset to request the line, rather than gpiomon.(To provide background for anyone else trying to follow along, the scenario is:
1. create a gpio-sim
2. request a line
3. destroy the gpio-sim
4. release the line.3 triggers this error:
May 24 11:11:12 firefly kernel: [ 200.027280] gpio_stub_drv gpiochip0: REMOVING GPIOCHIP WITH GPIOS STILL REQUESTEDand 4 triggers a crash - if the requested line holds an irq.)
I would point out:
/**
*gpiochip_remove()- unregister agpio_chip
* @gc: the chip to unregister
*
* Agpio_chipwith any GPIOs still requested may not be removed. */
voidgpiochip_remove(struct gpio_chip *gc)which is where that
dev_crit()is, so destroying the gpio-sim has already invalidated that contract.Anyway, it seems my problem is the forced removal of the gpiochip invalidates the irq that the line request is holding.
Not sure how best to deal with that.Moving the
edge_detector_stop()inside the "if (gdev->chip)" check of your patch does prevent crash.
But in that caseedge_detector_stop()does other cleanup that is no longer getting done.
Perhaps if the chip is gone then zeroline->irqprior to callingedge_detector_stop()?
Again, this is starting to feel like a hack for gpiolib not being good at telling the client that it has to pull the rug.
Though according the thegpiochip_remove()docs, it WONT pull the rug, so you get that.
Quoted 2 lines
Cheers,
Kent.
Interestingly enough, I did test it just like you and the "fix" seemed to address the issue. Upon a further look at the code, it's of course clear that the patch is wrong.
I wanted to debug the code to see what's happening exactly but it turned out that enabling the generation of DWARF data hid the issue as well even without any fix. It means that it's some kind of a memory corruption rather than a regular NULL-pointer dereference.
I'm not yet sure where the crash happens exactly other that it's in the irq domain code.
Anyway, I'll be back at it tomorrow.
Bart
Kent Gibson
reviewerQuoted 11 lines · Bartosz Golaszewski
Quoted 1 line · Kent Gibson
Interestingly enough, I did test it just like you and the "fix" seemed to address the issue. Upon a further look at the code, it's of course clear that the patch is wrong.
I wanted to debug the code to see what's happening exactly but it turned out that enabling the generation of DWARF data hid the issue as well even without any fix. It means that it's some kind of a memory corruption rather than a regular NULL-pointer dereference.
Quoted 5 lines
I'm not yet sure where the crash happens exactly other that it's in the irq domain code.
Anyway, I'll be back at it tomorrow.
I was also playing with a patch for gpiomon to have it add POLLERR to its poll() to see if it would notice the chip removal and exit. It didn't, it just stayed blocked, but it DID made the crash go away when I killed it. No idea why that would be. So yeah, weird things.
This was the patch, btw:
tools/gpiomon.c +4 -1
@@ -431,7 +431,7 @@ int main(int argc, char **argv) resolver->chips[i].path); pollfds[i].fd =gpiod_line_request_get_fd (requests[i]); - pollfds[i].events =POLLIN ; + pollfds[i].events =POLLIN |POLLERR ;gpiod_chip_close (chip); }
@@ -452,6 +452,9 @@ int main(int argc, char **argv)if (pollfds[i].revents ==0 )continue ; +if (pollfds[i].revents &POLLERR ) +die_perror ("error polling for events" ); + ret =gpiod_line_request_read_edge_events (requests[i], event_buffer,EVENT_BUF_SIZE );if (ret <0 )
I expect to add that, or something along those lines, if hot removal of chips from the kernel ever works like it should.
In the meantime it is just another curiosity.
Cheers,
Kent.