Skip to contentSkip to conversation

mm/backing-dev.c: fix crash when USB/SCSI device is detached

PATCH In review 민찬호 2 participants 7 messages Jan 2, 2012 – Jan 3, 2012

Subsystems (2)
lore
Share & tools
lore

민찬호

author
#0 lore raw

PATCH 1/1 mm/backing-dev.c: fix crash when USB/SCSI device is detached

from Chanho Min <[email protected]>

System may crash in backing-dev.c when removal SCSI device is detached. bdi task is killed by bdi_unregister()/'khubd', but task's point remains. Shortly afterward, If 'wb->wakeup_timer' is expired before del_timer()/bdi_forker_thread,
wakeup_timer_fn() may wake up the dead thread which cause the crash. 'bdi->wb.task' should be NULL as this patch.

---
 mm/backing-dev.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
mm/backing-dev.c +0 -0
index 71034f4..4378a5e 100644
@@ -607,6 +607,7 @@ static void bdi_wb_shutdown(struct backing_dev_info

*bdi)

        if (bdi->wb.task) {
                thaw_process(bdi->wb.task);
                kthread_stop(bdi->wb.task);
+               bdi->wb.task = NULL;
        }

}

Signature · 8 lines
1.7.0.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw
Quoted 8 lines · On Mon, Jan 02, 2012 at 06:38:21PM +0900, i? 1/2 i? 1/2 i? 1/2 i? 1/2 EGBP…

from Chanho Min <[email protected]>

System may crash in backing-dev.c when removal SCSI device is detached. bdi task is killed by bdi_unregister()/'khubd', but task's point remains. Shortly afterward, If 'wb->wakeup_timer' is expired before del_timer()/bdi_forker_thread,
wakeup_timer_fn() may wake up the dead thread which cause the crash. 'bdi->wb.task' should be NULL as this patch.

Is it some race condition between del_timer() and del_timer_sync()?

bdi_unregister() calls

del_timer_sync
bdi_wb_shutdown
kthread_stop

in turn, and del_timer_sync() should guarantee wakeup_timer_fn() is no longer called to access the stopped task.

Thanks,
Fengguang

Quoted 20 lines
---
 mm/backing-dev.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
mm/backing-dev.c +1 -0
index 71034f4..4378a5e 100644
@@ -607,6 +607,7 @@ static void bdi_wb_shutdown(struct backing_dev_info
 *bdi)
        if (bdi->wb.task) {
                thaw_process(bdi->wb.task);
                kthread_stop(bdi->wb.task);
+               bdi->wb.task = NULL;
        }
 }
Signature · 2 lines
-- 
1.7.0.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#2 lore raw
Quoted 12 lines · ����ȣ
Quoted 8 lines

from Chanho Min <[email protected]>

System may crash in backing-dev.c when removal SCSI device is detached. bdi task is killed by bdi_unregister()/'khubd', but task's point remains. Shortly afterward, If 'wb->wakeup_timer' is expired before del_timer()/bdi_forker_thread,
wakeup_timer_fn() may wake up the dead thread which cause the crash. 'bdi->wb.task' should be NULL as this patch.

Is it some race condition between del_timer() and del_timer_sync()?

Quoted 9 lines

bdi_unregister() calls

del_timer_sync
bdi_wb_shutdown
kthread_stop

in turn, and del_timer_sync() should guarantee wakeup_timer_fn() is no longer called to access the stopped task.

It is not race condition. This happens when USB is removed during write-access. bdi_wakeup_thread_delayed is called after kthread_stop, and timer is activated again.

bdi_unregister
kthread_stop
bdi_wakeup_thread_delayed (sys_write mostly calls this) timer fires

Anyway,Is this safeguard to prevent from waking up killed thread?

Thanks,
Chanho

Quoted 22 lines

Thanks,
Fengguang

Quoted 18 lines
---
 mm/backing-dev.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
mm/backing-dev.c +1 -0
index 71034f4..4378a5e 100644
@@ -607,6 +607,7 @@ static void bdi_wb_shutdown(struct backing_dev_info
 *bdi)
        if (bdi->wb.task) {
                thaw_process(bdi->wb.task);
                kthread_stop(bdi->wb.task);
+               bdi->wb.task = NULL;
        }
 }
Quoted 2 lines
Quoted 2 lines

--
1.7.0.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#6 lore raw
In reply to Wu Fengguang
Quoted 12 lines · ����ȣ
Quoted 8 lines

from Chanho Min <[email protected]>

System may crash in backing-dev.c when removal SCSI device is detached. bdi task is killed by bdi_unregister()/'khubd', but task's point remains. Shortly afterward, If 'wb->wakeup_timer' is expired before del_timer()/bdi_forker_thread,
wakeup_timer_fn() may wake up the dead thread which cause the crash. 'bdi->wb.task' should be NULL as this patch.

Is it some race condition between del_timer() and del_timer_sync()?

Quoted 9 lines

bdi_unregister() calls

del_timer_sync
bdi_wb_shutdown
kthread_stop

in turn, and del_timer_sync() should guarantee wakeup_timer_fn() is no longer called to access the stopped task.

It is not race condition. This happens when USB is removed during write-access. bdi_wakeup_thread_delayed is called after kthread_stop, and timer is activated again.

bdi_unregister
kthread_stop
bdi_wakeup_thread_delayed (sys_write mostly calls this) timer fires

Anyway,Is this safeguard to prevent from waking up killed thread?

Thanks,
Chanho

Quoted 22 lines

Thanks,
Fengguang

Quoted 18 lines
---
 mm/backing-dev.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
mm/backing-dev.c +1 -0
index 71034f4..4378a5e 100644
@@ -607,6 +607,7 @@ static void bdi_wb_shutdown(struct backing_dev_info
 *bdi)
        if (bdi->wb.task) {
                thaw_process(bdi->wb.task);
                kthread_stop(bdi->wb.task);
+               bdi->wb.task = NULL;
        }
 }
Quoted 2 lines
Quoted 2 lines

--
1.7.0.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw
In reply to Chanho Min
Quoted 22 lines · Chanho Min
Quoted 12 lines · On Mon, Jan 02, 2012 at 06:38:21PM +0900, i? 1/2 i? 1/2 i? 1/2 i? 1/2 EGBP…
Quoted 8 lines

from Chanho Min <[email protected]>

System may crash in backing-dev.c when removal SCSI device is detached. bdi task is killed by bdi_unregister()/'khubd', but task's point remains. Shortly afterward, If 'wb->wakeup_timer' is expired before del_timer()/bdi_forker_thread,
wakeup_timer_fn() may wake up the dead thread which cause the crash. 'bdi->wb.task' should be NULL as this patch.

Is it some race condition between del_timer() and del_timer_sync()?

Quoted 9 lines

bdi_unregister() calls

del_timer_sync
bdi_wb_shutdown
kthread_stop

in turn, and del_timer_sync() should guarantee wakeup_timer_fn() is no longer called to access the stopped task.

Quoted 7 lines

It is not race condition. This happens when USB is removed during write-access. bdi_wakeup_thread_delayed is called after kthread_stop, and timer is activated again.

bdi_unregister
kthread_stop
bdi_wakeup_thread_delayed (sys_write mostly calls this) timer fires

Ah OK, the timer could be restarted in the mean while, which breaks the synchronization rule in del_timer_sync().

I noticed a related fix is merged recently, does your test kernel contain this commit?

commit 7a401a972df8e184b3d1a3fc958c0a4ddee8d312

Author: Rabin Vincent <[email protected]>
Date:   Fri Nov 11 13:29:04 2011 +0100

backing-dev: ensure wakeup_timer is deleted

Quoted 1 line

Anyway,Is this safeguard to prevent from waking up killed thread?

This patch makes no guarantee wakeup_timer_fn() will see NULL bdi->wb.task before the task is stopped, so there is still race conditions. And still, the complete fix would be to prevent wakeup_timer_fn() from being called at all.

Thanks,
Fengguang

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
---
 mm/backing-dev.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
mm/backing-dev.c +1 -0
index 71034f4..4378a5e 100644
@@ -607,6 +607,7 @@ static void bdi_wb_shutdown(struct backing_dev_info
 *bdi)
        if (bdi->wb.task) {
                thaw_process(bdi->wb.task);
                kthread_stop(bdi->wb.task);
+               bdi->wb.task = NULL;
        }
 }
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

--
1.7.0.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#4 lore raw
Quoted 34 lines · Chanho Min
Quoted 5 lines
Quoted 5 lines · ����ȣ
Quoted 4 lines

from Chanho Min <[email protected]>

System may crash in backing-dev.c when removal SCSI device is detached. bdi task is killed by bdi_unregister()/'khubd', but task's point

remains.

Quoted 13 lines
Quoted 13 lines
Quoted 4 lines

Shortly afterward, If 'wb->wakeup_timer' is expired before del_timer()/bdi_forker_thread,
wakeup_timer_fn() may wake up the dead thread which cause the crash. 'bdi->wb.task' should be NULL as this patch.

Is it some race condition between del_timer() and del_timer_sync()?

bdi_unregister() calls

del_timer_sync
bdi_wb_shutdown
kthread_stop

Quoted 5 lines
Quoted 3 lines

in turn, and del_timer_sync() should guarantee wakeup_timer_fn() is no longer called to access the stopped task.

It is not race condition. This happens when USB is removed during write-

access.

Quoted 1 line

bdi_wakeup_thread_delayed is called after kthread_stop, and timer is

activated again.

Quoted 5 lines

bdi_unregister
kthread_stop
bdi_wakeup_thread_delayed (sys_write mostly calls this) timer fires

Quoted 6 lines

Ah OK, the timer could be restarted in the mean while, which breaks the synchronization rule in del_timer_sync().

I noticed a related fix is merged recently, does your test kernel contain this commit?

No, I will try to reproduce with this patch.
But, bdi_destroy is not called during write-access. Same result is expected.

Quoted 4 lines

commit 7a401a972df8e184b3d1a3fc958c0a4ddee8d312

Author: Rabin Vincent <[email protected]>
Date:   Fri Nov 11 13:29:04 2011 +0100
Quoted 8 lines

backing-dev: ensure wakeup_timer is deleted

Quoted 1 line

Anyway,Is this safeguard to prevent from waking up killed thread?

This patch makes no guarantee wakeup_timer_fn() will see NULL bdi->wb.task before the task is stopped, so there is still race conditions. And still, the complete fix would be to prevent wakeup_timer_fn() from being called at all.

If wakeup_timer_fn() see NULL bdi->wb.task, wakeup_timer_fn regards task as killed and wake up forker thread instead of the defined thread.
Is this intended behavior of the bdi?

Quoted 22 lines

Thanks,
Fengguang

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
---
 mm/backing-dev.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
mm/backing-dev.c +1 -0
index 71034f4..4378a5e 100644
@@ -607,6 +607,7 @@ static void bdi_wb_shutdown(struct backing_dev_info
 *bdi)
        if (bdi->wb.task) {
                thaw_process(bdi->wb.task);
                kthread_stop(bdi->wb.task);
+               bdi->wb.task = NULL;
        }
 }
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

--
1.7.0.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#5 lore raw
In reply to Wu Fengguang
Quoted 34 lines · Chanho Min
Quoted 5 lines
Quoted 5 lines · ����ȣ
Quoted 4 lines

from Chanho Min <[email protected]>

System may crash in backing-dev.c when removal SCSI device is detached. bdi task is killed by bdi_unregister()/'khubd', but task's point

remains.

Quoted 13 lines
Quoted 13 lines
Quoted 4 lines

Shortly afterward, If 'wb->wakeup_timer' is expired before del_timer()/bdi_forker_thread,
wakeup_timer_fn() may wake up the dead thread which cause the crash. 'bdi->wb.task' should be NULL as this patch.

Is it some race condition between del_timer() and del_timer_sync()?

bdi_unregister() calls

del_timer_sync
bdi_wb_shutdown
kthread_stop

Quoted 5 lines
Quoted 3 lines

in turn, and del_timer_sync() should guarantee wakeup_timer_fn() is no longer called to access the stopped task.

It is not race condition. This happens when USB is removed during write-

access.

Quoted 1 line

bdi_wakeup_thread_delayed is called after kthread_stop, and timer is

activated again.

Quoted 5 lines

bdi_unregister
kthread_stop
bdi_wakeup_thread_delayed (sys_write mostly calls this) timer fires

Quoted 6 lines

Ah OK, the timer could be restarted in the mean while, which breaks the synchronization rule in del_timer_sync().

I noticed a related fix is merged recently, does your test kernel contain this commit?

No, I will try to reproduce with this patch.
But, bdi_destroy is not called during write-access. Same result is expected.

Quoted 4 lines

commit 7a401a972df8e184b3d1a3fc958c0a4ddee8d312

Author: Rabin Vincent <[email protected]>
Date:   Fri Nov 11 13:29:04 2011 +0100
Quoted 8 lines

backing-dev: ensure wakeup_timer is deleted

Quoted 1 line

Anyway,Is this safeguard to prevent from waking up killed thread?

This patch makes no guarantee wakeup_timer_fn() will see NULL bdi->wb.task before the task is stopped, so there is still race conditions. And still, the complete fix would be to prevent wakeup_timer_fn() from being called at all.

If wakeup_timer_fn() see NULL bdi->wb.task, wakeup_timer_fn regards task as killed and wake up forker thread instead of the defined thread.
Is this intended behavior of the bdi?

Quoted 22 lines

Thanks,
Fengguang

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
---
 mm/backing-dev.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
mm/backing-dev.c +1 -0
index 71034f4..4378a5e 100644
@@ -607,6 +607,7 @@ static void bdi_wb_shutdown(struct backing_dev_info
 *bdi)
        if (bdi->wb.task) {
                thaw_process(bdi->wb.task);
                kthread_stop(bdi->wb.task);
+               bdi->wb.task = NULL;
        }
 }
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

--
1.7.0.4

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>