tty: fix data race in flush_to_ldisc
PATCH 1/1 tty: fix data race in flush_to_ldisc
The data race is found with KernelThreadSanitizer (on rev 21bdb584af8c):
ThreadSanitizer: data-race in release_tty
Write of size 8 by thread T325 (K2579):
release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688
tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920
__fput+0x15f/0x310 fs/file_table.c:207
____fput+0x1d/0x30 fs/file_table.c:243
task_work_run+0x115/0x130 kernel/task_work.c:123
do_notify_resume+0x73/0x80
tracehook_notify_resume include/linux/tracehook.h:190
do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757
int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326
Previous read of size 8 by thread T19 (K16):
flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472
process_one_work+0x47e/0x930 kernel/workqueue.c:2036
worker_thread+0xb0/0x900 kernel/workqueue.c:2170
kthread+0x150/0x170 kernel/kthread.c:207flush_to_ldisc reads port->itty and checks that it is not NULL, concurrently release_tty sets port->itty to NULL. It is possible that flush_to_ldisc loads port->itty once, ensures that it is not NULL, but then reloads it again and uses. The second load can already return NULL, which will cause a crash.
Set port->itty to NULL after shutting down flush_to_ldisc work.
--- drivers/tty/tty_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/tty/tty_io.c +1 -1
@@ -1684,9 +1684,9 @@ static void release_tty(struct tty_struct *tty, int idx)tty_free_termios (tty);tty_driver_remove_tty (tty->driver, tty); tty->port->itty =NULL ; +cancel_work_sync (&tty->port->buf.work);if (tty->link) tty->link->port->itty =NULL ; -cancel_work_sync (&tty->port->buf.work);tty_kref_put (tty->link);tty_kref_put (tty);
Signature · 1 line
2.5.0.457.gab17608
Quoted 37 lines · Dmitry Vyukov
The data race is found with KernelThreadSanitizer (on rev
21bdb584af8c):ThreadSanitizer: data-race in
release_ttyWrite of size 8 by thread T325 (K2579): release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688 tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920 __fput+0x15f/0x310 fs/file_table.c:207 ____fput+0x1d/0x30 fs/file_table.c:243 task_work_run+0x115/0x130 kernel/task_work.c:123 do_notify_resume+0x73/0x80 tracehook_notify_resume include/linux/tracehook.h:190 do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757 int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326 Previous read of size 8 by thread T19 (K16): flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472 process_one_work+0x47e/0x930 kernel/workqueue.c:2036 worker_thread+0xb0/0x900 kernel/workqueue.c:2170 kthread+0x150/0x170 kernel/kthread.c:207
flush_to_ldiscreadsport->ittyand checks that it is not NULL, concurrentlyrelease_ttysetsport->ittyto NULL. It is possible thatflush_to_ldiscloadsport->ittyonce, ensures that it is not NULL, but then reloads it again and uses. The second load can already return NULL, which will cause a crash.Set
port->ittyto NULL after shutting downflush_to_ldiscwork.Signed-off-by Dmitry Vyukov <[email protected]>--- drivers/tty/tty_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/tty/tty_io.c +0
-0
drivers/tty/tty_io.c static void release_tty(struct tty_struct *tty, int idx) · line 1689
Quoted 7 lines
drivers/tty/tty_io.c +1
-1@@ -1684,9 +1684,9 @@ static void release_tty(struct tty_struct *tty, int idx)tty_free_termios (tty);tty_driver_remove_tty (tty->driver, tty); tty->port->itty =NULL ; +cancel_work_sync (&tty->port->buf.work);if (tty->link) tty->link->port->itty =NULL ; -cancel_work_sync (&tty->port->buf.work);
This patch doesn't do anything.
Regards,
Peter Hurley
Quoted 4 lines
drivers/tty/tty_io.c +0
-0tty_kref_put (tty->link);tty_kref_put (tty);
Quoted 33 lines · Peter Hurley
Quoted 32 lines · Dmitry Vyukov
The data race is found with KernelThreadSanitizer (on rev
21bdb584af8c):ThreadSanitizer: data-race in
release_ttyWrite of size 8 by thread T325 (K2579): release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688 tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920 __fput+0x15f/0x310 fs/file_table.c:207 ____fput+0x1d/0x30 fs/file_table.c:243 task_work_run+0x115/0x130 kernel/task_work.c:123 do_notify_resume+0x73/0x80 tracehook_notify_resume include/linux/tracehook.h:190 do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757 int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326 Previous read of size 8 by thread T19 (K16): flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472 process_one_work+0x47e/0x930 kernel/workqueue.c:2036 worker_thread+0xb0/0x900 kernel/workqueue.c:2170 kthread+0x150/0x170 kernel/kthread.c:207
flush_to_ldiscreadsport->ittyand checks that it is not NULL, concurrentlyrelease_ttysetsport->ittyto NULL. It is possible thatflush_to_ldiscloadsport->ittyonce, ensures that it is not NULL, but then reloads it again and uses. The second load can already return NULL, which will cause a crash.Set
port->ittyto NULL after shutting downflush_to_ldiscwork.Signed-off-by Dmitry Vyukov <[email protected]>--- drivers/tty/tty_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 14 lines
drivers/tty/tty_io.c static void release_tty(struct tty_struct *tty, int idx) · line 1689
Quoted 12 lines
drivers/tty/tty_io.c +1
-1@@ -1684,9 +1684,9 @@ static void release_tty(struct tty_struct *tty, int idx)tty_free_termios (tty);tty_driver_remove_tty (tty->driver, tty); tty->port->itty =NULL ; +cancel_work_sync (&tty->port->buf.work);if (tty->link) tty->link->port->itty =NULL ; -cancel_work_sync (&tty->port->buf.work);This patch doesn't do anything.
It sets tty->link->port->itty to NULL _after_ waiting for flush_to_ldisc work to finish. At least this is the intention. Can you explain why it does not do anything?
Thank you.
Quoted 49 lines · Dmitry Vyukov
Quoted 33 lines · Peter Hurley
Quoted 32 lines · Dmitry Vyukov
The data race is found with KernelThreadSanitizer (on rev
21bdb584af8c):ThreadSanitizer: data-race in
release_ttyWrite of size 8 by thread T325 (K2579): release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688 tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920 __fput+0x15f/0x310 fs/file_table.c:207 ____fput+0x1d/0x30 fs/file_table.c:243 task_work_run+0x115/0x130 kernel/task_work.c:123 do_notify_resume+0x73/0x80 tracehook_notify_resume include/linux/tracehook.h:190 do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757 int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326 Previous read of size 8 by thread T19 (K16): flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472 process_one_work+0x47e/0x930 kernel/workqueue.c:2036 worker_thread+0xb0/0x900 kernel/workqueue.c:2170 kthread+0x150/0x170 kernel/kthread.c:207
flush_to_ldiscreadsport->ittyand checks that it is not NULL, concurrentlyrelease_ttysetsport->ittyto NULL. It is possible thatflush_to_ldiscloadsport->ittyonce, ensures that it is not NULL, but then reloads it again and uses. The second load can already return NULL, which will cause a crash.Set
port->ittyto NULL after shutting downflush_to_ldiscwork.Signed-off-by Dmitry Vyukov <[email protected]>--- drivers/tty/tty_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 14 lines
drivers/tty/tty_io.c static void release_tty(struct tty_struct *tty, int idx) · line 1689
Quoted 12 lines
drivers/tty/tty_io.c +1
-1@@ -1684,9 +1684,9 @@ static void release_tty(struct tty_struct *tty, int idx)tty_free_termios (tty);tty_driver_remove_tty (tty->driver, tty); tty->port->itty =NULL ; +cancel_work_sync (&tty->port->buf.work);if (tty->link) tty->link->port->itty =NULL ; -cancel_work_sync (&tty->port->buf.work);This patch doesn't do anything.
Quoted 3 lines
It sets
tty->link->port->ittyto NULL _after_ waiting forflush_to_ldiscwork to finish. At least this is the intention. Can you explain why it does not do anything?
Because tty->link is a different tty than that corresponding to the buffer work being cancelled.
Regards,
Peter Hurley
PATCH 1/1 tty: fix data race in flush_to_ldisc
The data race is found with KernelThreadSanitizer (on rev 21bdb584af8c):
ThreadSanitizer: data-race in release_tty
Write of size 8 by thread T325 (K2579):
release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688
tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920
__fput+0x15f/0x310 fs/file_table.c:207
____fput+0x1d/0x30 fs/file_table.c:243
task_work_run+0x115/0x130 kernel/task_work.c:123
do_notify_resume+0x73/0x80
tracehook_notify_resume include/linux/tracehook.h:190
do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757
int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326
Previous read of size 8 by thread T19 (K16):
flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472
process_one_work+0x47e/0x930 kernel/workqueue.c:2036
worker_thread+0xb0/0x900 kernel/workqueue.c:2170
kthread+0x150/0x170 kernel/kthread.c:207flush_to_ldisc reads port->itty and checks that it is not NULL, concurrently release_tty sets port->itty to NULL. It is possible that flush_to_ldisc loads port->itty once, ensures that it is not NULL, but then reloads it again and uses. The second load can already return NULL, which will cause a crash.
Use READ_ONCE/WRITE_ONCE to read/update port->itty.
--- drivers/tty/tty_buffer.c | 2 +- drivers/tty/tty_io.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
drivers/tty/tty_buffer.c +1 -1
@@ -469,7 +469,7 @@ static void flush_to_ldisc(struct work_struct *work)struct tty_struct *tty;struct tty_ldisc *disc; - tty = port->itty; + tty =READ_ONCE (port->itty);if (tty ==NULL )return ;
drivers/tty/tty_io.c +1 -1
@@ -1685,7 +1685,7 @@ static void release_tty(struct tty_struct *tty, int idx)tty_driver_remove_tty (tty->driver, tty); tty->port->itty =NULL ;if (tty->link) - tty->link->port->itty =NULL ; +WRITE_ONCE (tty->link->port->itty,NULL );cancel_work_sync (&tty->port->buf.work);tty_kref_put (tty->link);
Signature · 1 line
2.5.0.457.gab17608
Quoted 18 lines · Dmitry Vyukov
The data race is found with KernelThreadSanitizer (on rev
21bdb584af8c):ThreadSanitizer: data-race in
release_ttyWrite of size 8 by thread T325 (K2579): release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688 tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920 __fput+0x15f/0x310 fs/file_table.c:207 ____fput+0x1d/0x30 fs/file_table.c:243 task_work_run+0x115/0x130 kernel/task_work.c:123 do_notify_resume+0x73/0x80 tracehook_notify_resume include/linux/tracehook.h:190 do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757 int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326 Previous read of size 8 by thread T19 (K16): flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472 process_one_work+0x47e/0x930 kernel/workqueue.c:2036 worker_thread+0xb0/0x900 kernel/workqueue.c:2170 kthread+0x150/0x170 kernel/kthread.c:207
The stack traces are not really helpful in describing how the race occurs; I would leave it out of the changelog.
Quoted 7 lines
flush_to_ldiscreadsport->ittyand checks that it is not NULL, concurrentlyrelease_ttysetsport->ittyto NULL. It is possible thatflush_to_ldiscloadsport->ittyonce, ensures that it is not NULL, but then reloads it again and uses. The second load can already return NULL, which will cause a crash.Use READ_ONCE/WRITE_ONCE to read/update
port->itty.
See below.
Quoted 11 lines
Signed-off-by Dmitry Vyukov <[email protected]>--- drivers/tty/tty_buffer.c | 2 +- drivers/tty/tty_io.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)drivers/tty/tty_buffer.c +0
-0
drivers/tty/tty_buffer.c static void flush_to_ldisc(struct work_struct *work) · line 472
Quoted 5 lines
drivers/tty/tty_buffer.c +1
-1@@ -469,7 +469,7 @@ static void flush_to_ldisc(struct work_struct *work)struct tty_struct *tty;struct tty_ldisc *disc; - tty = port->itty; + tty =READ_ONCE (port->itty);
This is fine.
drivers/tty/tty_io.c static void release_tty(struct tty_struct *tty, int idx) · line 1688
Quoted 13 lines
drivers/tty/tty_buffer.c +0
-0if (tty ==NULL )return ;drivers/tty/tty_io.c +1
-1@@ -1685,7 +1685,7 @@ static void release_tty(struct tty_struct *tty, int idx)tty_driver_remove_tty (tty->driver, tty); tty->port->itty =NULL ;if (tty->link) - tty->link->port->itty =NULL ; +WRITE_ONCE (tty->link->port->itty,NULL );
This isn't doing anything useful.
1. The compiler can't push the store past the cancel_work_sync() (because the compiler has no visibility into cancel_work_sync()), and,
2. There's no effect if the compiler hoists the store higher in the release_tty() because the line discipline has already been closed and killed (so the tty_ldisc_ref() in flush_to_ldisc() returns NULL anyway).
Regards,
Peter Hurley
Quoted 4 lines
drivers/tty/tty_io.c +0
-0cancel_work_sync (&tty->port->buf.work);tty_kref_put (tty->link);
Quoted 20 lines · Peter Hurley
Quoted 18 lines · Dmitry Vyukov
The data race is found with KernelThreadSanitizer (on rev
21bdb584af8c):ThreadSanitizer: data-race in
release_ttyWrite of size 8 by thread T325 (K2579): release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688 tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920 __fput+0x15f/0x310 fs/file_table.c:207 ____fput+0x1d/0x30 fs/file_table.c:243 task_work_run+0x115/0x130 kernel/task_work.c:123 do_notify_resume+0x73/0x80 tracehook_notify_resume include/linux/tracehook.h:190 do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757 int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326 Previous read of size 8 by thread T19 (K16): flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472 process_one_work+0x47e/0x930 kernel/workqueue.c:2036 worker_thread+0xb0/0x900 kernel/workqueue.c:2170 kthread+0x150/0x170 kernel/kthread.c:207
Quoted 2 lines
The stack traces are not really helpful in describing how the race occurs; I would leave it out of the changelog.
ok
Quoted 45 lines
Quoted 7 lines
flush_to_ldiscreadsport->ittyand checks that it is not NULL, concurrentlyrelease_ttysetsport->ittyto NULL. It is possible thatflush_to_ldiscloadsport->ittyonce, ensures that it is not NULL, but then reloads it again and uses. The second load can already return NULL, which will cause a crash.Use READ_ONCE/WRITE_ONCE to read/update
port->itty.See below.
Quoted 11 lines
Signed-off-by Dmitry Vyukov <[email protected]>--- drivers/tty/tty_buffer.c | 2 +- drivers/tty/tty_io.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)drivers/tty/tty_buffer.c +0
-0drivers/tty/tty_buffer.c static void flush_to_ldisc(struct work_struct *work) · line 472
Quoted 5 lines
drivers/tty/tty_buffer.c +1
-1@@ -469,7 +469,7 @@ static void flush_to_ldisc(struct work_struct *work)struct tty_struct *tty;struct tty_ldisc *disc; - tty = port->itty; + tty =READ_ONCE (port->itty);This is fine.
drivers/tty/tty_io.c static void release_tty(struct tty_struct *tty, int idx) · line 1688
Quoted 13 lines
drivers/tty/tty_buffer.c +0
-0if (tty ==NULL )return ;drivers/tty/tty_io.c +1
-1@@ -1685,7 +1685,7 @@ static void release_tty(struct tty_struct *tty, int idx)tty_driver_remove_tty (tty->driver, tty); tty->port->itty =NULL ;if (tty->link) - tty->link->port->itty =NULL ; +WRITE_ONCE (tty->link->port->itty,NULL );This isn't doing anything useful.
Quoted 5 lines
1. The compiler can't push the store past the
cancel_work_sync()(because the compiler has no visibility intocancel_work_sync()), and,
2. There's no effect if the compiler hoists the store higher in therelease_tty()because the line discipline has already been closed and killed (so thetty_ldisc_ref()inflush_to_ldisc()returns NULL anyway).
OK, let me do one try at convincing you that WRITE_ONCE here is a good idea. If you are not convinced then I will remove it.
WRITE_ONCE/READ_ONCE for all shared memory accesses:
1. Make the code more readable but highlighting important aspects.
2. Required by relevant standards and relieve you, me and everybody else reading this code from spending time on proving that it cannot break (think of multi-file compilation mode, store tearing which compilers indeed known to do in some contexts, and compiler transformations that we don't know of).
3. Allow tooling that finds undoubtedly harmful bugs, like this one, or in fact, I've just mailed another fix for missed memory barriers in this file (also found by KTSAN).
I've described these aspects in more detail here:
https://github.com/google/ktsan/wiki/READ_ONCE-and-WRITE_ONCE
I don't see any negative aspects to it. Do you see any? Because if you see at least some value in at least on these points and don't see any negative aspects, then it is worth doing.
Signature · 11 lines
Dmitry Vyukov, Software Engineer, [email protected] Google Germany GmbH, Dienerstraße 12, 80331, München Geschäftsführer: Graham Law, Christine Elizabeth Flores Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind, leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen Sie die E-Mail und alle Anhänge. Vielen Dank. This e-mail is confidential. If you are not the right addressee please do not forward it, please inform the sender, and please erase this e-mail including any attachments. Thanks.
Quoted 21 lines · Dmitry Vyukov
Quoted 20 lines · Peter Hurley
Quoted 18 lines · Dmitry Vyukov
The data race is found with KernelThreadSanitizer (on rev
21bdb584af8c):ThreadSanitizer: data-race in
release_ttyWrite of size 8 by thread T325 (K2579): release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688 tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920 __fput+0x15f/0x310 fs/file_table.c:207 ____fput+0x1d/0x30 fs/file_table.c:243 task_work_run+0x115/0x130 kernel/task_work.c:123 do_notify_resume+0x73/0x80 tracehook_notify_resume include/linux/tracehook.h:190 do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757 int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326 Previous read of size 8 by thread T19 (K16): flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472 process_one_work+0x47e/0x930 kernel/workqueue.c:2036 worker_thread+0xb0/0x900 kernel/workqueue.c:2170 kthread+0x150/0x170 kernel/kthread.c:207
Quoted 4 lines
Quoted 2 lines
The stack traces are not really helpful in describing how the race occurs; I would leave it out of the changelog.
ok
Just to clarify; my comment is only wrt this particular patch. You may have other patches in the future with unclear execution paths that may require the call stacks. It's just that this particular race has invariant call stacks (for the relevant parts).
Quoted 45 lines
Quoted 45 lines
Quoted 7 lines
flush_to_ldiscreadsport->ittyand checks that it is not NULL, concurrentlyrelease_ttysetsport->ittyto NULL. It is possible thatflush_to_ldiscloadsport->ittyonce, ensures that it is not NULL, but then reloads it again and uses. The second load can already return NULL, which will cause a crash.Use READ_ONCE/WRITE_ONCE to read/update
port->itty.See below.
Quoted 11 lines
Signed-off-by Dmitry Vyukov <[email protected]>--- drivers/tty/tty_buffer.c | 2 +- drivers/tty/tty_io.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)drivers/tty/tty_buffer.c +0
-0drivers/tty/tty_buffer.c static void flush_to_ldisc(struct work_struct *work) · line 472
Quoted 5 lines
drivers/tty/tty_buffer.c +1
-1@@ -469,7 +469,7 @@ static void flush_to_ldisc(struct work_struct *work)struct tty_struct *tty;struct tty_ldisc *disc; - tty = port->itty; + tty =READ_ONCE (port->itty);This is fine.
drivers/tty/tty_io.c static void release_tty(struct tty_struct *tty, int idx) · line 1688
Quoted 13 lines
drivers/tty/tty_buffer.c +0
-0if (tty ==NULL )return ;drivers/tty/tty_io.c +1
-1@@ -1685,7 +1685,7 @@ static void release_tty(struct tty_struct *tty, int idx)tty_driver_remove_tty (tty->driver, tty); tty->port->itty =NULL ;if (tty->link) - tty->link->port->itty =NULL ; +WRITE_ONCE (tty->link->port->itty,NULL );This isn't doing anything useful.
Quoted 8 lines
Quoted 5 lines
1. The compiler can't push the store past the
cancel_work_sync()(because the compiler has no visibility intocancel_work_sync()), and,
2. There's no effect if the compiler hoists the store higher in therelease_tty()because the line discipline has already been closed and killed (so thetty_ldisc_ref()inflush_to_ldisc()returns NULL anyway).OK, let me do one try at convincing you that
WRITE_ONCEhere is a good idea. If you are not convinced then I will remove it.
FWIW, I'm not the gatekeeper wrt tree-wide code/best-practices changes; iow, convincing me is not a useful goal. But I think this would be a good topic/
presentation for either Kernel Summit or Linux Plumbers.
That said, I'll make some observations below that you should expect to read from other contributors/maintainers regarding your point-of-view.
Quoted 2 lines
WRITE_ONCE/READ_ONCE for all shared memory accesses:
1. Make the code more readable but highlighting important aspects.
Most would argue the additional macros detract from readability.
Quoted 5 lines
2. Required by relevant standards and relieve you, me and everybody else reading this code from spending time on proving that it cannot break (think of multi-file compilation mode, store tearing which compilers indeed known to do in some contexts, and compiler transformations that we don't know of).
Multi-file compilation and the compiler memory model are inherently incompatible with kernel code. Instrumenting code rather than specializing the tool (compiler) is progress in the wrong direction, imho.
Same with store tearing of primitive types.
Compiler "transforms" are an occasional hazard of kernel development, as are straight-up compiler bugs; in my limited experience, each occurs with equal frequency.
Quoted 1 line
3. Allow tooling that finds undoubtedly harmful bugs, like this one,
While the READ_ONCE() fix improves robustness, I wouldn't categorize this as a 'harmful' bug. I seriously doubt any compiler has generated a reload of port->itty in flush_to_ldisc(). As such, I would disagree with any submission to stable kernels for this fix.
Quoted 2 lines
or in fact, I've just mailed another fix for missed memory barriers in this file (also found by KTSAN).
I really appreciate you uncovering those. But note that READ_ONCE/ WRITE_ONCE aren't necessary to _find_ those bugs, but rather to eliminate false positives (which I can sympathize with).
And in fact, with every potential race, human analysis is required anyway.
Quoted 6 lines
I've described these aspects in more detail here:
https://github.com/google/ktsan/wiki/READ_ONCE-and-WRITE_ONCEI don't see any negative aspects to it. Do you see any? Because if you see at least some value in at least on these points and don't see any negative aspects, then it is worth doing.
Consider the converse suggestion; everywhere I even smell race I add READ_ONCE/WRITE_ONCE. Kernel code would become unreadable, (and slower on hot paths). Consequently, each change will need to be justified, and thus, we've arrived where we started: analyzing each race on its own merits and fixing real races.
Regards,
Peter Hurley
Quoted 139 lines · Peter Hurley
Quoted 21 lines · Dmitry Vyukov
Quoted 20 lines · Peter Hurley
Quoted 18 lines · Dmitry Vyukov
The data race is found with KernelThreadSanitizer (on rev
21bdb584af8c):ThreadSanitizer: data-race in
release_ttyWrite of size 8 by thread T325 (K2579): release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688 tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920 __fput+0x15f/0x310 fs/file_table.c:207 ____fput+0x1d/0x30 fs/file_table.c:243 task_work_run+0x115/0x130 kernel/task_work.c:123 do_notify_resume+0x73/0x80 tracehook_notify_resume include/linux/tracehook.h:190 do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757 int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326 Previous read of size 8 by thread T19 (K16): flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472 process_one_work+0x47e/0x930 kernel/workqueue.c:2036 worker_thread+0xb0/0x900 kernel/workqueue.c:2170 kthread+0x150/0x170 kernel/kthread.c:207Quoted 4 lines
Quoted 2 lines
The stack traces are not really helpful in describing how the race occurs; I would leave it out of the changelog.
ok
Just to clarify; my comment is only wrt this particular patch. You may have other patches in the future with unclear execution paths that may require the call stacks. It's just that this particular race has invariant call stacks (for the relevant parts).
Quoted 45 lines
Quoted 45 lines
Quoted 7 lines
flush_to_ldiscreadsport->ittyand checks that it is not NULL, concurrentlyrelease_ttysetsport->ittyto NULL. It is possible thatflush_to_ldiscloadsport->ittyonce, ensures that it is not NULL, but then reloads it again and uses. The second load can already return NULL, which will cause a crash.Use READ_ONCE/WRITE_ONCE to read/update
port->itty.See below.
Quoted 11 lines
Signed-off-by Dmitry Vyukov <[email protected]>--- drivers/tty/tty_buffer.c | 2 +- drivers/tty/tty_io.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)drivers/tty/tty_buffer.c +0
-0drivers/tty/tty_buffer.c static void flush_to_ldisc(struct work_struct *work) · line 472
Quoted 5 lines
drivers/tty/tty_buffer.c +1
-1@@ -469,7 +469,7 @@ static void flush_to_ldisc(struct work_struct *work)struct tty_struct *tty;struct tty_ldisc *disc; - tty = port->itty; + tty =READ_ONCE (port->itty);This is fine.
drivers/tty/tty_io.c static void release_tty(struct tty_struct *tty, int idx) · line 1688
Quoted 13 lines
drivers/tty/tty_buffer.c +0
-0if (tty ==NULL )return ;drivers/tty/tty_io.c +1
-1@@ -1685,7 +1685,7 @@ static void release_tty(struct tty_struct *tty, int idx)tty_driver_remove_tty (tty->driver, tty); tty->port->itty =NULL ;if (tty->link) - tty->link->port->itty =NULL ; +WRITE_ONCE (tty->link->port->itty,NULL );This isn't doing anything useful.
Quoted 8 lines
Quoted 5 lines
1. The compiler can't push the store past the
cancel_work_sync()(because the compiler has no visibility intocancel_work_sync()), and,
2. There's no effect if the compiler hoists the store higher in therelease_tty()because the line discipline has already been closed and killed (so thetty_ldisc_ref()inflush_to_ldisc()returns NULL anyway).OK, let me do one try at convincing you that
WRITE_ONCEhere is a good idea. If you are not convinced then I will remove it.FWIW, I'm not the gatekeeper wrt tree-wide code/best-practices changes; iow, convincing me is not a useful goal. But I think this would be a good topic/
presentation for either Kernel Summit or Linux Plumbers.That said, I'll make some observations below that you should expect to read from other contributors/maintainers regarding your point-of-view.
Quoted 2 lines
WRITE_ONCE/READ_ONCE for all shared memory accesses:
1. Make the code more readable but highlighting important aspects.Most would argue the additional macros detract from readability.
Quoted 5 lines
2. Required by relevant standards and relieve you, me and everybody else reading this code from spending time on proving that it cannot break (think of multi-file compilation mode, store tearing which compilers indeed known to do in some contexts, and compiler transformations that we don't know of).
Multi-file compilation and the compiler memory model are inherently incompatible with kernel code. Instrumenting code rather than specializing the tool (compiler) is progress in the wrong direction, imho.
Same with store tearing of primitive types.
Compiler "transforms" are an occasional hazard of kernel development, as are straight-up compiler bugs; in my limited experience, each occurs with equal frequency.
Quoted 1 line
3. Allow tooling that finds undoubtedly harmful bugs, like this one,
While the
READ_ONCE()fix improves robustness, I wouldn't categorize this as a 'harmful' bug. I seriously doubt any compiler has generated a reload ofport->ittyinflush_to_ldisc(). As such, I would disagree with any submission to stable kernels for this fix.Quoted 2 lines
or in fact, I've just mailed another fix for missed memory barriers in this file (also found by KTSAN).
I really appreciate you uncovering those. But note that READ_ONCE/
WRITE_ONCEaren't necessary to _find_ those bugs, but rather to eliminate false positives (which I can sympathize with).And in fact, with every potential race, human analysis is required anyway.
Quoted 6 lines
I've described these aspects in more detail here:
https://github.com/google/ktsan/wiki/READ_ONCE-and-WRITE_ONCEI don't see any negative aspects to it. Do you see any? Because if you see at least some value in at least on these points and don't see any negative aspects, then it is worth doing.
Quoted 5 lines
Consider the converse suggestion; everywhere I even smell race I add READ_ONCE/WRITE_ONCE. Kernel code would become unreadable, (and slower on hot paths). Consequently, each change will need to be justified, and thus, we've arrived where we started: analyzing each race on its own merits and fixing real races.
Thanks for the detailed explanation, Peter!
Signature · 11 lines
Dmitry Vyukov, Software Engineer, [email protected] Google Germany GmbH, Dienerstraße 12, 80331, München Geschäftsführer: Graham Law, Christine Elizabeth Flores Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Diese E-Mail ist vertraulich. Wenn Sie nicht der richtige Adressat sind, leiten Sie diese bitte nicht weiter, informieren Sie den Absender und löschen Sie die E-Mail und alle Anhänge. Vielen Dank. This e-mail is confidential. If you are not the right addressee please do not forward it, please inform the sender, and please erase this e-mail including any attachments. Thanks.
Quoted 50 lines · Peter Hurley
Quoted 49 lines · Dmitry Vyukov
Quoted 47 lines · Peter Hurley
Quoted 37 lines · Dmitry Vyukov
The data race is found with KernelThreadSanitizer (on rev
21bdb584af8c):ThreadSanitizer: data-race in
release_ttyWrite of size 8 by thread T325 (K2579): release_tty+0xf3/0x1c0 drivers/tty/tty_io.c:1688 tty_release+0x698/0x7c0 drivers/tty/tty_io.c:1920 __fput+0x15f/0x310 fs/file_table.c:207 ____fput+0x1d/0x30 fs/file_table.c:243 task_work_run+0x115/0x130 kernel/task_work.c:123 do_notify_resume+0x73/0x80 tracehook_notify_resume include/linux/tracehook.h:190 do_notify_resume+0x73/0x80 arch/x86/kernel/signal.c:757 int_signal+0x12/0x17 arch/x86/entry/entry_64.S:326 Previous read of size 8 by thread T19 (K16): flush_to_ldisc+0x29/0x300 drivers/tty/tty_buffer.c:472 process_one_work+0x47e/0x930 kernel/workqueue.c:2036 worker_thread+0xb0/0x900 kernel/workqueue.c:2170 kthread+0x150/0x170 kernel/kthread.c:207
flush_to_ldiscreadsport->ittyand checks that it is not NULL, concurrentlyrelease_ttysetsport->ittyto NULL. It is possible thatflush_to_ldiscloadsport->ittyonce, ensures that it is not NULL, but then reloads it again and uses. The second load can already return NULL, which will cause a crash.Set
port->ittyto NULL after shutting downflush_to_ldiscwork.Signed-off-by Dmitry Vyukov <[email protected]>--- drivers/tty/tty_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/tty/tty_io.c +0
-0drivers/tty/tty_io.c static void release_tty(struct tty_struct *tty, int idx) · line 1689
Quoted 7 lines
drivers/tty/tty_io.c +1
-1@@ -1684,9 +1684,9 @@ static void release_tty(struct tty_struct *tty, int idx)tty_free_termios (tty);tty_driver_remove_tty (tty->driver, tty); tty->port->itty =NULL ; +cancel_work_sync (&tty->port->buf.work);if (tty->link) tty->link->port->itty =NULL ; -cancel_work_sync (&tty->port->buf.work);This patch doesn't do anything.
Quoted 6 lines
Quoted 3 lines
It sets
tty->link->port->ittyto NULL _after_ waiting forflush_to_ldiscwork to finish. At least this is the intention. Can you explain why it does not do anything?Because
tty->linkis a different tty than that corresponding to the buffer work being cancelled.
Got it.
Please take a look at the updated version of the patch.