Fix races in thread relaxation
cover Fix races in thread relaxation
This should resolve the issue reported in [1] and also clean up a bit more of our now obsolete pipeline abstraction. Test case is planned for later.
Jan
[1] https://lore.kernel.org/xenomai/[email protected]/T/#t
Jan Kiszka (3):
cobalt: Move relax related lostage_wakeup work from stack to thread
cobalt: Drop obsolete pipeline_inband_work abstraction cobalt/posix/process: Sync on thread-hosted irq_work items before
exiting
.../kernel/dovetail/pipeline/inband_work.h | 28 ---------------- include/cobalt/kernel/rtdm/driver.h | 9 +++-- include/cobalt/kernel/thread.h | 10 ++++-- kernel/cobalt/posix/process.c | 5 +++ kernel/cobalt/rtdm/drvlib.c | 15 +++------ kernel/cobalt/rtdm/fd.c | 13 +++----- kernel/cobalt/thread.c | 33 +++++++------------ kernel/drivers/udd/udd.c | 12 +++---- 8 files changed, 45 insertions(+), 80 deletions(-) delete mode 100644 include/cobalt/kernel/dovetail/pipeline/inband_work.h
Signature · 1 line
2.43.0
PATCH 1/3 cobalt: Move relax related lostage_wakeup work from stack to thread
From: Jan Kiszka <[email protected]>
We cannot rely on the lostage_wakeup work being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As the lostage_wakeup work is currently on the stack, it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].
Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using irq_work_sync in the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.
[1] https://lore.kernel.org/xenomai/[email protected]/T/#t
--- include/cobalt/kernel/thread.h | 6 ++++++ kernel/cobalt/thread.c | 15 +++++---------- 2 files changed, 11 insertions(+), 10 deletions(-)
include/cobalt/kernel/thread.h +6 -0
@@ -64,6 +64,11 @@ struct lostage_signal {int signo, sigval; }; +struct lostage_wakeup { +struct pipeline_inband_work inband_work;/* Must be first. */ +struct task_struct *task; +}; +struct xnthread_init_attr {struct xnthread_personality *personality;cpumask_t affinity;
@@ -214,6 +219,7 @@ struct xnthread {u32 proghash;/* Hash value for exe_path */ #endif struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS ]; +struct lostage_wakeup relax_work; };static inline int xnthread_get_state (const struct xnthread *thread)
kernel/cobalt/thread.c +5 -10
@@ -1974,11 +1974,6 @@ int xnthread_harden(void) }EXPORT_SYMBOL_GPL (xnthread_harden); -struct lostage_wakeup { -struct pipeline_inband_work inband_work;/* Must be first. */ -struct task_struct *task; -}; -static void lostage_task_wakeup (struct pipeline_inband_work *inband_work) {struct lostage_wakeup *rq;
@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)void xnthread_relax (int notify,int reason) {struct task_struct *p = current; -struct lostage_wakeup wakework = { - .inband_work =PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup), - .task = p, - };struct xnthread *thread =xnthread_current ();int cpu__maybe_unused , suspension;kernel_siginfo_t si;
@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)*/ trace_cobalt_shadow_gorelax (reason); + thread->relax_work.inband_work = (struct pipeline_inband_work ) +PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup); + thread->relax_work.task = p; +/* * If you intend to change the following interrupt-free * sequence, /first/ make sure to check the special handling
@@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)*/ splmax ();trace_cobalt_lostage_request ("wakeup" , p); -pipeline_post_inband_work (&wakework); +pipeline_post_inband_work (&thread->relax_work);/* * Grab the nklock to synchronize the Linux task state * manipulation with handle_sigwake_event. This lock will be
Signature · 1 line
2.43.0
Re:[PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread
hi Jan, the issue reported in [1] has seemly not been solved in ipipe. Will this issue still be maintained on ipipe?
the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]).
linux kernel: 4.19.89
xenomai: 3.2.4 + patch[1/3]
[1] https://lore.kernel.org/xenomai/[email protected]/T/#t
[ 98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[ 98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[ 98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[ 98.182496] I-pipe domain: Linux
[ 98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[ 98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[ 98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[ 98.182564] RAX: dead000000000100 RBX: 000000000005f3e0 RCX: ffffc900012d2a88
[ 98.182577] RDX: 0000000000000000 RSI: ffffc900012d1840 RDI: 0000000000001032
[ 98.182591] RBP: 0000000000000001 R08: ffffc900012d2a88 R09: ffffc900012d2a78
[ 98.182605] R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900012d3708
[ 98.182620] R13: 0000000000000003 R14: ffffc900012d1840 R15: 0000000000000000
[ 98.182634] FS: 00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[ 98.182650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 98.182661] CR2: 000055738facc000 CR3: 000000012ad44005 CR4: 00000000001606e0
[ 98.182698] Call Trace:
[ 98.182708] __cobalt_cond_wait_prologue+0x26d/0x850
[ 98.182720] ? handle_root_syscall+0xd3/0x310
[ 98.182730] ? __cobalt_cond_wait_prologue+0x850/0x850
[ 98.182741] CoBaLt_cond_wait_prologue+0x1d/0x20
[ 98.182751] handle_head_syscall+0x106/0x3b0
[ 98.182762] ipipe_handle_syscall+0x66/0xc0
[ 98.182772] do_syscall_64+0x22/0x170
[ 98.182782] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 98.182793] RIP: 0033:0x7f888d29575b
[ 98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[ 98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246 ORIG_RAX: 0000000010000023
[ 98.182852] RAX: ffffffffffffffda RBX: 00007f888c56e6e0 RCX: 00007f888d29575b
[ 98.183195] RDX: 00007f888c3baba4 RSI: 00007f888c56e6e0 RDI: 00007f888c56e7f8
[ 98.183530] RBP: 00007f888c56e7f8 R08: 0000000000000000 R09: 00007f888c56e6d0
[ 98.183854] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f888c56e6e0
[ 98.184182] R13: 00007f888c3bad80 R14: 0000000000000001 R15: 00007f888c3babb0
[ 98.184500] Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c arc4 ecb md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs ccm dns_resolver fscache ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter vmwgfx sg ttm drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect sysimgblt serio_raw snd_pcsp snd_pcm snd_timer snd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc evdev aesni_intel aes_x86_64 crypto_simd cryptd glue_helper soundcore ac button binfmt_misc sunrpc [ 98.187446] ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod ata_generic ata_piix crc32c_intel mptspi mptscsih mptbase scsi_transport_spi libata scsi_mod e1000 psmouse i2c_piix4
Quoted 96 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
We cannot rely on the
lostage_wakeupwork being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As thelostage_wakeupwork is currently on the stack, it can become corrupt whenxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using
irq_work_syncin the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.[1] https://lore.kernel.org/xenomai/[email protected]/T/#t
Reported-by JerryJun <[email protected]>Signed-off-by Jan Kiszka <[email protected]>---
include/cobalt/kernel/thread.h| 6 ++++++kernel/cobalt/thread.c| 15 +++++----------2 files changed, 11 insertions(+), 10 deletions(-)include/cobalt/kernel/thread.h +6
-0@@ -64,6 +64,11 @@ struct lostage_signal {int signo, sigval; }; +struct lostage_wakeup { +struct pipeline_inband_work inband_work;/* Must be first. */ +struct task_struct *task; +}; +struct xnthread_init_attr {struct xnthread_personality *personality;cpumask_t affinity;@@ -214,6 +219,7 @@ struct xnthread {u32 proghash;/* Hash value for exe_path */ #endif struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS ]; +struct lostage_wakeup relax_work; };static inline int xnthread_get_state (const struct xnthread *thread)kernel/cobalt/thread.c +5
-10@@ -1974,11 +1974,6 @@ int xnthread_harden(void) }EXPORT_SYMBOL_GPL (xnthread_harden); -struct lostage_wakeup { -struct pipeline_inband_work inband_work;/* Must be first. */ -struct task_struct *task; -}; -static void lostage_task_wakeup (struct pipeline_inband_work *inband_work) {struct lostage_wakeup *rq;@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)void xnthread_relax (int notify,int reason) {struct task_struct *p = current; -struct lostage_wakeup wakework = { - .inband_work =PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup), - .task = p, - };struct xnthread *thread =xnthread_current ();int cpu__maybe_unused , suspension;kernel_siginfo_t si;@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)*/ trace_cobalt_shadow_gorelax (reason); + thread->relax_work.inband_work = (struct pipeline_inband_work ) +PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup); + thread->relax_work.task = p; +/* * If you intend to change the following interrupt-free * sequence, /first/ make sure to check the special handling @@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)*/ splmax ();trace_cobalt_lostage_request ("wakeup" , p); -pipeline_post_inband_work (&wakework); +pipeline_post_inband_work (&thread->relax_work);/* * Grab the nklock to synchronize the Linux task state * manipulation with handle_sigwake_event. This lock will be Signature · 1 line
2.43.0
Re: [PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread
Quoted 8 lines · JerryJun
hi Jan, the issue reported in [1] has seemly not been solved in ipipe. Will this issue still be maintained on ipipe? the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]). linux kernel: 4.19.89
xenomai: 3.2.4 + patch[1/3]
[1] https://lore.kernel.org/xenomai/[email protected]/T/#t
Sorry, that fix is still only on master (upcoming 3.3), not yet in stable trees. It's still not expected to be I-pipe/dovetail-dependent.
Let me recover a bit more from yesterday's Preempt RT merge party, then I will close that gap.
Jan
Quoted 134 lines
[ 98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[ 98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[ 98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[ 98.182496] I-pipe domain: Linux
[ 98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[ 98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[ 98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[ 98.182564] RAX:dead000000000100RBX:000000000005f3e0RCX:ffffc900012d2a88
[ 98.182577] RDX: 0000000000000000 RSI:ffffc900012d1840RDI: 0000000000001032
[ 98.182591] RBP: 0000000000000001 R08:ffffc900012d2a88R09:ffffc900012d2a78
[ 98.182605] R10: 0000000000000000 R11: 0000000000000000 R12:ffffc900012d3708
[ 98.182620] R13: 0000000000000003 R14:ffffc900012d1840R15: 0000000000000000
[ 98.182634] FS: 00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[ 98.182650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 98.182661] CR2:000055738facc000CR3:000000012ad44005CR4:00000000001606e0
[ 98.182698] Call Trace:
[ 98.182708]__cobalt_cond_wait_prologue+0x26d/0x850
[ 98.182720] ?handle_root_syscall+0xd3/0x310
[ 98.182730] ?__cobalt_cond_wait_prologue+0x850/0x850
[ 98.182741]CoBaLt_cond_wait_prologue+0x1d/0x20
[ 98.182751]handle_head_syscall+0x106/0x3b0
[ 98.182762]ipipe_handle_syscall+0x66/0xc0
[ 98.182772]do_syscall_64+0x22/0x170
[ 98.182782]entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 98.182793] RIP: 0033:0x7f888d29575b
[ 98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[ 98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246ORIG_RAX: 0000000010000023
[ 98.182852] RAX: ffffffffffffffda RBX:00007f888c56e6e0RCX:00007f888d29575b
[ 98.183195] RDX:00007f888c3baba4RSI:00007f888c56e6e0RDI:00007f888c56e7f8
[ 98.183530] RBP:00007f888c56e7f8R08: 0000000000000000 R09:00007f888c56e6d0
[ 98.183854] R10: 0000000000000000 R11: 0000000000000246 R12:00007f888c56e6e0
[ 98.184182] R13:00007f888c3bad80R14: 0000000000000001 R15:00007f888c3babb0
[ 98.184500] Modules linked in: fusenf_conntrack_netbios_nsnf_conntrack_broadcastxt_CTxt_tcpudpip6t_rpfilterip6t_REJECTnf_reject_ipv6ipt_REJECTnf_reject_ipv4xt_conntrackip6table_natnf_nat_ipv6ip6table_mangleip6table_rawip6table_securityiptable_natnf_nat_ipv4nf_natiptable_mangleiptable_rawiptable_securitynf_conntracknf_defrag_ipv6nf_defrag_ipv4libcrc32c arc4 ecb md4sha512_ssse3sha512_genericcmacnls_utf8cifs ccmdns_resolverfscacheip_setnf_tablesnfnetlinkip6table_filterip6_tablesiptable_filtervmwgfx sg ttmdrm_kms_helperdrmfb_sys_fopssyscopyarea sysfillrect sysimgbltserio_rawsnd_pcspsnd_pcmsnd_timersndcrct10dif_pclmulcrc32_pclmulghash_clmulni_intelpcbc evdevaesni_intelaes_x86_64crypto_simdcryptdglue_helpersoundcore ac buttonbinfmt_miscsunrpc [ 98.187446]ip_tablesx_tablesautofs4 ext4crc32c_genericcrc16 mbcache jbd2 fscryptosd_modata_genericata_piixcrc32c_intelmptspi mptscsih mptbasescsi_transport_spilibatascsi_mode1000 psmousei2c_piix4Quoted 96 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
We cannot rely on the
lostage_wakeupwork being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As thelostage_wakeupwork is currently on the stack, it can become corrupt whenxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using
irq_work_syncin the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.[1] https://lore.kernel.org/xenomai/[email protected]/T/#t
Reported-by JerryJun <[email protected]>Signed-off-by Jan Kiszka <[email protected]>---
include/cobalt/kernel/thread.h| 6 ++++++kernel/cobalt/thread.c| 15 +++++----------2 files changed, 11 insertions(+), 10 deletions(-)include/cobalt/kernel/thread.h +6
-0@@ -64,6 +64,11 @@ struct lostage_signal {int signo, sigval; }; +struct lostage_wakeup { +struct pipeline_inband_work inband_work;/* Must be first. */ +struct task_struct *task; +}; +struct xnthread_init_attr {struct xnthread_personality *personality;cpumask_t affinity;@@ -214,6 +219,7 @@ struct xnthread {u32 proghash;/* Hash value for exe_path */ #endif struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS ]; +struct lostage_wakeup relax_work; };static inline int xnthread_get_state (const struct xnthread *thread)kernel/cobalt/thread.c +5
-10@@ -1974,11 +1974,6 @@ int xnthread_harden(void) }EXPORT_SYMBOL_GPL (xnthread_harden); -struct lostage_wakeup { -struct pipeline_inband_work inband_work;/* Must be first. */ -struct task_struct *task; -}; -static void lostage_task_wakeup (struct pipeline_inband_work *inband_work) {struct lostage_wakeup *rq;@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)void xnthread_relax (int notify,int reason) {struct task_struct *p = current; -struct lostage_wakeup wakework = { - .inband_work =PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup), - .task = p, - };struct xnthread *thread =xnthread_current ();int cpu__maybe_unused , suspension;kernel_siginfo_t si;@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)*/ trace_cobalt_shadow_gorelax (reason); + thread->relax_work.inband_work = (struct pipeline_inband_work ) +PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup); + thread->relax_work.task = p; +/* * If you intend to change the following interrupt-free * sequence, /first/ make sure to check the special handling @@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)*/ splmax ();trace_cobalt_lostage_request ("wakeup" , p); -pipeline_post_inband_work (&wakework); +pipeline_post_inband_work (&thread->relax_work);/* * Grab the nklock to synchronize the Linux task state * manipulation with handle_sigwake_event. This lock will be Signature · 1 line
2.43.0
Signature · 2 lines
Siemens AG, Technology Linux Expert Center
Re:Re: [PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread
hi jan, in this patch you mentioned "As the lostage_wakeup work is currently on the stack,
it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one ", why it‘s not triggered it in the switchtest benchmark which involves "xnthread_relax" due to linux syscall ? may be you probably describe the triggered scene detailed? Or how can I trigger it in switchtest
by using xnthread_relax leaves earlier on the new CPU than the work was processed on the old one? thanks
Quoted 156 lines · Jan Kiszka
Quoted 8 lines · JerryJun
hi Jan, the issue reported in [1] has seemly not been solved in ipipe. Will this issue still be maintained on ipipe? the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]). linux kernel: 4.19.89
xenomai: 3.2.4 + patch[1/3]
[1] https://lore.kernel.org/xenomai/[email protected]/T/#tSorry, that fix is still only on master (upcoming 3.3), not yet in stable trees. It's still not expected to be I-pipe/dovetail-dependent.
Let me recover a bit more from yesterday's Preempt RT merge party, then I will close that gap.
Jan
Quoted 134 lines
[ 98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[ 98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[ 98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[ 98.182496] I-pipe domain: Linux
[ 98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[ 98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[ 98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[ 98.182564] RAX:dead000000000100RBX:000000000005f3e0RCX:ffffc900012d2a88
[ 98.182577] RDX: 0000000000000000 RSI:ffffc900012d1840RDI: 0000000000001032
[ 98.182591] RBP: 0000000000000001 R08:ffffc900012d2a88R09:ffffc900012d2a78
[ 98.182605] R10: 0000000000000000 R11: 0000000000000000 R12:ffffc900012d3708
[ 98.182620] R13: 0000000000000003 R14:ffffc900012d1840R15: 0000000000000000
[ 98.182634] FS: 00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[ 98.182650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 98.182661] CR2:000055738facc000CR3:000000012ad44005CR4:00000000001606e0
[ 98.182698] Call Trace:
[ 98.182708]__cobalt_cond_wait_prologue+0x26d/0x850
[ 98.182720] ?handle_root_syscall+0xd3/0x310
[ 98.182730] ?__cobalt_cond_wait_prologue+0x850/0x850
[ 98.182741]CoBaLt_cond_wait_prologue+0x1d/0x20
[ 98.182751]handle_head_syscall+0x106/0x3b0
[ 98.182762]ipipe_handle_syscall+0x66/0xc0
[ 98.182772]do_syscall_64+0x22/0x170
[ 98.182782]entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 98.182793] RIP: 0033:0x7f888d29575b
[ 98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[ 98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246ORIG_RAX: 0000000010000023
[ 98.182852] RAX: ffffffffffffffda RBX:00007f888c56e6e0RCX:00007f888d29575b
[ 98.183195] RDX:00007f888c3baba4RSI:00007f888c56e6e0RDI:00007f888c56e7f8
[ 98.183530] RBP:00007f888c56e7f8R08: 0000000000000000 R09:00007f888c56e6d0
[ 98.183854] R10: 0000000000000000 R11: 0000000000000246 R12:00007f888c56e6e0
[ 98.184182] R13:00007f888c3bad80R14: 0000000000000001 R15:00007f888c3babb0
[ 98.184500] Modules linked in: fusenf_conntrack_netbios_nsnf_conntrack_broadcastxt_CTxt_tcpudpip6t_rpfilterip6t_REJECTnf_reject_ipv6ipt_REJECTnf_reject_ipv4xt_conntrackip6table_natnf_nat_ipv6ip6table_mangleip6table_rawip6table_securityiptable_natnf_nat_ipv4nf_natiptable_mangleiptable_rawiptable_securitynf_conntracknf_defrag_ipv6nf_defrag_ipv4libcrc32c arc4 ecb md4sha512_ssse3sha512_genericcmacnls_utf8cifs ccmdns_resolverfscacheip_setnf_tablesnfnetlinkip6table_filterip6_tablesiptable_filtervmwgfx sg ttmdrm_kms_helperdrmfb_sys_fopssyscopyarea sysfillrect sysimgbltserio_rawsnd_pcspsnd_pcmsnd_timersndcrct10dif_pclmulcrc32_pclmulghash_clmulni_intelpcbc evdevaesni_intelaes_x86_64crypto_simdcryptdglue_helpersoundcore ac buttonbinfmt_miscsunrpc [ 98.187446]ip_tablesx_tablesautofs4 ext4crc32c_genericcrc16 mbcache jbd2 fscryptosd_modata_genericata_piixcrc32c_intelmptspi mptscsih mptbasescsi_transport_spilibatascsi_mode1000 psmousei2c_piix4Quoted 96 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
We cannot rely on the
lostage_wakeupwork being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As thelostage_wakeupwork is currently on the stack, it can become corrupt whenxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using
irq_work_syncin the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.[1] https://lore.kernel.org/xenomai/[email protected]/T/#t
Reported-by JerryJun <[email protected]>Signed-off-by Jan Kiszka <[email protected]>---
include/cobalt/kernel/thread.h| 6 ++++++kernel/cobalt/thread.c| 15 +++++----------2 files changed, 11 insertions(+), 10 deletions(-)include/cobalt/kernel/thread.h +6
-0@@ -64,6 +64,11 @@ struct lostage_signal {int signo, sigval; }; +struct lostage_wakeup { +struct pipeline_inband_work inband_work;/* Must be first. */ +struct task_struct *task; +}; +struct xnthread_init_attr {struct xnthread_personality *personality;cpumask_t affinity;@@ -214,6 +219,7 @@ struct xnthread {u32 proghash;/* Hash value for exe_path */ #endif struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS ]; +struct lostage_wakeup relax_work; };static inline int xnthread_get_state (const struct xnthread *thread)kernel/cobalt/thread.c +5
-10@@ -1974,11 +1974,6 @@ int xnthread_harden(void) }EXPORT_SYMBOL_GPL (xnthread_harden); -struct lostage_wakeup { -struct pipeline_inband_work inband_work;/* Must be first. */ -struct task_struct *task; -}; -static void lostage_task_wakeup (struct pipeline_inband_work *inband_work) {struct lostage_wakeup *rq;@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)void xnthread_relax (int notify,int reason) {struct task_struct *p = current; -struct lostage_wakeup wakework = { - .inband_work =PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup), - .task = p, - };struct xnthread *thread =xnthread_current ();int cpu__maybe_unused , suspension;kernel_siginfo_t si;@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)*/ trace_cobalt_shadow_gorelax (reason); + thread->relax_work.inband_work = (struct pipeline_inband_work ) +PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup); + thread->relax_work.task = p; +/* * If you intend to change the following interrupt-free * sequence, /first/ make sure to check the special handling @@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)*/ splmax ();trace_cobalt_lostage_request ("wakeup" , p); -pipeline_post_inband_work (&wakework); +pipeline_post_inband_work (&thread->relax_work);/* * Grab the nklock to synchronize the Linux task state * manipulation with handle_sigwake_event. This lock will be Signature · 1 line
2.43.0Signature · 2 lines
Siemens AG, Technology Linux Expert Center
Re: Re: [PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread
Quoted 5 lines · JerryJun
hi jan, in this patch you mentioned "As the
lostage_wakeupwork is currently on the stack,
it can become corrupt whenxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one ", why it‘s not triggered it in the switchtest benchmark which involves "xnthread_relax" due to linux syscall ? may be you probably describe the triggered scene detailed? Or how can I trigger it in switchtest
by usingxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one? thanks
It's about task destruction, not migration.
The reproducer did a lot of task creation/destruction to trigger that. It happened that the task was already gone while there was still a node in the irq work referencing the "dead" task.
Classical use after free due to missing/broken synchronization.
We're still missing a proper test for that scenario. To my best knowledge Clara will revisit this topic.
Florian
Quoted 159 lines
Quoted 156 lines · Jan Kiszka
Quoted 8 lines · JerryJun
hi Jan, the issue reported in [1] has seemly not been solved in ipipe. Will this issue still be maintained on ipipe? the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]). linux kernel: 4.19.89
xenomai: 3.2.4 + patch[1/3]
[1] https://lore.kernel.org/xenomai/[email protected]/T/#tSorry, that fix is still only on master (upcoming 3.3), not yet in stable trees. It's still not expected to be I-pipe/dovetail-dependent.
Let me recover a bit more from yesterday's Preempt RT merge party, then I will close that gap.
Jan
Quoted 134 lines
[ 98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[ 98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[ 98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[ 98.182496] I-pipe domain: Linux
[ 98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[ 98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[ 98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[ 98.182564] RAX:dead000000000100RBX:000000000005f3e0RCX:ffffc900012d2a88
[ 98.182577] RDX: 0000000000000000 RSI:ffffc900012d1840RDI: 0000000000001032
[ 98.182591] RBP: 0000000000000001 R08:ffffc900012d2a88R09:ffffc900012d2a78
[ 98.182605] R10: 0000000000000000 R11: 0000000000000000 R12:ffffc900012d3708
[ 98.182620] R13: 0000000000000003 R14:ffffc900012d1840R15: 0000000000000000
[ 98.182634] FS: 00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[ 98.182650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 98.182661] CR2:000055738facc000CR3:000000012ad44005CR4:00000000001606e0
[ 98.182698] Call Trace:
[ 98.182708]__cobalt_cond_wait_prologue+0x26d/0x850
[ 98.182720] ?handle_root_syscall+0xd3/0x310
[ 98.182730] ?__cobalt_cond_wait_prologue+0x850/0x850
[ 98.182741]CoBaLt_cond_wait_prologue+0x1d/0x20
[ 98.182751]handle_head_syscall+0x106/0x3b0
[ 98.182762]ipipe_handle_syscall+0x66/0xc0
[ 98.182772]do_syscall_64+0x22/0x170
[ 98.182782]entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 98.182793] RIP: 0033:0x7f888d29575b
[ 98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[ 98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246ORIG_RAX: 0000000010000023
[ 98.182852] RAX: ffffffffffffffda RBX:00007f888c56e6e0RCX:00007f888d29575b
[ 98.183195] RDX:00007f888c3baba4RSI:00007f888c56e6e0RDI:00007f888c56e7f8
[ 98.183530] RBP:00007f888c56e7f8R08: 0000000000000000 R09:00007f888c56e6d0
[ 98.183854] R10: 0000000000000000 R11: 0000000000000246 R12:00007f888c56e6e0
[ 98.184182] R13:00007f888c3bad80R14: 0000000000000001 R15:00007f888c3babb0
[ 98.184500] Modules linked in: fusenf_conntrack_netbios_nsnf_conntrack_broadcastxt_CTxt_tcpudpip6t_rpfilterip6t_REJECTnf_reject_ipv6ipt_REJECTnf_reject_ipv4xt_conntrackip6table_natnf_nat_ipv6ip6table_mangleip6table_rawip6table_securityiptable_natnf_nat_ipv4nf_natiptable_mangleiptable_rawiptable_securitynf_conntracknf_defrag_ipv6nf_defrag_ipv4libcrc32c arc4 ecb md4sha512_ssse3sha512_genericcmacnls_utf8cifs ccmdns_resolverfscacheip_setnf_tablesnfnetlinkip6table_filterip6_tablesiptable_filtervmwgfx sg ttmdrm_kms_helperdrmfb_sys_fopssyscopyarea sysfillrect sysimgbltserio_rawsnd_pcspsnd_pcmsnd_timersndcrct10dif_pclmulcrc32_pclmulghash_clmulni_intelpcbc evdevaesni_intelaes_x86_64crypto_simdcryptdglue_helpersoundcore ac buttonbinfmt_miscsunrpc [ 98.187446]ip_tablesx_tablesautofs4 ext4crc32c_genericcrc16 mbcache jbd2 fscryptosd_modata_genericata_piixcrc32c_intelmptspi mptscsih mptbasescsi_transport_spilibatascsi_mode1000 psmousei2c_piix4Quoted 96 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
We cannot rely on the
lostage_wakeupwork being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As thelostage_wakeupwork is currently on the stack, it can become corrupt whenxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using
irq_work_syncin the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.[1] https://lore.kernel.org/xenomai/[email protected]/T/#t
Reported-by JerryJun <[email protected]>Signed-off-by Jan Kiszka <[email protected]>---
include/cobalt/kernel/thread.h| 6 ++++++kernel/cobalt/thread.c| 15 +++++----------2 files changed, 11 insertions(+), 10 deletions(-)include/cobalt/kernel/thread.h +6
-0@@ -64,6 +64,11 @@ struct lostage_signal {int signo, sigval; }; +struct lostage_wakeup { +struct pipeline_inband_work inband_work;/* Must be first. */ +struct task_struct *task; +}; +struct xnthread_init_attr {struct xnthread_personality *personality;cpumask_t affinity;@@ -214,6 +219,7 @@ struct xnthread {u32 proghash;/* Hash value for exe_path */ #endif struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS ]; +struct lostage_wakeup relax_work; };static inline int xnthread_get_state (const struct xnthread *thread)kernel/cobalt/thread.c +5
-10@@ -1974,11 +1974,6 @@ int xnthread_harden(void) }EXPORT_SYMBOL_GPL (xnthread_harden); -struct lostage_wakeup { -struct pipeline_inband_work inband_work;/* Must be first. */ -struct task_struct *task; -}; -static void lostage_task_wakeup (struct pipeline_inband_work *inband_work) {struct lostage_wakeup *rq;@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)void xnthread_relax (int notify,int reason) {struct task_struct *p = current; -struct lostage_wakeup wakework = { - .inband_work =PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup), - .task = p, - };struct xnthread *thread =xnthread_current ();int cpu__maybe_unused , suspension;kernel_siginfo_t si;@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)*/ trace_cobalt_shadow_gorelax (reason); + thread->relax_work.inband_work = (struct pipeline_inband_work ) +PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup); + thread->relax_work.task = p; +/* * If you intend to change the following interrupt-free * sequence, /first/ make sure to check the special handling @@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)*/ splmax ();trace_cobalt_lostage_request ("wakeup" , p); -pipeline_post_inband_work (&wakework); +pipeline_post_inband_work (&thread->relax_work);/* * Grab the nklock to synchronize the Linux task state * manipulation with handle_sigwake_event. This lock will be Signature · 1 line
2.43.0Signature · 2 lines
Siemens AG, Technology Linux Expert Center
Re:Re: Re: [PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread
It seems to have something to do with migration.
it‘s not triggered it by "taskset -c 1 ./xeno-app" from "Attachment #2: xenomai-alchemy-app.tar.gz" which make the main thread and child thread on the same cpu. besides, there is an interesting problem that it‘s also not triggered it when replace alchemy with pthread(libcobalt) API from attachment.
Quoted 13 lines · Florian
Quoted 5 lines · JerryJun
hi jan, in this patch you mentioned "As the
lostage_wakeupwork is currently on the stack,
it can become corrupt whenxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one ", why it‘s not triggered it in the switchtest benchmark which involves "xnthread_relax" due to linux syscall ? may be you probably describe the triggered scene detailed? Or how can I trigger it in switchtest
by usingxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one? thanksIt's about task destruction, not migration.
The reproducer did a lot of task creation/destruction to trigger that. It happened that the task was already gone while there was still a node in the irq work referencing the "dead" task.
Quoted 167 lines
Classical use after free due to missing/broken synchronization.
We're still missing a proper test for that scenario. To my best knowledge Clara will revisit this topic.
Florian
Quoted 159 lines
Quoted 156 lines · Jan Kiszka
Quoted 8 lines · JerryJun
hi Jan, the issue reported in [1] has seemly not been solved in ipipe. Will this issue still be maintained on ipipe? the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]). linux kernel: 4.19.89
xenomai: 3.2.4 + patch[1/3]
[1] https://lore.kernel.org/xenomai/[email protected]/T/#tSorry, that fix is still only on master (upcoming 3.3), not yet in stable trees. It's still not expected to be I-pipe/dovetail-dependent.
Let me recover a bit more from yesterday's Preempt RT merge party, then I will close that gap.
Jan
Quoted 134 lines
[ 98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[ 98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[ 98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[ 98.182496] I-pipe domain: Linux
[ 98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[ 98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[ 98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[ 98.182564] RAX:dead000000000100RBX:000000000005f3e0RCX:ffffc900012d2a88
[ 98.182577] RDX: 0000000000000000 RSI:ffffc900012d1840RDI: 0000000000001032
[ 98.182591] RBP: 0000000000000001 R08:ffffc900012d2a88R09:ffffc900012d2a78
[ 98.182605] R10: 0000000000000000 R11: 0000000000000000 R12:ffffc900012d3708
[ 98.182620] R13: 0000000000000003 R14:ffffc900012d1840R15: 0000000000000000
[ 98.182634] FS: 00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[ 98.182650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 98.182661] CR2:000055738facc000CR3:000000012ad44005CR4:00000000001606e0
[ 98.182698] Call Trace:
[ 98.182708]__cobalt_cond_wait_prologue+0x26d/0x850
[ 98.182720] ?handle_root_syscall+0xd3/0x310
[ 98.182730] ?__cobalt_cond_wait_prologue+0x850/0x850
[ 98.182741]CoBaLt_cond_wait_prologue+0x1d/0x20
[ 98.182751]handle_head_syscall+0x106/0x3b0
[ 98.182762]ipipe_handle_syscall+0x66/0xc0
[ 98.182772]do_syscall_64+0x22/0x170
[ 98.182782]entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 98.182793] RIP: 0033:0x7f888d29575b
[ 98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[ 98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246ORIG_RAX: 0000000010000023
[ 98.182852] RAX: ffffffffffffffda RBX:00007f888c56e6e0RCX:00007f888d29575b
[ 98.183195] RDX:00007f888c3baba4RSI:00007f888c56e6e0RDI:00007f888c56e7f8
[ 98.183530] RBP:00007f888c56e7f8R08: 0000000000000000 R09:00007f888c56e6d0
[ 98.183854] R10: 0000000000000000 R11: 0000000000000246 R12:00007f888c56e6e0
[ 98.184182] R13:00007f888c3bad80R14: 0000000000000001 R15:00007f888c3babb0
[ 98.184500] Modules linked in: fusenf_conntrack_netbios_nsnf_conntrack_broadcastxt_CTxt_tcpudpip6t_rpfilterip6t_REJECTnf_reject_ipv6ipt_REJECTnf_reject_ipv4xt_conntrackip6table_natnf_nat_ipv6ip6table_mangleip6table_rawip6table_securityiptable_natnf_nat_ipv4nf_natiptable_mangleiptable_rawiptable_securitynf_conntracknf_defrag_ipv6nf_defrag_ipv4libcrc32c arc4 ecb md4sha512_ssse3sha512_genericcmacnls_utf8cifs ccmdns_resolverfscacheip_setnf_tablesnfnetlinkip6table_filterip6_tablesiptable_filtervmwgfx sg ttmdrm_kms_helperdrmfb_sys_fopssyscopyarea sysfillrect sysimgbltserio_rawsnd_pcspsnd_pcmsnd_timersndcrct10dif_pclmulcrc32_pclmulghash_clmulni_intelpcbc evdevaesni_intelaes_x86_64crypto_simdcryptdglue_helpersoundcore ac buttonbinfmt_miscsunrpc [ 98.187446]ip_tablesx_tablesautofs4 ext4crc32c_genericcrc16 mbcache jbd2 fscryptosd_modata_genericata_piixcrc32c_intelmptspi mptscsih mptbasescsi_transport_spilibatascsi_mode1000 psmousei2c_piix4Quoted 96 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
We cannot rely on the
lostage_wakeupwork being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As thelostage_wakeupwork is currently on the stack, it can become corrupt whenxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using
irq_work_syncin the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.[1] https://lore.kernel.org/xenomai/[email protected]/T/#t
Reported-by JerryJun <[email protected]>Signed-off-by Jan Kiszka <[email protected]>---
include/cobalt/kernel/thread.h| 6 ++++++kernel/cobalt/thread.c| 15 +++++----------2 files changed, 11 insertions(+), 10 deletions(-)include/cobalt/kernel/thread.h +6
-0@@ -64,6 +64,11 @@ struct lostage_signal {int signo, sigval; }; +struct lostage_wakeup { +struct pipeline_inband_work inband_work;/* Must be first. */ +struct task_struct *task; +}; +struct xnthread_init_attr {struct xnthread_personality *personality;cpumask_t affinity;@@ -214,6 +219,7 @@ struct xnthread {u32 proghash;/* Hash value for exe_path */ #endif struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS ]; +struct lostage_wakeup relax_work; };static inline int xnthread_get_state (const struct xnthread *thread)kernel/cobalt/thread.c +5
-10@@ -1974,11 +1974,6 @@ int xnthread_harden(void) }EXPORT_SYMBOL_GPL (xnthread_harden); -struct lostage_wakeup { -struct pipeline_inband_work inband_work;/* Must be first. */ -struct task_struct *task; -}; -static void lostage_task_wakeup (struct pipeline_inband_work *inband_work) {struct lostage_wakeup *rq;@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)void xnthread_relax (int notify,int reason) {struct task_struct *p = current; -struct lostage_wakeup wakework = { - .inband_work =PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup), - .task = p, - };struct xnthread *thread =xnthread_current ();int cpu__maybe_unused , suspension;kernel_siginfo_t si;@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)*/ trace_cobalt_shadow_gorelax (reason); + thread->relax_work.inband_work = (struct pipeline_inband_work ) +PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup); + thread->relax_work.task = p; +/* * If you intend to change the following interrupt-free * sequence, /first/ make sure to check the special handling @@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)*/ splmax ();trace_cobalt_lostage_request ("wakeup" , p); -pipeline_post_inband_work (&wakework); +pipeline_post_inband_work (&thread->relax_work);/* * Grab the nklock to synchronize the Linux task state * manipulation with handle_sigwake_event. This lock will be Signature · 1 line
2.43.0Signature · 2 lines
Siemens AG, Technology Linux Expert Center
#include <sched.h>
#include <alchemy/task.h>
#include <signal.h>
#include <pthread.h>
#include <stdio.h>
#define THREAD_CNT 30
static int global_exit_trigger = 0;
static void signal_handler()
{
global_exit_trigger = 1;
}static void *rt_main(void *arg)
{
while (!global_exit_trigger) {
usleep(1 * 1000 * 1000); // 10 sec
}
return NULL;
}static void threads_start(pthread_t *rt_tasks, int *thread_ids) {
char task_name[256];
cpu_set_t mask;
int ret;
struct sched_param param = {.sched_priority = 50};
pthread_attr_t attr; CPU_ZERO(&mask);
CPU_SET(1, &mask); pthread_attr_init(&attr); pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE); pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED); pthread_attr_setschedpolicy(&attr, SCHED_FIFO); pthread_attr_setschedparam(&attr, ¶m);
pthread_attr_setaffinity_np(&attr, sizeof(mask), &mask);
for (int i = 0; i < THREAD_CNT; i++) {
ret = pthread_create(&rt_tasks[i], &attr, &rt_main, NULL);
if (ret) {
printf("Failed to create task %d\n", i);
break;
}
}
}static void threads_stop(pthread_t *rt_tasks)
{
int ret;
for (int i = 0; i < THREAD_CNT; i++) {
pthread_cancel(rt_tasks[i]);
pthread_join(rt_tasks[i], NULL);
}
}int main(int argc, char **argv)
{
pthread_t rt_tasks[THREAD_CNT];
int thread_ids[THREAD_CNT];
signal(SIGINT, signal_handler);
printf("Starting\n");
while (!global_exit_trigger) {
threads_start(rt_tasks, thread_ids);
usleep(1*1000); // 1ms
threads_stop(rt_tasks);
}
}Re: Re: Re: [PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread
Quoted 2 lines · JerryJun
It seems to have something to do with migration.
it‘s not triggered it by "taskset -c 1 ./xeno-app" from "Attachment #2: xenomai-alchemy-app.tar.gz" which make the main thread and child thread on the same cpu. besides, there is an interesting problem that it‘s also not triggered it when replace alchemy with pthread(libcobalt) API from attachment.
The affinity just closed the race window. It was clearly a bug in the task shutdown sequence.
Quoted 182 lines
Quoted 180 lines · Florian
Quoted 5 lines · JerryJun
hi jan, in this patch you mentioned "As the
lostage_wakeupwork is currently on the stack,
it can become corrupt whenxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one ", why it‘s not triggered it in the switchtest benchmark which involves "xnthread_relax" due to linux syscall ? may be you probably describe the triggered scene detailed? Or how can I trigger it in switchtest
by usingxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one? thanksIt's about task destruction, not migration.
The reproducer did a lot of task creation/destruction to trigger that. It happened that the task was already gone while there was still a node in the irq work referencing the "dead" task.
Classical use after free due to missing/broken synchronization.
We're still missing a proper test for that scenario. To my best knowledge Clara will revisit this topic.
Florian
Quoted 159 lines
Quoted 156 lines · Jan Kiszka
Quoted 8 lines · JerryJun
hi Jan, the issue reported in [1] has seemly not been solved in ipipe. Will this issue still be maintained on ipipe? the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]). linux kernel: 4.19.89
xenomai: 3.2.4 + patch[1/3]
[1] https://lore.kernel.org/xenomai/[email protected]/T/#tSorry, that fix is still only on master (upcoming 3.3), not yet in stable trees. It's still not expected to be I-pipe/dovetail-dependent.
Let me recover a bit more from yesterday's Preempt RT merge party, then I will close that gap.
Jan
Quoted 134 lines
[ 98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[ 98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[ 98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[ 98.182496] I-pipe domain: Linux
[ 98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[ 98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[ 98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[ 98.182564] RAX:dead000000000100RBX:000000000005f3e0RCX:ffffc900012d2a88
[ 98.182577] RDX: 0000000000000000 RSI:ffffc900012d1840RDI: 0000000000001032
[ 98.182591] RBP: 0000000000000001 R08:ffffc900012d2a88R09:ffffc900012d2a78
[ 98.182605] R10: 0000000000000000 R11: 0000000000000000 R12:ffffc900012d3708
[ 98.182620] R13: 0000000000000003 R14:ffffc900012d1840R15: 0000000000000000
[ 98.182634] FS: 00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[ 98.182650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 98.182661] CR2:000055738facc000CR3:000000012ad44005CR4:00000000001606e0
[ 98.182698] Call Trace:
[ 98.182708]__cobalt_cond_wait_prologue+0x26d/0x850
[ 98.182720] ?handle_root_syscall+0xd3/0x310
[ 98.182730] ?__cobalt_cond_wait_prologue+0x850/0x850
[ 98.182741]CoBaLt_cond_wait_prologue+0x1d/0x20
[ 98.182751]handle_head_syscall+0x106/0x3b0
[ 98.182762]ipipe_handle_syscall+0x66/0xc0
[ 98.182772]do_syscall_64+0x22/0x170
[ 98.182782]entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 98.182793] RIP: 0033:0x7f888d29575b
[ 98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[ 98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246ORIG_RAX: 0000000010000023
[ 98.182852] RAX: ffffffffffffffda RBX:00007f888c56e6e0RCX:00007f888d29575b
[ 98.183195] RDX:00007f888c3baba4RSI:00007f888c56e6e0RDI:00007f888c56e7f8
[ 98.183530] RBP:00007f888c56e7f8R08: 0000000000000000 R09:00007f888c56e6d0
[ 98.183854] R10: 0000000000000000 R11: 0000000000000246 R12:00007f888c56e6e0
[ 98.184182] R13:00007f888c3bad80R14: 0000000000000001 R15:00007f888c3babb0
[ 98.184500] Modules linked in: fusenf_conntrack_netbios_nsnf_conntrack_broadcastxt_CTxt_tcpudpip6t_rpfilterip6t_REJECTnf_reject_ipv6ipt_REJECTnf_reject_ipv4xt_conntrackip6table_natnf_nat_ipv6ip6table_mangleip6table_rawip6table_securityiptable_natnf_nat_ipv4nf_natiptable_mangleiptable_rawiptable_securitynf_conntracknf_defrag_ipv6nf_defrag_ipv4libcrc32c arc4 ecb md4sha512_ssse3sha512_genericcmacnls_utf8cifs ccmdns_resolverfscacheip_setnf_tablesnfnetlinkip6table_filterip6_tablesiptable_filtervmwgfx sg ttmdrm_kms_helperdrmfb_sys_fopssyscopyarea sysfillrect sysimgbltserio_rawsnd_pcspsnd_pcmsnd_timersndcrct10dif_pclmulcrc32_pclmulghash_clmulni_intelpcbc evdevaesni_intelaes_x86_64crypto_simdcryptdglue_helpersoundcore ac buttonbinfmt_miscsunrpc [ 98.187446]ip_tablesx_tablesautofs4 ext4crc32c_genericcrc16 mbcache jbd2 fscryptosd_modata_genericata_piixcrc32c_intelmptspi mptscsih mptbasescsi_transport_spilibatascsi_mode1000 psmousei2c_piix4Quoted 96 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
We cannot rely on the
lostage_wakeupwork being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As thelostage_wakeupwork is currently on the stack, it can become corrupt whenxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using
irq_work_syncin the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.[1] https://lore.kernel.org/xenomai/[email protected]/T/#t
Reported-by JerryJun <[email protected]>Signed-off-by Jan Kiszka <[email protected]>---
include/cobalt/kernel/thread.h| 6 ++++++kernel/cobalt/thread.c| 15 +++++----------2 files changed, 11 insertions(+), 10 deletions(-)include/cobalt/kernel/thread.h +6
-0@@ -64,6 +64,11 @@ struct lostage_signal {int signo, sigval; }; +struct lostage_wakeup { +struct pipeline_inband_work inband_work;/* Must be first. */ +struct task_struct *task; +}; +struct xnthread_init_attr {struct xnthread_personality *personality;cpumask_t affinity;@@ -214,6 +219,7 @@ struct xnthread {u32 proghash;/* Hash value for exe_path */ #endif struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS ]; +struct lostage_wakeup relax_work; };static inline int xnthread_get_state (const struct xnthread *thread)kernel/cobalt/thread.c +5
-10@@ -1974,11 +1974,6 @@ int xnthread_harden(void) }EXPORT_SYMBOL_GPL (xnthread_harden); -struct lostage_wakeup { -struct pipeline_inband_work inband_work;/* Must be first. */ -struct task_struct *task; -}; -static void lostage_task_wakeup (struct pipeline_inband_work *inband_work) {struct lostage_wakeup *rq;@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)void xnthread_relax (int notify,int reason) {struct task_struct *p = current; -struct lostage_wakeup wakework = { - .inband_work =PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup), - .task = p, - };struct xnthread *thread =xnthread_current ();int cpu__maybe_unused , suspension;kernel_siginfo_t si;@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)*/ trace_cobalt_shadow_gorelax (reason); + thread->relax_work.inband_work = (struct pipeline_inband_work ) +PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup); + thread->relax_work.task = p; +/* * If you intend to change the following interrupt-free * sequence, /first/ make sure to check the special handling @@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)*/ splmax ();trace_cobalt_lostage_request ("wakeup" , p); -pipeline_post_inband_work (&wakework); +pipeline_post_inband_work (&thread->relax_work);/* * Grab the nklock to synchronize the Linux task state * manipulation with handle_sigwake_event. This lock will be Signature · 1 line
2.43.0Signature · 2 lines
Siemens AG, Technology Linux Expert Center
Re: [PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread
In reply to JerryJunQuoted 7 lines · JerryJun
hi jan, in this patch you mentioned "As the
lostage_wakeupwork is currently on the stack,
it can become corrupt whenxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one ", why it‘s not triggered it in the switchtest benchmark which involves "xnthread_relax" due to linux syscall ? may be you probably describe the triggered scene detailed? Or how can I trigger it in switchtest
by usingxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one? thanks
The switchtest primarily tests switching between already existing tasks. I don't think we stress there the continuous creation and deletion of task there yet.
Clara wanted to look into improving the test coverage of that particular scenario as well. You two should probably coordinate who will work on what aspect.
Jan
Quoted 159 lines
Quoted 156 lines · Jan Kiszka
Quoted 8 lines · JerryJun
hi Jan, the issue reported in [1] has seemly not been solved in ipipe. Will this issue still be maintained on ipipe? the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]). linux kernel: 4.19.89
xenomai: 3.2.4 + patch[1/3]
[1] https://lore.kernel.org/xenomai/[email protected]/T/#tSorry, that fix is still only on master (upcoming 3.3), not yet in stable trees. It's still not expected to be I-pipe/dovetail-dependent.
Let me recover a bit more from yesterday's Preempt RT merge party, then I will close that gap.
Jan
Quoted 134 lines
[ 98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[ 98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[ 98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[ 98.182496] I-pipe domain: Linux
[ 98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[ 98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[ 98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[ 98.182564] RAX:dead000000000100RBX:000000000005f3e0RCX:ffffc900012d2a88
[ 98.182577] RDX: 0000000000000000 RSI:ffffc900012d1840RDI: 0000000000001032
[ 98.182591] RBP: 0000000000000001 R08:ffffc900012d2a88R09:ffffc900012d2a78
[ 98.182605] R10: 0000000000000000 R11: 0000000000000000 R12:ffffc900012d3708
[ 98.182620] R13: 0000000000000003 R14:ffffc900012d1840R15: 0000000000000000
[ 98.182634] FS: 00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[ 98.182650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 98.182661] CR2:000055738facc000CR3:000000012ad44005CR4:00000000001606e0
[ 98.182698] Call Trace:
[ 98.182708]__cobalt_cond_wait_prologue+0x26d/0x850
[ 98.182720] ?handle_root_syscall+0xd3/0x310
[ 98.182730] ?__cobalt_cond_wait_prologue+0x850/0x850
[ 98.182741]CoBaLt_cond_wait_prologue+0x1d/0x20
[ 98.182751]handle_head_syscall+0x106/0x3b0
[ 98.182762]ipipe_handle_syscall+0x66/0xc0
[ 98.182772]do_syscall_64+0x22/0x170
[ 98.182782]entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 98.182793] RIP: 0033:0x7f888d29575b
[ 98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[ 98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246ORIG_RAX: 0000000010000023
[ 98.182852] RAX: ffffffffffffffda RBX:00007f888c56e6e0RCX:00007f888d29575b
[ 98.183195] RDX:00007f888c3baba4RSI:00007f888c56e6e0RDI:00007f888c56e7f8
[ 98.183530] RBP:00007f888c56e7f8R08: 0000000000000000 R09:00007f888c56e6d0
[ 98.183854] R10: 0000000000000000 R11: 0000000000000246 R12:00007f888c56e6e0
[ 98.184182] R13:00007f888c3bad80R14: 0000000000000001 R15:00007f888c3babb0
[ 98.184500] Modules linked in: fusenf_conntrack_netbios_nsnf_conntrack_broadcastxt_CTxt_tcpudpip6t_rpfilterip6t_REJECTnf_reject_ipv6ipt_REJECTnf_reject_ipv4xt_conntrackip6table_natnf_nat_ipv6ip6table_mangleip6table_rawip6table_securityiptable_natnf_nat_ipv4nf_natiptable_mangleiptable_rawiptable_securitynf_conntracknf_defrag_ipv6nf_defrag_ipv4libcrc32c arc4 ecb md4sha512_ssse3sha512_genericcmacnls_utf8cifs ccmdns_resolverfscacheip_setnf_tablesnfnetlinkip6table_filterip6_tablesiptable_filtervmwgfx sg ttmdrm_kms_helperdrmfb_sys_fopssyscopyarea sysfillrect sysimgbltserio_rawsnd_pcspsnd_pcmsnd_timersndcrct10dif_pclmulcrc32_pclmulghash_clmulni_intelpcbc evdevaesni_intelaes_x86_64crypto_simdcryptdglue_helpersoundcore ac buttonbinfmt_miscsunrpc [ 98.187446]ip_tablesx_tablesautofs4 ext4crc32c_genericcrc16 mbcache jbd2 fscryptosd_modata_genericata_piixcrc32c_intelmptspi mptscsih mptbasescsi_transport_spilibatascsi_mode1000 psmousei2c_piix4Quoted 96 lines · Jan Kiszka
From: Jan Kiszka <[email protected]>
We cannot rely on the
lostage_wakeupwork being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As thelostage_wakeupwork is currently on the stack, it can become corrupt whenxnthread_relaxleaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using
irq_work_syncin the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.[1] https://lore.kernel.org/xenomai/[email protected]/T/#t
Reported-by JerryJun <[email protected]>Signed-off-by Jan Kiszka <[email protected]>---
include/cobalt/kernel/thread.h| 6 ++++++kernel/cobalt/thread.c| 15 +++++----------2 files changed, 11 insertions(+), 10 deletions(-)include/cobalt/kernel/thread.h +6
-0@@ -64,6 +64,11 @@ struct lostage_signal {int signo, sigval; }; +struct lostage_wakeup { +struct pipeline_inband_work inband_work;/* Must be first. */ +struct task_struct *task; +}; +struct xnthread_init_attr {struct xnthread_personality *personality;cpumask_t affinity;@@ -214,6 +219,7 @@ struct xnthread {u32 proghash;/* Hash value for exe_path */ #endif struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS ]; +struct lostage_wakeup relax_work; };static inline int xnthread_get_state (const struct xnthread *thread)kernel/cobalt/thread.c +5
-10@@ -1974,11 +1974,6 @@ int xnthread_harden(void) }EXPORT_SYMBOL_GPL (xnthread_harden); -struct lostage_wakeup { -struct pipeline_inband_work inband_work;/* Must be first. */ -struct task_struct *task; -}; -static void lostage_task_wakeup (struct pipeline_inband_work *inband_work) {struct lostage_wakeup *rq;@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)void xnthread_relax (int notify,int reason) {struct task_struct *p = current; -struct lostage_wakeup wakework = { - .inband_work =PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup), - .task = p, - };struct xnthread *thread =xnthread_current ();int cpu__maybe_unused , suspension;kernel_siginfo_t si;@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)*/ trace_cobalt_shadow_gorelax (reason); + thread->relax_work.inband_work = (struct pipeline_inband_work ) +PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup); + thread->relax_work.task = p; +/* * If you intend to change the following interrupt-free * sequence, /first/ make sure to check the special handling @@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)*/ splmax ();trace_cobalt_lostage_request ("wakeup" , p); -pipeline_post_inband_work (&wakework); +pipeline_post_inband_work (&thread->relax_work);/* * Grab the nklock to synchronize the Linux task state * manipulation with handle_sigwake_event. This lock will be Signature · 1 line
2.43.0Signature · 2 lines
Siemens AG, Technology Linux Expert Center
Signature · 2 lines
Siemens AG, Technology Linux Expert Center
PATCH 2/3 cobalt: Drop obsolete pipeline_inband_work abstraction
In reply to Jan KiszkaFrom: Jan Kiszka <[email protected]>
We are now dovetail-only and can use irq_work directly.
--- .../kernel/dovetail/pipeline/inband_work.h | 28 ------------------- include/cobalt/kernel/rtdm/driver.h | 9 +++--- include/cobalt/kernel/thread.h | 6 ++-- kernel/cobalt/rtdm/drvlib.c | 15 ++++------ kernel/cobalt/rtdm/fd.c | 13 ++++----- kernel/cobalt/thread.c | 24 +++++++--------- kernel/drivers/udd/udd.c | 12 ++++---- 7 files changed, 33 insertions(+), 74 deletions(-) delete mode 100644 include/cobalt/kernel/dovetail/pipeline/inband_work.h
include/cobalt/kernel/dovetail/pipeline/inband_work.h deleted +0 -28
@@ -1,28 +0,0 @@ -/* -* SPDX-License-Identifier: GPL-2.0 -* -* Copyright (C) 2020 Philippe Gerum <[email protected]> -*/ - -#ifndef _COBALT_KERNEL_DOVETAIL_INBAND_WORK_H -#define _COBALT_KERNEL_DOVETAIL_INBAND_WORK_H - -#include <linux/irq_work.h> - -/* -* This field must be named inband_work and appear first in the -* container work struct. -*/ -struct pipeline_inband_work { -struct irq_work work; -}; - -#define PIPELINE_INBAND_WORK_INITIALIZER (__handler) \ - { \ - .work = IRQ_WORK_INIT((void (*)(struct irq_work *))__handler), \ - } - -#define pipeline_post_inband_work (__work) \ - irq_work_queue(&(__work)->inband_work.work) - -#endif /* !_COBALT_KERNEL_DOVETAIL_INBAND_WORK_H */
include/cobalt/kernel/rtdm/driver.h +4 -5
@@ -28,13 +28,13 @@#include <asm/atomic.h> #include <linux/cpumask.h> +#include <linux/irq_work.h> #include <linux/list.h> #include <linux/module.h> #include <linux/cdev.h> #include <linux/wait.h> #include <linux/notifier.h> #include <pipeline/lock.h> -#include <pipeline/inband_work.h> #include <xenomai/version.h> #include <cobalt/kernel/heap.h> #include <cobalt/kernel/sched.h>
@@ -909,7 +909,7 @@ typedef struct rtdm_nrtsig rtdm_nrtsig_t;typedef void (*rtdm_nrtsig_handler_t )(rtdm_nrtsig_t *nrt_sig,void *arg);struct rtdm_nrtsig { -struct pipeline_inband_work inband_work;/* Must be first */ +struct irq_work inband_work;rtdm_nrtsig_handler_t handler;void *arg; };
@@ -918,13 +918,12 @@ void rtdm_schedule_nrt_work(struct work_struct *lostage_work);/** @} rtdm_nrtsignal */ #ifndef DOXYGEN_CPP /* Avoid static inline tags for RTDM in doxygen */ -void __rtdm_nrtsig_execute (struct pipeline_inband_work *inband_work); +void __rtdm_nrtsig_execute (struct irq_work *inband_work);static inline void rtdm_nrtsig_init (rtdm_nrtsig_t *nrt_sig,rtdm_nrtsig_handler_t handler,void *arg) { - nrt_sig->inband_work = (struct pipeline_inband_work ) -PIPELINE_INBAND_WORK_INITIALIZER (__rtdm_nrtsig_execute); + nrt_sig->inband_work =IRQ_WORK_INIT (__rtdm_nrtsig_execute); nrt_sig->handler = handler; nrt_sig->arg = arg; }
include/cobalt/kernel/thread.h +3 -3
@@ -19,11 +19,11 @@#ifndef _COBALT_KERNEL_THREAD_H#define _COBALT_KERNEL_THREAD_H +#include <linux/irq_work.h> #include <linux/wait.h> #include <linux/sched.h> #include <linux/sched/rt.h> #include <pipeline/thread.h> -#include <pipeline/inband_work.h> #include <cobalt/kernel/list.h> #include <cobalt/kernel/stat.h> #include <cobalt/kernel/timer.h>
@@ -59,13 +59,13 @@ struct xnthread_personality;struct completion ;struct lostage_signal { -struct pipeline_inband_work inband_work;/* Must be first. */ +struct irq_work inband_work;struct task_struct *task;int signo, sigval; };struct lostage_wakeup { -struct pipeline_inband_work inband_work;/* Must be first. */ +struct irq_work inband_work;struct task_struct *task; };
kernel/cobalt/rtdm/drvlib.c +5 -10
@@ -31,7 +31,6 @@#include <linux/anon_inodes.h> #include <rtdm/driver.h> #include <rtdm/compat.h> -#include <pipeline/inband_work.h> #include "internal.h" #include <trace/events/cobalt-rtdm.h>
@@ -1616,7 +1615,7 @@ int rtdm_nrtsig_init(rtdm_nrtsig_t *nrt_sig, rtdm_nrtsig_handler_t handler,void rtdm_nrtsig_destroy (rtdm_nrtsig_t *nrt_sig);#endif /* DOXYGEN_CPP */ -void __rtdm_nrtsig_execute (struct pipeline_inband_work *inband_work) +void __rtdm_nrtsig_execute (struct irq_work *inband_work) {struct rtdm_nrtsig *nrt_sig;
@@ -1634,14 +1633,14 @@ EXPORT_SYMBOL_GPL(__rtdm_nrtsig_execute);*/ void rtdm_nrtsig_pend (rtdm_nrtsig_t *nrt_sig) { -pipeline_post_inband_work (nrt_sig); +irq_work_queue (&nrt_sig->inband_work); }EXPORT_SYMBOL_GPL (rtdm_nrtsig_pend);static LIST_HEAD (nrt_work_list);DEFINE_PRIVATE_XNLOCK (nrt_work_lock); -static void lostage_schedule_work (struct pipeline_inband_work *inband_work) +static void lostage_schedule_work (struct irq_work *inband_work) {struct work_struct *lostage_work;spl_t s;
@@ -1663,11 +1662,7 @@ static void lostage_schedule_work(struct pipeline_inband_work *inband_work)xnlock_put_irqrestore (&nrt_work_lock, s); } -static struct lostage_trigger_work { -struct pipeline_inband_work inband_work;/* Must be first. */ -} nrt_work = { - .inband_work =PIPELINE_INBAND_WORK_INITIALIZER (lostage_schedule_work), -}; +static DEFINE_IRQ_WORK(nrt_work, lostage_schedule_work);/** * Put a work task in Linux non real-time global workqueue from primary mode.
@@ -1686,7 +1681,7 @@ void rtdm_schedule_nrt_work(struct work_struct *lostage_work)xnlock_get_irqsave (&nrt_work_lock, s);list_add_tail (&lostage_work->entry, &nrt_work_list); -pipeline_post_inband_work (&nrt_work); +irq_work_queue (&nrt_work);xnlock_put_irqrestore (&nrt_work_lock, s); }
kernel/cobalt/rtdm/fd.c +5 -8
@@ -17,6 +17,8 @@* along with this program; if not, write to the Free Software * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. */ + +#include <linux/irq_work.h> #include <linux/list.h> #include <linux/err.h> #include <linux/slab.h>
@@ -29,7 +31,6 @@#include <cobalt/kernel/lock.h> #include <cobalt/kernel/ppd.h> #include <cobalt/kernel/time.h> -#include <pipeline/inband_work.h> #include <trace/events/cobalt-rtdm.h> #include <rtdm/compat.h> #include <rtdm/fd.h>
@@ -297,16 +298,12 @@ static int fd_cleanup_thread(void *data)return 0 ; } -static void lostage_trigger_close (struct pipeline_inband_work *inband_work) +static void lostage_trigger_close (struct irq_work *inband_work) {up (&rtdm_fd_cleanup_sem); } -static struct lostage_trigger_close { -struct pipeline_inband_work inband_work;/* Must be first. */ -} fd_closework = { - .inband_work =PIPELINE_INBAND_WORK_INITIALIZER (lostage_trigger_close), -}; +static DEFINE_IRQ_WORK(fd_closework, lostage_trigger_close);static void __put_fd (struct rtdm_fd *fd,spl_t s) {
@@ -331,7 +328,7 @@ static void __put_fd(struct rtdm_fd *fd, spl_t s)xnlock_put_irqrestore (&fdtree_lock, s);if (trigger) -pipeline_post_inband_work (&fd_closework); +irq_work_queue (&fd_closework); } }
kernel/cobalt/thread.c +10 -14
@@ -40,7 +40,6 @@#include <cobalt/kernel/lock.h> #include <cobalt/kernel/thread.h> #include <pipeline/kevents.h> -#include <pipeline/inband_work.h> #include <pipeline/sched.h> #include <trace/events/cobalt-core.h> #include "debug.h"
@@ -86,12 +85,12 @@ static inline void enlist_new_thread(struct xnthread *thread) }struct kthread_arg { -struct pipeline_inband_work inband_work;/* Must be first. */ +struct irq_work inband_work;struct xnthread *thread;struct completion *done; }; -static void do_parent_wakeup (struct pipeline_inband_work *inband_work) +static void do_parent_wakeup (struct irq_work *inband_work) {struct kthread_arg *ka;
@@ -150,9 +149,8 @@ static int map_kthread(struct xnthread *thread, struct kthread_arg *ka)trace_cobalt_lostage_request ("wakeup" , current); - ka->inband_work = (struct pipeline_inband_work ) -PIPELINE_INBAND_WORK_INITIALIZER (do_parent_wakeup); -pipeline_post_inband_work (ka); + ka->inband_work =IRQ_WORK_INIT (do_parent_wakeup); +irq_work_queue (&ka->inband_work);xnlock_get_irqsave (&nklock, s);
@@ -1974,7 +1972,7 @@ int xnthread_harden(void) }EXPORT_SYMBOL_GPL (xnthread_harden); -static void lostage_task_wakeup (struct pipeline_inband_work *inband_work) +static void lostage_task_wakeup (struct irq_work *inband_work) {struct lostage_wakeup *rq;struct task_struct *p;
@@ -2060,8 +2058,7 @@ void xnthread_relax(int notify, int reason)*/ trace_cobalt_shadow_gorelax (reason); - thread->relax_work.inband_work = (struct pipeline_inband_work ) -PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_wakeup); + thread->relax_work.inband_work =IRQ_WORK_INIT (lostage_task_wakeup); thread->relax_work.task = p;/*
@@ -2076,7 +2073,7 @@ void xnthread_relax(int notify, int reason)*/ splmax ();trace_cobalt_lostage_request ("wakeup" , p); -pipeline_post_inband_work (&thread->relax_work); +irq_work_queue (&thread->relax_work.inband_work);/* * Grab the nklock to synchronize the Linux task state * manipulation with handle_sigwake_event. This lock will be
@@ -2154,7 +2151,7 @@ void xnthread_relax(int notify, int reason) }EXPORT_SYMBOL_GPL (xnthread_relax); -static void lostage_task_signal (struct pipeline_inband_work *inband_work) +static void lostage_task_signal (struct irq_work *inband_work) {struct lostage_signal *rq;struct task_struct *p;
@@ -2387,15 +2384,14 @@ void __xnthread_signal(struct xnthread *thread, int sig, int arg)if (sigwork->task)return ; - sigwork->inband_work = (struct pipeline_inband_work ) -PIPELINE_INBAND_WORK_INITIALIZER (lostage_task_signal); + sigwork->inband_work =IRQ_WORK_INIT (lostage_task_signal); sigwork->task =xnthread_host_task (thread); sigwork->signo = sig; sigwork->sigval = sig ==SIGDEBUG ? arg | sigdebug_marker : arg;trace_cobalt_lostage_request ("signal" , sigwork->task); -pipeline_post_inband_work (sigwork); +irq_work_queue (&sigwork->inband_work); }void xnthread_signal (struct xnthread *thread,int sig,int arg)
kernel/drivers/udd/udd.c +6 -6
@@ -17,6 +17,8 @@* along with this program; if not, write to the Free Software * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. */ + +#include <linux/irq_work.h> #include <linux/module.h> #include <linux/init.h> #include <linux/slab.h>
@@ -24,7 +26,6 @@#include <rtdm/cobalt.h> #include <rtdm/driver.h> #include <rtdm/udd.h> -#include <pipeline/inband_work.h> struct udd_context {u32 event_count;
@@ -516,13 +517,13 @@ void udd_notify_event(struct udd_device *udd)EXPORT_SYMBOL_GPL (udd_notify_event);struct irqswitch_work { -struct pipeline_inband_work inband_work; +struct irq_work inband_work;rtdm_irq_t *irqh;int enabled;rtdm_event_t *done; }; -static void lostage_irqswitch_line (struct pipeline_inband_work *inband_work) +static void lostage_irqswitch_line (struct irq_work *inband_work) {struct irqswitch_work *rq;
@@ -550,8 +551,7 @@ static void switch_irq_line(rtdm_irq_t *irqh, int enable, rtdm_event_t *done)if (WARN_ON (rq ==NULL ))return ; - rq->inband_work = (struct pipeline_inband_work ) -PIPELINE_INBAND_WORK_INITIALIZER (lostage_irqswitch_line); + rq->inband_work =IRQ_WORK_INIT (lostage_irqswitch_line); rq->irqh = irqh; rq->enabled = enable; rq->done = done;
@@ -561,7 +561,7 @@ static void switch_irq_line(rtdm_irq_t *irqh, int enable, rtdm_event_t *done)* enabling/disabling IRQ lines from primary mode. Defer this * to the root context. */ -pipeline_post_inband_work (rq); +irq_work_queue (&rq->inband_work); }/**
Signature · 1 line
2.43.0
PATCH 3/3 cobalt/posix/process: Sync on thread-hosted irq_work items before exiting
In reply to Jan KiszkaFrom: Jan Kiszka <[email protected]>
Better safe than sorry: Due to migrations, there could theoretically still be signal or relaxation work pending on different CPU. To avoid reaping the carpet underneath of them, sync on all per-thread work_irq items when a former RT thread is terminating.
--- kernel/cobalt/posix/process.c | 5 +++++ 1 file changed, 5 insertions(+)
kernel/cobalt/posix/process.c +5 -0
@@ -838,6 +838,7 @@ static void __handle_taskexit_event(struct task_struct *p) {struct cobalt_ppd *sys_ppd;struct xnthread *thread; +unsigned int n;spl_t s;/*
@@ -861,6 +862,10 @@ static void __handle_taskexit_event(struct task_struct *p)xnthread_run_handler_stack (thread, exit_thread); +for (n =0 ; n <XNTHREAD_MAX_SIGNALS ; n++) +irq_work_sync (&thread->sigarray[n].inband_work); +irq_work_sync (&thread->relax_work.inband_work); +if (xnthread_test_state (thread,XNUSER )) {cobalt_umm_free (&cobalt_kernel_ppd.umm, thread->u_window); thread->u_window =NULL ;
Signature · 1 line
2.43.0
Quoted 5 lines · Jan Kiszka
This should resolve the issue reported in [1] and also clean up a bit more of our now obsolete pipeline abstraction. Test case is planned for later.
Jan
I can confirm that the reproducer for [1] does no longer crash.
Quoted 22 lines
[1] https://lore.kernel.org/xenomai/[email protected]/T/#t
Jan Kiszka (3):
cobalt: Move relax relatedlostage_wakeupwork from stack to thread
cobalt: Drop obsoletepipeline_inband_workabstraction cobalt/posix/process: Sync on thread-hostedirq_workitems before
exiting.../kernel/dovetail/pipeline/inband_work.h | 28 ---------------- include/cobalt/kernel/rtdm/driver.h | 9 +++-- include/cobalt/kernel/thread.h | 10 ++++-- kernel/cobalt/posix/process.c | 5 +++ kernel/cobalt/rtdm/drvlib.c | 15 +++------ kernel/cobalt/rtdm/fd.c | 13 +++----- kernel/cobalt/thread.c | 33 +++++++------------ kernel/drivers/udd/udd.c | 12 +++---- 8 files changed, 45 insertions(+), 80 deletions(-) delete mode 100644 include/cobalt/kernel/dovetail/pipeline/inband_work.hSignature · 1 line
2.43.0