Skip to contentSkip to conversation

Fix races in thread relaxation

PATCH In review Jan Kiszka 3 participants 12 messages Aug 30, 2024 – Oct 22, 2024

lore
Share & tools
lore

Jan Kiszka

author
#0 lore raw

cover Fix races in thread relaxation

This should resolve the issue reported in [1] and also clean up a bit more of our now obsolete pipeline abstraction. Test case is planned for later.

Jan

[1] https://lore.kernel.org/xenomai/[email protected]/T/#t

Jan Kiszka (3):
cobalt: Move relax related lostage_wakeup work from stack to thread
cobalt: Drop obsolete pipeline_inband_work abstraction cobalt/posix/process: Sync on thread-hosted irq_work items before
exiting

 .../kernel/dovetail/pipeline/inband_work.h    | 28 ----------------
 include/cobalt/kernel/rtdm/driver.h           |  9 +++--
 include/cobalt/kernel/thread.h                | 10 ++++--
 kernel/cobalt/posix/process.c                 |  5 +++
 kernel/cobalt/rtdm/drvlib.c                   | 15 +++------
 kernel/cobalt/rtdm/fd.c                       | 13 +++-----
 kernel/cobalt/thread.c                        | 33 +++++++------------
 kernel/drivers/udd/udd.c                      | 12 +++----
 8 files changed, 45 insertions(+), 80 deletions(-)
 delete mode 100644 include/cobalt/kernel/dovetail/pipeline/inband_work.h
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH 1/3 cobalt: Move relax related lostage_wakeup work from stack to thread

From: Jan Kiszka <[email protected]>

We cannot rely on the lostage_wakeup work being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As the lostage_wakeup work is currently on the stack, it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].

Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using irq_work_sync in the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.

[1] https://lore.kernel.org/xenomai/[email protected]/T/#t

---
 include/cobalt/kernel/thread.h |  6 ++++++
 kernel/cobalt/thread.c         | 15 +++++----------
 2 files changed, 11 insertions(+), 10 deletions(-)
include/cobalt/kernel/thread.h +6 -0
index 7f9a37d29a..22ed61168c 100644
@@ -64,6 +64,11 @@ struct lostage_signal {
 	int signo, sigval;
 };
 
+struct lostage_wakeup {
+	struct pipeline_inband_work inband_work; /* Must be first. */
+	struct task_struct *task;
+};
+
 struct xnthread_init_attr {
 	struct xnthread_personality *personality;
 	cpumask_t affinity;
@@ -214,6 +219,7 @@ struct xnthread {
 	u32 proghash;		/* Hash value for exe_path */
 #endif
 	struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS];
+	struct lostage_wakeup relax_work;
 };
 
 static inline int xnthread_get_state(const struct xnthread *thread)
kernel/cobalt/thread.c +5 -10
index 8738fadfce..7dffb081bd 100644
@@ -1974,11 +1974,6 @@ int xnthread_harden(void)
 }
 EXPORT_SYMBOL_GPL(xnthread_harden);
 
-struct lostage_wakeup {
-	struct pipeline_inband_work inband_work; /* Must be first. */
-	struct task_struct *task;
-};
-
 static void lostage_task_wakeup(struct pipeline_inband_work *inband_work)
 {
 	struct lostage_wakeup *rq;
@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)
 void xnthread_relax(int notify, int reason)
 {
 	struct task_struct *p = current;
-	struct lostage_wakeup wakework = {
-		.inband_work = PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup),
-		.task = p,
-	};
 	struct xnthread *thread = xnthread_current();
 	int cpu __maybe_unused, suspension;
 	kernel_siginfo_t si;
@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)
 	 */
 	trace_cobalt_shadow_gorelax(reason);
 
+	thread->relax_work.inband_work = (struct pipeline_inband_work)
+		PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup);
+	thread->relax_work.task = p;
+
 	/*
 	 * If you intend to change the following interrupt-free
 	 * sequence, /first/ make sure to check the special handling
@@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)
 	 */
 	splmax();
 	trace_cobalt_lostage_request("wakeup", p);
-	pipeline_post_inband_work(&wakework);
+	pipeline_post_inband_work(&thread->relax_work);
 	/*
 	 * Grab the nklock to synchronize the Linux task state
 	 * manipulation with handle_sigwake_event. This lock will be
Signature · 1 line
2.43.0
#2 lore raw

Re:[PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread

  hi Jan, the issue reported in [1] has seemly not been solved in ipipe.  Will this issue still be maintained on ipipe?
    the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]).
    linux kernel: 4.19.89
    xenomai: 3.2.4 + patch[1/3]

    [1] https://lore.kernel.org/xenomai/[email protected]/T/#t

[   98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[   98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[   98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[   98.182496] I-pipe domain: Linux
[   98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[   98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[   98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[   98.182564] RAX: dead000000000100 RBX: 000000000005f3e0 RCX: ffffc900012d2a88
[   98.182577] RDX: 0000000000000000 RSI: ffffc900012d1840 RDI: 0000000000001032
[   98.182591] RBP: 0000000000000001 R08: ffffc900012d2a88 R09: ffffc900012d2a78
[   98.182605] R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900012d3708
[   98.182620] R13: 0000000000000003 R14: ffffc900012d1840 R15: 0000000000000000
[   98.182634] FS:  00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[   98.182650] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   98.182661] CR2: 000055738facc000 CR3: 000000012ad44005 CR4: 00000000001606e0
[   98.182698] Call Trace:
[   98.182708]  __cobalt_cond_wait_prologue+0x26d/0x850
[   98.182720]  ? handle_root_syscall+0xd3/0x310
[   98.182730]  ? __cobalt_cond_wait_prologue+0x850/0x850
[   98.182741]  CoBaLt_cond_wait_prologue+0x1d/0x20
[   98.182751]  handle_head_syscall+0x106/0x3b0
[   98.182762]  ipipe_handle_syscall+0x66/0xc0
[   98.182772]  do_syscall_64+0x22/0x170
[   98.182782]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   98.182793] RIP: 0033:0x7f888d29575b
[   98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[   98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246 ORIG_RAX: 0000000010000023
[   98.182852] RAX: ffffffffffffffda RBX: 00007f888c56e6e0 RCX: 00007f888d29575b
[   98.183195] RDX: 00007f888c3baba4 RSI: 00007f888c56e6e0 RDI: 00007f888c56e7f8
[   98.183530] RBP: 00007f888c56e7f8 R08: 0000000000000000 R09: 00007f888c56e6d0
[   98.183854] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f888c56e6e0
[   98.184182] R13: 00007f888c3bad80 R14: 0000000000000001 R15: 00007f888c3babb0
[   98.184500] Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c arc4 ecb md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs ccm dns_resolver fscache ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter vmwgfx sg ttm drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect sysimgblt serio_raw snd_pcsp snd_pcm snd_timer snd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc evdev aesni_intel aes_x86_64 crypto_simd cryptd glue_helper soundcore ac button binfmt_misc sunrpc [   98.187446]  ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod ata_generic ata_piix crc32c_intel mptspi mptscsih mptbase scsi_transport_spi libata scsi_mod e1000 psmouse i2c_piix4

Quoted 96 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

We cannot rely on the lostage_wakeup work being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As the lostage_wakeup work is currently on the stack, it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].

Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using irq_work_sync in the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.

[1] https://lore.kernel.org/xenomai/[email protected]/T/#t

---
include/cobalt/kernel/thread.h | 6 ++++++
kernel/cobalt/thread.c | 15 +++++----------

2 files changed, 11 insertions(+), 10 deletions(-)
include/cobalt/kernel/thread.h +6 -0
index 7f9a37d29a..22ed61168c 100644
@@ -64,6 +64,11 @@ struct lostage_signal {
 	int signo, sigval;
 };
 
+struct lostage_wakeup {
+	struct pipeline_inband_work inband_work; /* Must be first. */
+	struct task_struct *task;
+};
+
 struct xnthread_init_attr {
 	struct xnthread_personality *personality;
 	cpumask_t affinity;
@@ -214,6 +219,7 @@ struct xnthread {
 	u32 proghash;		/* Hash value for exe_path */
 #endif
 	struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS];
+	struct lostage_wakeup relax_work;
 };
 
 static inline int xnthread_get_state(const struct xnthread *thread)
kernel/cobalt/thread.c +5 -10
index 8738fadfce..7dffb081bd 100644
@@ -1974,11 +1974,6 @@ int xnthread_harden(void)
 }
 EXPORT_SYMBOL_GPL(xnthread_harden);
 
-struct lostage_wakeup {
-	struct pipeline_inband_work inband_work; /* Must be first. */
-	struct task_struct *task;
-};
-
 static void lostage_task_wakeup(struct pipeline_inband_work *inband_work)
 {
 	struct lostage_wakeup *rq;
@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)
 void xnthread_relax(int notify, int reason)
 {
 	struct task_struct *p = current;
-	struct lostage_wakeup wakework = {
-		.inband_work = PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup),
-		.task = p,
-	};
 	struct xnthread *thread = xnthread_current();
 	int cpu __maybe_unused, suspension;
 	kernel_siginfo_t si;
@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)
 	 */
 	trace_cobalt_shadow_gorelax(reason);
 
+	thread->relax_work.inband_work = (struct pipeline_inband_work)
+		PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup);
+	thread->relax_work.task = p;
+
 	/*
 	 * If you intend to change the following interrupt-free
 	 * sequence, /first/ make sure to check the special handling
@@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)
 	 */
 	splmax();
 	trace_cobalt_lostage_request("wakeup", p);
-	pipeline_post_inband_work(&wakework);
+	pipeline_post_inband_work(&thread->relax_work);
 	/*
 	 * Grab the nklock to synchronize the Linux task state
 	 * manipulation with handle_sigwake_event. This lock will be
Signature · 1 line
2.43.0
author
#3 lore raw

Re: [PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread

Quoted 8 lines · JerryJun

  hi Jan, the issue reported in [1] has seemly not been solved in ipipe.  Will this issue still be maintained on ipipe? the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]). linux kernel: 4.19.89
    xenomai: 3.2.4 + patch[1/3]

    [1] https://lore.kernel.org/xenomai/[email protected]/T/#t

Sorry, that fix is still only on master (upcoming 3.3), not yet in stable trees. It's still not expected to be I-pipe/dovetail-dependent.

Let me recover a bit more from yesterday's Preempt RT merge party, then I will close that gap.

Jan

Quoted 134 lines

[   98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[   98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[   98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[   98.182496] I-pipe domain: Linux
[   98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[   98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[   98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[   98.182564] RAX: dead000000000100 RBX: 000000000005f3e0 RCX: ffffc900012d2a88
[   98.182577] RDX: 0000000000000000 RSI: ffffc900012d1840 RDI: 0000000000001032
[   98.182591] RBP: 0000000000000001 R08: ffffc900012d2a88 R09: ffffc900012d2a78
[   98.182605] R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900012d3708
[   98.182620] R13: 0000000000000003 R14: ffffc900012d1840 R15: 0000000000000000
[   98.182634] FS:  00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[   98.182650] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   98.182661] CR2: 000055738facc000 CR3: 000000012ad44005 CR4: 00000000001606e0
[   98.182698] Call Trace:
[   98.182708]  __cobalt_cond_wait_prologue+0x26d/0x850
[   98.182720]  ? handle_root_syscall+0xd3/0x310
[   98.182730]  ? __cobalt_cond_wait_prologue+0x850/0x850
[   98.182741]  CoBaLt_cond_wait_prologue+0x1d/0x20
[   98.182751]  handle_head_syscall+0x106/0x3b0
[   98.182762]  ipipe_handle_syscall+0x66/0xc0
[   98.182772]  do_syscall_64+0x22/0x170
[   98.182782]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   98.182793] RIP: 0033:0x7f888d29575b
[   98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[   98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246 ORIG_RAX: 0000000010000023
[   98.182852] RAX: ffffffffffffffda RBX: 00007f888c56e6e0 RCX: 00007f888d29575b
[   98.183195] RDX: 00007f888c3baba4 RSI: 00007f888c56e6e0 RDI: 00007f888c56e7f8
[   98.183530] RBP: 00007f888c56e7f8 R08: 0000000000000000 R09: 00007f888c56e6d0
[   98.183854] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f888c56e6e0
[   98.184182] R13: 00007f888c3bad80 R14: 0000000000000001 R15: 00007f888c3babb0
[   98.184500] Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c arc4 ecb md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs ccm dns_resolver fscache ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter vmwgfx sg ttm drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect sysimgblt serio_raw snd_pcsp snd_pcm snd_timer snd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc evdev aesni_intel aes_x86_64 crypto_simd cryptd glue_helper soundcore ac button binfmt_misc sunrpc [   98.187446]  ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod ata_generic ata_piix crc32c_intel mptspi mptscsih mptbase scsi_transport_spi libata scsi_mod e1000 psmouse i2c_piix4

Quoted 96 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

We cannot rely on the lostage_wakeup work being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As the lostage_wakeup work is currently on the stack, it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].

Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using irq_work_sync in the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.

[1] https://lore.kernel.org/xenomai/[email protected]/T/#t

---
include/cobalt/kernel/thread.h | 6 ++++++
kernel/cobalt/thread.c | 15 +++++----------

2 files changed, 11 insertions(+), 10 deletions(-)
include/cobalt/kernel/thread.h +6 -0
index 7f9a37d29a..22ed61168c 100644
@@ -64,6 +64,11 @@ struct lostage_signal {
 	int signo, sigval;
 };
 
+struct lostage_wakeup {
+	struct pipeline_inband_work inband_work; /* Must be first. */
+	struct task_struct *task;
+};
+
 struct xnthread_init_attr {
 	struct xnthread_personality *personality;
 	cpumask_t affinity;
@@ -214,6 +219,7 @@ struct xnthread {
 	u32 proghash;		/* Hash value for exe_path */
 #endif
 	struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS];
+	struct lostage_wakeup relax_work;
 };
 
 static inline int xnthread_get_state(const struct xnthread *thread)
kernel/cobalt/thread.c +5 -10
index 8738fadfce..7dffb081bd 100644
@@ -1974,11 +1974,6 @@ int xnthread_harden(void)
 }
 EXPORT_SYMBOL_GPL(xnthread_harden);
 
-struct lostage_wakeup {
-	struct pipeline_inband_work inband_work; /* Must be first. */
-	struct task_struct *task;
-};
-
 static void lostage_task_wakeup(struct pipeline_inband_work *inband_work)
 {
 	struct lostage_wakeup *rq;
@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)
 void xnthread_relax(int notify, int reason)
 {
 	struct task_struct *p = current;
-	struct lostage_wakeup wakework = {
-		.inband_work = PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup),
-		.task = p,
-	};
 	struct xnthread *thread = xnthread_current();
 	int cpu __maybe_unused, suspension;
 	kernel_siginfo_t si;
@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)
 	 */
 	trace_cobalt_shadow_gorelax(reason);
 
+	thread->relax_work.inband_work = (struct pipeline_inband_work)
+		PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup);
+	thread->relax_work.task = p;
+
 	/*
 	 * If you intend to change the following interrupt-free
 	 * sequence, /first/ make sure to check the special handling
@@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)
 	 */
 	splmax();
 	trace_cobalt_lostage_request("wakeup", p);
-	pipeline_post_inband_work(&wakework);
+	pipeline_post_inband_work(&thread->relax_work);
 	/*
 	 * Grab the nklock to synchronize the Linux task state
 	 * manipulation with handle_sigwake_event. This lock will be
Signature · 1 line
2.43.0
Signature · 2 lines
Siemens AG, Technology
Linux Expert Center
#4 lore raw

Re:Re: [PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread

hi jan, in this patch you mentioned "As the lostage_wakeup work is currently on the stack,
 it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one ", why it‘s not triggered it in the switchtest benchmark which involves "xnthread_relax" due to linux syscall ? may be you probably describe the triggered scene detailed? Or how can I trigger it in switchtest
 by using xnthread_relax leaves earlier on the new CPU than the work was processed on the old one? thanks

Quoted 156 lines · Jan Kiszka
Quoted 8 lines · JerryJun

  hi Jan, the issue reported in [1] has seemly not been solved in ipipe.  Will this issue still be maintained on ipipe? the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]). linux kernel: 4.19.89
    xenomai: 3.2.4 + patch[1/3]

    [1] https://lore.kernel.org/xenomai/[email protected]/T/#t

Sorry, that fix is still only on master (upcoming 3.3), not yet in stable trees. It's still not expected to be I-pipe/dovetail-dependent.

Let me recover a bit more from yesterday's Preempt RT merge party, then I will close that gap.

Jan

Quoted 134 lines

[   98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[   98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[   98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[   98.182496] I-pipe domain: Linux
[   98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[   98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[   98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[   98.182564] RAX: dead000000000100 RBX: 000000000005f3e0 RCX: ffffc900012d2a88
[   98.182577] RDX: 0000000000000000 RSI: ffffc900012d1840 RDI: 0000000000001032
[   98.182591] RBP: 0000000000000001 R08: ffffc900012d2a88 R09: ffffc900012d2a78
[   98.182605] R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900012d3708
[   98.182620] R13: 0000000000000003 R14: ffffc900012d1840 R15: 0000000000000000
[   98.182634] FS:  00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[   98.182650] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   98.182661] CR2: 000055738facc000 CR3: 000000012ad44005 CR4: 00000000001606e0
[   98.182698] Call Trace:
[   98.182708]  __cobalt_cond_wait_prologue+0x26d/0x850
[   98.182720]  ? handle_root_syscall+0xd3/0x310
[   98.182730]  ? __cobalt_cond_wait_prologue+0x850/0x850
[   98.182741]  CoBaLt_cond_wait_prologue+0x1d/0x20
[   98.182751]  handle_head_syscall+0x106/0x3b0
[   98.182762]  ipipe_handle_syscall+0x66/0xc0
[   98.182772]  do_syscall_64+0x22/0x170
[   98.182782]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   98.182793] RIP: 0033:0x7f888d29575b
[   98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[   98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246 ORIG_RAX: 0000000010000023
[   98.182852] RAX: ffffffffffffffda RBX: 00007f888c56e6e0 RCX: 00007f888d29575b
[   98.183195] RDX: 00007f888c3baba4 RSI: 00007f888c56e6e0 RDI: 00007f888c56e7f8
[   98.183530] RBP: 00007f888c56e7f8 R08: 0000000000000000 R09: 00007f888c56e6d0
[   98.183854] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f888c56e6e0
[   98.184182] R13: 00007f888c3bad80 R14: 0000000000000001 R15: 00007f888c3babb0
[   98.184500] Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c arc4 ecb md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs ccm dns_resolver fscache ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter vmwgfx sg ttm drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect sysimgblt serio_raw snd_pcsp snd_pcm snd_timer snd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc evdev aesni_intel aes_x86_64 crypto_simd cryptd glue_helper soundcore ac button binfmt_misc sunrpc [   98.187446]  ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod ata_generic ata_piix crc32c_intel mptspi mptscsih mptbase scsi_transport_spi libata scsi_mod e1000 psmouse i2c_piix4

Quoted 96 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

We cannot rely on the lostage_wakeup work being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As the lostage_wakeup work is currently on the stack, it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].

Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using irq_work_sync in the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.

[1] https://lore.kernel.org/xenomai/[email protected]/T/#t

---
include/cobalt/kernel/thread.h | 6 ++++++
kernel/cobalt/thread.c | 15 +++++----------

2 files changed, 11 insertions(+), 10 deletions(-)
include/cobalt/kernel/thread.h +6 -0
index 7f9a37d29a..22ed61168c 100644
@@ -64,6 +64,11 @@ struct lostage_signal {
 	int signo, sigval;
 };
 
+struct lostage_wakeup {
+	struct pipeline_inband_work inband_work; /* Must be first. */
+	struct task_struct *task;
+};
+
 struct xnthread_init_attr {
 	struct xnthread_personality *personality;
 	cpumask_t affinity;
@@ -214,6 +219,7 @@ struct xnthread {
 	u32 proghash;		/* Hash value for exe_path */
 #endif
 	struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS];
+	struct lostage_wakeup relax_work;
 };
 
 static inline int xnthread_get_state(const struct xnthread *thread)
kernel/cobalt/thread.c +5 -10
index 8738fadfce..7dffb081bd 100644
@@ -1974,11 +1974,6 @@ int xnthread_harden(void)
 }
 EXPORT_SYMBOL_GPL(xnthread_harden);
 
-struct lostage_wakeup {
-	struct pipeline_inband_work inband_work; /* Must be first. */
-	struct task_struct *task;
-};
-
 static void lostage_task_wakeup(struct pipeline_inband_work *inband_work)
 {
 	struct lostage_wakeup *rq;
@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)
 void xnthread_relax(int notify, int reason)
 {
 	struct task_struct *p = current;
-	struct lostage_wakeup wakework = {
-		.inband_work = PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup),
-		.task = p,
-	};
 	struct xnthread *thread = xnthread_current();
 	int cpu __maybe_unused, suspension;
 	kernel_siginfo_t si;
@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)
 	 */
 	trace_cobalt_shadow_gorelax(reason);
 
+	thread->relax_work.inband_work = (struct pipeline_inband_work)
+		PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup);
+	thread->relax_work.task = p;
+
 	/*
 	 * If you intend to change the following interrupt-free
 	 * sequence, /first/ make sure to check the special handling
@@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)
 	 */
 	splmax();
 	trace_cobalt_lostage_request("wakeup", p);
-	pipeline_post_inband_work(&wakework);
+	pipeline_post_inband_work(&thread->relax_work);
 	/*
 	 * Grab the nklock to synchronize the Linux task state
 	 * manipulation with handle_sigwake_event. This lock will be
Signature · 1 line
2.43.0
Signature · 2 lines
Siemens AG, Technology
Linux Expert Center
#5 lore raw

Re: Re: [PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread

Quoted 5 lines · JerryJun

hi jan, in this patch you mentioned "As the lostage_wakeup work is currently on the stack,
 it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one ", why it‘s not triggered it in the switchtest benchmark which involves "xnthread_relax" due to linux syscall ? may be you probably describe the triggered scene detailed? Or how can I trigger it in switchtest
 by using xnthread_relax leaves earlier on the new CPU than the work was processed on the old one? thanks

It's about task destruction, not migration.

The reproducer did a lot of task creation/destruction to trigger that. It happened that the task was already gone while there was still a node in the irq work referencing the "dead" task.

Classical use after free due to missing/broken synchronization.

We're still missing a proper test for that scenario. To my best knowledge Clara will revisit this topic.

Florian

Quoted 159 lines
Quoted 156 lines · Jan Kiszka
Quoted 8 lines · JerryJun

  hi Jan, the issue reported in [1] has seemly not been solved in ipipe.  Will this issue still be maintained on ipipe? the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]). linux kernel: 4.19.89
    xenomai: 3.2.4 + patch[1/3]

    [1] https://lore.kernel.org/xenomai/[email protected]/T/#t

Sorry, that fix is still only on master (upcoming 3.3), not yet in stable trees. It's still not expected to be I-pipe/dovetail-dependent.

Let me recover a bit more from yesterday's Preempt RT merge party, then I will close that gap.

Jan

Quoted 134 lines

[   98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[   98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[   98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[   98.182496] I-pipe domain: Linux
[   98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[   98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[   98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[   98.182564] RAX: dead000000000100 RBX: 000000000005f3e0 RCX: ffffc900012d2a88
[   98.182577] RDX: 0000000000000000 RSI: ffffc900012d1840 RDI: 0000000000001032
[   98.182591] RBP: 0000000000000001 R08: ffffc900012d2a88 R09: ffffc900012d2a78
[   98.182605] R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900012d3708
[   98.182620] R13: 0000000000000003 R14: ffffc900012d1840 R15: 0000000000000000
[   98.182634] FS:  00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[   98.182650] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   98.182661] CR2: 000055738facc000 CR3: 000000012ad44005 CR4: 00000000001606e0
[   98.182698] Call Trace:
[   98.182708]  __cobalt_cond_wait_prologue+0x26d/0x850
[   98.182720]  ? handle_root_syscall+0xd3/0x310
[   98.182730]  ? __cobalt_cond_wait_prologue+0x850/0x850
[   98.182741]  CoBaLt_cond_wait_prologue+0x1d/0x20
[   98.182751]  handle_head_syscall+0x106/0x3b0
[   98.182762]  ipipe_handle_syscall+0x66/0xc0
[   98.182772]  do_syscall_64+0x22/0x170
[   98.182782]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   98.182793] RIP: 0033:0x7f888d29575b
[   98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[   98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246 ORIG_RAX: 0000000010000023
[   98.182852] RAX: ffffffffffffffda RBX: 00007f888c56e6e0 RCX: 00007f888d29575b
[   98.183195] RDX: 00007f888c3baba4 RSI: 00007f888c56e6e0 RDI: 00007f888c56e7f8
[   98.183530] RBP: 00007f888c56e7f8 R08: 0000000000000000 R09: 00007f888c56e6d0
[   98.183854] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f888c56e6e0
[   98.184182] R13: 00007f888c3bad80 R14: 0000000000000001 R15: 00007f888c3babb0
[   98.184500] Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c arc4 ecb md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs ccm dns_resolver fscache ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter vmwgfx sg ttm drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect sysimgblt serio_raw snd_pcsp snd_pcm snd_timer snd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc evdev aesni_intel aes_x86_64 crypto_simd cryptd glue_helper soundcore ac button binfmt_misc sunrpc [   98.187446]  ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod ata_generic ata_piix crc32c_intel mptspi mptscsih mptbase scsi_transport_spi libata scsi_mod e1000 psmouse i2c_piix4

Quoted 96 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

We cannot rely on the lostage_wakeup work being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As the lostage_wakeup work is currently on the stack, it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].

Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using irq_work_sync in the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.

[1] https://lore.kernel.org/xenomai/[email protected]/T/#t

---
include/cobalt/kernel/thread.h | 6 ++++++
kernel/cobalt/thread.c | 15 +++++----------

2 files changed, 11 insertions(+), 10 deletions(-)
include/cobalt/kernel/thread.h +6 -0
index 7f9a37d29a..22ed61168c 100644
@@ -64,6 +64,11 @@ struct lostage_signal {
 	int signo, sigval;
 };
 
+struct lostage_wakeup {
+	struct pipeline_inband_work inband_work; /* Must be first. */
+	struct task_struct *task;
+};
+
 struct xnthread_init_attr {
 	struct xnthread_personality *personality;
 	cpumask_t affinity;
@@ -214,6 +219,7 @@ struct xnthread {
 	u32 proghash;		/* Hash value for exe_path */
 #endif
 	struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS];
+	struct lostage_wakeup relax_work;
 };
 
 static inline int xnthread_get_state(const struct xnthread *thread)
kernel/cobalt/thread.c +5 -10
index 8738fadfce..7dffb081bd 100644
@@ -1974,11 +1974,6 @@ int xnthread_harden(void)
 }
 EXPORT_SYMBOL_GPL(xnthread_harden);
 
-struct lostage_wakeup {
-	struct pipeline_inband_work inband_work; /* Must be first. */
-	struct task_struct *task;
-};
-
 static void lostage_task_wakeup(struct pipeline_inband_work *inband_work)
 {
 	struct lostage_wakeup *rq;
@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)
 void xnthread_relax(int notify, int reason)
 {
 	struct task_struct *p = current;
-	struct lostage_wakeup wakework = {
-		.inband_work = PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup),
-		.task = p,
-	};
 	struct xnthread *thread = xnthread_current();
 	int cpu __maybe_unused, suspension;
 	kernel_siginfo_t si;
@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)
 	 */
 	trace_cobalt_shadow_gorelax(reason);
 
+	thread->relax_work.inband_work = (struct pipeline_inband_work)
+		PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup);
+	thread->relax_work.task = p;
+
 	/*
 	 * If you intend to change the following interrupt-free
 	 * sequence, /first/ make sure to check the special handling
@@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)
 	 */
 	splmax();
 	trace_cobalt_lostage_request("wakeup", p);
-	pipeline_post_inband_work(&wakework);
+	pipeline_post_inband_work(&thread->relax_work);
 	/*
 	 * Grab the nklock to synchronize the Linux task state
 	 * manipulation with handle_sigwake_event. This lock will be
Signature · 1 line
2.43.0
Signature · 2 lines
Siemens AG, Technology
Linux Expert Center
#6 lore raw

Re:Re: Re: [PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread

It seems to have something to do with migration.
it‘s not triggered it by "taskset -c 1 ./xeno-app" from "Attachment #2: xenomai-alchemy-app.tar.gz" which make the main thread and child thread on the same cpu.  besides, there is an interesting problem that it‘s also not triggered it when replace alchemy with pthread(libcobalt) API from attachment.

Quoted 13 lines · Florian
Quoted 5 lines · JerryJun

hi jan, in this patch you mentioned "As the lostage_wakeup work is currently on the stack,
 it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one ", why it‘s not triggered it in the switchtest benchmark which involves "xnthread_relax" due to linux syscall ? may be you probably describe the triggered scene detailed? Or how can I trigger it in switchtest
 by using xnthread_relax leaves earlier on the new CPU than the work was processed on the old one? thanks

It's about task destruction, not migration.

The reproducer did a lot of task creation/destruction to trigger that. It happened that the task was already gone while there was still a node in the irq work referencing the "dead" task.

Quoted 167 lines

Classical use after free due to missing/broken synchronization.

We're still missing a proper test for that scenario. To my best knowledge Clara will revisit this topic.

Florian

Quoted 159 lines
Quoted 156 lines · Jan Kiszka
Quoted 8 lines · JerryJun

  hi Jan, the issue reported in [1] has seemly not been solved in ipipe.  Will this issue still be maintained on ipipe? the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]). linux kernel: 4.19.89
    xenomai: 3.2.4 + patch[1/3]

    [1] https://lore.kernel.org/xenomai/[email protected]/T/#t

Sorry, that fix is still only on master (upcoming 3.3), not yet in stable trees. It's still not expected to be I-pipe/dovetail-dependent.

Let me recover a bit more from yesterday's Preempt RT merge party, then I will close that gap.

Jan

Quoted 134 lines

[   98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[   98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[   98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[   98.182496] I-pipe domain: Linux
[   98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[   98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[   98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[   98.182564] RAX: dead000000000100 RBX: 000000000005f3e0 RCX: ffffc900012d2a88
[   98.182577] RDX: 0000000000000000 RSI: ffffc900012d1840 RDI: 0000000000001032
[   98.182591] RBP: 0000000000000001 R08: ffffc900012d2a88 R09: ffffc900012d2a78
[   98.182605] R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900012d3708
[   98.182620] R13: 0000000000000003 R14: ffffc900012d1840 R15: 0000000000000000
[   98.182634] FS:  00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[   98.182650] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   98.182661] CR2: 000055738facc000 CR3: 000000012ad44005 CR4: 00000000001606e0
[   98.182698] Call Trace:
[   98.182708]  __cobalt_cond_wait_prologue+0x26d/0x850
[   98.182720]  ? handle_root_syscall+0xd3/0x310
[   98.182730]  ? __cobalt_cond_wait_prologue+0x850/0x850
[   98.182741]  CoBaLt_cond_wait_prologue+0x1d/0x20
[   98.182751]  handle_head_syscall+0x106/0x3b0
[   98.182762]  ipipe_handle_syscall+0x66/0xc0
[   98.182772]  do_syscall_64+0x22/0x170
[   98.182782]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   98.182793] RIP: 0033:0x7f888d29575b
[   98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[   98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246 ORIG_RAX: 0000000010000023
[   98.182852] RAX: ffffffffffffffda RBX: 00007f888c56e6e0 RCX: 00007f888d29575b
[   98.183195] RDX: 00007f888c3baba4 RSI: 00007f888c56e6e0 RDI: 00007f888c56e7f8
[   98.183530] RBP: 00007f888c56e7f8 R08: 0000000000000000 R09: 00007f888c56e6d0
[   98.183854] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f888c56e6e0
[   98.184182] R13: 00007f888c3bad80 R14: 0000000000000001 R15: 00007f888c3babb0
[   98.184500] Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c arc4 ecb md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs ccm dns_resolver fscache ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter vmwgfx sg ttm drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect sysimgblt serio_raw snd_pcsp snd_pcm snd_timer snd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc evdev aesni_intel aes_x86_64 crypto_simd cryptd glue_helper soundcore ac button binfmt_misc sunrpc [   98.187446]  ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod ata_generic ata_piix crc32c_intel mptspi mptscsih mptbase scsi_transport_spi libata scsi_mod e1000 psmouse i2c_piix4

Quoted 96 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

We cannot rely on the lostage_wakeup work being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As the lostage_wakeup work is currently on the stack, it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].

Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using irq_work_sync in the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.

[1] https://lore.kernel.org/xenomai/[email protected]/T/#t

---
include/cobalt/kernel/thread.h | 6 ++++++
kernel/cobalt/thread.c | 15 +++++----------

2 files changed, 11 insertions(+), 10 deletions(-)
include/cobalt/kernel/thread.h +6 -0
index 7f9a37d29a..22ed61168c 100644
@@ -64,6 +64,11 @@ struct lostage_signal {
 	int signo, sigval;
 };
 
+struct lostage_wakeup {
+	struct pipeline_inband_work inband_work; /* Must be first. */
+	struct task_struct *task;
+};
+
 struct xnthread_init_attr {
 	struct xnthread_personality *personality;
 	cpumask_t affinity;
@@ -214,6 +219,7 @@ struct xnthread {
 	u32 proghash;		/* Hash value for exe_path */
 #endif
 	struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS];
+	struct lostage_wakeup relax_work;
 };
 
 static inline int xnthread_get_state(const struct xnthread *thread)
kernel/cobalt/thread.c +5 -10
index 8738fadfce..7dffb081bd 100644
@@ -1974,11 +1974,6 @@ int xnthread_harden(void)
 }
 EXPORT_SYMBOL_GPL(xnthread_harden);
 
-struct lostage_wakeup {
-	struct pipeline_inband_work inband_work; /* Must be first. */
-	struct task_struct *task;
-};
-
 static void lostage_task_wakeup(struct pipeline_inband_work *inband_work)
 {
 	struct lostage_wakeup *rq;
@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)
 void xnthread_relax(int notify, int reason)
 {
 	struct task_struct *p = current;
-	struct lostage_wakeup wakework = {
-		.inband_work = PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup),
-		.task = p,
-	};
 	struct xnthread *thread = xnthread_current();
 	int cpu __maybe_unused, suspension;
 	kernel_siginfo_t si;
@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)
 	 */
 	trace_cobalt_shadow_gorelax(reason);
 
+	thread->relax_work.inband_work = (struct pipeline_inband_work)
+		PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup);
+	thread->relax_work.task = p;
+
 	/*
 	 * If you intend to change the following interrupt-free
 	 * sequence, /first/ make sure to check the special handling
@@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)
 	 */
 	splmax();
 	trace_cobalt_lostage_request("wakeup", p);
-	pipeline_post_inband_work(&wakework);
+	pipeline_post_inband_work(&thread->relax_work);
 	/*
 	 * Grab the nklock to synchronize the Linux task state
 	 * manipulation with handle_sigwake_event. This lock will be
Signature · 1 line
2.43.0
Signature · 2 lines
Siemens AG, Technology
Linux Expert Center

#include <sched.h>
#include <alchemy/task.h>
#include <signal.h>
#include <pthread.h>
#include <stdio.h>

#define THREAD_CNT 30

static int global_exit_trigger = 0;

static void signal_handler()

{
	global_exit_trigger = 1;
}

static void *rt_main(void *arg)

{
	while (!global_exit_trigger) {
		usleep(1 * 1000 * 1000); // 10 sec
	}
	return NULL;
}

static void threads_start(pthread_t *rt_tasks, int *thread_ids) {

 	char task_name[256];
	cpu_set_t mask;
	int ret;
	struct sched_param param = {.sched_priority = 50};
	pthread_attr_t attr;

CPU_ZERO(&mask);

 	CPU_SET(1, &mask);
	pthread_attr_init(&attr);
	pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);
	pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED);
	pthread_attr_setschedpolicy(&attr, SCHED_FIFO);
	pthread_attr_setschedparam(&attr, &param);

pthread_attr_setaffinity_np(&attr, sizeof(mask), &mask);

	for (int i = 0; i < THREAD_CNT; i++) {
		ret = pthread_create(&rt_tasks[i], &attr, &rt_main, NULL);
		if (ret) {
			printf("Failed to create task %d\n", i);
			break;
		}
	}
}

static void threads_stop(pthread_t *rt_tasks)

{
	int ret;

	for (int i = 0; i < THREAD_CNT; i++) {
		pthread_cancel(rt_tasks[i]);
		pthread_join(rt_tasks[i], NULL);
	}
}

int main(int argc, char **argv)
{

 	pthread_t rt_tasks[THREAD_CNT];
	int thread_ids[THREAD_CNT];

	signal(SIGINT, signal_handler);

	printf("Starting\n");

	while (!global_exit_trigger) {
		threads_start(rt_tasks, thread_ids);
		usleep(1*1000); // 1ms
		threads_stop(rt_tasks);
	}
}
#7 lore raw

Re: Re: Re: [PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread

Quoted 2 lines · JerryJun

It seems to have something to do with migration.
it‘s not triggered it by "taskset -c 1 ./xeno-app" from "Attachment #2: xenomai-alchemy-app.tar.gz" which make the main thread and child thread on the same cpu.  besides, there is an interesting problem that it‘s also not triggered it when replace alchemy with pthread(libcobalt) API from attachment.

The affinity just closed the race window. It was clearly a bug in the task shutdown sequence.

Quoted 182 lines
Quoted 180 lines · Florian
Quoted 5 lines · JerryJun

hi jan, in this patch you mentioned "As the lostage_wakeup work is currently on the stack,
 it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one ", why it‘s not triggered it in the switchtest benchmark which involves "xnthread_relax" due to linux syscall ? may be you probably describe the triggered scene detailed? Or how can I trigger it in switchtest
 by using xnthread_relax leaves earlier on the new CPU than the work was processed on the old one? thanks

It's about task destruction, not migration.

The reproducer did a lot of task creation/destruction to trigger that. It happened that the task was already gone while there was still a node in the irq work referencing the "dead" task.

Classical use after free due to missing/broken synchronization.

We're still missing a proper test for that scenario. To my best knowledge Clara will revisit this topic.

Florian

Quoted 159 lines
Quoted 156 lines · Jan Kiszka
Quoted 8 lines · JerryJun

  hi Jan, the issue reported in [1] has seemly not been solved in ipipe.  Will this issue still be maintained on ipipe? the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]). linux kernel: 4.19.89
    xenomai: 3.2.4 + patch[1/3]

    [1] https://lore.kernel.org/xenomai/[email protected]/T/#t

Sorry, that fix is still only on master (upcoming 3.3), not yet in stable trees. It's still not expected to be I-pipe/dovetail-dependent.

Let me recover a bit more from yesterday's Preempt RT merge party, then I will close that gap.

Jan

Quoted 134 lines

[   98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[   98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[   98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[   98.182496] I-pipe domain: Linux
[   98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[   98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[   98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[   98.182564] RAX: dead000000000100 RBX: 000000000005f3e0 RCX: ffffc900012d2a88
[   98.182577] RDX: 0000000000000000 RSI: ffffc900012d1840 RDI: 0000000000001032
[   98.182591] RBP: 0000000000000001 R08: ffffc900012d2a88 R09: ffffc900012d2a78
[   98.182605] R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900012d3708
[   98.182620] R13: 0000000000000003 R14: ffffc900012d1840 R15: 0000000000000000
[   98.182634] FS:  00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[   98.182650] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   98.182661] CR2: 000055738facc000 CR3: 000000012ad44005 CR4: 00000000001606e0
[   98.182698] Call Trace:
[   98.182708]  __cobalt_cond_wait_prologue+0x26d/0x850
[   98.182720]  ? handle_root_syscall+0xd3/0x310
[   98.182730]  ? __cobalt_cond_wait_prologue+0x850/0x850
[   98.182741]  CoBaLt_cond_wait_prologue+0x1d/0x20
[   98.182751]  handle_head_syscall+0x106/0x3b0
[   98.182762]  ipipe_handle_syscall+0x66/0xc0
[   98.182772]  do_syscall_64+0x22/0x170
[   98.182782]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   98.182793] RIP: 0033:0x7f888d29575b
[   98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[   98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246 ORIG_RAX: 0000000010000023
[   98.182852] RAX: ffffffffffffffda RBX: 00007f888c56e6e0 RCX: 00007f888d29575b
[   98.183195] RDX: 00007f888c3baba4 RSI: 00007f888c56e6e0 RDI: 00007f888c56e7f8
[   98.183530] RBP: 00007f888c56e7f8 R08: 0000000000000000 R09: 00007f888c56e6d0
[   98.183854] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f888c56e6e0
[   98.184182] R13: 00007f888c3bad80 R14: 0000000000000001 R15: 00007f888c3babb0
[   98.184500] Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c arc4 ecb md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs ccm dns_resolver fscache ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter vmwgfx sg ttm drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect sysimgblt serio_raw snd_pcsp snd_pcm snd_timer snd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc evdev aesni_intel aes_x86_64 crypto_simd cryptd glue_helper soundcore ac button binfmt_misc sunrpc [   98.187446]  ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod ata_generic ata_piix crc32c_intel mptspi mptscsih mptbase scsi_transport_spi libata scsi_mod e1000 psmouse i2c_piix4

Quoted 96 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

We cannot rely on the lostage_wakeup work being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As the lostage_wakeup work is currently on the stack, it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].

Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using irq_work_sync in the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.

[1] https://lore.kernel.org/xenomai/[email protected]/T/#t

---
include/cobalt/kernel/thread.h | 6 ++++++
kernel/cobalt/thread.c | 15 +++++----------

2 files changed, 11 insertions(+), 10 deletions(-)
include/cobalt/kernel/thread.h +6 -0
index 7f9a37d29a..22ed61168c 100644
@@ -64,6 +64,11 @@ struct lostage_signal {
 	int signo, sigval;
 };
 
+struct lostage_wakeup {
+	struct pipeline_inband_work inband_work; /* Must be first. */
+	struct task_struct *task;
+};
+
 struct xnthread_init_attr {
 	struct xnthread_personality *personality;
 	cpumask_t affinity;
@@ -214,6 +219,7 @@ struct xnthread {
 	u32 proghash;		/* Hash value for exe_path */
 #endif
 	struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS];
+	struct lostage_wakeup relax_work;
 };
 
 static inline int xnthread_get_state(const struct xnthread *thread)
kernel/cobalt/thread.c +5 -10
index 8738fadfce..7dffb081bd 100644
@@ -1974,11 +1974,6 @@ int xnthread_harden(void)
 }
 EXPORT_SYMBOL_GPL(xnthread_harden);
 
-struct lostage_wakeup {
-	struct pipeline_inband_work inband_work; /* Must be first. */
-	struct task_struct *task;
-};
-
 static void lostage_task_wakeup(struct pipeline_inband_work *inband_work)
 {
 	struct lostage_wakeup *rq;
@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)
 void xnthread_relax(int notify, int reason)
 {
 	struct task_struct *p = current;
-	struct lostage_wakeup wakework = {
-		.inband_work = PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup),
-		.task = p,
-	};
 	struct xnthread *thread = xnthread_current();
 	int cpu __maybe_unused, suspension;
 	kernel_siginfo_t si;
@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)
 	 */
 	trace_cobalt_shadow_gorelax(reason);
 
+	thread->relax_work.inband_work = (struct pipeline_inband_work)
+		PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup);
+	thread->relax_work.task = p;
+
 	/*
 	 * If you intend to change the following interrupt-free
 	 * sequence, /first/ make sure to check the special handling
@@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)
 	 */
 	splmax();
 	trace_cobalt_lostage_request("wakeup", p);
-	pipeline_post_inband_work(&wakework);
+	pipeline_post_inband_work(&thread->relax_work);
 	/*
 	 * Grab the nklock to synchronize the Linux task state
 	 * manipulation with handle_sigwake_event. This lock will be
Signature · 1 line
2.43.0
Signature · 2 lines
Siemens AG, Technology
Linux Expert Center
author
#8 lore raw

Re: [PATCH 1/3] cobalt: Move relax related lostage_wakeup work from stack to thread

In reply to JerryJun
Quoted 7 lines · JerryJun

hi jan, in this patch you mentioned "As the lostage_wakeup work is currently on the stack,
 it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one ", why it‘s not triggered it in the switchtest benchmark which involves "xnthread_relax" due to linux syscall ? may be you probably describe the triggered scene detailed? Or how can I trigger it in switchtest
 by using xnthread_relax leaves earlier on the new CPU than the work was processed on the old one? thanks

The switchtest primarily tests switching between already existing tasks. I don't think we stress there the continuous creation and deletion of task there yet.

Clara wanted to look into improving the test coverage of that particular scenario as well. You two should probably coordinate who will work on what aspect.

Jan

Quoted 159 lines
Quoted 156 lines · Jan Kiszka
Quoted 8 lines · JerryJun

  hi Jan, the issue reported in [1] has seemly not been solved in ipipe.  Will this issue still be maintained on ipipe? the test case is available from Florian Bezdeka ([-- Attachment #2: xenomai-alchemy-app.tar.gz --]). linux kernel: 4.19.89
    xenomai: 3.2.4 + patch[1/3]

    [1] https://lore.kernel.org/xenomai/[email protected]/T/#t

Sorry, that fix is still only on master (upcoming 3.3), not yet in stable trees. It's still not expected to be I-pipe/dovetail-dependent.

Let me recover a bit more from yesterday's Preempt RT merge party, then I will close that gap.

Jan

Quoted 134 lines

[   98.182433] general protection fault: 0000 [#1] PREEMPT SMP NOPTI
[   98.182453] CPU: 1 PID: 69131 Comm: rt-task-20 Kdump: loaded Not tainted 4.19.89-ipipe-9-xenomai-3.2.4-proc-20240909 #12
[   98.182475] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[   98.182496] I-pipe domain: Linux
[   98.182507] RIP: 0010:xnsynch_sleep_on+0x15d/0x250
[   98.182517] Code: 30 12 00 00 3b 78 e8 7f 07 eb 4a 39 78 e8 7d 45 48 8b 86 50 12 00 00 48 8d b0 b8 ed ff ff 48 39 c8 75 e8 48 8b 86 48 12 00 00 <4c> 89 40 08 49 89 86 48 12 00 00 49 89 8e 50 12 00 00 4c 89 86 48
[   98.182552] RSP: 0018:ffffc90006327e30 EFLAGS: 00010046
[   98.182564] RAX: dead000000000100 RBX: 000000000005f3e0 RCX: ffffc900012d2a88
[   98.182577] RDX: 0000000000000000 RSI: ffffc900012d1840 RDI: 0000000000001032
[   98.182591] RBP: 0000000000000001 R08: ffffc900012d2a88 R09: ffffc900012d2a78
[   98.182605] R10: 0000000000000000 R11: 0000000000000000 R12: ffffc900012d3708
[   98.182620] R13: 0000000000000003 R14: ffffc900012d1840 R15: 0000000000000000
[   98.182634] FS:  00007f888c3bb700(0000) GS:ffff888133d00000(0000) knlGS:0000000000000000
[   98.182650] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   98.182661] CR2: 000055738facc000 CR3: 000000012ad44005 CR4: 00000000001606e0
[   98.182698] Call Trace:
[   98.182708]  __cobalt_cond_wait_prologue+0x26d/0x850
[   98.182720]  ? handle_root_syscall+0xd3/0x310
[   98.182730]  ? __cobalt_cond_wait_prologue+0x850/0x850
[   98.182741]  CoBaLt_cond_wait_prologue+0x1d/0x20
[   98.182751]  handle_head_syscall+0x106/0x3b0
[   98.182762]  ipipe_handle_syscall+0x66/0xc0
[   98.182772]  do_syscall_64+0x22/0x170
[   98.182782]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   98.182793] RIP: 0033:0x7f888d29575b
[   98.182801] Code: 8b 73 04 49 89 dc e8 54 f0 ff ff 48 8b 6c 24 10 b9 23 00 00 10 48 8d 54 24 44 45 31 c0 45 31 d2 48 89 de 89 c8 48 89 ef 0f 05 <8b> 7c 24 2c 31 f6 49 89 c5 89 c3 e8 25 f0 ff ff 4c 89 ff e8 7d e9
[   98.182837] RSP: 002b:00007f888c3bab60 EFLAGS: 00000246 ORIG_RAX: 0000000010000023
[   98.182852] RAX: ffffffffffffffda RBX: 00007f888c56e6e0 RCX: 00007f888d29575b
[   98.183195] RDX: 00007f888c3baba4 RSI: 00007f888c56e6e0 RDI: 00007f888c56e7f8
[   98.183530] RBP: 00007f888c56e7f8 R08: 0000000000000000 R09: 00007f888c56e6d0
[   98.183854] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f888c56e6e0
[   98.184182] R13: 00007f888c3bad80 R14: 0000000000000001 R15: 00007f888c3babb0
[   98.184500] Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT xt_tcpudp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c arc4 ecb md4 sha512_ssse3 sha512_generic cmac nls_utf8 cifs ccm dns_resolver fscache ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter vmwgfx sg ttm drm_kms_helper drm fb_sys_fops syscopyarea sysfillrect sysimgblt serio_raw snd_pcsp snd_pcm snd_timer snd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc evdev aesni_intel aes_x86_64 crypto_simd cryptd glue_helper soundcore ac button binfmt_misc sunrpc [   98.187446]  ip_tables x_tables autofs4 ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod ata_generic ata_piix crc32c_intel mptspi mptscsih mptbase scsi_transport_spi libata scsi_mod e1000 psmouse i2c_piix4

Quoted 96 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

We cannot rely on the lostage_wakeup work being scheduled only synchronously over the CPU of the relaxing thread. It thread might have a CPU migration and other wakeup events pending and resume on different CPU. As the lostage_wakeup work is currently on the stack, it can become corrupt when xnthread_relax leaves earlier on the new CPU than the work was processed on the old one - ugly crashes ahead as seen in [1].

Analogously to the signal wakeups, move the relax work into the xnthread structure to avoid such corruptions. We intentionally refrain from using irq_work_sync in the relaxation exit path to avoid starting to wait a CPU that is already busy again with a lengthy RT task.

[1] https://lore.kernel.org/xenomai/[email protected]/T/#t

---
include/cobalt/kernel/thread.h | 6 ++++++
kernel/cobalt/thread.c | 15 +++++----------

2 files changed, 11 insertions(+), 10 deletions(-)
include/cobalt/kernel/thread.h +6 -0
index 7f9a37d29a..22ed61168c 100644
@@ -64,6 +64,11 @@ struct lostage_signal {
 	int signo, sigval;
 };
 
+struct lostage_wakeup {
+	struct pipeline_inband_work inband_work; /* Must be first. */
+	struct task_struct *task;
+};
+
 struct xnthread_init_attr {
 	struct xnthread_personality *personality;
 	cpumask_t affinity;
@@ -214,6 +219,7 @@ struct xnthread {
 	u32 proghash;		/* Hash value for exe_path */
 #endif
 	struct lostage_signal sigarray[XNTHREAD_MAX_SIGNALS];
+	struct lostage_wakeup relax_work;
 };
 
 static inline int xnthread_get_state(const struct xnthread *thread)
kernel/cobalt/thread.c +5 -10
index 8738fadfce..7dffb081bd 100644
@@ -1974,11 +1974,6 @@ int xnthread_harden(void)
 }
 EXPORT_SYMBOL_GPL(xnthread_harden);
 
-struct lostage_wakeup {
-	struct pipeline_inband_work inband_work; /* Must be first. */
-	struct task_struct *task;
-};
-
 static void lostage_task_wakeup(struct pipeline_inband_work *inband_work)
 {
 	struct lostage_wakeup *rq;
@@ -2052,10 +2047,6 @@ void __xnthread_propagate_schedparam(struct xnthread *curr)
 void xnthread_relax(int notify, int reason)
 {
 	struct task_struct *p = current;
-	struct lostage_wakeup wakework = {
-		.inband_work = PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup),
-		.task = p,
-	};
 	struct xnthread *thread = xnthread_current();
 	int cpu __maybe_unused, suspension;
 	kernel_siginfo_t si;
@@ -2069,6 +2060,10 @@ void xnthread_relax(int notify, int reason)
 	 */
 	trace_cobalt_shadow_gorelax(reason);
 
+	thread->relax_work.inband_work = (struct pipeline_inband_work)
+		PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup);
+	thread->relax_work.task = p;
+
 	/*
 	 * If you intend to change the following interrupt-free
 	 * sequence, /first/ make sure to check the special handling
@@ -2081,7 +2076,7 @@ void xnthread_relax(int notify, int reason)
 	 */
 	splmax();
 	trace_cobalt_lostage_request("wakeup", p);
-	pipeline_post_inband_work(&wakework);
+	pipeline_post_inband_work(&thread->relax_work);
 	/*
 	 * Grab the nklock to synchronize the Linux task state
 	 * manipulation with handle_sigwake_event. This lock will be
Signature · 1 line
2.43.0
Signature · 2 lines
Siemens AG, Technology
Linux Expert Center
Signature · 2 lines
Siemens AG, Technology
Linux Expert Center
author
#9 lore raw

PATCH 2/3 cobalt: Drop obsolete pipeline_inband_work abstraction

In reply to Jan Kiszka

From: Jan Kiszka <[email protected]>

We are now dovetail-only and can use irq_work directly.

---
 .../kernel/dovetail/pipeline/inband_work.h    | 28 -------------------
 include/cobalt/kernel/rtdm/driver.h           |  9 +++---
 include/cobalt/kernel/thread.h                |  6 ++--
 kernel/cobalt/rtdm/drvlib.c                   | 15 ++++------
 kernel/cobalt/rtdm/fd.c                       | 13 ++++-----
 kernel/cobalt/thread.c                        | 24 +++++++---------
 kernel/drivers/udd/udd.c                      | 12 ++++----
 7 files changed, 33 insertions(+), 74 deletions(-)
 delete mode 100644 include/cobalt/kernel/dovetail/pipeline/inband_work.h
include/cobalt/kernel/dovetail/pipeline/inband_work.h deleted +0 -28
deleted file mode 100644
index a69bf651f0..0000000000
@@ -1,28 +0,0 @@
-/*
- * SPDX-License-Identifier: GPL-2.0
- *
- * Copyright (C) 2020 Philippe Gerum  <[email protected]>
- */
-
-#ifndef _COBALT_KERNEL_DOVETAIL_INBAND_WORK_H
-#define _COBALT_KERNEL_DOVETAIL_INBAND_WORK_H
-
-#include <linux/irq_work.h>
-
-/*
- * This field must be named inband_work and appear first in the
- * container work struct.
- */
-struct pipeline_inband_work {
-	struct irq_work work;
-};
-
-#define PIPELINE_INBAND_WORK_INITIALIZER(__handler)			\
-	{								\
-		.work = IRQ_WORK_INIT((void (*)(struct irq_work *))__handler), \
-	}
-
-#define pipeline_post_inband_work(__work)				\
-			irq_work_queue(&(__work)->inband_work.work)
-
-#endif /* !_COBALT_KERNEL_DOVETAIL_INBAND_WORK_H */
include/cobalt/kernel/rtdm/driver.h +4 -5
index 8fb326b1cf..604d54b2b7 100644
@@ -28,13 +28,13 @@
 
 #include <asm/atomic.h>
 #include <linux/cpumask.h>
+#include <linux/irq_work.h>
 #include <linux/list.h>
 #include <linux/module.h>
 #include <linux/cdev.h>
 #include <linux/wait.h>
 #include <linux/notifier.h>
 #include <pipeline/lock.h>
-#include <pipeline/inband_work.h>
 #include <xenomai/version.h>
 #include <cobalt/kernel/heap.h>
 #include <cobalt/kernel/sched.h>
@@ -909,7 +909,7 @@ typedef struct rtdm_nrtsig rtdm_nrtsig_t;
 typedef void (*rtdm_nrtsig_handler_t)(rtdm_nrtsig_t *nrt_sig, void *arg);
 
 struct rtdm_nrtsig {
-	struct pipeline_inband_work inband_work; /* Must be first */
+	struct irq_work inband_work;
 	rtdm_nrtsig_handler_t handler;
 	void *arg;
 };
@@ -918,13 +918,12 @@ void rtdm_schedule_nrt_work(struct work_struct *lostage_work);
 /** @} rtdm_nrtsignal */
 
 #ifndef DOXYGEN_CPP /* Avoid static inline tags for RTDM in doxygen */
-void __rtdm_nrtsig_execute(struct pipeline_inband_work *inband_work);
+void __rtdm_nrtsig_execute(struct irq_work *inband_work);
 
 static inline void rtdm_nrtsig_init(rtdm_nrtsig_t *nrt_sig,
 				    rtdm_nrtsig_handler_t handler, void *arg)
 {
-	nrt_sig->inband_work = (struct pipeline_inband_work)
-		PIPELINE_INBAND_WORK_INITIALIZER(__rtdm_nrtsig_execute);
+	nrt_sig->inband_work = IRQ_WORK_INIT(__rtdm_nrtsig_execute);
 	nrt_sig->handler = handler;
 	nrt_sig->arg = arg;
 }
include/cobalt/kernel/thread.h +3 -3
index 22ed61168c..04fad0e491 100644
@@ -19,11 +19,11 @@
 #ifndef _COBALT_KERNEL_THREAD_H
 #define _COBALT_KERNEL_THREAD_H
 
+#include <linux/irq_work.h>
 #include <linux/wait.h>
 #include <linux/sched.h>
 #include <linux/sched/rt.h>
 #include <pipeline/thread.h>
-#include <pipeline/inband_work.h>
 #include <cobalt/kernel/list.h>
 #include <cobalt/kernel/stat.h>
 #include <cobalt/kernel/timer.h>
@@ -59,13 +59,13 @@ struct xnthread_personality;
 struct completion;
 
 struct lostage_signal {
-	struct pipeline_inband_work inband_work; /* Must be first. */
+	struct irq_work inband_work;
 	struct task_struct *task;
 	int signo, sigval;
 };
 
 struct lostage_wakeup {
-	struct pipeline_inband_work inband_work; /* Must be first. */
+	struct irq_work inband_work;
 	struct task_struct *task;
 };
 
kernel/cobalt/rtdm/drvlib.c +5 -10
index e6e658c8c4..86788743fe 100644
@@ -31,7 +31,6 @@
 #include <linux/anon_inodes.h>
 #include <rtdm/driver.h>
 #include <rtdm/compat.h>
-#include <pipeline/inband_work.h>
 #include "internal.h"
 #include <trace/events/cobalt-rtdm.h>
 
@@ -1616,7 +1615,7 @@ int rtdm_nrtsig_init(rtdm_nrtsig_t *nrt_sig, rtdm_nrtsig_handler_t handler,
 void rtdm_nrtsig_destroy(rtdm_nrtsig_t *nrt_sig);
 #endif /* DOXYGEN_CPP */
 
-void __rtdm_nrtsig_execute(struct pipeline_inband_work *inband_work)
+void __rtdm_nrtsig_execute(struct irq_work *inband_work)
 {
 	struct rtdm_nrtsig *nrt_sig;
 
@@ -1634,14 +1633,14 @@ EXPORT_SYMBOL_GPL(__rtdm_nrtsig_execute);
  */
 void rtdm_nrtsig_pend(rtdm_nrtsig_t *nrt_sig)
 {
-	pipeline_post_inband_work(nrt_sig);
+	irq_work_queue(&nrt_sig->inband_work);
 }
 EXPORT_SYMBOL_GPL(rtdm_nrtsig_pend);
 
 static LIST_HEAD(nrt_work_list);
 DEFINE_PRIVATE_XNLOCK(nrt_work_lock);
 
-static void lostage_schedule_work(struct pipeline_inband_work *inband_work)
+static void lostage_schedule_work(struct irq_work *inband_work)
 {
 	struct work_struct *lostage_work;
 	spl_t s;
@@ -1663,11 +1662,7 @@ static void lostage_schedule_work(struct pipeline_inband_work *inband_work)
 	xnlock_put_irqrestore(&nrt_work_lock, s);
 }
 
-static struct lostage_trigger_work {
-	struct pipeline_inband_work inband_work; /* Must be first. */
-} nrt_work =  {
-	.inband_work = PIPELINE_INBAND_WORK_INITIALIZER(lostage_schedule_work),
-};
+static DEFINE_IRQ_WORK(nrt_work, lostage_schedule_work);
 
 /**
  * Put a work task in Linux non real-time global workqueue from primary mode.
@@ -1686,7 +1681,7 @@ void rtdm_schedule_nrt_work(struct work_struct *lostage_work)
 	xnlock_get_irqsave(&nrt_work_lock, s);
 
 	list_add_tail(&lostage_work->entry, &nrt_work_list);
-	pipeline_post_inband_work(&nrt_work);
+	irq_work_queue(&nrt_work);
 
 	xnlock_put_irqrestore(&nrt_work_lock, s);
 }
kernel/cobalt/rtdm/fd.c +5 -8
index 29555cb213..d5813a16c2 100644
@@ -17,6 +17,8 @@
  * along with this program; if not, write to the Free Software
  * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
  */
+
+#include <linux/irq_work.h>
 #include <linux/list.h>
 #include <linux/err.h>
 #include <linux/slab.h>
@@ -29,7 +31,6 @@
 #include <cobalt/kernel/lock.h>
 #include <cobalt/kernel/ppd.h>
 #include <cobalt/kernel/time.h>
-#include <pipeline/inband_work.h>
 #include <trace/events/cobalt-rtdm.h>
 #include <rtdm/compat.h>
 #include <rtdm/fd.h>
@@ -297,16 +298,12 @@ static int fd_cleanup_thread(void *data)
 	return 0;
 }
 
-static void lostage_trigger_close(struct pipeline_inband_work *inband_work)
+static void lostage_trigger_close(struct irq_work *inband_work)
 {
 	up(&rtdm_fd_cleanup_sem);
 }
 
-static struct lostage_trigger_close {
-	struct pipeline_inband_work inband_work; /* Must be first. */
-} fd_closework =  {
-	.inband_work = PIPELINE_INBAND_WORK_INITIALIZER(lostage_trigger_close),
-};
+static DEFINE_IRQ_WORK(fd_closework, lostage_trigger_close);
 
 static void __put_fd(struct rtdm_fd *fd, spl_t s)
 {
@@ -331,7 +328,7 @@ static void __put_fd(struct rtdm_fd *fd, spl_t s)
 		xnlock_put_irqrestore(&fdtree_lock, s);
 
 		if (trigger)
-			pipeline_post_inband_work(&fd_closework);
+			irq_work_queue(&fd_closework);
 	}
 }
 
kernel/cobalt/thread.c +10 -14
index 7dffb081bd..bdd0531404 100644
@@ -40,7 +40,6 @@
 #include <cobalt/kernel/lock.h>
 #include <cobalt/kernel/thread.h>
 #include <pipeline/kevents.h>
-#include <pipeline/inband_work.h>
 #include <pipeline/sched.h>
 #include <trace/events/cobalt-core.h>
 #include "debug.h"
@@ -86,12 +85,12 @@ static inline void enlist_new_thread(struct xnthread *thread)
 }
 
 struct kthread_arg {
-	struct pipeline_inband_work inband_work; /* Must be first. */
+	struct irq_work inband_work;
 	struct xnthread *thread;
 	struct completion *done;
 };
 
-static void do_parent_wakeup(struct pipeline_inband_work *inband_work)
+static void do_parent_wakeup(struct irq_work *inband_work)
 {
 	struct kthread_arg *ka;
 
@@ -150,9 +149,8 @@ static int map_kthread(struct xnthread *thread, struct kthread_arg *ka)
 
 	trace_cobalt_lostage_request("wakeup", current);
 
-	ka->inband_work = (struct pipeline_inband_work)
-		PIPELINE_INBAND_WORK_INITIALIZER(do_parent_wakeup);
-	pipeline_post_inband_work(ka);
+	ka->inband_work = IRQ_WORK_INIT(do_parent_wakeup);
+	irq_work_queue(&ka->inband_work);
 
 	xnlock_get_irqsave(&nklock, s);
 
@@ -1974,7 +1972,7 @@ int xnthread_harden(void)
 }
 EXPORT_SYMBOL_GPL(xnthread_harden);
 
-static void lostage_task_wakeup(struct pipeline_inband_work *inband_work)
+static void lostage_task_wakeup(struct irq_work *inband_work)
 {
 	struct lostage_wakeup *rq;
 	struct task_struct *p;
@@ -2060,8 +2058,7 @@ void xnthread_relax(int notify, int reason)
 	 */
 	trace_cobalt_shadow_gorelax(reason);
 
-	thread->relax_work.inband_work = (struct pipeline_inband_work)
-		PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_wakeup);
+	thread->relax_work.inband_work = IRQ_WORK_INIT(lostage_task_wakeup);
 	thread->relax_work.task = p;
 
 	/*
@@ -2076,7 +2073,7 @@ void xnthread_relax(int notify, int reason)
 	 */
 	splmax();
 	trace_cobalt_lostage_request("wakeup", p);
-	pipeline_post_inband_work(&thread->relax_work);
+	irq_work_queue(&thread->relax_work.inband_work);
 	/*
 	 * Grab the nklock to synchronize the Linux task state
 	 * manipulation with handle_sigwake_event. This lock will be
@@ -2154,7 +2151,7 @@ void xnthread_relax(int notify, int reason)
 }
 EXPORT_SYMBOL_GPL(xnthread_relax);
 
-static void lostage_task_signal(struct pipeline_inband_work *inband_work)
+static void lostage_task_signal(struct irq_work *inband_work)
 {
 	struct lostage_signal *rq;
 	struct task_struct *p;
@@ -2387,15 +2384,14 @@ void __xnthread_signal(struct xnthread *thread, int sig, int arg)
 	if (sigwork->task)
 		return;
 
-	sigwork->inband_work = (struct pipeline_inband_work)
-			PIPELINE_INBAND_WORK_INITIALIZER(lostage_task_signal);
+	sigwork->inband_work = IRQ_WORK_INIT(lostage_task_signal);
 	sigwork->task = xnthread_host_task(thread);
 	sigwork->signo = sig;
 	sigwork->sigval = sig == SIGDEBUG ? arg | sigdebug_marker : arg;
 
 	trace_cobalt_lostage_request("signal", sigwork->task);
 
-	pipeline_post_inband_work(sigwork);
+	irq_work_queue(&sigwork->inband_work);
 }
 
 void xnthread_signal(struct xnthread *thread, int sig, int arg)
kernel/drivers/udd/udd.c +6 -6
index b1efcebd0f..c7b8f5c7bf 100644
@@ -17,6 +17,8 @@
  * along with this program; if not, write to the Free Software
  * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
  */
+
+#include <linux/irq_work.h>
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/slab.h>
@@ -24,7 +26,6 @@
 #include <rtdm/cobalt.h>
 #include <rtdm/driver.h>
 #include <rtdm/udd.h>
-#include <pipeline/inband_work.h>
 
 struct udd_context {
 	u32 event_count;
@@ -516,13 +517,13 @@ void udd_notify_event(struct udd_device *udd)
 EXPORT_SYMBOL_GPL(udd_notify_event);
 
 struct irqswitch_work {
-	struct pipeline_inband_work inband_work;
+	struct irq_work inband_work;
 	rtdm_irq_t *irqh;
 	int enabled;
 	rtdm_event_t *done;
 };
 
-static void lostage_irqswitch_line(struct pipeline_inband_work *inband_work)
+static void lostage_irqswitch_line(struct irq_work *inband_work)
 {
 	struct irqswitch_work *rq;
 
@@ -550,8 +551,7 @@ static void switch_irq_line(rtdm_irq_t *irqh, int enable, rtdm_event_t *done)
 	if (WARN_ON(rq == NULL))
 		return;
 
-	rq->inband_work = (struct pipeline_inband_work)
-		PIPELINE_INBAND_WORK_INITIALIZER(lostage_irqswitch_line);
+	rq->inband_work = IRQ_WORK_INIT(lostage_irqswitch_line);
 	rq->irqh = irqh;
 	rq->enabled = enable;
 	rq->done = done;
@@ -561,7 +561,7 @@ static void switch_irq_line(rtdm_irq_t *irqh, int enable, rtdm_event_t *done)
 	 * enabling/disabling IRQ lines from primary mode. Defer this
 	 * to the root context.
 	 */
-	pipeline_post_inband_work(rq);
+	irq_work_queue(&rq->inband_work);
 }
 
 /**
Signature · 1 line
2.43.0
author
#10 lore raw

PATCH 3/3 cobalt/posix/process: Sync on thread-hosted irq_work items before exiting

In reply to Jan Kiszka

From: Jan Kiszka <[email protected]>

Better safe than sorry: Due to migrations, there could theoretically still be signal or relaxation work pending on different CPU. To avoid reaping the carpet underneath of them, sync on all per-thread work_irq items when a former RT thread is terminating.

---
 kernel/cobalt/posix/process.c | 5 +++++
 1 file changed, 5 insertions(+)
kernel/cobalt/posix/process.c +5 -0
index 7cfaa723e6..8b4efcab40 100644
@@ -838,6 +838,7 @@ static void __handle_taskexit_event(struct task_struct *p)
 {
 	struct cobalt_ppd *sys_ppd;
 	struct xnthread *thread;
+	unsigned int n;
 	spl_t s;
 
 	/*
@@ -861,6 +862,10 @@ static void __handle_taskexit_event(struct task_struct *p)
 
 	xnthread_run_handler_stack(thread, exit_thread);
 
+	for (n = 0; n < XNTHREAD_MAX_SIGNALS; n++)
+		irq_work_sync(&thread->sigarray[n].inband_work);
+	irq_work_sync(&thread->relax_work.inband_work);
+
 	if (xnthread_test_state(thread, XNUSER)) {
 		cobalt_umm_free(&cobalt_kernel_ppd.umm, thread->u_window);
 		thread->u_window = NULL;
Signature · 1 line
2.43.0
Tested-by on the v1 series
#11 lore raw
In reply to Jan Kiszka
Quoted 5 lines · Jan Kiszka

This should resolve the issue reported in [1] and also clean up a bit more of our now obsolete pipeline abstraction. Test case is planned for later.

Jan

I can confirm that the reproducer for [1] does no longer crash.

Quoted 22 lines

[1] https://lore.kernel.org/xenomai/[email protected]/T/#t

Jan Kiszka (3):
cobalt: Move relax related lostage_wakeup work from stack to thread
cobalt: Drop obsolete pipeline_inband_work abstraction cobalt/posix/process: Sync on thread-hosted irq_work items before
exiting

 .../kernel/dovetail/pipeline/inband_work.h    | 28 ----------------
 include/cobalt/kernel/rtdm/driver.h           |  9 +++--
 include/cobalt/kernel/thread.h                | 10 ++++--
 kernel/cobalt/posix/process.c                 |  5 +++
 kernel/cobalt/rtdm/drvlib.c                   | 15 +++------
 kernel/cobalt/rtdm/fd.c                       | 13 +++-----
 kernel/cobalt/thread.c                        | 33 +++++++------------
 kernel/drivers/udd/udd.c                      | 12 +++----
 8 files changed, 45 insertions(+), 80 deletions(-)
 delete mode 100644 include/cobalt/kernel/dovetail/pipeline/inband_work.h
Signature · 1 line
2.43.0