genirq: Fix IRQ threads VS cpuset

When a cpuset isolated partition is created / updated or destroyed, the IRQ threads are affine blindly to all the non-isolated CPUs. And this happens without taking into account the IRQ thread initial affinity that becomes ignored.

For example in a system with 8 CPUs, if an IRQ and its kthread are initially affine to CPU 5, creating an isolated partition with only CPU 2 inside will eventually end up affining the IRQ kthread to all CPUs but CPU 2 (that is CPUs 0,1,3-7), losing the kthread preference for CPU 5.

Besides the blind re-affinity, this doesn't take care of the actual low level interrupt which isn't migrated. As of today the only way to isolate non managed interrupts, along with their kthreads, is to overwrite their affinity separately, for example through /proc/irq/

To avoid doing that manually, future development should focus on updating the IRQs affinity whenever cpuset isolated partitions are updated.

In the meantime, cpuset shouldn't fiddle with IRQ threads directly. To prevent from that, set the PF_NO_SETAFFINITY flag to them.

Signed-off-by Frederic Weisbecker <[email protected]>

---
  kernel/irq/manage.c | 26 +++++++++++++-------------
  1 file changed, 13 insertions(+), 13 deletions(-)

kernel/irq/manage.c +3 -3

index 400856abf672..76e2cbe21d1f 100644
@@ -176,7 +176,7 @@ bool irq_can_set_affinity_usr(unsigned int irq)
  }
 
  /**
- * irq_set_thread_affinity - Notify irq threads to adjust affinity
+ * irq_thread_update_affinity - Notify irq threads to adjust affinity
   * @desc:	irq descriptor which has affinity changed
   *
   * Just set IRQTF_AFFINITY and delegate the affinity setting to the

@@ -184,7 +184,7 @@ bool irq_can_set_affinity_usr(unsigned int irq)
   * we hold desc->lock and this code can be called from hard interrupt
   * context.
   */
-static void irq_set_thread_affinity(struct irq_desc *desc)
+static void irq_thread_update_affinity(struct irq_desc *desc)
  {
  	struct irqaction *action;
 

@@ -283,7 +283,7 @@ int irq_do_set_affinity(struct irq_data *data, const struct cpumask *mask,
  		fallthrough;
  	case IRQ_SET_MASK_OK_NOCOPY:
  		irq_validate_effective_affinity(data);
-		irq_set_thread_affinity(desc);
+		irq_thread_update_affinity(desc);
  		ret = 0;
  	}
 

kernel/irq/manage.c +4 -10

  #else
  static inline void irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *action) { }
+static inline void irq_thread_set_affinity(struct task_struct *t,
+					   struct irq_desc *desc) { }
  #endif
 
  static int irq_wait_for_interrupt(struct irq_desc *desc,

@@ -1221,6 +1229,7 @@ static void wake_up_and_wait_for_irq_thread_ready(struct irq_desc *desc,
  	if (!action || !action->thread)
  		return;
 
+	irq_thread_set_affinity(action->thread, desc);
  	wake_up_process(action->thread);
  	wait_event(desc->wait_for_threads,
  		   test_bit(IRQTF_READY, &action->thread_flags));

@@ -1405,16 +1414,7 @@ setup_irq_thread(struct irqaction *new, unsigned int irq, bool secondary)
  	 * references an already freed task_struct.
  	 */
  	new->thread = get_task_struct(t);
-	/*
-	 * Tell the thread to set its affinity. This is
-	 * important for shared interrupt handlers as we do
-	 * not invoke setup_affinity() for the secondary
-	 * handlers as everything is already set up. Even for
-	 * interrupts marked with IRQF_NO_BALANCE this is
-	 * correct as we want the thread to move to the cpu(s)
-	 * on which the requesting code placed the interrupt.
-	 */
-	set_bit(IRQTF_AFFINITY, &new->thread_flags);
+
  	return 0;
  }
 

kernel/irq/manage.c static void irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *a · line 1043

Quoted 5 lines

kernel/irq/manage.c +5 -0

+static inline void irq_thread_set_affinity(struct task_struct *t,
+					   struct irq_desc *desc)
+{
+	kthread_bind_mask(t, irq_data_get_effective_affinity_mask(&desc->irq_data));
+}

According to irq_thread_check_affinity(), accessing the cpumask returned from irq_data_get_effective_affinity_mask(&desc->irq_data) requires taking desc->lock to ensure its stability. Do we need something similar here? Other than that, it looks good to me.

Quoted 9 lines · Waiman Long

kernel/irq/manage.c static void irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *a · line 1043

Quoted 5 lines

kernel/irq/manage.c +5 -0

+static inline void irq_thread_set_affinity(struct task_struct *t,
+					   struct irq_desc *desc)
+{
+	kthread_bind_mask(t, irq_data_get_effective_affinity_mask(&desc->irq_data));
+}

According to irq_thread_check_affinity(), accessing the cpumask returned from irq_data_get_effective_affinity_mask(&desc->irq_data) requires taking desc->lock to ensure its stability. Do we need something similar here? Other than that, it looks good to me.

That's during interrupt setup so it should be stable (famous last words)

When a cpuset isolated partition is created / updated or destroyed, the IRQ threads are affine blindly to all the non-isolated CPUs. And this happens without taking into account the IRQ thread initial affinity that becomes ignored.

For example in a system with 8 CPUs, if an IRQ and its kthread are initially affine to CPU 5, creating an isolated partition with only CPU 2 inside will eventually end up affining the IRQ kthread to all CPUs but CPU 2 (that is CPUs 0,1,3-7), losing the kthread preference for CPU 5.

Besides the blind re-affinity, this doesn't take care of the actual low level interrupt which isn't migrated. As of today the only way to isolate non managed interrupts, along with their kthreads, is to overwrite their affinity separately, for example through /proc/irq/

To avoid doing that manually, future development should focus on updating the IRQs affinity whenever cpuset isolated partitions are updated.

In the meantime, cpuset shouldn't fiddle with IRQ threads directly. To prevent from that, set the PF_NO_SETAFFINITY flag to them.

Signed-off-by Frederic Weisbecker <[email protected]>

---
  kernel/irq/manage.c | 26 +++++++++++++-------------
  1 file changed, 13 insertions(+), 13 deletions(-)

kernel/irq/manage.c +12 -7

index 400856abf672..76e2cbe21d1f 100644
@@ -176,7 +176,7 @@ bool irq_can_set_affinity_usr(unsigned int irq)
  }
 
  /**
- * irq_set_thread_affinity - Notify irq threads to adjust affinity
+ * irq_thread_update_affinity - Notify irq threads to adjust affinity
   * @desc:	irq descriptor which has affinity changed
   *
   * Just set IRQTF_AFFINITY and delegate the affinity setting to the

@@ -184,7 +184,7 @@ bool irq_can_set_affinity_usr(unsigned int irq)
   * we hold desc->lock and this code can be called from hard interrupt
   * context.
   */
-static void irq_set_thread_affinity(struct irq_desc *desc)
+static void irq_thread_update_affinity(struct irq_desc *desc)
  {
  	struct irqaction *action;
 

@@ -283,7 +283,7 @@ int irq_do_set_affinity(struct irq_data *data, const struct cpumask *mask,
  		fallthrough;
  	case IRQ_SET_MASK_OK_NOCOPY:
  		irq_validate_effective_affinity(data);
-		irq_set_thread_affinity(desc);
+		irq_thread_update_affinity(desc);
  		ret = 0;
  	}
 

@@ -1035,8 +1035,16 @@ static void irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *a
  		set_cpus_allowed_ptr(current, mask);
  	free_cpumask_var(mask);
  }
+
+static inline void irq_thread_set_affinity(struct task_struct *t,
+					   struct irq_desc *desc)
+{
+	kthread_bind_mask(t, irq_data_get_effective_affinity_mask(&desc->irq_data));
+}
  #else
  static inline void irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *action) { }
+static inline void irq_thread_set_affinity(struct task_struct *t,
+					   struct irq_desc *desc) { }
  #endif
 
  static int irq_wait_for_interrupt(struct irq_desc *desc,

@@ -1221,6 +1229,7 @@ static void wake_up_and_wait_for_irq_thread_ready(struct irq_desc *desc,
  	if (!action || !action->thread)
  		return;
 
+	irq_thread_set_affinity(action->thread, desc);
  	wake_up_process(action->thread);
  	wait_event(desc->wait_for_threads,
  		   test_bit(IRQTF_READY, &action->thread_flags));

@@ -1405,16 +1414,7 @@ setup_irq_thread(struct irqaction *new, unsigned int irq, bool secondary)
  	 * references an already freed task_struct.
  	 */
  	new->thread = get_task_struct(t);
-	/*
-	 * Tell the thread to set its affinity. This is
-	 * important for shared interrupt handlers as we do
-	 * not invoke setup_affinity() for the secondary

Quoted 16 lines · Frederic Weisbecker

When a cpuset isolated partition is created / updated or destroyed, the IRQ threads are affine blindly to all the non-isolated CPUs. And this happens without taking into account the IRQ thread initial affinity that becomes ignored.

For example in a system with 8 CPUs, if an IRQ and its kthread are initially affine to CPU 5, creating an isolated partition with only CPU 2 inside will eventually end up affining the IRQ kthread to all CPUs but CPU 2 (that is CPUs 0,1,3-7), losing the kthread preference for CPU 5.

Besides the blind re-affinity, this doesn't take care of the actual low level interrupt which isn't migrated. As of today the only way to isolate non managed interrupts, along with their kthreads, is to overwrite their affinity separately, for example through /proc/irq/

Quoted 8 lines

To avoid doing that manually, future development should focus on updating the IRQs affinity whenever cpuset isolated partitions are updated.

In the meantime, cpuset shouldn't fiddle with IRQ threads directly. To prevent from that, set the PF_NO_SETAFFINITY flag to them.

Signed-off-by Frederic Weisbecker <[email protected]>

This patch landed in today's linux-next as commit 844dcacab287 ("genirq:
Fix interrupt threads affinity vs. cpuset isolated partitions"). In my tests I found that it triggers a warnings on some of my test systems. This is example of such warning:

------------[ cut here ]------------
WARNING: CPU: 0 PID: 1 at kernel/kthread.c:599 kthread_bind_mask+0x2c/0x84

Quoted 26 lines · Marek Szyprowski

Quoted 16 lines · Frederic Weisbecker

When a cpuset isolated partition is created / updated or destroyed, the IRQ threads are affine blindly to all the non-isolated CPUs. And this happens without taking into account the IRQ thread initial affinity that becomes ignored.

For example in a system with 8 CPUs, if an IRQ and its kthread are initially affine to CPU 5, creating an isolated partition with only CPU 2 inside will eventually end up affining the IRQ kthread to all CPUs but CPU 2 (that is CPUs 0,1,3-7), losing the kthread preference for CPU 5.

Besides the blind re-affinity, this doesn't take care of the actual low level interrupt which isn't migrated. As of today the only way to isolate non managed interrupts, along with their kthreads, is to overwrite their affinity separately, for example through /proc/irq/

Quoted 8 lines

To avoid doing that manually, future development should focus on updating the IRQs affinity whenever cpuset isolated partitions are updated.

In the meantime, cpuset shouldn't fiddle with IRQ threads directly. To prevent from that, set the PF_NO_SETAFFINITY flag to them.

Signed-off-by Frederic Weisbecker <[email protected]>

Quoted 7 lines

This patch landed in today's linux-next as commit 844dcacab287 ("genirq:
Fix interrupt threads affinity vs. cpuset isolated partitions"). In my tests I found that it triggers a warnings on some of my test systems. This is example of such warning:

------------[ cut here ]------------
WARNING: CPU: 0 PID: 1 at kernel/kthread.c:599 kthread_bind_mask+0x2c/0x84

Erm, does this means that the IRQ thread got awaken before the first official wakeup in wake_up_and_wait_for_irq_thread_ready()? This looks wrong...

irq_startup() may be called on a few occcasions before. So perhaps the IRQ already fired and woke up the kthread once before the "official" first wake up?

There seem to be some initialization ordering issue here...

Quoted 8 lines · Frederic Weisbecker

Erm, does this means that the IRQ thread got awaken before the first official wakeup in wake_up_and_wait_for_irq_thread_ready()? This looks wrong...

irq_startup() may be called on a few occcasions before. So perhaps the IRQ already fired and woke up the kthread once before the "official" first wake up?

There seem to be some initialization ordering issue here...

That's unavoidable because of locking and hen and egg ordering problems.

When the thread is created the interrupt is not yet started up and therefore effective affinity is not known. So doing a bind there is pointless.

The action has to be made visible _before_ starting it up as once the interrupt is unmasked it might fire. That's even more true for shared interrupts.

The wakeup/bind/... muck cannot be invoked with the descriptor lock held, so it has to be done _after_ the thing went live.

So yes an interrupt which fires before kthread_bind() is invoked might exactly have that effect. It wakes it from the kthread() wait and it goes straight through to the thread function.

That's why the original code just set the affinity bit and let the thread itself handle it.

There are three options to solve that:

1) Bind the thread right after kthread_create() to a random housekeeping task and let move itself over to the real place once it came up (at this point the affinity is established)

2) Serialize the setup so that the thread (even, when woken up early) get's stuck in UNINTERRUPTIBLE state _before_ it can reach wait_for_interrupt() which waits with INTERRUPTIBLE

3) Teach kthread_create() that the thread is subject to being bound to something later on and implement that serialization there.

#1 is pretty straight forward. See untested below.

#2 is ugly (I tried)

#3 could be useful in general, but that needs more thoughts

kernel/irq/handle.c +9 -1

@@ -133,7 +133,15 @@ void __irq_wake_thread(struct irq_desc *
 	 */
 	atomic_inc(&desc->threads_active);
 
-	wake_up_process(action->thread);
+	/*
+	 * This might be a premature wakeup before the thread reached the
+	 * thread function and set the IRQTF_READY bit. It's waiting in
+	 * kthread code with state UNINTERRUPTIBLE. Once it reaches the
+	 * thread function it waits with INTERRUPTIBLE. The wakeup is not
+	 * lost in that case because the thread is guaranteed to observe
+	 * the RUN flag before it goes to sleep in wait_for_interrupt().
+	 */
+	wake_up_state(action->thread, TASK_INTERRUPTIBLE);
 }
 
 static DEFINE_STATIC_KEY_FALSE(irqhandler_duration_check_enabled);

kernel/irq/manage.c +7 -11

@@ -1026,16 +1026,8 @@ static void irq_thread_check_affinity(st
 	set_cpus_allowed_ptr(current, mask);
 	free_cpumask_var(mask);
 }
-
-static inline void irq_thread_set_affinity(struct task_struct *t,
-					   struct irq_desc *desc)
-{
-	kthread_bind_mask(t, irq_data_get_effective_affinity_mask(&desc->irq_data));
-}
 #else
 static inline void irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *action) { }
-static inline void irq_thread_set_affinity(struct task_struct *t,
-					   struct irq_desc *desc) { }
 #endif
 
 static int irq_wait_for_interrupt(struct irq_desc *desc,

@@ -1220,7 +1212,6 @@ static void wake_up_and_wait_for_irq_thr
 	if (!action || !action->thread)
 		return;
 
-	irq_thread_set_affinity(action->thread, desc);
 	wake_up_process(action->thread);
 	wait_event(desc->wait_for_threads,
 		   test_bit(IRQTF_READY, &action->thread_flags));

@@ -1389,26 +1380,35 @@ static void irq_nmi_teardown(struct irq_
 static int
 setup_irq_thread(struct irqaction *new, unsigned int irq, bool secondary)
 {
+	/*
+	 * At this point interrupt affinity is not known. just assume that
+	 * the current CPU is not isolated and valid to bring the thread
+	 * up. The thread will move itself over to the right place once the
+	 * whole setup is complete.
+	 */
+	unsigned int cpu = raw_smp_processor_id();
 	struct task_struct *t;
 
-	if (!secondary) {
-		t = kthread_create(irq_thread, new, "irq/%d-%s", irq,

kernel/irq/manage.c static void irq_nmi_teardown(struct irq_

Quoted 5 lines · Thomas Gleixner

kernel/irq/manage.c +4 -1

+	if (!secondary)
+		t = kthread_create_on_cpu(irq_thread, new, cpu, "irq/%d-%s", irq, new->name);
+	else
+		t = kthread_create_on_cpu(irq_thread, new, cpu, "irq/%d-s-%s",
-		irq, new->name);

Right I though about something like that, it involved:

kthread_bind_mask(t, cpu_possible_mask);

Which do you prefer? Also do you prefer such a fixup or should I refactor my patches you merged?

Quoted 33 lines · Frederic Weisbecker

Quoted 24 lines · Marek Szyprowski

Quoted 23 lines · Frederic Weisbecker

When a cpuset isolated partition is created / updated or destroyed, the IRQ threads are affine blindly to all the non-isolated CPUs. And this happens without taking into account the IRQ thread initial affinity that becomes ignored.

For example in a system with 8 CPUs, if an IRQ and its kthread are initially affine to CPU 5, creating an isolated partition with only CPU 2 inside will eventually end up affining the IRQ kthread to all CPUs but CPU 2 (that is CPUs 0,1,3-7), losing the kthread preference for CPU 5.

Besides the blind re-affinity, this doesn't take care of the actual low level interrupt which isn't migrated. As of today the only way to isolate non managed interrupts, along with their kthreads, is to overwrite their affinity separately, for example through /proc/irq/

To avoid doing that manually, future development should focus on updating the IRQs affinity whenever cpuset isolated partitions are updated.

In the meantime, cpuset shouldn't fiddle with IRQ threads directly. To prevent from that, set the PF_NO_SETAFFINITY flag to them.

Quoted 8 lines

Quoted 1 line

Signed-off-by Frederic Weisbecker <[email protected]>

This patch landed in today's linux-next as commit 844dcacab287 ("genirq:
Fix interrupt threads affinity vs. cpuset isolated partitions"). In my tests I found that it triggers a warnings on some of my test systems. This is example of such warning:

------------[ cut here ]------------
WARNING: CPU: 0 PID: 1 at kernel/kthread.c:599 kthread_bind_mask+0x2c/0x84

Quoted 8 lines

Erm, does this means that the IRQ thread got awaken before the first official wakeup in wake_up_and_wait_for_irq_thread_ready()? This looks wrong...

irq_startup() may be called on a few occcasions before. So perhaps the IRQ already fired and woke up the kthread once before the "official" first wake up?

There seem to be some initialization ordering issue here...

That's unavoidable because of locking and hen and egg ordering problems.

When the thread is created the interrupt is not yet started up and therefore effective affinity is not known. So doing a bind there is pointless.

The action has to be made visible _before_ starting it up as once the interrupt is unmasked it might fire. That's even more true for shared interrupts.

The wakeup/bind/... muck cannot be invoked with the descriptor lock held, so it has to be done _after_ the thing went live.

So yes an interrupt which fires before kthread_bind() is invoked might exactly have that effect. It wakes it from the kthread() wait and it goes straight through to the thread function.

That's why the original code just set the affinity bit and let the thread itself handle it.

There are three options to solve that:

1) Bind the thread right after kthread_create() to a random housekeeping task and let move itself over to the real place once it came up (at this point the affinity is established)

2) Serialize the setup so that the thread (even, when woken up early) get's stuck in UNINTERRUPTIBLE state _before_ it can reach wait_for_interrupt() which waits with INTERRUPTIBLE

3) Teach kthread_create() that the thread is subject to being bound to something later on and implement that serialization there.

#1 is pretty straight forward. See untested below.

#2 is ugly (I tried)

#3 could be useful in general, but that needs more thoughts

Thanks

tglx
---

kernel/irq/handle.c +9 -1

@@ -133,7 +133,15 @@ void __irq_wake_thread(struct irq_desc *
  	 */
  	atomic_inc(&desc->threads_active);
 
-	wake_up_process(action->thread);
+	/*
+	 * This might be a premature wakeup before the thread reached the
+	 * thread function and set the IRQTF_READY bit. It's waiting in
+	 * kthread code with state UNINTERRUPTIBLE. Once it reaches the
+	 * thread function it waits with INTERRUPTIBLE. The wakeup is not
+	 * lost in that case because the thread is guaranteed to observe
+	 * the RUN flag before it goes to sleep in wait_for_interrupt().
+	 */
+	wake_up_state(action->thread, TASK_INTERRUPTIBLE);
  }
 
  static DEFINE_STATIC_KEY_FALSE(irqhandler_duration_check_enabled);

kernel/irq/manage.c +14 -19

@@ -1026,16 +1026,8 @@ static void irq_thread_check_affinity(st
  	set_cpus_allowed_ptr(current, mask);
  	free_cpumask_var(mask);
  }
-
-static inline void irq_thread_set_affinity(struct task_struct *t,
-					   struct irq_desc *desc)
-{
-	kthread_bind_mask(t, irq_data_get_effective_affinity_mask(&desc->irq_data));
-}
  #else
  static inline void irq_thread_check_affinity(struct irq_desc *desc, struct irqaction *action) { }
-static inline void irq_thread_set_affinity(struct task_struct *t,
-					   struct irq_desc *desc) { }
  #endif
 
  static int irq_wait_for_interrupt(struct irq_desc *desc,

@@ -1220,7 +1212,6 @@ static void wake_up_and_wait_for_irq_thr
  	if (!action || !action->thread)
  		return;
 
-	irq_thread_set_affinity(action->thread, desc);
  	wake_up_process(action->thread);
  	wait_event(desc->wait_for_threads,
  		   test_bit(IRQTF_READY, &action->thread_flags));

@@ -1389,26 +1380,35 @@ static void irq_nmi_teardown(struct irq_
  static int
  setup_irq_thread(struct irqaction *new, unsigned int irq, bool secondary)
  {
+	/*
+	 * At this point interrupt affinity is not known. just assume that
+	 * the current CPU is not isolated and valid to bring the thread
+	 * up. The thread will move itself over to the right place once the
+	 * whole setup is complete.
+	 */
+	unsigned int cpu = raw_smp_processor_id();
  	struct task_struct *t;
 
-	if (!secondary) {
-		t = kthread_create(irq_thread, new, "irq/%d-%s", irq,
-				   new->name);
-	} else {
-		t = kthread_create(irq_thread, new, "irq/%d-s-%s", irq,
-				   new->name);
-	}
+	if (!secondary)
+		t = kthread_create_on_cpu(irq_thread, new, cpu, "irq/%d-%s", irq, new->name);
+	else
+		t = kthread_create_on_cpu(irq_thread, new, cpu, "irq/%d-s-%s", irq, new->name);
 
  	if (IS_ERR(t))
  		return PTR_ERR(t);
 
  	/*
-	 * We keep the reference to the task struct even if
-	 * the thread dies to avoid that the interrupt code
-	 * references an already freed task_struct.
+	 * We keep the reference to the task struct even if the thread dies
+	 * to avoid that the interrupt code references an already freed
+	 * task_struct.
  	 */

Quoted 4 lines · Frederic Weisbecker

In the meantime, cpuset shouldn't fiddle with IRQ threads directly. To prevent from that, set the PF_NO_SETAFFINITY flag to them.

Signed-off-by Frederic Weisbecker <[email protected]>

This patch landed in today's linux-next as commit 844dcacab287 ("genirq:
Fix interrupt threads affinity vs. cpuset isolated partitions"). In my tests I found that it triggers a warnings on some of my test systems. This is example of such warning:

------------[ cut here ]------------
WARNING: CPU: 0 PID: 1 at kernel/kthread.c:599 kthread_bind_mask+0x2c/0x84
Modules linked in:
CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 
6.18.0-rc1-00031-g844dcacab287 #16177 PREEMPT
Hardware name: Samsung Exynos (Flattened Device Tree)
Call trace:
  unwind_backtrace from show_stack+0x10/0x14
  show_stack from dump_stack_lvl+0x68/0x88
  dump_stack_lvl from __warn+0x80/0x1d0
  __warn from warn_slowpath_fmt+0x1b0/0x1bc
  warn_slowpath_fmt from kthread_bind_mask+0x2c/0x84
  kthread_bind_mask from wake_up_and_wait_for_irq_thread_ready+0x3c/0xd4
  wake_up_and_wait_for_irq_thread_ready from __setup_irq+0x3e8/0x894

See here for previous patch:

	https://lore.kernel.org/lkml/[email protected]/

Quoted 3 lines · Frederic Weisbecker

See here for previous patch:

	https://lore.kernel.org/lkml/[email protected]/

It would be really useful if you would have 'PATCH V2' in the subject lines of the series so it's easy distinguishable for humans and tools.