v4.16.18-rt11

Dear RT folks!

I'm pleased to announce the v4.16.18-rt11 patch set.

Changes since v4.16.18-rt10:

- Finally fix the SIMD locking on arm64.

- Revert the srcu based notifier in crypto code. This is no longer required since the rwsem on RT allows multiple reader.

- Add a per-CPU lock in crypto's cryptd per-CPU queue. Reported by Mike Galbraith.

- Add a per-CPU lock in crypto's scompress code. Patch by Mike Galbraith.

- Disable runtime EFI services on ARM64. It can be enabled again via command line. The getTime function needs around 10ms to complete on one of my machines. The setVariable might take even longer. It is disabled by default, it can be enabled via `efi=runtime' on the command line.

Known issues
- A warning triggered in "rcu_note_context_switch" originated from SyS_timer_gettime(). The issue was always there, it is now visible. Reported by Grygorii Strashko and Daniel Wagner.

The delta patch against v4.16.18-rt10 is appended below and can be found here:

     https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.16/incr/patch-4.16.18-rt10-rt11.patch.xz

You can get this release via the git tree at:

    git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git v4.16.18-rt11

The RT patch against v4.16.18 can be found here:

    https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.16/older/patch-4.16.18-rt11.patch.xz

The split quilt queue is available at:

    https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.16/older/patches-4.16.18-rt11.tar.xz

Sebastian

1. arm64/acpi/perf: move pmu allocation to an early CPU up hook
2. sched: Introduce raw_cond_resched_lock()
3. arm, KVM: convert vgic_irq.irq_lock to raw_spinlock_t

With these applied, 4 socket TaiShan 2280 box boots shiny new -rt11 gripe free, and has been tossed into SUSE's kvm build-bot slave pit, where it is presumably performing acceptably, given its boss keeps giving it more work to do. (I see only lack of smoke/flame)

1 should fly, 2 and 3 may well die.. as box does without them.

-Mike

RT cannot allocate while irqs are disabled.

  BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:974
  in_atomic(): 0, irqs_disabled(): 128, pid: 25, name: cpuhp/0
  CPU: 0 PID: 25 Comm: cpuhp/0 Not tainted 4.16.18-rt10-rt #2
  Hardware name: Huawei TaiShan 2280 /BC11SPCD, BIOS 1.32 08/22/2017
  Call trace:
   dump_backtrace+0x0/0x188
   show_stack+0x24/0x30
   dump_stack+0x9c/0xd0
   ___might_sleep+0x124/0x188
   rt_spin_lock+0x40/0x80
   pcpu_alloc+0x104/0x7a0
   __alloc_percpu_gfp+0x38/0x48
   __armpmu_alloc+0x44/0x168
   armpmu_alloc_atomic+0x1c/0x28
   arm_pmu_acpi_cpu_starting+0x1cc/0x210
   cpuhp_invoke_callback+0xb8/0x820
   cpuhp_thread_fun+0xc0/0x1e0
   smpboot_thread_fn+0x1ac/0x2c8
   kthread+0x134/0x138
   ret_from_fork+0x10/0x18

Move the allocation to CPUHP_BP_PREPARE_DYN, where we'll be preemptible, thus no longer needing GFP_ATOMIC.

---
 drivers/perf/arm_pmu_acpi.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

@@ -135,10 +135,10 @@ static struct arm_pmu *arm_pmu_acpi_find
 		return pmu;
 	}
 
-	pmu = armpmu_alloc_atomic();
+	pmu = armpmu_alloc();
 	if (!pmu) {
 		pr_warn("Unable to allocate PMU for CPU%d\n",
-			smp_processor_id());
+			raw_smp_processor_id());
 		return NULL;
 	}
 

@@ -283,7 +283,7 @@ static int arm_pmu_acpi_init(void)
 	if (ret)
 		return ret;
 
-	ret = cpuhp_setup_state(CPUHP_AP_PERF_ARM_ACPI_STARTING,
+	ret = cpuhp_setup_state(CPUHP_BP_PREPARE_DYN,
 				"perf/arm/pmu_acpi:starting",
 				arm_pmu_acpi_cpu_starting, NULL);
 

(bah, make a clean The End to adventures in arm-land)

RT cannot allocate while irqs are disabled.

  BUG: sleeping function called from invalid context at kernel/locking/rtmutex.c:974
  in_atomic(): 0, irqs_disabled(): 128, pid: 25, name: cpuhp/0
  CPU: 0 PID: 25 Comm: cpuhp/0 Not tainted 4.16.18-rt10-rt #2
  Hardware name: Huawei TaiShan 2280 /BC11SPCD, BIOS 1.32 08/22/2017
  Call trace:
   dump_backtrace+0x0/0x188
   show_stack+0x24/0x30
   dump_stack+0x9c/0xd0
   ___might_sleep+0x124/0x188
   rt_spin_lock+0x40/0x80
   pcpu_alloc+0x104/0x7a0
   __alloc_percpu_gfp+0x38/0x48
   __armpmu_alloc+0x44/0x168
   armpmu_alloc_atomic+0x1c/0x28
   arm_pmu_acpi_cpu_starting+0x1cc/0x210
   cpuhp_invoke_callback+0xb8/0x820
   cpuhp_thread_fun+0xc0/0x1e0
   smpboot_thread_fn+0x1ac/0x2c8
   kthread+0x134/0x138
   ret_from_fork+0x10/0x18

Do the allocation and other preparation for probe along with the other CPUHP_PERF_{ARCH}_PREPARE stages, where we'll be preemptible, thus no longer requiring a GFP_ATOMIC allocation either.

---
 drivers/perf/arm_pmu_acpi.c |   12 ++++++------
 include/linux/cpuhotplug.h  |    2 +-
 2 files changed, 7 insertions(+), 7 deletions(-)

@@ -135,10 +135,10 @@ static struct arm_pmu *arm_pmu_acpi_find
 		return pmu;
 	}
 
-	pmu = armpmu_alloc_atomic();
+	pmu = armpmu_alloc();
 	if (!pmu) {
 		pr_warn("Unable to allocate PMU for CPU%d\n",
-			smp_processor_id());
+			raw_smp_processor_id());
 		return NULL;
 	}
 

@@ -185,7 +185,7 @@ static bool pmu_irq_matches(struct arm_p
  * coming up. The perf core won't open events while a hotplug event is in
  * progress.
  */
-static int arm_pmu_acpi_cpu_starting(unsigned int cpu)
+static int arm_pmu_acpi_cpu_prepare(unsigned int cpu)
 {
 	struct arm_pmu *pmu;
 	struct pmu_hw_events __percpu *hw_events;

@@ -283,9 +283,9 @@ static int arm_pmu_acpi_init(void)
 	if (ret)
 		return ret;
 
-	ret = cpuhp_setup_state(CPUHP_AP_PERF_ARM_ACPI_STARTING,
-				"perf/arm/pmu_acpi:starting",
-				arm_pmu_acpi_cpu_starting, NULL);
+	ret = cpuhp_setup_state(CPUHP_PERF_ARM_PMU_ACPI_PREPARE,
+				"perf/arm/pmu_acpi:prepare",
+				arm_pmu_acpi_cpu_prepare, NULL);
 
 	return ret;
 }

@@ -29,6 +29,7 @@ enum cpuhp_state {
 	CPUHP_PERF_PREPARE,
 	CPUHP_PERF_X86_PREPARE,
 	CPUHP_PERF_X86_AMD_UNCORE_PREP,
+	CPUHP_PERF_ARM_PMU_ACPI_PREPARE,
 	CPUHP_PERF_BFIN,
 	CPUHP_PERF_POWER,
 	CPUHP_PERF_SUPERH,

@@ -114,7 +115,6 @@ enum cpuhp_state {
 	CPUHP_AP_ARM_VFP_STARTING,
 	CPUHP_AP_ARM64_DEBUG_MONITORS_STARTING,
 	CPUHP_AP_PERF_ARM_HW_BREAKPOINT_STARTING,
-	CPUHP_AP_PERF_ARM_ACPI_STARTING,
 	CPUHP_AP_PERF_ARM_STARTING,
 	CPUHP_AP_ARM_L2X0_STARTING,
 	CPUHP_AP_ARM_ARCH_TIMER_STARTING,

Nope, it's an ex-candidate. Having found/run kvm unit tests, I discovered that while the above fixes boot time splat, it somehow manages to break kvm pmu tests, so needs staring at.

-Mike

FYI, per kvm unit tests, 4.16-rt definitely has more kvm issues.

huawei5:/abuild/mike/kvm-unit-tests # uname -r 4.16.18-rt11-rt huawei5:/abuild/mike/kvm-unit-tests # ./run_tests.sh PASS selftest-setup (2 tests)
FAIL selftest-vectors-kernel
FAIL selftest-vectors-user
PASS selftest-smp (65 tests)
PASS pci-test (1 tests)
PASS pmu (3 tests)
FAIL gicv2-ipi
FAIL gicv3-ipi
FAIL gicv2-active
FAIL gicv3-active
PASS psci (4 tests)
FAIL timer huawei5:/abuild/mike/kvm-unit-tests #

4.14-rt passes all tests. The above is with the kvm raw_spinlock_t conversion patch applied, but the 4.12 based SLERT tree I cloned to explore arm-land in the first place shows only one timer failure, and has/needs it applied as well, which would seem to vindicate it.

huawei5:/abuild/mike/kvm-unit-tests # uname -r 4.12.14-0.gec0b559-rt huawei5:/abuild/mike/kvm-unit-tests # ./run_tests.sh PASS selftest-setup (2 tests)
PASS selftest-vectors-kernel (2 tests)
PASS selftest-vectors-user (2 tests)
PASS selftest-smp (65 tests)
PASS pci-test (1 tests)
PASS pmu (3 tests)
PASS gicv2-ipi (3 tests)
PASS gicv3-ipi (3 tests)
PASS gicv2-active (1 tests)
PASS gicv3-active (1 tests)
PASS psci (4 tests)
FAIL timer (8 tests, 1 unexpected failures) huawei5:/abuild/mike/kvm-unit-tests #

FWIW, this single timer failure wass inspired by something in the 4-15 merge window. A 4.14-rt based 4.15-rt updated to include recent fixes reproduces the exact same (ie my colleagues imported it into SLE). The rest landed in 4.16.. staring at which is not proving the least bit enlightening.

-Mike

But it's not RT, or rather most of it isn't...

As noted, the single timer failure is an RT issue of some sort, and remains. The rest I bisected in @stable with the attached config, and confirmed that revert fixes up 4.16-rt as well (modulo singleton).

a9c0e12ebee56ef06b7eccdbc73bab71d0018df8 is the first bad commit commit a9c0e12ebee56ef06b7eccdbc73bab71d0018df8

Author: Marc Zyngier <[email protected]>
Date:   Mon Oct 23 17:11:20 2017 +0100

KVM: arm/arm64: Only clean the dcache on translation fault

The only case where we actually need to perform a dcache maintenance is when we map the page for the first time, and subsequent permission faults do not require cache maintenance. Let's make it conditional on not being a permission fault (and thus a translation fault).

:040000 040000 951e77e6ec8df405f4bad59086ad1416c480ce3c 946dd071aa755606eaa5103d870bc471cc07258f M virt

git bisect start
# good: [a8ec862fd39d9adb88469eb8b9125daccc1c8335] Linux 4.15.18
git bisect good a8ec862fd39d9adb88469eb8b9125daccc1c8335
# bad: [62e9ccfaaedffd057e921fca976f9f7f71c9b254] Linux 4.16.18
git bisect bad 62e9ccfaaedffd057e921fca976f9f7f71c9b254
# good: [d8a5b80568a9cb66810e75b182018e9edb68e8ff] Linux 4.15
git bisect good d8a5b80568a9cb66810e75b182018e9edb68e8ff
# good: [fe53d1443a146326b49d57fe6336b5c2a725223f] Merge tag 'armsoc-drivers' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect good fe53d1443a146326b49d57fe6336b5c2a725223f
# good: [9a61df9e5f7471fe5be3e02bd0bed726b2761a54] Merge tag 'kbuild-v4.16-2' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
git bisect good 9a61df9e5f7471fe5be3e02bd0bed726b2761a54
# bad: [c4f4d2f917729e9b7b8bb452bf4971be93e7a15f] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
git bisect bad c4f4d2f917729e9b7b8bb452bf4971be93e7a15f
# bad: [97ace515f01439d4cf6e898b4094040dc12d36e7] Merge tag 'armsoc-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect bad 97ace515f01439d4cf6e898b4094040dc12d36e7
# bad: [9f416319f40cd857d2bb517630e5855a905ef3fb] arm64: fix unwind_frame() for filtered out fn for function graph tracing git bisect bad 9f416319f40cd857d2bb517630e5855a905ef3fb
# bad: [405cacc947f7b58969b2a8ab1568c2d98b245308] drm/i915/vlv: Add cdclk workaround for DSI
git bisect bad 405cacc947f7b58969b2a8ab1568c2d98b245308
# good: [810f4600ec5ee79c68dcbb136ed26a652df46348] Merge tag 'kvm-s390-next-4.16-3' of git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux
git bisect good 810f4600ec5ee79c68dcbb136ed26a652df46348
# bad: [1ab03c072feb579c9fd116de25be2b211e6bff6a] Merge tag 'kvm-ppc-next-4.16-2' of git://git.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc
git bisect bad 1ab03c072feb579c9fd116de25be2b211e6bff6a
# bad: [87cedc6be55954c6efd6eca2e694132513f65a2a] kvm: x86: remove efer_reload entry in kvm_vcpu_stat
git bisect bad 87cedc6be55954c6efd6eca2e694132513f65a2a
# good: [fefb876b9b96fa7e4ed3d906979ea45b4cf07349] arm64: KVM: PTE/PMD S2 XN bit definition
git bisect good fefb876b9b96fa7e4ed3d906979ea45b4cf07349
# bad: [b276f1b3b1eb52697f6645bb98f7d32ffb89df69] KVM: arm/arm64: Fix trailing semicolon
git bisect bad b276f1b3b1eb52697f6645bb98f7d32ffb89df69
# bad: [17ab9d57debaa53d665651e425a0efc4a893c039] KVM: arm/arm64: Drop vcpu parameter from guest cache maintenance operartions git bisect bad 17ab9d57debaa53d665651e425a0efc4a893c039
# bad: [a9c0e12ebee56ef06b7eccdbc73bab71d0018df8] KVM: arm/arm64: Only clean the dcache on translation fault
git bisect bad a9c0e12ebee56ef06b7eccdbc73bab71d0018df8
# good: [d0e22b4ac3ba23c611739f554392bf5e217df49f] KVM: arm/arm64: Limit icache invalidation to prefetch aborts
git bisect good d0e22b4ac3ba23c611739f554392bf5e217df49f

# first bad commit: [a9c0e12ebee56ef06b7eccdbc73bab71d0018df8] KVM: arm/arm64: Only clean the dcache on translation fault

On Tue, 31 Jul 2018 19:28:49 +0100,
Mike Galbraith <[email protected]> wrote:

Hi Mike,

Is it something that is reproducible with the current mainline (non-RT)?

Pretty worrying. What HW is that on?

	M.

Jazz is not dead, it just smell funny.

These waters are a bit muddy, it's config dependent. I'm trying to generate a reproducing !RT config for -rc7 as we speak. If I build openSUSE/master-default, it does NOT reproduce. That with the bisect config just finished building, and.... yup, it reproduced (attached).

Every RT tree 4.16..master.today reproduces, and is fixed up (modulo the RT specific failure) by reverting the fingered commit.

Box is a 4 node/64 core TaiShan 2280.

-Mike

Thanks for that.

Is that what is also known as D05/HIP07, with 64 Cortex-A72? If so, I think we have one somewhere... I'll try to reproduce it once I'm back into the office (things seem to run just fine on my A72-based laptop, so it definitely smells of being config dependent).

Cheers,

	M.

Jazz is not dead, it just smell funny.

No idea, our rent-a-box web client shows nothing informative.

-Mike

Cheers. The dependency is THP, which is disabled in RT.

@@ -632,7 +632,10 @@ CONFIG_DEFAULT_MMAP_MIN_ADDR=65536
 CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
 CONFIG_MEMORY_FAILURE=y
 # CONFIG_HWPOISON_INJECT is not set
-# CONFIG_TRANSPARENT_HUGEPAGE is not set
+CONFIG_TRANSPARENT_HUGEPAGE=y
+CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
+# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
+CONFIG_TRANSPARENT_HUGE_PAGECACHE=y
 CONFIG_CLEANCACHE=y
 CONFIG_FRONTSWAP=y
 CONFIG_CMA=y

@@ -4306,6 +4309,7 @@ CONFIG_RAS=y
 # CONFIG_ANDROID is not set
 # CONFIG_LIBNVDIMM is not set
 CONFIG_DAX=y
+# CONFIG_DEV_DAX is not set
 CONFIG_NVMEM=y
 # CONFIG_MTK_EFUSE is not set
 # CONFIG_QCOM_QFPROM is not set

@@ -5175,6 +5179,7 @@ CONFIG_DECOMPRESS_XZ=y
 CONFIG_DECOMPRESS_LZO=y
 CONFIG_DECOMPRESS_LZ4=y
 CONFIG_GENERIC_ALLOCATOR=y
+CONFIG_RADIX_TREE_MULTIORDER=y
 CONFIG_ASSOCIATIVE_ARRAY=y
 CONFIG_HAS_IOMEM=y
 CONFIG_HAS_IOPORT_MAP=y

Hi Mike,

Could you give that patchlet[1] a go? It solves a similar issue for me on a different platform.

Thanks,

	M.

[1] https://lists.cs.columbia.edu/pipermail/kvmarm/2018-August/032469.html

Jazz is not dead. It just smells funny...

Yup, all better.

-Mike

Now I'm not sure it actually fixes anything. Alex's test case seem to have opened a new can of worms... In tracing hell at the moment.

Thanks,

	M.

Jazz is not dead. It just smells funny...

I'll keep the test box reserved a while longer. When you get it's other leg stuffed into the casket, if you want testing, holler.

-Mike

Add raw_cond_resched_lock() infrastructure.

---
 include/linux/sched.h |   15 +++++++++++++++
 kernel/sched/core.c   |   20 ++++++++++++++++++++
 2 files changed, 35 insertions(+)

@@ -1779,12 +1779,18 @@ static inline int _cond_resched(void) {
 })
 
 extern int __cond_resched_lock(spinlock_t *lock);
+extern int __raw_cond_resched_lock(raw_spinlock_t *lock);
 
 #define cond_resched_lock(lock) ({				\
 	___might_sleep(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET);\
 	__cond_resched_lock(lock);				\
 })
 
+#define raw_cond_resched_lock(lock) ({				\
+	___might_sleep(__FILE__, __LINE__, PREEMPT_LOCK_OFFSET);\
+	__raw_cond_resched_lock(lock);				\
+})
+
 #ifndef CONFIG_PREEMPT_RT_FULL
 extern int __cond_resched_softirq(void);
 

@@ -1817,6 +1823,15 @@ static inline int spin_needbreak(spinloc
 #else
 	return 0;
 #endif
+}
+
+static inline int raw_spin_needbreak(raw_spinlock_t *lock)
+{
+#ifdef CONFIG_PREEMPT
+	return raw_spin_is_contended(lock);
+#else
+	return 0;
+#endif
 }
 
 static __always_inline bool need_resched(void)

@@ -5065,6 +5065,26 @@ int __cond_resched_lock(spinlock_t *lock
 }
 EXPORT_SYMBOL(__cond_resched_lock);
 
+int __raw_cond_resched_lock(raw_spinlock_t *lock)
+{
+	int resched = should_resched(PREEMPT_LOCK_OFFSET);
+	int ret = 0;
+
+	lockdep_assert_held(lock);
+
+	if (raw_spin_needbreak(lock) || resched) {
+		raw_spin_unlock(lock);
+		if (resched)
+			preempt_schedule_common();
+		else
+			cpu_relax();
+		ret = 1;
+		raw_spin_lock(lock);
+	}
+	return ret;
+}
+EXPORT_SYMBOL(__raw_cond_resched_lock);
+
 #ifndef CONFIG_PREEMPT_RT_FULL
 int __sched __cond_resched_softirq(void)
 {

b103cc3f10c0 ("KVM: arm/arm64: Avoid timer save/restore in vcpu entry/exit") requires vgic_irq.irq_lock be converted to raw_spinlock_t.

Problem: kvm_preempt_ops.sched_in = kvm_sched_in;
kvm_sched_in()

      kvm_arch_vcpu_load()
         kvm_timer_vcpu_load() <- b103cc3f10c0 addition
            kvm_timer_vcpu_load_gic()
               kvm_vgic_map_is_active()
                  spin_lock_irqsave(&irq->irq_lock, flags);

Quoting virt/kvm/arm/vgic/vgic.c, locking order is...

kvm->lock (mutex)

    its->cmd_lock (mutex)
      its->its_lock (mutex)
        vgic_cpu->ap_list_lock         must be taken with IRQs disabled
          kvm->lpi_list_lock           must be taken with IRQs disabled
            vgic_irq->irq_lock         must be taken with IRQs disabled

...meaning vgic_dist.lpi_list_lock and vgic_cpu.ap_list_lock must be converted as well.

---
 include/kvm/arm_vgic.h           |    6 -
 virt/kvm/arm/vgic/vgic-debug.c   |    4 -
 virt/kvm/arm/vgic/vgic-init.c    |    8 +-
 virt/kvm/arm/vgic/vgic-its.c     |   22 +++----
 virt/kvm/arm/vgic/vgic-mmio-v2.c |   14 ++--
 virt/kvm/arm/vgic/vgic-mmio-v3.c |   10 +--
 virt/kvm/arm/vgic/vgic-mmio.c    |   34 +++++------
 virt/kvm/arm/vgic/vgic-v2.c      |    4 -
 virt/kvm/arm/vgic/vgic-v3.c      |    8 +-
 virt/kvm/arm/vgic/vgic.c         |  120 +++++++++++++++++++--------------------
 10 files changed, 115 insertions(+), 115 deletions(-)

@@ -96,7 +96,7 @@ enum vgic_irq_config {
 };
 
 struct vgic_irq {
-	spinlock_t irq_lock;		/* Protects the content of the struct */
+	raw_spinlock_t irq_lock;	/* Protects the content of the struct */
 	struct list_head lpi_list;	/* Used to link all LPIs together */
 	struct list_head ap_list;
 

@@ -243,7 +243,7 @@ struct vgic_dist {
 	u64			propbaser;
 
 	/* Protects the lpi_list and the count value below. */
-	spinlock_t		lpi_list_lock;
+	raw_spinlock_t		lpi_list_lock;
 	struct list_head	lpi_list_head;
 	int			lpi_list_count;
 

@@ -296,7 +296,7 @@ struct vgic_cpu {
 	unsigned int used_lrs;
 	struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS];
 
-	spinlock_t ap_list_lock;	/* Protects the ap_list */
+	raw_spinlock_t ap_list_lock;	/* Protects the ap_list */
 
 	/*
 	 * List of IRQs that this VCPU should consider because they are either

@@ -228,9 +228,9 @@ static int vgic_debug_show(struct seq_fi
 		irq = &kvm->arch.vgic.spis[iter->intid - VGIC_NR_PRIVATE_IRQS];
 	}
 
-	spin_lock_irqsave(&irq->irq_lock, flags);
+	raw_spin_lock_irqsave(&irq->irq_lock, flags);
 	print_irq_state(s, irq, vcpu);
-	spin_unlock_irqrestore(&irq->irq_lock, flags);
+	raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 
 	return 0;
 }

@@ -64,7 +64,7 @@ void kvm_vgic_early_init(struct kvm *kvm
 	struct vgic_dist *dist = &kvm->arch.vgic;
 
 	INIT_LIST_HEAD(&dist->lpi_list_head);
-	spin_lock_init(&dist->lpi_list_lock);
+	raw_spin_lock_init(&dist->lpi_list_lock);
 }
 
 /**

@@ -80,7 +80,7 @@ void kvm_vgic_vcpu_early_init(struct kvm
 	int i;
 
 	INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
-	spin_lock_init(&vgic_cpu->ap_list_lock);
+	raw_spin_lock_init(&vgic_cpu->ap_list_lock);
 
 	/*
 	 * Enable and configure all SGIs to be edge-triggered and

@@ -90,7 +90,7 @@ void kvm_vgic_vcpu_early_init(struct kvm
 		struct vgic_irq *irq = &vgic_cpu->private_irqs[i];
 
 		INIT_LIST_HEAD(&irq->ap_list);
-		spin_lock_init(&irq->irq_lock);
+		raw_spin_lock_init(&irq->irq_lock);
 		irq->intid = i;
 		irq->vcpu = NULL;
 		irq->target_vcpu = vcpu;

@@ -214,7 +214,7 @@ static int kvm_vgic_dist_init(struct kvm
 
 		irq->intid = i + VGIC_NR_PRIVATE_IRQS;
 		INIT_LIST_HEAD(&irq->ap_list);
-		spin_lock_init(&irq->irq_lock);
+		raw_spin_lock_init(&irq->irq_lock);
 		irq->vcpu = NULL;
 		irq->target_vcpu = vcpu0;
 		kref_init(&irq->refcount);

@@ -65,14 +65,14 @@ static struct vgic_irq *vgic_add_lpi(str
 
 	INIT_LIST_HEAD(&irq->lpi_list);
 	INIT_LIST_HEAD(&irq->ap_list);
-	spin_lock_init(&irq->irq_lock);
+	raw_spin_lock_init(&irq->irq_lock);
 
 	irq->config = VGIC_CONFIG_EDGE;
 	kref_init(&irq->refcount);
 	irq->intid = intid;
 	irq->target_vcpu = vcpu;
 
-	spin_lock_irqsave(&dist->lpi_list_lock, flags);
+	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
 
 	/*
 	 * There could be a race with another vgic_add_lpi(), so we need to

@@ -100,7 +100,7 @@ static struct vgic_irq *vgic_add_lpi(str
 	dist->lpi_list_count++;
 
 out_unlock:
-	spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
+	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
 
 	/*
 	 * We "cache" the configuration table entries in our struct vgic_irq's.

@@ -287,7 +287,7 @@ static int update_lpi_config(struct kvm
 	if (ret)
 		return ret;
 
-	spin_lock_irqsave(&irq->irq_lock, flags);
+	raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
 	if (!filter_vcpu || filter_vcpu == irq->target_vcpu) {
 		irq->priority = LPI_PROP_PRIORITY(prop);

@@ -299,7 +299,7 @@ static int update_lpi_config(struct kvm
 		}
 	}
 
-	spin_unlock_irqrestore(&irq->irq_lock, flags);
+	raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 
 	if (irq->hw)
 		return its_prop_update_vlpi(irq->host_irq, prop, needs_inv);

@@ -332,7 +332,7 @@ static int vgic_copy_lpi_list(struct kvm
 	if (!intids)
 		return -ENOMEM;
 
-	spin_lock_irqsave(&dist->lpi_list_lock, flags);
+	raw_spin_lock_irqsave(&dist->lpi_list_lock, flags);
 	list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
 		if (i == irq_count)
 			break;

@@ -341,7 +341,7 @@ static int vgic_copy_lpi_list(struct kvm
 			continue;
 		intids[i++] = irq->intid;
 	}
-	spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
+	raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags);
 
 	*intid_ptr = intids;
 	return i;

@@ -352,9 +352,9 @@ static int update_affinity(struct vgic_i
 	int ret = 0;
 	unsigned long flags;
 
-	spin_lock_irqsave(&irq->irq_lock, flags);
+	raw_spin_lock_irqsave(&irq->irq_lock, flags);
 	irq->target_vcpu = vcpu;
-	spin_unlock_irqrestore(&irq->irq_lock, flags);
+	raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 
 	if (irq->hw) {
 		struct its_vlpi_map map;

@@ -455,7 +455,7 @@ static int its_sync_lpi_pending_table(st
 		}
 
 		irq = vgic_get_irq(vcpu->kvm, NULL, intids[i]);
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 		irq->pending_latch = pendmask & (1U << bit_nr);
 		vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
 		vgic_put_irq(vcpu->kvm, irq);

@@ -612,7 +612,7 @@ static int vgic_its_trigger_msi(struct k
 		return irq_set_irqchip_state(irq->host_irq,
 					     IRQCHIP_STATE_PENDING, true);
 
-	spin_lock_irqsave(&irq->irq_lock, flags);
+	raw_spin_lock_irqsave(&irq->irq_lock, flags);
 	irq->pending_latch = true;
 	vgic_queue_irq_unlock(kvm, irq, flags);
 

@@ -100,7 +100,7 @@ static void vgic_mmio_write_sgir(struct
 
 		irq = vgic_get_irq(source_vcpu->kvm, vcpu, intid);
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 		irq->pending_latch = true;
 		irq->source |= 1U << source_vcpu->vcpu_id;
 

@@ -144,13 +144,13 @@ static void vgic_mmio_write_target(struc
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, NULL, intid + i);
 		int target;
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
 		irq->targets = (val >> (i * 8)) & cpu_mask;
 		target = irq->targets ? __ffs(irq->targets) : 0;
 		irq->target_vcpu = kvm_get_vcpu(vcpu->kvm, target);
 
-		spin_unlock_irqrestore(&irq->irq_lock, flags);
+		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 		vgic_put_irq(vcpu->kvm, irq);
 	}
 }

@@ -183,13 +183,13 @@ static void vgic_mmio_write_sgipendc(str
 	for (i = 0; i < len; i++) {
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
 		irq->source &= ~((val >> (i * 8)) & 0xff);
 		if (!irq->source)
 			irq->pending_latch = false;
 
-		spin_unlock_irqrestore(&irq->irq_lock, flags);
+		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 		vgic_put_irq(vcpu->kvm, irq);
 	}
 }

@@ -205,7 +205,7 @@ static void vgic_mmio_write_sgipends(str
 	for (i = 0; i < len; i++) {
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
 		irq->source |= (val >> (i * 8)) & 0xff;
 

@@ -213,7 +213,7 @@ static void vgic_mmio_write_sgipends(str
 			irq->pending_latch = true;
 			vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
 		} else {
-			spin_unlock_irqrestore(&irq->irq_lock, flags);
+			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 		}
 		vgic_put_irq(vcpu->kvm, irq);
 	}

@@ -145,13 +145,13 @@ static void vgic_mmio_write_irouter(stru
 	if (!irq)
 		return;
 
-	spin_lock_irqsave(&irq->irq_lock, flags);
+	raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
 	/* We only care about and preserve Aff0, Aff1 and Aff2. */
 	irq->mpidr = val & GENMASK(23, 0);
 	irq->target_vcpu = kvm_mpidr_to_vcpu(vcpu->kvm, irq->mpidr);
 
-	spin_unlock_irqrestore(&irq->irq_lock, flags);
+	raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 	vgic_put_irq(vcpu->kvm, irq);
 }
 

@@ -252,7 +252,7 @@ static void vgic_v3_uaccess_write_pendin
 	for (i = 0; i < len * 8; i++) {
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 		if (test_bit(i, &val)) {
 			/*
 			 * pending_latch is set irrespective of irq type

@@ -263,7 +263,7 @@ static void vgic_v3_uaccess_write_pendin
 			vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
 		} else {
 			irq->pending_latch = false;
-			spin_unlock_irqrestore(&irq->irq_lock, flags);
+			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 		}
 
 		vgic_put_irq(vcpu->kvm, irq);

@@ -845,7 +845,7 @@ void vgic_v3_dispatch_sgi(struct kvm_vcp
 
 		irq = vgic_get_irq(vcpu->kvm, c_vcpu, sgi);
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 		irq->pending_latch = true;
 
 		vgic_queue_irq_unlock(vcpu->kvm, irq, flags);

@@ -75,7 +75,7 @@ void vgic_mmio_write_senable(struct kvm_
 	for_each_set_bit(i, &val, len * 8) {
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 		irq->enabled = true;
 		vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
 

@@ -94,11 +94,11 @@ void vgic_mmio_write_cenable(struct kvm_
 	for_each_set_bit(i, &val, len * 8) {
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
 		irq->enabled = false;
 
-		spin_unlock_irqrestore(&irq->irq_lock, flags);
+		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 		vgic_put_irq(vcpu->kvm, irq);
 	}
 }

@@ -115,10 +115,10 @@ unsigned long vgic_mmio_read_pending(str
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 		unsigned long flags;
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 		if (irq_is_pending(irq))
 			value |= (1U << i);
-		spin_unlock_irqrestore(&irq->irq_lock, flags);
+		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 
 		vgic_put_irq(vcpu->kvm, irq);
 	}

@@ -170,7 +170,7 @@ void vgic_mmio_write_spending(struct kvm
 	for_each_set_bit(i, &val, len * 8) {
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 		if (irq->hw)
 			vgic_hw_irq_spending(vcpu, irq, is_uaccess);
 		else

@@ -217,14 +217,14 @@ void vgic_mmio_write_cpending(struct kvm
 	for_each_set_bit(i, &val, len * 8) {
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
 		if (irq->hw)
 			vgic_hw_irq_cpending(vcpu, irq, is_uaccess);
 		else
 			irq->pending_latch = false;
 
-		spin_unlock_irqrestore(&irq->irq_lock, flags);
+		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 		vgic_put_irq(vcpu->kvm, irq);
 	}
 }

@@ -266,7 +266,7 @@ static void vgic_mmio_change_active(stru
 	unsigned long flags;
 	struct kvm_vcpu *requester_vcpu = vgic_get_mmio_requester_vcpu();
 
-	spin_lock_irqsave(&irq->irq_lock, flags);
+	raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
 	/*
 	 * If this virtual IRQ was written into a list register, we

@@ -287,7 +287,7 @@ static void vgic_mmio_change_active(stru
 	while (irq->vcpu && /* IRQ may have state in an LR somewhere */
 	       irq->vcpu != requester_vcpu && /* Current thread is not the VCPU thread */
 	       irq->vcpu->cpu != -1) /* VCPU thread is running */
-		cond_resched_lock(&irq->irq_lock);
+		raw_cond_resched_lock(&irq->irq_lock);
 
 	if (irq->hw)
 		vgic_hw_irq_change_active(vcpu, irq, active, !requester_vcpu);

@@ -297,7 +297,7 @@ static void vgic_mmio_change_active(stru
 	if (irq->active)
 		vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
 	else
-		spin_unlock_irqrestore(&irq->irq_lock, flags);
+		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 }
 
 /*

@@ -436,10 +436,10 @@ void vgic_mmio_write_priority(struct kvm
 	for (i = 0; i < len; i++) {
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 		/* Narrow the priority range to what we actually support */
 		irq->priority = (val >> (i * 8)) & GENMASK(7, 8 - VGIC_PRI_BITS);
-		spin_unlock_irqrestore(&irq->irq_lock, flags);
+		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 
 		vgic_put_irq(vcpu->kvm, irq);
 	}

@@ -485,14 +485,14 @@ void vgic_mmio_write_config(struct kvm_v
 			continue;
 
 		irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
 		if (test_bit(i * 2 + 1, &val))
 			irq->config = VGIC_CONFIG_EDGE;
 		else
 			irq->config = VGIC_CONFIG_LEVEL;
 
-		spin_unlock_irqrestore(&irq->irq_lock, flags);
+		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 		vgic_put_irq(vcpu->kvm, irq);
 	}
 }

@@ -541,12 +541,12 @@ void vgic_write_irq_line_level_info(stru
 		 * restore irq config before line level.
 		 */
 		new_level = !!(val & (1U << i));
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 		irq->line_level = new_level;
 		if (new_level)
 			vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
 		else
-			spin_unlock_irqrestore(&irq->irq_lock, flags);
+			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 
 		vgic_put_irq(vcpu->kvm, irq);
 	}

@@ -85,7 +85,7 @@ void vgic_v2_fold_lr_state(struct kvm_vc
 
 		irq = vgic_get_irq(vcpu->kvm, vcpu, intid);
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
 		/* Always preserve the active bit */
 		irq->active = !!(val & GICH_LR_ACTIVE_BIT);

@@ -132,7 +132,7 @@ void vgic_v2_fold_lr_state(struct kvm_vc
 				vgic_irq_set_phys_active(irq, false);
 		}
 
-		spin_unlock_irqrestore(&irq->irq_lock, flags);
+		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 		vgic_put_irq(vcpu->kvm, irq);
 	}
 

@@ -75,7 +75,7 @@ void vgic_v3_fold_lr_state(struct kvm_vc
 		if (!irq)	/* An LPI could have been unmapped. */
 			continue;
 
-		spin_lock_irqsave(&irq->irq_lock, flags);
+		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
 		/* Always preserve the active bit */
 		irq->active = !!(val & ICH_LR_ACTIVE_BIT);

@@ -123,7 +123,7 @@ void vgic_v3_fold_lr_state(struct kvm_vc
 				vgic_irq_set_phys_active(irq, false);
 		}
 
-		spin_unlock_irqrestore(&irq->irq_lock, flags);
+		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 		vgic_put_irq(vcpu->kvm, irq);
 	}
 

@@ -335,9 +335,9 @@ int vgic_v3_lpi_sync_pending_status(stru
 
 	status = val & (1 << bit_nr);
 
-	spin_lock_irqsave(&irq->irq_lock, flags);
+	raw_spin_lock_irqsave(&irq->irq_lock, flags);
 	if (irq->target_vcpu != vcpu) {
-		spin_unlock_irqrestore(&irq->irq_lock, flags);
+		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 		goto retry;
 	}
 	irq->pending_latch = status;

The thing missing from the Changelog is the analysis that all the work done under these locks is indeed properly bounded and cannot cause excessive latencies.

True, I have no idea what worst case hold times are. Nothing poked me dead in the eye when looking around in completely alien code, nor did cyclictest inspire concern running on box with no base of comparison.

I do know that latency is now < infinity, a modest improvement ;-)

-Mike

(arm-land adventures 1/3 take2 will have to wait, my cup runeth over)

v4.14..v4.15 timer handling changes including calling kvm_timer_vcpu_load() during kvm_preempt_ops.sched_in and taking vgic_dist.lpi_list_lock in the timer interrupt handler required locks for which locking rules/context had been changed be converted to raw_spinlock_t...

Quoting virt/kvm/arm/vgic/vgic.c:
* Locking order is always:
* kvm->lock (mutex)
* its->cmd_lock (mutex)
* its->its_lock (mutex)

 *       vgic_cpu->ap_list_lock         must be taken with IRQs disabled
 *         kvm->lpi_list_lock           must be taken with IRQs disabled
 *           vgic_irq->irq_lock         must be taken with IRQs disabled

*
* As the ap_list_lock might be taken from the timer interrupt handler,
* we have to disable IRQs before taking this lock and everything lower
* than it.

...and fixed the obvious bricking consequence of those changes for RT, but left an RT specific kvm unit test timer failure in its wake. Handling phys_timer in hard interrupt context as expected cures that failure.

Pre:
PASS selftest-setup (2 tests)
PASS selftest-vectors-kernel (2 tests)
PASS selftest-vectors-user (2 tests)
PASS selftest-smp (65 tests)
PASS pci-test (1 tests)
PASS pmu (3 tests)
PASS gicv2-ipi (3 tests)
PASS gicv3-ipi (3 tests)
PASS gicv2-active (1 tests)
PASS gicv3-active (1 tests)
PASS psci (4 tests)
FAIL timer (8 tests, 1 unexpected failures)

Post:
PASS selftest-setup (2 tests)
PASS selftest-vectors-kernel (2 tests)
PASS selftest-vectors-user (2 tests)
PASS selftest-smp (65 tests)
PASS pci-test (1 tests)
PASS pmu (3 tests)
PASS gicv2-ipi (3 tests)
PASS gicv3-ipi (3 tests)
PASS gicv2-active (1 tests)
PASS gicv3-active (1 tests)
PASS psci (4 tests)
PASS timer (8 tests)

---
 virt/kvm/arm/arch_timer.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

@@ -634,7 +634,7 @@ void kvm_timer_vcpu_init(struct kvm_vcpu
 	hrtimer_init(&timer->bg_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
 	timer->bg_timer.function = kvm_bg_timer_expire;
 
-	hrtimer_init(&timer->phys_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
+	hrtimer_init(&timer->phys_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD);
 	timer->phys_timer.function = kvm_phys_timer_expire;
 
 	vtimer->irq.irq = default_vtimer_irq.irq;

I take it that this should be added to v4.16-rt and marked stable-rt?

-- Steve

Yeah, barring way sexier ideas of course.

-Mike

Gah, wait. WRT marking for stable-rt, only the pmu allocation splat fix is applicable to @stable-rt, kvm locking woes start at 4.15.

-Mike

None of that series got picked up, so the pmu patch is moot as well. The warning is harmless anyway.

Besides, there are more interesting fish in the arm64 sea than kvm.

virgin 4.16.18-rt12-rt

[  537.236131] ITS queue timeout (65440 65504 4640)
[  537.236150] ITS cmd its_build_inv_cmd failed
[  537.236160] sched: RT throttling activated
[ 1229.180124] ITS queue timeout (65440 65504 3744)
[ 1229.180134] ITS cmd its_build_inv_cmd failed

(converts those to trace_printk [invalidating virgin source warantee])

# tracer: nop
#
# _-----=> irqs-off
# / _----=> need-resched

#                            | /  _----=> need-resched_lazy
#                            || / _---=> hardirq/softirq
#                            ||| / _--=> preempt-depth
#                            |||| /     delay
#           TASK-PID   CPU#  |||||    TIMESTAMP  FUNCTION
#              | |       |   |||||       |         |
          <idle>-0     [000] d..h2..    84.967153: its_wait_for_range_completion+0xa8/0xe0: ITS queue timeout (65440 65504 96)
          <idle>-0     [000] d..h2..    84.967155: its_send_single_command+0xd4/0x130: ITS cmd its_build_inv_cmd failed

(starts cyclictest+kbuild)

[ 9609.602489] NOHZ: local_softirq_pending 10
[10089.311713] NOHZ: local_softirq_pending 80
[10089.311735] NOHZ: local_softirq_pending 80
[10097.315809] NOHZ: local_softirq_pending 80
[10112.262436] NOHZ: local_softirq_pending 10
[10118.966066] NOHZ: local_softirq_pending 80
[10121.211106] NOHZ: local_softirq_pending 80
[10121.287093] NOHZ: local_softirq_pending 80
[10329.501202] NOHZ: local_softirq_pending 80
[10329.501231] NOHZ: local_softirq_pending 80

T:60 (17291) P:99 I:31000 C: 18128 Min: 5 Act: 12 Avg: 107 Max: 1581163 <== Thar she blows!

irq/464-eth0-Tx-3955 [060] d...111 11015.059743: its_wait_for_range_completion+0xa8/0xe0: ITS queue timeout (65440 65504 5280) irq/464-eth0-Tx-3955 [060] d...111 11015.059749: <stack trace>
=> irq_chip_unmask_parent+0x2c/0x38
=> its_unmask_msi_irq+0x28/0x38
=> unmask_irq.part.4+0x30/0x50
=> unmask_threaded_irq+0x4c/0x58
=> irq_finalize_oneshot.part.2+0x9c/0x100
=> irq_forced_thread_fn+0x60/0x98
=> irq_thread+0x148/0x1e0
=> kthread+0x140/0x148
=> ret_from_fork+0x10/0x18
irq/464-eth0-Tx-3955 [060] d...111 11015.059751: its_send_single_command+0xd4/0x130: ITS cmd its_build_inv_cmd failed irq/464-eth0-Tx-3955 [060] d...111 11015.059752: <stack trace>
=> its_unmask_msi_irq+0x28/0x38
=> unmask_irq.part.4+0x30/0x50
=> unmask_threaded_irq+0x4c/0x58
=> irq_finalize_oneshot.part.2+0x9c/0x100
=> irq_forced_thread_fn+0x60/0x98
=> irq_thread+0x148/0x1e0
=> kthread+0x140/0x148
=> ret_from_fork+0x10/0x18
irq/464-eth0-Tx-3955 [060] d..h311 11015.059764: update_curr_rt+0x280/0x320: sched: RT throttling activated irq/464-eth0-Tx-3955 [060] d..h311 11015.059765: <stack trace>
=> update_process_times+0x48/0x60
=> tick_sched_handle.isra.5+0x34/0x70
=> tick_sched_timer+0x54/0xa8
=> __hrtimer_run_queues+0xf8/0x388
=> hrtimer_interrupt+0xe4/0x258
=> arch_timer_handler_phys+0x38/0x58
=> handle_percpu_devid_irq+0xa8/0x2c8
=> generic_handle_irq+0x34/0x50
=> __handle_domain_irq+0x8c/0x100
=> gic_handle_irq+0x80/0x18c
=> el1_irq+0xb4/0x13c
=> _raw_spin_unlock_irq+0x2c/0x78
=> irq_finalize_oneshot.part.2+0x6c/0x100
=> irq_forced_thread_fn+0x60/0x98
=> irq_thread+0x148/0x1e0
=> kthread+0x140/0x148
=> ret_from_fork+0x10/0x18

          <idle>-0     [008] d..h3.. 11015.374565: sched_rt_period_timer+0x2f0/0x420: sched: RT throttling deactivated
          <idle>-0     [008] d..h3.. 11015.374570: <stack trace>

=> arch_timer_handler_phys+0x38/0x58
=> handle_percpu_devid_irq+0xa8/0x2c8
=> generic_handle_irq+0x34/0x50
=> __handle_domain_irq+0x8c/0x100
=> gic_handle_irq+0x80/0x18c
=> el1_irq+0xb4/0x13c
=> arch_cpu_idle+0x2c/0x1d0
=> default_idle_call+0x30/0x48
=> do_idle+0x1c4/0x218
=> cpu_startup_entry+0x2c/0x30
=> secondary_start_kernel+0x184/0x1d0

FWIW, I had thought that something 4.16..master had made that business go entirely away, but it only became much more infrequent. So much for plan to slowly sneak up on whatever than was via ~bisect.

I just released v4.18 based RT. Please repost the kvm/RT patches once things have settled.

Sebastian