Skip to contentSkip to conversation

kernel/irq: Add irqbalance01

PATCH In review Richard Palethorpe 4 participants 4 messages Sep 16, 2021 – Oct 11, 2021 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

PATCH v2 1/1 kernel/irq: Add irqbalance01

Add first test specifically targeting interrupts and IRQ management.

This includes some comments inline because I think the parsing code is unavoidably confusing.

Note on the CPU mask parsing; there is already some code for parsing and manipulating bitmaps in the LTP. However it is absurdly complicated and we don't need actual bitmaps. In fact an array of bytes is more flexible.

---

V2:
* Read proc files in at most 1-page blocks

As Cyril suggested this looks like the best way. seq_file will only expand the internal buffer above a page if a record does not fit in one page. In this case a record is the per-cpus counts for an irq. However it won't expand the buffer for multiple records. It just returns whatver fits in the buffer.

* Read in CPU masks from the end of the buffer

If nr_cpus is not a multiple of 4 then one of the hex digits has some padding. We either need to pad irq_affinity as well (which creates complication) or ignore the padding. It's easier to ignore it if we start at the end.

* Print headers and some minor format changes to printing
* Add a few more parsing checks
* Remove freeing of buffers

 runtest/irq                         |   1 +
 testcases/kernel/irq/.gitignore     |   1 +
 testcases/kernel/irq/Makefile       |   7 +
 testcases/kernel/irq/irqbalance01.c | 316 ++++++++++++++++++++++++++++
 4 files changed, 325 insertions(+)
 create mode 100644 runtest/irq
 create mode 100644 testcases/kernel/irq/.gitignore
 create mode 100644 testcases/kernel/irq/Makefile
 create mode 100644 testcases/kernel/irq/irqbalance01.c
runtest/irq new file +1 -0
new file mode 100644
index 000000000..56d0d23c8
@@ -0,0 +1 @@
+irqbalance01 irqbalance01
testcases/kernel/irq/.gitignore new file +1 -0
new file mode 100644
index 000000000..8ed69a99c
@@ -0,0 +1 @@
+irqbalance01
testcases/kernel/irq/Makefile new file +7 -0
new file mode 100644
index 000000000..aa51da7cb
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+top_srcdir		?= ../../..
+
+include $(top_srcdir)/include/mk/testcases.mk
+
+include $(top_srcdir)/include/mk/generic_leaf_target.mk
testcases/kernel/irq/irqbalance01.c new file +316 -0

316 lines not on this page · raw message

Signature · 1 line
2.31.1
#2 lore raw

PATCH v2 1/1 kernel/irq: Add irqbalance01

Hi Richie,

Changes to this version (it's actually v3) LGTM.
I suppose Cyril wants to recheck it, thus not merging it yet.

Thanks!

Kind regards,
Petr

testcases/kernel/irq/.gitignore line 1

Quoted 3 lines

+++ b/testcases/kernel/irq/.gitignore

testcases/kernel/irq/.gitignore new file +1 -0
@@ -0,0 +1 @@
+irqbalance01

Very nit: we usually put '/' at the beginning to: /irqbalance01
(to apply files only in testcases/kernel/irq/, but that's irrelevant in most cases as there are no subdirectories). If we even bother, it can be changed during merge.

#3 lore raw
In reply to Richard Palethorpe

Hi!

testcases/kernel/irq/irqbalance01.c line 200

Quoted 12 lines
testcases/kernel/irq/irqbalance01.c new file +12 -0
+	/* Read the CPU affinity masks for each IRQ. The first CPU is in the
+	 * right most (least significant) bit. See bitmap_string() in the kernel
+	 * (%*pb)
+	 */
+	for (row = 0; row < nr_irqs; row++) {
+		sprintf(path, "/proc/irq/%u/smp_affinity", irq_ids[row]);
+		buf = read_proc_file(path);
+		c = buf;
+		col = 0;
+
+		while (*c != '\0')
+			c++;

A minor nit, we alredy know the lenght, but we do not return it from the read_proc_file() we may as well change this so taht the read_proc_file returns both buffer pointer and the size. E.g. add size_t *len parameter to the read_proc_file() and set it if it's not NULL.

Other than that the parsing looks good now.

Quoted 10 lines
testcases/kernel/irq/irqbalance01.c new file +10 -0
+static void evidence_of_change(void)
+{
+	size_t row, col, changed = 0;
+
+	for (row = 0; row < nr_irqs; row++) {
+		for (col = 0; col < nr_cpus; col++) {
+			if (!irq_stats[row * nr_cpus + col])
+				continue;
+
+			if (irq_affinity[row * nr_cpus + col] == ALLOW)

testcases/kernel/irq/irqbalance01.c line 291

Quoted 10 lines
testcases/kernel/irq/irqbalance01.c new file +10 -0
+				continue;
+
+			changed++;
+		}
+	}
+
+	tst_res(changed ? TPASS : TFAIL,
+		"Heuristic: Detected %zu irq-cpu pairs have been dissallowed",
+		changed);
+}

I'm still unusure about this part though.

The test fails on my workstation where the IRQs are nicely distributed by the hardware/BIOS.

Maybe we should check if IRQs are distributed somehow evenly and PASS the test if that is the case as well.

In my case there is 27 IRQs, 18 of them have non-zero counters and there is 12 CPUs. For active IRQs that gives 18/12 = 1.5 IRQ per CPU and there are 7 CPUs that handle 2 IRQs, 4 CPUs that handle 1 IRQ and 1 CPU that doesn't handle anything. I wonder if we can figure out some kind of heuristic, but I guess that in the end it wouldn't be worth the effort.

Signature · 5 lines
Cyril Hrubis
[email protected]

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp
#4 lore raw

Hello Cyril,

Quoted 10 lines · Cyril Hrubis

Hi!

Quoted 9 lines
testcases/kernel/irq/irqbalance01.c new file +9 -0
+	/* Read the CPU affinity masks for each IRQ. The first CPU is in the
+	 * right most (least significant) bit. See bitmap_string() in the kernel
+	 * (%*pb)
+	 */
+	for (row = 0; row < nr_irqs; row++) {
+		sprintf(path, "/proc/irq/%u/smp_affinity", irq_ids[row]);
+		buf = read_proc_file(path);
+		c = buf;
+		col = 0;
Quoted 8 lines

testcases/kernel/irq/irqbalance01.c line 200

Quoted 3 lines
testcases/kernel/irq/irqbalance01.c new file +3 -0
+
+		while (*c != '\0')
+			c++;

A minor nit, we alredy know the lenght, but we do not return it from the read_proc_file() we may as well change this so taht the read_proc_file returns both buffer pointer and the size. E.g. add size_t *len parameter to the read_proc_file() and set it if it's not NULL.

+1

Quoted 26 lines

Other than that the parsing looks good now.

Quoted 10 lines
testcases/kernel/irq/irqbalance01.c new file +10 -0
+static void evidence_of_change(void)
+{
+	size_t row, col, changed = 0;
+
+	for (row = 0; row < nr_irqs; row++) {
+		for (col = 0; col < nr_cpus; col++) {
+			if (!irq_stats[row * nr_cpus + col])
+				continue;
+
+			if (irq_affinity[row * nr_cpus + col] == ALLOW)

testcases/kernel/irq/irqbalance01.c line 291

Quoted 10 lines
testcases/kernel/irq/irqbalance01.c new file +10 -0
+				continue;
+
+			changed++;
+		}
+	}
+
+	tst_res(changed ? TPASS : TFAIL,
+		"Heuristic: Detected %zu irq-cpu pairs have been dissallowed",
+		changed);
+}

I'm still unusure about this part though.

Quoted 2 lines

The test fails on my workstation where the IRQs are nicely distributed by the hardware/BIOS.

You don't need irqbalance then :-p

The original bug this is intened to avoid was caused by the irqbalance deamon silently failing.

Quoted 3 lines

Maybe we should check if IRQs are distributed somehow evenly and PASS the test if that is the case as well.

My plan was to have a second heuristic based on the IRQ distribution. However the intention was to make detection more sensitve to bad setups. I think what you are suggesting is the opposite. So that users can blindly run the test and let it figure out if it is appropriate for the hardware.

Quoted 7 lines

In my case there is 27 IRQs, 18 of them have non-zero counters and there is 12 CPUs. For active IRQs that gives 18/12 = 1.5 IRQ per CPU and there are 7 CPUs that handle 2 IRQs, 4 CPUs that handle 1 IRQ and 1 CPU that doesn't handle anything. I wonder if we can figure out some kind of heuristic, but I guess that in the end it wouldn't be worth the effort.

I general I think it is best to write tests which figure out if they are in the right environment. This makes all our lives easier. The problem with this test in particular is the number of assumptions we have to make about what a good IRQ distirbution looks like.

IRQs are (usually) evenly distributed for "performance". However not all IRQs are equal, you might get better performance through an uneven distribution. Especially if you only care about the performance some tasks and not others. Also there are some reasons to isolate some CPUs from IRQs regardless of performance.

Having said that we can probably find a heuristic for what a very bad setup looks like i.e. (almost) all NIC IRQs on one CPU when there are 30+ CPUs and multiple NICs. That is one CPU is totally saturated with IRQs from different NICs. This is a bit speculative so I'd like some feedback from testing/community before putting effort into that.

Signature · 5 lines
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp