Skip to contentSkip to conversation

proc/stat: Separate out individual irq counts into /proc/stat_irqs

PATCH Superseded Waiman Long 4 participants 6 messages Apr 19, 2018 Proc filesystem

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 proc/stat: Separate out individual irq counts into /proc/stat_irqs

It was found that reading /proc/stat could be time consuming on systems with a lot of irqs. For example, reading /proc/stat in a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs. In that particular case, the majority of the CPU cycles for reading /proc/stat was spent in the kstat_irqs() function. Therefore, application performance can be impacted if the application reads /proc/stat rather frequently.

The "intr" line within /proc/stat contains a sum total of all the irqs that have happened followed by a list of irq counts for each individual irq number. In many cases, the first number is good enough. The individual irq counts may not provide that much more information.

In order to avoid this kind of performance issue, all these individual irq counts are now separated into a new /proc/stat_irqs file. The sum total irq count will stay in /proc/stat and be duplicated in /proc/stat_irqs. Applications that need to look up individual irq counts will now have to look into /proc/stat_irqs instead of /proc/stat.

---
 fs/proc/stat.c | 48 +++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 41 insertions(+), 7 deletions(-)
fs/proc/stat.c +41 -7
index 59749df..79e3c03 100644
@@ -155,11 +155,6 @@ static int show_stat(struct seq_file *p, void *v)
 		seq_putc(p, '\n');
 	}
 	seq_put_decimal_ull(p, "intr ", (unsigned long long)sum);
-
-	/* sum again ? it could be updated? */
-	for_each_irq_nr(j)
-		seq_put_decimal_ull(p, " ", kstat_irqs_usr(j));
-
 	seq_printf(p,
 		"\nctxt %llu\n"
 		"btime %llu\n"
@@ -181,15 +176,46 @@ static int show_stat(struct seq_file *p, void *v)
 	return 0;
 }
 
+/*
+ * Showing individual irq counts can be expensive if there are a lot of
+ * irqs. So it is done in a separate procfs file to reduce performance
+ * overhead of reading other statistical counts.
+ */
+static int show_stat_irqs(struct seq_file *p, void *v)
+{
+	int i, j;
+	u64 sum = 0;
+
+	for_each_possible_cpu(i) {
+		sum += kstat_cpu_irqs_sum(i);
+		sum += arch_irq_stat_cpu(i);
+	}
+	sum += arch_irq_stat();
+
+	seq_put_decimal_ull(p, "intr ", (unsigned long long)sum);
+
+	for_each_irq_nr(j)
+		seq_put_decimal_ull(p, " ", kstat_irqs_usr(j));
+
+	seq_putc(p, '\n');
+
+	return 0;
+}
+
 static int stat_open(struct inode *inode, struct file *file)
 {
 	size_t size = 1024 + 128 * num_online_cpus();
 
-	/* minimum size to display an interrupt count : 2 bytes */
-	size += 2 * nr_irqs;
 	return single_open_size(file, show_stat, NULL, size);
 }
 
+static int stat_irqs_open(struct inode *inode, struct file *file)
+{
+	size_t size = 1024 + 16 * nr_irqs;
+
+	return single_open_size(file, show_stat_irqs, NULL, size);
+}
+
 static const struct file_operations proc_stat_operations = {
 	.open		= stat_open,
 	.read		= seq_read,
@@ -197,9 +223,17 @@ static int stat_open(struct inode *inode, struct file *file)
 	.release	= single_release,
 };
 
+static const struct file_operations proc_stat_irqs_operations = {
+	.open		= stat_irqs_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
 static int __init proc_stat_init(void)
 {
 	proc_create("stat", 0, NULL, &proc_stat_operations);
+	proc_create("stat_irqs", 0, NULL, &proc_stat_irqs_operations);
 	return 0;
 }
 fs_initcall(proc_stat_init);
Signature · 1 line
1.8.3.1
#1 lore raw
Quoted 19 lines · Waiman Long

It was found that reading /proc/stat could be time consuming on systems with a lot of irqs. For example, reading /proc/stat in a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs. In that particular case, the majority of the CPU cycles for reading /proc/stat was spent in the kstat_irqs() function. Therefore, application performance can be impacted if the application reads /proc/stat rather frequently.

The "intr" line within /proc/stat contains a sum total of all the irqs that have happened followed by a list of irq counts for each individual irq number. In many cases, the first number is good enough. The individual irq counts may not provide that much more information.

In order to avoid this kind of performance issue, all these individual irq counts are now separated into a new /proc/stat_irqs file. The sum total irq count will stay in /proc/stat and be duplicated in /proc/stat_irqs. Applications that need to look up individual irq counts will now have to look into /proc/stat_irqs instead of /proc/stat.

Quoted 4 lines
---
 fs/proc/stat.c | 48 +++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 41 insertions(+), 7 deletions(-)

Also please update Documentation/filesystems/proc.txt.

thanks,

Signature · 1 line
~Randy
#2 lore raw
Quoted 20 lines · Randy Dunlap
Quoted 19 lines · Waiman Long

It was found that reading /proc/stat could be time consuming on systems with a lot of irqs. For example, reading /proc/stat in a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs. In that particular case, the majority of the CPU cycles for reading /proc/stat was spent in the kstat_irqs() function. Therefore, application performance can be impacted if the application reads /proc/stat rather frequently.

The "intr" line within /proc/stat contains a sum total of all the irqs that have happened followed by a list of irq counts for each individual irq number. In many cases, the first number is good enough. The individual irq counts may not provide that much more information.

In order to avoid this kind of performance issue, all these individual irq counts are now separated into a new /proc/stat_irqs file. The sum total irq count will stay in /proc/stat and be duplicated in /proc/stat_irqs. Applications that need to look up individual irq counts will now have to look into /proc/stat_irqs instead of /proc/stat.

Quoted 7 lines
Quoted 4 lines
---
 fs/proc/stat.c | 48 +++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 41 insertions(+), 7 deletions(-)

Also please update Documentation/filesystems/proc.txt.

thanks,

Right. I forgot to do that. Will send out v2 to fix that.

-Longman

#3 lore raw
In reply to Waiman Long
Quoted 13 lines · On Thu, 19 Apr 2018 13:09:29 -0400 Waiman Long <[email protected]> wrote:

It was found that reading /proc/stat could be time consuming on systems with a lot of irqs. For example, reading /proc/stat in a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs. In that particular case, the majority of the CPU cycles for reading /proc/stat was spent in the kstat_irqs() function. Therefore, application performance can be impacted if the application reads /proc/stat rather frequently.

The "intr" line within /proc/stat contains a sum total of all the irqs that have happened followed by a list of irq counts for each individual irq number. In many cases, the first number is good enough. The individual irq counts may not provide that much more information.

Quoted 6 lines

In order to avoid this kind of performance issue, all these individual irq counts are now separated into a new /proc/stat_irqs file. The sum total irq count will stay in /proc/stat and be duplicated in /proc/stat_irqs. Applications that need to look up individual irq counts will now have to look into /proc/stat_irqs instead of /proc/stat.

(cc /proc maintainer)

It's a non-backward-compatible change. For something which has existing for so long, it would be a mighty task to demonstrate that no existing userspace will be disrupted by this change.

So we need to think again. A new interface which omits the per-IRQ stats might be acceptable.

Or, conceivably, a new /proc knob which disables the per-IRQ stats in /proc/stat. That would allow operators to opt in to this disabling and would avoid the need to alter whatever-application-it-is-that-is-having-trouble. This seems a bit ugly though.

Also, the changelog is rather vague. "application performance can be impacted". Well, *are* applications impacted? What is the real-world performance gain which this change provides, in a real-world workload?

#4 lore raw
Quoted 21 lines · Andrew Morton
Quoted 13 lines · On Thu, 19 Apr 2018 13:09:29 -0400 Waiman Long <[email protected]> wrote:

It was found that reading /proc/stat could be time consuming on systems with a lot of irqs. For example, reading /proc/stat in a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs. In that particular case, the majority of the CPU cycles for reading /proc/stat was spent in the kstat_irqs() function. Therefore, application performance can be impacted if the application reads /proc/stat rather frequently.

The "intr" line within /proc/stat contains a sum total of all the irqs that have happened followed by a list of irq counts for each individual irq number. In many cases, the first number is good enough. The individual irq counts may not provide that much more information.

Quoted 6 lines

In order to avoid this kind of performance issue, all these individual irq counts are now separated into a new /proc/stat_irqs file. The sum total irq count will stay in /proc/stat and be duplicated in /proc/stat_irqs. Applications that need to look up individual irq counts will now have to look into /proc/stat_irqs instead of /proc/stat.

Quoted 8 lines

(cc /proc maintainer)

It's a non-backward-compatible change. For something which has existing for so long, it would be a mighty task to demonstrate that no existing userspace will be disrupted by this change.

So we need to think again. A new interface which omits the per-IRQ stats might be acceptable.

OK, how about a new /proc/stat2 file that is the same as /proc/stat except that it omits per-IRQ stats. BTW, do you have any suggestion for a better name?

Quoted 9 lines

Or, conceivably, a new /proc knob which disables the per-IRQ stats in /proc/stat. That would allow operators to opt in to this disabling and would avoid the need to alter whatever-application-it-is-that-is-having-trouble. This seems a bit ugly though.

Also, the changelog is rather vague. "application performance can be impacted". Well, *are* applications impacted? What is the real-world performance gain which this change provides, in a real-world workload?

Will clarify that in the next version.

-Longman

#5 lore raw
In reply to Andrew Morton
Quoted 22 lines · Andrew Morton
Quoted 13 lines · On Thu, 19 Apr 2018 13:09:29 -0400 Waiman Long <[email protected]> wrote:

It was found that reading /proc/stat could be time consuming on systems with a lot of irqs. For example, reading /proc/stat in a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs. In that particular case, the majority of the CPU cycles for reading /proc/stat was spent in the kstat_irqs() function. Therefore, application performance can be impacted if the application reads /proc/stat rather frequently.

The "intr" line within /proc/stat contains a sum total of all the irqs that have happened followed by a list of irq counts for each individual irq number. In many cases, the first number is good enough. The individual irq counts may not provide that much more information.

Quoted 6 lines

In order to avoid this kind of performance issue, all these individual irq counts are now separated into a new /proc/stat_irqs file. The sum total irq count will stay in /proc/stat and be duplicated in /proc/stat_irqs. Applications that need to look up individual irq counts will now have to look into /proc/stat_irqs instead of /proc/stat.

Quoted 8 lines

(cc /proc maintainer)

It's a non-backward-compatible change. For something which has existing for so long, it would be a mighty task to demonstrate that no existing userspace will be disrupted by this change.

So we need to think again. A new interface which omits the per-IRQ stats might be acceptable.

Here is profile of open+read+close /proc/stat

30% is taking mutex only to print "0".

+   98.80%     0.04%  a.out    [kernel.vmlinux]  [k] entry_SYSCALL_64                                                                            ▒
+   98.75%     0.10%  a.out    [kernel.vmlinux]  [k] do_syscall_64                                                                               ▒
+   95.56%     0.04%  a.out    libc-2.25.so      [.] __GI___libc_read                                                                            ◆
+   95.09%     0.01%  a.out    [kernel.vmlinux]  [k] sys_read                                                                                    ▒
+   95.04%     0.03%  a.out    [kernel.vmlinux]  [k] vfs_read                                                                                    ▒
+   94.98%     0.05%  a.out    [kernel.vmlinux]  [k] proc_reg_read                                                                               ▒
+   94.98%     0.00%  a.out    [kernel.vmlinux]  [k] __vfs_read                                                                                  ▒
+   94.92%     0.06%  a.out    [kernel.vmlinux]  [k] seq_read                                                                                    ▒
+   94.52%     3.65%  a.out    [kernel.vmlinux]  [k] show_stat                                                                                   ▒
+   48.62%     2.59%  a.out    [kernel.vmlinux]  [k] kstat_irqs_usr                                                                              ▒
+   33.52%     9.55%  a.out    [kernel.vmlinux]  [k] seq_put_decimal_ull                                                                         ▒
+   19.63%    19.59%  a.out    [kernel.vmlinux]  [k] memcpy_erms                                                                                 ▒
+   17.34%     9.53%  a.out    [kernel.vmlinux]  [k] kstat_irqs                                                                                  ▒
-   15.45%    15.43%  a.out    [kernel.vmlinux]  [k] mutex_lock                                                                                  ▒
     15.43% __GI___libc_read                                                                                                                     ▒
        entry_SYSCALL_64                                                                                                                         ▒
        do_syscall_64                                                                                                                            ▒
        sys_read                                                                                                                                 ▒
        vfs_read                                                                                                                                 ▒
        __vfs_read                                                                                                                               ▒
        proc_reg_read                                                                                                                            ▒
      - seq_read                                                                                                                                 ▒
         - 15.41% show_stat                                                                                                                      ▒
              kstat_irqs_usr                                                                                                                     ▒
              mutex_lock                                                                                                                         ▒
+   13.32%    13.27%  a.out    [kernel.vmlinux]  [k] mutex_unlock                                                                                ▒
+    4.60%     1.35%  a.out    [kernel.vmlinux]  [k] cpumask_next                                                                                ▒
+    3.03%     3.03%  a.out    [kernel.vmlinux]  [k] __radix_tree_lookup                                                                         ▒
+    2.96%     0.08%  a.out    [kernel.vmlinux]  [k] seq_printf                                                                                  ▒
+    2.92%     0.02%  a.out    libc-2.25.so      [.] __GI___libc_open                                                                            ▒
+    2.89%     0.07%  a.out    [kernel.vmlinux]  [k] seq_vprintf                                                                                 ▒
+    2.81%     0.70%  a.out    [kernel.vmlinux]  [k] vsnprintf                                                                                   ▒
+    2.66%     2.66%  a.out    [kernel.vmlinux]  [k] _find_next_bit                                                                              ▒
+    2.42%     1.36%  a.out    [kernel.vmlinux]  [k] num_to_str                                                                                  ▒
+    2.41%     0.19%  a.out    [kernel.vmlinux]  [k] get_idle_time                                                                               ▒

+ 2.39% 0.02% a.out [kernel.vmlinux] [k] do_sys_open