Skip to contentSkip to conversation

proc/stat: Separate out individual irq counts into /proc/stat_irqs

PATCH Alexey Dobriyan 5 participants 12 messages Apr 19, 2018 – May 2, 2018 1 message not in the archive

lore
Share & tools
lore
#0 lore raw
Quoted 2 lines

Therefore, application performance can be impacted if the application reads /proc/stat rather frequently.

[nods]
Text interfaces can be designed in a very stupid way.

Quoted 2 lines

For example, reading /proc/stat in a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs.

Is this top(1)? What is this application doing?
If it needs percpu usage stats, then maybe /proc/stat should be converted away from single_open() so that core seq_file code doesn't generate everything at once.

Review

Quoted 5 lines
(quoted diff) +0 -5
-
-	/* sum again ? it could be updated? */
-	for_each_irq_nr(j)
-		seq_put_decimal_ull(p, " ", kstat_irqs_usr(j));
-

This is direct userspace breakage.

Proper fix is to start strategic switch away from /proc. It is a fun toy but its time has come.

#1 lore raw
Quoted 11 lines · Alexey Dobriyan
Quoted 2 lines

Therefore, application performance can be impacted if the application reads /proc/stat rather frequently.

[nods]
Text interfaces can be designed in a very stupid way.

Quoted 2 lines

For example, reading /proc/stat in a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs.

Is this top(1)? What is this application doing?
If it needs percpu usage stats, then maybe /proc/stat should be converted away from single_open() so that core seq_file code doesn't generate everything at once.

The application is actually a database benchmarking tool used by a customer. The reading of /proc/stat is an artifact of the benchmarking tool that can actually be turned off. Without doing that, about 20% of CPU time were spent reading /proc/stat and the trashing of cachelines slowed the benchmark number quite significantly. However, I was also told that there are legitimate cases where reading /proc/stat was necessary in some of their applications.

Quoted 6 lines

Review

Quoted 5 lines
(quoted diff) +0 -5
-
-	/* sum again ? it could be updated? */
-	for_each_irq_nr(j)
-		seq_put_decimal_ull(p, " ", kstat_irqs_usr(j));
-

This is direct userspace breakage.

Yes, I am aware of that. That is the cost of improving the performance of applications that read /proc/stat, but don't need the individual irq counts.

Quoted 3 lines

Proper fix is to start strategic switch away from /proc. It is a fun toy but its time has come.

Migration from procfs is easier said then done. Many existing customers are reluctant to do that.

-Longman

#2 lore raw
Quoted 8 lines · Waiman Long
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines

Therefore, application performance can be impacted if the application reads /proc/stat rather frequently.

[nods]
Text interfaces can be designed in a very stupid way.

Quoted 2 lines

For example, reading /proc/stat in a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs.

Quoted 7 lines
Quoted 4 lines

Is this top(1)? What is this application doing?
If it needs percpu usage stats, then maybe /proc/stat should be converted away from single_open() so that core seq_file code doesn't generate everything at once.

The application is actually a database benchmarking tool used by a customer.

So it probably needs lines before "intr" line.

Quoted 14 lines

The reading of /proc/stat is an artifact of the benchmarking tool that can actually be turned off. Without doing that, about 20% of CPU time were spent reading /proc/stat and the trashing of cachelines slowed the benchmark number quite significantly. However, I was also told that there are legitimate cases where reading /proc/stat was necessary in some of their applications.

Quoted 6 lines

Review

Quoted 5 lines
(quoted diff) +0 -5
-
-	/* sum again ? it could be updated? */
-	for_each_irq_nr(j)
-		seq_put_decimal_ull(p, " ", kstat_irqs_usr(j));
-

This is direct userspace breakage.

Quoted 3 lines

Yes, I am aware of that. That is the cost of improving the performance of applications that read /proc/stat, but don't need the individual irq counts.

Yeah, but all it takes is one script which cares.

I have an idea.

Maintain "maximum registered irq #", it should be much smaller than "nr_irqs":

intr 4245359 151 0 0 0 0 0 0 0 38 0 0 0 0 0 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 44330 182364 57741 0 0 0 0 0 0 0 0 85 89124 0 0 0 0 0 323360 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Or maintain array of registered irqs and iterate over them only.

I have another idea.

perf record shows mutex_lock/mutex_unlock at the top. Most of them are irq mutex not seqfile mutex as there are many more interrupts than reads. Take it once.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 61 lines · Waiman Long
Quoted 33 lines · Alexey Dobriyan
Quoted 6 lines · Waiman Long
Quoted 5 lines · Alexey Dobriyan
Quoted 2 lines

Therefore, application performance can be impacted if the application reads /proc/stat rather frequently.

[nods]
Text interfaces can be designed in a very stupid way.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

For example, reading /proc/stat in a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs.

Is this top(1)? What is this application doing?
If it needs percpu usage stats, then maybe /proc/stat should be converted away from single_open() so that core seq_file code doesn't generate everything at once.

The application is actually a database benchmarking tool used by a customer.

So it probably needs lines before "intr" line.

Quoted 13 lines

The reading of /proc/stat is an artifact of the benchmarking tool that can actually be turned off. Without doing that, about 20% of CPU time were spent reading /proc/stat and the trashing of cachelines slowed the benchmark number quite significantly. However, I was also told that there are legitimate cases where reading /proc/stat was necessary in some of their applications.

Quoted 6 lines

Review

Quoted 5 lines
(quoted diff) +0 -5
-
-	/* sum again ? it could be updated? */
-	for_each_irq_nr(j)
-		seq_put_decimal_ull(p, " ", kstat_irqs_usr(j));
-

This is direct userspace breakage.

Quoted 3 lines

Yes, I am aware of that. That is the cost of improving the performance of applications that read /proc/stat, but don't need the individual irq counts.

Quoted 8 lines

Yeah, but all it takes is one script which cares.

I have an idea.

Maintain "maximum registered irq #", it should be much smaller than "nr_irqs":

intr 4245359 151 0 0 0 0 0 0 0 38 0 0 0 0 0 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 44330 182364 57741 0 0 0 0 0 0 0 0 85 89124 0 0 0 0 0 323360 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Yes, that can probably help.

This is the data from the problematic skylake server:

model name : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
56 sosreport-carevalo.02076935-20180413085327/proc/stat
Interrupts: 5370
Interrupts without "0" entries: 1011

There are still quite a large number of non-zero entries, though.

Quoted 1 line

Or maintain array of registered irqs and iterate over them only.

Right, we can allocate a bitmap of used irqs to do that.

Quoted 3 lines

I have another idea.

Quoted 8 lines
Quoted 4 lines

perf record shows mutex_lock/mutex_unlock at the top. Most of them are irq mutex not seqfile mutex as there are many more interrupts than reads. Take it once.

How many cpus are in your test system? In that skylake server, it was the per-cpu summing operation of the irq counts that was consuming most of the time for reading /proc/stat. I think we can certainly try to optimize the lock taking.

It's 16x(NR_IRQS: 4352, nr_irqs: 960, preallocated irqs: 16) Given that irq registering is rare operation, maintaining sorted array of irq should be the best option.

Quoted 3 lines

For the time being, I think I am going to have a clone /proc/stat2 as suggested in my earlier email. Alternatively, I can put that somewhere in sysfs if you have a good idea of where I can put it.

sysfs is strictly one-value-per-file.

Quoted 2 lines

I will also look into ways to optimize the current per-IRQ stats handling, but it will come later.

There is always a time-honored way of ioctl(2) switching irq info off /proc supports that.

There are many options.

#5 lore raw
Quoted 73 lines · Alexey Dobriyan
Quoted 58 lines · Waiman Long
Quoted 33 lines · Alexey Dobriyan
Quoted 6 lines · Waiman Long
Quoted 5 lines · Alexey Dobriyan
Quoted 2 lines

Therefore, application performance can be impacted if the application reads /proc/stat rather frequently.

[nods]
Text interfaces can be designed in a very stupid way.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

For example, reading /proc/stat in a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs.

Is this top(1)? What is this application doing?
If it needs percpu usage stats, then maybe /proc/stat should be converted away from single_open() so that core seq_file code doesn't generate everything at once.

The application is actually a database benchmarking tool used by a customer.

So it probably needs lines before "intr" line.

Quoted 13 lines

The reading of /proc/stat is an artifact of the benchmarking tool that can actually be turned off. Without doing that, about 20% of CPU time were spent reading /proc/stat and the trashing of cachelines slowed the benchmark number quite significantly. However, I was also told that there are legitimate cases where reading /proc/stat was necessary in some of their applications.

Quoted 6 lines

Review

Quoted 5 lines
(quoted diff) +0 -5
-
-	/* sum again ? it could be updated? */
-	for_each_irq_nr(j)
-		seq_put_decimal_ull(p, " ", kstat_irqs_usr(j));
-

This is direct userspace breakage.

Quoted 3 lines

Yes, I am aware of that. That is the cost of improving the performance of applications that read /proc/stat, but don't need the individual irq counts.

Quoted 8 lines

Yeah, but all it takes is one script which cares.

I have an idea.

Maintain "maximum registered irq #", it should be much smaller than "nr_irqs":

intr 4245359 151 0 0 0 0 0 0 0 38 0 0 0 0 0 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 44330 182364 57741 0 0 0 0 0 0 0 0 85 89124 0 0 0 0 0 323360 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Yes, that can probably help.

This is the data from the problematic skylake server:

model name : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
56 sosreport-carevalo.02076935-20180413085327/proc/stat
Interrupts: 5370
Interrupts without "0" entries: 1011

There are still quite a large number of non-zero entries, though.

Quoted 1 line

Or maintain array of registered irqs and iterate over them only.

Right, we can allocate a bitmap of used irqs to do that.

Quoted 2 lines

I have another idea.

Quoted 8 lines
Quoted 4 lines

perf record shows mutex_lock/mutex_unlock at the top. Most of them are irq mutex not seqfile mutex as there are many more interrupts than reads. Take it once.

How many cpus are in your test system? In that skylake server, it was the per-cpu summing operation of the irq counts that was consuming most of the time for reading /proc/stat. I think we can certainly try to optimize the lock taking.

It's 16x(NR_IRQS: 4352, nr_irqs: 960, preallocated irqs: 16) Given that irq registering is rare operation, maintaining sorted array of irq should be the best option.

Quoted 3 lines

For the time being, I think I am going to have a clone /proc/stat2 as suggested in my earlier email. Alternatively, I can put that somewhere in sysfs if you have a good idea of where I can put it.

Quoted 8 lines

sysfs is strictly one-value-per-file.

Quoted 2 lines

I will also look into ways to optimize the current per-IRQ stats handling, but it will come later.

There is always a time-honored way of ioctl(2) switching irq info off /proc supports that.

There are many options.

OK, it is good to know. Do you have any existing code snippet in the kernel that I can use as reference on how to use ioctl(2) switching?

I will look into how to optimize the existing per-IRQ stats code first before venturing into cloning /proc/stat.

Cheers,
Longman

#6 lore raw

Hi,

Quoted 26 lines
Quoted 26 lines
Quoted 5 lines
Quoted 1 line

Or maintain array of registered irqs and iterate over them only.

Right, we can allocate a bitmap of used irqs to do that.

Quoted 2 lines

I have another idea.

Quoted 8 lines
Quoted 4 lines

perf record shows mutex_lock/mutex_unlock at the top. Most of them are irq mutex not seqfile mutex as there are many more interrupts than reads. Take it once.

How many cpus are in your test system? In that skylake server, it was the per-cpu summing operation of the irq counts that was consuming most of the time for reading /proc/stat. I think we can certainly try to optimize the lock taking.

It's 16x(NR_IRQS: 4352, nr_irqs: 960, preallocated irqs: 16) Given that irq registering is rare operation, maintaining sorted array of irq should be the best option.

Quoted 3 lines

For the time being, I think I am going to have a clone /proc/stat2 as suggested in my earlier email. Alternatively, I can put that somewhere in sysfs if you have a good idea of where I can put it.

sysfs is strictly one-value-per-file.

Quoted 2 lines

I will also look into ways to optimize the current per-IRQ stats handling, but it will come later.

There is always a time-honored way of ioctl(2) switching irq info off /proc supports that.

Quoted 7 lines
Quoted 1 line

There are many options.

OK, it is good to know. Do you have any existing code snippet in the kernel that I can use as reference on how to use ioctl(2) switching?

I will look into how to optimize the existing per-IRQ stats code first before venturing into cloning /proc/stat.

Can we not just remove per-IRQ stats from /proc/stat (since I gather from this discussion it isn't scalable), and just have applications that need per-IRQ stats use /proc/interrupts ?

thanks,

- Joel

#7 lore raw
Quoted 3 lines · Joel Fernandes (Google)

Can we not just remove per-IRQ stats from /proc/stat (since I gather from this discussion it isn't scalable), and just have applications that need per-IRQ stats use /proc/interrupts ?

If you can prove noone is using them in /proc/stat...

#8 lore raw
Quoted 6 lines · Alexey Dobriyan
Quoted 3 lines · Joel Fernandes (Google)

Can we not just remove per-IRQ stats from /proc/stat (since I gather from this discussion it isn't scalable), and just have applications that need per-IRQ stats use /proc/interrupts ?

If you can prove noone is using them in /proc/stat...

And you can't even stick WARN into /proc/stat to find out.

#9 lore raw
Quoted 9 lines · Alexey Dobriyan
Quoted 6 lines · Joel Fernandes (Google)
Quoted 3 lines

Can we not just remove per-IRQ stats from /proc/stat (since I gather from this discussion it isn't scalable), and just have applications that need per-IRQ stats use /proc/interrupts ?

If you can prove noone is using them in /proc/stat...

And you can't even stick WARN into /proc/stat to find out.

FWIW, removing per irq counts from /proc/stat would break some of our scripts. We could adapt to that, but everybody else would have to as well, so I'm afraid it's not going to be possible.

It would probably be better to extract out the stats that you're actually interested in to a new file.

#10 lore raw
Quoted 7 lines · David Rientjes
Quoted 5 lines · Alexey Dobriyan
Quoted 5 lines · Joel Fernandes (Google)
Quoted 3 lines

Can we not just remove per-IRQ stats from /proc/stat (since I gather from this discussion it isn't scalable), and just have applications that need per-IRQ stats use /proc/interrupts ?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

If you can prove noone is using them in /proc/stat...

And you can't even stick WARN into /proc/stat to find out.

FWIW, removing per irq counts from /proc/stat would break some of our scripts. We could adapt to that, but everybody else would have to as well, so I'm afraid it's not going to be possible.

Excellent!

Quoted 2 lines

It would probably be better to extract out the stats that you're actually interested in to a new file.

This is the worst scenario. Individual IRQ stats are going to live in 2 places. And /proc/stat still would be slow.

#11 lore raw
Quoted 17 lines · On Tue, 24 Apr 2018 09:18:59 +0300 Alexey Dobriyan <[email protected]> wrote:
Quoted 7 lines · David Rientjes
Quoted 5 lines · Alexey Dobriyan
Quoted 5 lines · Joel Fernandes (Google)
Quoted 3 lines

Can we not just remove per-IRQ stats from /proc/stat (since I gather from this discussion it isn't scalable), and just have applications that need per-IRQ stats use /proc/interrupts ?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

If you can prove noone is using them in /proc/stat...

And you can't even stick WARN into /proc/stat to find out.

FWIW, removing per irq counts from /proc/stat would break some of our scripts. We could adapt to that, but everybody else would have to as well, so I'm afraid it's not going to be possible.

Quoted 7 lines

Excellent!

Quoted 2 lines

It would probably be better to extract out the stats that you're actually interested in to a new file.

This is the worst scenario. Individual IRQ stats are going to live in 2 places. And /proc/stat still would be slow.

No, a new /proc/stat2 which omits the `intr' line would be fast(er). Although if we're going to do such a thing we might choose to reorganize and prune /proc/stat2 even further, and actually put some thought into it - the current one is a bit of a dog's breakfast.

Dumb question(s):

a) if we moved the `intr' line to the very end of /proc/stat, would anything break? Maybe.

b) if an application were then to read stuff from /proc/stat and were to stop reading before it read the `intr' line, would its read from /proc/stat still be slow?

c) if the answer to b) is "yes" then can we change that? Only go off and prepare the `intr' line if the application is really truly reading it in?

#12 lore raw
In reply to Alexey Dobriyan
Quoted 21 lines · Alexey Dobriyan
Quoted 21 lines

Yes, that can probably help.

This is the data from the problematic skylake server:

model name : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
56 sosreport-carevalo.02076935-20180413085327/proc/stat
Interrupts: 5370
Interrupts without "0" entries: 1011

There are still quite a large number of non-zero entries, though.

Quoted 1 line

Or maintain array of registered irqs and iterate over them only.

Right, we can allocate a bitmap of used irqs to do that.

Quoted 6 lines

I have another idea.

perf record shows mutex_lock/mutex_unlock at the top. Most of them are irq mutex not seqfile mutex as there are many more interrupts than reads. Take it once.

Quoted 8 lines
Quoted 4 lines

How many cpus are in your test system? In that skylake server, it was the per-cpu summing operation of the irq counts that was consuming most of the time for reading /proc/stat. I think we can certainly try to optimize the lock taking.

It's 16x(NR_IRQS: 4352, nr_irqs: 960, preallocated irqs: 16) Given that irq registering is rare operation, maintaining sorted array of irq should be the best option.

BTW, the skylake server is 2-socket 24-core 48-thread.

Cheers,
Longman