proc/stat: Separate out individual irq counts into /proc/stat_irqs
Quoted 2 lines
Therefore, application performance can be impacted if the application reads
/proc/statrather frequently.
[nods]
Text interfaces can be designed in a very stupid way.
Quoted 2 lines
For example, reading
/proc/statin a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs.
Is this top(1)? What is this application doing?
If it needs percpu usage stats, then maybe /proc/stat should be converted away from single_open() so that core seq_file code doesn't generate everything at once.
Review
Quoted 5 lines
(quoted diff) +0
-5- - /* sum again ? it could be updated? */ - for_each_irq_nr(j) - seq_put_decimal_ull(p, " ", kstat_irqs_usr(j)); -
This is direct userspace breakage.
Proper fix is to start strategic switch away from /proc. It is a fun toy but its time has come.
Quoted 11 lines · Alexey Dobriyan
Quoted 2 lines
Therefore, application performance can be impacted if the application reads
/proc/statrather frequently.[nods]
Text interfaces can be designed in a very stupid way.Quoted 2 lines
For example, reading
/proc/statin a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs.Is this top(1)? What is this application doing?
If it needs percpu usage stats, then maybe/proc/statshould be converted away fromsingle_open()so that coreseq_filecode doesn't generate everything at once.
The application is actually a database benchmarking tool used by a customer. The reading of /proc/stat is an artifact of the benchmarking tool that can actually be turned off. Without doing that, about 20% of CPU time were spent reading /proc/stat and the trashing of cachelines slowed the benchmark number quite significantly. However, I was also told that there are legitimate cases where reading /proc/stat was necessary in some of their applications.
Quoted 6 lines
Review
Quoted 5 lines
(quoted diff) +0
-5- - /* sum again ? it could be updated? */ - for_each_irq_nr(j) - seq_put_decimal_ull(p, " ", kstat_irqs_usr(j)); -This is direct userspace breakage.
Yes, I am aware of that. That is the cost of improving the performance of applications that read /proc/stat, but don't need the individual irq counts.
Quoted 3 lines
Proper fix is to start strategic switch away from /proc. It is a fun toy but its time has come.
Migration from procfs is easier said then done. Many existing customers are reluctant to do that.
-Longman
Quoted 8 lines · Waiman Long
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines
Therefore, application performance can be impacted if the application reads
/proc/statrather frequently.[nods]
Text interfaces can be designed in a very stupid way.Quoted 2 lines
For example, reading
/proc/statin a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs.
Quoted 7 lines
Quoted 4 lines
Is this top(1)? What is this application doing?
If it needs percpu usage stats, then maybe/proc/statshould be converted away fromsingle_open()so that coreseq_filecode doesn't generate everything at once.The application is actually a database benchmarking tool used by a customer.
So it probably needs lines before "intr" line.
Quoted 14 lines
The reading of
/proc/statis an artifact of the benchmarking tool that can actually be turned off. Without doing that, about 20% of CPU time were spent reading/proc/statand the trashing of cachelines slowed the benchmark number quite significantly. However, I was also told that there are legitimate cases where reading/proc/statwas necessary in some of their applications.Quoted 6 lines
Review
Quoted 5 lines
(quoted diff) +0
-5- - /* sum again ? it could be updated? */ - for_each_irq_nr(j) - seq_put_decimal_ull(p, " ", kstat_irqs_usr(j)); -This is direct userspace breakage.
Quoted 3 lines
Yes, I am aware of that. That is the cost of improving the performance of applications that read
/proc/stat, but don't need the individual irq counts.
Yeah, but all it takes is one script which cares.
I have an idea.
Maintain "maximum registered irq #", it should be much smaller than "nr_irqs":
intr 4245359 151 0 0 0 0 0 0 0 38 0 0 0 0 0 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 44330 182364 57741 0 0 0 0 0 0 0 0 85 89124 0 0 0 0 0 323360 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Or maintain array of registered irqs and iterate over them only.
I have another idea.
perf record shows mutex_lock/mutex_unlock at the top. Most of them are irq mutex not seqfile mutex as there are many more interrupts than reads. Take it once.
Quoted 61 lines · Waiman Long
Quoted 33 lines · Alexey Dobriyan
Quoted 6 lines · Waiman Long
Quoted 5 lines · Alexey Dobriyan
Quoted 2 lines
Therefore, application performance can be impacted if the application reads
/proc/statrather frequently.[nods]
Text interfaces can be designed in a very stupid way.Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
For example, reading
/proc/statin a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs.Is this top(1)? What is this application doing?
If it needs percpu usage stats, then maybe/proc/statshould be converted away fromsingle_open()so that coreseq_filecode doesn't generate everything at once.The application is actually a database benchmarking tool used by a customer.
So it probably needs lines before "intr" line.
Quoted 13 lines
The reading of
/proc/statis an artifact of the benchmarking tool that can actually be turned off. Without doing that, about 20% of CPU time were spent reading/proc/statand the trashing of cachelines slowed the benchmark number quite significantly. However, I was also told that there are legitimate cases where reading/proc/statwas necessary in some of their applications.Quoted 6 lines
Review
Quoted 5 lines
(quoted diff) +0
-5- - /* sum again ? it could be updated? */ - for_each_irq_nr(j) - seq_put_decimal_ull(p, " ", kstat_irqs_usr(j)); -This is direct userspace breakage.
Quoted 3 lines
Yes, I am aware of that. That is the cost of improving the performance of applications that read
/proc/stat, but don't need the individual irq counts.Quoted 8 lines
Yeah, but all it takes is one script which cares.
I have an idea.
Maintain "maximum registered irq #", it should be much smaller than "
nr_irqs":intr 4245359 151 0 0 0 0 0 0 0 38 0 0 0 0 0 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 44330 182364 57741 0 0 0 0 0 0 0 0 85 89124 0 0 0 0 0 323360 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Yes, that can probably help.
This is the data from the problematic skylake server:
model name : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
56 sosreport-carevalo.02076935-20180413085327/proc/stat
Interrupts: 5370
Interrupts without "0" entries: 1011There are still quite a large number of non-zero entries, though.
Quoted 1 line
Or maintain array of registered irqs and iterate over them only.
Right, we can allocate a bitmap of used irqs to do that.
Quoted 3 lines
I have another idea.
Quoted 8 lines
Quoted 4 lines
perf record shows mutex_lock/mutex_unlock at the top. Most of them are irq mutex not seqfile mutex as there are many more interrupts than reads. Take it once.
How many cpus are in your test system? In that skylake server, it was the per-cpu summing operation of the irq counts that was consuming most of the time for reading
/proc/stat. I think we can certainly try to optimize the lock taking.
It's 16x(NR_IRQS: 4352, nr_irqs: 960, preallocated irqs: 16) Given that irq registering is rare operation, maintaining sorted array of irq should be the best option.
Quoted 3 lines
For the time being, I think I am going to have a clone
/proc/stat2as suggested in my earlier email. Alternatively, I can put that somewhere in sysfs if you have a good idea of where I can put it.
sysfs is strictly one-value-per-file.
Quoted 2 lines
I will also look into ways to optimize the current per-IRQ stats handling, but it will come later.
There is always a time-honored way of ioctl(2) switching irq info off /proc supports that.
There are many options.
Quoted 73 lines · Alexey Dobriyan
Quoted 58 lines · Waiman Long
Quoted 33 lines · Alexey Dobriyan
Quoted 6 lines · Waiman Long
Quoted 5 lines · Alexey Dobriyan
Quoted 2 lines
Therefore, application performance can be impacted if the application reads
/proc/statrather frequently.[nods]
Text interfaces can be designed in a very stupid way.Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
For example, reading
/proc/statin a certain 2-socket Skylake server took about 4.6ms because it had over 5k irqs.Is this top(1)? What is this application doing?
If it needs percpu usage stats, then maybe/proc/statshould be converted away fromsingle_open()so that coreseq_filecode doesn't generate everything at once.The application is actually a database benchmarking tool used by a customer.
So it probably needs lines before "intr" line.
Quoted 13 lines
The reading of
/proc/statis an artifact of the benchmarking tool that can actually be turned off. Without doing that, about 20% of CPU time were spent reading/proc/statand the trashing of cachelines slowed the benchmark number quite significantly. However, I was also told that there are legitimate cases where reading/proc/statwas necessary in some of their applications.Quoted 6 lines
Review
Quoted 5 lines
(quoted diff) +0
-5- - /* sum again ? it could be updated? */ - for_each_irq_nr(j) - seq_put_decimal_ull(p, " ", kstat_irqs_usr(j)); -This is direct userspace breakage.
Quoted 3 lines
Yes, I am aware of that. That is the cost of improving the performance of applications that read
/proc/stat, but don't need the individual irq counts.Quoted 8 lines
Yeah, but all it takes is one script which cares.
I have an idea.
Maintain "maximum registered irq #", it should be much smaller than "
nr_irqs":intr 4245359 151 0 0 0 0 0 0 0 38 0 0 0 0 0 0 0 0 0 64 0 0 0 0 0 0 0 0 0 0 44330 182364 57741 0 0 0 0 0 0 0 0 85 89124 0 0 0 0 0 323360 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Yes, that can probably help.
This is the data from the problematic skylake server:
model name : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
56 sosreport-carevalo.02076935-20180413085327/proc/stat
Interrupts: 5370
Interrupts without "0" entries: 1011There are still quite a large number of non-zero entries, though.
Quoted 1 line
Or maintain array of registered irqs and iterate over them only.
Right, we can allocate a bitmap of used irqs to do that.
Quoted 2 lines
I have another idea.
Quoted 8 lines
Quoted 4 lines
perf record shows mutex_lock/mutex_unlock at the top. Most of them are irq mutex not seqfile mutex as there are many more interrupts than reads. Take it once.
How many cpus are in your test system? In that skylake server, it was the per-cpu summing operation of the irq counts that was consuming most of the time for reading
/proc/stat. I think we can certainly try to optimize the lock taking.It's 16x(NR_IRQS: 4352,
nr_irqs: 960, preallocated irqs: 16) Given that irq registering is rare operation, maintaining sorted array of irq should be the best option.Quoted 3 lines
For the time being, I think I am going to have a clone
/proc/stat2as suggested in my earlier email. Alternatively, I can put that somewhere in sysfs if you have a good idea of where I can put it.
Quoted 8 lines
sysfs is strictly one-value-per-file.
Quoted 2 lines
I will also look into ways to optimize the current per-IRQ stats handling, but it will come later.
There is always a time-honored way of ioctl(2) switching irq info off /proc supports that.
There are many options.
OK, it is good to know. Do you have any existing code snippet in the kernel that I can use as reference on how to use ioctl(2) switching?
I will look into how to optimize the existing per-IRQ stats code first before venturing into cloning /proc/stat.
Cheers,
Longman
Quoted 21 lines · Alexey Dobriyan
Quoted 21 lines
Yes, that can probably help.
This is the data from the problematic skylake server:
model name : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
56 sosreport-carevalo.02076935-20180413085327/proc/stat
Interrupts: 5370
Interrupts without "0" entries: 1011There are still quite a large number of non-zero entries, though.
Quoted 1 line
Or maintain array of registered irqs and iterate over them only.
Right, we can allocate a bitmap of used irqs to do that.
Quoted 6 lines
I have another idea.
perf record shows mutex_lock/mutex_unlock at the top. Most of them are irq mutex not seqfile mutex as there are many more interrupts than reads. Take it once.
Quoted 8 lines
Quoted 4 lines
How many cpus are in your test system? In that skylake server, it was the per-cpu summing operation of the irq counts that was consuming most of the time for reading
/proc/stat. I think we can certainly try to optimize the lock taking.It's 16x(NR_IRQS: 4352,
nr_irqs: 960, preallocated irqs: 16) Given that irq registering is rare operation, maintaining sorted array of irq should be the best option.
BTW, the skylake server is 2-socket 24-core 48-thread.
Cheers,
Longman
Hi,
Quoted 26 lines
Quoted 26 lines
Quoted 5 lines
Quoted 1 line
Or maintain array of registered irqs and iterate over them only.
Right, we can allocate a bitmap of used irqs to do that.
Quoted 2 lines
I have another idea.
Quoted 8 lines
Quoted 4 lines
perf record shows mutex_lock/mutex_unlock at the top. Most of them are irq mutex not seqfile mutex as there are many more interrupts than reads. Take it once.
How many cpus are in your test system? In that skylake server, it was the per-cpu summing operation of the irq counts that was consuming most of the time for reading
/proc/stat. I think we can certainly try to optimize the lock taking.It's 16x(NR_IRQS: 4352,
nr_irqs: 960, preallocated irqs: 16) Given that irq registering is rare operation, maintaining sorted array of irq should be the best option.Quoted 3 lines
For the time being, I think I am going to have a clone
/proc/stat2as suggested in my earlier email. Alternatively, I can put that somewhere in sysfs if you have a good idea of where I can put it.sysfs is strictly one-value-per-file.
Quoted 2 lines
I will also look into ways to optimize the current per-IRQ stats handling, but it will come later.
There is always a time-honored way of ioctl(2) switching irq info off /proc supports that.
Quoted 7 lines
Quoted 1 line
There are many options.
OK, it is good to know. Do you have any existing code snippet in the kernel that I can use as reference on how to use ioctl(2) switching?
I will look into how to optimize the existing per-IRQ stats code first before venturing into cloning
/proc/stat.
Can we not just remove per-IRQ stats from /proc/stat (since I gather from this discussion it isn't scalable), and just have applications that need per-IRQ stats use /proc/interrupts ?
thanks,
- Joel
Quoted 3 lines · Joel Fernandes (Google)
Can we not just remove per-IRQ stats from
/proc/stat(since I gather from this discussion it isn't scalable), and just have applications that need per-IRQ stats use/proc/interrupts?
If you can prove noone is using them in /proc/stat...
Quoted 6 lines · Alexey Dobriyan
Quoted 3 lines · Joel Fernandes (Google)
Can we not just remove per-IRQ stats from
/proc/stat(since I gather from this discussion it isn't scalable), and just have applications that need per-IRQ stats use/proc/interrupts?If you can prove noone is using them in
/proc/stat...
And you can't even stick WARN into /proc/stat to find out.
Quoted 9 lines · Alexey Dobriyan
Quoted 6 lines · Joel Fernandes (Google)
Quoted 3 lines
Can we not just remove per-IRQ stats from
/proc/stat(since I gather from this discussion it isn't scalable), and just have applications that need per-IRQ stats use/proc/interrupts?If you can prove noone is using them in
/proc/stat...And you can't even stick WARN into
/proc/statto find out.
FWIW, removing per irq counts from /proc/stat would break some of our scripts. We could adapt to that, but everybody else would have to as well, so I'm afraid it's not going to be possible.
It would probably be better to extract out the stats that you're actually interested in to a new file.
Quoted 7 lines · David Rientjes
Quoted 5 lines · Alexey Dobriyan
Quoted 5 lines · Joel Fernandes (Google)
Quoted 3 lines
Can we not just remove per-IRQ stats from
/proc/stat(since I gather from this discussion it isn't scalable), and just have applications that need per-IRQ stats use/proc/interrupts?
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
If you can prove noone is using them in
/proc/stat...And you can't even stick WARN into
/proc/statto find out.FWIW, removing per irq counts from
/proc/statwould break some of our scripts. We could adapt to that, but everybody else would have to as well, so I'm afraid it's not going to be possible.
Excellent!
Quoted 2 lines
It would probably be better to extract out the stats that you're actually interested in to a new file.
This is the worst scenario. Individual IRQ stats are going to live in 2 places. And /proc/stat still would be slow.
Quoted 17 lines · On Tue, 24 Apr 2018 09:18:59 +0300 Alexey Dobriyan <[email protected]> wrote:
Quoted 7 lines · David Rientjes
Quoted 5 lines · Alexey Dobriyan
Quoted 5 lines · Joel Fernandes (Google)
Quoted 3 lines
Can we not just remove per-IRQ stats from
/proc/stat(since I gather from this discussion it isn't scalable), and just have applications that need per-IRQ stats use/proc/interrupts?Quoted 8 lines
Quoted 4 lines
Quoted 1 line
If you can prove noone is using them in
/proc/stat...And you can't even stick WARN into
/proc/statto find out.FWIW, removing per irq counts from
/proc/statwould break some of our scripts. We could adapt to that, but everybody else would have to as well, so I'm afraid it's not going to be possible.
Quoted 7 lines
Excellent!
Quoted 2 lines
It would probably be better to extract out the stats that you're actually interested in to a new file.
This is the worst scenario. Individual IRQ stats are going to live in 2 places. And
/proc/statstill would be slow.
No, a new /proc/stat2 which omits the `intr' line would be fast(er). Although if we're going to do such a thing we might choose to reorganize and prune /proc/stat2 even further, and actually put some thought into it - the current one is a bit of a dog's breakfast.
Dumb question(s):
a) if we moved the `intr' line to the very end of /proc/stat, would anything break? Maybe.
b) if an application were then to read stuff from /proc/stat and were to stop reading before it read the `intr' line, would its read from /proc/stat still be slow?
c) if the answer to b) is "yes" then can we change that? Only go off and prepare the `intr' line if the application is really truly reading it in?