APIC lookups
Hi,
I've noticed that kvm_irq_delivery_to_apic() is locating the destination APIC by running through kvm_for_each_vcpu() which becomes a scalability issue with a large number if vcpus.
I'm thinking about speeding that up using a radix tree for lookups, and was wondering if it sounds right.
Signature · 1 line
Sasha.
Quoted 9 lines · Sasha Levin
Hi,
I've noticed that
kvm_irq_delivery_to_apic()is locating the destination APIC by running throughkvm_for_each_vcpu()which becomes a scalability issue with a large number if vcpus.I'm thinking about speeding that up using a radix tree for lookups, and was wondering if it sounds right.
Can also go with a simple sorted array since APIC id is limited to 8 bits, but it might need to change if we plan on supporting more than 254 vcpus in the future.
Signature · 1 line
Sasha.
Quoted 9 lines · Sasha Levin
Hi,
I've noticed that
kvm_irq_delivery_to_apic()is locating the destination APIC by running throughkvm_for_each_vcpu()which becomes a scalability issue with a large number if vcpus.I'm thinking about speeding that up using a radix tree for lookups, and was wondering if it sounds right.
We have to call kvm_apic_match_dest() on each apic to see if it should get the message. Single message can be sent to more than one apic. It is likely possible to optimize common case of physical addressing fixed destination, but then just use array of 256 elements, no need for a tree. Do you see this function in profiling?
--
Gleb.
Quoted 7 lines · Gleb Natapov
Quoted 6 lines · Sasha Levin
Hi,
I've noticed that
kvm_irq_delivery_to_apic()is locating the destination APIC by running throughkvm_for_each_vcpu()which becomes a scalability issue with a large number if vcpus.
Quoted 7 lines
Quoted 3 lines
I'm thinking about speeding that up using a radix tree for lookups, and was wondering if it sounds right.
We have to call
kvm_apic_match_dest()on each apic to see if it should get the message. Single message can be sent to more than one apic. It is likely possible to optimize common case of physical addressing fixed destination, but then just use array of 256 elements, no need for a tree.
I think it's also possible to handle it for logical addressing as well, instead of a simple compare we just need to go through all the IDs that would 'and' with the dest.
Quoted 1 line
Do you see this function in profiling?
I was running profiling to see which functions get much slower during regular operation (not boot) when you run with large amount of vcpus, and this was one of them.
Though this is probably due to the method we use to find lowest priority and not the lookups themselves.
Signature · 1 line
Sasha.
Quoted 11 lines · Sasha Levin
Quoted 10 lines · Gleb Natapov
Quoted 9 lines · Sasha Levin
Hi,
I've noticed that
kvm_irq_delivery_to_apic()is locating the destination APIC by running throughkvm_for_each_vcpu()which becomes a scalability issue with a large number if vcpus.I'm thinking about speeding that up using a radix tree for lookups, and was wondering if it sounds right.
Quoted 9 lines
Quoted 4 lines
We have to call
kvm_apic_match_dest()on each apic to see if it should get the message. Single message can be sent to more than one apic. It is likely possible to optimize common case of physical addressing fixed destination, but then just use array of 256 elements, no need for a tree.I think it's also possible to handle it for logical addressing as well, instead of a simple compare we just need to go through all the IDs that would 'and' with the dest.
There are two kinds of logical addressing: flat and cluster. And I see nothing that prevents different CPUs be in different mode.
It is better to cache lookup result in irq routing entry to speedup following interrupts.
Quoted 9 lines
Quoted 1 line
Do you see this function in profiling?
I was running profiling to see which functions get much slower during regular operation (not boot) when you run with large amount of vcpus, and this was one of them.
Though this is probably due to the method we use to find lowest priority and not the lookups themselves.
Currently we round robin between all cpus on each interrupt when lowest priority delivery is used. We should do it on each N interrupts where N >> 1.
--
Gleb.
Quoted 17 lines · Gleb Natapov
Quoted 16 lines · Sasha Levin
Quoted 14 lines · Gleb Natapov
Quoted 9 lines · Sasha Levin
Hi,
I've noticed that
kvm_irq_delivery_to_apic()is locating the destination APIC by running throughkvm_for_each_vcpu()which becomes a scalability issue with a large number if vcpus.I'm thinking about speeding that up using a radix tree for lookups, and was wondering if it sounds right.
We have to call
kvm_apic_match_dest()on each apic to see if it should get the message. Single message can be sent to more than one apic. It is likely possible to optimize common case of physical addressing fixed destination, but then just use array of 256 elements, no need for a tree.
Quoted 7 lines
Quoted 4 lines
I think it's also possible to handle it for logical addressing as well, instead of a simple compare we just need to go through all the IDs that would 'and' with the dest.
There are two kinds of logical addressing: flat and cluster. And I see nothing that prevents different CPUs be in different mode.
Hm... I thought that when using logical addressing it's either flat or cluster, not both.
In that case - yes, let's skip that.
Quoted 9 lines
It is better to cache lookup result in irq routing entry to speedup following interrupts.
Quoted 6 lines
Quoted 1 line
Do you see this function in profiling?
I was running profiling to see which functions get much slower during regular operation (not boot) when you run with large amount of vcpus, and this was one of them.
Quoted 5 lines
Quoted 3 lines
Though this is probably due to the method we use to find lowest priority and not the lookups themselves.
Currently we round robin between all cpus on each interrupt when lowest priority delivery is used. We should do it on each N interrupts where N >> 1.
I'll try that and see how it improves performance.
Signature · 1 line
Sasha.