Feedback on 4.9 performance after PTI fixes
Hi,
I managed to take a bit of time to run some more tests on PTI both native and hosted in KVM, on stable versions built with CONFIG_PAGE_TABLE_ISOLATION=y. Here it's 4.9.75, used both on the host and the VM. I could compare pti=on/off both in the host and the VM. A single CPU was exposed in the VM.
It was running on my laptop (core i7 3320M at 2.6 GHz, 3.3 GHz single core turbo).
The test was run on haproxy's ability to forward connections. The results are below :
Host | Guest | conn/s | ratio_to_host | ratio_to_VM | Notes
---------+---------+---------+---------------+--------------+----------------
pti=off | - | 27400 | 100.0% | - | host reference pti=off | pti=off | 24200 | 88.3% | 100.0% | VM reference pti=off | pti=on | 13300 | 48.5% | 55.0% | pti=on | - | 23800 | 86.9% | - | protected host pti=on | pti=off | 23100 | 84.3% | 95.5% | pti=on | pti=on | 13300 | 48.5% | 55.0% |
The ratio_to_host column shows the performance relative to the host with pti=off. The ratio_to_VM column shows the performance relative to the VM running with pti=off in a host also having pti=off (ie:
performance before upgrading the systems).
On this test we see a few things :
- the performance impact on the native host is around 13%
- the highest performance impact on VMs comes from having PTI on the guest kernel (-45%). At this point it makes no difference whether the host kernel has it or not.
- the host kernel's protection has a very limited impact on the guest system's performance (-4.5%), which is probably nice for some cloud users who might want to take the risk of turning the protection off on their VMs.
The impact inside VMs is quite big but it's not where we usuall install processes sensitive to syscall performance. I could find an even higher impact on a packet generation program dropping from 2.5 Mpps to 600kpps in the VM after the fix, but it doesn't make much sense to do this in VMs so I don't really care.
I have not yet tried the retpoline patches.
Regards,
Willy
Quoted 3 lines · Willy Tarreau
- the highest performance impact on VMs comes from having PTI on the guest kernel (-45%). At this point it makes no difference whether the host kernel has it or not.
Hi Willy,
out of curiosity, is the pcid/invpcid flags exposed to and used by your guest CPU? It might very well that the PCID optimisations are not used by the guests here, and it might be worth either checking on bare metal or with the PCID optimisations enabled.
Regards,
Signature · 1 line
Yves-Alexis
Quoted 11 lines · Yves-Alexis Perez
Quoted 3 lines · Willy Tarreau
- the highest performance impact on VMs comes from having PTI on the guest kernel (-45%). At this point it makes no difference whether the host kernel has it or not.
Hi Willy,
out of curiosity, is the pcid/invpcid flags exposed to and used by your guest CPU? It might very well that the PCID optimisations are not used by the guests here, and it might be worth either checking on bare metal or with the PCID optimisations enabled.
More details on this: https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU
Regards,
Signature · 1 line
Yves-Alexis
Hi Yves-Alexis,
Quoted 11 lines · Yves-Alexis Perez
Quoted 3 lines · Willy Tarreau
- the highest performance impact on VMs comes from having PTI on the guest kernel (-45%). At this point it makes no difference whether the host kernel has it or not.
Hi Willy,
out of curiosity, is the pcid/invpcid flags exposed to and used by your guest CPU? It might very well that the PCID optimisations are not used by the guests here, and it might be worth either checking on bare metal or with the PCID optimisations enabled.
You're totally right, I discovered during my later developments that indeed PCID is not exposed there. So we take the hit of a full TLB flush twice per syscall.
Willy
Quoted 3 lines · Willy Tarreau
You're totally right, I discovered during my later developments that indeed PCID is not exposed there. So we take the hit of a full TLB flush twice per syscall.
So I really think it might make sense to redo the tests with PCID, because the
assumptions you're basing your patch series on might actually not hold.
Regards,
Signature · 1 line
Yves-Alexis
Quoted 7 lines · Yves-Alexis Perez
Quoted 3 lines · Willy Tarreau
You're totally right, I discovered during my later developments that indeed PCID is not exposed there. So we take the hit of a full TLB flush twice per syscall.
So I really think it might make sense to redo the tests with PCID, because the
assumptions you're basing your patch series on might actually not hold.
I'll have to do it on the bare-metal server soon anyway.
Cheers,
Willy
Quoted 6 lines · Willy Tarreau
Quoted 5 lines · Yves-Alexis Perez
Quoted 3 lines · Willy Tarreau
You're totally right, I discovered during my later developments that indeed PCID is not exposed there. So we take the hit of a full TLB flush twice per syscall.
Quoted 6 lines
Quoted 2 lines
So I really think it might make sense to redo the tests with PCID, because the assumptions you're basing your patch series on might actually not hold.
I'll have to do it on the bare-metal server soon anyway.
Cheers, Willy
Today, I performed a stress test against various kernels using HAPEE 1.7-r1 (Enterprise version of HAProxy) on CentOS 7.4 and here is the executive summary:
RedHat Kernel with patches increases latency by 275%, drops capacity by 34% and completely saturates CPU resources. Kernels 4.9.75 and 4.14.12 doesn't increase latency and doesn't drop the capacity compared to 4.9.56. Kernel 4.14.12 doesn't bring significant improvements over 4.9.75. Moreover, kernel packages came with some microcode updates which did something on the CPUs during the boot process. I don't have a lot of details about those microcode updates.
Here some info about the test environment:
I used one server as a test haproxy node and configured a pool with a single server, where another haproxy server was acting as a server. I used another machine as a generator to send 80K/sec HTTP requests. I didn't have the time to set up a distributed stress test and as a result, I could only generate 80K/sec requests - A single HAProxy node can serve much more requests. The server, which was behind haproxy, served all requests from memory and without any external dependencies(Disk/Network IO). The request and the response fit in one TCP packet.
All servers were bare-metal, had Intel E5-2620v3 @ 2.40GHz (with PCID support) and connected to the same switch with 10GbE interfaces. Due to network restrictions, I couldn't use the 2nd interface and a result the traffic traveled twice from one interface. As a generator tool, I used wrk2, and I set 80K/second as maximum request rate.
I have also compared two production haproxy servers running RedHat kernels with and without patches and the server with the patches spent 50% more at CPU at system and softirq levels. Those productions server have been handling customer traffic since Friday, so I have 100% confidence in the CPU increase. I will do the same for two production servers with 4.9.56 and 4.9.75 kernels.
Moreover, I have observed similar drop of performance with RedHat kernel on different workloads, such as MySQL, cassandra and graphite. On the later, I observed a big performance win with 4.14.12.
Several people mentioned to me that kernel 4.14 should provide much better performance due to the Longer-lived TLB Entries with PCID feature. So far, I haven't seen in my haproxy test. This does not mean that it doesn't have better performance; absence of evidence is not evidence of absence :-)
All above are specific to my workload and setup; other workloads may see less or more impact on the performance.
Last but not least, I would like to take this opportunity to say a huge THANK YOU to all developers/engineers involved in making Linux more secure and safe to use.
Cheers,
Pavlos
From: Willy Tarreau
Quoted 1 line
Sent: 07 January 2018 10:19
...
Quoted 5 lines
The impact inside VMs is quite big but it's not where we usuall install processes sensitive to syscall performance. I could find an even higher impact on a packet generation program dropping from 2.5 Mpps to 600kpps in the VM after the fix, but it doesn't make much sense to do this in VMs so I don't really care.
Why not?
It will be related to the cost of sending (and probably receiving) network traffic in a VM.
This is something that is done a lot.
Maybe not packet generation, but a UDP/IP benchmark inside a VM would be sensible.
It may well be that moderate ethernet packet rates cause a massive performance drop when the host kernel has PTI enabled.
David
Hi again,
updating the table after Yves-Alexis' comment on PCID. Rerunning the test with -cpu=Haswell to enable PCID gave me much better numbers :
Quoted 14 lines · Willy Tarreau
Hi,
I managed to take a bit of time to run some more tests on PTI both native and hosted in KVM, on stable versions built with
CONFIG_PAGE_TABLE_ISOLATION=y. Here it's 4.9.75, used both on the host and the VM. I could compare pti=on/off both in the host and the VM. A single CPU was exposed in the VM.It was running on my laptop (core i7 3320M at 2.6 GHz, 3.3 GHz single core turbo).
The test was run on haproxy's ability to forward connections. The results are below :
Quoted 8 lines
Host | Guest | conn/s |
ratio_to_host|ratio_to_VM| Notes
---------+---------+---------+---------------+--------------+----------------pti=off | - | 27400 | 100.0% | - | host reference pti=off | pti=off | 24200 | 88.3% | 100.0% | VM reference pti=off | pti=on | 13300 | 48.5% | 55.0% | pti=on | - | 23800 | 86.9% | - | protected host pti=on | pti=off | 23100 | 84.3% | 95.5% | pti=on | pti=on | 13300 | 48.5% | 55.0% |
New run :
Host | Guest | conn/s | ratio | Notes
---------+---------+---------+--------+----------------
pti=off | pti=off | 23100 | 100.0% | VM reference without PTI pti=off | pti=on | 19700 | 85.2% | VM with PTI and PCID pti=off | pti=on | 12700 | 55.0% | VM with PTI without PCID
So the performance cut in half was indeed caused by the lack of PCID here. With it the impact is much less, though still important.
Willy