i386: turn off l3-cache property by default

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

So switch l3-cache off by default, and add a compat clause for the range of machine types where it was on.

Signed-off-by Denis Plotnikov <[email protected]>

Reviewed-by Roman Kagan <[email protected]>

---
 include/hw/i386/pc.h | 7 ++++++-
 target/i386/cpu.c    | 2 +-
 2 files changed, 7 insertions(+), 2 deletions(-)

include/hw/i386/pc.h +6 -1

index 087d184..1d2dcae 100644
@@ -375,7 +375,12 @@ bool e820_get_entry(int, uint32_t, uint64_t *, uint64_t *);
         .driver   = TYPE_X86_CPU,\
         .property = "x-hv-max-vps",\
         .value    = "0x40",\
-    },
+    },\
+    {\
+        .driver   = TYPE_X86_CPU,\
+        .property = "l3-cache",\
+        .value    = "on",\
+    },\
 
 #define PC_COMPAT_2_9 \
     HW_COMPAT_2_9 \

target/i386/cpu.c +1 -1

index 1edcf29..95a51bd 100644
@@ -4154,7 +4154,7 @@ static Property x86_cpu_properties[] = {
     DEFINE_PROP_STRING("hv-vendor-id", X86CPU, hyperv_vendor_id),
     DEFINE_PROP_BOOL("cpuid-0xb", X86CPU, enable_cpuid_0xb, true),
     DEFINE_PROP_BOOL("lmce", X86CPU, enable_lmce, false),
-    DEFINE_PROP_BOOL("l3-cache", X86CPU, enable_l3_cache, true),
+    DEFINE_PROP_BOOL("l3-cache", X86CPU, enable_l3_cache, false),
     DEFINE_PROP_BOOL("kvm-no-smi-migration", X86CPU, kvm_no_smi_migration,
                      false),
     DEFINE_PROP_BOOL("vmware-cpuid-freq", X86CPU, vmware_cpuid_freq, true),

Signature · 1 line

2.7.4

Quoted 9 lines

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

Quoted 8 lines

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

Quoted 9 lines · Michael S. Tsirkin

Quoted 9 lines

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

Quoted 8 lines

Quoted 8 lines

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

Hi Denis,

thanks for the report. I think there are two cases:

1) The dedicated pCPU case: do you still get the performance degradation with dedicated pCPUs?

2) The non-dedicated pCPU case: do you still get the performance degradation with threads=1? If not, why do you have sibling vCPUs at all, if you don't have a dedicated physical CPU for each vCPU?

Quoted 9 lines · Michael S. Tsirkin

Quoted 9 lines

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

Quoted 8 lines

Quoted 8 lines

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

Hi Denis,

thanks for the report. I think there are two cases:

1) The dedicated pCPU case: do you still get the performance degradation with dedicated pCPUs?

2) The non-dedicated pCPU case: do you still get the performance degradation with threads=1? If not, why do you have sibling vCPUs at all, if you don't have a dedicated physical CPU for each vCPU?

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

So switch l3-cache off by default, and add a compat clause for the range of machine types where it was on.

Signed-off-by Denis Plotnikov <[email protected]>

Reviewed-by Roman Kagan <[email protected]>

---
 include/hw/i386/pc.h | 7 ++++++-
 target/i386/cpu.c    | 2 +-
 2 files changed, 7 insertions(+), 2 deletions(-)

include/hw/i386/pc.h +6 -1

index 087d184..1d2dcae 100644
@@ -375,7 +375,12 @@ bool e820_get_entry(int, uint32_t, uint64_t *, uint64_t *);
         .driver   = TYPE_X86_CPU,\
         .property = "x-hv-max-vps",\
         .value    = "0x40",\
-    },
+    },\
+    {\
+        .driver   = TYPE_X86_CPU,\
+        .property = "l3-cache",\
+        .value    = "on",\
+    },\
 
 #define PC_COMPAT_2_9 \
     HW_COMPAT_2_9 \

target/i386/cpu.c +1 -1

index 1edcf29..95a51bd 100644
@@ -4154,7 +4154,7 @@ static Property x86_cpu_properties[] = {
     DEFINE_PROP_STRING("hv-vendor-id", X86CPU, hyperv_vendor_id),
     DEFINE_PROP_BOOL("cpuid-0xb", X86CPU, enable_cpuid_0xb, true),
     DEFINE_PROP_BOOL("lmce", X86CPU, enable_lmce, false),
-    DEFINE_PROP_BOOL("l3-cache", X86CPU, enable_l3_cache, true),
+    DEFINE_PROP_BOOL("l3-cache", X86CPU, enable_l3_cache, false),
     DEFINE_PROP_BOOL("kvm-no-smi-migration", X86CPU, kvm_no_smi_migration,
                      false),
     DEFINE_PROP_BOOL("vmware-cpuid-freq", X86CPU, vmware_cpuid_freq, true),

Signature · 1 line

2.7.4

Hi,

Quoted 34 lines · Denis Plotnikov

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

Quoted 9 lines

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

There's one thing I don't understand in your test case: if you just found out that Linux will behave worse if it assumes that the VCPUs are sharing a L3 cache, why are you configuring a 8-core VCPU topology explicitly?

Do you still see a difference in the numbers if you use "-smp 8" with no "cores" and "threads" options?

Quoted 51 lines · Eduardo Habkost

Hi,

Quoted 34 lines · Denis Plotnikov

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

Quoted 9 lines

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

There's one thing I don't understand in your test case: if you just found out that Linux will behave worse if it assumes that the VCPUs are sharing a L3 cache, why are you configuring a 8-core VCPU topology explicitly?

Quoted 3 lines

Do you still see a difference in the numbers if you use "-smp 8" with no "cores" and "threads" options?

This is quite simple. A lot of software licenses are bound to the amount of CPU __sockets__. Thus it is mandatory in a lot of cases to set topology with 1 socket/xx cores to reduce the amount of money necessary to be paid for the software.

-----Original Message-----
From: Eduardo Habkost [mailto:[email protected]]
Sent: Wednesday, November 29, 2017 5:13 AM
To: Denis V. Lunev; longpeng; Michael S. Tsirkin
Cc: Denis Plotnikov; [email protected]; [email protected]; [email protected]; [email protected]; Gonglei (Arei); huangpeng; Zhaoshenglong; [email protected]
Subject: Re: [Qemu-devel] [PATCH] i386: turn off l3-cache property by default

[CCing the people who were copied in the original patch that enabled l3cache]

Quoted 4 lines · Eduardo Habkost

Quoted 3 lines

Hi,

On Fri, Nov 24, 2017 at 04:26:50PM +0300, Denis Plotnikov wrote:

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

Quoted 31 lines

Quoted 31 lines

Quoted 31 lines

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

Quoted 5 lines

Quoted 5 lines

There's one thing I don't understand in your test case: if you just found out that Linux will behave worse if it assumes that the VCPUs are sharing a L3 cache, why are you configuring a 8-core VCPU topology explicitly?

Quoted 7 lines

Quoted 3 lines

Do you still see a difference in the numbers if you use "-smp 8" with no "cores" and "threads" options?

This is quite simple. A lot of software licenses are bound to the amount of CPU __sockets__. Thus it is mandatory in a lot of cases to set topology with 1 socket/xx cores to reduce the amount of money necessary to be paid for the software.

In this case it looks like we're talking about the expected meaning of "cores=N". My first interpretation would be that the user obviously want the guest to see the multiple cores sharing a L3 cache, because that's how real CPUs normally work. But I see why you have different expectations.

Numbers on dedicated-pCPU scenarios would be helpful to guide the decision. I wouldn't like to cause a performance regression for users that fine-tuned vCPU topology and set up CPU pinning.

--
Eduardo

Quoted 11 lines · Denis V. Lunev

Quoted 10 lines · Eduardo Habkost

Quoted 9 lines · Denis Plotnikov

Quoted 8 lines

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

Yes, that's one thing.

The other reason for enabling L3 cache is the performance of accessing memory.

We tested it by Stream benchmark, the performance is better with L3-cache=on.

Quoted 31 lines

Quoted 31 lines

Quoted 31 lines

Quoted 31 lines

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

For currently public cloud providers, they usually provide different instances, Including sharing instances and dedicated instances.

And the public cloud tenants usually want the L3 cache, even bigger is better.

Basically all performance tuning target to specific scenarios, we only need to ensure benefit in most scenes.

-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Wednesday, November 29, 2017 1:56 PM
To: Gonglei (Arei)
Cc: Eduardo Habkost; Denis V. Lunev; longpeng; Michael S. Tsirkin; Denis Plotnikov; [email protected]; [email protected]; [email protected]; huangpeng; Zhaoshenglong
Subject: Re: [Qemu-devel] [PATCH] i386: turn off l3-cache property by default

Quoted 4 lines · Gonglei (Arei)

Quoted 4 lines · Denis V. Lunev

Quoted 3 lines · Eduardo Habkost

Quoted 2 lines · Denis Plotnikov

Quoted 1 line

Commit 14c985cffa "target-i386: present virtual L3 cache info for

vcpus"

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking

up

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

a task on a sibling CPU, the task was put onto the target CPU's

runqueue

Quoted 7 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

Yes, that's one thing.

The other reason for enabling L3 cache is the performance of accessing

memory.

I guess you're talking about the super-smart buffer size tuning glibc does in its memcpy and friends. We try to control that with an atomic test for memcpy, and we didn't notice a difference. We'll need to double-check...

Quoted 1 line

We tested it by Stream benchmark, the performance is better with

L3-cache=on.

This one: https://www.cs.virginia.edu/stream/ ? Thanks, we'll have a look, too.

[CCing the people who were copied in the original patch that enabled l3cache]

Quoted 19 lines · Denis V. Lunev

Quoted 18 lines · Eduardo Habkost

Hi,

Quoted 15 lines · Denis Plotnikov

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

Quoted 15 lines

Quoted 15 lines

Quoted 15 lines

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

Quoted 14 lines

Quoted 14 lines

Quoted 9 lines

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

There's one thing I don't understand in your test case: if you just found out that Linux will behave worse if it assumes that the VCPUs are sharing a L3 cache, why are you configuring a 8-core VCPU topology explicitly?

Quoted 7 lines

Quoted 3 lines

Do you still see a difference in the numbers if you use "-smp 8" with no "cores" and "threads" options?

This is quite simple. A lot of software licenses are bound to the amount of CPU __sockets__. Thus it is mandatory in a lot of cases to set topology with 1 socket/xx cores to reduce the amount of money necessary to be paid for the software.

In this case it looks like we're talking about the expected meaning of "cores=N". My first interpretation would be that the user obviously want the guest to see the multiple cores sharing a L3 cache, because that's how real CPUs normally work. But I see why you have different expectations.

Numbers on dedicated-pCPU scenarios would be helpful to guide the decision. I wouldn't like to cause a performance regression for users that fine-tuned vCPU topology and set up CPU pinning.

Quoted 23 lines · Eduardo Habkost

[CCing the people who were copied in the original patch that enabled l3cache]

Quoted 19 lines · Denis V. Lunev

Quoted 18 lines · Eduardo Habkost

Hi,

Quoted 15 lines · Denis Plotnikov

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

Agree. :)

Our testing VM is Suse11 guest with idle=poll at that time and now I realize

that Suse11 has a BUG in its scheduler.

For REHL 7.3 or upstream kernel, in ttwu_queue_remote(), a RES IPI is issued if rq->idle is not polling:
'''
static void ttwu_queue_remote(struct task_struct *p, int cpu)

{
	struct rq *rq = cpu_rq(cpu);

	if (llist_add(&p->wake_entry, &cpu_rq(cpu)->wake_list)) {
		if (!set_nr_if_polling(rq->idle))
			smp_send_reschedule(cpu);
		else
			trace_sched_wake_idle_without_ipi(cpu);
	}
}

'''

But for Suse11, it does not check, it send a RES IPI unconditionally.

Quoted 15 lines

Quoted 15 lines

Quoted 15 lines

Quoted 15 lines

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

I guess this issue could be resolved by disable the SD_WAKE_AFFINE.

As Gonglei said:
1. the L3 cache relates to the user experience.
2. the glibc would get the cache info by CPUID directly, and relates to the memory performance.

What's more, the L3 cache relates to the sched_domain which is important to the (load) balancer when system is busy.

All this doesn't mean the patch is insignificant, I just think we should do more research before decide. I'll do some tests, thanks. :)

Quoted 26 lines · Longpeng (Mike)

Quoted 23 lines · Eduardo Habkost

[CCing the people who were copied in the original patch that enabled l3cache]

Quoted 19 lines · Denis V. Lunev

Quoted 18 lines · Eduardo Habkost

Hi,

Quoted 15 lines · Denis Plotnikov

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

Quoted 3 lines

Agree. :)

Our testing VM is Suse11 guest with idle=poll at that time and now I realize

                                      ^^^^^^^^^

Oh, that's a whole lot of a difference! I wish you mentioned that in that patch.

Quoted 40 lines

that Suse11 has a BUG in its scheduler.

For REHL 7.3 or upstream kernel, in ttwu_queue_remote(), a RES IPI is issued if rq->idle is not polling:
'''
static void ttwu_queue_remote(struct task_struct *p, int cpu)

{
	struct rq *rq = cpu_rq(cpu);

	if (llist_add(&p->wake_entry, &cpu_rq(cpu)->wake_list)) {
		if (!set_nr_if_polling(rq->idle))
			smp_send_reschedule(cpu);
		else
			trace_sched_wake_idle_without_ipi(cpu);
	}
}

'''

But for Suse11, it does not check, it send a RES IPI unconditionally.

Quoted 19 lines

Quoted 19 lines

Quoted 19 lines

Quoted 19 lines

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

Quoted 1 line

I guess this issue could be resolved by disable the SD_WAKE_AFFINE.

But that requires extra tuning in the guest which is even less likely to happen in the cloud case when VM admin != host admin.

Quoted 10 lines

As Gonglei said:
1. the L3 cache relates to the user experience.
2. the glibc would get the cache info by CPUID directly, and relates to the memory performance.

What's more, the L3 cache relates to the sched_domain which is important to the (load) balancer when system is busy.

All this doesn't mean the patch is insignificant, I just think we should do more research before decide. I'll do some tests, thanks. :)

Looking forward to it, thanks!

Roman.

Quoted 23 lines · Eduardo Habkost

[CCing the people who were copied in the original patch that enabled l3cache]

Quoted 19 lines · Denis V. Lunev

Quoted 18 lines · Eduardo Habkost

Hi,

Quoted 15 lines · Denis Plotnikov

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

Agree. :)

Our testing VM is Suse11 guest with idle=poll at that time and now I realize that Suse11 has a BUG in its scheduler.

For REHL 7.3 or upstream kernel, in ttwu_queue_remote(), a RES IPI is issued if rq->idle is not polling:
'''
static void ttwu_queue_remote(struct task_struct *p, int cpu)

{
	struct rq *rq = cpu_rq(cpu);

	if (llist_add(&p->wake_entry, &cpu_rq(cpu)->wake_list)) {
		if (!set_nr_if_polling(rq->idle))
			smp_send_reschedule(cpu);
		else
			trace_sched_wake_idle_without_ipi(cpu);
	}
}

'''

But for Suse11, it does not check, it send a RES IPI unconditionally.

Quoted 16 lines

Quoted 16 lines

Quoted 16 lines

Quoted 16 lines

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

I guess this issue could be resolved by disable the SD_WAKE_AFFINE.

As Gonglei said:
1. the L3 cache relates to the user experience.

2. the glibc would get the cache info by CPUID directly, and relates to the memory performance.

What's more, the L3 cache relates to the sched_domain which is important to the (load) balancer when system is busy.

All this doesn't mean the patch is insignificant, I just think we should do more research before decide. I'll do some tests, thanks. :)

Quoted 33 lines

Quoted 15 lines

Quoted 15 lines

Quoted 10 lines

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

There's one thing I don't understand in your test case: if you just found out that Linux will behave worse if it assumes that the VCPUs are sharing a L3 cache, why are you configuring a 8-core VCPU topology explicitly?

Quoted 7 lines

Quoted 3 lines

Do you still see a difference in the numbers if you use "-smp 8" with no "cores" and "threads" options?

This is quite simple. A lot of software licenses are bound to the amount of CPU __sockets__. Thus it is mandatory in a lot of cases to set topology with 1 socket/xx cores to reduce the amount of money necessary to be paid for the software.

In this case it looks like we're talking about the expected meaning of "cores=N". My first interpretation would be that the user obviously want the guest to see the multiple cores sharing a L3 cache, because that's how real CPUs normally work. But I see why you have different expectations.

Numbers on dedicated-pCPU scenarios would be helpful to guide the decision. I wouldn't like to cause a performance regression for users that fine-tuned vCPU topology and set up CPU pinning.

Signature · 2 lines

Regards,
Longpeng(Mike)

Quoted 46 lines · Longpeng (Mike)

Quoted 23 lines · Eduardo Habkost

[CCing the people who were copied in the original patch that enabled l3cache]

Quoted 19 lines · Denis V. Lunev

Quoted 18 lines · Eduardo Habkost

Hi,

Quoted 15 lines · Denis Plotnikov

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

Agree. :)

Our testing VM is Suse11 guest with idle=poll at that time and now I realize that Suse11 has a BUG in its scheduler.

For REHL 7.3 or upstream kernel, in ttwu_queue_remote(), a RES IPI is issued if rq->idle is not polling:
'''
static void ttwu_queue_remote(struct task_struct *p, int cpu)

{
	struct rq *rq = cpu_rq(cpu);

	if (llist_add(&p->wake_entry, &cpu_rq(cpu)->wake_list)) {
		if (!set_nr_if_polling(rq->idle))
			smp_send_reschedule(cpu);
		else
			trace_sched_wake_idle_without_ipi(cpu);
	}
}

'''

Quoted 1 line

But for Suse11, it does not check, it send a RES IPI unconditionally.

So, does that mean no Linux guest benefits from the l3-cache=on default except SuSE 11 guests?

Quoted 22 lines

Quoted 17 lines

Quoted 17 lines

Quoted 17 lines

Quoted 17 lines

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

Quoted 4 lines

I guess this issue could be resolved by disable the SD_WAKE_AFFINE.

As Gonglei said:
1. the L3 cache relates to the user experience.

This is true, in a way: I have seen a fair share of user reports where they incorrectly blame the L3 cache absence or the L3 cache size for performance problems.

Quoted 2 lines

2. the glibc would get the cache info by CPUID directly, and relates to the memory performance.

I'm interested in numbers that demonstrate that.

Quoted 6 lines

What's more, the L3 cache relates to the sched_domain which is important to the (load) balancer when system is busy.

All this doesn't mean the patch is insignificant, I just think we should do more research before decide. I'll do some tests, thanks. :)

Yes, we need more data. But if we find out that there are no cases where the l3-cache=on default actually improves performance, I will be willing to apply this patch.

IMO, the long term solution is to make Linux guests not misbehave when we stop lying about the L3 cache. Maybe we could provide a "IPIs are expensive, please avoid them" hint in the KVM CPUID leaf?

Quoted 39 lines

Quoted 33 lines

Quoted 15 lines

Quoted 15 lines

Quoted 10 lines

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

There's one thing I don't understand in your test case: if you just found out that Linux will behave worse if it assumes that the VCPUs are sharing a L3 cache, why are you configuring a 8-core VCPU topology explicitly?

Quoted 7 lines

Quoted 3 lines

Do you still see a difference in the numbers if you use "-smp 8" with no "cores" and "threads" options?

This is quite simple. A lot of software licenses are bound to the amount of CPU __sockets__. Thus it is mandatory in a lot of cases to set topology with 1 socket/xx cores to reduce the amount of money necessary to be paid for the software.

In this case it looks like we're talking about the expected meaning of "cores=N". My first interpretation would be that the user obviously want the guest to see the multiple cores sharing a L3 cache, because that's how real CPUs normally work. But I see why you have different expectations.

Numbers on dedicated-pCPU scenarios would be helpful to guide the decision. I wouldn't like to cause a performance regression for users that fine-tuned vCPU topology and set up CPU pinning.

Signature · 2 lines

Regards,
Longpeng(Mike)

Quoted 42 lines · Eduardo Habkost

Quoted 41 lines · Longpeng (Mike)

Quoted 11 lines · Eduardo Habkost

Quoted 10 lines · Denis V. Lunev

Quoted 9 lines · Eduardo Habkost

Quoted 8 lines · Denis Plotnikov

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

Agree. :)

Our testing VM is Suse11 guest with idle=poll at that time and now I realize that Suse11 has a BUG in its scheduler.

For REHL 7.3 or upstream kernel, in ttwu_queue_remote(), a RES IPI is issued if rq->idle is not polling:
'''
static void ttwu_queue_remote(struct task_struct *p, int cpu)

{
	struct rq *rq = cpu_rq(cpu);

	if (llist_add(&p->wake_entry, &cpu_rq(cpu)->wake_list)) {
		if (!set_nr_if_polling(rq->idle))
			smp_send_reschedule(cpu);
		else
			trace_sched_wake_idle_without_ipi(cpu);
	}
}

'''

Quoted 5 lines

Quoted 1 line

But for Suse11, it does not check, it send a RES IPI unconditionally.

So, does that mean no Linux guest benefits from the l3-cache=on default except SuSE 11 guests?

Not only that, there is another scenario:

static void ttwu_queue(...)

{
	if (...two cpus NOT sharing L3-cache) {
		...
		ttwu_queue_remote(p, cpu, wake_flags);
		return;
	}
	...
	ttwu_do_activate(rq, p, wake_flags, &rf); <--*Here*
	...
}

In ttwu_do_activate(), there are also some opportunities with low probability to do not send RES IPI even if the target cpu isn't in IDLE polling state.

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Quoted 11 lines

Quoted 11 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

I guess this issue could be resolved by disable the SD_WAKE_AFFINE.

Quoted 11 lines

Quoted 2 lines

As Gonglei said:
1. the L3 cache relates to the user experience.

This is true, in a way: I have seen a fair share of user reports where they incorrectly blame the L3 cache absence or the L3 cache size for performance problems.

Quoted 2 lines

2. the glibc would get the cache info by CPUID directly, and relates to the memory performance.

I'm interested in numbers that demonstrate that.

Sorry I have no numbers in hand currently :(

I'll do some tests these days, please give me some time.

Quoted 10 lines

Quoted 5 lines

What's more, the L3 cache relates to the sched_domain which is important to the (load) balancer when system is busy.

All this doesn't mean the patch is insignificant, I just think we should do more research before decide. I'll do some tests, thanks. :)

Yes, we need more data. But if we find out that there are no cases where the l3-cache=on default actually improves performance, I will be willing to apply this patch.

That's a good thing if we find the truth, it's free. :)

OTOH, I think we should notice that: Linux is designed on real hardware, maybe there're some other problems if QEMU lacks some related features. If we search 'cpus_share_cache' in the Linux kernel, we can see that it's also used by Block Layer.

Quoted 4 lines

IMO, the long term solution is to make Linux guests not misbehave when we stop lying about the L3 cache. Maybe we could provide a "IPIs are expensive, please avoid them" hint in the KVM CPUID leaf?

Maybe more PV features could be digged.

Quoted 1 line · Eduardo Habkost

Quoted 4 lines

Quoted 4 lines

IMO, the long term solution is to make Linux guests not misbehave when we stop lying about the L3 cache. Maybe we could provide a "IPIs are expensive, please avoid them" hint in the KVM CPUID leaf?

We already have it, it's the hypervisor bit ;) Seriously, I'm unaware of hypervisors where IPIs aren't expensive.

Quoted 5 lines

Quoted 5 lines

IMO, the long term solution is to make Linux guests not misbehave when we stop lying about the L3 cache. Maybe we could provide a "IPIs are expensive, please avoid them" hint in the KVM CPUID leaf?

We already have it, it's the hypervisor bit ;) Seriously, I'm unaware of hypervisors where IPIs aren't expensive.

Quoted 8 lines · Roman Kagan

Quoted 5 lines

Quoted 5 lines

IMO, the long term solution is to make Linux guests not misbehave when we stop lying about the L3 cache. Maybe we could provide a "IPIs are expensive, please avoid them" hint in the KVM CPUID leaf?

We already have it, it's the hypervisor bit ;) Seriously, I'm unaware of hypervisors where IPIs aren't expensive.

In theory, AMD's AVIC should optimize IPIs to running vCPUs. Amazon's recently posted patches to disable HLT and MWAIT exits might tilt the balance in favor of IPIs even for Intel APICv (where sending the IPI is expensive, but receiving it isn't).

Being able to tie this to Amazon's other proposal, the "DEDICATED" CPUID bit, would be nice. My plan was to disable all three of MWAIT/HLT/PAUSE when setting the dedicated bit.

Quoted 63 lines · Longpeng (Mike)

Quoted 42 lines · Eduardo Habkost

Quoted 41 lines · Longpeng (Mike)

Quoted 11 lines · Eduardo Habkost

Quoted 10 lines · Denis V. Lunev

Quoted 9 lines · Eduardo Habkost

Quoted 8 lines · Denis Plotnikov

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

Agree. :)

Our testing VM is Suse11 guest with idle=poll at that time and now I realize that Suse11 has a BUG in its scheduler.

For REHL 7.3 or upstream kernel, in ttwu_queue_remote(), a RES IPI is issued if rq->idle is not polling:
'''
static void ttwu_queue_remote(struct task_struct *p, int cpu)

{
	struct rq *rq = cpu_rq(cpu);

	if (llist_add(&p->wake_entry, &cpu_rq(cpu)->wake_list)) {
		if (!set_nr_if_polling(rq->idle))
			smp_send_reschedule(cpu);
		else
			trace_sched_wake_idle_without_ipi(cpu);
	}
}

'''

Quoted 5 lines

Quoted 1 line

But for Suse11, it does not check, it send a RES IPI unconditionally.

So, does that mean no Linux guest benefits from the l3-cache=on default except SuSE 11 guests?

Not only that, there is another scenario:

static void ttwu_queue(...)

{
	if (...two cpus NOT sharing L3-cache) {
		...
		ttwu_queue_remote(p, cpu, wake_flags);
		return;
	}
	...
	ttwu_do_activate(rq, p, wake_flags, &rf); <--*Here*
	...
}

Quoted 2 lines

In ttwu_do_activate(), there are also some opportunities with low probability to do not send RES IPI even if the target cpu isn't in IDLE polling state.

Well it isn't so low actually, what you need is to keep the cpus busy switching tasks. In that case it's not uncommon that the task being woken up on a remote cpu has accumulated more vruntime than the task already running on that cpu; in that case the new task won't preempt the current task and the IPI won't be issued. E.g. on a RHEL 7.4 guest we saw:

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

The workload where it bites is mostly idle guest, with chains of dependent wakeups, i.e. with little parallelism:

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

Quoted 1 line

Quoted 1 line

Quoted 1 line

I guess this issue could be resolved by disable the SD_WAKE_AFFINE.

Actually, it's SD_WAKE_AFFINE that's being effectively defeated by this l3-cache, because the scheduler thinks that the cpus that share the last-level cache are close enough that a dependent task can be woken up on a sibling cpu.

Quoted 10 lines

Quoted 10 lines

Quoted 2 lines

As Gonglei said:
1. the L3 cache relates to the user experience.

This is true, in a way: I have seen a fair share of user reports where they incorrectly blame the L3 cache absence or the L3 cache size for performance problems.

Quoted 2 lines

2. the glibc would get the cache info by CPUID directly, and relates to the memory performance.

Quoted 1 line

Quoted 1 line

I'm interested in numbers that demonstrate that.

Me too. I vaguely remember debugging a memcpy degradation in the guest (on the Parallels proprietary hypervisor), that turned out being due a combination of l3 cache size and the cpu topology exposed to the guest, which caused glibc to choose an inadequate buffuer size.

Quoted 3 lines

Sorry I have no numbers in hand currently :(

I'll do some tests these days, please give me some time.

We'll try to get some data on this, too.

Quoted 11 lines

Quoted 10 lines

Quoted 5 lines

What's more, the L3 cache relates to the sched_domain which is important to the (load) balancer when system is busy.

All this doesn't mean the patch is insignificant, I just think we should do more research before decide. I'll do some tests, thanks. :)

Yes, we need more data. But if we find out that there are no cases where the l3-cache=on default actually improves performance, I will be willing to apply this patch.

Quoted 11 lines

That's a good thing if we find the truth, it's free. :)

OTOH, I think we should notice that: Linux is designed on real hardware, maybe there're some other problems if QEMU lacks some related features. If we search 'cpus_share_cache' in the Linux kernel, we can see that it's also used by Block Layer.

Quoted 4 lines

IMO, the long term solution is to make Linux guests not misbehave when we stop lying about the L3 cache. Maybe we could provide a "IPIs are expensive, please avoid them" hint in the KVM CPUID leaf?

We already have it, it's the hypervisor bit ;) Seriously, I'm unaware of hypervisors where IPIs aren't expensive.

Quoted 1 line

Maybe more PV features could be digged.

One problem with this is that PV features are hard to get into other guest OSes or existing Linux guests.

Roman.

[CCing the people who were copied in the original patch that enabled l3cache]

Quoted 45 lines · Eduardo Habkost

Quoted 44 lines · Denis Plotnikov

Quoted 43 lines

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

Quoted 7 lines

Quoted 7 lines

There's one thing I don't understand in your test case: if you just found out that Linux will behave worse if it assumes that the VCPUs are sharing a L3 cache, why are you configuring a 8-core VCPU topology explicitly?

Do you still see a difference in the numbers if you use "-smp 8" with no "cores" and "threads" options?

Quoted 4 lines

This is quite simple. A lot of software licenses are bound to the amount of CPU __sockets__. Thus it is mandatory in a lot of cases to set topology with 1 socket/xx cores to reduce the amount of money necessary to be paid for the software.

In this case it looks like we're talking about the expected meaning of "cores=N". My first interpretation would be that the user obviously want the guest to see the multiple cores sharing a L3 cache, because that's how real CPUs normally work. But I see why you have different expectations.

Numbers on dedicated-pCPU scenarios would be helpful to guide the decision. I wouldn't like to cause a performance regression for users that fine-tuned vCPU topology and set up CPU pinning.

Quoted 2 lines · Eduardo Habkost

[CCing the people who were copied in the original patch that enabled l3cache]

Thanks, and sorry to have forgotten to do so in the patch!

Quoted 46 lines · Denis V. Lunev

Quoted 45 lines · Eduardo Habkost

Quoted 44 lines · Denis Plotnikov

Quoted 43 lines

Commit 14c985cffa "target-i386: present virtual L3 cache info for vcpus" introduced and set by default exposing l3 to the guest.

The motivation behind it was that in the Linux scheduler, when waking up a task on a sibling CPU, the task was put onto the target CPU's runqueue directly, without sending a reschedule IPI. Reduction in the IPI count led to performance gain.

However, this isn't the whole story. Once the task is on the target CPU's runqueue, it may have to preempt the current task on that CPU, be it the idle task putting the CPU to sleep or just another running task. For that a reschedule IPI will have to be issued, too. Only when that other CPU is running a normal task for too little time, the fairness constraints will prevent the preemption and thus the IPI.

This boils down to the improvement being only achievable in workloads with many actively switching tasks. We had no access to the (proprietary?) SAP HANA benchmark the commit referred to, but the pattern is also reproduced with "perf bench sched messaging -g 1" on 1 socket, 8 cores vCPU topology, we see indeed:

l3-cache #res IPI /s #time / 10000 loops
off 560K 1.8 sec
on 40K 0.9 sec

Now there's a downside: with L3 cache the Linux scheduler is more eager to wake up tasks on sibling CPUs, resulting in unnecessary cross-vCPU interactions and therefore exessive halts and IPIs. E.g. "perf bench sched pipe -i 100000" gives

l3-cache #res IPI /s #HLT /s #time /100000 loops
off 200 (no K) 230 0.2 sec
on 400K 330K 0.5 sec

In a more realistic test, we observe 15% degradation in VM density (measured as the number of VMs, each running Drupal CMS serving 2 http requests per second to its main page, with 95%-percentile response latency under 100 ms) with l3-cache=on.

We think that mostly-idle scenario is more common in cloud and personal usage, and should be optimized for by default; users of highly loaded VMs should be able to tune them up themselves.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

There's one thing I don't understand in your test case: if you just found out that Linux will behave worse if it assumes that the VCPUs are sharing a L3 cache, why are you configuring a 8-core VCPU topology explicitly?

Do you still see a difference in the numbers if you use "-smp 8" with no "cores" and "threads" options?

No we don't. The guest scheduler makes no optimizations (which turn out pessimizations) in this case.

Quoted 11 lines

Quoted 4 lines

This is quite simple. A lot of software licenses are bound to the amount of CPU __sockets__. Thus it is mandatory in a lot of cases to set topology with 1 socket/xx cores to reduce the amount of money necessary to be paid for the software.

In this case it looks like we're talking about the expected meaning of "cores=N". My first interpretation would be that the user obviously want the guest to see the multiple cores sharing a L3 cache, because that's how real CPUs normally work. But I see why you have different expectations.

Quoted 3 lines

Numbers on dedicated-pCPU scenarios would be helpful to guide the decision. I wouldn't like to cause a performance regression for users that fine-tuned vCPU topology and set up CPU pinning.

We're speaking about the default setting; those who do fine-tuning are not going to lose the ability to configure l3-cache.

Dedicated pCPU scenarios are the opposite of density ones. Besides I'm still struggling to see why dedicated pCPUs would change anything in the guest scheduler behavior. (I hope Denis (Plotnikov) should be able to redo the measurements with dedicated pCPUs and provide the numbers regardless of my opinion though.)

Our main problem at the moment is that our users get a performance degradation in our default (arguably suboptimal) configuration when transitioning to a newer version of QEMU, that's why we're suggesting to set the default to match the pre-l3-cache behavior. (Well, they would if we didn't apply this change to our version of QEMU.)

Roman.

Quoted 5 lines

There's one thing I don't understand in your test case: if you just found out that Linux will behave worse if it assumes that the VCPUs are sharing a L3 cache, why are you configuring a 8-core VCPU topology explicitly?

Quoted 3 lines

Do you still see a difference in the numbers if you use "-smp 8" with no "cores" and "threads" options?

This is quite simple. A lot of software licenses are bound to the amount of CPU __sockets__. Thus it is mandatory in a lot of cases to set topology with 1 socket/xx cores to reduce the amount of money necessary to be paid for the software.

Quoted 8 lines · Denis V. Lunev

Quoted 8 lines

There's one thing I don't understand in your test case: if you just found out that Linux will behave worse if it assumes that the VCPUs are sharing a L3 cache, why are you configuring a 8-core VCPU topology explicitly?

Do you still see a difference in the numbers if you use "-smp 8" with no "cores" and "threads" options?

Quoted 4 lines

This is quite simple. A lot of software licenses are bound to the amount of CPU __sockets__. Thus it is mandatory in a lot of cases to set topology with 1 socket/xx cores to reduce the amount of money necessary to be paid for the software.

This answers the first question but not the second one.

If the answer to the second one is negative, then I don't understand why changing the default makes sense.

I would expect qemu by default to be as close to emulating a physical system as possible. If one has to deviate from that for some workloads, that is fine, but probably not a good default.