Skip to contentSkip to conversation

perf/x86: event scheduling cleanups

RFC PATCH In review Peter Zijlstra 7 participants 48 messages Mar 14, 2019 – Apr 3, 2019

Subsystems (2)
lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

cover perf/x86: event scheduling cleanups

These here patches are the result of chasing a bug in the TSX Force Abort patches.

The first patch is simple and fixes the issue and should be marked /urgent and backported.

The rest is the result of having a hard look at how all that event scheduling stuff worked while trying to figure out wth was going wrong.

Stephane, since you were involved with writing much of that code, can you please have a good look? I _think_ I got it, but... :-)

author maintainer
#1 lore raw

PATCH 1/8 perf/x86/intel: Fix memory corruption

Through:

validate_event()

    x86_pmu.get_event_constraints(.idx=-1)
      tfa_get_event_constraints()
        dyn_constraint()

We use cpuc->constraint_list[-1], which is an obvious out-of-bound access.

In this case, simply skip the TFA constraint code, there is no event constraint with just PMC3, therefore the code will never result in the empty set.

Fixes 400816f60c54 ("perf/x86/intel: Implement support for TSX Force Abort")
---
 arch/x86/events/intel/core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/events/intel/core.c +1 -1
@@ -3410,7 +3410,7 @@ tfa_get_event_constraints(struct cpu_hw_
 	/*
 	 * Without TFA we must not use PMC3.
 	 */
-	if (!allow_tsx_force_abort && test_bit(3, c->idxmsk)) {
+	if (!allow_tsx_force_abort && test_bit(3, c->idxmsk) && idx >= 0) {
 		c = dyn_constraint(cpuc, c, idx);
 		c->idxmsk64 &= ~(1ULL << 3);
 		c->weight--;
#2 lore raw

[tip:perf/urgent] perf/x86/intel: Fix memory corruption

perf/x86/intel: Fix memory corruption
Commit-ID:  ede271b059463731cbd6dffe55ffd70d7dbe8392
Gitweb:     https://git.kernel.org/tip/ede271b059463731cbd6dffe55ffd70d7dbe8392
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Thu, 14 Mar 2019 14:01:14 +0100
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Fri, 15 Mar 2019 12:22:51 +0100

perf/x86/intel: Fix memory corruption

Through:

validate_event()

    x86_pmu.get_event_constraints(.idx=-1)
      tfa_get_event_constraints()
        dyn_constraint()

cpuc->constraint_list[-1] is used, which is an obvious out-of-bound access.

In this case, simply skip the TFA constraint code, there is no event constraint with just PMC3, therefore the code will never result in the empty set.

---
 arch/x86/events/intel/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/events/intel/core.c +1 -1
index 35102ecdfc8d..92dfeb343a6a 100644
@@ -3410,7 +3410,7 @@ tfa_get_event_constraints(struct cpu_hw_events *cpuc, int idx,
 	/*
 	 * Without TFA we must not use PMC3.
 	 */
-	if (!allow_tsx_force_abort && test_bit(3, c->idxmsk)) {
+	if (!allow_tsx_force_abort && test_bit(3, c->idxmsk) && idx >= 0) {
 		c = dyn_constraint(cpuc, c, idx);
 		c->idxmsk64 &= ~(1ULL << 3);
 		c->weight--;
#3 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

In reply to Peter Zijlstra
Quoted 31 lines · Peter Zijlstra

Through:

validate_event()

    x86_pmu.get_event_constraints(.idx=-1)
      tfa_get_event_constraints()
        dyn_constraint()

We use cpuc->constraint_list[-1], which is an obvious out-of-bound access.

In this case, simply skip the TFA constraint code, there is no event constraint with just PMC3, therefore the code will never result in the empty set.

Fixes 400816f60c54 ("perf/x86/intel: Implement support for TSX Force Abort")
---
 arch/x86/events/intel/core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

arch/x86/events/intel/core.c tfa_get_event_constraints(struct cpu_hw_ · line 3416

Quoted 8 lines
arch/x86/events/intel/core.c +1 -1
@@ -3410,7 +3410,7 @@ tfa_get_event_constraints(struct cpu_hw_
        /*
         * Without TFA we must not use PMC3.
         */
-       if (!allow_tsx_force_abort && test_bit(3, c->idxmsk)) {
+       if (!allow_tsx_force_abort && test_bit(3, c->idxmsk) && idx >= 0) {
                c = dyn_constraint(cpuc, c, idx);
                c->idxmsk64 &= ~(1ULL << 3);
                c->weight--;

I was not cc'd on the patch that added allow_tsx_force_abort, so I will give some comments here.
If I understand the goal of the control parameter it is to turn on/off the TFA workaround and thus determine
whether or not PMC3 is available. I don't know why you would need to make this a runtime tunable. That
seems a bit dodgy. But given the code you have here right now, we have to deal with it. A sysadmin could
flip the control at any time, including when PMC3 is already in used by some events. I do not see the code
that schedules out all the events on all CPUs once PMC3 becomes unavailable. You cannot just rely on the
next context-switch or timer tick for multiplexing.

author maintainer
#4 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 7 lines · Stephane Eranian
Quoted 7 lines
arch/x86/events/intel/core.c +0 -1
@@ -3410,7 +3410,7 @@ tfa_get_event_constraints(struct cpu_hw_
        /*
         * Without TFA we must not use PMC3.
         */
-       if (!allow_tsx_force_abort && test_bit(3, c->idxmsk)) {
Quoted 8 lines

arch/x86/events/intel/core.c tfa_get_event_constraints(struct cpu_hw_ · line 3416

Quoted 6 lines
arch/x86/events/intel/core.c +1 -0
+       if (!allow_tsx_force_abort && test_bit(3, c->idxmsk) && idx >= 0) {
                c = dyn_constraint(cpuc, c, idx);
                c->idxmsk64 &= ~(1ULL << 3);
                c->weight--;

I was not cc'd on the patch that added allow_tsx_force_abort, so I

Yeah, that never was public :-( I didn't particularly like that, but that's the way it is.

Quoted 6 lines

will give some comments here.

If I understand the goal of the control parameter it is to turn on/off the TFA workaround and thus determine whether or not PMC3 is available. I don't know why you would need to make this a runtime tunable.

Not quite; the control on its own doesn't directly write the MSR. And even when the work-around is allowed, we'll not set the MSR unless there is also demand for PMC3.

It is a runtime tunable because boot parameters suck.

Quoted 6 lines

That seems a bit dodgy. But given the code you have here right now, we have to deal with it. A sysadmin could flip the control at any time, including when PMC3 is already in used by some events. I do not see the code that schedules out all the events on all CPUs once PMC3 becomes unavailable. You cannot just rely on the next context-switch or timer tick for multiplexing.

Yeah, meh. You're admin, you can 'fix' it. In practise I don't expect most people to care about the knob, and the few people that do, should be able to make it work.

#5 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 24 lines · Peter Zijlstra
Quoted 7 lines · Stephane Eranian
Quoted 7 lines
arch/x86/events/intel/core.c +0 -1
@@ -3410,7 +3410,7 @@ tfa_get_event_constraints(struct cpu_hw_
        /*
         * Without TFA we must not use PMC3.
         */
-       if (!allow_tsx_force_abort && test_bit(3, c->idxmsk)) {
Quoted 8 lines

arch/x86/events/intel/core.c tfa_get_event_constraints(struct cpu_hw_ · line 3416

Quoted 6 lines
arch/x86/events/intel/core.c +1 -0
+       if (!allow_tsx_force_abort && test_bit(3, c->idxmsk) && idx >= 0) {
                c = dyn_constraint(cpuc, c, idx);
                c->idxmsk64 &= ~(1ULL << 3);
                c->weight--;

I was not cc'd on the patch that added allow_tsx_force_abort, so I

Yeah, that never was public :-( I didn't particularly like that, but that's the way it is.

Quoted 1 line

will give some comments here.

Quoted 9 lines
Quoted 4 lines

If I understand the goal of the control parameter it is to turn on/off the TFA workaround and thus determine whether or not PMC3 is available. I don't know why you would need to make this a runtime tunable.

Not quite; the control on its own doesn't directly write the MSR. And even when the work-around is allowed, we'll not set the MSR unless there is also demand for PMC3.

Trying to understand this better here. When the workaround is enabled (tfa=0), you lose
PMC3 and transactions operate normally. When it is disabled (tfa=1), transactions are
all aborted and PMC3 is available. If you are saying that when there is a PMU event
requesting PMC3, then you need PMC3 avail, so you set the MSR so that tfa=1 forcing
all transactions to abort. But in that case, you are modifying the execution of the workload
when you are monitoring it, assuming it uses TSX. You want lowest overhead and no
modifications to how the workload operates, otherwise how representative is the data you are
collecting? I understand that there is no impact on apps not using TSX, well, except on context
switch where you have to toggle that MSR. But for workloads using TSX, there is potentially
an impact.

Quoted 9 lines

It is a runtime tunable because boot parameters suck.

Quoted 6 lines

That seems a bit dodgy. But given the code you have here right now, we have to deal with it. A sysadmin could flip the control at any time, including when PMC3 is already in used by some events. I do not see the code that schedules out all the events on all CPUs once PMC3 becomes unavailable. You cannot just rely on the next context-switch or timer tick for multiplexing.

Quoted 3 lines

Yeah, meh. You're admin, you can 'fix' it. In practise I don't expect most people to care about the knob, and the few people that do, should be able to make it work.

I don't understand how this can work reliably. You have a knob to toggle that MSR. Then, you have another one inside perf_events and then the sysadmin has to make sure nobody (incl. NMI watchdog) is using the PMU when this all happens.
How can this be a practical solution? Am I missing something here?

author maintainer
#6 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 9 lines · Stephane Eranian
Quoted 4 lines

Not quite; the control on its own doesn't directly write the MSR. And even when the work-around is allowed, we'll not set the MSR unless there is also demand for PMC3.

Trying to understand this better here. When the workaround is enabled (tfa=0), you lose PMC3 and transactions operate normally.

When it is disabled (tfa=1), transactions are all aborted and PMC3 is available.

Right, but we don't expose tfa.

Quoted 3 lines

If you are saying that when there is a PMU event requesting PMC3, then you need PMC3 avail, so you set the MSR so that tfa=1 forcing all transactions to abort.

Right, so when allow_tfa=1 (default), we only set tfa=1 when PMC3 is requested.

This has the advantage that,

TSX only workload -> works
perf 4 counteres -> works

Only when you need both of them, do you get 'trouble'.

Quoted 2 lines

But in that case, you are modifying the execution of the workload when you are monitoring it, assuming it uses TSX.

We assume you are not in fact using TSX, not a lot of code does. If you do use TSX a lot, and you don't want to interfere, you have to set allow_tfa=0 and live with one counter less.

Any which way around you turn this stone, it sucks.

Quoted 2 lines

You want lowest overhead and no modifications to how the workload operates, otherwise how representative is the data you are collecting?

Sure; but there are no good choices here. This 'fix' will break something. We figured TSX+4-counter-perf was the least common scenario.

We konw of people that rely on 4 counter being present; you want to explain to them how when doing an update their program suddently doesn't work anymore?

Or you want to default to tfa=1; but then you have to explain to those people relying on TSX why their workload stopped working.

Quoted 2 lines

I understand that there is no impact on apps not using TSX, well, except on context switch where you have to toggle that MSR.

There is no additional code in the context switch; only the perf event scheduling code prods at the MSR.

Quoted 1 line

But for workloads using TSX, there is potentially an impact.

Yes, well, if you're a TSX _and_ perf user, you now have an extra knob to play with; that's not something I can do anything about. We're forced to make a choice here.

Quoted 7 lines
Quoted 3 lines

Yeah, meh. You're admin, you can 'fix' it. In practise I don't expect most people to care about the knob, and the few people that do, should be able to make it work.

I don't understand how this can work reliably.

You have a knob to toggle that MSR.

No, we don't have this knob.

Quoted 1 line

Then, you have another one inside perf_events

Only this knob exists allow_tfa.

Quoted 2 lines

and then the sysadmin has to make sure nobody (incl. NMI watchdog) is using the PMU when this all happens.

You're very unlucky if the watchdog runs on PMC3, normally it runs on Fixed1 or something. Esp early after boot. (Remember, we schedule fixed counters first, and then general purpose counters, with a preference for lower counters).

Anyway, you can trivially switch it off if you want.

Quoted 1 line

How can this be a practical solution? Am I missing something here?

It works just fine; it is unfortunate that we have this interaction but that's not something we can do anything about. We're forced to deal with this.

But if you're a TSX+perf user, have your boot scripts do:

echo 0 > /sys/bus/event_source/devices/cpu/allow_tsx_force_abort

and you'll not use PMC3 and TSX will be 'awesome'. If you don't give a crap about TSX (most people), just boot and be happy.

If you do care about TSX+perf and want to dynamically toggle for some reason, you just have to be a little careful.

#7 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 84 lines · Peter Zijlstra
Quoted 9 lines · Stephane Eranian
Quoted 4 lines

Not quite; the control on its own doesn't directly write the MSR. And even when the work-around is allowed, we'll not set the MSR unless there is also demand for PMC3.

Trying to understand this better here. When the workaround is enabled (tfa=0), you lose PMC3 and transactions operate normally.

When it is disabled (tfa=1), transactions are all aborted and PMC3 is available.

Right, but we don't expose tfa.

Quoted 3 lines

If you are saying that when there is a PMU event requesting PMC3, then you need PMC3 avail, so you set the MSR so that tfa=1 forcing all transactions to abort.

Right, so when allow_tfa=1 (default), we only set tfa=1 when PMC3 is requested.

This has the advantage that,

TSX only workload -> works
perf 4 counteres -> works

Only when you need both of them, do you get 'trouble'.

Quoted 2 lines

But in that case, you are modifying the execution of the workload when you are monitoring it, assuming it uses TSX.

We assume you are not in fact using TSX, not a lot of code does. If you do use TSX a lot, and you don't want to interfere, you have to set allow_tfa=0 and live with one counter less.

Any which way around you turn this stone, it sucks.

Quoted 2 lines

You want lowest overhead and no modifications to how the workload operates, otherwise how representative is the data you are collecting?

Sure; but there are no good choices here. This 'fix' will break something. We figured TSX+4-counter-perf was the least common scenario.

We konw of people that rely on 4 counter being present; you want to explain to them how when doing an update their program suddently doesn't work anymore?

Or you want to default to tfa=1; but then you have to explain to those people relying on TSX why their workload stopped working.

Quoted 2 lines

I understand that there is no impact on apps not using TSX, well, except on context switch where you have to toggle that MSR.

There is no additional code in the context switch; only the perf event scheduling code prods at the MSR.

Quoted 1 line

But for workloads using TSX, there is potentially an impact.

Yes, well, if you're a TSX _and_ perf user, you now have an extra knob to play with; that's not something I can do anything about. We're forced to make a choice here.

Quoted 7 lines
Quoted 3 lines

Yeah, meh. You're admin, you can 'fix' it. In practise I don't expect most people to care about the knob, and the few people that do, should be able to make it work.

I don't understand how this can work reliably.

You have a knob to toggle that MSR.

No, we don't have this knob.

Quoted 1 line

Then, you have another one inside perf_events

Only this knob exists allow_tfa.

Quoted 2 lines

and then the sysadmin has to make sure nobody (incl. NMI watchdog) is using the PMU when this all happens.

You're very unlucky if the watchdog runs on PMC3, normally it runs on Fixed1 or something. Esp early after boot. (Remember, we schedule fixed counters first, and then general purpose counters, with a preference for lower counters).

Quoted 8 lines

Anyway, you can trivially switch it off if you want.

Quoted 1 line

How can this be a practical solution? Am I missing something here?

It works just fine; it is unfortunate that we have this interaction but that's not something we can do anything about. We're forced to deal with this.

Right now, if I do:

echo 0 > /sys/bus/event_source/devices/cpu/allow_tsx_force_abort

Then I don't have the guarantee on when there will be no abort when I return from the echo.
the MSR is accessed only on PMU scheduling. I would expect a sysadmin to want some guarantee
if this is to be switched on/off at runtime. If not, then having a boot time option is better in my opinion.

This other bit I noticed is that cpuc->tfa_shadow is used to avoid the wrmsr(), but I don't see the
code that makes sure the init value (0) matches the value of the MSR. Is this MSR guarantee to be
zero on reset? How about on kexec()?

Quoted 9 lines

But if you're a TSX+perf user, have your boot scripts do:

echo 0 > /sys/bus/event_source/devices/cpu/allow_tsx_force_abort

and you'll not use PMC3 and TSX will be 'awesome'. If you don't give a crap about TSX (most people), just boot and be happy.

If you do care about TSX+perf and want to dynamically toggle for some reason, you just have to be a little careful.

#8 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 105 lines · Stephane Eranian
Quoted 84 lines · Peter Zijlstra
Quoted 9 lines · Stephane Eranian
Quoted 4 lines

Not quite; the control on its own doesn't directly write the MSR. And even when the work-around is allowed, we'll not set the MSR unless there is also demand for PMC3.

Trying to understand this better here. When the workaround is enabled (tfa=0), you lose PMC3 and transactions operate normally.

When it is disabled (tfa=1), transactions are all aborted and PMC3 is available.

Right, but we don't expose tfa.

Quoted 3 lines

If you are saying that when there is a PMU event requesting PMC3, then you need PMC3 avail, so you set the MSR so that tfa=1 forcing all transactions to abort.

Right, so when allow_tfa=1 (default), we only set tfa=1 when PMC3 is requested.

This has the advantage that,

TSX only workload -> works
perf 4 counteres -> works

Only when you need both of them, do you get 'trouble'.

Quoted 2 lines

But in that case, you are modifying the execution of the workload when you are monitoring it, assuming it uses TSX.

We assume you are not in fact using TSX, not a lot of code does. If you do use TSX a lot, and you don't want to interfere, you have to set allow_tfa=0 and live with one counter less.

Any which way around you turn this stone, it sucks.

Quoted 2 lines

You want lowest overhead and no modifications to how the workload operates, otherwise how representative is the data you are collecting?

Sure; but there are no good choices here. This 'fix' will break something. We figured TSX+4-counter-perf was the least common scenario.

We konw of people that rely on 4 counter being present; you want to explain to them how when doing an update their program suddently doesn't work anymore?

Or you want to default to tfa=1; but then you have to explain to those people relying on TSX why their workload stopped working.

Quoted 2 lines

I understand that there is no impact on apps not using TSX, well, except on context switch where you have to toggle that MSR.

There is no additional code in the context switch; only the perf event scheduling code prods at the MSR.

Quoted 1 line

But for workloads using TSX, there is potentially an impact.

Yes, well, if you're a TSX _and_ perf user, you now have an extra knob to play with; that's not something I can do anything about. We're forced to make a choice here.

Quoted 7 lines
Quoted 3 lines

Yeah, meh. You're admin, you can 'fix' it. In practise I don't expect most people to care about the knob, and the few people that do, should be able to make it work.

I don't understand how this can work reliably.

You have a knob to toggle that MSR.

No, we don't have this knob.

Quoted 1 line

Then, you have another one inside perf_events

Only this knob exists allow_tfa.

Quoted 2 lines

and then the sysadmin has to make sure nobody (incl. NMI watchdog) is using the PMU when this all happens.

You're very unlucky if the watchdog runs on PMC3, normally it runs on Fixed1 or something. Esp early after boot. (Remember, we schedule fixed counters first, and then general purpose counters, with a preference for lower counters).

Quoted 8 lines

Anyway, you can trivially switch it off if you want.

Quoted 1 line

How can this be a practical solution? Am I missing something here?

It works just fine; it is unfortunate that we have this interaction but that's not something we can do anything about. We're forced to deal with this.

Right now, if I do:

echo 0 > /sys/bus/event_source/devices/cpu/allow_tsx_force_abort

Then I don't have the guarantee on when there will be no abort when I return from the echo.
the MSR is accessed only on PMU scheduling. I would expect a sysadmin to want some guarantee
if this is to be switched on/off at runtime. If not, then having a boot time option is better in my opinion.

Quoted 6 lines

This other bit I noticed is that cpuc->tfa_shadow is used to avoid the wrmsr(), but I don't see the
code that makes sure the init value (0) matches the value of the MSR. Is this MSR guarantee to be
zero on reset? How about on kexec()?

Furthermore, depending on what is measured on each CPU, i.e., PMC3 is used or not,
the TFA may be set to true or false dynamically. So if you have a TSX workload it may
be impacted depending on which CPU it runs on. I don't think users would like that approach.

Quoted 9 lines
Quoted 9 lines

But if you're a TSX+perf user, have your boot scripts do:

echo 0 > /sys/bus/event_source/devices/cpu/allow_tsx_force_abort

and you'll not use PMC3 and TSX will be 'awesome'. If you don't give a crap about TSX (most people), just boot and be happy.

If you do care about TSX+perf and want to dynamically toggle for some reason, you just have to be a little careful.

author maintainer
#9 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

In reply to Stephane Eranian
Quoted 9 lines · Stephane Eranian

Right now, if I do:

echo 0 > /sys/bus/event_source/devices/cpu/allow_tsx_force_abort

Then I don't have the guarantee on when there will be no abort when I return from the echo. the MSR is accessed only on PMU scheduling. I would expect a sysadmin to want some guarantee if this is to be switched on/off at runtime. If not, then having a boot time option is better in my opinion.

Something like cycling the nmi watchdog or:

perf stat -a -e cycles sleep 1

should be enough to force reschedule the events on every CPU.

Again, I'm not adverse to 'fixing' this if it can be done with limited LoC. But I don't really see this as critical.

Quoted 4 lines

This other bit I noticed is that cpuc->tfa_shadow is used to avoid the wrmsr(), but I don't see the code that makes sure the init value (0) matches the value of the MSR. Is this MSR guarantee to be zero on reset?

That was my understanding.

Quoted 1 line

How about on kexec()?

Good point, we might want to fix that.

author maintainer
#10 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 26 lines · Peter Zijlstra
Quoted 9 lines · Stephane Eranian

Right now, if I do:

echo 0 > /sys/bus/event_source/devices/cpu/allow_tsx_force_abort

Then I don't have the guarantee on when there will be no abort when I return from the echo. the MSR is accessed only on PMU scheduling. I would expect a sysadmin to want some guarantee if this is to be switched on/off at runtime. If not, then having a boot time option is better in my opinion.

Something like cycling the nmi watchdog or:

perf stat -a -e cycles sleep 1

should be enough to force reschedule the events on every CPU.

Again, I'm not adverse to 'fixing' this if it can be done with limited LoC. But I don't really see this as critical.

Quoted 4 lines

This other bit I noticed is that cpuc->tfa_shadow is used to avoid the wrmsr(), but I don't see the code that makes sure the init value (0) matches the value of the MSR. Is this MSR guarantee to be zero on reset?

Quoted 5 lines

That was my understanding.

Quoted 1 line

How about on kexec()?

Good point, we might want to fix that.

Something like the below perhaps?

---
Subject: perf/x86/intel: Initialize TFA MSR

Stephane reported that we don't initialize the TFA MSR, which could lead to trouble if the RESET value is not 0 or on kexec.

---
 arch/x86/events/intel/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/events/intel/core.c +6 -0
index 8baa441d8000..2d3caf2d1384 100644
@@ -3575,6 +3575,12 @@ static void intel_pmu_cpu_starting(int cpu)
 
 	cpuc->lbr_sel = NULL;
 
+	if (x86_pmu.flags & PMU_FL_TFA) {
+		WARN_ON_ONCE(cpuc->tfa_shadow);
+		cpuc->tfa_shadow = ~0ULL;
+		intel_set_tfa(cpuc, false);
+	}
+
 	if (x86_pmu.version > 1)
 		flip_smm_bit(&x86_pmu.attr_freeze_on_smi);
 
#11 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 4 lines · Peter Zijlstra

Subject: perf/x86/intel: Initialize TFA MSR

Stephane reported that we don't initialize the TFA MSR, which could lead to trouble if the RESET value is not 0 or on kexec.

That sentence doesn't parse.

Stephane reported that the TFA MSR is not initialized by the kernel, but the TFA bit could set by firmware or as a leftover from a kexec, which makes the state inconsistent.

Quoted 11 lines
---
 arch/x86/events/intel/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/events/intel/core.c +0 -0
index 8baa441d8000..2d3caf2d1384 100644

arch/x86/events/intel/core.c static void intel_pmu_cpu_starting(int cpu) · line 3579

Quoted 5 lines
arch/x86/events/intel/core.c +2 -0
@@ -3575,6 +3575,12 @@ static void intel_pmu_cpu_starting(int cpu)
 
 	cpuc->lbr_sel = NULL;
 
+	if (x86_pmu.flags & PMU_FL_TFA) {
+		WARN_ON_ONCE(cpuc->tfa_shadow);

Hmm. I wouldn't warn here as this is a legit state for kexec/kdump and I don't think we can figure out whether this comes directly from the firmware.

Thanks,

tglx

author maintainer
#12 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 24 lines · Thomas Gleixner
Quoted 4 lines · Peter Zijlstra

Subject: perf/x86/intel: Initialize TFA MSR

Stephane reported that we don't initialize the TFA MSR, which could lead to trouble if the RESET value is not 0 or on kexec.

That sentence doesn't parse.

Stephane reported that the TFA MSR is not initialized by the kernel, but the TFA bit could set by firmware or as a leftover from a kexec, which makes the state inconsistent.

Quoted 12 lines
---
 arch/x86/events/intel/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/events/intel/core.c +0 -0
index 8baa441d8000..2d3caf2d1384 100644
@@ -3575,6 +3575,12 @@ static void intel_pmu_cpu_starting(int cpu)
 
Quoted 8 lines

arch/x86/events/intel/core.c static void intel_pmu_cpu_starting(int cpu) · line 3579

Quoted 4 lines
arch/x86/events/intel/core.c +2 -0
 	cpuc->lbr_sel = NULL;
 
+	if (x86_pmu.flags & PMU_FL_TFA) {
+		WARN_ON_ONCE(cpuc->tfa_shadow);

Hmm. I wouldn't warn here as this is a legit state for kexec/kdump and I don't think we can figure out whether this comes directly from the firmware.

Even on kexec, the cpuc will be freshly allocated and zerod I think. We only inherit hardware state, not software state.

#13 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 34 lines · Peter Zijlstra
Quoted 24 lines · Thomas Gleixner
Quoted 4 lines · Peter Zijlstra

Subject: perf/x86/intel: Initialize TFA MSR

Stephane reported that we don't initialize the TFA MSR, which could lead to trouble if the RESET value is not 0 or on kexec.

That sentence doesn't parse.

Stephane reported that the TFA MSR is not initialized by the kernel, but the TFA bit could set by firmware or as a leftover from a kexec, which makes the state inconsistent.

Quoted 12 lines
---
 arch/x86/events/intel/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/events/intel/core.c +0 -0
index 8baa441d8000..2d3caf2d1384 100644
@@ -3575,6 +3575,12 @@ static void intel_pmu_cpu_starting(int cpu)
 
Quoted 8 lines

arch/x86/events/intel/core.c static void intel_pmu_cpu_starting(int cpu) · line 3579

Quoted 4 lines
arch/x86/events/intel/core.c +2 -0
 	cpuc->lbr_sel = NULL;
 
+	if (x86_pmu.flags & PMU_FL_TFA) {
+		WARN_ON_ONCE(cpuc->tfa_shadow);

Hmm. I wouldn't warn here as this is a legit state for kexec/kdump and I don't think we can figure out whether this comes directly from the firmware.

Quoted 2 lines

Even on kexec, the cpuc will be freshly allocated and zerod I think. We only inherit hardware state, not software state.

Ouch, misread the patch of course ... What about cpu hotplug? Does that free/alloc or reuse?

Thanks,

tglx

author maintainer
#14 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 35 lines · Thomas Gleixner
Quoted 34 lines · Peter Zijlstra
Quoted 32 lines · Thomas Gleixner
Quoted 4 lines · Peter Zijlstra

Subject: perf/x86/intel: Initialize TFA MSR

Stephane reported that we don't initialize the TFA MSR, which could lead to trouble if the RESET value is not 0 or on kexec.

That sentence doesn't parse.

Stephane reported that the TFA MSR is not initialized by the kernel, but the TFA bit could set by firmware or as a leftover from a kexec, which makes the state inconsistent.

Quoted 11 lines
---
 arch/x86/events/intel/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/events/intel/core.c +0 -0
index 8baa441d8000..2d3caf2d1384 100644

arch/x86/events/intel/core.c static void intel_pmu_cpu_starting(int cpu) · line 3579

Quoted 5 lines
arch/x86/events/intel/core.c +2 -0
@@ -3575,6 +3575,12 @@ static void intel_pmu_cpu_starting(int cpu)
 
 	cpuc->lbr_sel = NULL;
 
+	if (x86_pmu.flags & PMU_FL_TFA) {
+		WARN_ON_ONCE(cpuc->tfa_shadow);

Hmm. I wouldn't warn here as this is a legit state for kexec/kdump and I don't think we can figure out whether this comes directly from the firmware.

Quoted 5 lines
Quoted 2 lines

Even on kexec, the cpuc will be freshly allocated and zerod I think. We only inherit hardware state, not software state.

Ouch, misread the patch of course ... What about cpu hotplug? Does that free/alloc or reuse?

re-use I think, but since we kill all events before taking the CPU down, it _should_ have cleared the MSR while doing that.

Might be best to have someone test this.. someone with ucode and a machine etc.. :-)

#15 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 2 lines · Peter Zijlstra

Might be best to have someone test this.. someone with ucode and a machine etc.. :-)

If your own company can't oblige you here Peter :-) I'm sure I can make some time to test it.

#16 lore raw

RE: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Is the request to check whether the msr gets reset to default upon reboot of the machine ?

-----Original Message-----
From: Tony Jones [mailto:[email protected]]
Sent: Thursday, March 21, 2019 12:43 PM
To: Peter Zijlstra <[email protected]>; Thomas Gleixner <[email protected]>
Cc: Stephane Eranian <[email protected]>; Ingo Molnar <[email protected]>; Jiri Olsa <[email protected]>; LKML <[email protected]>; DSouza, Nelson <[email protected]>
Subject: Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 2 lines · Peter Zijlstra

Might be best to have someone test this.. someone with ucode and a machine etc.. :-)

If your own company can't oblige you here Peter :-) I'm sure I can make some time to test it.

author maintainer
#17 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 1 line · Nelson

Is the request to check whether the msr gets reset to default upon reboot of the machine ?

basically:

- apply patch
- start workload with 4 counter (on all CPUs), such that tfa-msr=1
- try the following:
o cpu hotplug
o kexec

to see if the new WARN will trigger -- it should not.

#18 lore raw

RE: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Attached cpu hotplug test output while perf is running in the background. No WARN messages seen.

When I run the kexec command, it boots to bios. Haven't used kexec before. Still trying to figure that one out.

-----Original Message-----
From: Peter Zijlstra [mailto:[email protected]]
Sent: Thursday, March 21, 2019 1:07 PM
To: DSouza, Nelson <[email protected]>
Cc: Tony Jones <[email protected]>; Thomas Gleixner <[email protected]>; Stephane Eranian <[email protected]>; Ingo Molnar <[email protected]>; Jiri Olsa <[email protected]>; LKML <[email protected]>
Subject: Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 1 line · Nelson

Is the request to check whether the msr gets reset to default upon reboot of the machine ?

basically:

- apply patch
- start workload with 4 counter (on all CPUs), such that tfa-msr=1
- try the following:
o cpu hotplug
o kexec

to see if the new WARN will trigger -- it should not.

root@jf1-otc-3AR3-33:~# cat /sys/devices/cpu/allow_tsx_force_abort
1
root@jf1-otc-3AR3-33:~# perf stat -e cs,'{cache-misses,cache-references,branches,branch-misses}' --no-merge -a sleep 60 &
[1] 3025
root@jf1-otc-3AR3-33:~# echo 0 > /sys/devices/system/cpu/cpu4/online
root@jf1-otc-3AR3-33:~# echo 0 > /sys/devices/system/cpu/cpu5/online
root@jf1-otc-3AR3-33:~# echo 0 > /sys/devices/system/cpu/cpu6/online
root@jf1-otc-3AR3-33:~# echo 0 > /sys/devices/system/cpu/cpu7/online
root@jf1-otc-3AR3-33:~#
 Performance counter stats for 'system wide':
             4,800      cs
     <not counted>      cache-misses
     <not counted>      cache-references
     <not counted>      branches
     <not counted>      branch-misses
      60.009098150 seconds time elapsed

[1]+ Done perf stat -e cs,'{cache-misses,cache-references,branches,branch-misses}' --no-merge -a sleep 60

#19 lore raw

RE: [PATCH 1/8] perf/x86/intel: Fix memory corruption

kexec was borked in 5.0+ kernel I was using.

Switched to 5.0.3 stable and kexec works as expected. No warnings seen with a kexec boot.

-----Original Message-----
From: DSouza, Nelson
Sent: Thursday, March 21, 2019 4:16 PM
To: Peter Zijlstra <[email protected]>
Cc: Tony Jones <[email protected]>; Thomas Gleixner <[email protected]>; Stephane Eranian <[email protected]>; Ingo Molnar <[email protected]>; Jiri Olsa <[email protected]>; LKML <[email protected]>
Subject: RE: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Attached cpu hotplug test output while perf is running in the background. No WARN messages seen.

When I run the kexec command, it boots to bios. Haven't used kexec before. Still trying to figure that one out.

-----Original Message-----
From: Peter Zijlstra [mailto:[email protected]]
Sent: Thursday, March 21, 2019 1:07 PM
To: DSouza, Nelson <[email protected]>
Cc: Tony Jones <[email protected]>; Thomas Gleixner <[email protected]>; Stephane Eranian <[email protected]>; Ingo Molnar <[email protected]>; Jiri Olsa <[email protected]>; LKML <[email protected]>
Subject: Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 1 line · Nelson

Is the request to check whether the msr gets reset to default upon reboot of the machine ?

basically:

- apply patch
- start workload with 4 counter (on all CPUs), such that tfa-msr=1
- try the following:
o cpu hotplug
o kexec

to see if the new WARN will trigger -- it should not.

#20 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

In reply to Thomas Gleixner
Quoted 4 lines · Thomas Gleixner
Quoted 2 lines · Peter Zijlstra

Subject: perf/x86/intel: Initialize TFA MSR

Quoted 9 lines
Quoted 2 lines

Stephane reported that we don't initialize the TFA MSR, which could lead to trouble if the RESET value is not 0 or on kexec.

That sentence doesn't parse.

Stephane reported that the TFA MSR is not initialized by the kernel, but the TFA bit could set by firmware or as a leftover from a kexec, which makes the state inconsistent.

Correct. This is what I meant.
The issue is what does the kernel guarantee when it boots?

I see:
static bool allow_tsx_force_abort = true;

Therefore you must ensure the MSR is set to reflect that state on boot. So you have to force it to that value to be in sync which is what your new patch is doing.

Quoted 24 lines
Quoted 11 lines
---
 arch/x86/events/intel/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/events/intel/core.c +0 -0
index 8baa441d8000..2d3caf2d1384 100644

arch/x86/events/intel/core.c static void intel_pmu_cpu_starting(int cpu) · line 3579

Quoted 5 lines
arch/x86/events/intel/core.c +2 -0
@@ -3575,6 +3575,12 @@ static void intel_pmu_cpu_starting(int cpu)
 
      cpuc->lbr_sel = NULL;
 
+     if (x86_pmu.flags & PMU_FL_TFA) {
+             WARN_ON_ONCE(cpuc->tfa_shadow);

Hmm. I wouldn't warn here as this is a legit state for kexec/kdump and I don't think we can figure out whether this comes directly from the firmware.

Thanks,

tglx

#21 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 17 lines · Stephane Eranian
Quoted 4 lines · Thomas Gleixner
Quoted 2 lines · Peter Zijlstra

Subject: perf/x86/intel: Initialize TFA MSR

Quoted 9 lines
Quoted 2 lines

Stephane reported that we don't initialize the TFA MSR, which could lead to trouble if the RESET value is not 0 or on kexec.

That sentence doesn't parse.

Stephane reported that the TFA MSR is not initialized by the kernel, but the TFA bit could set by firmware or as a leftover from a kexec, which makes the state inconsistent.

Correct. This is what I meant.
The issue is what does the kernel guarantee when it boots?

Quoted 6 lines

I see:
static bool allow_tsx_force_abort = true;

Therefore you must ensure the MSR is set to reflect that state on boot. So you have to force it to that value to be in sync which is what your new patch is doing.

The initial state should be that the MSR TFA bit is 0. The software state is a different beast.

allow_tsx_force_abort

false Do not set MSR TFA bit (Make TSX work with PMC3) and
exclude PMC3 from being used.

true Set the MSR TFA bit when PMC3 is used by perf, clear it
when PMC3 is not longer in use.

Now, if the firmware or the kexec has the TFA bit set in the MSR and PMC3 is not in use then TSX always aborts pointlessly. It's not a fatal isseu, but it's inconsistent.

So independent of the state of allow_tsx_force_abort the kernel has to clear the MSR TSA bit when the CPUs are brought up.

The state of allow_tsx_force_abort is solely relevant after CPUs coming up to decide whether PMC3 can be used by perf or not.

Thanks,

tglx

#22 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 29 lines · Thomas Gleixner
Quoted 17 lines · Stephane Eranian
Quoted 4 lines · Thomas Gleixner
Quoted 2 lines · Peter Zijlstra

Subject: perf/x86/intel: Initialize TFA MSR

Quoted 9 lines
Quoted 2 lines

Stephane reported that we don't initialize the TFA MSR, which could lead to trouble if the RESET value is not 0 or on kexec.

That sentence doesn't parse.

Stephane reported that the TFA MSR is not initialized by the kernel, but the TFA bit could set by firmware or as a leftover from a kexec, which makes the state inconsistent.

Correct. This is what I meant.
The issue is what does the kernel guarantee when it boots?

Quoted 6 lines

I see:
static bool allow_tsx_force_abort = true;

Therefore you must ensure the MSR is set to reflect that state on boot. So you have to force it to that value to be in sync which is what your new patch is doing.

The initial state should be that the MSR TFA bit is 0. The software state is a different beast.

Quoted 8 lines

allow_tsx_force_abort

false Do not set MSR TFA bit (Make TSX work with PMC3) and
exclude PMC3 from being used.

true Set the MSR TFA bit when PMC3 is used by perf, clear it
when PMC3 is not longer in use.

I would expect this description to be included in the source code where the allow_tsx_force_abort variable is defined and somewhere in the kernel Documentation
because it is not trivial to understand what the control actually does and the guarantees
you have when you toggle it.

Quoted 14 lines

Now, if the firmware or the kexec has the TFA bit set in the MSR and PMC3 is not in use then TSX always aborts pointlessly. It's not a fatal isseu, but it's inconsistent.

So independent of the state of allow_tsx_force_abort the kernel has to clear the MSR TSA bit when the CPUs are brought up.

The state of allow_tsx_force_abort is solely relevant after CPUs coming up to decide whether PMC3 can be used by perf or not.

Thanks,

tglx

author maintainer
#23 lore raw

Re: [PATCH 1/8] perf/x86/intel: Fix memory corruption

Quoted 32 lines · Stephane Eranian
Quoted 31 lines · Thomas Gleixner
Quoted 17 lines · Stephane Eranian
Quoted 4 lines · Thomas Gleixner
Quoted 2 lines · Peter Zijlstra

Subject: perf/x86/intel: Initialize TFA MSR

Quoted 9 lines
Quoted 2 lines

Stephane reported that we don't initialize the TFA MSR, which could lead to trouble if the RESET value is not 0 or on kexec.

That sentence doesn't parse.

Stephane reported that the TFA MSR is not initialized by the kernel, but the TFA bit could set by firmware or as a leftover from a kexec, which makes the state inconsistent.

Correct. This is what I meant.
The issue is what does the kernel guarantee when it boots?

Quoted 6 lines

I see:
static bool allow_tsx_force_abort = true;

Therefore you must ensure the MSR is set to reflect that state on boot. So you have to force it to that value to be in sync which is what your new patch is doing.

The initial state should be that the MSR TFA bit is 0. The software state is a different beast.

allow_tsx_force_abort

Quoted 8 lines
Quoted 6 lines

false Do not set MSR TFA bit (Make TSX work with PMC3) and
exclude PMC3 from being used.

true Set the MSR TFA bit when PMC3 is used by perf, clear it
when PMC3 is not longer in use.

I would expect this description to be included in the source code where the allow_tsx_force_abort variable is defined

That part is easy I suppose.

Quoted 3 lines

and somewhere in the kernel Documentation because it is not trivial to understand what the control actually does and the guarantees you have when you toggle it.

But here we seem to be sorely lacking, that is, there is no sysfs/perf documentation at all.

In any case, feel free to send a patch :-)

#24 lore raw

[tip:perf/urgent] perf/x86/intel: Initialize TFA MSR

In reply to Peter Zijlstra
perf/x86/intel: Initialize TFA MSR
Commit-ID:  d7262457e35dbe239659e62654e56f8ddb814bed
Gitweb:     https://git.kernel.org/tip/d7262457e35dbe239659e62654e56f8ddb814bed
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Thu, 21 Mar 2019 13:38:49 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 3 Apr 2019 11:40:32 +0200

perf/x86/intel: Initialize TFA MSR

Stephane reported that the TFA MSR is not initialized by the kernel, but the TFA bit could set by firmware or as a leftover from a kexec, which makes the state inconsistent.

---
 arch/x86/events/intel/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/events/intel/core.c +6 -0
index 1539647ea39d..f61dcbef20ff 100644
@@ -3575,6 +3575,12 @@ static void intel_pmu_cpu_starting(int cpu)
 
 	cpuc->lbr_sel = NULL;
 
+	if (x86_pmu.flags & PMU_FL_TFA) {
+		WARN_ON_ONCE(cpuc->tfa_shadow);
+		cpuc->tfa_shadow = ~0ULL;
+		intel_set_tfa(cpuc, false);
+	}
+
 	if (x86_pmu.version > 1)
 		flip_smm_bit(&x86_pmu.attr_freeze_on_smi);
 
#25 lore raw

Re: [tip:perf/urgent] perf/x86/intel: Initialize TFA MSR

Quoted 13 lines · tip-bot for Peter Zijlstra
Commit-ID:  d7262457e35dbe239659e62654e56f8ddb814bed
Gitweb:     https://git.kernel.org/tip/d7262457e35dbe239659e62654e56f8ddb814bed
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Thu, 21 Mar 2019 13:38:49 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 3 Apr 2019 11:40:32 +0200

perf/x86/intel: Initialize TFA MSR

Stephane reported that the TFA MSR is not initialized by the kernel, but the TFA bit could set by firmware or as a leftover from a kexec, which makes the state inconsistent.

Quoted 13 lines

This lacks:

1) Fixes tag

2) Cc: stable ....

Sigh.

Quoted 22 lines
---
 arch/x86/events/intel/core.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/events/intel/core.c +6 -0
index 1539647ea39d..f61dcbef20ff 100644
@@ -3575,6 +3575,12 @@ static void intel_pmu_cpu_starting(int cpu)
 
 	cpuc->lbr_sel = NULL;
 
+	if (x86_pmu.flags & PMU_FL_TFA) {
+		WARN_ON_ONCE(cpuc->tfa_shadow);
+		cpuc->tfa_shadow = ~0ULL;
+		intel_set_tfa(cpuc, false);
+	}
+
 	if (x86_pmu.version > 1)
 		flip_smm_bit(&x86_pmu.attr_freeze_on_smi);
 
#26 lore raw

Re: [tip:perf/urgent] perf/x86/intel: Initialize TFA MSR

Quoted 29 lines · Thomas Gleixner
Quoted 13 lines · tip-bot for Peter Zijlstra
Commit-ID:  d7262457e35dbe239659e62654e56f8ddb814bed
Gitweb:     https://git.kernel.org/tip/d7262457e35dbe239659e62654e56f8ddb814bed
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Thu, 21 Mar 2019 13:38:49 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 3 Apr 2019 11:40:32 +0200

perf/x86/intel: Initialize TFA MSR

Stephane reported that the TFA MSR is not initialized by the kernel, but the TFA bit could set by firmware or as a leftover from a kexec, which makes the state inconsistent.

Quoted 13 lines
Quoted 7 lines

This lacks:

1) Fixes tag

2) Cc: stable ....

Sigh.

It would also be nice to know what a "TFA" bit is without having to go find a copy of the Intel documentation.

Vince

author maintainer
#27 lore raw

RFC PATCH 2/8 perf/x86/intel: Simplify intel_tfa_commit_scheduling()

In reply to Peter Zijlstra

validate_group() calls x86_schedule_events(.assign=NULL) and therefore will not call intel_tfa_commit_scheduling(). So there is no point in checking cpuc->is_fake, we'll never get there.

---
 arch/x86/events/intel/core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/events/intel/core.c +1 -1
@@ -2015,7 +2015,7 @@ static void intel_tfa_commit_scheduling(
 	/*
 	 * We're going to use PMC3, make sure TFA is set before we touch it.
 	 */
-	if (cntr == 3 && !cpuc->is_fake)
+	if (cntr == 3)
 		intel_set_tfa(cpuc, true);
 }
 
author maintainer
#28 lore raw

RFC PATCH 3/8 perf/x86: Simplify x86_pmu.get_constraints() interface

In reply to Peter Zijlstra

There is a special case for validate_events() where we'll call x86_pmu.get_constraints(.idx=-1). It's purpose, up until recent, seems to be to avoid taking a previous constraint from cpuc->event_constraint[] in intel_get_event_constraints().

(I could not find any other get_event_constraints() implementation using @idx)

However, since that cpuc is freshly allocated, that array will in fact be initialized with NULL pointers, achieving the very same effect.

Therefore remove this exception.

---
 arch/x86/events/core.c       |    2 +-
 arch/x86/events/intel/core.c |    8 +++-----
 2 files changed, 4 insertions(+), 6 deletions(-)
arch/x86/events/core.c +1 -1
@@ -2031,7 +2031,7 @@ static int validate_event(struct perf_ev
 	if (IS_ERR(fake_cpuc))
 		return PTR_ERR(fake_cpuc);
 
-	c = x86_pmu.get_event_constraints(fake_cpuc, -1, event);
+	c = x86_pmu.get_event_constraints(fake_cpuc, 0, event);
 
 	if (!c || !c->weight)
 		ret = -EINVAL;
arch/x86/events/intel/core.c +3 -5
@@ -2931,11 +2931,9 @@ static struct event_constraint *
 intel_get_event_constraints(struct cpu_hw_events *cpuc, int idx,
 			    struct perf_event *event)
 {
-	struct event_constraint *c1 = NULL;
-	struct event_constraint *c2;
+	struct event_constraint *c1, *c2;
 
-	if (idx >= 0) /* fake does < 0 */
-		c1 = cpuc->event_constraint[idx];
+	c1 = cpuc->event_constraint[idx];
 
 	/*
 	 * first time only
@@ -3410,7 +3408,7 @@ tfa_get_event_constraints(struct cpu_hw_
 	/*
 	 * Without TFA we must not use PMC3.
 	 */
-	if (!allow_tsx_force_abort && test_bit(3, c->idxmsk) && idx >= 0) {
+	if (!allow_tsx_force_abort && test_bit(3, c->idxmsk)) {
 		c = dyn_constraint(cpuc, c, idx);
 		c->idxmsk64 &= ~(1ULL << 3);
 		c->weight--;
Reviewed-by on v1 3/8
#29 lore raw

Re: [RFC][PATCH 3/8] perf/x86: Simplify x86_pmu.get_constraints() interface

Quoted 9 lines · Peter Zijlstra

There is a special case for validate_events() where we'll call x86_pmu.get_constraints(.idx=-1). It's purpose, up until recent, seems to be to avoid taking a previous constraint from cpuc->event_constraint[] in intel_get_event_constraints().

(I could not find any other get_event_constraints() implementation using @idx)

Quoted 6 lines

However, since that cpuc is freshly allocated, that array will in fact be initialized with NULL pointers, achieving the very same effect.

Therefore remove this exception.

Looks good to me.

Quoted 43 lines
---
 arch/x86/events/core.c       |    2 +-
 arch/x86/events/intel/core.c |    8 +++-----
 2 files changed, 4 insertions(+), 6 deletions(-)
arch/x86/events/core.c +1 -1
@@ -2031,7 +2031,7 @@ static int validate_event(struct perf_ev
        if (IS_ERR(fake_cpuc))
                return PTR_ERR(fake_cpuc);
 
-       c = x86_pmu.get_event_constraints(fake_cpuc, -1, event);
+       c = x86_pmu.get_event_constraints(fake_cpuc, 0, event);
 
        if (!c || !c->weight)
                ret = -EINVAL;
arch/x86/events/intel/core.c +3 -5
@@ -2931,11 +2931,9 @@ static struct event_constraint *
 intel_get_event_constraints(struct cpu_hw_events *cpuc, int idx,
                            struct perf_event *event)
 {
-       struct event_constraint *c1 = NULL;
-       struct event_constraint *c2;
+       struct event_constraint *c1, *c2;
 
-       if (idx >= 0) /* fake does < 0 */
-               c1 = cpuc->event_constraint[idx];
+       c1 = cpuc->event_constraint[idx];
 
        /*
         * first time only
@@ -3410,7 +3408,7 @@ tfa_get_event_constraints(struct cpu_hw_
        /*
         * Without TFA we must not use PMC3.
         */
-       if (!allow_tsx_force_abort && test_bit(3, c->idxmsk) && idx >= 0) {
+       if (!allow_tsx_force_abort && test_bit(3, c->idxmsk)) {
                c = dyn_constraint(cpuc, c, idx);
                c->idxmsk64 &= ~(1ULL << 3);
                c->weight--;
author maintainer
#30 lore raw

RFC PATCH 4/8 perf/x86: Remove PERF_X86_EVENT_COMMITTED

In reply to Peter Zijlstra

The flag PERF_X86_EVENT_COMMITTED is used to find uncommitted events for which to call put_event_constraint() when scheduling fails.

These are the newly added events to the list, and must form, per definition, the tail of cpuc->event_list[]. By computing the list index of the last successfull schedule, then iteration can start there and the flag is redundant.

There are only 3 callers of x86_schedule_events(), notably:

- x86_pmu_add()
- x86_pmu_commit_txn()
- validate_group()

For x86_pmu_add(), cpuc->n_events isn't updated until after schedule_events() succeeds, therefore cpuc->n_events points to the desired index.

For x86_pmu_commit_txn(), cpuc->n_events is updated, but we can trivially compute the desired value with cpuc->n_txn -- the number of events added in this transaction.

For validate_group(), we can make the rule for x86_pmu_add() work by simply setting cpuc->n_events to 0 before calling schedule_events().

---
 arch/x86/events/core.c       |   21 ++++++---------------
 arch/x86/events/perf_event.h |    1 -
 2 files changed, 6 insertions(+), 16 deletions(-)
arch/x86/events/core.c +6 -15
@@ -925,19 +925,16 @@ int x86_schedule_events(struct cpu_hw_ev
 	if (!unsched && assign) {
 		for (i = 0; i < n; i++) {
 			e = cpuc->event_list[i];
-			e->hw.flags |= PERF_X86_EVENT_COMMITTED;
 			if (x86_pmu.commit_scheduling)
 				x86_pmu.commit_scheduling(cpuc, i, assign[i]);
 		}
 	} else {
-		for (i = 0; i < n; i++) {
+		i = cpuc->n_events;
+		if (cpuc->txn_flags & PERF_PMU_TXN_ADD)
+			i -= cpuc->n_txn;
+
+		for (; i < n; i++) {
 			e = cpuc->event_list[i];
-			/*
-			 * do not put_constraint() on comitted events,
-			 * because they are good to go
-			 */
-			if ((e->hw.flags & PERF_X86_EVENT_COMMITTED))
-				continue;
 
 			/*
 			 * release events that failed scheduling
@@ -1372,11 +1369,6 @@ static void x86_pmu_del(struct perf_even
 	int i;
 
 	/*
-	 * event is descheduled
-	 */
-	event->hw.flags &= ~PERF_X86_EVENT_COMMITTED;
-
-	/*
 	 * If we're called during a txn, we only need to undo x86_pmu.add.
 	 * The events never got scheduled and ->cancel_txn will truncate
 	 * the event_list.
@@ -2079,8 +2071,7 @@ static int validate_group(struct perf_ev
 	if (n < 0)
 		goto out;
 
-	fake_cpuc->n_events = n;
-
+	fake_cpuc->n_events = 0;
 	ret = x86_pmu.schedule_events(fake_cpuc, n, NULL);
 
 out:
arch/x86/events/perf_event.h +0 -1
@@ -61,7 +61,6 @@ struct event_constraint {
 #define PERF_X86_EVENT_PEBS_LDLAT	0x0001 /* ld+ldlat data address sampling */
 #define PERF_X86_EVENT_PEBS_ST		0x0002 /* st data address sampling */
 #define PERF_X86_EVENT_PEBS_ST_HSW	0x0004 /* haswell style datala, store */
-#define PERF_X86_EVENT_COMMITTED	0x0008 /* event passed commit_txn */
 #define PERF_X86_EVENT_PEBS_LD_HSW	0x0010 /* haswell style datala, load */
 #define PERF_X86_EVENT_PEBS_NA_HSW	0x0020 /* haswell style datala, unknown */
 #define PERF_X86_EVENT_EXCL		0x0040 /* HT exclusivity on counter */
Reviewed-by on v1 4/8
#31 lore raw

Re: [RFC][PATCH 4/8] perf/x86: Remove PERF_X86_EVENT_COMMITTED

Quoted 11 lines · Peter Zijlstra

The flag PERF_X86_EVENT_COMMITTED is used to find uncommitted events for which to call put_event_constraint() when scheduling fails.

These are the newly added events to the list, and must form, per definition, the tail of cpuc->event_list[]. By computing the list index of the last successfull schedule, then iteration can start there and the flag is redundant.

There are only 3 callers of x86_schedule_events(), notably:

Quoted 8 lines

- x86_pmu_add()
- x86_pmu_commit_txn()
- validate_group()

For x86_pmu_add(), cpuc->n_events isn't updated until after schedule_events() succeeds, therefore cpuc->n_events points to the desired index.

Correct.

Quoted 4 lines

For x86_pmu_commit_txn(), cpuc->n_events is updated, but we can trivially compute the desired value with cpuc->n_txn -- the number of events added in this transaction.

I suggest you put this explanation in the code so that it is easier to understand.

Quoted 4 lines

For validate_group(), we can make the rule for x86_pmu_add() work by simply setting cpuc->n_events to 0 before calling schedule_events().

Quoted 57 lines
---
 arch/x86/events/core.c       |   21 ++++++---------------
 arch/x86/events/perf_event.h |    1 -
 2 files changed, 6 insertions(+), 16 deletions(-)
arch/x86/events/core.c +6 -15
@@ -925,19 +925,16 @@ int x86_schedule_events(struct cpu_hw_ev
        if (!unsched && assign) {
                for (i = 0; i < n; i++) {
                        e = cpuc->event_list[i];
-                       e->hw.flags |= PERF_X86_EVENT_COMMITTED;
                        if (x86_pmu.commit_scheduling)
                                x86_pmu.commit_scheduling(cpuc, i, assign[i]);
                }
        } else {
-               for (i = 0; i < n; i++) {
+               i = cpuc->n_events;
+               if (cpuc->txn_flags & PERF_PMU_TXN_ADD)
+                       i -= cpuc->n_txn;
+
+               for (; i < n; i++) {
                        e = cpuc->event_list[i];
-                       /*
-                        * do not put_constraint() on comitted events,
-                        * because they are good to go
-                        */
-                       if ((e->hw.flags & PERF_X86_EVENT_COMMITTED))
-                               continue;
 
                        /*
                         * release events that failed scheduling
@@ -1372,11 +1369,6 @@ static void x86_pmu_del(struct perf_even
        int i;
 
        /*
-        * event is descheduled
-        */
-       event->hw.flags &= ~PERF_X86_EVENT_COMMITTED;
-
-       /*
         * If we're called during a txn, we only need to undo x86_pmu.add.
         * The events never got scheduled and ->cancel_txn will truncate
         * the event_list.
@@ -2079,8 +2071,7 @@ static int validate_group(struct perf_ev
        if (n < 0)
                goto out;
 
-       fake_cpuc->n_events = n;
-
+       fake_cpuc->n_events = 0;
        ret = x86_pmu.schedule_events(fake_cpuc, n, NULL);
 
 out:

arch/x86/events/perf_event.h struct event_constraint { · line 64

Quoted 4 lines
arch/x86/events/perf_event.h +0 -1
@@ -61,7 +61,6 @@ struct event_constraint {
 #define PERF_X86_EVENT_PEBS_LDLAT      0x0001 /* ld+ldlat data address sampling */
 #define PERF_X86_EVENT_PEBS_ST         0x0002 /* st data address sampling */
 #define PERF_X86_EVENT_PEBS_ST_HSW     0x0004 /* haswell style datala, store */
-#define PERF_X86_EVENT_COMMITTED       0x0008 /* event passed commit_txn */

I would put a placeholder saying that bit 3 is available or renumbered the other masks below

Quoted 5 lines
arch/x86/events/perf_event.h +0 -0
 #define PERF_X86_EVENT_PEBS_LD_HSW     0x0010 /* haswell style datala, load */
 #define PERF_X86_EVENT_PEBS_NA_HSW     0x0020 /* haswell style datala, unknown */
 #define PERF_X86_EVENT_EXCL            0x0040 /* HT exclusivity on counter */
author maintainer
#32 lore raw

Re: [RFC][PATCH 4/8] perf/x86: Remove PERF_X86_EVENT_COMMITTED

Quoted 8 lines · Stephane Eranian
Quoted 4 lines · Peter Zijlstra

For x86_pmu_commit_txn(), cpuc->n_events is updated, but we can trivially compute the desired value with cpuc->n_txn -- the number of events added in this transaction.

I suggest you put this explanation in the code so that it is easier to understand.

Right, I actually attempted writing that comment a few times but every time it became a mess. I'll try again, because you're quite right, this is a bit magical.

Quoted 6 lines
Quoted 6 lines
arch/x86/events/core.c +5 -1
-               for (i = 0; i < n; i++) {
+               i = cpuc->n_events;
+               if (cpuc->txn_flags & PERF_PMU_TXN_ADD)
+                       i -= cpuc->n_txn;
+
+               for (; i < n; i++) {
author maintainer
#33 lore raw

Re: [RFC][PATCH 4/8] perf/x86: Remove PERF_X86_EVENT_COMMITTED

Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Stephane Eranian
Quoted 4 lines · Peter Zijlstra

For x86_pmu_commit_txn(), cpuc->n_events is updated, but we can trivially compute the desired value with cpuc->n_txn -- the number of events added in this transaction.

Quoted 6 lines
Quoted 2 lines

I suggest you put this explanation in the code so that it is easier to understand.

Right, I actually attempted writing that comment a few times but every time it became a mess. I'll try again, because you're quite right, this is a bit magical.

Ha! I found the comment, it's in patch 8 :-)

author maintainer Reviewed-by Stephane Eranian
#34 lore raw

Re: [RFC][PATCH 4/8] perf/x86: Remove PERF_X86_EVENT_COMMITTED

In reply to Stephane Eranian

Subject: perf/x86: Remove PERF_X86_EVENT_COMMITTED
From: Peter Zijlstra <[email protected]>
Date: Thu Mar 14 12:58:52 CET 2019

The flag PERF_X86_EVENT_COMMITTED is used to find uncommitted events for which to call put_event_constraint() when scheduling fails.

These are the newly added events to the list, and must form, per definition, the tail of cpuc->event_list[]. By computing the list index of the last successfull schedule, then iteration can start there and the flag is redundant.

There are only 3 callers of x86_schedule_events(), notably:

- x86_pmu_add()
- x86_pmu_commit_txn()
- validate_group()

For x86_pmu_add(), cpuc->n_events isn't updated until after schedule_events() succeeds, therefore cpuc->n_events points to the desired index.

For x86_pmu_commit_txn(), cpuc->n_events is updated, but we can trivially compute the desired value with cpuc->n_txn -- the number of events added in this transaction.

For validate_group(), we can make the rule for x86_pmu_add() work by simply setting cpuc->n_events to 0 before calling schedule_events().

---
 arch/x86/events/core.c       |   28 +++++++++++++---------------
 arch/x86/events/perf_event.h |   19 +++++++++----------
 2 files changed, 22 insertions(+), 25 deletions(-)
arch/x86/events/core.c +13 -15
@@ -925,19 +925,23 @@ int x86_schedule_events(struct cpu_hw_ev
 	if (!unsched && assign) {
 		for (i = 0; i < n; i++) {
 			e = cpuc->event_list[i];
-			e->hw.flags |= PERF_X86_EVENT_COMMITTED;
 			if (x86_pmu.commit_scheduling)
 				x86_pmu.commit_scheduling(cpuc, i, assign[i]);
 		}
 	} else {
-		for (i = 0; i < n; i++) {
+		/*
+		 * In a transaction cpuc->n_events is already updated, but we
+		 * can use cpuc->n_txn know how many new events there are.
+		 *
+		 * Outside of a transaction, cpuc->n_events is not yet updated,
+		 * and indicates how many events how many events are scheduled.
+		 */
+		i = cpuc->n_events;
+		if (cpuc->txn_flags & PERF_PMU_TXN_ADD)
+			i -= cpuc->n_txn;
+
+		for (; i < n; i++) {
 			e = cpuc->event_list[i];
-			/*
-			 * do not put_constraint() on comitted events,
-			 * because they are good to go
-			 */
-			if ((e->hw.flags & PERF_X86_EVENT_COMMITTED))
-				continue;
 
 			/*
 			 * release events that failed scheduling
@@ -1372,11 +1376,6 @@ static void x86_pmu_del(struct perf_even
 	int i;
 
 	/*
-	 * event is descheduled
-	 */
-	event->hw.flags &= ~PERF_X86_EVENT_COMMITTED;
-
-	/*
 	 * If we're called during a txn, we only need to undo x86_pmu.add.
 	 * The events never got scheduled and ->cancel_txn will truncate
 	 * the event_list.
@@ -2079,8 +2078,7 @@ static int validate_group(struct perf_ev
 	if (n < 0)
 		goto out;
 
-	fake_cpuc->n_events = n;
-
+	fake_cpuc->n_events = 0;
 	ret = x86_pmu.schedule_events(fake_cpuc, n, NULL);
 
 out:
arch/x86/events/perf_event.h +9 -10
@@ -55,22 +55,21 @@ struct event_constraint {
 	int	overlap;
 	int	flags;
 };
+
 /*
  * struct hw_perf_event.flags flags
  */
 #define PERF_X86_EVENT_PEBS_LDLAT	0x0001 /* ld+ldlat data address sampling */
 #define PERF_X86_EVENT_PEBS_ST		0x0002 /* st data address sampling */
 #define PERF_X86_EVENT_PEBS_ST_HSW	0x0004 /* haswell style datala, store */
-#define PERF_X86_EVENT_COMMITTED	0x0008 /* event passed commit_txn */
-#define PERF_X86_EVENT_PEBS_LD_HSW	0x0010 /* haswell style datala, load */
-#define PERF_X86_EVENT_PEBS_NA_HSW	0x0020 /* haswell style datala, unknown */
-#define PERF_X86_EVENT_EXCL		0x0040 /* HT exclusivity on counter */
-#define PERF_X86_EVENT_DYNAMIC		0x0080 /* dynamic alloc'd constraint */
-#define PERF_X86_EVENT_RDPMC_ALLOWED	0x0100 /* grant rdpmc permission */
-#define PERF_X86_EVENT_EXCL_ACCT	0x0200 /* accounted EXCL event */
-#define PERF_X86_EVENT_AUTO_RELOAD	0x0400 /* use PEBS auto-reload */
-#define PERF_X86_EVENT_LARGE_PEBS	0x0800 /* use large PEBS */
-
+#define PERF_X86_EVENT_PEBS_LD_HSW	0x0008 /* haswell style datala, load */
+#define PERF_X86_EVENT_PEBS_NA_HSW	0x0010 /* haswell style datala, unknown */
+#define PERF_X86_EVENT_EXCL		0x0020 /* HT exclusivity on counter */
+#define PERF_X86_EVENT_DYNAMIC		0x0040 /* dynamic alloc'd constraint */
+#define PERF_X86_EVENT_RDPMC_ALLOWED	0x0080 /* grant rdpmc permission */
+#define PERF_X86_EVENT_EXCL_ACCT	0x0100 /* accounted EXCL event */
+#define PERF_X86_EVENT_AUTO_RELOAD	0x0200 /* use PEBS auto-reload */
+#define PERF_X86_EVENT_LARGE_PEBS	0x0400 /* use large PEBS */
 
 struct amd_nb {
 	int nb_id;  /* NorthBridge id */
author maintainer
#35 lore raw

RFC PATCH 5/8 perf/x86/intel: Optimize intel_get_excl_constraints()

In reply to Peter Zijlstra

Avoid the POPCNT by noting we can decrement the weight for each cleared bit.

---
 arch/x86/events/intel/core.c |   22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)
arch/x86/events/intel/core.c +13 -9
@@ -2838,7 +2838,7 @@ intel_get_excl_constraints(struct cpu_hw
 	struct intel_excl_cntrs *excl_cntrs = cpuc->excl_cntrs;
 	struct intel_excl_states *xlo;
 	int tid = cpuc->excl_thread_id;
-	int is_excl, i;
+	int is_excl, i, w;
 
 	/*
 	 * validating a group does not require
@@ -2894,36 +2894,40 @@ intel_get_excl_constraints(struct cpu_hw
 	 * SHARED   : sibling counter measuring non-exclusive event
 	 * UNUSED   : sibling counter unused
 	 */
+	w = c->weight;
 	for_each_set_bit(i, c->idxmsk, X86_PMC_IDX_MAX) {
 		/*
 		 * exclusive event in sibling counter
 		 * our corresponding counter cannot be used
 		 * regardless of our event
 		 */
-		if (xlo->state[i] == INTEL_EXCL_EXCLUSIVE)
+		if (xlo->state[i] == INTEL_EXCL_EXCLUSIVE) {
 			__clear_bit(i, c->idxmsk);
+			w--;
+			continue;
+		}
 		/*
 		 * if measuring an exclusive event, sibling
 		 * measuring non-exclusive, then counter cannot
 		 * be used
 		 */
-		if (is_excl && xlo->state[i] == INTEL_EXCL_SHARED)
+		if (is_excl && xlo->state[i] == INTEL_EXCL_SHARED) {
 			__clear_bit(i, c->idxmsk);
+			w--;
+			continue;
+		}
 	}
 
 	/*
-	 * recompute actual bit weight for scheduling algorithm
-	 */
-	c->weight = hweight64(c->idxmsk64);
-
-	/*
 	 * if we return an empty mask, then switch
 	 * back to static empty constraint to avoid
 	 * the cost of freeing later on
 	 */
-	if (c->weight == 0)
+	if (!w)
 		c = &emptyconstraint;
 
+	c->weight = w;
+
 	return c;
 }
 
Reviewed-by on v1 5/8
#36 lore raw

Re: [RFC][PATCH 5/8] perf/x86/intel: Optimize intel_get_excl_constraints()

Quoted 5 lines · Peter Zijlstra

Avoid the POPCNT by noting we can decrement the weight for each cleared bit.

Looks good to me.

Quoted 66 lines
---
 arch/x86/events/intel/core.c |   22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)
arch/x86/events/intel/core.c +13 -9
@@ -2838,7 +2838,7 @@ intel_get_excl_constraints(struct cpu_hw
        struct intel_excl_cntrs *excl_cntrs = cpuc->excl_cntrs;
        struct intel_excl_states *xlo;
        int tid = cpuc->excl_thread_id;
-       int is_excl, i;
+       int is_excl, i, w;
 
        /*
         * validating a group does not require
@@ -2894,36 +2894,40 @@ intel_get_excl_constraints(struct cpu_hw
         * SHARED   : sibling counter measuring non-exclusive event
         * UNUSED   : sibling counter unused
         */
+       w = c->weight;
        for_each_set_bit(i, c->idxmsk, X86_PMC_IDX_MAX) {
                /*
                 * exclusive event in sibling counter
                 * our corresponding counter cannot be used
                 * regardless of our event
                 */
-               if (xlo->state[i] == INTEL_EXCL_EXCLUSIVE)
+               if (xlo->state[i] == INTEL_EXCL_EXCLUSIVE) {
                        __clear_bit(i, c->idxmsk);
+                       w--;
+                       continue;
+               }
                /*
                 * if measuring an exclusive event, sibling
                 * measuring non-exclusive, then counter cannot
                 * be used
                 */
-               if (is_excl && xlo->state[i] == INTEL_EXCL_SHARED)
+               if (is_excl && xlo->state[i] == INTEL_EXCL_SHARED) {
                        __clear_bit(i, c->idxmsk);
+                       w--;
+                       continue;
+               }
        }
 
        /*
-        * recompute actual bit weight for scheduling algorithm
-        */
-       c->weight = hweight64(c->idxmsk64);
-
-       /*
         * if we return an empty mask, then switch
         * back to static empty constraint to avoid
         * the cost of freeing later on
         */
-       if (c->weight == 0)
+       if (!w)
                c = &emptyconstraint;
 
+       c->weight = w;
+
        return c;
 }
 
author maintainer
#37 lore raw

RFC PATCH 6/8 perf/x86: Clear ->event_constraint[] on put

In reply to Peter Zijlstra

The current code unconditionally clears cpuc->event_constraint[i] before calling get_event_constraints(.idx=i). The only site that cares is intel_get_event_constraints() where the c1 load will always be NULL.

However, always calling get_event_constraints() on all events is wastefull, most times it will return the exact same result. Therefore retain the logic in intel_get_event_constraints() and change the generic code to only clear the constraint on put.

---
 arch/x86/events/core.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/events/core.c +3 -1
@@ -858,7 +858,6 @@ int x86_schedule_events(struct cpu_hw_ev
 		x86_pmu.start_scheduling(cpuc);
 
 	for (i = 0, wmin = X86_PMC_IDX_MAX, wmax = 0; i < n; i++) {
-		cpuc->event_constraint[i] = NULL;
 		c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
 		cpuc->event_constraint[i] = c;
 
@@ -941,6 +940,8 @@ int x86_schedule_events(struct cpu_hw_ev
 			 */
 			if (x86_pmu.put_event_constraints)
 				x86_pmu.put_event_constraints(cpuc, e);
+
+			cpuc->event_constraint[i] = NULL;
 		}
 	}
 
@@ -1404,6 +1405,7 @@ static void x86_pmu_del(struct perf_even
 		cpuc->event_list[i-1] = cpuc->event_list[i];
 		cpuc->event_constraint[i-1] = cpuc->event_constraint[i];
 	}
+	cpuc->event_constraint[i-1] = NULL;
 	--cpuc->n_events;
 
 	perf_event_update_userpage(event);
Reviewed-by on v1 6/8
#38 lore raw

Re: [RFC][PATCH 6/8] perf/x86: Clear ->event_constraint[] on put

Quoted 6 lines · Peter Zijlstra

The current code unconditionally clears cpuc->event_constraint[i] before calling get_event_constraints(.idx=i). The only site that cares is intel_get_event_constraints() where the c1 load will always be NULL.

Quoted 6 lines

However, always calling get_event_constraints() on all events is wastefull, most times it will return the exact same result. Therefore retain the logic in intel_get_event_constraints() and change the generic code to only clear the constraint on put.

I thought there was caching of the constraint in case it was static (or unconstrained)
to avoid walking the constraint table each time between invocations on the same group_sched_in() call. But the way the c1 vs. c2 logic is written I don't see it. In which case, this could be another opportunity.

Looks good to me.

Quoted 33 lines
---
 arch/x86/events/core.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/events/core.c +3 -1
@@ -858,7 +858,6 @@ int x86_schedule_events(struct cpu_hw_ev
                x86_pmu.start_scheduling(cpuc);
 
        for (i = 0, wmin = X86_PMC_IDX_MAX, wmax = 0; i < n; i++) {
-               cpuc->event_constraint[i] = NULL;
                c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
                cpuc->event_constraint[i] = c;
 
@@ -941,6 +940,8 @@ int x86_schedule_events(struct cpu_hw_ev
                         */
                        if (x86_pmu.put_event_constraints)
                                x86_pmu.put_event_constraints(cpuc, e);
+
+                       cpuc->event_constraint[i] = NULL;
                }
        }
 
@@ -1404,6 +1405,7 @@ static void x86_pmu_del(struct perf_even
                cpuc->event_list[i-1] = cpuc->event_list[i];
                cpuc->event_constraint[i-1] = cpuc->event_constraint[i];
        }
+       cpuc->event_constraint[i-1] = NULL;
        --cpuc->n_events;
 
        perf_event_update_userpage(event);
author maintainer
#39 lore raw

Re: [RFC][PATCH 6/8] perf/x86: Clear ->event_constraint[] on put

Quoted 12 lines · Stephane Eranian
Quoted 11 lines · Peter Zijlstra

The current code unconditionally clears cpuc->event_constraint[i] before calling get_event_constraints(.idx=i). The only site that cares is intel_get_event_constraints() where the c1 load will always be NULL.

However, always calling get_event_constraints() on all events is wastefull, most times it will return the exact same result. Therefore retain the logic in intel_get_event_constraints() and change the generic code to only clear the constraint on put.

Quoted 4 lines
Quoted 1 line

I thought there was caching of the constraint in case it was static (or unconstrained)

Yeah, I was under the same impression :-/

Quoted 3 lines

to avoid walking the constraint table each time between invocations on the same group_sched_in() call. But the way the c1 vs. c2 logic is written I don't see it. In which case, this could be another opportunity.

Right, the next patch attempts this.

author maintainer
#40 lore raw

RFC PATCH 7/8 perf/x86: Optimize x86_schedule_events()

In reply to Peter Zijlstra

Now that cpuc->event_constraint[] is retained, we can avoid calling get_event_constraints() over and over again.

---
 arch/x86/events/core.c       |   25 +++++++++++++++++++++----
 arch/x86/events/intel/core.c |    3 ++-
 2 files changed, 23 insertions(+), 5 deletions(-)
arch/x86/events/core.c +21 -4
@@ -844,6 +844,12 @@ int perf_assign_events(struct event_cons
 }
 EXPORT_SYMBOL_GPL(perf_assign_events);
 
+static inline bool is_ht_workaround_active(struct cpu_hw_events *cpuc)
+{
+	return is_ht_workaround_enabled() && !cpuc->is_fake &&
+	       READ_ONCE(cpuc->excl_cntrs->exclusive_present);
+}
+
 int x86_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign)
 {
 	struct event_constraint *c;
@@ -858,8 +864,20 @@ int x86_schedule_events(struct cpu_hw_ev
 		x86_pmu.start_scheduling(cpuc);
 
 	for (i = 0, wmin = X86_PMC_IDX_MAX, wmax = 0; i < n; i++) {
-		c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
-		cpuc->event_constraint[i] = c;
+		c = cpuc->event_constraint[i];
+
+		/*
+		 * Request constraints for new events; or for those events that
+		 * have a dynamic constraint due to the HT workaround -- for
+		 * those the constraint can change due to scheduling activity
+		 * on the other sibling.
+		 */
+		if (!c || ((c->flags & PERF_X86_EVENT_DYNAMIC) &&
+			   is_ht_workaround_active(cpuc))) {
+
+			c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
+			cpuc->event_constraint[i] = c;
+		}
 
 		wmin = min(wmin, c->weight);
 		wmax = max(wmax, c->weight);
@@ -903,8 +921,7 @@ int x86_schedule_events(struct cpu_hw_ev
 		 * N/2 counters can be used. This helps with events with
 		 * specific counter constraints.
 		 */
-		if (is_ht_workaround_enabled() && !cpuc->is_fake &&
-		    READ_ONCE(cpuc->excl_cntrs->exclusive_present))
+		if (is_ht_workaround_active(cpuc))
 			gpmax /= 2;
 
 		unsched = perf_assign_events(cpuc->event_constraint, n, wmin,
arch/x86/events/intel/core.c +2 -1
@@ -2945,7 +2945,8 @@ intel_get_event_constraints(struct cpu_h
 	 * - dynamic constraint: handled by intel_get_excl_constraints()
 	 */
 	c2 = __intel_get_event_constraints(cpuc, idx, event);
-	if (c1 && (c1->flags & PERF_X86_EVENT_DYNAMIC)) {
+	if (c1) {
+	        WARN_ON_ONCE(!(c1->flags & PERF_X86_EVENT_DYNAMIC));
 		bitmap_copy(c1->idxmsk, c2->idxmsk, X86_PMC_IDX_MAX);
 		c1->weight = c2->weight;
 		c2 = c1;
#41 lore raw

Re: [RFC][PATCH 7/8] perf/x86: Optimize x86_schedule_events()

Quoted 35 lines · Peter Zijlstra

Now that cpuc->event_constraint[] is retained, we can avoid calling get_event_constraints() over and over again.

---
 arch/x86/events/core.c       |   25 +++++++++++++++++++++----
 arch/x86/events/intel/core.c |    3 ++-
 2 files changed, 23 insertions(+), 5 deletions(-)
arch/x86/events/core.c +10 -2
@@ -844,6 +844,12 @@ int perf_assign_events(struct event_cons
 }
 EXPORT_SYMBOL_GPL(perf_assign_events);
 
+static inline bool is_ht_workaround_active(struct cpu_hw_events *cpuc)
+{
+       return is_ht_workaround_enabled() && !cpuc->is_fake &&
+              READ_ONCE(cpuc->excl_cntrs->exclusive_present);
+}
+
 int x86_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign)
 {
        struct event_constraint *c;
@@ -858,8 +864,20 @@ int x86_schedule_events(struct cpu_hw_ev
                x86_pmu.start_scheduling(cpuc);
 
        for (i = 0, wmin = X86_PMC_IDX_MAX, wmax = 0; i < n; i++) {
-               c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
-               cpuc->event_constraint[i] = c;
+               c = cpuc->event_constraint[i];
+
+               /*
+                * Request constraints for new events; or for those events that

arch/x86/events/core.c int x86_schedule_events(struct cpu_hw_ev · line 880

Quoted 10 lines
arch/x86/events/core.c +10 -0
+                * have a dynamic constraint due to the HT workaround -- for
+                * those the constraint can change due to scheduling activity
+                * on the other sibling.
+                */
+               if (!c || ((c->flags & PERF_X86_EVENT_DYNAMIC) &&
+                          is_ht_workaround_active(cpuc))) {
+
+                       c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
+                       cpuc->event_constraint[i] = c;
+               }

On this one, I think there may be a problem with events with shared_regs constraints.
Constraint is dynamic as it depends on other events which share the same MSR, yet it
is not marked as DYNAMIC. But this may be okay because these other events are all
on the same CPU and thus scheduled during the same ctx_sched_in(). Yet with the swapping in intel_alt_er(), we need to double-check that we cannot reuse a constraint
which could be stale. I believe this is okay, just double-check.

Quoted 27 lines
arch/x86/events/core.c +1 -2
                wmin = min(wmin, c->weight);
                wmax = max(wmax, c->weight);
@@ -903,8 +921,7 @@ int x86_schedule_events(struct cpu_hw_ev
                 * N/2 counters can be used. This helps with events with
                 * specific counter constraints.
                 */
-               if (is_ht_workaround_enabled() && !cpuc->is_fake &&
-                   READ_ONCE(cpuc->excl_cntrs->exclusive_present))
+               if (is_ht_workaround_active(cpuc))
                        gpmax /= 2;
 
                unsched = perf_assign_events(cpuc->event_constraint, n, wmin,
arch/x86/events/intel/core.c +2 -1
@@ -2945,7 +2945,8 @@ intel_get_event_constraints(struct cpu_h
         * - dynamic constraint: handled by intel_get_excl_constraints()
         */
        c2 = __intel_get_event_constraints(cpuc, idx, event);
-       if (c1 && (c1->flags & PERF_X86_EVENT_DYNAMIC)) {
+       if (c1) {
+               WARN_ON_ONCE(!(c1->flags & PERF_X86_EVENT_DYNAMIC));
                bitmap_copy(c1->idxmsk, c2->idxmsk, X86_PMC_IDX_MAX);
                c1->weight = c2->weight;
                c2 = c1;
author maintainer
#42 lore raw

Re: [RFC][PATCH 7/8] perf/x86: Optimize x86_schedule_events()

Quoted 16 lines · Stephane Eranian
Quoted 15 lines · Peter Zijlstra
arch/x86/events/core.c +9 -2
@@ -858,8 +864,20 @@ int x86_schedule_events(struct cpu_hw_ev
                x86_pmu.start_scheduling(cpuc);
 
        for (i = 0, wmin = X86_PMC_IDX_MAX, wmax = 0; i < n; i++) {
-               c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
-               cpuc->event_constraint[i] = c;
+               c = cpuc->event_constraint[i];
+
+               /*
+                * Request constraints for new events; or for those events that
+                * have a dynamic constraint due to the HT workaround -- for
+                * those the constraint can change due to scheduling activity
+                * on the other sibling.
+                */
+               if (!c || ((c->flags & PERF_X86_EVENT_DYNAMIC) &&
Quoted 8 lines

arch/x86/events/core.c int x86_schedule_events(struct cpu_hw_ev · line 880

Quoted 5 lines
arch/x86/events/core.c +5 -0
+                          is_ht_workaround_active(cpuc))) {
+
+                       c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
+                       cpuc->event_constraint[i] = c;
+               }

On this one, I think there may be a problem with events with shared_regs constraints.

Hmm...

Quoted 2 lines

Constraint is dynamic as it depends on other events which share the same MSR, yet it is not marked as DYNAMIC.

it returns &emptyconstraint or a table constraint, depending on register state.

Quoted 6 lines

But this may be okay because these other events are all on the same CPU and thus scheduled during the same ctx_sched_in(). Yet with the swapping in intel_alt_er(), we need to double-check that we cannot reuse a constraint which could be stale.

I believe this is okay, just double-check.

I'm not sure I see a problem.

So if we're the first event on a shared register, we claim the register and scheduling succeeds (barring other constraints).

If we're the second event on a shared register (and have conflicting register state), we get the empty constraint. This _will_ cause scheduling to fail. We'll not cache the state and punt it back to the core code.

So no future scheduling pass will come to see a shared reg constraint that could've changed.

Now, there is indeed the intel_alt_er() thing, which slightly complicates this; then suppose we schedule an event on RSP0, another on RSP1, then remove the RSP0 one. Even in that case, the remaining RSP1 event will not change its constraint, since intel_fixup_er() rewrites the event to be a native RSP1 event.

So that too reduces to the prior case.

That said; I have simplified the above condition to:

@@ -858,8 +858,17 @@ int x86_schedule_events(struct cpu_hw_ev
x86_pmu.start_scheduling(cpuc);

for (i = 0, wmin = X86_PMC_IDX_MAX, wmax = 0; i < n; i++) {
- c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
- cpuc->event_constraint[i] = c;
+ c = cpuc->event_constraint[i];
+
+ /*
+ * Request constraints for new events; or for those events that
+ * have a dynamic constraint -- for those the constraint can
+ * change due to external factors (sibling state, allow_tfa).
+ */
+ if (!c || (c->flags & PERF_X86_EVENT_DYNAMIC)) {
+ c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
+ cpuc->event_constraint[i] = c;
+ }

wmin = min(wmin, c->weight);
wmax = max(wmax, c->weight);

Because any dynamic event can change from one moment to the next.

#43 lore raw

Re: [RFC][PATCH 7/8] perf/x86: Optimize x86_schedule_events()

Quoted 55 lines · Peter Zijlstra
Quoted 16 lines · Stephane Eranian
Quoted 15 lines · Peter Zijlstra
arch/x86/events/core.c +9 -2
@@ -858,8 +864,20 @@ int x86_schedule_events(struct cpu_hw_ev
                x86_pmu.start_scheduling(cpuc);
 
        for (i = 0, wmin = X86_PMC_IDX_MAX, wmax = 0; i < n; i++) {
-               c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
-               cpuc->event_constraint[i] = c;
+               c = cpuc->event_constraint[i];
+
+               /*
+                * Request constraints for new events; or for those events that
+                * have a dynamic constraint due to the HT workaround -- for
+                * those the constraint can change due to scheduling activity
+                * on the other sibling.
+                */
+               if (!c || ((c->flags & PERF_X86_EVENT_DYNAMIC) &&
Quoted 8 lines

arch/x86/events/core.c int x86_schedule_events(struct cpu_hw_ev · line 880

Quoted 5 lines
arch/x86/events/core.c +5 -0
+                          is_ht_workaround_active(cpuc))) {
+
+                       c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
+                       cpuc->event_constraint[i] = c;
+               }

On this one, I think there may be a problem with events with shared_regs constraints.

Hmm...

Quoted 2 lines

Constraint is dynamic as it depends on other events which share the same MSR, yet it is not marked as DYNAMIC.

it returns &emptyconstraint or a table constraint, depending on register state.

Quoted 6 lines

But this may be okay because these other events are all on the same CPU and thus scheduled during the same ctx_sched_in(). Yet with the swapping in intel_alt_er(), we need to double-check that we cannot reuse a constraint which could be stale.

I believe this is okay, just double-check.

I'm not sure I see a problem.

So if we're the first event on a shared register, we claim the register and scheduling succeeds (barring other constraints).

If we're the second event on a shared register (and have conflicting register state), we get the empty constraint. This _will_ cause scheduling to fail. We'll not cache the state and punt it back to the core code.

So no future scheduling pass will come to see a shared reg constraint that could've changed.

Quoted 8 lines

Now, there is indeed the intel_alt_er() thing, which slightly complicates this; then suppose we schedule an event on RSP0, another on RSP1, then remove the RSP0 one. Even in that case, the remaining RSP1 event will not change its constraint, since intel_fixup_er() rewrites the event to be a native RSP1 event.

So that too reduces to the prior case.

I came the same conclusion later yesterday. I think this is okay.

Quoted 11 lines

That said; I have simplified the above condition to:

arch/x86/events/core.c +2 -2
@@ -858,8 +858,17 @@ int x86_schedule_events(struct cpu_hw_ev
                x86_pmu.start_scheduling(cpuc);
 
        for (i = 0, wmin = X86_PMC_IDX_MAX, wmax = 0; i < n; i++) {
-               c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
-               cpuc->event_constraint[i] = c;
+               c = cpuc->event_constraint[i];
+

arch/x86/events/core.c int x86_schedule_events(struct cpu_hw_ev · line 872

Quoted 10 lines
arch/x86/events/core.c +9 -0
+               /*
+                * Request constraints for new events; or for those events that
+                * have a dynamic constraint -- for those the constraint can
+                * change due to external factors (sibling state, allow_tfa).
+                */
+               if (!c || (c->flags & PERF_X86_EVENT_DYNAMIC)) {
+                       c = x86_pmu.get_event_constraints(cpuc, i, cpuc->event_list[i]);
+                       cpuc->event_constraint[i] = c;
+               }
 

Right now DYNAMIC is only casued by the HT bug, but it could change later on but the logic here would remain. If HT workaround is disabled, then no evnt is tagged with
DYNAMIC.

Quoted 4 lines
arch/x86/events/core.c +0 -0
                wmin = min(wmin, c->weight);
                wmax = max(wmax, c->weight);

Because any dynamic event can change from one moment to the next.

I agree.

author maintainer
#44 lore raw

RFC PATCH 8/8 perf/x86: Add sanity checks to x86_schedule_events()

In reply to Peter Zijlstra
---
 arch/x86/events/core.c |   21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)
arch/x86/events/core.c +15 -6
@@ -855,17 +855,30 @@ int x86_schedule_events(struct cpu_hw_ev
 	struct event_constraint *c;
 	unsigned long used_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
 	struct perf_event *e;
-	int i, wmin, wmax, unsched = 0;
+	int n0, i, wmin, wmax, unsched = 0;
 	struct hw_perf_event *hwc;
 
 	bitmap_zero(used_mask, X86_PMC_IDX_MAX);
 
+	/*
+	 * Compute the number of events already present; see x86_pmu_add(),
+	 * validate_group() and x86_pmu_commit_txn(). For the former two
+	 * cpuc->n_events hasn't been updated yet, while for the latter
+	 * cpuc->n_txn contains the number of events added in the current
+	 * transaction.
+	 */
+	n0 = cpuc->n_events;
+	if (cpuc->txn_flags & PERF_PMU_TXN_ADD)
+		n0 -= cpuc->n_txn;
+
 	if (x86_pmu.start_scheduling)
 		x86_pmu.start_scheduling(cpuc);
 
 	for (i = 0, wmin = X86_PMC_IDX_MAX, wmax = 0; i < n; i++) {
 		c = cpuc->event_constraint[i];
 
+		WARN_ON_ONCE((c && i >= n0) || (!c && i < n0));
+
 		/*
 		 * Request constraints for new events; or for those events that
 		 * have a dynamic constraint due to the HT workaround -- for
@@ -945,11 +958,7 @@ int x86_schedule_events(struct cpu_hw_ev
 				x86_pmu.commit_scheduling(cpuc, i, assign[i]);
 		}
 	} else {
-		i = cpuc->n_events;
-		if (cpuc->txn_flags & PERF_PMU_TXN_ADD)
-			i -= cpuc->n_txn;
-
-		for (; i < n; i++) {
+		for (i = n0; i < n; i++) {
 			e = cpuc->event_list[i];
 
 			/*
#45 lore raw
In reply to Peter Zijlstra
Quoted 4 lines · Peter Zijlstra

These here patches are the result of chasing a bug in the TSX Force Abort patches.

Quoted 9 lines

The first patch is simple and fixes the issue and should be marked /urgent and backported.

The rest is the result of having a hard look at how all that event scheduling stuff worked while trying to figure out wth was going wrong.

Stephane, since you were involved with writing much of that code, can you please have a good look? I _think_ I got it, but... :-)

Will do today.
But you did not send me PATCH 1/8 apparently.

#46 lore raw
Quoted 9 lines · Stephane Eranian
Quoted 7 lines · Peter Zijlstra

These here patches are the result of chasing a bug in the TSX Force Abort patches.

The first patch is simple and fixes the issue and should be marked /urgent and backported.

Quoted 8 lines
Quoted 6 lines

The rest is the result of having a hard look at how all that event scheduling stuff worked while trying to figure out wth was going wrong.

Stephane, since you were involved with writing much of that code, can you please have a good look? I _think_ I got it, but... :-)

Will do today.
But you did not send me PATCH 1/8 apparently.

Correction : I got it. Mail client had just moved it somewhere else!

author maintainer
#47 lore raw
Quoted 13 lines · Stephane Eranian
Quoted 12 lines · Stephane Eranian
Quoted 10 lines · Peter Zijlstra

These here patches are the result of chasing a bug in the TSX Force Abort patches.

The first patch is simple and fixes the issue and should be marked /urgent and backported.

The rest is the result of having a hard look at how all that event scheduling stuff worked while trying to figure out wth was going wrong.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Stephane, since you were involved with writing much of that code, can you please have a good look? I _think_ I got it, but... :-)

Will do today.
But you did not send me PATCH 1/8 apparently.

Correction : I got it. Mail client had just moved it somewhere else!

Also please look at all 8, I'm reasonably sure 1/8 is correct, but it is rather more ugly than I'd wish. So the remaining 7 clean it all up (and effectively revert 1 again) but are much more invasive.