Performance regression in v3.14

After updating my main system from v3.13 to v3.14.2, I found that the git bash-completion was extremely sluggish. Completing a file name would take roughly six rather than one second on this Haswell machine (i7-4770). (Other things, such as git rebase, also felt slower, but the completion issue was much more obvious and easy to measure).

I managed to reproduce the problem using the following minimal construct

cat dmesg.repeat | while read x; do true; done

where dmesg.repeat is simply dmesg concatenated together to an equivalent number of lines as produced by git ls-files in the kernel-source tree root (45k), and where the actual processing of each line has been removed.

Most of the time I get:

        $ time cat dmesg.repeat | while read x; do true; done

        real    0m6.091s
        user    0m3.674s
        sys     0m2.447s

but sometimes it only takes one second.

        $ time cat dmesg.repeat | while read x; do true; done

        real    0m1.100s
        user    0m0.544s
        sys     0m0.570s

I don't seem to be able to reproduce the problem on 3.13 where the pipe always takes about one second to finish.

Taking all but one core offline seems to make the problem go away, and so does using the performance rather than powersave governor of the intel_pstate cpufreq driver (on at least one of two online cores).

Moving the mouse cursor makes to loop finish faster, and so does switching to a another terminal to print cpufreq/cpuinfo_cur_freq which was around cpuinfo_min_freq several times (when tracing, see below).

I could not reproduce the problem when using perf record, but I can get function-profile traces using ftrace (in which case the loop takes about 60 seconds instead of six seconds to finish).

Comparing the traces I see a lot of functions taking ten times longer to finish, but I guess that's expected if this is indeed a cpufreq issue.

Since this is my main machine (and only multi-core machine at the moment) I'm not able to bisect this myself. And for the same reason I have not verified that the problem persists in v3.15-rc.

I don't see any cpufreq patches in the v3.14.3 stable queue nor anything obviously related and marked for stable in v3.15-rc.

Any ideas about what might be going on?

Cc'ing Dirk who is taking care of intel-pstate driver.

Quoted 41 lines · Johan Hovold

After updating my main system from v3.13 to v3.14.2, I found that the git bash-completion was extremely sluggish. Completing a file name would take roughly six rather than one second on this Haswell machine (i7-4770). (Other things, such as git rebase, also felt slower, but the completion issue was much more obvious and easy to measure).

I managed to reproduce the problem using the following minimal construct

cat dmesg.repeat | while read x; do true; done

where dmesg.repeat is simply dmesg concatenated together to an equivalent number of lines as produced by git ls-files in the kernel-source tree root (45k), and where the actual processing of each line has been removed.

Most of the time I get:

        $ time cat dmesg.repeat | while read x; do true; done

        real    0m6.091s
        user    0m3.674s
        sys     0m2.447s

but sometimes it only takes one second.

        $ time cat dmesg.repeat | while read x; do true; done

        real    0m1.100s
        user    0m0.544s
        sys     0m0.570s

I don't seem to be able to reproduce the problem on 3.13 where the pipe always takes about one second to finish.

Taking all but one core offline seems to make the problem go away, and so does using the performance rather than powersave governor of the intel_pstate cpufreq driver (on at least one of two online cores).

Moving the mouse cursor makes to loop finish faster, and so does switching to a another terminal to print cpufreq/cpuinfo_cur_freq which was around cpuinfo_min_freq several times (when tracing, see below).

I tried to take a look at the diff for cpufreq between 3.13 and 3.14.2 and couldn't pin point on any change which might cause it. Don't have a clue of what's going on. I don't know how to help you on this.

Normally I test my stuff on a ARM board and I don't remember facing any such behavior there. There might be something wrong with intel-pstate as well..

Also, can you try to use acpi-cpufreq instead? And see how that is behaving?

So I guess that idle-active difference is normal for acpi-cpufreq and that the problem only arises in or with the intel_pstate driver.

Cc'ing Dirk who is taking care of intel-pstate driver.

Quoted 49 lines

After updating my main system from v3.13 to v3.14.2, I found that the git bash-completion was extremely sluggish. Completing a file name would take roughly six rather than one second on this Haswell machine (i7-4770). (Other things, such as git rebase, also felt slower, but the completion issue was much more obvious and easy to measure).

I managed to reproduce the problem using the following minimal construct

cat dmesg.repeat | while read x; do true; done

where dmesg.repeat is simply dmesg concatenated together to an equivalent number of lines as produced by git ls-files in the kernel-source tree root (45k), and where the actual processing of each line has been removed.

Most of the time I get:

         $ time cat dmesg.repeat | while read x; do true; done

         real    0m6.091s
         user    0m3.674s
         sys     0m2.447s

but sometimes it only takes one second.

         $ time cat dmesg.repeat | while read x; do true; done

         real    0m1.100s
         user    0m0.544s
         sys     0m0.570s

I don't seem to be able to reproduce the problem on 3.13 where the pipe always takes about one second to finish.

Taking all but one core offline seems to make the problem go away, and so does using the performance rather than powersave governor of the intel_pstate cpufreq driver (on at least one of two online cores).

Moving the mouse cursor makes to loop finish faster, and so does switching to a another terminal to print cpufreq/cpuinfo_cur_freq which was around cpuinfo_min_freq several times (when tracing, see below).

I could not reproduce the problem when using perf record, but I can get function-profile traces using ftrace (in which case the loop takes about 60 seconds instead of six seconds to finish).

Comparing the traces I see a lot of functions taking ten times longer to finish, but I guess that's expected if this is indeed a cpufreq issue.

Quoted 8 lines

Since this is my main machine (and only multi-core machine at the moment) I'm not able to bisect this myself. And for the same reason I have not verified that the problem persists in v3.15-rc.

I don't see any cpufreq patches in the v3.14.3 stable queue nor anything obviously related and marked for stable in v3.15-rc.

Any ideas about what might be going on?

I tried to take a look at the diff for cpufreq between 3.13 and 3.14.2 and couldn't pin point on any change which might cause it. Don't have a clue of what's going on. I don't know how to help you on this.

Normally I test my stuff on a ARM board and I don't remember facing any such behavior there. There might be something wrong with intel-pstate as well..

Also, can you try to use acpi-cpufreq instead? And see how that is behaving?

--
viresh

Quoted 2 lines · Viresh Kumar

Cc'ing Dirk who is taking care of intel-pstate driver.

Thanks Viresh I had seen this thread.

I am looking into it

Quoted 71 lines · Johan Hovold

Quoted 49 lines

After updating my main system from v3.13 to v3.14.2, I found that the git bash-completion was extremely sluggish. Completing a file name would take roughly six rather than one second on this Haswell machine (i7-4770). (Other things, such as git rebase, also felt slower, but the completion issue was much more obvious and easy to measure).

I managed to reproduce the problem using the following minimal construct

cat dmesg.repeat | while read x; do true; done

where dmesg.repeat is simply dmesg concatenated together to an equivalent number of lines as produced by git ls-files in the kernel-source tree root (45k), and where the actual processing of each line has been removed.

Most of the time I get:

         $ time cat dmesg.repeat | while read x; do true; done

         real    0m6.091s
         user    0m3.674s
         sys     0m2.447s

but sometimes it only takes one second.

         $ time cat dmesg.repeat | while read x; do true; done

         real    0m1.100s
         user    0m0.544s
         sys     0m0.570s

I don't seem to be able to reproduce the problem on 3.13 where the pipe always takes about one second to finish.

Taking all but one core offline seems to make the problem go away, and so does using the performance rather than powersave governor of the intel_pstate cpufreq driver (on at least one of two online cores).

Moving the mouse cursor makes to loop finish faster, and so does switching to a another terminal to print cpufreq/cpuinfo_cur_freq which was around cpuinfo_min_freq several times (when tracing, see below).

I could not reproduce the problem when using perf record, but I can get function-profile traces using ftrace (in which case the loop takes about 60 seconds instead of six seconds to finish).

Comparing the traces I see a lot of functions taking ten times longer to finish, but I guess that's expected if this is indeed a cpufreq issue.

Quoted 8 lines

Since this is my main machine (and only multi-core machine at the moment) I'm not able to bisect this myself. And for the same reason I have not verified that the problem persists in v3.15-rc.

I don't see any cpufreq patches in the v3.14.3 stable queue nor anything obviously related and marked for stable in v3.15-rc.

Any ideas about what might be going on?

I tried to take a look at the diff for cpufreq between 3.13 and 3.14.2 and couldn't pin point on any change which might cause it. Don't have a clue of what's going on. I don't know how to help you on this.

Normally I test my stuff on a ARM board and I don't remember facing any such behavior there. There might be something wrong with intel-pstate as well..

Also, can you try to use acpi-cpufreq instead? And see how that is behaving?

--
viresh

Quoted 6 lines · Dirk Brandewie

Quoted 2 lines · Viresh Kumar

Cc'ing Dirk who is taking care of intel-pstate driver.

Thanks Viresh I had seen this thread.

Quoted 1 line

I am looking into it

Any updates on this, Dirk? 3.14 is still basically unusable with the intel_pstate driver.

Any fixes or workarounds posted elsewhere that I can apply in the meantime?

I tried applying your (rejected) patch "intel_pstate: Remove C0 tracking" posted here:

	https://lkml.org/lkml/2014/5/8/574

to v3.14.4 and it fixes the problem as expected.

So we have a commit fcb6a15c2e7e ("intel_pstate: Take core C0 time into account for core busy calculation") that went into v3.14-rc2 (and was even marked for *stable*) that first broke Greg KH's system:

	https://lkml.org/lkml/2014/2/19/626

That was apparently fixed by e66c17683746 ("intel_pstate: Change busy calculation to use fixed point math."), but still left v3.14 basically unusable for lower-intensity workloads such as my bash-completion example and other reported regressions:

	https://bugzilla.kernel.org/show_bug.cgi?id=75121

Sure there may be issues with v3.13 not hitting the lowest frequencies but at least the system was *usable*.

In my opinion there's really no other option than to restore the 3.13 behaviour by effectively reverting fcb6a15c2e7e ("intel_pstate: Take core C0 time into account for core busy calculation") until you have figured out a way to take C0 into account without breaking things too badly.

Quoted 21 lines · Johan Hovold

I tried applying your (rejected) patch "intel_pstate: Remove C0 tracking" posted here:

	https://lkml.org/lkml/2014/5/8/574

to v3.14.4 and it fixes the problem as expected.

So we have a commit fcb6a15c2e7e ("intel_pstate: Take core C0 time into account for core busy calculation") that went into v3.14-rc2 (and was even marked for *stable*) that first broke Greg KH's system:

	https://lkml.org/lkml/2014/2/19/626

That was apparently fixed by e66c17683746 ("intel_pstate: Change busy calculation to use fixed point math."), but still left v3.14 basically unusable for lower-intensity workloads such as my bash-completion example and other reported regressions:

	https://bugzilla.kernel.org/show_bug.cgi?id=75121

Quoted 8 lines

Sure there may be issues with v3.13 not hitting the lowest frequencies but at least the system was *usable*.

In my opinion there's really no other option than to restore the 3.13 behaviour by effectively reverting fcb6a15c2e7e ("intel_pstate: Take core C0 time into account for core busy calculation") until you have figured out a way to take C0 into account without breaking things too badly.

Hi all,

My posts before and now are only relevant to why C0 tracking can't be removed. Maybe I need to elaborate on it a little bit more.

In a nutshell, without C0 tracking, the intel_pstate is effectively performance governor in terms of frequency control.

Why? Without C0 trakcing, the machinery of the freq control is as I formed:
last_freq_average / last_requested_freq ==> setpoint which can be virtually formed into:
last_freq_average / last_requested_freq * last_C0_pct ==> setpoint * last_C0_pct

which said, the control machinery will increase the frequency at ANY frequency at ANY C0_pct (which is the CPU utilization), since setpoint is less then 100 percent.

And a few iterations
later, we will reach max (possible) frequency,
then we are effectively performance governor
(highest frequency all the time).

That is not true. Yes, and due to the setpoint being less than 100, which is needed or the driver won't work at all, there is a tendency to drive the target pstate upwards. However that is tempered by both the PID proportional gain, and ultimately integer math. More importantly, the CPU itself tells the driver when it is operating below the target pstate and driver responds.

Additionally, the tendency to drive up the target pstate too much is exasperated by some extra rounding up at a couple of spots. Dirk has a pending fix.

Quoted 4 lines

And a few iterations
later, we will reach max (possible) frequency, then we are effectively performance governor (highest frequency all the time).

Please do not confuse highest target pstate with highest frequency. They are not the same. The processor itself can back off.

[ +CC: Greg, Doug, Stratos, Yuyang ]

Quoted 7 lines · Johan Hovold

Quoted 6 lines · Dirk Brandewie

Quoted 2 lines · Viresh Kumar

Cc'ing Dirk who is taking care of intel-pstate driver.

Thanks Viresh I had seen this thread.

Quoted 7 lines

Quoted 1 line

I am looking into it

Any updates on this, Dirk? 3.14 is still basically unusable with the intel_pstate driver.

Any fixes or workarounds posted elsewhere that I can apply in the meantime?

Another week and still no reply, Dirk?

I tried applying your (rejected) patch "intel_pstate: Remove C0 tracking" posted here:

	https://lkml.org/lkml/2014/5/8/574

to v3.14.4 and it fixes the problem as expected.

So we have a commit fcb6a15c2e7e ("intel_pstate: Take core C0 time into account for core busy calculation") that went into v3.14-rc2 (and was even marked for *stable*) that first broke Greg KH's system:

	https://lkml.org/lkml/2014/2/19/626

That was apparently fixed by e66c17683746 ("intel_pstate: Change busy calculation to use fixed point math."), but still left v3.14 basically unusable for lower-intensity workloads such as my bash-completion example and other reported regressions:

	https://bugzilla.kernel.org/show_bug.cgi?id=75121

Sure there may be issues with v3.13 not hitting the lowest frequencies but at least the system was *usable*.

In my opinion there's really no other option than to restore the 3.13 behaviour by effectively reverting fcb6a15c2e7e ("intel_pstate: Take core C0 time into account for core busy calculation") until you have figured out a way to take C0 into account without breaking things too badly.

Quoted 43 lines · Johan Hovold

[ +CC: Greg, Doug, Stratos, Yuyang ]

Quoted 7 lines · Johan Hovold

Quoted 6 lines · Dirk Brandewie

Quoted 2 lines · Viresh Kumar

Cc'ing Dirk who is taking care of intel-pstate driver.

Thanks Viresh I had seen this thread.

Quoted 7 lines

Quoted 1 line

I am looking into it

Any updates on this, Dirk? 3.14 is still basically unusable with the intel_pstate driver.

Any fixes or workarounds posted elsewhere that I can apply in the meantime?

Another week and still no reply, Dirk?

I tried applying your (rejected) patch "intel_pstate: Remove C0 tracking" posted here:

	https://lkml.org/lkml/2014/5/8/574

to v3.14.4 and it fixes the problem as expected.

So we have a commit fcb6a15c2e7e ("intel_pstate: Take core C0 time into account for core busy calculation") that went into v3.14-rc2 (and was even marked for *stable*) that first broke Greg KH's system:

	https://lkml.org/lkml/2014/2/19/626

That was apparently fixed by e66c17683746 ("intel_pstate: Change busy calculation to use fixed point math."), but still left v3.14 basically unusable for lower-intensity workloads such as my bash-completion example and other reported regressions:

	https://bugzilla.kernel.org/show_bug.cgi?id=75121

Sure there may be issues with v3.13 not hitting the lowest frequencies but at least the system was *usable*.

Quoted 5 lines

In my opinion there's really no other option than to restore the 3.13 behaviour by effectively reverting fcb6a15c2e7e ("intel_pstate: Take core C0 time into account for core busy calculation") until you have figured out a way to take C0 into account without breaking things too badly.

Dirk has posted some patches, do they fix the problem for you?

Quoted 43 lines · Johan Hovold

[ +CC: Greg, Doug, Stratos, Yuyang ]

Quoted 7 lines · Johan Hovold

Quoted 6 lines · Dirk Brandewie

Quoted 2 lines · Viresh Kumar

Cc'ing Dirk who is taking care of intel-pstate driver.

Thanks Viresh I had seen this thread.

Quoted 7 lines

Quoted 1 line

I am looking into it

Any updates on this, Dirk? 3.14 is still basically unusable with the intel_pstate driver.

Any fixes or workarounds posted elsewhere that I can apply in the meantime?

Another week and still no reply, Dirk?

I tried applying your (rejected) patch "intel_pstate: Remove C0 tracking" posted here:

	https://lkml.org/lkml/2014/5/8/574

to v3.14.4 and it fixes the problem as expected.

So we have a commit fcb6a15c2e7e ("intel_pstate: Take core C0 time into account for core busy calculation") that went into v3.14-rc2 (and was even marked for *stable*) that first broke Greg KH's system:

	https://lkml.org/lkml/2014/2/19/626

That was apparently fixed by e66c17683746 ("intel_pstate: Change busy calculation to use fixed point math."), but still left v3.14 basically unusable for lower-intensity workloads such as my bash-completion example and other reported regressions:

	https://bugzilla.kernel.org/show_bug.cgi?id=75121

Sure there may be issues with v3.13 not hitting the lowest frequencies but at least the system was *usable*.

Quoted 5 lines

In my opinion there's really no other option than to restore the 3.13 behaviour by effectively reverting fcb6a15c2e7e ("intel_pstate: Take core C0 time into account for core busy calculation") until you have figured out a way to take C0 into account without breaking things too badly.

Dirk has posted some patches, do they fix the problem for you?