Skip to contentSkip to conversation

v4.18-rc1 on droid 4: very bad CPU performance

DISCUSSION Pavel Machek 2 participants 11 messages Jun 21, 2018 – Jul 4, 2018

lore
Share & tools
lore
#0 lore raw

Hi!

V4.18 is slower than it should be.

user at devuan:~$ time cat /dev/urandom | head -c 10000000 | bzip2 -9 - | wc -c
10044291
52.73user 2.40system 61.53 (1m1.534s) elapsed 89.60%CPU
user at devuan:~$ uname -a
Linux devuan 4.18.0-rc1-87964-gfa19934-dirty #743 SMP Sun Jun 17 19:26:37 CEST 2018 armv7l GNU/Linux

That bzip should take 12 seconds, not minute.

Any ideas? Do you see it, too?

Best regards,

Pavel

Signature · 9 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180621/aae8df5a/attachment.sig>
#1 lore raw
Quoted 12 lines · Pavel Machek

Hi!

V4.18 is slower than it should be.

user at devuan:~$ time cat /dev/urandom | head -c 10000000 | bzip2 -9 - | wc -c
10044291
52.73user 2.40system 61.53 (1m1.534s) elapsed 89.60%CPU
user at devuan:~$ uname -a
Linux devuan 4.18.0-rc1-87964-gfa19934-dirty #743 SMP Sun Jun 17 19:26:37 CEST 2018 armv7l GNU/Linux

Quoted 3 lines

That bzip should take 12 seconds, not minute.

Any ideas? Do you see it, too?

I've noticed firefox is super slow.. Git bisect time?

Regards,

Tony

#2 lore raw
Quoted 13 lines · Tony Lindgren
Quoted 12 lines · Pavel Machek

Hi!

V4.18 is slower than it should be.

user at devuan:~$ time cat /dev/urandom | head -c 10000000 | bzip2 -9 - | wc -c
10044291
52.73user 2.40system 61.53 (1m1.534s) elapsed 89.60%CPU
user at devuan:~$ uname -a
Linux devuan 4.18.0-rc1-87964-gfa19934-dirty #743 SMP Sun Jun 17 19:26:37 CEST 2018 armv7l GNU/Linux

Quoted 5 lines
Quoted 3 lines

That bzip should take 12 seconds, not minute.

Any ideas? Do you see it, too?

I've noticed firefox is super slow.. Git bisect time?

Hmm not happening for me at least on duovero and droid4, I'm getting about 20s for v4.17 and v4.18-rc1.

Regards,

Tony

#3 lore raw

Hi!

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

V4.18 is slower than it should be.

user at devuan:~$ time cat /dev/urandom | head -c 10000000 | bzip2 -9 - | wc -c
10044291
52.73user 2.40system 61.53 (1m1.534s) elapsed 89.60%CPU
user at devuan:~$ uname -a
Linux devuan 4.18.0-rc1-87964-gfa19934-dirty #743 SMP Sun Jun 17 19:26:37 CEST 2018 armv7l GNU/Linux

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

That bzip should take 12 seconds, not minute.

Any ideas? Do you see it, too?

I've noticed firefox is super slow.. Git bisect time?

Hmm not happening for me at least on duovero and droid4, I'm getting about 20s for v4.17 and v4.18-rc1.

Something weird is going on here. My notes say it should be 12 seconds.

It is indeed 20 seconds for v4.17, but it is 12 seconds in v4.14.

v4.14 is:
# CONFIG_CPU_FREQ is not set
CONFIG_CPU_IDLE=y
# CONFIG_CPU_IDLE_GOV_LADDER is not set
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_ARM_CPUIDLE is not set

In v4.18, I have:

CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_ATTR_SET=y
CONFIG_CPU_FREQ_GOV_COMMON=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y

CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_MENU=y
CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED=y

Pavel

Signature · 9 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180625/32dadc81/attachment.sig>
#4 lore raw
Quoted 41 lines · Pavel Machek

Hi!

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

V4.18 is slower than it should be.

user at devuan:~$ time cat /dev/urandom | head -c 10000000 | bzip2 -9 - | wc -c
10044291
52.73user 2.40system 61.53 (1m1.534s) elapsed 89.60%CPU
user at devuan:~$ uname -a
Linux devuan 4.18.0-rc1-87964-gfa19934-dirty #743 SMP Sun Jun 17 19:26:37 CEST 2018 armv7l GNU/Linux

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

That bzip should take 12 seconds, not minute.

Any ideas? Do you see it, too?

I've noticed firefox is super slow.. Git bisect time?

Hmm not happening for me at least on duovero and droid4, I'm getting about 20s for v4.17 and v4.18-rc1.

Something weird is going on here. My notes say it should be 12 seconds.

It is indeed 20 seconds for v4.17, but it is 12 seconds in v4.14.

v4.14 is:
# CONFIG_CPU_FREQ is not set
CONFIG_CPU_IDLE=y
# CONFIG_CPU_IDLE_GOV_LADDER is not set
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_ARM_CPUIDLE is not set

In v4.18, I have:

CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_ATTR_SET=y
CONFIG_CPU_FREQ_GOV_COMMON=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y

Quoted 3 lines

CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_MENU=y
CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED=y

I gave it a quick try here with v4.14 and it was 21 seconds on 4430 duovero. But this was with omap2plus_defconfig.

Regards,

Tony

#5 lore raw
Quoted 42 lines · Tony Lindgren
Quoted 41 lines · Pavel Machek

Hi!

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

V4.18 is slower than it should be.

user at devuan:~$ time cat /dev/urandom | head -c 10000000 | bzip2 -9 - | wc -c
10044291
52.73user 2.40system 61.53 (1m1.534s) elapsed 89.60%CPU
user at devuan:~$ uname -a
Linux devuan 4.18.0-rc1-87964-gfa19934-dirty #743 SMP Sun Jun 17 19:26:37 CEST 2018 armv7l GNU/Linux

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

That bzip should take 12 seconds, not minute.

Any ideas? Do you see it, too?

I've noticed firefox is super slow.. Git bisect time?

Hmm not happening for me at least on duovero and droid4, I'm getting about 20s for v4.17 and v4.18-rc1.

Something weird is going on here. My notes say it should be 12 seconds.

It is indeed 20 seconds for v4.17, but it is 12 seconds in v4.14.

v4.14 is:
# CONFIG_CPU_FREQ is not set
CONFIG_CPU_IDLE=y
# CONFIG_CPU_IDLE_GOV_LADDER is not set
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_ARM_CPUIDLE is not set

In v4.18, I have:

CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_ATTR_SET=y
CONFIG_CPU_FREQ_GOV_COMMON=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y

Quoted 6 lines
Quoted 3 lines

CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_MENU=y
CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED=y

I gave it a quick try here with v4.14 and it was 21 seconds on 4430 duovero. But this was with omap2plus_defconfig.

Weird. In v4.18 I seen times from 57 seconds to 27 seconds to 20 seconds. It seems to be consistent after boot.

v4.14 is fast... 12 second. v4.15 (all-v4.15 branch) -- no boot. all-v4.16 -- 15.7 seconds. all-v4.18 with "all" config: time varies (20, 24, 27, 57 seconds). all-v4.18 with "droid4" config: 20 seconds.

What needs to be done to get cpufreq to work? One of my hypothesis is that cpu is running at wrong rate. I have it enabled in config, but no luck:

user at devuan:~$ zcat /proc/config.gz | grep CPU_FREQ CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_ATTR_SET=y CONFIG_CPU_FREQ_GOV_COMMON=y
CONFIG_CPU_FREQ_STAT=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set CONFIG_CPU_FREQ_GOV_PERFORMANCE=y CONFIG_CPU_FREQ_GOV_POWERSAVE=y CONFIG_CPU_FREQ_GOV_USERSPACE=y CONFIG_CPU_FREQ_GOV_ONDEMAND=y CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
# CONFIG_CPU_FREQ_GOV_SCHEDUTIL is not set
user at devuan:~$ sudo cpufreq-info
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq at vger.kernel.org, please. analyzing CPU 0:
no or unknown cpufreq driver is active on this CPU
maximum transition latency: 0.00 ms.
analyzing CPU 1:
no or unknown cpufreq driver is active on this CPU maximum transition latency: 0.00 ms.
user at devuan:~$

Best regards,

Pavel

Signature · 9 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180627/387e589d/attachment.sig>
#6 lore raw
In reply to Tony Lindgren
Quoted 42 lines · Tony Lindgren
Quoted 41 lines · Pavel Machek

Hi!

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

V4.18 is slower than it should be.

user at devuan:~$ time cat /dev/urandom | head -c 10000000 | bzip2 -9 - | wc -c
10044291
52.73user 2.40system 61.53 (1m1.534s) elapsed 89.60%CPU
user at devuan:~$ uname -a
Linux devuan 4.18.0-rc1-87964-gfa19934-dirty #743 SMP Sun Jun 17 19:26:37 CEST 2018 armv7l GNU/Linux

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

That bzip should take 12 seconds, not minute.

Any ideas? Do you see it, too?

I've noticed firefox is super slow.. Git bisect time?

Hmm not happening for me at least on duovero and droid4, I'm getting about 20s for v4.17 and v4.18-rc1.

Something weird is going on here. My notes say it should be 12 seconds.

It is indeed 20 seconds for v4.17, but it is 12 seconds in v4.14.

v4.14 is:
# CONFIG_CPU_FREQ is not set
CONFIG_CPU_IDLE=y
# CONFIG_CPU_IDLE_GOV_LADDER is not set
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_ARM_CPUIDLE is not set

In v4.18, I have:

CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_ATTR_SET=y
CONFIG_CPU_FREQ_GOV_COMMON=y
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y

Quoted 6 lines
Quoted 3 lines

CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_MENU=y
CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED=y

I gave it a quick try here with v4.14 and it was 21 seconds on 4430 duovero. But this was with omap2plus_defconfig.

Ok, so I was able to create .config where bzip2 takes 12 seconds, on v4.18...

In my normal config, it takes between 20 seconds and minute. Numbers seem to be stable between boots.

In "fast" config:
processor : 1
model name : ARMv7 Processor rev 3 (v7l)
BogoMIPS : 2393.70
Features : half thumb fastmult vfp edsp thumbee neon vfpv3 tls
vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x1
CPU part : 0xc09
CPU revision : 3

12 to 20 seconds is .. pretty significant slowdown. Is it possible that "fast" config misses errata workaround with impact this significant?

Best regards,
Pavel

Signature · 16 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: delme.gz
Type: application/gzip
Size: 24600 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180627/daff125a/attachment-0001.gz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180627/daff125a/attachment-0001.sig>
#7 lore raw
In reply to Tony Lindgren
Quoted 14 lines · Tony Lindgren
Quoted 13 lines · Tony Lindgren
Quoted 12 lines · Pavel Machek

Hi!

V4.18 is slower than it should be.

user at devuan:~$ time cat /dev/urandom | head -c 10000000 | bzip2 -9 - | wc -c
10044291
52.73user 2.40system 61.53 (1m1.534s) elapsed 89.60%CPU
user at devuan:~$ uname -a
Linux devuan 4.18.0-rc1-87964-gfa19934-dirty #743 SMP Sun Jun 17 19:26:37 CEST 2018 armv7l GNU/Linux

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

That bzip should take 12 seconds, not minute.

Any ideas? Do you see it, too?

I've noticed firefox is super slow.. Git bisect time?

Hmm not happening for me at least on duovero and droid4, I'm getting about 20s for v4.17 and v4.18-rc1.

Okay, so I know what was wrong with my config. I had l2cache disabled -- and this solved it:

+CONFIG_OUTER_CACHE=y
+CONFIG_OUTER_CACHE_SYNC=y
+CONFIG_CACHE_L2X0=y
+CONFIG_PL310_ERRATA_588369=y
+CONFIG_PL310_ERRATA_727915=y
+CONFIG_PL310_ERRATA_753970=y
+CONFIG_PL310_ERRATA_769419=y

Now I'm at 12 seconds.

user at devuan:~$ time cat /dev/urandom | head -c 10000000 | bzip2 -9 - | wc -c 10043993
11.41user 0.44system 11.63 (0m11.637s) elapsed 101.97%CPU
user at devuan:~$ uname -a
Linux devuan 4.18.0-rc2-88315-g3452793-dirty #756 SMP Wed Jun 27 22:16:59 CEST 2018 armv7l GNU/Linux user at devuan:~$

I was about to add "facepalm" emoji here, as disabling l2 cache is major mistake. OTOH if you also see 20 seconds, then either there is another problem, or you need to enable l2 cache, too (and I don't have to feel too bad for misconfiguring it).

Best regards,

Pavel

Signature · 9 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180627/8a34fb90/attachment.sig>
#8 lore raw
Quoted 27 lines · Pavel Machek
Quoted 14 lines · Tony Lindgren
Quoted 13 lines · Tony Lindgren
Quoted 12 lines · Pavel Machek

Hi!

V4.18 is slower than it should be.

user at devuan:~$ time cat /dev/urandom | head -c 10000000 | bzip2 -9 - | wc -c
10044291
52.73user 2.40system 61.53 (1m1.534s) elapsed 89.60%CPU
user at devuan:~$ uname -a
Linux devuan 4.18.0-rc1-87964-gfa19934-dirty #743 SMP Sun Jun 17 19:26:37 CEST 2018 armv7l GNU/Linux

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

That bzip should take 12 seconds, not minute.

Any ideas? Do you see it, too?

I've noticed firefox is super slow.. Git bisect time?

Hmm not happening for me at least on duovero and droid4, I'm getting about 20s for v4.17 and v4.18-rc1.

Okay, so I know what was wrong with my config. I had l2cache disabled -- and this solved it:

Review

Quoted 5 lines
(quoted diff) +5 -0
+CONFIG_OUTER_CACHE=y
+CONFIG_OUTER_CACHE_SYNC=y
+CONFIG_CACHE_L2X0=y
+CONFIG_PL310_ERRATA_588369=y
+CONFIG_PL310_ERRATA_727915=y

We have these in omap2plus_defconfig.

Review

Quoted 2 lines
(quoted diff) +2 -0
+CONFIG_PL310_ERRATA_753970=y
+CONFIG_PL310_ERRATA_769419=y

These two should not be needed from what I recall. Have not checked against processor revisions or errata though. But I verified that my duovero time stays at 21 seconds also with these enabled. It seems to run at 600MHz so it's slower compared to droid4.

Quoted 9 lines

Now I'm at 12 seconds.

user at devuan:~$ time cat /dev/urandom | head -c 10000000 | bzip2 -9 - | wc -c 10043993
11.41user 0.44system 11.63 (0m11.637s) elapsed 101.97%CPU
user at devuan:~$ uname -a
Linux devuan 4.18.0-rc2-88315-g3452793-dirty #756 SMP Wed Jun 27 22:16:59 CEST 2018 armv7l GNU/Linux user at devuan:~$

Quoted 4 lines

I was about to add "facepalm" emoji here, as disabling l2 cache is major mistake. OTOH if you also see 20 seconds, then either there is another problem, or you need to enable l2 cache, too (and I don't have to feel too bad for misconfiguring it).

OK. I checked and duovero runs only at 600MHz. I can only get down to 19 seconds with omap2plus_defconfig if I disable CONFIG_CPU_FREQ so the max speed is maintained from bootloader. Otherwise it runs at most at 1GHz, I think any higher speeds need smartreflex and voltage scaling working which we don't have in the mainline kernel. Not sure what happens with higher speeds with mainline kernel or if they are unsafe in the long run.

Anyways, care to post your .config somewhere so I can see what the numbers are for me with it?

Regards,

Tony

#9 lore raw

Hi!

Quoted 14 lines
Quoted 4 lines

I was about to add "facepalm" emoji here, as disabling l2 cache is major mistake. OTOH if you also see 20 seconds, then either there is another problem, or you need to enable l2 cache, too (and I don't have to feel too bad for misconfiguring it).

OK. I checked and duovero runs only at 600MHz. I can only get down to 19 seconds with omap2plus_defconfig if I disable CONFIG_CPU_FREQ so the max speed is maintained from bootloader. Otherwise it runs at most at 1GHz, I think any higher speeds need smartreflex and voltage scaling working which we don't have in the mainline kernel. Not sure what happens with higher speeds with mainline kernel or if they are unsafe in the long run.

Quoted 2 lines

Anyways, care to post your .config somewhere so I can see what the numbers are for me with it?

I'm attaching config I'm using for v4.18. In current boot, bzip2 takes 16 seconds.

user at devuan:~$ time cat /dev/urandom | head -c 10000000 | sudo nice -n -19 bzip2 -9 - | wc -c
10044705
15.68user 0.53system 15.93 (0m15.931s) elapsed 101.82%CPU

I guess I should go back to v4.17 to get 12 seconds..?!

Confused,
Pavel

Signature · 16 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.gz
Type: application/gzip
Size: 29025 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180704/2d45ee9c/attachment-0001.gz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180704/2d45ee9c/attachment-0001.sig>
#10 lore raw
In reply to Tony Lindgren

Hi!

Quoted 11 lines

OK. I checked and duovero runs only at 600MHz. I can only get down to 19 seconds with omap2plus_defconfig if I disable CONFIG_CPU_FREQ so the max speed is maintained from bootloader. Otherwise it runs at most at 1GHz, I think any higher speeds need smartreflex and voltage scaling working which we don't have in the mainline kernel. Not sure what happens with higher speeds with mainline kernel or if they are unsafe in the long run.

Anyways, care to post your .config somewhere so I can see what the numbers are for me with it?

After reboot (same config as in previous mail), before x / ofono / unicsy_demo /... were started:

user at devuan:~$ uname -a
Linux devuan 4.18.0-rc3-88788-g528947b #758 SMP Wed Jul 4 10:05:21 CEST 2018 armv7l GNU/Linux
user at devuan:~$ time cat /dev/urandom | head -c 10000000 | bzip2 -9 - | wc -c
10044041
11.50user 0.37system 11.57 (0m11.570s) elapsed 102.63%CPU user at devuan:~$

droid 4.

Pavel

Signature · 9 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20180704/b53d01f6/attachment.sig>