Skip to contentSkip to conversation

qt5: use ccache for building host code

PATCH New Thomas Petazzoni 2 participants 9 messages Oct 20, 2018 – Nov 30, 2018

lore
Share & tools
lore

Thomas Petazzoni

author
#0 lore raw

PATCH 1/1 qt5: use ccache for building host code

qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache when building all its host code (especially qmake). This means that even with ccache enabled and a hot cache, it still takes a long time to build qt5base.

Before this patch, building qt5base takes:

- 446 seconds with a cold ccache
- 185 seconds with a hot ccache

This is because the ccache is not used for host code.

After this patch, building qt5base takes:

- 450 seconds with a cold ccache
- 15 seconds with a hot ccache

Signed-off-by Thomas Petazzoni <[email protected]>
---
 package/qt5/qt5base/qt5base.mk | 8 ++++++++
 1 file changed, 8 insertions(+)
package/qt5/qt5base/qt5base.mk +8 -0
index ab2b883fc4..d9f7af7196 100644
@@ -264,6 +264,13 @@ define QT5BASE_CONFIGURE_ARCH_CONFIG
 endef
 endif
 
+# This allows to use ccache when available
+define QT5BASE_CONFIGURE_HOSTCC
+	$(SED) 's,^QMAKE_COMPILER\s*=.*,QMAKE_COMPILER=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
+	$(SED) 's,^QMAKE_CC\s*=.*,QMAKE_CC=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
+	$(SED) 's,^QMAKE_CXX\s*=.*,QMAKE_CXX=$(HOSTCXX),' $(@D)/mkspecs/common/g++-base.conf
+endef
+
 define QT5BASE_CONFIGURE_CMDS
 	mkdir -p $(@D)/mkspecs/devices/linux-buildroot-g++/
 	sed 's/@EGLFS_DEVICE@/$(QT5BASE_EGLFS_DEVICE)/g' \

@@ -274,6 +281,7 @@ define QT5BASE_CONFIGURE_CMDS
 	$(QT5BASE_CONFIGURE_CONFIG_FILE)
 	touch $(QT5BASE_ARCH_CONFIG_FILE)
 	$(QT5BASE_CONFIGURE_ARCH_CONFIG)
+	$(QT5BASE_CONFIGURE_HOSTCC)
 	(cd $(@D); \
 		$(TARGET_MAKE_ENV) \
 		PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
Signature · 1 line
2.14.4
author
#1 lore raw

PATCH 1/1 qt5: use ccache for building host code

Hello Peter,

Quoted 10 lines · Thomas Petazzoni

qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache when building all its host code (especially qmake). This means that even with ccache enabled and a hot cache, it still takes a long time to build qt5base.

Before this patch, building qt5base takes:

- 446 seconds with a cold ccache
- 185 seconds with a hot ccache

Quoted 8 lines

This is because the ccache is not used for host code.

After this patch, building qt5base takes:

- 450 seconds with a cold ccache
- 15 seconds with a hot ccache

Signed-off-by Thomas Petazzoni <[email protected]>

Any comment on this patch ?

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH 1/1 qt5: use ccache for building host code

Hello Thomas,

Quoted 15 lines · Thomas Petazzoni

Hello Peter,

Quoted 12 lines · Thomas Petazzoni

qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache when building all its host code (especially qmake). This means that even with ccache enabled and a hot cache, it still takes a long time to build qt5base.

Before this patch, building qt5base takes:

- 446 seconds with a cold ccache
- 185 seconds with a hot ccache

This is because the ccache is not used for host code.

Quoted 8 lines
Quoted 6 lines

After this patch, building qt5base takes:

- 450 seconds with a cold ccache
- 15 seconds with a hot ccache

Signed-off-by Thomas Petazzoni <[email protected]>

Any comment on this patch ?

Sorry, not an regular ccache user (maybe an mistake ;-) ), so I did not examine this patch yet, nice speedup, some questions at he patch reply...

Regards,
Peter

Quoted 4 lines

Thanks!

Thomas

#3 lore raw

PATCH 1/1 qt5: use ccache for building host code

In reply to Thomas Petazzoni

Hello Thomas,

Quoted 27 lines · Thomas Petazzoni

qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache when building all its host code (especially qmake). This means that even with ccache enabled and a hot cache, it still takes a long time to build qt5base.

Before this patch, building qt5base takes:

- 446 seconds with a cold ccache
- 185 seconds with a hot ccache

This is because the ccache is not used for host code.

After this patch, building qt5base takes:

- 450 seconds with a cold ccache
- 15 seconds with a hot ccache

Signed-off-by Thomas Petazzoni <[email protected]>
---
 package/qt5/qt5base/qt5base.mk | 8 ++++++++
 1 file changed, 8 insertions(+)
package/qt5/qt5base/qt5base.mk +0 -0
index ab2b883fc4..d9f7af7196 100644

package/qt5/qt5base/qt5base.mk define QT5BASE_CONFIGURE_ARCH_CONFIG · line 273

Quoted 10 lines
package/qt5/qt5base/qt5base.mk +7 -0
@@ -264,6 +264,13 @@ define QT5BASE_CONFIGURE_ARCH_CONFIG
 endef
 endif
 
+# This allows to use ccache when available
+define QT5BASE_CONFIGURE_HOSTCC
+	$(SED) 's,^QMAKE_COMPILER\s*=.*,QMAKE_COMPILER=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
+	$(SED) 's,^QMAKE_CC\s*=.*,QMAKE_CC=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
+	$(SED) 's,^QMAKE_CXX\s*=.*,QMAKE_CXX=$(HOSTCXX),' $(@D)/mkspecs/common/g++-base.conf
+endef
+

With this sed commands applied, a qmake host compile is always using ccache (same behavior already for the cross/target compile?)?

Regards,
Peter

Quoted 11 lines
package/qt5/qt5base/qt5base.mk +1 -0
 define QT5BASE_CONFIGURE_CMDS
 	mkdir -p $(@D)/mkspecs/devices/linux-buildroot-g++/
 	sed 's/@EGLFS_DEVICE@/$(QT5BASE_EGLFS_DEVICE)/g' \

@@ -274,6 +281,7 @@ define QT5BASE_CONFIGURE_CMDS
 	$(QT5BASE_CONFIGURE_CONFIG_FILE)
 	touch $(QT5BASE_ARCH_CONFIG_FILE)
 	$(QT5BASE_CONFIGURE_ARCH_CONFIG)
+	$(QT5BASE_CONFIGURE_HOSTCC)
 	(cd $(@D); \
 		$(TARGET_MAKE_ENV) \
 		PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
author
#4 lore raw

PATCH 1/1 qt5: use ccache for building host code

Hello Peter,

Thanks for looking at this patch!

Quoted 10 lines · Peter Seiderer

package/qt5/qt5base/qt5base.mk define QT5BASE_CONFIGURE_ARCH_CONFIG · line 273

Quoted 7 lines
package/qt5/qt5base/qt5base.mk +7 -0
+# This allows to use ccache when available
+define QT5BASE_CONFIGURE_HOSTCC
+	$(SED) 's,^QMAKE_COMPILER\s*=.*,QMAKE_COMPILER=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
+	$(SED) 's,^QMAKE_CC\s*=.*,QMAKE_CC=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
+	$(SED) 's,^QMAKE_CXX\s*=.*,QMAKE_CXX=$(HOSTCXX),' $(@D)/mkspecs/common/g++-base.conf
+endef
+

With this sed commands applied, a qmake host compile is always using ccache (same behavior already for the cross/target compile?)?

I'm not sure to understand what you mean here. This change ensures that qmake is built with $(HOSTCC) / $(HOSTCXX) instead of gcc/g++.

HOSTCC is set to "gcc" and HOSTCXX to "g++", except when BR2_CCACHE=y, in which case HOSTCC is "ccache gcc" and HOSTCXX is "ccache g++".

So basically, this patch ensures that qmake is built with what Buildroot considers to be the "host compiler", which may or may not have ccache enabled.

Does this answer your question, or perhaps I misunderstood it completely ?

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

PATCH 1/1 qt5: use ccache for building host code

Hello Thomas,

Quoted 23 lines · Thomas Petazzoni

Hello Peter,

Thanks for looking at this patch!

Quoted 10 lines · Peter Seiderer

package/qt5/qt5base/qt5base.mk define QT5BASE_CONFIGURE_ARCH_CONFIG · line 273

Quoted 7 lines
package/qt5/qt5base/qt5base.mk +7 -0
+# This allows to use ccache when available
+define QT5BASE_CONFIGURE_HOSTCC
+	$(SED) 's,^QMAKE_COMPILER\s*=.*,QMAKE_COMPILER=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
+	$(SED) 's,^QMAKE_CC\s*=.*,QMAKE_CC=$(HOSTCC),' $(@D)/mkspecs/common/g++-base.conf
+	$(SED) 's,^QMAKE_CXX\s*=.*,QMAKE_CXX=$(HOSTCXX),' $(@D)/mkspecs/common/g++-base.conf
+endef
+

With this sed commands applied, a qmake host compile is always using ccache (same behavior already for the cross/target compile?)?

I'm not sure to understand what you mean here. This change ensures that qmake is built with $(HOSTCC) / $(HOSTCXX) instead of gcc/g++.

HOSTCC is set to "gcc" and HOSTCXX to "g++", except when BR2_CCACHE=y, in which case HOSTCC is "ccache gcc" and HOSTCXX is "ccache g++".

Quoted 6 lines

So basically, this patch ensures that qmake is built with what Buildroot considers to be the "host compiler", which may or may not have ccache enabled.

Does this answer your question, or perhaps I misunderstood it completely ?

Thanks for the answer (you got it right)...., so in case I use the buildroot-generated qmake I will always use ccache (if configured), even if used outside of the buildroot build system, right?

The $(HOSTCC) is expanded at the sed command running time, g++-base.conf will contain 'QMAKE_COMPILER=gcc' or 'QMAKE_COMPILER=ccache gcc'...

Not sure if this matters for some use cases...

Regards,
Peter

Quoted 4 lines

Thanks!

Thomas

author
#6 lore raw

PATCH 1/1 qt5: use ccache for building host code

Hello Peter,

Quoted 3 lines · Peter Seiderer

Thanks for the answer (you got it right)...., so in case I use the buildroot-generated qmake I will always use ccache (if configured), even if used outside of the buildroot build system, right?

Well, in the context of Buildroot, qmake is essentially used to build code for the target, using the cross-compiler. In this case, ccache was already used, because invoking ccache is hidden inside the toolchain wrapper, so there's no way around it :-)

However, I guess with the right qmake invocations, you can make it build code for the host as well. And then yes, in this case, I assume it would now start using ccache.

Quoted 5 lines

The $(HOSTCC) is expanded at the sed command running time, g++-base.conf will contain 'QMAKE_COMPILER=gcc' or 'QMAKE_COMPILER=ccache gcc'...

Not sure if this matters for some use cases...

What are you worried about? The fact that there is a space in the QMAKE_COMPILER variable value ?

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#7 lore raw

PATCH 1/1 qt5: use ccache for building host code

Hello Thomas,

Quoted 8 lines · Thomas Petazzoni

Hello Peter,

Quoted 3 lines · Peter Seiderer

Thanks for the answer (you got it right)...., so in case I use the buildroot-generated qmake I will always use ccache (if configured), even if used outside of the buildroot build system, right?

Quoted 9 lines

Well, in the context of Buildroot, qmake is essentially used to build code for the target, using the cross-compiler. In this case, ccache was already used, because invoking ccache is hidden inside the toolchain wrapper, so there's no way around it :-)

However, I guess with the right qmake invocations, you can make it build code for the host as well. And then yes, in this case, I assume it would now start using ccache.

So host ccache usage will behave the same as target usage..., O.k.

Quoted 4 lines
Quoted 4 lines

The $(HOSTCC) is expanded at the sed command running time, g++-base.conf will contain 'QMAKE_COMPILER=gcc' or 'QMAKE_COMPILER=ccache gcc'...

Ups, missed the context here, the above was only to be sure it is expanded at configure time...

Quoted 1 line
Quoted 1 line

Not sure if this matters for some use cases...

And this should be at the context host ccache usage...

Quoted 3 lines

What are you worried about? The fact that there is a space in the QMAKE_COMPILER variable value ?

No worries, you can add my:

Reviewed-by Peter Seiderer <[email protected]>

Regards,
Peter

Quoted 4 lines

Best regards,

Thomas

author Applied
#8 lore raw

PATCH 1/1 qt5: use ccache for building host code

In reply to Thomas Petazzoni

Hello,

Quoted 14 lines · Thomas Petazzoni

qt5 currently doesn't use HOSTCC/HOSTCXX, so it doesn't use ccache when building all its host code (especially qmake). This means that even with ccache enabled and a hot cache, it still takes a long time to build qt5base.

Before this patch, building qt5base takes:

- 446 seconds with a cold ccache
- 185 seconds with a hot ccache

This is because the ccache is not used for host code.

After this patch, building qt5base takes:

Quoted 7 lines

- 450 seconds with a cold ccache
- 15 seconds with a hot ccache

Signed-off-by Thomas Petazzoni <[email protected]>
---
 package/qt5/qt5base/qt5base.mk | 8 ++++++++
 1 file changed, 8 insertions(+)

Applied to next.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com