Skip to contentSkip to conversation

Set the CMake BUILD_TESTING option to OFF as the default setting

DISCUSSION Fabio Berton 2 participants 7 messages Apr 23, 2024 – Jun 3, 2024

lore
Share & tools
lore
#0 lore raw

I've noticed that some recipes, like json-c [1], which use the cmake bbclass, are generating files that aren't being used. This is because CMake sets the BUILD_TESTING option to ON by default. According to the CMake documentation [2], when CTest is included, as json-c does here [3], the module automatically creates a |BUILD_TESTING| option. This option determines whether to enable testing support, and it is ON by default.

In the json-c example, the tests used by the do_install_ptest task are always generated. Thus, ${B}/tests/ exists even when ptests are not included in DISTRO_FEATURES.

As the behavior of CTest/CMake to build tests by default is rather surprising, we are wondering whether disabling this option by default is feasible in the OpenEmbedded context. Recipes expecting tests to be built (e.g. because of ptest support) would then turn on the flag explicitly.

For example, adding this to cmake.bbclass:

|OECMAKE_BUILD_TESTING ??= "false" EXTRA_OECMAKE:append = "\\ ${@bb.utils.contains('OECMAKE_BUILD_TESTING', 'false', '-DBUILD_TESTING=OFF', '', d)} \\ " |

and then set the OECMAKE_BUILD_TESTING in the recipe that requires BUILD_TESTING=ON

Do you have any suggestions on which steps we should take to verify that this does not introduce any regressions?

1 - https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/json-c/json-c_0.17.bb?h=master

2 - https://cmake.org/cmake/help/latest/module/CTest.html

3 - https://github.com/json-c/json-c/blob/master/CMakeLists.txt#L19

Regards,

Fabio Berton

#1 lore raw

I'm not sure I like the idea. This is going to break ptests or other usage of CTest in cmake-based recipes that aren't under our control. Building tests is also a test in itself even if you don't run them. I'd say unless this adds significantly to build times we should leave it as it is.

Alex

Quoted 40 lines · Fabio Berton via lists.openembedded.org

I've noticed that some recipes, like json-c [1], which use the cmake bbclass, are generating files that aren't being used. This is because CMake sets the BUILD_TESTING option to ON by default. According to the CMake documentation [2], when CTest is included, as json-c does here [3], the module automatically creates a BUILD_TESTING option. This option determines whether to enable testing support, and it is ON by default.

In the json-c example, the tests used by the do_install_ptest task are always generated. Thus, ${B}/tests/ exists even when ptests are not included in DISTRO_FEATURES.

As the behavior of CTest/CMake to build tests by default is rather surprising, we are wondering whether disabling this option by default is feasible in the OpenEmbedded context. Recipes expecting tests to be built (e.g. because of ptest support) would then turn on the flag explicitly.

For example, adding this to cmake.bbclass:

OECMAKE_BUILD_TESTING ??= "false"

EXTRA_OECMAKE:append = "\\
    ${@bb.utils.contains('OECMAKE_BUILD_TESTING', 'false', '-DBUILD_TESTING=OFF', '', d)} \\

"

and then set the OECMAKE_BUILD_TESTING in the recipe that requires BUILD_TESTING=ON

Do you have any suggestions on which steps we should take to verify that this does not introduce any regressions?

1 - https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/json-c/json-c_0.17.bb?h=master

2 - https://cmake.org/cmake/help/latest/module/CTest.html

3 - https://github.com/json-c/json-c/blob/master/CMakeLists.txt#L19

Regards,

Fabio Berton

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198621): https://lists.openembedded.org/g/openembedded-core/message/198621
Mute This Topic: https://lists.openembedded.org/mt/105688417/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw

Hi Alex,

What should I test to measure build time, all recipes that use cmake from OE-Core and meta-openembedded? Is there a way to build all recipes, or should I find all recipes that use cmake and run bitbake <all-recipes>?

Regards,

Fabio Berton

Quoted 49 lines · Alexander Kanavin

I'm not sure I like the idea. This is going to break ptests or other usage of CTest in cmake-based recipes that aren't under our control. Building tests is also a test in itself even if you don't run them. I'd say unless this adds significantly to build times we should leave it as it is.

Alex

Quoted 39 lines · Fabio Berton via lists.openembedded.org

I've noticed that some recipes, like json-c [1], which use the cmake bbclass, are generating files that aren't being used. This is because CMake sets the BUILD_TESTING option to ON by default. According to the CMake documentation [2], when CTest is included, as json-c does here [3], the module automatically creates a BUILD_TESTING option. This option determines whether to enable testing support, and it is ON by default.

In the json-c example, the tests used by the do_install_ptest task are always generated. Thus, ${B}/tests/ exists even when ptests are not included in DISTRO_FEATURES.

As the behavior of CTest/CMake to build tests by default is rather surprising, we are wondering whether disabling this option by default is feasible in the OpenEmbedded context. Recipes expecting tests to be built (e.g. because of ptest support) would then turn on the flag explicitly.

For example, adding this to cmake.bbclass:

OECMAKE_BUILD_TESTING ??= "false"

EXTRA_OECMAKE:append = "\\
     ${@bb.utils.contains('OECMAKE_BUILD_TESTING', 'false', '-DBUILD_TESTING=OFF', '', d)} \\

"

and then set the OECMAKE_BUILD_TESTING in the recipe that requires BUILD_TESTING=ON

Do you have any suggestions on which steps we should take to verify that this does not introduce any regressions?

1 -https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/json-c/json-c_0.17.bb?h=master

2 -https://cmake.org/cmake/help/latest/module/CTest.html

3 -https://github.com/json-c/json-c/blob/master/CMakeLists.txt#L19

Regards,

Fabio Berton

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198621):https://lists.openembedded.org/g/openembedded-core/message/198621 Mute This Topic:https://lists.openembedded.org/mt/105688417/1686489
Group Owner:[email protected] Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw

You can build everything with 'bitbake world' (and it's going to be long).

Build times are recorded into tmp/buildstats, if that is enabled (see default local.conf to see how if you're not already using that).

Alex

Quoted 61 lines · Fabio Berton

Hi Alex,

What should I test to measure build time, all recipes that use cmake from OE-Core and meta-openembedded? Is there a way to build all recipes, or should I find all recipes that use cmake and run bitbake <all-recipes>?

Regards,

Fabio Berton

On 4/23/2024 2:01 PM, Alexander Kanavin wrote:

I'm not sure I like the idea. This is going to break ptests or other usage of CTest in cmake-based recipes that aren't under our control. Building tests is also a test in itself even if you don't run them. I'd say unless this adds significantly to build times we should leave it as it is.

Alex

On Tue, 23 Apr 2024 at 14:31, Fabio Berton via lists.openembedded.org <[email protected]> wrote:

I've noticed that some recipes, like json-c [1], which use the cmake bbclass, are generating files that aren't being used. This is because CMake sets the BUILD_TESTING option to ON by default. According to the CMake documentation [2], when CTest is included, as json-c does here [3], the module automatically creates a BUILD_TESTING option. This option determines whether to enable testing support, and it is ON by default.

In the json-c example, the tests used by the do_install_ptest task are always generated. Thus, ${B}/tests/ exists even when ptests are not included in DISTRO_FEATURES.

As the behavior of CTest/CMake to build tests by default is rather surprising, we are wondering whether disabling this option by default is feasible in the OpenEmbedded context. Recipes expecting tests to be built (e.g. because of ptest support) would then turn on the flag explicitly.

For example, adding this to cmake.bbclass:

OECMAKE_BUILD_TESTING ??= "false"

EXTRA_OECMAKE:append = "\\
    ${@bb.utils.contains('OECMAKE_BUILD_TESTING', 'false', '-DBUILD_TESTING=OFF', '', d)} \\

"

and then set the OECMAKE_BUILD_TESTING in the recipe that requires BUILD_TESTING=ON

Do you have any suggestions on which steps we should take to verify that this does not introduce any regressions?

1 - https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/json-c/json-c_0.17.bb?h=master

2 - https://cmake.org/cmake/help/latest/module/CTest.html

3 - https://github.com/json-c/json-c/blob/master/CMakeLists.txt#L19

Regards,

Fabio Berton

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198621): https://lists.openembedded.org/g/openembedded-core/message/198621
Mute This Topic: https://lists.openembedded.org/mt/105688417/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#4 lore raw

Hi Alex,

I ran a few builds disabling CMake DBUILD_TESTING and the build time was about the same. I'm not sure if by building plenty of recipes we can see a difference in build time, since bitbake needs to build all the recipe dependencies that use cmake bbclass.

Looking for some recipes that use ptest, I see examples like this:

- meta/recipes-multimedia/gstreamer/gstreamer1.0_1.24.3.bb: PACKAGECONFIG ??= "${@bb.utils.contains('PTEST_ENABLED', '1', 'tests', '', d)}"
- meta/recipes-core/glib-networking/glib-networking_2.78.1.bb: PACKAGECONFIG ??= "gnutls environment ${@bb.utils.contains('PTEST_ENABLED', '1', 'tests', '', d)}"
- meta/recipes-support/libnl/libnl_3.9.0.bb: do_compile_ptest to compile only when PTEST_ENABLED is enabled
- ... (and several more)
- one interesting one is meta/recipes-devtools/flex/flex_2.6.4.bb: ${@bb.utils.contains('PTEST_ENABLED', '1', '', 'file://disable-tests.patch', d)} they decided to apply a patch to disable tests when ptest is off

Based on this, I would say that we don't want to build tests unless ptests are enabled (so the argument that building tests also tests the code is moot). In turn, it may be sensible to use the default CMake option to disable building of tests.

Thanks,

Fabio

#5 lore raw
Quoted 1 line · Fabio Berton via lists.openembedded.org

Based on this, I would say that we don't want to build tests unless ptests are enabled (so the argument that building tests also tests the code is moot). In turn, it may be sensible to use the default CMake option to disable building of tests.

Sorry, but I do not follow. Building tests does test the code, as it exposes API problems and incompatible versions etc.

Why should we do something else than what cmake does by default?

Alex

#6 lore raw
In reply to Fabio Berton
Quoted 1 line · Fabio Berton via lists.openembedded.org

Based on this, I would say that we don't want to build tests unless ptests are enabled (so the argument that building tests also tests the code is moot). In turn, it may be sensible to use the default CMake option to disable building of tests.

It's actually more subtle. If a recipe has ptest support, then it can use that to enable/disable building of tests. If a recipe has no ptest support, then there's not global rule on what to do. Some recipes disable them, others simply keep things at default settings. I wouldn't want to change a global default for all cmake recipes unless there is compelling reason, because users may be relying on that default, and it will break things for them needlessly.

The correct thing to do, if you observe tests that are being built but are not being run, is to add ptest support to the recipe.

Alex