Skip to contentSkip to conversation

Baseline test plan and bootrr

DISCUSSION Guillaume Tucker 2 participants 3 messages Feb 12, 2019 – May 2, 2019 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 15 lines · Guillaume Tucker via Groups.Io

Hi,

Currently, boot tests entirely rely on the LAVA logic to detect when a login prompt has been reached. While this mostly works, it does not guarantee that the platform is actually usable or that everything went smoothly during the boot.

For this reason, it would seem useful to introduce a "baseline" test plan which would essentially boot and then do some fast checks to verify that nothing is obviously broken. This can include things like grepping the kernel log for any errors and checking that drivers have been initialised correctly. There's a lot that can be done in a few seconds with a basic ramdisk.

Quoted 8 lines

So we could have a list of regular expressions to detect any issues in the kernel log and report a LAVA test case result for each of them. The log fragment associated with each match should also be available to include the actual errors in a report. Doing this on the device means we keep the test definition in the same location as the other test plans, and we can run it like any test suite.

Any thoughts on this part? Does anyone see any issue with having a series of patterns to grep the kernel log and have a test case result for each of them in LAVA? (i.e. pass if pattern was not matched...)

Guillaume

#2 lore raw
Quoted 27 lines · Guillaume Tucker
Quoted 15 lines · Guillaume Tucker via Groups.Io

Hi,

Currently, boot tests entirely rely on the LAVA logic to detect when a login prompt has been reached. While this mostly works, it does not guarantee that the platform is actually usable or that everything went smoothly during the boot.

For this reason, it would seem useful to introduce a "baseline" test plan which would essentially boot and then do some fast checks to verify that nothing is obviously broken. This can include things like grepping the kernel log for any errors and checking that drivers have been initialised correctly. There's a lot that can be done in a few seconds with a basic ramdisk.

Quoted 8 lines

So we could have a list of regular expressions to detect any issues in the kernel log and report a LAVA test case result for each of them. The log fragment associated with each match should also be available to include the actual errors in a report. Doing this on the device means we keep the test definition in the same location as the other test plans, and we can run it like any test suite.

Quoted 4 lines

Any thoughts on this part? Does anyone see any issue with having a series of patterns to grep the kernel log and have a test case result for each of them in LAVA? (i.e. pass if pattern was not matched...)

I think this is a good idea.

The difficulty comes in looking for device/board specific stuff, but that shouldn't stop us from doing something generic that looks for the obvious stuff.

It could also look for (and count) kernel errors, warnings, backtraces, etc. etc.

Kevin

#3 lore raw
Quoted 34 lines · Kevin Hilman
Quoted 27 lines · Guillaume Tucker
Quoted 15 lines · Guillaume Tucker via Groups.Io

Hi,

Currently, boot tests entirely rely on the LAVA logic to detect when a login prompt has been reached. While this mostly works, it does not guarantee that the platform is actually usable or that everything went smoothly during the boot.

For this reason, it would seem useful to introduce a "baseline" test plan which would essentially boot and then do some fast checks to verify that nothing is obviously broken. This can include things like grepping the kernel log for any errors and checking that drivers have been initialised correctly. There's a lot that can be done in a few seconds with a basic ramdisk.

Quoted 8 lines

So we could have a list of regular expressions to detect any issues in the kernel log and report a LAVA test case result for each of them. The log fragment associated with each match should also be available to include the actual errors in a report. Doing this on the device means we keep the test definition in the same location as the other test plans, and we can run it like any test suite.

Quoted 4 lines

Any thoughts on this part? Does anyone see any issue with having a series of patterns to grep the kernel log and have a test case result for each of them in LAVA? (i.e. pass if pattern was not matched...)

Quoted 9 lines

I think this is a good idea.

The difficulty comes in looking for device/board specific stuff, but that shouldn't stop us from doing something generic that looks for the obvious stuff.

It could also look for (and count) kernel errors, warnings, backtraces, etc. etc.

Thanks for the feedback so far. I've now created an initial PR to start testing this on staging:

https://github.com/kernelci/kernelci-core/pull/55

I believe the bootrr script should be part of the rootfs before we can merge this and roll it out in production.

Guillaume