Skip to contentSkip to conversation

doc: kselftest: update for clarity on running kselftests in CI rings

PATCH In review Shuah Khan 3 participants 5 messages Aug 27, 2019 – Aug 29, 2019

Subsystems (3)
lore
Share & tools
lore

Shuah Khan

author maintainer
#0 lore raw

PATCH v2 1/1 doc: kselftest: update for clarity on running kselftests in CI rings

Update to add clarity and recommendations on running newer kselftests on older kernels vs. matching the kernel and kselftest revisions.

The recommendation is "Match kernel revision and kselftest."

---
Changes since v1: Fixed "WARNING: Title underline too short."

 Documentation/dev-tools/kselftest.rst | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
Documentation/dev-tools/kselftest.rst +25 -0
index 25604904fa6e..308506c5e8fa 100644
@@ -12,6 +12,31 @@ write new tests using the framework on Kselftest wiki:
 
 https://kselftest.wiki.kernel.org/
 
+Recommendations on running kselftests in Continuous Integration test rings
+==========================================================================
+
+It is recommended that users run Kselftest from the same release. Running
+newer Kselftest on older kernels isn't recommended for the following
+reasons:
+
+- Kselftest from mainline and linux-next might not be stable enough to run
+  on stable kernels.
+- Kselftests detect feature dependencies at run-time and skip tests if a
+  feature and/or configuration they test aren't enabled. Running newer
+  tests on older kernels could result in a few too many skipped/failed
+  conditions. It becomes difficult to evaluate the results.
+- Newer tests provide better coverage. However, users should make a judgement
+  call on coverage vs. run to run consistency and being able to compare
+  run to run results on older kernels.
+
+Recommendations:
+
+Match kernel revision and kselftest. Especially important for LTS and
+Stable kernel Continuous Integration test rings.
+
+Hot-plug tests
+==============
+
 On some systems, hot-plug tests could hang forever waiting for cpu and
 memory to be ready to be offlined. A special hot-plug target is created
 to run the full range of hot-plug tests. In default mode, hot-plug tests run
Signature · 1 line
2.20.1
maintainer
#1 lore raw
Quoted 8 lines · Shuah Khan

Update to add clarity and recommendations on running newer kselftests on older kernels vs. matching the kernel and kselftest revisions.

The recommendation is "Match kernel revision and kselftest."

---
Changes since v1: Fixed "WARNING: Title underline too short."

I have a few more changes and would like to make and send a v3 after the LPC's Testing and Fuzzing kselftest discussion.

Holding off on this patch for now.

thanks,
-- Shuah

#2 lore raw
Quoted 7 lines · shuah
Quoted 5 lines · Shuah Khan

Update to add clarity and recommendations on running newer kselftests on older kernels vs. matching the kernel and kselftest revisions.

The recommendation is "Match kernel revision and kselftest."

Quoted 8 lines
Quoted 3 lines

---
Changes since v1: Fixed "WARNING: Title underline too short."

I have a few more changes and would like to make and send a v3 after the LPC's Testing and Fuzzing kselftest discussion.

Holding off on this patch for now.

Is this just because you are busy, or because you expect what you want to say to change after the discussion?

If it is because you expect what you want to say here to change, I am surprised. This seems like pretty good, straightforward advice. From where I stand this seems like it makes the documentation better without making anything worse, and so this change should probably be included.

In anycase, your call. I just don't think anyone is going to dispute what you are saying here :-)

Cheers!

Reviewed-by on v2 1/1
#3 lore raw
In reply to Shuah Khan
Quoted 7 lines · Shuah Khan

Update to add clarity and recommendations on running newer kselftests on older kernels vs. matching the kernel and kselftest revisions.

The recommendation is "Match kernel revision and kselftest."

One minor comment below.

Quoted 22 lines

---
Changes since v1: Fixed "WARNING: Title underline too short."

 Documentation/dev-tools/kselftest.rst | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
Documentation/dev-tools/kselftest.rst +8 -0
index 25604904fa6e..308506c5e8fa 100644
@@ -12,6 +12,31 @@ write new tests using the framework on Kselftest wiki:
 
 https://kselftest.wiki.kernel.org/
 
+Recommendations on running kselftests in Continuous Integration test rings
+==========================================================================
+
+It is recommended that users run Kselftest from the same release. Running
+newer Kselftest on older kernels isn't recommended for the following
+reasons:
+
+- Kselftest from mainline and linux-next might not be stable enough to run

Documentation/dev-tools/kselftest.rst write new tests using the framework on Kselftest wiki: · line 32

Quoted 10 lines
Documentation/dev-tools/kselftest.rst +10 -0
+  on stable kernels.
+- Kselftests detect feature dependencies at run-time and skip tests if a
+  feature and/or configuration they test aren't enabled. Running newer
+  tests on older kernels could result in a few too many skipped/failed
+  conditions. It becomes difficult to evaluate the results.
+- Newer tests provide better coverage. However, users should make a judgement
+  call on coverage vs. run to run consistency and being able to compare
+  run to run results on older kernels.
+
+Recommendations:

I think this should be the "note" directive. You already have a list of recommendations above; this seems like a TLDR section, or an important note or tip.

Quoted 13 lines
Documentation/dev-tools/kselftest.rst +7 -1
+
+Match kernel revision and kselftest. Especially important for LTS and
+Stable kernel Continuous Integration test rings.
+
+Hot-plug tests
+==============
+
 On some systems, hot-plug tests could hang forever waiting for cpu and
 memory to be ready to be offlined. A special hot-plug target is created
 to run the full range of hot-plug tests. In default mode, hot-plug tests run
--

2.20.1

maintainer
#4 lore raw
Quoted 50 lines · Brendan Higgins
Quoted 7 lines · Shuah Khan

Update to add clarity and recommendations on running newer kselftests on older kernels vs. matching the kernel and kselftest revisions.

The recommendation is "Match kernel revision and kselftest."

One minor comment below.

Quoted 22 lines

---
Changes since v1: Fixed "WARNING: Title underline too short."

  Documentation/dev-tools/kselftest.rst | 25 +++++++++++++++++++++++++
  1 file changed, 25 insertions(+)
Documentation/dev-tools/kselftest.rst +8 -0
index 25604904fa6e..308506c5e8fa 100644
@@ -12,6 +12,31 @@ write new tests using the framework on Kselftest wiki:
 
  https://kselftest.wiki.kernel.org/
 
+Recommendations on running kselftests in Continuous Integration test rings
+==========================================================================
+
+It is recommended that users run Kselftest from the same release. Running
+newer Kselftest on older kernels isn't recommended for the following
+reasons:
+
+- Kselftest from mainline and linux-next might not be stable enough to run

Documentation/dev-tools/kselftest.rst write new tests using the framework on Kselftest wiki: · line 32

Quoted 10 lines
Documentation/dev-tools/kselftest.rst +10 -0
+  on stable kernels.
+- Kselftests detect feature dependencies at run-time and skip tests if a
+  feature and/or configuration they test aren't enabled. Running newer
+  tests on older kernels could result in a few too many skipped/failed
+  conditions. It becomes difficult to evaluate the results.
+- Newer tests provide better coverage. However, users should make a judgement
+  call on coverage vs. run to run consistency and being able to compare
+  run to run results on older kernels.
+
+Recommendations:

I think this should be the "note" directive. You already have a list of recommendations above; this seems like a TLDR section, or an important note or tip.

Quoted 14 lines
Quoted 11 lines
Documentation/dev-tools/kselftest.rst +7 -1
+
+Match kernel revision and kselftest. Especially important for LTS and
+Stable kernel Continuous Integration test rings.
+
+Hot-plug tests
+==============
+
  On some systems, hot-plug tests could hang forever waiting for cpu and
  memory to be ready to be offlined. A special hot-plug target is created
  to run the full range of hot-plug tests. In default mode, hot-plug tests run
--
Quoted 2 lines

2.20.1

Thanks Brendan. I am holding off on this patch for now. I will have to make some updates. I will fold your comment in when I send an updated patch.

thanks,
-- Shuah