Skip to contentSkip to conversation

selftests/bpf: Make docs tests fail more reliably

PATCH Superseded Joe Stringer 2 participants 3 messages Apr 10, 2021 – Apr 13, 2021

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 selftests/bpf: Make docs tests fail more reliably

Previously, if rst2man caught errors, then these would be ignored and the output file would be written anyway. This would allow developers to introduce regressions in the docs comments in the BPF headers.

Additionally, even if you instruct rst2man to fail out, it will still write out to the destination target file, so if you ran the tests twice in a row it would always pass. Use a temporary file for the initial run to ensure that if rst2man fails out under "--strict" mode, subsequent runs will not automatically pass.

Tested via ./tools/testing/selftests/bpf/test_doc_build.sh

---
 tools/testing/selftests/bpf/Makefile.docs     | 3 ++-
 tools/testing/selftests/bpf/test_doc_build.sh | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/Makefile.docs +2 -1
index ccf260021e83..a918790c8f9c 100644
@@ -52,7 +52,8 @@ $(OUTPUT)%.$2: $(OUTPUT)%.rst
 ifndef RST2MAN_DEP
 	$$(error "rst2man not found, but required to generate man pages")
 endif
-	$$(QUIET_GEN)rst2man $$< > $$@
+	$$(QUIET_GEN)rst2man --strict $$< > [email protected]
+	$$(QUIET_GEN)mv [email protected] $$@
 
 docs-clean-$1:
 	$$(call QUIET_CLEAN, eBPF_$1-manpage)
tools/testing/selftests/bpf/test_doc_build.sh +1 -0
index 7eb940a7b2eb..ed12111cd2f0 100755
@@ -1,5 +1,6 @@
 #!/bin/bash
 # SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+set -e
 
 # Assume script is located under tools/testing/selftests/bpf/. We want to start
 # build attempts from the top of kernel repository.
Signature · 1 line
2.27.0
#1 lore raw
Quoted 24 lines · Joe Stringer

Previously, if rst2man caught errors, then these would be ignored and the output file would be written anyway. This would allow developers to introduce regressions in the docs comments in the BPF headers.

Additionally, even if you instruct rst2man to fail out, it will still write out to the destination target file, so if you ran the tests twice in a row it would always pass. Use a temporary file for the initial run to ensure that if rst2man fails out under "--strict" mode, subsequent runs will not automatically pass.

Tested via ./tools/testing/selftests/bpf/test_doc_build.sh

---
 tools/testing/selftests/bpf/Makefile.docs     | 3 ++-
 tools/testing/selftests/bpf/test_doc_build.sh | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/Makefile.docs +0 -0
index ccf260021e83..a918790c8f9c 100644

tools/testing/selftests/bpf/Makefile.docs $(OUTPUT)%.$2: $(OUTPUT)%.rst · line 56

Quoted 6 lines
tools/testing/selftests/bpf/Makefile.docs +2 -1
@@ -52,7 +52,8 @@ $(OUTPUT)%.$2: $(OUTPUT)%.rst
 ifndef RST2MAN_DEP
        $$(error "rst2man not found, but required to generate man pages")
 endif
-       $$(QUIET_GEN)rst2man $$< > $$@
+       $$(QUIET_GEN)rst2man --strict $$< > [email protected]
+       $$(QUIET_GEN)mv [email protected] $$@

if something goes wrong this .tmp file will be laying around, so we should at least add it to .gitignore?

Quoted 17 lines
tools/testing/selftests/bpf/Makefile.docs +0 -0
 docs-clean-$1:
        $$(call QUIET_CLEAN, eBPF_$1-manpage)
tools/testing/selftests/bpf/test_doc_build.sh +1 -1
index 7eb940a7b2eb..ed12111cd2f0 100755
@@ -1,5 +1,6 @@
 #!/bin/bash
 # SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+set -e
 
 # Assume script is located under tools/testing/selftests/bpf/. We want to start
 # build attempts from the top of kernel repository.
--

2.27.0

#2 lore raw
Quoted 21 lines · Andrii Nakryiko
Quoted 19 lines · Joe Stringer

Previously, if rst2man caught errors, then these would be ignored and the output file would be written anyway. This would allow developers to introduce regressions in the docs comments in the BPF headers.

Additionally, even if you instruct rst2man to fail out, it will still write out to the destination target file, so if you ran the tests twice in a row it would always pass. Use a temporary file for the initial run to ensure that if rst2man fails out under "--strict" mode, subsequent runs will not automatically pass.

Tested via ./tools/testing/selftests/bpf/test_doc_build.sh

---
 tools/testing/selftests/bpf/Makefile.docs     | 3 ++-
 tools/testing/selftests/bpf/test_doc_build.sh | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)
Quoted 14 lines

tools/testing/selftests/bpf/Makefile.docs $(OUTPUT)%.$2: $(OUTPUT)%.rst · line 56

Quoted 11 lines
tools/testing/selftests/bpf/Makefile.docs +2 -1
index ccf260021e83..a918790c8f9c 100644
@@ -52,7 +52,8 @@ $(OUTPUT)%.$2: $(OUTPUT)%.rst
 ifndef RST2MAN_DEP
        $$(error "rst2man not found, but required to generate man pages")
 endif
-       $$(QUIET_GEN)rst2man $$< > $$@
+       $$(QUIET_GEN)rst2man --strict $$< > [email protected]
+       $$(QUIET_GEN)mv [email protected] $$@

if something goes wrong this .tmp file will be laying around, so we should at least add it to .gitignore?

Right, doesn't look like it's there yet. Will fix it.

I also received out-of-band feedback to use --exit-status rather than --strict so I'll fix that up as well for a v2.

Cheers,
Joe