Add unit tests for kernel-doc parser and output

Hi Jon,

This series comes after my previous one:

    https://lore.kernel.org/linux-doc/[email protected]/

and depends on it. It is focusing on providing a more comprehensive selftests/unittests for kernel-doc.

It also adds a tool to allow executing all tests inside tools/unittests altogether:

    $ ./tools/unittests/run.py
    Ran 35 tests in 0.081s

OK (expected failures=4)
test_kdoc_parser:
CToKdocItem:
test_gen_func2: OK
CToMan:
test_man_func1: OK
KdocItemToMan:
test_man_func2: OK
KdocItemToRest:
test_rst_func2: OK
TestSelfValidate:
test_parse_pass: OK

test_kdoc_test_schema:
TestYAMLSchemaValidation:
test_kdoc_test_yaml_followsschema: OK

test_kdoc_xforms:
TestDifferentReplacements:
test_strip_multiple_acquires: OK
test_sub_count_parameter: OK
test_sub_mixed_placeholders: OK
test_sub_multiple_placeholders: OK
test_sub_no_placeholder: OK
test_sub_single_placeholder: OK
test_sub_with_capture: OK
test_sub_zero_placeholder: OK
TestMultipleMacros:
test_acquires_multiple: OK
test_acquires_nested_paren: OK
test_acquires_simple: OK
test_mixed_macros: OK
test_must_hold: OK
test_must_hold_shared: OK
test_no_false_positive: OK
test_no_function: OK
test_no_macro_remains: OK
TestRealUsecases:
test_functions_with_acquires_and_releases: OK
test_raw_struct_group: OK
test_raw_struct_group_tagged: OK
test_struct_group: OK
test_struct_group_attr: OK
test_struct_group_tagged_with_private: OK
test_struct_kcov: OK
test_vars_stackdepot: OK

Ran 35 tests

most of the efforts here is to generate a test_kdoc_parser which contains 6 test classes. The first one is mostly meant to do a selftest on itself, checking if its checker works. The other 5 covers different test scenarios:

- C to KdocItem;
- C to rst;
- C to man;
- KdocItem to rst;
- KdocItem to man.

Such tests are dynamically generated from a YAML file (at tools/unittests/kdoc-test.yaml).

There is another test to check if such file follows the schema defined for such tests, which is at ./tools/unittests/kdoc-test-schema.yaml.

Yet, as Jani already commented, populating the YAML file can be tricky, specially with regards to the KdocItem internal representation. To make easier to populate it, I added support at kernel-doc to generate a file for us.

The tool uses the already-existing "--man", "--rst" and a new "--kdoc-item" to define what will be stored.

If we want all tree, for instance, for all kernel-doc comments from, let's say, the Xe driver, one can do:

    $ ./scripts/kernel-doc -y xe.yaml drivers/gpu/drm/xe --man --rst --kdoc-item

To execute all such tests altogether is as simple as:

    $ time tools/unittests/test_kdoc_parser.py --yaml xe.yaml
    Ran 3398 tests in 1.071s

OK (expected failures=4)
test_kdoc_parser:
CToKdocItem:
test_gen_test_drivers_gpu_drm_xe_abi_guc_actions_abi_h_000: OK
test_gen_test_drivers_gpu_drm_xe_abi_guc_actions_abi_h_001: OK
test_gen_test_drivers_gpu_drm_xe_abi_guc_actions_slpc_abi_h_000: OK
...

        test_rst_test_drivers_gpu_drm_xe_xe_wa_h_001:                            OK
        test_rst_test_drivers_gpu_drm_xe_xe_wopcm_c_000:                         OK
        test_rst_test_drivers_gpu_drm_xe_xe_wopcm_c_001:                         OK
        test_rst_test_drivers_gpu_drm_xe_xe_wopcm_types_h_000:                   OK
    TestSelfValidate:
        test_parse_pass:                                                         OK


    Ran 3398 tests

    real    0m2.964s
    user    0m2.885s
    sys     0m0.047s

The tool also allows filtering tests by its name, but this is after creating all dynamic tests(*):

    $ time tools/unittests/test_kdoc_parser.py --yaml foo.yaml -k v4l2-mem2mem_c
    Ran 15 tests in 0.007s

    OK
    test_kdoc_parser:
    CToKdocItem:
        test_gen_test_drivers_media_v4l2-core_v4l2-mem2mem_c_000:  OK
        test_gen_test_drivers_media_v4l2-core_v4l2-mem2mem_c_001:  OK
        test_gen_test_drivers_media_v4l2-core_v4l2-mem2mem_c_002:  OK
        test_gen_test_drivers_media_v4l2-core_v4l2-mem2mem_c_003:  OK
        test_gen_test_drivers_media_v4l2-core_v4l2-mem2mem_c_004:  OK
    KdocItemToMan:
        test_man_test_drivers_media_v4l2-core_v4l2-mem2mem_c_000:  OK
        test_man_test_drivers_media_v4l2-core_v4l2-mem2mem_c_001:  OK
        test_man_test_drivers_media_v4l2-core_v4l2-mem2mem_c_002:  OK
        test_man_test_drivers_media_v4l2-core_v4l2-mem2mem_c_003:  OK
        test_man_test_drivers_media_v4l2-core_v4l2-mem2mem_c_004:  OK
    KdocItemToRest:
        test_rst_test_drivers_media_v4l2-core_v4l2-mem2mem_c_000:  OK
        test_rst_test_drivers_media_v4l2-core_v4l2-mem2mem_c_001:  OK
        test_rst_test_drivers_media_v4l2-core_v4l2-mem2mem_c_002:  OK
        test_rst_test_drivers_media_v4l2-core_v4l2-mem2mem_c_003:  OK
        test_rst_test_drivers_media_v4l2-core_v4l2-mem2mem_c_004:  OK


    Ran 15 tests

    real    0m2.000s
    user    0m1.935s
    sys     0m0.056s

(*) so most of the 2 seconds here were spent with unittest
discover and dynamic test creation, and 7 ms to run them :-)

Some notes:

1. currenty, the logic can't generate tests on some cases, as it wants one source code per each KdocItem/man/rst output. The logic with stores source code is a litle bit lazy doing its job, as it add items there even when the parser fails to create a KdocItem due to bad shaped kernel-doc markups.

2. At the worse-case scenario, if one creates a huge YAML file with all tests that pass (1):

    $ ./scripts/kernel-doc -y all.yaml --man --rst --kdoc-item .

And run it:

    $ time tools/unittests/test_kdoc_parser.py --yaml all.yaml

    Ran 229688 tests

    FAILED (failures=1433)

    real    5m3.631s
    user    4m59.525s
    sys     0m2.124s

1433/229688 fails (0.6% of the tests)

I didn't check why, but, even with that, we can probably still use it for regression tests, calling it before/after some critical changes and see what changes. It would take ~5 mins to run all such tests, which doesn't sound too bad, even if one needs to run it twice. to detect what changed.

Mauro Carvalho Chehab (15):
unittests: unittest_helper: store verbosity at the environment
unittests: rename kdoc_xforms.py
unittests: add a runner to execute all unittests
docs: kdoc_files: use a class to group config parameters
docs: kdoc_files: move output symbols logic to kdoc_output
docs: kdoc_item: fix initial value for parameterdesc_start_lines
docs: kdoc_item: add support to generate a KdocItem from a dict
docs: kdoc_item: fix a typo on sections_start_lines
docs: unittests: add a parser to test kernel-doc parser logic
docs: add a schema to help creating unittests for kernel-doc
docs: add a simple kdoc-test.yaml together with a validation tool
docs: test_kdoc_parser: add support for dynamic test creation
docs: add a new file to write kernel-doc output to a YAML file
docs: kernel-doc: add support to store output on a YAML file
unittests: test_kdoc_parser: add command line arg to read a YAML file

 tools/docs/kernel-doc                         |  48 +-
 tools/lib/python/kdoc/kdoc_files.py           | 104 ++--
 tools/lib/python/kdoc/kdoc_item.py            |  39 +-
 tools/lib/python/kdoc/kdoc_output.py          |  23 +-
 tools/lib/python/kdoc/kdoc_parser.py          |  33 +-
 tools/lib/python/kdoc/kdoc_yaml_file.py       | 155 ++++++
 tools/lib/python/unittest_helper.py           |  17 +-
 tools/unittests/kdoc-test-schema.yaml         | 156 ++++++
 tools/unittests/kdoc-test.yaml                | 154 ++++++
 tools/unittests/run.py                        |  17 +
 tools/unittests/test_kdoc_parser.py           | 523 ++++++++++++++++++
 tools/unittests/test_kdoc_test_schema.py      |  94 ++++
 .../{kdoc_xforms.py => test_kdoc_xforms.py}   |   0
 13 files changed, 1308 insertions(+), 55 deletions(-)
 create mode 100644 tools/lib/python/kdoc/kdoc_yaml_file.py
 create mode 100644 tools/unittests/kdoc-test-schema.yaml
 create mode 100644 tools/unittests/kdoc-test.yaml
 create mode 100755 tools/unittests/run.py
 create mode 100755 tools/unittests/test_kdoc_parser.py
 create mode 100755 tools/unittests/test_kdoc_test_schema.py
 rename tools/unittests/{kdoc_xforms.py => test_kdoc_xforms.py} (100%)

2.52.0