Fix public/private kernel-doc issues
Subsystems (2)
cover Fix public/private kernel-doc issues
Hi Jon,
This patch set fixes public/private handling inside comments.
The current logic has currently an issue: it breaks nested structs if private: is not followed by public:.
It also solves a currently undefined behavior: now, public/private are applied only to the current and inner statements. So, if one does:
struct {
struct {
/* private: */
int a;
struct {
int b;
};
};
int c;};
The output will be:
struct {
int c;};
Besides making sensing itself, the new behavior solves an issue when struct_group() macros are used, as the private from one struct_group won't propagate to other ones nor to the struct encapsulating one or more struct_group().
This series start with 2 unittests to detect the problem. If you apply those two patches and run the unit test, you'll be able to see the issue.
After applying the changes, it renames the unit test, making it test directly the code there, instead of the current wrapper at kdoc_re, and add more tests meant to check the tokenizer itself.
-
As commented previously, I'm now working on a patchset using the tokenizer for NestedMatch. I have already some code here, but not properly tested yet.
Anyway, the idea is to keep submitting it in small chunks, to make it easier to review.
Mauro Carvalho Chehab (8):
docs: python: add helpers to run unit tests
unittests: add a testbench to check public/private kdoc comments
docs: kdoc: don't add broken comments inside prototypes
docs: kdoc: properly handle empty enum arguments
docs: kdoc_re: add a C tokenizer
docs: kdoc: use tokenizer to handle comments on structs
unittests: test_private: modify it to use CTokenizer directly
unittests: test_tokenizer: check if the tokenizer works
Documentation/tools/python.rst | 2 + Documentation/tools/unittest.rst | 24 ++ tools/lib/python/kdoc/kdoc_parser.py | 25 +- tools/lib/python/kdoc/kdoc_re.py | 234 ++++++++++++++ tools/lib/python/unittest_helper.py | 353 ++++++++++++++++++++ tools/unittests/test_tokenizer.py | 462 +++++++++++++++++++++++++++ 6 files changed, 1088 insertions(+), 12 deletions(-) create mode 100644 Documentation/tools/unittest.rst create mode 100755 tools/lib/python/unittest_helper.py create mode 100755 tools/unittests/test_tokenizer.py
Signature · 1 line
2.52.0
PATCH 1/8 docs: python: add helpers to run unit tests
While python internal libraries have support for unit tests, its output is not nice. Add a helper module to improve its output.
I wrote this module last year while testing some scripts I used internally. The initial skeleton was generated with the help of LLM tools, but it was higly modified to ensure that it will work as I would expect.
Signed-off-by Mauro Carvalho Chehab <[email protected]>
--- Documentation/tools/python.rst | 2 + Documentation/tools/unittest.rst | 24 ++ tools/lib/python/unittest_helper.py | 353 ++++++++++++++++++++++++++++ 3 files changed, 379 insertions(+) create mode 100644 Documentation/tools/unittest.rst create mode 100755 tools/lib/python/unittest_helper.py
Documentation/tools/python.rst +2 -0
@@ -11,3 +11,5 @@ Python libraries feat kdoc kabi + + unittest
Documentation/tools/unittest.rst new file +24 -0
@@ -0,0 +1,24 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============== +Python unittest +=============== + +Checking consistency of python modules can be complex. Sometimes, it is +useful to define a set of unit tests to help checking them. + +While the actual test implementation is usecase dependent, Python already +provides a standard way to add unit tests by using ``import unittest``. + +Using such class, requires setting up a test suite. Also, the default format +is a little bit ackward. To improve it and provide a more uniform way to +report errors, some unittest classes and functions are defined. + + +Unittest helper module +====================== + +.. automodule:: lib.python.unittest_helper + :members: + :show-inheritance: + :undoc-members:
tools/lib/python/unittest_helper.py new file +353 -0
353 lines not on this page · raw message
Signature · 1 line
2.52.0
PATCH 2/8 unittests: add a testbench to check public/private kdoc comments
In reply to Mauro Carvalho ChehabAdd unit tests to check if the public/private and comments strip is working properly.
Running it shows that, on several cases, public/private is not doing what it is expected:
test_private:
TestPublicPrivate:
test balanced_inner_private: OK
test balanced_non_greddy_private: OK
test balanced_private: OK
test no private: OK
test unbalanced_inner_private: FAIL
test unbalanced_private: FAIL
test unbalanced_struct_group_tagged_with_private: FAIL
test unbalanced_two_struct_group_tagged_first_with_private: FAIL
test unbalanced_without_end_of_line: FAIL
Ran 9 tests
FAILED (failures=5)
Signed-off-by Mauro Carvalho Chehab <[email protected]>
--- tools/unittests/test_private.py | 331 ++++++++++++++++++++++++++++++++ 1 file changed, 331 insertions(+) create mode 100755 tools/unittests/test_private.py
tools/unittests/test_private.py new file +331 -0
@@ -0,0 +1,331 @@ +#!/usr/bin/env python3 + +""" +Unit tests for struct/union member extractor class. +""" + + +import os +import re +import unittest +import sys + +from unittest.mock import MagicMock + +SRC_DIR = os.path.dirname(os.path.realpath(__file__)) +sys.path.insert(0, os.path.join(SRC_DIR, "../lib/python")) + +from kdoc.kdoc_parser import trim_private_members +from unittest_helper import run_unittest + +# +# List of tests. +# +# The code will dynamically generate one test for each key on this dictionary. +# + +#: Tests to check if CTokenizer is handling properly public/private comments. +TESTS_PRIVATE = { + # + # Simplest case: no private. Ensure that trimming won't affect struct + # + "no private": { + "source": """ + struct foo { + int a; + int b; + int c; + }; + """, + "trimmed": """ + struct foo { + int a; + int b; + int c; + }; + """, + }, + + # + # Play "by the books" by always having a public in place + # + + "balanced_private": { + "source": """ + struct foo { + int a; + /* private: */ + int b; + /* public: */ + int c; + }; + """, + "trimmed": """ + struct foo { + int a; + int c; + }; + """, + }, + + "balanced_non_greddy_private": { + "source": """ + struct foo { + int a; + /* private: */ + int b; + /* public: */ + int c; + /* private: */ + int d; + /* public: */ + int e; + + }; + """, + "trimmed": """ + struct foo { + int a; + int c; + int e; + }; + """, + }, + + "balanced_inner_private": { + "source": """ + struct foo { + struct { + int a; + /* private: ignore below */ + int b; + /* public: but this should not be ignored */ + }; + int b; + }; + """, + "trimmed": """ + struct foo { + struct { + int a; + }; + int b; + }; + """, + }, + + # + # Test what happens if there's no public after private place + # + + "unbalanced_private": { + "source": """ + struct foo { + int a; + /* private: */ + int b; + int c; + }; + """, + "trimmed": """ + struct foo { + int a; + }; + """, + }, + + "unbalanced_inner_private": { + "source": """ + struct foo { + struct { + int a; + /* private: ignore below */ + int b; + /* but this should not be ignored */ + }; + int b; + }; + """, + "trimmed": """ + struct foo { + struct { + int a; + }; + int b; + }; + """, + }, + + "unbalanced_struct_group_tagged_with_private": { + "source": """ + struct page_pool_params { + struct_group_tagged(page_pool_params_fast, fast, + unsigned int order; + unsigned int pool_size; + int nid; + struct device *dev; + struct napi_struct *napi; + enum dma_data_direction dma_dir; + unsigned int max_len; + unsigned int offset; + }; + struct_group_tagged(page_pool_params_slow, slow, + struct net_device *netdev; + unsigned int queue_idx; + unsigned int flags; + /* private: used by test code only */ + void (*init_callback)(netmem_ref netmem, void *arg); + void *init_arg; + }; + }; + """, + "trimmed": """ + struct page_pool_params { + struct_group_tagged(page_pool_params_fast, fast, + unsigned int order; + unsigned int pool_size; + int nid; + struct device *dev; + struct napi_struct *napi; + enum dma_data_direction dma_dir; + unsigned int max_len; + unsigned int offset; + }; + struct_group_tagged(page_pool_params_slow, slow, + struct net_device *netdev; + unsigned int queue_idx; + unsigned int flags; + }; + }; + """, + }, + + "unbalanced_two_struct_group_tagged_first_with_private": { + "source": """ + struct page_pool_params { + struct_group_tagged(page_pool_params_slow, slow, + struct net_device *netdev; + unsigned int queue_idx; + unsigned int flags; + /* private: used by test code only */ + void (*init_callback)(netmem_ref netmem, void *arg); + void *init_arg; + }; + struct_group_tagged(page_pool_params_fast, fast, + unsigned int order; + unsigned int pool_size; + int nid; + struct device *dev; + struct napi_struct *napi; + enum dma_data_direction dma_dir; + unsigned int max_len; + unsigned int offset; + }; + }; + """, + "trimmed": """ + struct page_pool_params { + struct_group_tagged(page_pool_params_slow, slow, + struct net_device *netdev; + unsigned int queue_idx; + unsigned int flags; + }; + struct_group_tagged(page_pool_params_fast, fast, + unsigned int order; + unsigned int pool_size; + int nid; + struct device *dev; + struct napi_struct *napi; + enum dma_data_direction dma_dir; + unsigned int max_len; + unsigned int offset; + }; + }; + """, + }, + "unbalanced_without_end_of_line": { + "source": """ \ + struct page_pool_params { \ + struct_group_tagged(page_pool_params_slow, slow, \ + struct net_device *netdev; \ + unsigned int queue_idx; \ + unsigned int flags; + /* private: used by test code only */ + void (*init_callback)(netmem_ref netmem, void *arg); \ + void *init_arg; \ + }; \ + struct_group_tagged(page_pool_params_fast, fast, \ + unsigned int order; \ + unsigned int pool_size; \ + int nid; \ + struct device *dev; \ + struct napi_struct *napi; \ + enum dma_data_direction dma_dir; \ + unsigned int max_len; \ + unsigned int offset; \ + }; \ + }; + """, + "trimmed": """ + struct page_pool_params { + struct_group_tagged(page_pool_params_slow, slow, + struct net_device *netdev; + unsigned int queue_idx; + unsigned int flags; + }; + struct_group_tagged(page_pool_params_fast, fast, + unsigned int order; + unsigned int pool_size; + int nid; + struct device *dev; + struct napi_struct *napi; + enum dma_data_direction dma_dir; + unsigned int max_len; + unsigned int offset; + }; + }; + """, + }, +} + + +class TestPublicPrivate(unittest.TestCase): + """ + Main test class. Populated dynamically at runtime. + """ + + def setUp(self): + self.maxDiff = None + + def add_test(cls, name, source, trimmed): + """ + Dynamically add a test to the class + """ + def test(cls): + result = trim_private_members(source) + + result = re.sub(r"\s++", " ", result).strip() + expected = re.sub(r"\s++", " ", trimmed).strip() + + msg = f"failed when parsing this source:\n" + source + + cls.assertEqual(result, expected, msg=msg) + + test.__name__ = f'test {name}' + + setattr(TestPublicPrivate, test.__name__, test) + + +# +# Populate TestPublicPrivate class +# +test_class = TestPublicPrivate() +for name, test in TESTS_PRIVATE.items(): + test_class.add_test(name, test["source"], test["trimmed"]) + + +# +# main +# +if __name__ == "__main__": + run_unittest(__file__)
Signature · 1 line
2.52.0
PATCH 3/8 docs: kdoc: don't add broken comments inside prototypes
In reply to Mauro Carvalho ChehabParsing a file like drivers/scsi/isci/host.h, which contains broken kernel-doc markups makes it create a prototype that contains unmatched end comments.
That causes, for instance, struct sci_power_control to be shown this this prototype:
struct sci_power_control {
* it is not. */ bool timer_started;
*/ struct sci_timer timer;
* requesters field. */ u8 phys_waiting;
*/ u8 phys_granted_power;
* mapped into requesters via struct sci_phy.phy_index */ struct isci_phy *requesters[SCI_MAX_PHYS];
};as comments won't start with "/*" anymore.
Fix the logic to detect such cases, and keep adding the comments inside it.
Signed-off-by Mauro Carvalho Chehab <[email protected]>
--- tools/lib/python/kdoc/kdoc_parser.py | 6 ++++++ 1 file changed, 6 insertions(+)
tools/lib/python/kdoc/kdoc_parser.py +6 -0
@@ -1355,6 +1355,12 @@ class KernelDoc: elif doc_content.search(line): self.emit_msg(ln, f"Incorrect use of kernel-doc format: {line}") self.state = state.PROTO + + # + # Don't let it add partial comments at the code, as breaks the + # logic meant to remove comments from prototypes. + # + self.process_proto_type(ln, "/**\n" + line) # else ... ?? def process_inline_text(self, ln, line):
Signature · 1 line
2.52.0
PATCH 4/8 docs: kdoc: properly handle empty enum arguments
In reply to Mauro Carvalho ChehabDepending on how the enum proto is written, a comma at the end may incorrectly make kernel-doc parse an arg like " ".
Strip spaces before checking if arg is empty.
Signed-off-by Mauro Carvalho Chehab <[email protected]>
--- tools/lib/python/kdoc/kdoc_parser.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
tools/lib/python/kdoc/kdoc_parser.py +3 -2
@@ -810,9 +810,10 @@ class KernelDoc: member_set = set() members = KernRe(r'\([^;)]*\)').sub('', members) for arg in members.split(','): - if not arg: - continue arg = KernRe(r'^\s*(\w+).*').sub(r'\1', arg) + if not arg.strip(): + continue + self.entry.parameterlist.append(arg) if arg not in self.entry.parameterdescs: self.entry.parameterdescs[arg] = self.undescribed
Signature · 1 line
2.52.0
PATCH 5/8 docs: kdoc_re: add a C tokenizer
In reply to Mauro Carvalho ChehabHandling C code purely using regular expressions doesn't work well.
Add a C tokenizer to help doing it the right way.
The tokenizer was written using as basis the Python re documentation tokenizer example from:
https://docs.python.org/3/library/re.html#writing-a-tokenizer
Signed-off-by Mauro Carvalho Chehab <[email protected]>
--- tools/lib/python/kdoc/kdoc_re.py | 234 +++++++++++++++++++++++++++++++ 1 file changed, 234 insertions(+)
tools/lib/python/kdoc/kdoc_re.py +234 -0
@@ -141,6 +141,240 @@ class KernRe: return self.last_match.groups() +class TokType(): + + @staticmethod + def __str__(val): + """Return the name of an enum value""" + return TokType._name_by_val.get(val, f"UNKNOWN({val})") + +class CToken(): + """ + Data class to define a C token. + """ + + # Tokens that can be used by the parser. Works like an C enum. + + COMMENT = 0 #: A standard C or C99 comment, including delimiter. + STRING = 1 #: A string, including quotation marks. + CHAR = 2 #: A character, including apostophes. + NUMBER = 3 #: A number. + PUNC = 4 #: A puntuation mark: ``;`` / ``,`` / ``.``. + BEGIN = 5 #: A begin character: ``{`` / ``[`` / ``(``. + END = 6 #: A end character: ``}`` / ``]`` / ``)``. + CPP = 7 #: A preprocessor macro. + HASH = 8 #: The hash character - useful to handle other macros. + OP = 9 #: A C operator (add, subtract, ...). + STRUCT = 10 #: A ``struct`` keyword. + UNION = 11 #: An ``union`` keyword. + ENUM = 12 #: A ``struct`` keyword. + TYPEDEF = 13 #: A ``typedef`` keyword. + NAME = 14 #: A name. Can be an ID or a type. + SPACE = 15 #: Any space characters, including new lines + + MISMATCH = 255 #: an error indicator: should never happen in practice. + + # Dict to convert from an enum interger into a string. + _name_by_val = {v: k for k, v in dict(vars()).items() if isinstance(v, int)} + + # Dict to convert from string to an enum-like integer value. + _name_to_val = {k: v for v, k in _name_by_val.items()} + + @staticmethod + def to_name(val): + """Convert from an integer value from CToken enum into a string""" + + return CToken._name_by_val.get(val, f"UNKNOWN({val})") + + @staticmethod + def from_name(name): + """Convert a string into a CToken enum value""" + if name in CToken._name_to_val: + return CToken._name_to_val[name] + + return CToken.MISMATCH + + def __init__(self, kind, value, pos, + brace_level, paren_level, bracket_level): + self.kind = kind + self.value = value + self.pos = pos + self.brace_level = brace_level + self.paren_level = paren_level + self.bracket_level = bracket_level + + def __repr__(self): + name = self.to_name(self.kind) + if isinstance(self.value, str): + value = '"' + self.value + '"' + else: + value = self.value + + return f"CToken({name}, {value}, {self.pos}, " \ + f"{self.brace_level}, {self.paren_level}, {self.bracket_level})" + +#: Tokens to parse C code. +TOKEN_LIST = [ + (CToken.COMMENT, r"//[^\n]*|/\*[\s\S]*?\*/"), + + (CToken.STRING, r'"(?:\\.|[^"\\])*"'), + (CToken.CHAR, r"'(?:\\.|[^'\\])'"), + + (CToken.NUMBER, r"0[xX][0-9a-fA-F]+[uUlL]*|0[0-7]+[uUlL]*|" + r"[0-9]+(\.[0-9]*)?([eE][+-]?[0-9]+)?[fFlL]*"), + + (CToken.PUNC, r"[;,\.]"), + + (CToken.BEGIN, r"[\[\(\{]"), + + (CToken.END, r"[\]\)\}]"), + + (CToken.CPP, r"#\s*(define|include|ifdef|ifndef|if|else|elif|endif|undef|pragma)\b"), + + (CToken.HASH, r"#"), + + (CToken.OP, r"\+\+|\-\-|\->|==|\!=|<=|>=|&&|\|\||<<|>>|\+=|\-=|\*=|/=|%=" + r"|&=|\|=|\^=|=|\+|\-|\*|/|%|<|>|&|\||\^|~|!|\?|\:"), + + (CToken.STRUCT, r"\bstruct\b"), + (CToken.UNION, r"\bunion\b"), + (CToken.ENUM, r"\benum\b"), + (CToken.TYPEDEF, r"\bkinddef\b"), + + (CToken.NAME, r"[A-Za-z_][A-Za-z0-9_]*"), + + (CToken.SPACE, r"[\s]+"), + + (CToken.MISMATCH,r"."), +] + +#: Handle C continuation lines. +RE_CONT = KernRe(r"\\\n") + +RE_COMMENT_START = KernRe(r'/\*\s*') + +#: tokenizer regex. Will be filled at the first CTokenizer usage. +re_scanner = None + +class CTokenizer(): + """ + Scan C statements and definitions and produce tokens. + + When converted to string, it drops comments and handle public/private + values, respecting depth. + """ + + # This class is inspired and follows the basic concepts of: + # https://docs.python.org/3/library/re.html#writing-a-tokenizer + + def _tokenize(self, source): + """ + Interactor that parses ``source``, splitting it into tokens, as defined + at ``self.TOKEN_LIST``. + + The interactor returns a CToken class object. + """ + + # Handle continuation lines. Note that kdoc_parser already has a + # logic to do that. Still, let's keep it for completeness, as we might + # end re-using this tokenizer outsize kernel-doc some day - or we may + # eventually remove from there as a future cleanup. + source = RE_CONT.sub("", source) + + brace_level = 0 + paren_level = 0 + bracket_level = 0 + + for match in re_scanner.finditer(source): + kind = CToken.from_name(match.lastgroup) + pos = match.start() + value = match.group() + + if kind == CToken.MISMATCH: + raise RuntimeError(f"Unexpected token '{value}' on {pos}:\n\t{source}") + elif kind == CToken.BEGIN: + if value == '(': + paren_level += 1 + elif value == '[': + bracket_level += 1 + else: # value == '{' + brace_level += 1 + + elif kind == CToken.END: + if value == ')' and paren_level > 0: + paren_level -= 1 + elif value == ']' and bracket_level > 0: + bracket_level -= 1 + elif brace_level > 0: # value == '}' + brace_level -= 1 + + yield CToken(kind, value, pos, + brace_level, paren_level, bracket_level) + + def __init__(self, source): + """ + Create a regular expression to handle TOKEN_LIST. + + While I generally don't like using regex group naming via: + (?P<name>...) + + in this particular case, it makes sense, as we can pick the name + when matching a code via re_scanner(). + """ + global re_scanner + + if not re_scanner: + re_tokens = [] + + for kind, pattern in TOKEN_LIST: + name = CToken.to_name(kind) + re_tokens.append(f"(?P<{name}>{pattern})") + + re_scanner = KernRe("|".join(re_tokens), re.MULTILINE | re.DOTALL) + + self.tokens = [] + for tok in self._tokenize(source): + self.tokens.append(tok) + + def __str__(self): + out="" + show_stack = [True] + + for tok in self.tokens: + if tok.kind == CToken.BEGIN: + show_stack.append(show_stack[-1]) + + elif tok.kind == CToken.END: + prev = show_stack[-1] + if len(show_stack) > 1: + show_stack.pop() + + if not prev and show_stack[-1]: + # + # Try to preserve indent + # + out += "\t" * (len(show_stack) - 1) + + out += str(tok.value) + continue + + elif tok.kind == CToken.COMMENT: + comment = RE_COMMENT_START.sub("", tok.value) + + if comment.startswith("private:"): + show_stack[-1] = False + show = False + elif comment.startswith("public:"): + show_stack[-1] = True + + continue + + if show_stack[-1]: + out += str(tok.value) + + return out + + #: Nested delimited pairs (brackets and parenthesis) DELIMITER_PAIRS = { '{': '}',
Signature · 1 line
2.52.0
PATCH 6/8 docs: kdoc: use tokenizer to handle comments on structs
In reply to Mauro Carvalho ChehabBetter handle comments inside structs. After those changes, all unittests now pass:
test_private:
TestPublicPrivate:
test balanced_inner_private: OK
test balanced_non_greddy_private: OK
test balanced_private: OK
test no private: OK
test unbalanced_inner_private: OK
test unbalanced_private: OK
test unbalanced_struct_group_tagged_with_private: OK
test unbalanced_two_struct_group_tagged_first_with_private: OK
test unbalanced_without_end_of_line: OK
Ran 9 tests
Signed-off-by Mauro Carvalho Chehab <[email protected]>
--- tools/lib/python/kdoc/kdoc_parser.py | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-)
tools/lib/python/kdoc/kdoc_parser.py +4 -10
@@ -13,7 +13,7 @@ import sys import re from pprint import pformat -from kdoc.kdoc_re import NestedMatch, KernRe +from kdoc.kdoc_re import NestedMatch, KernRe, CTokenizer from kdoc.kdoc_item import KdocItem #
@@ -84,15 +84,9 @@ def trim_private_members(text): """ Remove ``struct``/``enum`` members that have been marked "private". """ - # First look for a "public:" block that ends a private region, then - # handle the "private until the end" case. - # - text = KernRe(r'/\*\s*private:.*?/\*\s*public:.*?\*/', flags=re.S).sub('', text) - text = KernRe(r'/\*\s*private:.*', flags=re.S).sub('', text) - # - # We needed the comments to do the above, but now we can take them out. - # - return KernRe(r'\s*/\*.*?\*/\s*', flags=re.S).sub('', text).strip() + + tokens = CTokenizer(text) + return str(tokens) class state: """
Signature · 1 line
2.52.0
PATCH 7/8 unittests: test_private: modify it to use CTokenizer directly
In reply to Mauro Carvalho ChehabChange the logic to use the tokenizer directly. This allows adding more unit tests to check the validty of the tokenizer itself.
Signed-off-by Mauro Carvalho Chehab <[email protected]>
---
.../{test_private.py => test_tokenizer.py} | 76 +++++++++++++------
1 file changed, 52 insertions(+), 24 deletions(-)
rename tools/unittests/{test_private.py => test_tokenizer.py} (85%)tools/unittests/test_tokenizer.py renamed from tools/unittests/test_private.py +52 -24
@@ -15,20 +15,44 @@ from unittest.mock import MagicMock SRC_DIR = os.path.dirname(os.path.realpath(__file__)) sys.path.insert(0, os.path.join(SRC_DIR, "../lib/python")) -from kdoc.kdoc_parser import trim_private_members +from kdoc.kdoc_re import CTokenizer from unittest_helper import run_unittest + + # # List of tests. # # The code will dynamically generate one test for each key on this dictionary. # +def make_private_test(name, data): + """ + Create a test named ``name`` using parameters given by ``data`` dict. + """ + + def test(self): + """In-lined lambda-like function to run the test""" + tokens = CTokenizer(data["source"]) + result = str(tokens) + + # + # Avoid whitespace false positives + # + result = re.sub(r"\s++", " ", result).strip() + expected = re.sub(r"\s++", " ", data["trimmed"]).strip() + + msg = f"failed when parsing this source:\n{data['source']}" + self.assertEqual(result, expected, msg=msg) + + return test + #: Tests to check if CTokenizer is handling properly public/private comments. TESTS_PRIVATE = { # # Simplest case: no private. Ensure that trimming won't affect struct # + "__run__": make_private_test, "no private": { "source": """ struct foo {
@@ -288,41 +312,45 @@ TESTS_PRIVATE = { }, } +#: Dict containing all test groups fror CTokenizer +TESTS = { + "TestPublicPrivate": TESTS_PRIVATE, +} -class TestPublicPrivate(unittest.TestCase): - """ - Main test class. Populated dynamically at runtime. - """ +def setUp(self): + self.maxDiff = None - def setUp(self): - self.maxDiff = None +def build_test_class(group_name, table): + """ + Dynamically creates a class instance using type() as a generator + for a new class derivated from unittest.TestCase. - def add_test(cls, name, source, trimmed): - """ - Dynamically add a test to the class - """ - def test(cls): - result = trim_private_members(source) + We're opting to do it inside a function to avoid the risk of + changing the globals() dictionary. + """ - result = re.sub(r"\s++", " ", result).strip() - expected = re.sub(r"\s++", " ", trimmed).strip() + class_dict = { + "setUp": setUp + } - msg = f"failed when parsing this source:\n" + source + run = table["__run__"] - cls.assertEqual(result, expected, msg=msg) + for test_name, data in table.items(): + if test_name == "__run__": + continue - test.__name__ = f'test {name}' + class_dict[f"test_{test_name}"] = run(test_name, data) - setattr(TestPublicPrivate, test.__name__, test) + cls = type(group_name, (unittest.TestCase,), class_dict) + return cls.__name__, cls # -# Populate TestPublicPrivate class +# Create classes and add them to the global dictionary # -test_class = TestPublicPrivate() -for name, test in TESTS_PRIVATE.items(): - test_class.add_test(name, test["source"], test["trimmed"]) - +for group, table in TESTS.items(): + t = build_test_class(group, table) + globals()[t[0]] = t[1] # # main
Signature · 1 line
2.52.0
PATCH 8/8 unittests: test_tokenizer: check if the tokenizer works
In reply to Mauro Carvalho ChehabAdd extra tests to check if the tokenizer is working properly.
Signed-off-by Mauro Carvalho Chehab <[email protected]>
--- tools/lib/python/kdoc/kdoc_re.py | 4 +- tools/unittests/test_tokenizer.py | 109 +++++++++++++++++++++++++++++- 2 files changed, 108 insertions(+), 5 deletions(-)
tools/lib/python/kdoc/kdoc_re.py +2 -2
@@ -194,8 +194,8 @@ class CToken(): return CToken.MISMATCH - def __init__(self, kind, value, pos, - brace_level, paren_level, bracket_level): + def __init__(self, kind, value=None, pos=0, + brace_level=0, paren_level=0, bracket_level=0): self.kind = kind self.value = value self.pos = pos
tools/unittests/test_tokenizer.py +106 -3
128 lines not on this page · raw message
Signature · 1 line
2.52.0