kernel-doc: make it parse new functions and structs
cover kernel-doc: make it parse new functions and structs
There are some patches at linux-next improving support and documentation for context analysis. Those adds a new set of macros that are not supported by the pure C Sphinx parser.
Add support by them, extending the usage of NestedMatch to cover more complex macros.
This series is more complex than originally planned, as we found a couple of hidden bugs at kernel-doc.
The series were co-developed with Randy, with came up with the original patch. I ended helping identifying kernel-doc issues and doing some changes to make the parser more reliable.
At least on my machine, by applying this patch and asking kernel-doc to parse the entire Kernel tree reduced the time from 36 seconds to 35 seconds, so it actually made it a little bit faster.
It should now be possible to document all sorts of function macros, as one of the bugs is that we were replacing macros with blank strings, due to the incorrect usage of function_xforms.
Mauro Carvalho Chehab (12):
docs: kdoc_re: add support for groups()
docs: kdoc_re: don't go past the end of a line
docs: kdoc_parser: move var transformers to the beginning
docs: kdoc_parser: don't mangle with function defines
docs: kdoc_parser: add functions support for NestedMatch
docs: kdoc_parser: use NestedMatch to handle __attribute__ on
functions
docs: kdoc_parser: fix variable regexes to work with size_t
docs: kdoc_parser: fix the default_value logic for variables
docs: kdoc_parser: add some debug for variable parsing
docs: kdoc_parser: don't exclude defaults from prototype
docs: kdoc_parser: fix parser to support multi-word types
docs: kdoc_parser: add support for LIST_HEAD
Randy Dunlap (2):
docs: kdoc_parser: ignore context analysis and lock attributes
kdoc_parser: handle struct member macro VIRTIO_DECLARE_FEATURES(name)
tools/lib/python/kdoc/kdoc_parser.py | 103 +++++++++++++++++++-------- tools/lib/python/kdoc/kdoc_re.py | 9 ++- 2 files changed, 81 insertions(+), 31 deletions(-)
Signature · 1 line
2.52.0
PATCH 1/14 docs: kdoc_re: add support for groups()
Add an equivalent to re groups() method.
This is useful on debug messages.
--- tools/lib/python/kdoc/kdoc_re.py | 7 +++++++ 1 file changed, 7 insertions(+)
tools/lib/python/kdoc/kdoc_re.py +7 -0
@@ -103,6 +103,13 @@ class KernRe: return self.last_match.group(num) + def groups(self): + """ + Returns the group results of the last match + """ + + return self.last_match.groups() + class NestedMatch: """
Signature · 1 line
2.52.0
PATCH 2/14 docs: kdoc_re: don't go past the end of a line
In reply to Mauro Carvalho ChehabThe logic which checks if the line ends with ";" is currently
broken: it may try to read past the buffer.
Fix it by checking before trying to access line[pos].
--- tools/lib/python/kdoc/kdoc_re.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/python/kdoc/kdoc_re.py +1 -1
@@ -261,7 +261,7 @@ class NestedMatch: out += new_sub # Drop end ';' if any - if line[pos] == ';': + if pos < len(line) and line[pos] == ';': pos += 1 cur_pos = pos
Signature · 1 line
2.52.0
PATCH 3/14 docs: kdoc_parser: move var transformers to the beginning
In reply to Mauro Carvalho ChehabJust like functions and structs had their transform variables placed at the beginning, move variable transforms to there as well.
No functional changes.
--- tools/lib/python/kdoc/kdoc_parser.py | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-)
tools/lib/python/kdoc/kdoc_parser.py +13 -10
@@ -194,6 +194,18 @@ function_xforms = [ (KernRe(r"__attribute__\s*\(\((?:[\w\s]+(?:\([^)]*\))?\s*,?)+\)\)\s+"), ""), ] +# +# Transforms for variable prototypes +# +var_xforms = [ + (KernRe(r"__read_mostly"), ""), + (KernRe(r"__ro_after_init"), ""), + (KernRe(r"(?://.*)$"), ""), + (KernRe(r"(?:/\*.*\*/)"), ""), + (KernRe(r";$"), ""), + (KernRe(r"=.*"), ""), +] + # # Apply a set of transforms to a block of text. #
@@ -955,15 +967,6 @@ class KernelDoc: ] OPTIONAL_VAR_ATTR = "^(?:" + "|".join(VAR_ATTRIBS) + ")?" - sub_prefixes = [ - (KernRe(r"__read_mostly"), ""), - (KernRe(r"__ro_after_init"), ""), - (KernRe(r"(?://.*)$"), ""), - (KernRe(r"(?:/\*.*\*/)"), ""), - (KernRe(r";$"), ""), - (KernRe(r"=.*"), ""), - ] - # # Store the full prototype before modifying it #
@@ -987,7 +990,7 @@ class KernelDoc: # Drop comments and macros to have a pure C prototype # if not declaration_name: - for r, sub in sub_prefixes: + for r, sub in var_xforms: proto = r.sub(sub, proto) proto = proto.rstrip()
Signature · 1 line
2.52.0
PATCH 4/14 docs: kdoc_parser: don't mangle with function defines
In reply to Mauro Carvalho ChehabMangling with #defines is not nice, as we may end removing the macro names, preventing several macros from being properly documented.
Also, on defines, we have something like:
#define foo(a1, a2, a3, ...) \ /* some real implementation */
The prototype part (first line on this example) won't contain any macros, so no need to apply any regexes on it.
With that, move the apply_transforms() logic to ensure that it will be called only on functions.
--- tools/lib/python/kdoc/kdoc_parser.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
tools/lib/python/kdoc/kdoc_parser.py +7 -5
@@ -166,7 +166,7 @@ struct_nested_prefixes = [ # # Transforms for function prototypes # -function_xforms = [ +function_xforms = [ (KernRe(r"^static +"), ""), (KernRe(r"^extern +"), ""), (KernRe(r"^asmlinkage +"), ""),
@@ -1049,10 +1049,7 @@ class KernelDoc: found = func_macro = False return_type = '' decl_type = 'function' - # - # Apply the initial transformations. - # - prototype = apply_transforms(function_xforms, prototype) + # # If we have a macro, remove the "#define" at the front. #
@@ -1071,6 +1068,11 @@ class KernelDoc: declaration_name = r.group(1) func_macro = True found = True + else: + # + # Apply the initial transformations. + # + prototype = apply_transforms(function_xforms, prototype) # Yes, this truly is vile. We are looking for: # 1. Return type (may be nothing if we're looking at a macro)
Signature · 1 line
2.52.0
PATCH 5/14 docs: kdoc_parser: add functions support for NestedMatch
In reply to Mauro Carvalho ChehabSome annotations macros may have nested parenthesis, causing normal regex parsing to fail.
Extend apply_transforms to also use NestedMatch and add support for nested functions.
--- tools/lib/python/kdoc/kdoc_parser.py | 39 ++++++++++++++++++---------- 1 file changed, 25 insertions(+), 14 deletions(-)
tools/lib/python/kdoc/kdoc_parser.py +25 -14
@@ -155,7 +155,7 @@ struct_xforms = [ (KernRe(r'DEFINE_DMA_UNMAP_LEN\s*\(' + struct_args_pattern + r'\)', re.S), r'__u32 \1'), ] # -# Regexes here are guaranteed to have the end delimiter matching +# Struct regexes here are guaranteed to have the end delimiter matching # the start delimiter. Yet, right now, only one replace group # is allowed. #
@@ -163,6 +163,13 @@ struct_nested_prefixes = [ (re.compile(r'\bSTRUCT_GROUP\('), r'\1'), ] +# +# Function Regexes here are guaranteed to have the end delimiter matching +# the start delimiter. +# +function_nested_prefixes = [ +] + # # Transforms for function prototypes #
@@ -206,14 +213,6 @@ var_xforms = [ (KernRe(r"=.*"), ""), ] -# -# Apply a set of transforms to a block of text. -# -def apply_transforms(xforms, text): - for search, subst in xforms: - text = search.sub(subst, text) - return text - # # A little helper to get rid of excess white space #
@@ -400,6 +399,8 @@ class KernelDoc: # Place all potential outputs into an array self.entries = [] + self.nested = NestedMatch() + # # We need Python 3.7 for its "dicts remember the insertion # order" guarantee
@@ -497,6 +498,16 @@ class KernelDoc: # State flags self.state = state.NORMAL + def apply_transforms(self, regex_xforms, nested_xforms, text): + """Apply a set of transforms to a block of text.""" + for search, subst in regex_xforms: + text = search.sub(subst, text) + + for search, sub in nested_xforms: + text = self.nested.sub(search, sub, text) + + return text.strip() + def push_parameter(self, ln, decl_type, param, dtype, org_arg, declaration_name): """
@@ -865,11 +876,9 @@ class KernelDoc: # Go through the list of members applying all of our transformations. # members = trim_private_members(members) - members = apply_transforms(struct_xforms, members) + members = self.apply_transforms(struct_xforms, struct_nested_prefixes, + members) - nested = NestedMatch() - for search, sub in struct_nested_prefixes: - members = nested.sub(search, sub, members) # # Deal with embedded struct and union members, and drop enums entirely. #
@@ -1072,7 +1081,9 @@ class KernelDoc: # # Apply the initial transformations. # - prototype = apply_transforms(function_xforms, prototype) + prototype = self.apply_transforms(function_xforms, + function_nested_prefixes, + prototype) # Yes, this truly is vile. We are looking for: # 1. Return type (may be nothing if we're looking at a macro)
Signature · 1 line
2.52.0
PATCH 6/14 docs: kdoc_parser: use NestedMatch to handle __attribute__ on functions
In reply to Mauro Carvalho ChehabSome annotations macros may have nested parenthesis, causing normal regex parsing to fail. The __attribute__ regex is currently very complex to try to avoid that, but it doesn't catch all cases.
Ensure that the parenthesis will be properly handled by using the NestedMatch() logic.
--- tools/lib/python/kdoc/kdoc_parser.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/python/kdoc/kdoc_parser.py +1 -1
@@ -168,6 +168,7 @@ struct_nested_prefixes = [ # the start delimiter. # function_nested_prefixes = [ + (re.compile(r"__attribute__\s*\("), ""), ] #
@@ -198,7 +199,6 @@ function_xforms = [ (KernRe(r"__diagnose_as\s*\(\s*\S+\s*(?:,\s*\d+\s*)*\) +"), ""), (KernRe(r"DECL_BUCKET_PARAMS\s*\(\s*(\S+)\s*,\s*(\S+)\s*\)"), r"\1, \2"), (KernRe(r"__attribute_const__ +"), ""), - (KernRe(r"__attribute__\s*\(\((?:[\w\s]+(?:\([^)]*\))?\s*,?)+\)\)\s+"), ""), ] #
Signature · 1 line
2.52.0
PATCH 7/14 docs: kdoc_parser: fix variable regexes to work with size_t
In reply to Mauro Carvalho ChehabThe regular expressions meant to pick variable types are too
naive: they forgot that the type word may contain underlines.
--- tools/lib/python/kdoc/kdoc_parser.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/lib/python/kdoc/kdoc_parser.py +2 -2
@@ -1010,14 +1010,14 @@ class KernelDoc: default_val = None - r= KernRe(OPTIONAL_VAR_ATTR + r"\w.*\s+(?:\*+)?([\w_]+)\s*[\d\]\[]*\s*(=.*)?") + r= KernRe(OPTIONAL_VAR_ATTR + r"[\w_]*\s+(?:\*+)?([\w_]+)\s*[\d\]\[]*\s*(=.*)?") if r.match(proto): if not declaration_name: declaration_name = r.group(1) default_val = r.group(2) else: - r= KernRe(OPTIONAL_VAR_ATTR + r"(?:\w.*)?\s+(?:\*+)?(?:[\w_]+)\s*[\d\]\[]*\s*(=.*)?") + r= KernRe(OPTIONAL_VAR_ATTR + r"(?:[\w_]*)?\s+(?:\*+)?(?:[\w_]+)\s*[\d\]\[]*\s*(=.*)?") if r.match(proto): default_val = r.group(1)
Signature · 1 line
2.52.0
PATCH 8/14 docs: kdoc_parser: fix the default_value logic for variables
In reply to Mauro Carvalho ChehabThe indentation is wrong for the second regex, which causes problems on variables with defaults.
--- tools/lib/python/kdoc/kdoc_parser.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/lib/python/kdoc/kdoc_parser.py +2 -2
@@ -1018,9 +1018,9 @@ class KernelDoc: default_val = r.group(2) else: r= KernRe(OPTIONAL_VAR_ATTR + r"(?:[\w_]*)?\s+(?:\*+)?(?:[\w_]+)\s*[\d\]\[]*\s*(=.*)?") - if r.match(proto): - default_val = r.group(1) + if r.match(proto): + default_val = r.group(1) if not declaration_name: self.emit_msg(ln,f"{proto}: can't parse variable") return
Signature · 1 line
2.52.0
PATCH 9/14 docs: kdoc_parser: add some debug for variable parsing
In reply to Mauro Carvalho ChehabThis is a new parser that we're still fine-tuning. Add some extra debug messages to help addressing issues over there.
--- tools/lib/python/kdoc/kdoc_parser.py | 11 +++++++++++ 1 file changed, 11 insertions(+)
tools/lib/python/kdoc/kdoc_parser.py +11 -0
@@ -1016,11 +1016,19 @@ class KernelDoc: declaration_name = r.group(1) default_val = r.group(2) + + self.config.log.debug("Variable proto parser: %s from '%s'", + r.groups(), proto) + else: r= KernRe(OPTIONAL_VAR_ATTR + r"(?:[\w_]*)?\s+(?:\*+)?(?:[\w_]+)\s*[\d\]\[]*\s*(=.*)?") if r.match(proto): default_val = r.group(1) + + if default_val: + self.config.log.debug("default: '%s'", default_val) + if not declaration_name: self.emit_msg(ln,f"{proto}: can't parse variable") return
@@ -1028,6 +1036,9 @@ class KernelDoc: if default_val: default_val = default_val.lstrip("=").strip() + self.config.log.debug("'%s' variable prototype: '%s', default: %s", + declaration_name, proto, default_val) + self.output_declaration("var", declaration_name, full_proto=full_proto, default_val=default_val,
Signature · 1 line
2.52.0
PATCH 10/14 docs: kdoc_parser: don't exclude defaults from prototype
In reply to Mauro Carvalho ChehabIf we do that, the defaults won't be parsed.
--- tools/lib/python/kdoc/kdoc_parser.py | 1 - 1 file changed, 1 deletion(-)
tools/lib/python/kdoc/kdoc_parser.py +0 -1
@@ -210,7 +210,6 @@ var_xforms = [ (KernRe(r"(?://.*)$"), ""), (KernRe(r"(?:/\*.*\*/)"), ""), (KernRe(r";$"), ""), - (KernRe(r"=.*"), ""), ] #
Signature · 1 line
2.52.0
PATCH 11/14 docs: kdoc_parser: fix parser to support multi-word types
In reply to Mauro Carvalho ChehabThe regular expression currently expects a single word for the type, but it may be something like "struct foo".
Add support for it.
--- tools/lib/python/kdoc/kdoc_parser.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/lib/python/kdoc/kdoc_parser.py +2 -2
@@ -1009,7 +1009,7 @@ class KernelDoc: default_val = None - r= KernRe(OPTIONAL_VAR_ATTR + r"[\w_]*\s+(?:\*+)?([\w_]+)\s*[\d\]\[]*\s*(=.*)?") + r= KernRe(OPTIONAL_VAR_ATTR + r"\s*[\w_\s]*\s+(?:\*+)?([\w_]+)\s*[\d\]\[]*\s*(=.*)?") if r.match(proto): if not declaration_name: declaration_name = r.group(1)
@@ -1020,7 +1020,7 @@ class KernelDoc: r.groups(), proto) else: - r= KernRe(OPTIONAL_VAR_ATTR + r"(?:[\w_]*)?\s+(?:\*+)?(?:[\w_]+)\s*[\d\]\[]*\s*(=.*)?") + r= KernRe(OPTIONAL_VAR_ATTR + r"(?:[\w_\s]*)?\s+(?:\*+)?(?:[\w_]+)\s*[\d\]\[]*\s*(=.*)?") if r.match(proto): default_val = r.group(1)
Signature · 1 line
2.52.0
PATCH 12/14 docs: kdoc_parser: ignore context analysis and lock attributes
In reply to Mauro Carvalho ChehabFrom: Randy Dunlap <[email protected]>
Drop all context analysis and lock (tracking) attributes to avoid kernel-doc warnings.
Documentation/core-api/kref:328: ../include/linux/kref.h:72: WARNING: Invalid C declaration: Expected end of definition. [error at 96]
int kref_put_mutex (struct kref *kref, void (*release)(struct kref *kref), struct mutex *mutex) __cond_acquires(true# mutex) ------------------------------------------------------------------------------------------------^Documentation/core-api/kref:328: ../include/linux/kref.h:94: WARNING: Invalid C declaration: Expected end of definition. [error at 92]
int kref_put_lock (struct kref *kref, void (*release)(struct kref *kref), spinlock_t *lock) __cond_acquires(true# lock) --------------------------------------------------------------------------------------------^
The regex is suggested by Mauro; mine was too greedy. Thanks. Updated context analysis and lock macros list provided by PeterZ. Thanks.
--- tools/lib/python/kdoc/kdoc_parser.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
tools/lib/python/kdoc/kdoc_parser.py +15 -0
@@ -84,6 +84,8 @@ struct_xforms = [ (KernRe(r'\s*__aligned\s*\([^;]*\)', re.S), ' '), (KernRe(r'\s*__counted_by\s*\([^;]*\)', re.S), ' '), (KernRe(r'\s*__counted_by_(le|be)\s*\([^;]*\)', re.S), ' '), + (KernRe(r'\s*__guarded_by\s*\([^\)]*\)', re.S), ' '), + (KernRe(r'\s*__pt_guarded_by\s*\([^\)]*\)', re.S), ' '), (KernRe(r'\s*__packed\s*', re.S), ' '), (KernRe(r'\s*CRYPTO_MINALIGN_ATTR', re.S), ' '), (KernRe(r'\s*__private', re.S), ' '),
@@ -168,6 +170,16 @@ struct_nested_prefixes = [ # the start delimiter. # function_nested_prefixes = [ + (re.compile(r"__cond_acquires\s*\("), ""), + (re.compile(r"__cond_releases\s*\("), ""), + (re.compile(r"__acquires\s*\("), ""), + (re.compile(r"__releases\s*\("), ""), + (re.compile(r"__must_hold\s*\("), ""), + (re.compile(r"__must_not_hold\s*\("), ""), + (re.compile(r"__must_hold_shared\s*\("), ""), + (re.compile(r"__cond_acquires_shared\s*\("), ""), + (re.compile(r"__acquires_shared\s*\("), ""), + (re.compile(r"__releases_shared\s*\("), ""), (re.compile(r"__attribute__\s*\("), ""), ]
@@ -198,6 +210,7 @@ function_xforms = [ (KernRe(r"__(?:re)?alloc_size\s*\(\s*\d+\s*(?:,\s*\d+\s*)?\) +"), ""), (KernRe(r"__diagnose_as\s*\(\s*\S+\s*(?:,\s*\d+\s*)*\) +"), ""), (KernRe(r"DECL_BUCKET_PARAMS\s*\(\s*(\S+)\s*,\s*(\S+)\s*\)"), r"\1, \2"), + (KernRe(r"__no_context_analysis\s*"), ""), (KernRe(r"__attribute_const__ +"), ""), ]
@@ -207,6 +220,8 @@ function_xforms = [ var_xforms = [ (KernRe(r"__read_mostly"), ""), (KernRe(r"__ro_after_init"), ""), + (KernRe(r'\s*__guarded_by\s*\([^\)]*\)', re.S), ""), + (KernRe(r'\s*__pt_guarded_by\s*\([^\)]*\)', re.S), ""), (KernRe(r"(?://.*)$"), ""), (KernRe(r"(?:/\*.*\*/)"), ""), (KernRe(r";$"), ""),
Signature · 1 line
2.52.0
PATCH 13/14 docs: kdoc_parser: add support for LIST_HEAD
In reply to Mauro Carvalho ChehabConvert LIST_HEAD into struct list_head when handling its prototype.
--- tools/lib/python/kdoc/kdoc_parser.py | 1 + 1 file changed, 1 insertion(+)
tools/lib/python/kdoc/kdoc_parser.py +1 -0
@@ -222,6 +222,7 @@ var_xforms = [ (KernRe(r"__ro_after_init"), ""), (KernRe(r'\s*__guarded_by\s*\([^\)]*\)', re.S), ""), (KernRe(r'\s*__pt_guarded_by\s*\([^\)]*\)', re.S), ""), + (KernRe(r"LIST_HEAD\(([\w_]+)\)"), r"struct list_head \1"), (KernRe(r"(?://.*)$"), ""), (KernRe(r"(?:/\*.*\*/)"), ""), (KernRe(r";$"), ""),
Signature · 1 line
2.52.0
PATCH 14/14 kdoc_parser: handle struct member macro VIRTIO_DECLARE_FEATURES(name)
In reply to Mauro Carvalho ChehabFrom: Randy Dunlap <[email protected]>
Parse the macro VIRTIO_DECLARE_FEATURES(name) and expand it to its definition. These prevents one build warning:
WARNING: include/linux/virtio.h:188 struct member 'VIRTIO_DECLARE_FEATURES(features' not described in 'virtio_device'
--- tools/lib/python/kdoc/kdoc_parser.py | 1 + 1 file changed, 1 insertion(+)
tools/lib/python/kdoc/kdoc_parser.py +1 -0
@@ -155,6 +155,7 @@ struct_xforms = [ struct_args_pattern + r'\)', re.S), r'\1 \2[]'), (KernRe(r'DEFINE_DMA_UNMAP_ADDR\s*\(' + struct_args_pattern + r'\)', re.S), r'dma_addr_t \1'), (KernRe(r'DEFINE_DMA_UNMAP_LEN\s*\(' + struct_args_pattern + r'\)', re.S), r'__u32 \1'), + (KernRe(r'VIRTIO_DECLARE_FEATURES\(([\w_]+)\)'), r'union { u64 \1; u64 \1_array[VIRTIO_FEATURES_U64S]; }'), ] # # Struct regexes here are guaranteed to have the end delimiter matching
Signature · 1 line
2.52.0