Skip to contentSkip to conversation

docs: Document list of undocumented commands

PATCH Superseded Hans Ulrich Niedermann 1 participant 5 messages Apr 24, 2020 – May 5, 2020

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover docs: Document list of undocumented commands

This patchset adds a script to the buildsystem which determines the lists of grub commands implemented in the source code and the list of grub commands documented in grub.info and documents the differences.

For that matter, the script will autogenerate a list of the undocumented commands to be included in grub.info, so that the undocumented commands are at least listed there so users have a point to start researching. If the command is at least mentioned somewhere withing grub.texi, that will be noted.

The two commands which were documented in grub.info but not implemented in the source code are removed from grub.texi in separate patches in the patchset.

For conflict free merging, this patchset requires my gitignore patchset as it adds a gitignore pattern for the file it generates.

The main differences to the previous versions ('' and 'v2) of this patchset are:

* Added usage comments to the check-commands.py script

* Use more descriptive variable names in check-commands.py

* If a command is mentioned somewhere in grub.texi, we now add a remark about that to the autogenerated "this command is undocumented" documentation stub.

Hans Ulrich Niedermann (3):
docs: Add grub command documentation checks
docs: Remove docs for non-existing pxe_unload command
docs: Remove docs for non-existing uppermem command

 .gitignore             |   1 +
 docs/Makefile.am       |  16 +++-
 docs/check-commands.py | 205 +++++++++++++++++++++++++++++++++++++++++
 docs/grub.texi         |  28 +-----
 4 files changed, 223 insertions(+), 27 deletions(-)
 create mode 100644 docs/check-commands.py
Signature · 1 line
2.25.3
#1 lore raw

PATCH v3 1/3 docs: Add grub command documentation checks

This adds the docs/check-commands.py script which checks whether grub commands implemented in the git source tree are documented in docs/grub.texi and vice versa.

During a standard "make" command, BUILT_SOURCES makes sure that docs/undocumented-commands.texi will be created (if it does not exist yet) or updated (if it differs from what the script would generate) or be left alone otherwise.

If you run "make grub.info" in the docs/ subdirectory, the BUILT_SOURCES trick to create or update undocumented-commands.texi will not work (this is an Automake limitation). So if you want to avoid the "all" make target, you need to explictly run "make update-undoc" first.

The generated docs/undocumented-commands.texi will document each otherwise undocumented command by describing that it is undocumented, and by adding that it is mentioned somewhere else in the info page if that happens to be the case.

The build time requirements of the make target's build rule are already required by other parts of the grub buildsystem:

* $(CMP)
compare two files by content

* $(PYTHON)
check-commands.py has been written to run on Python 2 and 3 (tested with Python 2.7 and Python 3.7)

The docs/check-commands.py script runs fast enough to not perceivably slow down a "make" on the grub source tree. The script basically reads and more or less greps through docs/grub.texi and all *.c source files, and that happens very quickly.

---
 .gitignore             |   1 +
 docs/Makefile.am       |  16 +++-
 docs/check-commands.py | 205 +++++++++++++++++++++++++++++++++++++++++
 docs/grub.texi         |   4 +
 4 files changed, 223 insertions(+), 3 deletions(-)
 create mode 100644 docs/check-commands.py
.gitignore +1 -0
index 678e829aa..ed019dd44 100644
@@ -93,6 +93,7 @@ widthspec.bin
 /docs/*.info-[0-9]*
 /docs/stamp-1
 /docs/stamp-vti
+/docs/undocumented-commands.texi
 /docs/version-dev.texi
 /docs/version.texi
 /ehci_test
docs/Makefile.am +13 -3
index 93eb39627..8e8500226 100644
@@ -2,8 +2,18 @@ AUTOMAKE_OPTIONS = subdir-objects
 
 # AM_MAKEINFOFLAGS = --no-split --no-validate
 info_TEXINFOS = grub.texi grub-dev.texi
-grub_TEXINFOS = fdl.texi
-
-EXTRA_DIST = font_char_metrics.png font_char_metrics.txt
+grub_TEXINFOS = fdl.texi undocumented-commands.texi
 
+EXTRA_DIST = check-commands.py font_char_metrics.png font_char_metrics.txt
 
+CLEANFILES = undocumented-commands.texi
+BUILT_SOURCES = update-undoc
+update-undoc:
+	$(PYTHON) check-commands.py > undocumented-commands.texi.tmp
+	@if test -f $(srcdir)/undocumented-commands.texi && $(CMP) undocumented-commands.texi.tmp $(srcdir)/undocumented-commands.texi; then \
+		echo "Not updating undocumented-commands.texi: is up to date"; \
+		rm -f undocumented-commands.texi.tmp; \
+	else \
+		echo "Updating undocumented-commands.texi"; \
+		mv -f undocumented-commands.texi.tmp $(srcdir)/undocumented-commands.texi; \
+	fi
docs/check-commands.py new file +205 -0

205 lines not on this page · raw message

docs/grub.texi +4 -0
index d6408d242..0865ffa17 100644
@@ -3764,6 +3764,7 @@ shell}.
 * General commands::
 * Command-line and menu entry commands::
 * Networking commands::
+* Undocumented commands::
 @end menu
 
 
@@ -5636,6 +5637,9 @@ is given, use default list of servers.
 @end deffn
 
 
+@include undocumented-commands.texi
+
+
 @node Internationalisation
 @chapter Internationalisation
 
Signature · 1 line
2.25.3
#2 lore raw

Re: [PATCH 1/3 v3] docs: Add grub command documentation checks

Quoted 15 lines · On Fri, 24 Apr 2020 15:50:54 +0200 Hans Ulrich Niedermann…

This adds the docs/check-commands.py script which checks whether grub commands implemented in the git source tree are documented in docs/grub.texi and vice versa.

During a standard "make" command, BUILT_SOURCES makes sure that docs/undocumented-commands.texi will be created (if it does not exist yet) or updated (if it differs from what the script would generate) or be left alone otherwise.

If you run "make grub.info" in the docs/ subdirectory, the BUILT_SOURCES trick to create or update undocumented-commands.texi will not work (this is an Automake limitation). So if you want to avoid the "all" make target, you need to explictly run "make update-undoc" first.

Quoted 4 lines

The generated docs/undocumented-commands.texi will document each otherwise undocumented command by describing that it is undocumented, and by adding that it is mentioned somewhere else in the info page if that happens to be the case.

The large RFC patchset for TXT boot just made me notice one potential issue with this way of hooking that script into the build process:

Nobody will notice during patch reviews when a patch changes the set of commands in the source code without changing the grub.texi documentation accordingly.

The following are the technical details about why, and how we could force missing documentation changes to show up during patch reviews.

What this patch does (requiring manual code/doc review)
=======================================================

As of this patch, the docs/check-commands.py script is called and docs/undocumented-commands.texi is generated from scratch, and there are no checks on the actual results:

* Every "make" build will generate a new docs/undocumented-commands.texi file which will then be included from docs/grub.texi to be part of the generated grub info page.

(This file should be in $(builddir), not in $(srcdir), but the
older and current build rules for info pages only work in $(srcdir), so docs/undocumented-commands.texi is created in $(srcdir) to stay compatible with this inherited weirdness.)

* If a patch adds or removes commands in the source code, a different docs/undocumented-commands.texi will be generated to reflect that.

However, if nobody happens to notice that the patch changes the set of implemented commands, nobody will notice that documentation updates would be in order.

So patch reviews will not directly show that commands have been added or removed from the codebase and that the documentation should perhaps be adapted to reflect those changes.

What this could be changed to: Forced grub command doc review
=============================================================

If the grub project wants enforce that every single patch which changes the set of implemented commands to also change the set of commands documented in docs/grub.texi accordingly, an alternative approach enforcing this would be the following:

* Add docs/undocumented-commands.texi to the git repository.

* Still have every "make" build update the

    docs/undocumented-commands.texi in the $(srcdir).

* Now, if a patch (set) adds adds or removes grub commands in the source code, the generated docs/undocumented-commands.texi will be different from the docs/undocumented-command.texi from the git repo and thus this difference will show up in patches.

This of course assumes that someone will at some time run "make" after applying the respective patch. I would expect a "make" run to be a part of all test processes, so I would expect this assumption to hold.

* If the patch review process results in the decision that the patch should actually add undocumented grub commands to or remove documented grub commands from the source code, then the changes to docs/undocumented-commands.texi can be committed along with the change to the source code.

However, if the patch review process results in the decision that the source code and the documentation should remain in sync, the further divergence will show up here, and a documentation update to docs/grub.texi to reflect the changes in the source code can be asked for in a revised version of the patch.

* However, this forces changes to docs/grub.texi to be in the same patch as the changes to the grub source code.

If you want it to be possible to have a patch set which contains first a patch with the code change and then another patch with the documentation update, this will result in a lot of ugliness with "make" changing git tracked files to different content. In that case, I would strongly advise to stay with the manual doc review process.

Should there be any patches changing doc/check-commands.py which generates doc/undocumented-commands.texi, these generated changes must be contained the same patch for the same reason.

Conclusion
==========

Which of these two methods is preferable for grub is for other people to decide. Both are valid models. I just needed to present that these two alternative methods exist and can be chosen from.

Quoted 312 lines

The build time requirements of the make target's build rule are already required by other parts of the grub buildsystem:

* $(CMP)
compare two files by content

* $(PYTHON)
check-commands.py has been written to run on Python 2 and 3 (tested with Python 2.7 and Python 3.7)

The docs/check-commands.py script runs fast enough to not perceivably slow down a "make" on the grub source tree. The script basically reads and more or less greps through docs/grub.texi and all *.c source files, and that happens very quickly.

---
 .gitignore             |   1 +
 docs/Makefile.am       |  16 +++-
 docs/check-commands.py | 205
+++++++++++++++++++++++++++++++++++++++++ docs/grub.texi         |
4 + 4 files changed, 223 insertions(+), 3 deletions(-)
 create mode 100644 docs/check-commands.py
.gitignore +1 -0
index 678e829aa..ed019dd44 100644
@@ -93,6 +93,7 @@ widthspec.bin
 /docs/*.info-[0-9]*
 /docs/stamp-1
 /docs/stamp-vti
+/docs/undocumented-commands.texi
 /docs/version-dev.texi
 /docs/version.texi
 /ehci_test
docs/Makefile.am +2 -3
index 93eb39627..8e8500226 100644
@@ -2,8 +2,18 @@ AUTOMAKE_OPTIONS = subdir-objects
 
 # AM_MAKEINFOFLAGS = --no-split --no-validate
 info_TEXINFOS = grub.texi grub-dev.texi
-grub_TEXINFOS = fdl.texi
-
-EXTRA_DIST = font_char_metrics.png font_char_metrics.txt
+grub_TEXINFOS = fdl.texi undocumented-commands.texi
 
+EXTRA_DIST = check-commands.py font_char_metrics.png

font_char_metrics.txt

docs/Makefile.am +5 -0
+CLEANFILES = undocumented-commands.texi
+BUILT_SOURCES = update-undoc
+update-undoc:
+	$(PYTHON) check-commands.py > undocumented-commands.texi.tmp
+	@if test -f $(srcdir)/undocumented-commands.texi && $(CMP)

undocumented-commands.texi.tmp $(srcdir)/undocumented-commands.texi; then \

docs/Makefile.am +1 -0
+		echo "Not updating undocumented-commands.texi: is up

to date"; \

docs/Makefile.am +4 -0
+		rm -f undocumented-commands.texi.tmp; \
+	else \
+		echo "Updating undocumented-commands.texi"; \
+		mv -f undocumented-commands.texi.tmp

$(srcdir)/undocumented-commands.texi; \

docs/Makefile.am +1 -0
+	fi
docs/check-commands.py new file +3 -0
new file mode 100644
index 000000000..4174b954a
@@ -0,0 +1,205 @@
+#!/usr/bin/env python
+#
+# check-commands.py - compare commands defined in grub.texi the *.c

sources +#

docs/check-commands.py new file +4 -0
+# Usage:
+#     check-commands.py > undocumented-commands.texi.new
+#
+# There are no command line parameters. The locations of grub.texi

and +# the *.c source files are determined relative to the location of the +# script file. The only output happens on stdout, in texinfo format +# for inclusione into grub.texi via '@include'.

docs/check-commands.py new file +17 -0
+
+from __future__ import print_function
+
+import os
+import re
+
+import sys
+
+
+class CommandChecker:
+
+    def __init__(self):
+        srcdir, self.prog = os.path.split(__file__)
+        self.top_srcdir = os.path.dirname(os.path.abspath(srcdir))
+
+    def read_texi_text_commands(self, texi_filename):
+        texi_pathname = os.path.join(self.top_srcdir, 'docs',

texi_filename)

docs/check-commands.py new file +10 -0
+        command_re = re.compile(r'@command\{([a-zA-Z0-9_-]+)\}')
+        commands = set()
+        with open(texi_pathname) as texi:
+            for line in texi.readlines():
+                for m in command_re.finditer(line):
+                    commands.add(m[1])
+        return commands
+
+    def read_texi_command_menu(self, texi_filename):
+        texi_pathname = os.path.join(self.top_srcdir, 'docs',

texi_filename)

docs/check-commands.py new file +5 -0
+        menuentry_re = re.compile(r'\* ([^:]+)::')
+        commands_node_re = re.compile(r'^@node .+ commands$')
+        commands = set()
+
+        # Avoid using enum for the state machine state (for

compatibility)

docs/check-commands.py new file +31 -0
+        waiting_for_node = True
+        waiting_for_menu = False
+        collecting_commands = False
+
+        with open(texi_pathname) as texi:
+            for line in texi.readlines():
+                line = line.strip()
+                if line.startswith('@comment'):
+                    continue
+                if waiting_for_node:
+                    if commands_node_re.match(line):
+                        waiting_for_node = False
+                        waiting_for_menu = True
+                        continue
+                if waiting_for_menu:
+                    if line == '@menu':
+                        waiting_for_menu = False
+                        collecting_commands = True
+                        continue
+                if collecting_commands:
+                    if line == '@end menu':
+                        collecting_commands = False
+                        waiting_for_node = True
+                        continue
+                    m = menuentry_re.match(line)
+                    commands.add(m.group(1))
+        return commands
+
+    def read_src_commands(self):
+        top = os.path.join(self.top_srcdir, 'grub-core')
+        register_re =

re.compile(r'grub_register_(command|command_p1|extcmd)\s*\("([a-z0-9A-Z_\[]+)",')

docs/check-commands.py new file +28 -0
+        commands = set()
+        for dirpath, dirnames, filenames in os.walk(top):
+            for filename in filenames:
+                filepath = os.path.join(dirpath, filename)
+                filepath_ext = os.path.splitext(filepath)[1]
+                if filepath_ext == '.c':
+                    with open(filepath) as cfile:
+                        for line in cfile.readlines():
+                            for m in register_re.finditer(line):
+                                commands.add(m.group(2))
+        return commands
+
+
+def write_undocumented_commands(undocumented_commands,
+                                mentioned_commands):
+    print("""\
+@node Undocumented commands
+@section The list of undocumented commands
+""")
+
+    if not undocumented_commands:
+        print("""
+There appear to be no undocumented commands at this time.
+""")
+        return
+
+    print("""\
+These commands are implemented in the grub software but still need

to be +documented and sorted into categories.
+""")

docs/check-commands.py new file +2 -0
+
+    sorted_undocumented_commands =

sorted(list(undocumented_commands)) +
+ # Fit the longest command into the format string for the menu
entries

docs/check-commands.py new file +9 -0
+    maxlen_str = sorted(list(undocumented_commands),
+                        key=lambda cmd: len(cmd),
+                        reverse=True)[0]
+    fmt = '* %%-%ds %%s' % (2+len(maxlen_str))
+
+    print("@menu")
+    for cmd in sorted_undocumented_commands:
+        if cmd in mentioned_commands:
+            print(fmt % ("%s::" % cmd, "Undocumented command (but

mentioned somewhere)"))

docs/check-commands.py new file +10 -0
+        else:
+            print(fmt % ("%s::" % cmd, "Undocumented command"))
+    print("@end menu")
+    print()
+
+    for cmd in sorted_undocumented_commands:
+        print("@node %s" % cmd)
+        print("@subsection %s" % cmd)
+        print()
+        print("The grub command @command{%s} has not been documented

properly yet." % cmd)

docs/check-commands.py new file +21 -0
+        if cmd in mentioned_commands:
+            print("""
+However, the @command{%s} command @emph{is} mentioned
+somewhere within this info file, so you might still find some
+interesting information there.""" % cmd)
+        print()
+
+
+def print_set(st):
+    for n, item in enumerate(sorted(list(st)), start=1):
+        print("@comment", "  %d." % n, item)
+
+
+def main():
+    cc = CommandChecker()
+
+    print("""\
+@comment Automatically generated by the %s script.
+@comment Do not modify this generated file.
+@comment
+@comment If you want to document some of the commands listed below,

feel +@comment free to copy over some of the @nodes and @subsections below +@comment into the grub.texi file proper and re-generate this file. +@comment""" % cc.prog)

docs/check-commands.py new file +6 -0
+
+    # This gives a few good hints about what commands are at least
+    # mentioned somewhere which can be helpful for finding out more
+    # about commands.
+    texi_text_commands = cc.read_texi_text_commands('grub.texi')
+    print("@comment", "Commands mentioned somewhere in the grub.texi

text:")

docs/check-commands.py new file +4 -0
+    print_set(texi_text_commands)
+
+    texi_node_commands = cc.read_texi_command_menu('grub.texi')
+    # print("@comment", "Commands documented in grub.texi

node/subsection:")

docs/check-commands.py new file +10 -0
+    # print_set(texi_node_commands)
+
+    src_commands = cc.read_src_commands()
+    # print("@comment", "Commands registered in grub source:")
+    # print_set(src_commands)
+
+    node_without_src = texi_node_commands - src_commands
+    if node_without_src:
+        print("@comment",
+              "Cmds documented in grub.texi node/subsection but not

registered in source:")

docs/check-commands.py new file +12 -0
+        print_set(node_without_src)
+        print("@comment")
+    else:
+        print("""\
+@comment There appear to be no commands documented in a grub.texi
+@comment node/subsection but not registered in grub source.
+@comment""")
+
+    src_without_node = src_commands - texi_node_commands
+    if src_without_node:
+        print("@comment",
+              "Cmds registered in source but not documented in

grub.texi node/subsection:")

docs/check-commands.py new file +12 -0
+        print_set(src_without_node)
+    else:
+        print("""\
+@comment All commands registered in the source appear to have been
+@comment documented in a grub.texi node/subsection. Congratulations!
+@comment""")
+
+    print()
+
+    write_undocumented_commands(src_without_node, texi_text_commands)
+
+    # Once grub.texi actually documents all commands, we can

uncomment

docs/check-commands.py new file +3 -0
+    # this and actually fail if the set of implemented commands and
+    # the set of documented commands differ in any way.
+    # if ((len(node_without_src) > 0) or (len(src_without_node) >

0)):

docs/check-commands.py new file +7 -0
+    #     sys.exit(1)
+
+    sys.exit(0)
+
+
+if __name__ == '__main__':
+    main()
docs/grub.texi +4 -0
index d6408d242..0865ffa17 100644
@@ -3764,6 +3764,7 @@ shell}.
 * General commands::
 * Command-line and menu entry commands::
 * Networking commands::
+* Undocumented commands::
 @end menu
 
 
@@ -5636,6 +5637,9 @@ is given, use default list of servers.
 @end deffn
 
 
+@include undocumented-commands.texi
+
+
 @node Internationalisation
 @chapter Internationalisation
 
#3 lore raw

PATCH v3 2/3 docs: Remove docs for non-existing pxe_unload command

In reply to Hans Ulrich Niedermann

Remove the documentation of the pxe_unload command from the docs/grub.texi file.

The pxe_unload command is not implemented in the grub source at this time at all. It appears to have been removed in commit 71a78acb0648d3d73c95ab0f021f907499aacc0 (from 2011-07-05).

---
 docs/grub.texi | 11 -----------
 1 file changed, 11 deletions(-)
docs/grub.texi +0 -11
index 0865ffa17..0f7886bb6 100644
@@ -4012,7 +4012,6 @@ you forget a command, you can run the command @command{help}
 * password_pbkdf2::             Set a hashed password
 * play::                        Play a tune
 * probe::                       Retrieve device info
-* pxe_unload::                  Unload the PXE environment
 * rdmsr::                       Read values from model-specific registers
 * read::                        Read user input
 * reboot::                      Reboot your computer
@@ -4863,16 +4862,6 @@ The option @option{--part-uuid} is currently only implemented for GPT-formatted
 @end deffn
 
 
-@node pxe_unload
-@subsection pxe_unload
-
-@deffn Command pxe_unload
-Unload the PXE environment (@pxref{Network}).
-
-This command is only available on PC BIOS systems.
-@end deffn
-
-
 @node rdmsr
 @subsection rdmsr
 
Signature · 1 line
2.25.3
#4 lore raw

PATCH v3 3/3 docs: Remove docs for non-existing uppermem command

In reply to Hans Ulrich Niedermann

Remove all documentation of and mentions of the uppermem command from the docs/grub.texi file.

The uppermem command is not implemented in the GRUB source at all and appears to not have been implemented despite there having been plans to do so.

To reduce user confusion, this even removes the paragraph describing how GRUB's uppermem command was supposed to complement the Linux kernel's mem= parameter.

---
 docs/grub.texi | 13 -------------
 1 file changed, 13 deletions(-)
docs/grub.texi +0 -13
index 0f7886bb6..d51ad0e25 100644
@@ -1114,12 +1114,6 @@ grub> @kbd{initrd16 /initrd}
 Finally, run the command @command{boot} (@pxref{boot}).
 @end enumerate
 
-@strong{Caution:} If you use an initrd and specify the @samp{mem=}
-option to the kernel to let it use less than actual memory size, you
-will also have to specify the same memory size to GRUB. To let GRUB know
-the size, run the command @command{uppermem} @emph{before} loading the
-kernel. @xref{uppermem}, for more information.
-
 
 @node NetBSD
 @subsection NetBSD
@@ -4031,7 +4025,6 @@ you forget a command, you can run the command @command{help}
 * true::                        Do nothing, successfully
 * trust::                       Add public key to list of trusted keys
 * unset::                       Unset an environment variable
-* uppermem::                    Set the upper memory size
 @comment * vbeinfo::                     List available video modes
 * verify_detached::             Verify detached digital signature
 * videoinfo::                   List available video modes
@@ -5349,12 +5342,6 @@ Unset the environment variable @var{envvar}.
 @end deffn
 
 
-@node uppermem
-@subsection uppermem
-
-This command is not yet implemented for GRUB 2, although it is planned.
-
-
 @ignore
 @node vbeinfo
 @subsection vbeinfo
Signature · 1 line
2.25.3