Skip to contentSkip to conversation

Use Doxygen and sphinx for html documentation

PATCH Superseded Luca Fancellu 7 participants 34 messages Apr 6, 2021 – Apr 8, 2021

v1 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

cover Use Doxygen and sphinx for html documentation

This serie introduce doxygen in the sphinx html docs generation. One benefit is to keep most of the documentation in the source files of xen so that it's more maintainable, on the other hand there are some limitation of doxygen that must be addressed modifying the current codebase (for example doxygen can't parse anonymous structure/union).

To reproduce the documentation xen must be compiled because most of the headers are generated on compilation time from the makefiles.

Here follows the steps to generate the sphinx html docs, some package may be required on your machine, everything is suggested by the autoconf script.
Here I'm building the arm64 docs (the only introduced for now by this serie):

./configure
make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" make -C docs XEN_TARGET_ARCH="arm64" sphinx-html

now in docs/sphinx/html/ we have the generated docs starting from the index.html page.

Luca Fancellu (3):
docs: add doxygen support for html documentation
docs: hypercalls sphinx skeleton for generated html docs/doxygen: doxygen documentation for grant_table.h

 .gitignore                                    |    7 +
 config/Docs.mk.in                             |    2 +
 docs/Makefile                                 |   51 +-
 docs/conf.py                                  |   48 +-
 docs/configure                                |  258 ++
 docs/configure.ac                             |   15 +
 docs/hypercall-interfaces/arm32.rst           |    4 +
 docs/hypercall-interfaces/arm64.rst           |   33 +
 .../arm64/grant_tables.rst                    |    8 +
 docs/hypercall-interfaces/index.rst.in        |    7 +
 docs/hypercall-interfaces/x86_64.rst          |    4 +
 docs/index.rst                                |    8 +
 docs/xen-doxygen/customdoxygen.css            |   36 +
 docs/xen-doxygen/doxy_input.list              |    1 +
 docs/xen-doxygen/doxy_input_template.in       |   21 +
 docs/xen-doxygen/footer.html                  |   21 +
 docs/xen-doxygen/header.html                  |   56 +
 docs/xen-doxygen/mainpage.md                  |    5 +
 docs/xen-doxygen/xen_project_logo_165x67.png  |  Bin 0 -> 18223 bytes
 docs/xen.doxyfile.in                          | 2314 +++++++++++++++++
 m4/ax_python_module.m4                        |   56 +
 m4/docs_tool.m4                               |    9 +
 xen/include/public/grant_table.h              |   79 +-
 23 files changed, 3007 insertions(+), 36 deletions(-)
 create mode 100644 docs/hypercall-interfaces/arm32.rst
 create mode 100644 docs/hypercall-interfaces/arm64.rst
 create mode 100644 docs/hypercall-interfaces/arm64/grant_tables.rst
 create mode 100644 docs/hypercall-interfaces/index.rst.in
 create mode 100644 docs/hypercall-interfaces/x86_64.rst
 create mode 100644 docs/xen-doxygen/customdoxygen.css
 create mode 100644 docs/xen-doxygen/doxy_input.list
 create mode 100644 docs/xen-doxygen/doxy_input_template.in
 create mode 100644 docs/xen-doxygen/footer.html
 create mode 100644 docs/xen-doxygen/header.html
 create mode 100644 docs/xen-doxygen/mainpage.md
 create mode 100644 docs/xen-doxygen/xen_project_logo_165x67.png
 create mode 100644 docs/xen.doxyfile.in
 create mode 100644 m4/ax_python_module.m4
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/3 docs: add doxygen support for html documentation

Add doxygen support to build html documentation with sphinx, to do that the following modification are applied:

1) Modify docs/configure.ac and consequently configure script to check, through the ./configure script, the presence in the system of the sphinx-build binary, if it is found then it checks also the presence of the doxygen binary, breathe and sphinx-rtd-theme python packages.
Doxygen and the packages are required only if sphinx-build is found, otherwise the Makefile will simply skip the sphinx html generation.
The ax_python_module.m4 support is needed to check for python packages.
2) Add doxygen templates and configuration file
3) Modify docs/Makefile to call in sequence doxygen and sphinx-build, the doxygen configuration file will be modified to include the xen absolute path and
a list of folder to check.
A doxygen_input.h file is generated to include every header listed in the doxy_input.list file.

---
 .gitignore                                   |    6 +
 config/Docs.mk.in                            |    2 +
 docs/Makefile                                |   47 +-
 docs/conf.py                                 |   48 +-
 docs/configure                               |  258 ++
 docs/configure.ac                            |   15 +
 docs/xen-doxygen/customdoxygen.css           |   36 +
 docs/xen-doxygen/doxy_input.list             |    0
 docs/xen-doxygen/doxy_input_template.in      |   21 +
 docs/xen-doxygen/footer.html                 |   21 +
 docs/xen-doxygen/header.html                 |   56 +
 docs/xen-doxygen/mainpage.md                 |    5 +
 docs/xen-doxygen/xen_project_logo_165x67.png |  Bin 0 -> 18223 bytes
 docs/xen.doxyfile.in                         | 2314 ++++++++++++++++++
 m4/ax_python_module.m4                       |   56 +
 m4/docs_tool.m4                              |    9 +
 16 files changed, 2888 insertions(+), 6 deletions(-)
 create mode 100644 docs/xen-doxygen/customdoxygen.css
 create mode 100644 docs/xen-doxygen/doxy_input.list
 create mode 100644 docs/xen-doxygen/doxy_input_template.in
 create mode 100644 docs/xen-doxygen/footer.html
 create mode 100644 docs/xen-doxygen/header.html
 create mode 100644 docs/xen-doxygen/mainpage.md
 create mode 100644 docs/xen-doxygen/xen_project_logo_165x67.png
 create mode 100644 docs/xen.doxyfile.in
 create mode 100644 m4/ax_python_module.m4
.gitignore +6 -0
index 1c2fa1530b..a5eebf1213 100644
@@ -58,6 +58,12 @@ docs/man7/
 docs/man8/
 docs/pdf/
 docs/txt/
+docs/doxygen-output
+docs/sphinx
+docs/xen.doxyfile
+docs/xen.doxyfile.tmp
+docs/doxygen_input.h
+docs/doxygen_input.h.tmp
 extras/mini-os*
 install/*
 stubdom/*-minios-config.mk
config/Docs.mk.in +2 -0
index e76e5cd5ff..dfd4a02838 100644
@@ -7,3 +7,5 @@ POD2HTML            := @POD2HTML@
 POD2TEXT            := @POD2TEXT@
 PANDOC              := @PANDOC@
 PERL                := @PERL@
+SPHINXBUILD         := @SPHINXBUILD@
+DOXYGEN             := @DOXYGEN@
docs/Makefile +44 -3

65 lines not on this page · raw message

docs/conf.py +45 -3

83 lines not on this page · raw message

docs/configure +258 -0

294 lines not on this page · raw message

docs/configure.ac +15 -0
index c2e5edd3b3..a2ff55f30a 100644
@@ -20,6 +20,7 @@ m4_include([../m4/docs_tool.m4])
 m4_include([../m4/path_or_fail.m4])
 m4_include([../m4/features.m4])
 m4_include([../m4/paths.m4])
+m4_include([../m4/ax_python_module.m4])
 
 AX_XEN_EXPAND_CONFIG()
 
@@ -29,6 +30,20 @@ AX_DOCS_TOOL_PROG([POD2HTML], [pod2html])
 AX_DOCS_TOOL_PROG([POD2TEXT], [pod2text])
 AX_DOCS_TOOL_PROG([PANDOC], [pandoc])
 
+# If sphinx is installed, make sure to have also the dependencies to build
+# Sphinx documentation.
+AC_CHECK_PROGS([SPHINXBUILD], [sphinx-build], [no])
+    AS_IF([test "x$SPHINXBUILD" = xno],
+        [AC_MSG_WARN(sphinx-build is not available so sphinx documentation \
+won't be built)],
+        [
+            AC_PATH_PROG([SPHINXBUILD], [sphinx-build])
+            AX_DOCS_TOOL_REQ_PROG([DOXYGEN], [doxygen])
+            AX_PYTHON_MODULE([breathe],[yes])
+            AX_PYTHON_MODULE([sphinx_rtd_theme], [yes])
+        ]
+    )
+
 AC_ARG_VAR([PERL], [Path to Perl parser])
 AX_PATH_PROG_OR_FAIL([PERL], [perl])
 
docs/xen-doxygen/customdoxygen.css new file +36 -0
new file mode 100644
index 0000000000..4735e41cf5
@@ -0,0 +1,36 @@
+/* Custom CSS for Doxygen-generated HTML
+ * Copyright (c) 2015 Intel Corporation
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+code {
+  font-family: Monaco,Menlo,Consolas,"Courier New",monospace;
+  background-color: #D8D8D8;
+  padding: 0 0.25em 0 0.25em;
+}
+
+pre.fragment {
+  display: block;
+  font-family: Monaco,Menlo,Consolas,"Courier New",monospace;
+  padding: 1rem;
+  word-break: break-all;
+  word-wrap: break-word;
+  white-space: pre;
+  background-color: #D8D8D8;
+}
+
+#projectlogo
+{
+  vertical-align: middle;
+}
+
+#projectname
+{
+  font: 200% Tahoma, Arial,sans-serif;
+  color: #3D578C;
+}
+
+#projectbrief
+{
+  color: #3D578C;
+}
docs/xen-doxygen/doxy_input.list new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
docs/xen-doxygen/doxy_input_template.in new file +21 -0
new file mode 100644
index 0000000000..fc7565ab5b
@@ -0,0 +1,21 @@
+#define __XEN__
+
+#include "@XEN_BASE@/xen/include/xen/config.h"
+
+#if defined(CONFIG_X86_64)
+
+#define __x86_64__ 1
+
+#elif defined(CONFIG_ARM_64)
+
+#define __aarch64__ 1
+
+#elif defined(CONFIG_ARM_32)
+
+#define __arm__ 1
+
+#else
+
+#error Architecture not supported/recognized.
+
+#endif
docs/xen-doxygen/footer.html new file +21 -0
new file mode 100644
index 0000000000..a24bf2b9b4
@@ -0,0 +1,21 @@
+<!-- HTML footer for doxygen 1.8.13-->
+<!-- start footer part -->
+<!--BEGIN GENERATE_TREEVIEW-->
+<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
+  <ul>
+    $navpath
+    <li class="footer">$generatedby
+    <a href="http://www.doxygen.org/index.html">
+    <img class="footer" src="$relpath^doxygen.png" alt="doxygen"/></a> $doxygenversion </li>
+  </ul>
+</div>
+<!--END GENERATE_TREEVIEW-->
+<!--BEGIN !GENERATE_TREEVIEW-->
+<hr class="footer"/><address class="footer"><small>
+$generatedby &#160;<a href="http://www.doxygen.org/index.html">
+<img class="footer" src="$relpath^doxygen.png" alt="doxygen"/>
+</a> $doxygenversion
+</small></address>
+<!--END !GENERATE_TREEVIEW-->
+</body>
+</html>
docs/xen-doxygen/header.html new file +56 -0

56 lines not on this page · raw message

docs/xen-doxygen/mainpage.md new file +5 -0
new file mode 100644
index 0000000000..ff548b87fc
@@ -0,0 +1,5 @@
+# API Documentation   {#index}
+
+## Introduction
+
+## Licensing
docs/xen-doxygen/xen_project_logo_165x67.png new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..7244959d59cdeb9f23c5202160ea45508dfc7265
GIT binary patch
docs/xen.doxyfile.in new file +2314 -0

2,314 lines not on this page · raw message

m4/ax_python_module.m4 new file +56 -0

57 lines not on this page · raw message

m4/docs_tool.m4 +9 -0
index 3e8814ac8d..39aa348026 100644
@@ -15,3 +15,12 @@ dnl
         AC_MSG_WARN([$2 is not available so some documentation won't be built])
     ])
 ])
+
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.17.1

Part of the diff is not shown here. View the raw message.

#2 lore raw

PATCH 2/3 docs: hypercalls sphinx skeleton for generated html

In reply to Luca Fancellu

Create a skeleton for the documentation about hypercalls

---
 .gitignore                             |  1 +
 docs/Makefile                          |  4 ++++
 docs/hypercall-interfaces/arm32.rst    |  4 ++++
 docs/hypercall-interfaces/arm64.rst    | 32 ++++++++++++++++++++++++++
 docs/hypercall-interfaces/index.rst.in |  7 ++++++
 docs/hypercall-interfaces/x86_64.rst   |  4 ++++
 docs/index.rst                         |  8 +++++++
 7 files changed, 60 insertions(+)
 create mode 100644 docs/hypercall-interfaces/arm32.rst
 create mode 100644 docs/hypercall-interfaces/arm64.rst
 create mode 100644 docs/hypercall-interfaces/index.rst.in
 create mode 100644 docs/hypercall-interfaces/x86_64.rst
.gitignore +1 -0
index a5eebf1213..dd3ca96b6f 100644
@@ -64,6 +64,7 @@ docs/xen.doxyfile
 docs/xen.doxyfile.tmp
 docs/doxygen_input.h
 docs/doxygen_input.h.tmp
+docs/hypercall-interfaces/index.rst
 extras/mini-os*
 install/*
 stubdom/*-minios-config.mk
docs/Makefile +4 -0
index dd651e9a93..2f29ca7e78 100644
@@ -65,6 +65,9 @@ build: html txt pdf man-pages figs
 sphinx-html: $(DOXY_DEPS) $(DOXY_LIST_SOURCES)
 ifneq ($(SPHINXBUILD),)
 	$(DOXYGEN) xen.doxyfile
+	@echo "Generating hypercall-interfaces/index.rst"
+	@sed -e "s,@XEN_TARGET_ARCH@,$(XEN_TARGET_ARCH),g" \
+		hypercall-interfaces/index.rst.in > hypercall-interfaces/index.rst
 	XEN_ROOT=$(realpath $(XEN_ROOT)) $(SPHINXBUILD) -b html . sphinx/html
 else
 	@echo "Sphinx is not installed; skipping sphinx-html documentation."
@@ -113,6 +116,7 @@ clean: clean-man-pages
 	rm -f xen.doxyfile.tmp
 	rm -f doxygen_input.h
 	rm -f doxygen_input.h.tmp
+	rm -f hypercall-interfaces/index.rst
 
 .PHONY: distclean
 distclean: clean
docs/hypercall-interfaces/arm32.rst new file +4 -0
new file mode 100644
index 0000000000..4e973fbbaf
@@ -0,0 +1,4 @@
+.. SPDX-License-Identifier: CC-BY-4.0
+
+Hypercall Interfaces - arm32
+============================
docs/hypercall-interfaces/arm64.rst new file +32 -0
new file mode 100644
index 0000000000..5e701a2adc
@@ -0,0 +1,32 @@
+.. SPDX-License-Identifier: CC-BY-4.0
+
+Hypercall Interfaces - arm64
+============================
+
+Starting points
+---------------
+.. toctree::
+   :maxdepth: 2
+
+
+
+Functions
+---------
+
+
+Structs
+-------
+
+
+Enums and sets of #defines
+--------------------------
+
+
+Typedefs
+--------
+
+
+Enum values and individual #defines
+-----------------------------------
+
+
docs/hypercall-interfaces/index.rst.in new file +7 -0
new file mode 100644
index 0000000000..e4dcc5db8d
@@ -0,0 +1,7 @@
+.. SPDX-License-Identifier: CC-BY-4.0
+
+Hypercall Interfaces
+====================
+
+.. toctree::
+   @XEN_TARGET_ARCH@
docs/hypercall-interfaces/x86_64.rst new file +4 -0
new file mode 100644
index 0000000000..3ed70dff95
@@ -0,0 +1,4 @@
+.. SPDX-License-Identifier: CC-BY-4.0
+
+Hypercall Interfaces - x86_64
+=============================
docs/index.rst +8 -0
index b75487a05d..52226a42d8 100644
@@ -53,6 +53,14 @@ kind of development environment.
    hypervisor-guide/index
 
 
+Hypercall Interfaces documentation
+----------------------------------
+
+.. toctree::
+   :maxdepth: 2
+
+   hypercall-interfaces/index
+
 Miscellanea
 -----------
 
Signature · 1 line
2.17.1
#3 lore raw

PATCH 3/3 docs/doxygen: doxygen documentation for grant_table.h

In reply to Luca Fancellu

Modification to include/public/grant_table.h:

1) Change anonymous structure to be named structure, because doxygen can't deal with them.
2) Add doxygen tags to:
- Create Grant tables section
- include variables in the generated documentation
3) Add .rst file for grant table for Arm64

---
 docs/hypercall-interfaces/arm64.rst           |  1 +
 .../arm64/grant_tables.rst                    |  8 ++
 docs/xen-doxygen/doxy_input.list              |  1 +
 xen/include/public/grant_table.h              | 79 ++++++++++++-------
 4 files changed, 59 insertions(+), 30 deletions(-)
 create mode 100644 docs/hypercall-interfaces/arm64/grant_tables.rst
docs/hypercall-interfaces/arm64.rst +1 -0
index 5e701a2adc..c30a7142b1 100644
@@ -8,6 +8,7 @@ Starting points
 .. toctree::
    :maxdepth: 2
 
+   arm64/grant_tables
 
 
 Functions
docs/hypercall-interfaces/arm64/grant_tables.rst new file +8 -0
new file mode 100644
index 0000000000..8955ec5812
@@ -0,0 +1,8 @@
+.. SPDX-License-Identifier: CC-BY-4.0
+
+Grant Tables
+============
+
+.. doxygengroup:: grant_table
+   :project: Xen
+   :members:
docs/xen-doxygen/doxy_input.list +1 -0
index e69de29bb2..233d692fa7 100644
@@ -0,0 +1 @@
+xen/include/public/grant_table.h
xen/include/public/grant_table.h +49 -30

191 lines not on this page · raw message

Signature · 1 line
2.17.1
#4 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 4 lines · Luca Fancellu

Modification to include/public/grant_table.h:

1) Change anonymous structure to be named structure, because doxygen can't deal with them.

Especially in the form presented (adding further name space clutter for consumers to fall over) I object to this, most notably ...

xen/include/public/grant_table.h DEFINE_XEN_GUEST_HANDLE(gnttab_swap_grant_ref_t); · line 602

Quoted 6 lines
xen/include/public/grant_table.h +1 -1
@@ -584,7 +599,7 @@ DEFINE_XEN_GUEST_HANDLE(gnttab_swap_grant_ref_t);
  * page granted to the calling domain by a foreign domain.
  */
 struct gnttab_cache_flush {
-    union {
+    union a {

... this one.

Jan

#5 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 9 lines · Jan Beulich
Quoted 4 lines · Luca Fancellu

Modification to include/public/grant_table.h:

1) Change anonymous structure to be named structure, because doxygen can't deal with them.

Especially in the form presented (adding further name space clutter for consumers to fall over) I object to this, most notably ...

Quoted 8 lines

xen/include/public/grant_table.h DEFINE_XEN_GUEST_HANDLE(gnttab_swap_grant_ref_t); · line 602

Quoted 6 lines
xen/include/public/grant_table.h +1 -1
@@ -584,7 +599,7 @@ DEFINE_XEN_GUEST_HANDLE(gnttab_swap_grant_ref_t);
  * page granted to the calling domain by a foreign domain.
  */
 struct gnttab_cache_flush {
-    union {
+    union a {

... this one.

Hi Jan,

It is unfortunate that none of these tools support anonymous structs or unions well. (You might recall we also had issues with the older kernel-doc series too, although a bit different.)

It is difficult to provide a good name here, a suggestion would be more than welcome. I agree with you that calling it "a" is a bad idea: "a" becomes a globally-visible union name.

Maybe we could call it: StructName_MemberName, so in this case:

union gnttab_cache_flush_a

It makes sure it is unique and doesn't risk clashing with anything else. We can follow this pattern elsewhere as well.

Any better suggestions?

#6 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 19 lines · Stefano Stabellini
Quoted 9 lines · Jan Beulich
Quoted 4 lines · Luca Fancellu

Modification to include/public/grant_table.h:

1) Change anonymous structure to be named structure, because doxygen can't deal with them.

Especially in the form presented (adding further name space clutter for consumers to fall over) I object to this, most notably ...

Quoted 8 lines

xen/include/public/grant_table.h DEFINE_XEN_GUEST_HANDLE(gnttab_swap_grant_ref_t); · line 602

Quoted 6 lines
xen/include/public/grant_table.h +1 -1
@@ -584,7 +599,7 @@ DEFINE_XEN_GUEST_HANDLE(gnttab_swap_grant_ref_t);
  * page granted to the calling domain by a foreign domain.
  */
 struct gnttab_cache_flush {
-    union {
+    union a {

... this one.

Quoted 3 lines

It is unfortunate that none of these tools support anonymous structs or unions well. (You might recall we also had issues with the older kernel-doc series too, although a bit different.)

While I wouldn't veto such changes, I think it is a very bad approach to make adjustments like this to cover for a docs tool shortcoming. Is it entirely unreasonable to have the tool fixed? In fact, if the issue was run into before, isn't it a bad sign that the tool hasn't been fixed already, and we merely need to require a certain minimum version?

Quoted 4 lines

It is difficult to provide a good name here, a suggestion would be more than welcome. I agree with you that calling it "a" is a bad idea: "a" becomes a globally-visible union name.

Quoted 8 lines

Maybe we could call it: StructName_MemberName, so in this case:

union gnttab_cache_flush_a

It makes sure it is unique and doesn't risk clashing with anything else. We can follow this pattern elsewhere as well.

Any better suggestions?

First and foremost any new additions ought to use a xen_, Xen_, or XEN_ prefix. For the specific case here, since "a" is already a rather bad choice for a member name (What does it stand for? In lieu of any better name we typically use "u" in such cases.), the badness shouldn't be extended. Sadly the ref as being one way of expressing the target MFN is also not accompanied by a GFN (as it ought to be), but an address. Otherwise I would have suggested to abstract the similar union also used by struct gnttab_copy. In the end I can't see many alternatives to something like xen_gnttab_cache_flush_target or xen_gnttab_cache_flush_ref_or_addr.

Jan

#7 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 48 lines · Jan Beulich
Quoted 19 lines · Stefano Stabellini
Quoted 9 lines · Jan Beulich
Quoted 4 lines · Luca Fancellu

Modification to include/public/grant_table.h:

1) Change anonymous structure to be named structure, because doxygen can't deal with them.

Especially in the form presented (adding further name space clutter for consumers to fall over) I object to this, most notably ...

Quoted 8 lines

xen/include/public/grant_table.h DEFINE_XEN_GUEST_HANDLE(gnttab_swap_grant_ref_t); · line 602

Quoted 6 lines
xen/include/public/grant_table.h +1 -1
@@ -584,7 +599,7 @@ DEFINE_XEN_GUEST_HANDLE(gnttab_swap_grant_ref_t);
 * page granted to the calling domain by a foreign domain.
 */
 struct gnttab_cache_flush {
-    union {
+    union a {

... this one.

Quoted 3 lines

It is unfortunate that none of these tools support anonymous structs or unions well. (You might recall we also had issues with the older kernel-doc series too, although a bit different.)

While I wouldn't veto such changes, I think it is a very bad approach to make adjustments like this to cover for a docs tool shortcoming. Is it entirely unreasonable to have the tool fixed? In fact, if the issue was run into before, isn't it a bad sign that the tool hasn't been fixed already, and we merely need to require a certain minimum version?

Quoted 4 lines

It is difficult to provide a good name here, a suggestion would be more than welcome. I agree with you that calling it "a" is a bad idea: "a" becomes a globally-visible union name.

Quoted 8 lines

Maybe we could call it: StructName_MemberName, so in this case:

union gnttab_cache_flush_a

It makes sure it is unique and doesn't risk clashing with anything else. We can follow this pattern elsewhere as well.

Any better suggestions?

Quoted 10 lines

First and foremost any new additions ought to use a xen_, Xen_, or XEN_ prefix. For the specific case here, since "a" is already a rather bad choice for a member name (What does it stand for? In lieu of any better name we typically use "u" in such cases.), the badness shouldn't be extended. Sadly the ref as being one way of expressing the target MFN is also not accompanied by a GFN (as it ought to be), but an address. Otherwise I would have suggested to abstract the similar union also used by struct gnttab_copy. In the end I can't see many alternatives to something like xen_gnttab_cache_flush_target or xen_gnttab_cache_flush_ref_or_addr.

Hi Jan,

Thank you for your feedback, I agree with you all that “a” is not really a good name, I will be happy to change it if we define a pattern.

Just to be sure that we are in the same page, are you suggesting to modify the name In this way?

struct gnttab_cache_flush {
- union {

+    union xen_gnttab_cache_flush_a {
        uint64_t dev_bus_addr;
        grant_ref_t ref;
    } a;

Following this kind of pattern: xen_<upper struct name>_<member name> ?

Cheers,
Luca

Quoted 2 lines

Jan

#8 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 11 lines · Luca Fancellu

Just to be sure that we are in the same page, are you suggesting to modify the name In this way?

struct gnttab_cache_flush {

xen/include/public/grant_table.h +1 -1
-    union {
+    union xen_gnttab_cache_flush_a {
        uint64_t dev_bus_addr;
        grant_ref_t ref;
    } a;

Following this kind of pattern: xen_<upper struct name>_<member name> ?

While in general I would be fine with this scheme, for field names like "a" or "u" it doesn't fit well imo. I'm also unconvinced this would be scalable to the case where there's further struct/union nesting.

Jan

#9 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 11 lines · Jan Beulich
Quoted 10 lines · Luca Fancellu

Just to be sure that we are in the same page, are you suggesting to modify the name In this way?

struct gnttab_cache_flush {

xen/include/public/grant_table.h +1 -1
-    union {
+    union xen_gnttab_cache_flush_a {
        uint64_t dev_bus_addr;
        grant_ref_t ref;
    } a;
Quoted 4 lines
Quoted 1 line

Following this kind of pattern: xen_<upper struct name>_<member name> ?

While in general I would be fine with this scheme, for field names like "a" or "u" it doesn't fit well imo.

"a" is a bad name anyway, even for the member. We can take the opportunity to find a better name. Almost anything would be better than "a". Maybe "refaddr"?

Quoted 2 lines

I'm also unconvinced this would be
scalable to the case where there's further struct/union nesting.

How many of these instances of multilevel nesting do we have? Luca might know. Probably not many? They could be special-cased.

#10 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 12 lines · Stefano Stabellini
Quoted 11 lines · Jan Beulich
Quoted 10 lines · Luca Fancellu

Just to be sure that we are in the same page, are you suggesting to modify the name In this way?

struct gnttab_cache_flush {

(quoted diff) +1 -1
-    union {
+    union xen_gnttab_cache_flush_a {
        uint64_t dev_bus_addr;
        grant_ref_t ref;
    } a;
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Following this kind of pattern: xen_<upper struct name>_<member name> ?

While in general I would be fine with this scheme, for field names like "a" or "u" it doesn't fit well imo.

"a" is a bad name anyway, even for the member. We can take the opportunity to find a better name. Almost anything would be better than "a". Maybe "refaddr"?

We need to be careful with changing _anything_ in the public interface. Consumers importing our headers directly (as was e.g. done for the old linux-2.6.18-xen.hg tree) could break with a field name change as much as with any other changes to what had been made available to them.

Jan

#11 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

In reply to Stefano Stabellini
Quoted 26 lines · Stefano Stabellini
Quoted 11 lines · Jan Beulich
Quoted 10 lines · Luca Fancellu

Just to be sure that we are in the same page, are you suggesting to modify the name In this way?

struct gnttab_cache_flush {

(quoted diff) +1 -1
-    union {
+    union xen_gnttab_cache_flush_a {
       uint64_t dev_bus_addr;
       grant_ref_t ref;
   } a;
Quoted 4 lines
Quoted 1 line

Following this kind of pattern: xen_<upper struct name>_<member name> ?

While in general I would be fine with this scheme, for field names like "a" or "u" it doesn't fit well imo.

"a" is a bad name anyway, even for the member. We can take the opportunity to find a better name. Almost anything would be better than "a". Maybe "refaddr"?

Quoted 5 lines
Quoted 2 lines

I'm also unconvinced this would be
scalable to the case where there's further struct/union nesting.

How many of these instances of multilevel nesting do we have? Luca might know. Probably not many? They could be special-cased.

There are not many multilevel nesting instances of anonymous struct/union and the maximum level of nesting I found in the public headers is 2:

union {
	union/struct {
		…
	} <name>
} <name>

I also see that in the majority of cases the unions have not meaningful names like “a” or “u” as member name, instead struct names are fine, It could be fine to keep the meaningful name the same for the struct type name and use the pattern for the non-meaningful ones as long as the names doesn’t create compilation errors?

Example:

struct upper_level {
	union {
		struct {
		
		} meaningful_name1;
		struct {
		
		} meaningful_name2;
	} u;
};

becomes:

struct upper_level {
	union upper_level_u {
		struct meaningful_name1 {
		
		} meaningful_name1;
		struct meaningful_name2 {
		
		} meaningful_name2;
	} u;
};

Doing this will help a lot the documentation side because the html page will show the "struct upper_level" with inside the “union upper_level_u" and inside again the two struct “meaningful_name1" and “meaningful_name2", and from the point of view of the developer it can tell her/him exactly the name of the member to access when writing code (apart from the upper_level_u that can be accessed by u, but we can’t clearly change it).
If this sounds difficult to understand when reading, please generate the documentation and have a look on the page in one side and the source code in another.

Moreover the change should be back-compatible since we are not changing member names.

Cheers,

Luca

#12 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 59 lines · Luca Fancellu
Quoted 26 lines · Stefano Stabellini
Quoted 11 lines · Jan Beulich
Quoted 10 lines · Luca Fancellu

Just to be sure that we are in the same page, are you suggesting to modify the name In this way?

struct gnttab_cache_flush {

(quoted diff) +1 -1
-    union {
+    union xen_gnttab_cache_flush_a {
       uint64_t dev_bus_addr;
       grant_ref_t ref;
   } a;
Quoted 4 lines
Quoted 1 line

Following this kind of pattern: xen_<upper struct name>_<member name> ?

While in general I would be fine with this scheme, for field names like "a" or "u" it doesn't fit well imo.

"a" is a bad name anyway, even for the member. We can take the opportunity to find a better name. Almost anything would be better than "a". Maybe "refaddr"?

Quoted 5 lines
Quoted 2 lines

I'm also unconvinced this would be
scalable to the case where there's further struct/union nesting.

How many of these instances of multilevel nesting do we have? Luca might know. Probably not many? They could be special-cased.

There are not many multilevel nesting instances of anonymous struct/union and the maximum level of nesting I found in the public headers is 2:

union {
	union/struct {
		…
	} <name>
} <name>

I also see that in the majority of cases the unions have not meaningful names like “a” or “u” as member name, instead struct names are fine, It could be fine to keep the meaningful name the same for the struct type name and use the pattern for the non-meaningful ones as long as the names doesn’t create compilation errors?

Example:

struct upper_level {
	union {
		struct {
		
		} meaningful_name1;
		struct {
		
		} meaningful_name2;
	} u;
};

becomes:

Quoted 10 lines
struct upper_level {
	union upper_level_u {
		struct meaningful_name1 {
		
		} meaningful_name1;
		struct meaningful_name2 {
		
		} meaningful_name2;
	} u;
};

As you say - as long as there aren't any compilation issues. Two top level struct-s could have identically named struct/union fields without tag, in which case your approach outlined above will fail. And even if there was no such case right now, the case would need to be covered in whatever naming model was to be used (except, of course, the one without any names).

Jan

#13 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

In reply to Luca Fancellu

Hi Luca,

Quoted 59 lines · Luca Fancellu
Quoted 26 lines · Stefano Stabellini
Quoted 11 lines · Jan Beulich
Quoted 10 lines · Luca Fancellu

Just to be sure that we are in the same page, are you suggesting to modify the name In this way?

struct gnttab_cache_flush {

(quoted diff) +1 -1
-    union {
+    union xen_gnttab_cache_flush_a {
        uint64_t dev_bus_addr;
        grant_ref_t ref;
    } a;
Quoted 4 lines
Quoted 1 line

Following this kind of pattern: xen_<upper struct name>_<member name> ?

While in general I would be fine with this scheme, for field names like "a" or "u" it doesn't fit well imo.

"a" is a bad name anyway, even for the member. We can take the opportunity to find a better name. Almost anything would be better than "a". Maybe "refaddr"?

Quoted 5 lines
Quoted 2 lines

I'm also unconvinced this would be
scalable to the case where there's further struct/union nesting.

How many of these instances of multilevel nesting do we have? Luca might know. Probably not many? They could be special-cased.

There are not many multilevel nesting instances of anonymous struct/union and the maximum level of nesting I found in the public headers is 2:

union {
	union/struct {
		…
	} <name>
} <name>

I also see that in the majority of cases the unions have not meaningful names like “a” or “u” as member name, instead struct names are fine, It could be fine to keep the meaningful name the same for the struct type name and use the pattern for the non-meaningful ones as long as the names doesn’t create compilation errors?

Example:

struct upper_level {
	union {
		struct {
		
		} meaningful_name1;
		struct {
		
		} meaningful_name2;
	} u;
};

becomes:

Quoted 10 lines
struct upper_level {
	union upper_level_u {
		struct meaningful_name1 {
		
		} meaningful_name1;
		struct meaningful_name2 {
		
		} meaningful_name2;
	} u;
};

If I understand correctly your proposal, the name of the structure would be the name of the field. The name of the fields are usually pretty generic so you will likely end up to redefine the structure name.

Unless we want to provide random name, the only safe naming would be to define the structure as upper_level_u_meaningful_name{1, 2}. But, this is going to be pretty awful to read.

But I am still a bit puzzled by the fact doxygen is not capable to deal with anynomous/unamed union. How do other projects deal with them?

Quoted 4 lines

Doing this will help a lot the documentation side because the html page will show the "struct upper_level" with inside the “union upper_level_u" and inside again the two struct “meaningful_name1" and “meaningful_name2", and from the point of view of the developer it can tell her/him exactly the name of the member to access when writing code (apart from the upper_level_u that can be accessed by u, but we can’t clearly change it).

I don't quite understand the last point. Wouldn't the developper see the field name? So how is it going to be different from seeing the structure name?

Quoted 1 line

If this sounds difficult to understand when reading, please generate the documentation and have a look on the page in one side and the source code in another.

Just to be clear, do you mean understanding what you wrote or a developper trying to understand the code?

Cheers,

Signature · 1 line
Julien Grall
#14 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 73 lines · Julien Grall

Hi Luca,

Quoted 63 lines · Luca Fancellu
Quoted 26 lines · Stefano Stabellini
Quoted 11 lines · Jan Beulich
Quoted 10 lines · Luca Fancellu

Just to be sure that we are in the same page, are you suggesting to modify the name In this way?

struct gnttab_cache_flush {
-    union {
+    union xen_gnttab_cache_flush_a {
        uint64_t dev_bus_addr;
        grant_ref_t ref;
    } a;

Quoted 4 lines
Quoted 1 line

Following this kind of pattern: xen_<upper struct name>_<member name> ?

While in general I would be fine with this scheme, for field names like "a" or "u" it doesn't fit well imo.

"a" is a bad name anyway, even for the member. We can take the opportunity to find a better name. Almost anything would be better than "a". Maybe "refaddr"?

Quoted 5 lines
Quoted 2 lines

I'm also unconvinced this would be
scalable to the case where there's further struct/union nesting.

How many of these instances of multilevel nesting do we have? Luca might know. Probably not many? They could be special-cased.

There are not many multilevel nesting instances of anonymous struct/union and the maximum level of nesting I found in the public headers is 2:

union {
    union/struct {
        …
    } <name>
} <name>

I also see that in the majority of cases the unions have not meaningful names like “a” or “u” as member name, instead struct names are fine, It could be fine to keep the meaningful name the same for the struct type name and use the pattern for the non-meaningful ones as long as the names doesn’t create compilation errors?

Example:

struct upper_level {
    union {
        struct {

        } meaningful_name1;
        struct {

        } meaningful_name2;
    } u;
};

becomes:

struct upper_level {
    union upper_level_u {
        struct meaningful_name1 {

Quoted 6 lines

        } meaningful_name1;
        struct meaningful_name2 {

        } meaningful_name2;
    } u;
};

Quoted 5 lines

If I understand correctly your proposal, the name of the structure would be the name of the field. The name of the fields are usually pretty generic so you will likely end up to redefine the structure name.

Unless we want to provide random name, the only safe naming would be to define the structure as upper_level_u_meaningful_name{1, 2}. But, this is going to be pretty awful to read.

But I am still a bit puzzled by the fact doxygen is not capable to deal with anynomous/unamed union. How do other projects deal with them?

While going through the list of anynomous union in Xen, I noticed we also have something like:

struct test {
    union {
        int a;
        int b;
    };
};

We can't name them because of syntactic reasons. What's your plan for them?

Cheers,

Signature · 1 line
Julien Grall
#15 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 84 lines · Julien Grall
Quoted 68 lines · Julien Grall

Hi Luca,

Quoted 66 lines · Luca Fancellu
Quoted 26 lines · Stefano Stabellini
Quoted 11 lines · Jan Beulich
Quoted 10 lines · Luca Fancellu

Just to be sure that we are in the same page, are you suggesting to modify the name In this way?

struct gnttab_cache_flush {

(quoted diff) +1 -1
-    union {
+    union xen_gnttab_cache_flush_a {
        uint64_t dev_bus_addr;
        grant_ref_t ref;
    } a;
Quoted 4 lines
Quoted 1 line

Following this kind of pattern: xen_<upper struct name>_<member name> ?

While in general I would be fine with this scheme, for field names like "a" or "u" it doesn't fit well imo.

"a" is a bad name anyway, even for the member. We can take the opportunity to find a better name. Almost anything would be better than "a". Maybe "refaddr"?

Quoted 5 lines
Quoted 2 lines

I'm also unconvinced this would be
scalable to the case where there's further struct/union nesting.

How many of these instances of multilevel nesting do we have? Luca might know. Probably not many? They could be special-cased.

There are not many multilevel nesting instances of anonymous struct/union and the maximum level of nesting I found in the public headers is 2:

union {
    union/struct {
        …
    } <name>
} <name>

I also see that in the majority of cases the unions have not meaningful names like “a” or “u” as member name, instead struct names are fine, It could be fine to keep the meaningful name the same for the struct type name and use the pattern for the non-meaningful ones as long as the names doesn’t create compilation errors?

Example:

struct upper_level {
    union {
        struct {

        } meaningful_name1;
        struct {

        } meaningful_name2;
    } u;
};

becomes:

struct upper_level {
    union upper_level_u {
        struct meaningful_name1 {

        } meaningful_name1;
        struct meaningful_name2 {
Quoted 6 lines
Quoted 3 lines
        } meaningful_name2;
    } u;
};

If I understand correctly your proposal, the name of the structure would be the name of the field. The name of the fields are usually pretty generic so you will likely end up to redefine the structure name. Unless we want to provide random name, the only safe naming would be to define the structure as upper_level_u_meaningful_name{1, 2}. But, this is going to be pretty awful to read. But I am still a bit puzzled by the fact doxygen is not capable to deal with anynomous/unamed union. How do other projects deal with them?

While going through the list of anynomous union in Xen, I noticed we also have something like:

Quoted 8 lines
struct test {
   union {
       int a;
       int b;
   };
};

We can't name them because of syntactic reasons. What's your plan for them?

I would say that if the fields a and b are not meant to be described in the documentation, they can be hidden, so there is no need to change the structure itself but we might just add some comment containing Doxygen tags for skipping them.
In Zephyr they have some kind of structures like that.

Quoted 5 lines

Cheers,

Signature · 1 line
Julien Grall
#16 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

In reply to Julien Grall
Quoted 72 lines · Julien Grall

Hi Luca,

Quoted 52 lines · Luca Fancellu
Quoted 24 lines · Stefano Stabellini
Quoted 11 lines · Jan Beulich
Quoted 10 lines · Luca Fancellu

Just to be sure that we are in the same page, are you suggesting to modify the name In this way?

struct gnttab_cache_flush {

(quoted diff) +1 -1
-    union {
+    union xen_gnttab_cache_flush_a {
       uint64_t dev_bus_addr;
       grant_ref_t ref;
   } a;
Quoted 4 lines
Quoted 1 line

Following this kind of pattern: xen_<upper struct name>_<member name> ?

While in general I would be fine with this scheme, for field names like "a" or "u" it doesn't fit well imo.

"a" is a bad name anyway, even for the member. We can take the opportunity to find a better name. Almost anything would be better than "a". Maybe "refaddr"?

Quoted 5 lines
Quoted 2 lines

I'm also unconvinced this would be
scalable to the case where there's further struct/union nesting.

How many of these instances of multilevel nesting do we have? Luca might know. Probably not many? They could be special-cased.

There are not many multilevel nesting instances of anonymous struct/union and the maximum level of nesting I found in the public headers is 2:

union {
	union/struct {
		…
	} <name>
} <name>

I also see that in the majority of cases the unions have not meaningful names like “a” or “u” as member name, instead struct names are fine, It could be fine to keep the meaningful name the same for the struct type name and use the pattern for the non-meaningful ones as long as the names doesn’t create compilation errors?
Example:

struct upper_level {
	union {
		struct {
		
		} meaningful_name1;
		struct {
		
		} meaningful_name2;
	} u;
};
becomes:
struct upper_level {
	union upper_level_u {
Quoted 8 lines
		struct meaningful_name1 {
		
		} meaningful_name1;
		struct meaningful_name2 {
		
		} meaningful_name2;
	} u;
};

If I understand correctly your proposal, the name of the structure would be the name of the field. The name of the fields are usually pretty generic so you will likely end up to redefine the structure name.

Unless we want to provide random name, the only safe naming would be to define the structure as upper_level_u_meaningful_name{1, 2}. But, this is going to be pretty awful to read.

Quoted 7 lines

But I am still a bit puzzled by the fact doxygen is not capable to deal with anynomous/unamed union. How do other projects deal with them?

Quoted 3 lines

Doing this will help a lot the documentation side because the html page will show the "struct upper_level" with inside the “union upper_level_u" and inside again the two struct “meaningful_name1" and “meaningful_name2", and from the point of view of the developer it can tell her/him exactly the name of the member to access when writing code (apart from the upper_level_u that can be accessed by u, but we can’t clearly change it).

I don't quite understand the last point. Wouldn't the developper see the field name? So how is it going to be different from seeing the structure name?

The developer, that is using the documentation generated with sphinx+doxygen, will see the structure name and not the field name because this is the way sphinx+doxygen is rendering the code structures. You can see it in the generated documentation using this serie.

Quoted 4 lines
Quoted 2 lines

If this sounds difficult to understand when reading, please generate the documentation and have a look on the page in one side and the source code in another.

Just to be clear, do you mean understanding what you wrote or a developper trying to understand the code?

I meant understanding what I wrote, because I know it’s difficult to describe the concept without visualising the html page, it would have been much easier to attach an image to clarify.

Quoted 5 lines

Cheers,

Signature · 1 line
Julien Grall
#17 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

In reply to Luca Fancellu

Hi,

Quoted 4 lines · Luca Fancellu

Modification to include/public/grant_table.h:

1) Change anonymous structure to be named structure, because doxygen can't deal with them.

What do you mean by can't deal with them? I had a quick try with doxygen build and couldn't find any failure (although I haven't looked at the output).

Cheers,

Signature · 1 line
Julien Grall
#18 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 13 lines · Julien Grall

Hi,

Quoted 3 lines · Luca Fancellu

Modification to include/public/grant_table.h:
1) Change anonymous structure to be named structure, because doxygen can't deal with them.

What do you mean by can't deal with them? I had a quick try with doxygen build and couldn't find any failure (although I haven't looked at the output).

Hi Julien,

The problem is that Doxygen can’t generate proper documentation for anonymous union/structure, it ends up with warning and/or producing wrong documentation like changing names or giving field description to the wrong field.

Cheers,
Luca

Quoted 4 lines

Cheers,

Signature · 1 line
Julien Grall
#19 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Hi Luca,

Quoted 11 lines · Luca Fancellu
Quoted 11 lines · Julien Grall

Hi,

Quoted 3 lines · Luca Fancellu

Modification to include/public/grant_table.h:
1) Change anonymous structure to be named structure, because doxygen can't deal with them.

Quoted 7 lines
Quoted 2 lines

What do you mean by can't deal with them? I had a quick try with doxygen build and couldn't find any failure (although I haven't looked at the output).

Hi Julien,

The problem is that Doxygen can’t generate proper documentation for anonymous union/structure, it ends up with warning and/or producing wrong documentation like changing names or giving field description to the wrong field.

I might do something wrong because I cannot spot any significant difference in the doxygen ouput if I switch back to anonymous union. Would you mind to post more details (such as a diff) on how doxygen doesn't generate properly documentation?

Cheers,

Signature · 1 line
Julien Grall
#20 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 16 lines · Julien Grall

Hi Luca,

Quoted 9 lines · Luca Fancellu
Quoted 9 lines · Julien Grall

Hi,

Quoted 3 lines · Luca Fancellu

Modification to include/public/grant_table.h:
1) Change anonymous structure to be named structure, because doxygen can't deal with them.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

What do you mean by can't deal with them? I had a quick try with doxygen build and couldn't find any failure (although I haven't looked at the output).

Hi Julien,
The problem is that Doxygen can’t generate proper documentation for anonymous union/structure, it ends up with warning and/or producing wrong documentation like changing names or giving field description to the wrong field.

I might do something wrong because I cannot spot any significant difference in the doxygen ouput if I switch back to anonymous union. Would you mind to post more details (such as a diff) on how doxygen doesn't generate properly documentation?

Hi Julien,

Here the explanation of the problem: https://vovkos.github.io/doxyrest/manual/unnamed-structs.html

Clearly the proposed solution is not suitable because they are just hiding the anonymous union/struct field from the documentation.

I did two test:

1) with the anonymous structure:

struct gnttab_cache_flush {

   union {
       uint64_t dev_bus_addr;
       grant_ref_t ref;

} a;

I get a warning from sphinx (because the XML output of Doxygen is not in a good shape) when I generate the documentation, here follow the output:

$ make -C docs XEN_TARGET_ARCH="arm64" sphinx-html
make: Entering directory '/home/user/prj_xen/xen/docs'
Generating xen.doxyfile
mv xen.doxyfile.tmp xen.doxyfile
Generating doxygen_input.h
/usr/bin/doxygen xen.doxyfile
Generating hypercall-interfaces/index.rst
XEN_ROOT=/home/user/prj_xen/xen /usr/bin/sphinx-build -b html . sphinx/html
Running Sphinx v1.6.7
making output directory...
loading pickled environment... not yet created
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 21 source files that are out of date
updating environment: 21 added, 0 changed, 0 removed
reading sources... [100%] misc/xen-makefiles/makefiles
/home/user/prj_xen/xen/docs/hypercall-interfaces/arm64/grant_tables.rst:6: WARNING: Invalid definition: Expected end of definition. [error at 18]
  gnttab_cache_flush.a
  ------------------^
looking for now-outdated files... none found
pickling environment... done
checking consistency... /home/user/prj_xen/xen/docs/hypercall-interfaces/arm32.rst: WARNING: document isn't included in any toctree
/home/user/prj_xen/xen/docs/hypercall-interfaces/x86_64.rst: WARNING: document isn't included in any toctree
/home/user/prj_xen/xen/docs/misc/kconfig.rst: WARNING: document isn't included in any toctree
/home/user/prj_xen/xen/docs/misc/kconfig-language.rst: WARNING: document isn't included in any toctree
/home/user/prj_xen/xen/docs/misc/kconfig-macro-language.rst: WARNING: document isn't included in any toctree
/home/user/prj_xen/xen/docs/misc/xen-makefiles/makefiles.rst: WARNING: document isn't included in any toctree
done
preparing documents... done
writing output... [100%] misc/xen-makefiles/makefiles
generating indices... genindex
writing additional pages... search
copying images... [100%] admin-guide/xen-overview.drawio.svg
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded, 7 warnings.
make: Leaving directory '/home/user/prj_xen/xen/docs’

And checking the generated html page html/hypercall-interfaces/arm64/grant_tables.html you can see that there is a union without name or fields right above "union __guest_handle_gnttab_cache_flush_t".

2) without the anonymous structure:

struct gnttab_cache_flush {
- union {

+    union a {
       uint64_t dev_bus_addr;
       grant_ref_t ref;

} a;

I don’t get the warning from sphinx and looking in the html/hypercall-interfaces/arm64/grant_tables.html page I can see the proper documentation for the union a right above "union __guest_handle_gnttab_cache_flush_t”.

Let me know if you get different results.

Cheers,
Luca

Quoted 4 lines

Cheers,

Signature · 1 line
Julien Grall
#21 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

In reply to Luca Fancellu

Luca Fancellu writes ("Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h"):

Quoted 2 lines

The problem is that Doxygen can’t generate proper documentation for anonymous union/structure, it ends up with warning and/or producing wrong documentation like changing names or giving field description to the wrong field.

This does not seem like it would be an impossibly hard feature to add to doxygen.

Ian.

#22 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Hi Ian,

Quoted 8 lines · Ian Jackson

Luca Fancellu writes ("Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h"):

Quoted 2 lines

The problem is that Doxygen can’t generate proper documentation for anonymous union/structure, it ends up with warning and/or producing wrong documentation like changing names or giving field description to the wrong field.

This does not seem like it would be an impossibly hard feature to add to doxygen.

Modifying doxygen is not really in our planned efforts and if someone does it that would put an hard constraint on the version of doxygen possible to use.

But is adding names to anonymous elements really an issue here ?

If we agree on names or on a convention for name the result will not impact the code or backward compatibility.

Regards
Bertrand

Quoted 2 lines

Ian.

#23 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Quoted 6 lines · Bertrand Marquis
Quoted 6 lines · Ian Jackson

Luca Fancellu writes ("Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h"):

Quoted 2 lines

The problem is that Doxygen can’t generate proper documentation for anonymous union/structure, it ends up with warning and/or producing wrong documentation like changing names or giving field description to the wrong field.

Quoted 6 lines
Quoted 2 lines

This does not seem like it would be an impossibly hard feature to add to doxygen.

Modifying doxygen is not really in our planned efforts and if someone does it that would put an hard constraint on the version of doxygen possible to use.

But is adding names to anonymous elements really an issue here ?

It's clutter in the code base, making things harder to read (even if just slightly). It's certainly odd to make such source changes just for a doc tool. If changing doxygen is not an option for you, how about pre-processing the header and inserting the names the tool wants, before handing the result as input to it?

Jan

#24 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Hi,

Quoted 14 lines · Jan Beulich
Quoted 11 lines · Bertrand Marquis
Quoted 8 lines · Ian Jackson

Luca Fancellu writes ("Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h"):

Quoted 2 lines

The problem is that Doxygen can’t generate proper documentation for anonymous union/structure, it ends up with warning and/or producing wrong documentation like changing names or giving field description to the wrong field.

This does not seem like it would be an impossibly hard feature to add to doxygen.

Modifying doxygen is not really in our planned efforts and if someone does it that would put an hard constraint on the version of doxygen possible to use.

Quoted 7 lines
Quoted 1 line

But is adding names to anonymous elements really an issue here ?

It's clutter in the code base, making things harder to read (even if just slightly). It's certainly odd to make such source changes just for a doc tool. If changing doxygen is not an option for you, how about pre-processing the header and inserting the names the tool wants, before handing the result as input to it?

Introducing a new tool or find a pre-processing solution is for sure possible but will be more complex and less error prone.

Also as said in my mail to Julien, passing this issue of anonymous entries now might just push the problem as we might have to solve it later if we want to become MISRA compliant (which is also something we are looking in FuSa).

Regards
Bertrand

Quoted 3 lines

Jan

#25 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

In reply to Bertrand Marquis
Quoted 4 lines · Bertrand Marquis

Hi Ian,

Quoted 2 lines · Ian Jackson
Quoted 8 lines
Quoted 6 lines

Luca Fancellu writes ("Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h"):

Quoted 2 lines

The problem is that Doxygen can’t generate proper documentation for anonymous union/structure, it ends up with warning and/or producing wrong documentation like changing names or giving field description to the wrong field.

This does not seem like it would be an impossibly hard feature to add to doxygen.

Modifying doxygen is not really in our planned efforts and if someone does it that would put an hard constraint on the version of doxygen possible to use.

Are you saying that anyone who want to use doxygen has to waive off the use of anonymous union/struct? Is it the only thing doxygen can't deal with?

Quoted 2 lines

But is adding names to anonymous elements really an issue here ?
If we agree on names or on a convention for name the result will not impact the code or backward compatibility.

I think the naming is only the tip of the problem. One advantage of anymous union/struct is you make clear they are not meant to be used outside of the context. So they should mostly be seen as an easy way to access some part of the "parent" structure directly. Therefore, IMHO, they don't deserve to be documented separately.

In fact, this is the first thing I noticed when building the documentation because 'union a' was in global index.

Cheers,

Signature · 1 line
Julien Grall
#26 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Hi Julien,

Quoted 9 lines · Julien Grall
Quoted 4 lines · Bertrand Marquis

Hi Ian,

Quoted 3 lines · Ian Jackson

Luca Fancellu writes ("Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h"):

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

The problem is that Doxygen can’t generate proper documentation for anonymous union/structure, it ends up with warning and/or producing wrong documentation like changing names or giving field description to the wrong field.

This does not seem like it would be an impossibly hard feature to add to doxygen.

Modifying doxygen is not really in our planned efforts and if someone does it that would put an hard constraint on the version of doxygen possible to use.

Are you saying that anyone who want to use doxygen has to waive off the use of anonymous union/struct? Is it the only thing doxygen can't deal with?

That is the main one we came into while doing this but there might be other going forward, hard to be sure at this stage.

Quoted 5 lines
Quoted 3 lines

But is adding names to anonymous elements really an issue here ?
If we agree on names or on a convention for name the result will not impact the code or backward compatibility.

I think the naming is only the tip of the problem. One advantage of anymous union/struct is you make clear they are not meant to be used outside of the context. So they should mostly be seen as an easy way to access some part of the "parent" structure directly. Therefore, IMHO, they don't deserve to be documented separately.

Somehow in the documentation when you have a union you will need to document that it is a union and the possible entries. Having a name to refer to it sounds to me a lot easier than making it anonymous.

One way or an other most standards like MISRA are forbidding anonymous entries as they cannot be referred to.

Quoted 2 lines

In fact, this is the first thing I noticed when building the documentation because 'union a' was in global index.

Definitely I agree the “a” is not a good solution and we need to find meaningful names.
But this is in fact true for the sub-element in the structure (from which the name was taken), using “a” as an identifier is not really explanatory of what that is. “u” for union can be see as a standard though.

This is why i think we should put names which a meaning but this is not always easy to find.

Cheers
Bertrand

Quoted 5 lines

Cheers,

Signature · 1 line
Julien Grall
#27 lore raw

Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h

Bertrand Marquis writes ("Re: [PATCH 3/3] docs/doxygen: doxygen documentation for grant_table.h"):

Quoted 1 line

Somehow in the documentation when you have a union you will need to document that it is a union and the possible entries.

I would expect to find the documentation for an anonymous struct or union folded into the documentation of the containing structure, just as it is in the source.

Quoted 1 line

One way or an other most standards like MISRA are forbidding anonymous entries as they cannot be referred to.

An anonymous union or struct like this is always the type of a single field in a containing aggegate type. So if one needs to speak of it, one can specify the container's type and the field. So it *can* be named.

I am assuming we don't have *unused* anonymous structs and unions.

Ian.

#28 lore raw
In reply to Luca Fancellu
Quoted 17 lines · Luca Fancellu

This serie introduce doxygen in the sphinx html docs generation. One benefit is to keep most of the documentation in the source files of xen so that it's more maintainable, on the other hand there are some limitation of doxygen that must be addressed modifying the current codebase (for example doxygen can't parse anonymous structure/union).

To reproduce the documentation xen must be compiled because most of the headers are generated on compilation time from the makefiles.

Here follows the steps to generate the sphinx html docs, some package may be required on your machine, everything is suggested by the autoconf script.
Here I'm building the arm64 docs (the only introduced for now by this serie):

Quoted 7 lines

./configure
make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" make -C docs XEN_TARGET_ARCH="arm64" sphinx-html

now in docs/sphinx/html/ we have the generated docs starting from the index.html page.

Do you have a sample rendered output?

The plan was to try and use Linux's kernel-doc plugin for Sphinx, which is very doxygen-like.  Did you experiment with this option?

~Andrew

#29 lore raw
Quoted 23 lines · Andrew Cooper
Quoted 22 lines · Luca Fancellu

This serie introduce doxygen in the sphinx html docs generation. One benefit is to keep most of the documentation in the source files of xen so that it's more maintainable, on the other hand there are some limitation of doxygen that must be addressed modifying the current codebase (for example doxygen can't parse anonymous structure/union).

To reproduce the documentation xen must be compiled because most of the headers are generated on compilation time from the makefiles.

Here follows the steps to generate the sphinx html docs, some package may be required on your machine, everything is suggested by the autoconf script.
Here I'm building the arm64 docs (the only introduced for now by this serie):

./configure
make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" make -C docs XEN_TARGET_ARCH="arm64" sphinx-html

Quoted 7 lines
Quoted 2 lines

now in docs/sphinx/html/ we have the generated docs starting from the index.html page.

Do you have a sample rendered output?

The plan was to try and use Linux's kernel-doc plugin for Sphinx, which is very doxygen-like.  Did you experiment with this option?

As you probably know the end goal for Luca (and the Xen FuSa SIG as a whole) is to generate all FuSa documents, including requirements docs, interface docs, etc.

FuSa requires us to follow the famous V model, where the high level requirements are linked to the lower level requirements, which are linked to the interface docs, which are linked all the way down to the code.

Maintaining the linking is difficult and typically done with expensive proprietary FuSa tools.

Fortunately, an engineer working with the Zephyr project developed a set of scripts for Doxygen that are able to generate the required FuSa docs and also links from in-code comments and markdown/rst docs in the tree.

This is great work, and in the FuSa SIG we thought it would be best to align ourselves with Zephyr to be able to pull our efforts together on the tooling side instead of doing the same thing again on our own.

This is the reason why we ended up with Doxygen.

#30 lore raw
Quoted 46 lines · Stefano Stabellini
Quoted 23 lines · Andrew Cooper
Quoted 22 lines · Luca Fancellu

This serie introduce doxygen in the sphinx html docs generation. One benefit is to keep most of the documentation in the source files of xen so that it's more maintainable, on the other hand there are some limitation of doxygen that must be addressed modifying the current codebase (for example doxygen can't parse anonymous structure/union).

To reproduce the documentation xen must be compiled because most of the headers are generated on compilation time from the makefiles.

Here follows the steps to generate the sphinx html docs, some package may be required on your machine, everything is suggested by the autoconf script.
Here I'm building the arm64 docs (the only introduced for now by this serie):

./configure
make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" make -C docs XEN_TARGET_ARCH="arm64" sphinx-html

Quoted 6 lines
Quoted 2 lines

now in docs/sphinx/html/ we have the generated docs starting from the index.html page.

Do you have a sample rendered output?

The plan was to try and use Linux's kernel-doc plugin for Sphinx, which is very doxygen-like.  Did you experiment with this option?

As you probably know the end goal for Luca (and the Xen FuSa SIG as a whole) is to generate all FuSa documents, including requirements docs, interface docs, etc.

FuSa requires us to follow the famous V model, where the high level requirements are linked to the lower level requirements, which are linked to the interface docs, which are linked all the way down to the code.

Maintaining the linking is difficult and typically done with expensive proprietary FuSa tools.

Fortunately, an engineer working with the Zephyr project developed a set of scripts for Doxygen that are able to generate the required FuSa docs and also links from in-code comments and markdown/rst docs in the tree.

Quoted 5 lines

This is great work, and in the FuSa SIG we thought it would be best to align ourselves with Zephyr to be able to pull our efforts together on the tooling side instead of doing the same thing again on our own.

This is the reason why we ended up with Doxygen.

So are we're saying that Doxygen is a hard dependency because there is an extension for Doxygen to generate other safety docs?

~Andrew

#31 lore raw

Hi Andrew,

Quoted 49 lines · Andrew Cooper
Quoted 46 lines · Stefano Stabellini
Quoted 23 lines · Andrew Cooper
Quoted 22 lines · Luca Fancellu

This serie introduce doxygen in the sphinx html docs generation. One benefit is to keep most of the documentation in the source files of xen so that it's more maintainable, on the other hand there are some limitation of doxygen that must be addressed modifying the current codebase (for example doxygen can't parse anonymous structure/union).

To reproduce the documentation xen must be compiled because most of the headers are generated on compilation time from the makefiles.

Here follows the steps to generate the sphinx html docs, some package may be required on your machine, everything is suggested by the autoconf script.
Here I'm building the arm64 docs (the only introduced for now by this serie):

./configure
make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" make -C docs XEN_TARGET_ARCH="arm64" sphinx-html

Quoted 6 lines
Quoted 2 lines

now in docs/sphinx/html/ we have the generated docs starting from the index.html page.

Do you have a sample rendered output?

The plan was to try and use Linux's kernel-doc plugin for Sphinx, which is very doxygen-like. Did you experiment with this option?

As you probably know the end goal for Luca (and the Xen FuSa SIG as a whole) is to generate all FuSa documents, including requirements docs, interface docs, etc.

FuSa requires us to follow the famous V model, where the high level requirements are linked to the lower level requirements, which are linked to the interface docs, which are linked all the way down to the code.

Maintaining the linking is difficult and typically done with expensive proprietary FuSa tools.

Fortunately, an engineer working with the Zephyr project developed a set of scripts for Doxygen that are able to generate the required FuSa docs and also links from in-code comments and markdown/rst docs in the tree.

Quoted 8 lines
Quoted 5 lines

This is great work, and in the FuSa SIG we thought it would be best to align ourselves with Zephyr to be able to pull our efforts together on the tooling side instead of doing the same thing again on our own.

This is the reason why we ended up with Doxygen.

So are we're saying that Doxygen is a hard dependency because there is an extension for Doxygen to generate other safety docs?

hard no as we could find other solution but the current strategy we take together with Zephyr project is based on using Doxygen for requirement linking.

Also an other argument is that the documentation generated is actually nice and that could also be useful for developers and users (see [1] for Zephyr doc).

Our global idea is also that using doxygen we can extract a big part of the documentation (which will partly be used as certification) directly from the code which will make it a lot easier for developers to maintain.

Regards
Bertrand

[1] https://docs.zephyrproject.org/latest/

Quoted 3 lines

~Andrew

#32 lore raw
In reply to Luca Fancellu
Quoted 17 lines · Luca Fancellu

This serie introduce doxygen in the sphinx html docs generation. One benefit is to keep most of the documentation in the source files of xen so that it's more maintainable, on the other hand there are some limitation of doxygen that must be addressed modifying the current codebase (for example doxygen can't parse anonymous structure/union).

To reproduce the documentation xen must be compiled because most of the headers are generated on compilation time from the makefiles.

Here follows the steps to generate the sphinx html docs, some package may be required on your machine, everything is suggested by the autoconf script.
Here I'm building the arm64 docs (the only introduced for now by this serie):

Quoted 4 lines

./configure
make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" make -C docs XEN_TARGET_ARCH="arm64" sphinx-html

I have tried this instruction and got:

make: Entering directory '/home/ANT.AMAZON.COM/jgrall/works/oss/xen/docs' xen.doxyfile
make: xen.doxyfile: Command not found
Makefile:67: recipe for target 'sphinx-html' failed
make: *** [sphinx-html] Error 127
make: Leaving directory '/home/ANT.AMAZON.COM/jgrall/works/oss/xen/docs'

AFAICT, $DOXYGEN will only get defined when sphinx-build is installed.
When sphinx-build is not installed SPHINXBUILD will be equal to 'no', but the Makefile check for emptiness.

Cheers,

Signature · 1 line
Julien Grall
#33 lore raw
Quoted 35 lines · Julien Grall
Quoted 10 lines · Luca Fancellu

This serie introduce doxygen in the sphinx html docs generation. One benefit is to keep most of the documentation in the source files of xen so that it's more maintainable, on the other hand there are some limitation of doxygen that must be addressed modifying the current codebase (for example doxygen can't parse anonymous structure/union).
To reproduce the documentation xen must be compiled because most of the headers are generated on compilation time from the makefiles.
Here follows the steps to generate the sphinx html docs, some

Quoted 8 lines

package may be required on your machine, everything is suggested
by the autoconf script.
Here I'm building the arm64 docs (the only introduced for now by this serie):
./configure
make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" menuconfig make -C xen XEN_TARGET_ARCH="arm64" CROSS_COMPILE="aarch64-linux-gnu-" make -C docs XEN_TARGET_ARCH="arm64" sphinx-html

I have tried this instruction and got:

make: Entering directory '/home/ANT.AMAZON.COM/jgrall/works/oss/xen/docs' xen.doxyfile
make: xen.doxyfile: Command not found
Makefile:67: recipe for target 'sphinx-html' failed
make: *** [sphinx-html] Error 127
make: Leaving directory '/home/ANT.AMAZON.COM/jgrall/works/oss/xen/docs'

Quoted 3 lines

AFAICT, $DOXYGEN will only get defined when sphinx-build is installed.
When sphinx-build is not installed SPHINXBUILD will be equal to 'no', but the Makefile check for emptiness.

Hi Julien,

Thank you for spotting it, I’ll fix it in the v2 patch

Cheers,
Luca

Quoted 4 lines

Cheers,

Signature · 1 line
Julien Grall