Linux man-pages Makefile portability

Hi Ingo,

Would you mind checking if the current Makefile works on your OpenBSD?
I think it depends on gmake(1), but I'd like to state that in the dependencies (which I need to write in a new INSTALL file). Maybe, if it's not too hard, we can tweak the Makefile to make it work with make(1POSIX).

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/>

Thanks,

Alex

Alejandro Colomar
<http://www.alejandro-colomar.es/>

Hi Alejandro,

Alejandro Colomar wrote on Sun, Jun 19, 2022 at 07:52:33PM +0200:

After doing

   $ git clone https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/
   $ cd man-pages

i have a hard time finding any targets in the ./Makefile that i could possibly run for testing...

I'm certainly not going to run targets like "all" or "install". I don't think testing the "dist" target makes much sense because nobody is going to build your distribution tarballs on OpenBSD. I'm not interested in testing the "src" group of targets; i think we discussed in the past that i dislike that whole idea.

The html target could potentially be interesting, but
https://savannah.nongnu.org/projects/man2html/
and the repo linked to from there,
https://cvs.savannah.nongnu.org/viewvc/man2html/
gives me the impression this is abandonware that did not receive any maintenance in 19 years. Besides, an OpenBSD port does not currently exist and i don't feel like creating one just for testing.

The only remaining target i can readily see that might be fit for testing is "lint", which does seem quite interesting indeed.

But look at this:

   $ make lint
   $

It does absolutely nothing.

Running "make -d g1 lint" to show the input graph tells me there is a rule

lint:
@:

so i'm not surprised it does nothing.

Next, i tried

   $ make lint-mandoc
  make: don't know how to make lint-mandoc
  Stop in /co/man-pages

The target simply doesn't exist.

My impression is your Makefile is full of POSIX violations and utterly non-portable and gmake(1)-only. But that's probably OK. I mean, the purpose of your project is to document parts of Linux. So requiring a tool that almost all Linux users have readily available and that users of other systems can install it they want to does not seem unreasonable to me. (It's a completely different story for a project like GNU troff that is about the opposite of Linux only and that basically all UNIX systems want to be able to use in production.)

If i use GNU make instead of our native make, i do get output that might be potentially meaningful, even on OpenBSD-current:

   $ gmake lint-mandoc
  INSTALL tmp/lint/
  INSTALL tmp/lint/man0/
  LINT (mandoc)   tmp/lint/man0/sysexits.h.0.lint.mandoc.touch
  INSTALL tmp/lint/man1/
  LINT (mandoc)   tmp/lint/man1/getent.1.lint.mandoc.touch
  LINT (mandoc)   tmp/lint/man1/iconv.1.lint.mandoc.touch
  LINT (mandoc)   tmp/lint/man1/intro.1.lint.mandoc.touch
  LINT (mandoc)   tmp/lint/man1/ldd.1.lint.mandoc.touch
  LINT (mandoc)   tmp/lint/man1/locale.1.lint.mandoc.touch
  LINT (mandoc)   tmp/lint/man1/localedef.1.lint.mandoc.touch
  LINT (mandoc)   tmp/lint/man1/memusage.1.lint.mandoc.touch
  mandoc: man1/memusage.1:187:2: WARNING: empty block: UR
  gmake: *** [Makefile:468: tmp/lint/man1/memusage.1.lint.mandoc.touch] Error 2

That makes sense to me given that i also get

   $ mandoc -T lint man1/memusage.1                               
  mandoc: man1/memusage.1:187:2: WARNING: empty block: UR

when running the mandoc linter manually.

That sounds reasonable to me.

Yes, i think if you want people to test and use that Makefile, then documentating in some way what it is supposed to do and how it is supposed to be used would make sense. Sorry if i somehow missed the documentation, i looked at README and MAINTAINER_NOTES to no avail.

By the way, this surprised me in the README file:

Consider using multiple threads (at least 2) when installing these man pages, as the Makefile is optimized for multiple threads:
"make -j install".

I did not do measurements, but are you sure this is sane advice? For an installation procedure, i would have expected execution speed to be limited by disk I/O and i would have expected CPU consumption to be negligible. Did you do measurements to confirm the relevance of this advice?

Finally, i don't think it is particularly good style that "make all" implies "make install", even if it is clearly documented, which it is. That seems very unusual and not at all what people would expect. Usually, "make all" does specifically *not* modify the production parts of the system but only *builds* the software such that it can later be installed with a separate command. Arguably, having "make all" imply "make uninstall" is even worse. I don't think anybody would expect that running "make all" would attempt to delete stuff from their production system!

I see lots and lots of syntax in the Makefile where i don't even know what it means. I think if you want to make it POSIX compatible, that would likely amount to throwing it all away and starting over with a complete rewrite from scratch. It doesn't look like there is much you would be able to keep.

There is yet another reason why i don't consider it a serious issue if the Makefile is for GNU make(1) only. It is not a rare occurence that i use your work - i have repeatedly looked at it when working on stuff in OpenBSD to find out how Linux defines some features for comparison. But i never even noticed there is a Makefile in your distribution. Commands like

   $ man -M /co/man-pages open

work perfectly fine on my system to view the Linux open(2) manual, nicely formatted, with no need for installation or a Makefile. Even when i put up a copy at

https://man.bsd.lv/Linux-5.13/open

just copying the eight man* directories and running the makewhatis(8) command distributed with mandoc is completely sufficient, no further steps are needed for installation, not even on the production web server.

It seems likely to me that other non-Linux users interested in your work don't need the Makefile either.

Yours,
Ingo

Hi Ingo,

Thanks for trying!

Sorry, I need to document the targets.

The only interesting targets for users are 'dist', 'install', and uninstall'. You could test the install target with DESTDIR=/tmp/somedir

You don't even need to be root to test that in /tmp.

<https://www.gnu.org/prep/standards/html_node/DESTDIR.html>

Makes sense.

I remember.

I used that because the archaic Makefile present in the tree when I joined was using it (and I didn't know any other tools):

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/Makefile?id=b32a6935592118ce4acfddf95e52fca8fbd3aded>

But that Makefile was clearly unused since no-one knows.

If you know a better tool, I could start using it. Maybe I could use groff(1) directly, with grohtml(1).

Thanks :-)

I guess it's due to the use of $(foreach ). I guess it's a GNU extension and make(1POSIX) ignores that creating an empty string. Since the Makefile uses a lot of functions[1], I guess it's not easy to make it portable.

The lint part is mostly implemented between lines 432 and 486.

[1]: addsuffix, wildcard, foreach, filter, patsubst, sort, shell, basename, notdir, info. Not sure how many of those are supported by your make(1); maybe none?

Yeah, I guess that it's fine, since the extensions are useful, and they are virtually everywhere in Linux.

I was wondering if I could make it portable with little changes, but it seems not, and not worth even trying.

Yes, although they could definitely simplify the build system a little bit. It's one of the most complex builds I've ever seen.

No, you didn't. I expected autocomplete to help, and targets to be obvious in what they mean (as I followed GNU conventions as close as possible so that they would be intuitive), but if your make doesn't understand my Makefile, autocomplete is obviously not going to help.

Sorry :/

Yes, I did:

[

$ sudo make uninstall >/dev/null
$ time sudo make install >/dev/null

real 0m7.722s
user 0m0.004s
sys 0m0.004s

$ sudo make uninstall >/dev/null
$ time sudo make -j2 install >/dev/null

real 0m4.634s
user 0m0.005s
sys 0m0.004s

$ sudo make uninstall >/dev/null
$ time sudo make -j install >/dev/null

real 0m3.170s
user 0m0.000s
sys 0m0.009s

$ touch man2/membarrier.2
$ time sudo make install >/dev/null

real 0m1.766s
user 0m0.005s
sys 0m0.004s

$ touch man2/membarrier.2
$ time sudo make -j2 install >/dev/null

real 0m1.735s
user 0m0.004s
sys 0m0.004s

$ touch man2/membarrier.2
$ time sudo make -j install >/dev/null

real 0m1.737s
user 0m0.001s
sys 0m0.008s
]

You can see that just for the $(shell ...) and all the text functions that process the filenames to be installed it is taking almost 1.7 s, even if it only needs to update a single page. That's a lot, but it has it's advantages (generating the file list on-the-fly; no ./configure).

Then, the actual installation of the ~2.5k pages (most of them are link pages), takes another 1.4 s in multi-process mode, and 6 s in single-process mode (so at least 4.6 s that are not I/O). Maybe it's make(1) that has a hard time traversing the tree... I don't know where the bottleneck is, but it's clearly there.

I completely agree. That predates me, and I wanted to change it for a long time, but I was worried that I might be breaking others' stuff.

But, since you called it, I feel supported to remove that behavior.

I guess that the Makefile before I touched is was so basic that you could only uninstall everything or install everything, whoever wrote that in the Makefile wanted to just type those two with just one command:

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/Makefile?id=b32a6935592118ce4acfddf95e52fca8fbd3aded>

But that Makefile had one good thing: it was very readable, especially compared to mine; but it was no more than a couple of sh(1) scripts embedded in it.

Nah, I think I'll keep it. Although if I'm shown a simpler POSIX Makefile that achieves the same behavior, I'd consider it.

Thanks! I do check yours too :)

BTW, did you check the changes to queue.3? I guess you could improve yours in a similar manner.

<https://linux-man-pages.blogspot.com/2020/11/man-pages-509-is-released.html>

Also, if you have been following the addition of pages about types, and would like to comment, you'll be welcome!

<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=178eaf37e2e971cae88bd4d3f124ede0afbb1015>

Heh!

How do you generate your HTML pages? mandoc(1)? They are nice.

Makes sense. I'm happy that I didn't break anything by making heavy use of GNU extensions.

Cheers,
Alex

Alejandro Colomar
<http://www.alejandro-colomar.es/>

Hi Alejandro,

Alejandro Colomar wrote on Mon, Jun 20, 2022 at 12:23:45AM +0200:

I see, now it makes sense to me.

I'm relieved i won't have to dig into man2html. :-)

I'm not saying "noone knows"; some packaging tools on some Linux distributions might very well rely on the Makefile - i don't know. All i'm saying is i don't readily see a reason why people *not* running Linux might need the Makefile...

I don't think using groff to generate HTML output from manual pages is a good idea. It generates low-quality HTML code, and it is impossible to fix because of the basic concept how groff works.

You see, both mdoc(7) and modern HTML are semantic markup languages. Groff, however, does the transformation in a two-step process. In the first step, it throws away *all* the semantic information, retaining only *physical* formatting cues. And then in the second step, it tries to generate HTML from those physical formatting instructions. That cannot possibly work. At best it might generate early-1990ies-style HTML 1 or HTML 2 code with no support for CSS whatsoever.
That's an unavoidable consequence of the basic design of grohtml(1).

For man(7), the problem is not quite as bad as for mdoc(7) because the man(7) language is a 1970ies-style, purely presentational markup language that conveys no semantic information in the first place, so there isn't really much that could get lost during the intermediate step. Then again, i think nowadays, it makes more sense to use a converter that can handle *both* languages well instead of one that is kind of nearly good enough for man(7) but really inadequate for mdoc(7).

[...]

[...]

The OpenBSD implementation of make(1) is much more powerful than POSIX make, but according to the manual page, you are right that none of these keywords are supported by OpenBSD make, let alone by POSIX make.

I wouldn't go that far; i guess then you never looked at build systems of large software like X11 or firefox or boost... Even the OpenBSD ports build framework is more complicated than the groff build system. Then again, the groff build system *does* have a lot of abstraction, indirection, and complexity *for such a small piece of software*. :-/ But we're disgressing.

[ ... regarding installation instructions ...]

I almost never use autocomplete except for the names of commands that are installed system-wide and for the names of files in the local file system. I didn't even know it is possible to use autocompletion for make targets, and i dislike the idea. But don't worry! Your build system *is* complicated for a package that actually doesn't need to build anything, but not so bad that i didn't find what i looked for. :)

As a side remark, i consider it bad style to use dependencies during
installation: dependencies are for the build stage, not for the installation stage. When i say "make install", i just want *all* the files installed unconditionally for two reasons: On the one hand, dependency handling is error-prone and it would be bad if some file does not get installed due to the notorious problem of oversights in dependency handling (and dependency handling in parallel Makefiles is even more fragile than in serial ones). On the other hand, "make install" also has the purpose of repairing an installation that got broken in some way or other, and skipping some files because the build system *thinks* they are probably still installed properly defeats the purpose IMHO.

As another aside, i consider using .so bad style. It is unnecessarily fragile. Using hard links on the file system level (see ln(1)) is significantly more robust. With mandoc(1), you don't need links at all, but i admit traditional man(1) implementations including man-db still require them for manual pages having more than one name.

I see. So you need multiple processors purely for dealing with make(1) overhead... Gee! :-/

[...]

I think in OpenBSD, these changes would get vetoed by large numbers of developers because they violate the way OpenBSD manual pages are organized in several ways:

1. Your queue(7) manual page is placed in the wrong section. It is purely about an API provided by a library for the C language. Such information unambiguously belongs in section 3 and certainly not in section 7. It is not even an edge case; it is perfectly clear what the correct section is.

2. Your file names and .TH names violate the OpenBSD convention that section 2 and 3 manual pages must be named after functions or macros. For example, the page name "slist" is not acceptable because no sname() function or macro exists.

3. Splitting the page up into multiple pages is a bad idea for two reasons: it results in significant duplication of information and it splits information about interfaces so closely related to each other that most of their features are identical across multiple pages.

You would have no chance of getting anything like that committed to OpenBSD.

BSD doesn't have manual pages about types, and i don't think there is a significant benefit from having them. Most standard types are trivial and can easily be looked up in the header files with no need for separate documentation. In the unusual case that a type has non-trivial syntax and/or semantics, it can be documented in the manual page of the most closely related API function; for example, "struct pollfd" is documented in our poll(2) page. That makes it easy to find with the usual "man -k Vt=typename" search command.

Documenting a non-trivial type separately from the functions using it is counter-productive and makes the documentation hard to read because programmers *never* need to use a type defined by a library unless they also want to use related API functions. Actually, i find it better to *not* add type names as names of manual pages because that way, the classical syntax "man functname" can be used to search for function and macro names and the advanced syntax "man -k Vt=typename" to search for types, with less potential for confusion.

So in OpenBSD, your pages about types would get vetoed on the grounds of "pages not named after functions or macros" as well as on the grounds of "these pages do not document any function or macro; instead of creating a new page, put the information where it belongs."

That said, other projects are of course free to have such pages if they want to. The mandoc(1) program is also able to handle paths like "man3/id_t.3type". It will consider that page to be *both* in section "3" (as specified by the directory name) and in section "3type" (as specified by the file name and by the .TH macro). I would consider it better style to keep section names consistent, i.e. to use either "man3/id_t.3" .TH id_t 3 or "man3type/id_t.3type" .TH id_t 3type, but it's not a big deal: since many systems (in particular various Linux distros) suffer from such inconsistencies, handling such inconsistencies gracefully is an important feature that certainly won't get removed.

https://man.openbsd.org/mandoc.1#HTML_Output
https://man.openbsd.org/man.cgi.8

I think that's the usual way to generate HTML from manual pages nowadays. The following sites also use mandoc for HTML output:

* https://www.freebsd.org/cgi/man.cgi
* https://manpages.debian.org/
* https://man.archlinux.org/
* https://man.voidlinux.org/

Some of these have their own CGI handling and/or database code, but they all use the mandoc parser and HTML generator.

Yours,
Ingo

[added Branden, as he was involved in discussions regarding man3type;
Branden, you might want to visit this thread from the begining, as I only copied the minimum to reply; it's in linux-man@]

Hi, Ingo! and Branden!

Oh, Michael seemed a bit surprised that I started patching the Makefile, as if the features I was patching hadn't been used in a very long time.

Anyway, I fixed make all. I never liked it. Now it builds all that it can build, which now is HTML pages, and in the future will probably include PDF pages too. It was that, or make it a no-op, so I thought HTML+PDF was more useful.

Well, the Makefile is basically meant to install (copy) the files to the system, so since you copy them to somewhere, gmake install mandir=... should work, but cp(1) -r tends to be as useful for such a simple package.

[...]

Hmm, I'll try both and see. Thanks.

Heh, then to be compatible with BSD make(1) I guess I'd have to hardcode the page names, or use suffix rules, none of which convinces me.

Maybe suffix rules could work, but I'd have to stop testing the EXAMPLES, which is the most complex part of the Makefile.

I'll consider using them, or at least Substitution References[1] if it adds compatibility, at least for the simplest tasks, such as make install.

[1]: <https://www.gnu.org/software/make/manual/html_node/Substitution-Refs.html#Substitution-Refs>

Oh, I use autocomplete everyday for things like arguments to git commands, and I really feel it when I'm in a system where I don't have such help.

:)

I've had doubts about that, and in the past I tended to do the same as you suggest, not because of fear to broken deps, but for making sure I don't create temporary files owned by root. But in this case, where there are thousands of files to install, there's an important time difference between installing just the diff and installing the whole repo, so I asked the following question[2] just to confirm my doubts, and added the deps.

Regarding the possibility of broken deps, I believe the solution is to fix the Makefile, not to assume that it can't be done right and make it dumb; and I try hard to make sure my Makefiles work in multi-process mode. There's always a chance that I got some corner case wrong, but this case it's pretty low (and if someone doesn't trust my Makefiles to behave well with -j, I don't force to use it, but I recommend it very much :)).

Regarding make install having a secondary purpose of being kind of a reinstall, I disagree. I tend to write an explicit make reinstall target for that purpose (implemented as \t$(MAKE) uninstall\n\t$(MAKE) install); I didn't write it yet for the man-pages, but I'm going to add it now.

[2]: <https://stackoverflow.com/questions/70901364/should-make-install-depend-on-compilation>

I also had that feeling at first. I just leave it there because of "don't fix it if it ain't broke" and it just works. .so has a good side, which is that the Makefile is simpler, as it doesn't need to create links.

Yupee! :/

I'll see if I can reduce that overhead without losing features. Maybe I improve compatibility in the way. :)

See 2.

Heh, I agree! I would have put them in section 7, but I was new to the project, and didn't want to change things too much at the time. Since queue(3) was in man3, I kept the tradition, and the child pages were kept in man3. Probably I should have put them in man7, but blame history, not me :)

Buuut, is it me, or I see a contradiction with point 1, which claims that queue(3) should be in man3? We don't have a slist() function/macro, but we don't have a queue() one either (maybe historically there was one and I don't know it, but I guess not. My systems say:

alx@devuan:/usr/include$ grepc queue
alx@devuan:/usr/include$

alx@debian:/usr/include$ grepc queue
alx@debian:/usr/include$

Should now queue() be in man3 or man7?

Actually, I didn't duplicate information at all, AFAIR. It was already _very_ duplicated in the same queue(3) page, so I just splitted it at the right points. I only had to cut the page into many little ones, then translate the pages from mdoc(7) to man(7), and then fix minor style issues.

See:

$ wc -l man3/circleq.3 man3/list.3 man3/slist.3 man3/stailq.3 man3/tailq.3 man7/queue.7
  318 man3/circleq.3
  306 man3/list.3
  317 man3/slist.3
  375 man3/stailq.3
  395 man3/tailq.3
  133 man7/queue.7
 1844 total
$ git checkout man-pages-5.08 >/dev/null 2>&1
$ wc -l man3/queue.3
1231 man3/queue.3

The difference is just source code overhead; the text is almost the same.

Maybe you could still simplify your queue(7) page in a different way, without splitting it; it is very repetitive.

Heh, I know. The only thing that was well received from my side in that list was a bug report about some exec(3) function (about alloca(3)).

Oh, yes, I envy your headers. They are really readable!
But glibc source code is not as friendly, and it took me a long long time to get used to search things in those headers (I still can't find some things in their code; function definitions, for example, are very cryptic in some cases).

Also, some programmers, especially when starting (but I know of many programmers that are "senior" and still have serious issues with types), would benefit from documentation specific about types. That would help understand their limitations, and what a type is appropriate for or not.

Oh, that's where man(7) sucks :)

I wouldn't need these:

$ sed -n /^man_lsfunc/,/^}/p <scripts/bash_aliases
man_lsfunc()
{
	if [ $# -lt 1 ]; then
		>&2 echo "Usage: ${FUNCNAME[0]} <manpage|manNdir>...";
		return $EX_USAGE;
	fi

	for arg in "$@"; do
		man_section "$arg" 'SYNOPSIS';
	done \
	|sed_rm_ccomments \
	|pcregrep -Mn '(?s)^ [\w ]+ \**\w+\([\w\s(,)[\]*]*?(...)?\s*\); *$' \
	|grep '^[0-9]' \
	|sed -E 's/syscall\(SYS_(\w*),?/\1(/' \
	|sed -E 's/^[^(]+ \**(\w+)\(.*/\1/' \
	|uniq;
}
$ sed -n /^man_lsvar/,/^}/p <scripts/bash_aliases
man_lsvar()
{
	if [ $# -lt 1 ]; then
		>&2 echo "Usage: ${FUNCNAME[0]} <manpage|manNdir>...";
		return $EX_USAGE;
	fi

	for arg in "$@"; do
		man_section "$arg" 'SYNOPSIS';
	done \
	|sed_rm_ccomments \
	|pcregrep -Mv '(?s)^ [\w ]+ \**\w+\([\w\s(,)[\]*]+?(...)?\s*\); *$' \
	|pcregrep -Mn \
	  -e '(?s)^ +extern [\w ]+ \**\(\*+[\w ]+\)\([\w\s(,)[\]*]+?\s*\); *$' \
	  -e '^ +extern [\w ]+ \**[\w ]+; *$' \
	|grep '^[0-9]' \
	|grep -v 'typedef' \
	|sed -E 's/^[0-9]+: +extern [^(]+ \**\(\*+(\w* )?(\w+)\)\(.*/\2/' \
	|sed    's/^[0-9]\+: \+extern .* \**\(\w\+\); */\1/' \
	|uniq;
}

And they're not perfect...

That's not true. I've needed (or better phrased, wanted) types, even when I wasn't using any APIs that used them. The reason was that I was designing an API, and wanted to use the most appropriate types for my functions.

Having had documentation about types would have helped *a lot* at the time.

Many programmers don't know all the differences between size_t and ssize_t, and for example that ISO C only provides one of them (the other is added by POSIX), and I know of one *great* programmer that learnt the difference between those types from me a few months ago :).

One could search in the standard documents about the types, but I guess we will agree that those documents are not very friendly, especially for beginners.

Another case where I've found my type pages very useful was when I contributed[3] to iwyu(1)[4]. Having to read the POSIX or ISO C documents would have been crazy (I had to do it anyway, to write the pages, but I don't want to repeat that process again ;)).

[3]: <https://github.com/include-what-you-use/include-what-you-use/pull/930>
[4]: <https://include-what-you-use.org/>

We don't have that feature in man(7), so the closest thing that I do is to grep in the glibc and BSDs source code with grepc(1)[3], and also grep -rn ... inside the man-pages repo.

[5]: <http://www.alejandro-colomar.es/src/alx/alx/grepc.git/>

The first argument, I agree, and it's why I didn't use section 3, but subsection 3type.

The second, I disagree for the reasons above, but can understand why others might disagree with me. Maybe I can convince you :)

I considered[6] using man3type, and used man3 in the end just because when in doubt I opted for the smallest change. Knowing that it breaks mandoc(1), I'll definitely move to <man3type/>.

[6]: <https://lore.kernel.org/linux-man/[email protected]/T/#u>

Yes, since there's no compilation, make install is basically a wrapper around cp -r.

It has nice features, such as reduced install time by checking timestamps, but that's more useful to me as a maintainer (since I install several times a minute in some cases), and not so much for end users, where a few seconds are not important.

Interesting. Thanks!

Cheers,

Alex

Alejandro Colomar
<http://www.alejandro-colomar.es/>

Hi Ingo!

[...]

I fixed it:
<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/?id=451a27a78d51973b01bfb5d3b1e0ec081d9161e1>

And Debian seems to work fine with man3type/ and man2type/ out of the box, so I prefer it this way. I hope that other projects follow the example; and that packagers/distributions also create subsection directories, and don't undo my work.

Cheers,

Alex

Alejandro Colomar
<http://www.alejandro-colomar.es/>

Hi Alejandro,

Alejandro Colomar wrote on Thu, Jul 21, 2022 at 04:17:18PM +0200:

I didn't mean to say man3/id_t.3type "breaks mandoc". Quite to the contrary, the above quotation explains that mandoc copes with it.

However, when it comes to robustness with respect to *other* man(1) implementations apart from mandoc and man-db, i suspect the most portable and reliable way is using man1 - man9 only with no suffixes, consistent suffixes like "man3type/id_t.3type" are probably medium portability and medium reliability, and inconsistent suffixes like "man3/id_t.3type" and "man3type/id_t.3" are likely the least portable and the most fragile.

So your change is an improvement.

The system making the heaviest use of section suffixes i'm aware of is Solaris:

SunOS unstable11s 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise

  entities      man3dat       man3mvec      man3sysevent    man4b
  fr            man3dax       man3nsl       man3tcl         man5
  fr.ISO8859-1  man3devid     man3nvpair    man3tecla       man5oldap
  fr.UTF-8      man3devinfo   man3oldap     man3tiff        man5openssl
  it            man3dlpi      man3openssl   man3tsol        man7
  it.ISO8859-1  man3dns_sd    man3p         man3uuid        man7d
  it.UTF-8      man3elf       man3pam       man3volmgt      man7fs
  ja_JP.UTF-8   man3exacct    man3pcap      man3x           man7i
  man-index     man3ext       man3perl      man3x11         man7ipp
  man.cf        man3f         man3pi        man3xau         man7m
  man1          man3fcoe      man3picl      man3xaw         man7openssl
  man1b         man3fm        man3picltree  man3xcb         man7p
  man1c         man3fstyp     man3plot      man3xcomposite  man8
  man1m         man3gen       man3pool      man3xcurses     man8oldap
  man1oldap     man3gss       man3proc      man3xcursor     man8s
  man1openssl   man3hbaapi    man3project   man3xevie       man9
  man1s         man3head      man3rad       man3xext        man9e
  man1t         man3iscsit    man3reparse   man3xi          man9f
  man2          man3kstat     man3resolv    man3xinerama    man9p
  man3          man3kvm       man3rpc       man3xmu         man9s
  man3archive   man3layout    man3sasl      man3xnet        pl
  man3c         man3ldap      man3scf       man3xrandr      pl.ISO8859-2
  man3c_db      man3lgrp      man3sec       man3xss         pl.UTF-8
  man3cc4       man3lib       man3sip       man3xt          ru.KOI8-R
  man3cfgadm    man3m         man3slp       man3xtsol       ru.UTF-8
  man3cmi       man3mail      man3snmp      man3xtst        zh_CN.UTF-8
  man3commputil man3malloc    man3socket    man3xv
  man3contract  man3mlib      man3srpt      man3xxf86vm
  man3cpc       man3mp        man3ssh2      man3zonestat
  man3curses    man3mpapi     man3stmf      man4

Inside these directories, they are *mostly* using the convention "keep both section names consistent", so i do think that is good to follow. Even Solaris isn't perfect in that respect, though, they have, for example,

/usr/share/man/man3cc4/cartpol.3

but on first sight, i only found about two dozen such examples.

I did not scrutinize your change in detail, but had a brief look at it and did not see any obvious problems.

On Linux, undoing it would make very little sense to me because i expect that all man(1) programs commonly used on Linux can cope with section suffixes, and in particular with consistent use of session suffixes. Besides, undoing it properly is hardly possibly for a packager. If would require changing all these:

- directory names
- file names
- .TH macros
- and manual page cross references

So for better or worse, the only sane option for a packages is to follow your lead.

Yours,
Ingo

Hi Ingo!

On 7/22/22 18:59, Ingo Schwarze wrote:
[...]

Yeah, I didn't mean break as in "mandoc(1) goes nuts, or crashes", but rather as in "it doesn't do what I wanted it to do".

:)

Wow!
Although it's interesting to know that this list exists:
I can check it when trying to come up with a section name.

I guess Illumos shares this subsectioning scheme.

Do you know from the top of your head if any of those is dedicated to constants such as NULL, PATH_MAX, or BUFSIZ?

In my git clone of Illumos, I can't find anything relevant:

alx@asus5775:~/src/illumos$ find  | grep -i null | grep man
./illumos-gate/usr/src/man/man9f/nulldev.9f
./illumos-gate/usr/src/man/man4d/null.4d
./illumos-gate/usr/src/man/man4d/nulldriver.4d

I actually make it as easy as I can to modify directory names in the Makefile through the standard GNU directory variables.

I don't expect them to change filenames, even if they change dirs. It wouldn't make sense at all.

The same as fith file names. I expect them to want the pages to be in subsections, even if they wouldn't like the directory structure.

That's a more difficult one. They would need to run a sed script for every release. A good reason to not modify it.

If they change dirs and don't change this, they'd break .so links.

:)

Cheers,

Alex

Alejandro Colomar
<http://www.alejandro-colomar.es/>

Hi Alejandro,

Alejandro Colomar wrote on Fri, Jul 22, 2022 at 07:37:14PM +0200:

Even that is an overstatement. The difference between being in section 3p only and being in both section 3p and 3 is barely user-visible.

Here is a manual page that is in section 3p and in section 3p only, on OpenBSD-current:

   $ man -w warnings
  /usr/share/man/man3p/warnings.3p
   $ grep  '^\.TH' $(man -w warnings)
  .TH warnings 3p "2021-03-02" "perl v5.32.1" "Perl Programmers Reference Guide"

But lo and behold:

   $ man -s 3 -w warnings
  /usr/share/man/man3p/warnings.3p

Mandoc still finds the page in section 3 because "3" is a substring of "3p", and that not only makes sense from the string processing perspective but also from a logical perspective because section 3p is a particular corner of section 3 after all.

To cause mandoc to *not* find the page in section 3, the user would have to type something like

   $ man -k 'sec~3$' -a warnings
  man: nothing appropriate

Even i don't normally use such advanced syntax (regex matching on section names using the "sec" search keyword and the explicit regex operator and the -a logical operator for the conjunction of two search criteria). I doubt whether man-db even supports similar features...

[...]

I would somewhat advise against that. While i do think that consistency is good *if* you decide to use a section suffix, i'd still recommend to use section suffixes sparingly, at least in operating systems that use them sparingly now. They provide relatively little value, make the top directory of the manpath larger and less readable, and they are in particular *not* well suited to moving stuff out of the non-suffix directories that users may not wish to see by default. For example, suppose you created a directory man3py to document python library functions. As explained above, these pages will *still* show up even when people type "man 3" or "man 3p" rather than "man 3py".

More or less, according to
https://src.illumos.org/source/xref/illumos-gate/usr/src/man/ - it's not identical though.

I doubt it, for two reasons. On the one hand, my impression is that at least in Solaris, section suffixes are not so much used for logical subdivision of sections but more according to provenance; for example, man1b is for BSD compat features, man1s is for commands specific to SunOS, man3c is for libc, man3ext is for an "extended library", man3p is for the Sun performance library (available for both C and FORTRAN 95), and so on.

On the other hand, naming manual pages after symbolic constants or after type names is so unsual that i doubt any scheme exists for that. The most widely used way to look up manual pages by the names of symbolic constants or type names probably is using macro keys as implemented in the mandoc version of apropos(1). That is used by most FreeBSD, OpenBSD, Alpine Linux, and Void Linux. I admit that doesn't qualify as "widely used", but "most widely used" is probably true all the same. ;-)

Yours,
Ingo

[TO += Colin; CC += man-dv-devel@;
there's a bug in man(1)'s autocomplete]

Hi Ingo (and Colin),

I don't think there's any benefit of keeping $(mandir) contain only the standard man? subdirs. Everybody already knows about them, so nobody needs to read the output of ls(1) in such a system. On the other hand, reading the output of ls(1) in a system with free use of subsections provides useful information. If one needs to filter a bit, standard Unix tools come into play.

But the good thing is they can be put the last in the list of pages to be shown by man(1), so they won't hide another page.

In my Debian system, man(1) shows:

1 n l 8 3 0 2 3posix 3pm 3perl 3am 5 4 9 6 7

3type and 3const could go at the end of that, to avoid hiding pages in section 7.

While testing some related stuff to add to my arguments, I just understood of a bug that I knew existed but never thought too much about it, and related to one that Branden reported to me recently:

man 3 foo[TAB]

autocomplete is smart enough to only search pages in section 3 (I don't know if it finds pages in subsections from 3; I guess it does).

man [-s]3type foo[TAB]

autocomplete should search in 3type, but it doesn't quite work.
However, I noticed there's a difference in behavior between 3posix and 3type (3posix only exists as a suffix, but pages live in man3; 3type is a correct subsection, using man3type and .3type), so maybe my work can help fix that bug in the autocomplete scripts.

I know you don't use autocomplete for those things, but it comes in handy to me :)

Actually, man3type exists in several systems. Solaris has it (I guess Illumos too), and I've seen it in other systems (something from Oracle, IIRC). libbsd(7) also documents types, although they put them in the global namespace, which I think you and I agree that it's not quite right because of "documentation about a non-function in man3".

That leaves out man(7). And types tend to be not very well documented if they are documented as part of a function page. And they also tend to be documented several times (out of sync, of course).

Cheers,

Alex

Alejandro Colomar
Linux man-pages comaintainer; http://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Hi Alejandro,

Alejandro Colomar wrote on Sun, Jul 24, 2022 at 01:09:23PM +0200:

[...]

Admitted, it's a weak argument. Having lots of useless directories does not do *significant* harm. Then again, in general, i would say it is good practice to not add needless files or directories, even when they are not particularly harmful. They still make find(1) and ls -R output and Makefiles longer. Not doing needless things is a general aim of design economy.

Besides, when there is not much to gain in the first place, even weak counter-arguments wield non-zero weight.

I flatly deny that argument. The Solaris list of suffixes deleted above does not tell me anything of value. It's just a list of random sources of software, often with arbitrary partitions, and many of the suffixes are non-descriptive.

[...]

Not on the Solaris 11 system i have access to:

SunOS unstable11s 5.11 11.3 sun4u sparc SUNW,SPARC-Enterprise

[ no output]

/usr/share/man/*type*: No such file or directory

/usr/gnu/share/man/*type*: No such file or directory

Not according to https://src.illumos.org/source/search?path=3type
nor according to
https://src.illumos.org/source/xref/illumos-gate/usr/src/man/ .

Again, Solaris suffixes do *not* indicate logical subdivisions, but only a rather fuzzy, arbitrary, and inconsistent "where we got these files from", at least if i understand correctly.

That argument has no merit. The pre-eminent rule is "all documentation of function libraries belongs in section 3" and explaining data types (typically defined by the function libraries in the same header files as the functions) are clearly part of that.

The rule "all library documentation should be organized by functions" is secondary to that, and the rule "all section 3 names should be function names" is merely a corollary to that.

So even if you throw the rule "all library dicumentation should be organized by functions" overboard, that is still no excuse for moving part of the library documentation out of section 3. It also makes sense to keep it all together in the same section because it is needed by exactly the same people (programmers) in exactly the same way (when reading and writing code and wondering what words contained in the code mean).

Yes. Searching for preprocessor constants and searching for data type names are essentially semantic search features. So it is your choice to pick a 197x-era markup language that does not provide semantic markup but only physical markup. But than it feels irrational to me to turn around and complain not getting semantic search. Unless you are a Prime Minister, you cannot have your cake and eat it.

Trying to work around the lack of semantic markup by moving everything into the manual page names feels like very poor design to me.

We disagree about that, and i won't repeat my full explanation why. I'll repeat only this one aspect: There are few syntax elements where context matters as much as for types; types live by how they are used. Separating their description from the functions using them is a disservice to users, forcing them to jump around various pages instead of having explained together what belongs together.

Any kind of documentation needs discipline and diligence and maintenance, and none of it can ever be perfect. But that's no excuse for artificially tearing apart what users need together.

Yours,
Ingo

Perhaps not for long...

It will not surprise, but might horrify, Ingo to learn that I have an idea for how to add semantic markup to man(7).

Consider this hypothetical example.

  $ cat man3/man-pages.man
  .DC type B
  .DC field I
  $ cat man3/tm.3type
  .so man3/man-pages.man
[...]
  .SH DESCRIPTION
  .TG type "struct tm"
  describes time, broken down into distinct components.
  .PP
  .TG field tm_isdst
  describes wether daylight saving time is in effect at the time
  described.
[...]

Here, "DC" means "define class", a class of tags. "TG", if one could not guess, declares a tag of the type in its first argument with the remaining arguments being the content thus tagged.

Returning to "DC", we see that it takes a second argument naming a macro to call which will then apply any desired presentational markup to style the tagged word. This second argument need not be present. In other words, tagged content need not be visually distinct from its surroundings. Even in that event, it can still be useful; see #1 below.

Further, it will be obvious to the experienced *roff user that the macro called by DC to style the applicable arguments given to TG need not even be part of the man(7) language.

You could populate "man-pages.man" like this.

  $ cat man3/man-pages.man
  .de CW
  .  ie t \&\f[CR]\\$*\f[]
  .  el   \&\\$*
  ..
  .DC type CW
  .DC field I

This technique breaks the stranglehold of the man(7) font selection macros. (You're still limited by the output device's font repertoire, however.) If rendering to PostScript or PDF, you could decide to style certain tags in Zapf Chancery Medium italic, if you wished. (I cannot warrant that you won't get yelled at.)

Here are a few perhaps less obvious things this approach would offer.

1. It enables keyword search by tag. Whatever does the searching need only look for "TG" calls, match the class argument, and return the remainder. A search could be narrowed by limiting both the class _and_ the keyword arguments of course, perhaps to answer questions like "what pages use 'stat' as data type?".

2. Degraded operation for other/older man(7) implementations is

    straightforward.  'DC' can be completely ignored.  'TG' can be
    defined as follows.

    .de TG
    \&\\$*
    ..

    or, for truly bloody-minded portability, thus.

    .de TG
    \&\\$1 \\$2 \\$3 \\$4 \\$5 \\$6 \\$7 \\$8 \\$9
    ..

3. Everyday man(7) page authors need only learn 'TG' and the available list of keywords for the suite of man pages to which they are contributing. Hammering out the repertoire of available tag classes and the surely monumental bikeshedding of text styling decisions to be associated with each tag class is delegated to the project that chooses to define them. The man(7) macro package itself will impose no policy and may not even define any tag classes to start with. (groff would have some for its own man pages, of course, as I would expect Linux man-pages to do.)

4. Site admins offended at the styling decisions undertaken by various projects could reliably override them by editing the files sourced by the relevant man pages. Maybe those should live in /etc rather than the man page hierarchy proper.

5. Misspelling a tag class or using an unavailable one is an error that would be easily diagnosed and reported.

To reiterate, groff man(7) would impose no policy regarding the tag classes or their rendering on anyone. It similarly would escape the ongoing problem that mdoc(7) chose for itself by administering centralized authoritative lists of standards documents, operating system releases, and other lexica. Tagful man(7) pages under my proposal would opt into whatever keyword/class discipline they desire, or not at all.

I am not wedded to the nomenclature for the included files, nor the DC or TG macros, except to note that the macro names are available. (DT, putatively for "define tag", is not. It is already taken.)

I stand ready for the hail of rotten tomatoes.

Regards,
Branden

Hi Branden,

I hope you are not surprised that i'll try to answer constructively.

Even though i'm not sure how relevent such an early design draft is for linux-man@ or man-db-devel@, i'm not trimming the distribution list just yet, but you might perhaps wish to consider choosing one appropriate forum rather than cross-posting a design effort that is likely to be long and difficult to three lists.

G. Branden Robinson wrote on Sun, Jul 24, 2022 at 12:29:47PM -0500:

It does not surprise me, you have already moved ahead with adding new syntax to man(7), though so far in a piece-meal manner. It would only horrify me if this were done badly.

Since this is an early draft, it is not a surpise that i will raise several points of the kind "won't work as you apparently hope" below. That's not intended as denigration. In an early design phase, it is normal that not every idea is viable. The job is to find the working ideas in a larger pool of potential ideas and hammer them into a working whole of minimal complexity that does the job.

There is a danger of collisions here between the helper files of different software projects. That's maybe manageable; page names can already collide right now. Do such helper files really need to be section-specific? mdoc(7) teaches that in practice, there are basically two markup domains: command line utilities and library functions. A possible third domain, kernel configuartion directives, leads a fringe existence at best. In practice, combining both domains in one namespace poses no problems.

Besides, you will hardly be surprised that i deem making the styles configurable detrimental in the first place. The whole point of having such styles is enabling a uniform, consistent user experience.

This utterly breaks compatibility.
On formatters not suuporting .TG, the most important content of the dpocument will simply disappear.

Even if you would somehow manage to get a fallback

.de TG
\&\\$2
..

in place - and so far, i have no idea how you could do that portably on an arbitrary and potentially unknown formatter - that would leave "struct tm" with no markup whatsoever.

One obvious idea to trivially improve the design is to use the equivalent syntax

.TG type
.I "struct tm"

with the syntax constraint ".TG needs to be followed by a font macro" and the semantic effect "in that case, the custom style overrides the default style", reminiscent of how CSS can be used to override the default styles of HTML elements.

This suggestion for improvement is not yet a definitive recommendation though, it is merely another idea for the pool.

Let me also note that the markup overhead for your proposed scheme is so far at least three times as large as for mdoc(7). With modc(7), marking up a work usually requires one two-letter macro as the markup overhead. Your scheme requites the tag (which will usually be at least four letters, sometimes more) in addition.

For spacing and punctuation control, mdoc(7) usually requires no furthes overhead, except occasionally two letters for Ns. It's not yet clear how your scheme will fare in that respect because you proposed no mechanism for spacing and punction control yet, but that will likely require some overhead, too.

I fear you are missing the main aspect that makes using the man(7) font macros complicated: spacing and punctuation.

Let us start with a moderately simple, everyday example:

.Ft double Fn atan2 "double y" "double x"

What would that become with your proposed syntax?

.TG type double
.TG funcname atan2
(
.TG funcarg "double y"
,
.TG funcarg "double x"
)

What is your plan for that? Recommending \c everywhere? Or mdoc(7)-style punctuation handling like

.TG type double
.TG funcname atan2 NS
.TG funcarg ( "double y" ,
.TG funcarg "double x" )

Practical work shows these trivial cases of spacing around common punctuation like parentheses and commas are not the only ones that occur. Sometimes, spacing needs to be inserted or suppressed in ways that differ from default rules. mdoc(7) has several mechanisms for that, the most important one being .Ns. Your new scheme might choose a different approach, but i predict how well that mechanism is going to be will be one determining factor of how pleasant or unpleasant the language will become for authors and maintainers.

Sure, just like

man -k Vt=stat

with mandoc.

No, that is plain wrong. It results in

type struct tm describes time, broken down into distinct components. field tm_isdst describes wether daylight saving time is in effect at the time described.

You certainly do not want to print out \\$1.

That would exclude any kind of mdoc(7)-style punctuation handling, making spacing control much harder to implement. I'm not saying impossible, but harder. One way out might be to use \\$2 only and require quoting of multi-word arguments.

I fear that is a terrible idea. The DocBook design fiasco teaches us that even a single committee is easily capable of utterly screwing up markup consistency. Leaving the markup lexicon up to individual projects guarantees hopeless global inconsistency. Different projects will mark up different syntax elements, assign different tags for the same purpose, and assign conflicting styles to the same tags.

Consequently, semantic searching will *not* work at all. You cannot even search for a function name if one group tags it as "function" and another as "funcname" and another as "sub" and another as "procname". Rendering will be wildly inconsistent because every group of pages, large or small, comes with its own styles - and larger projects are unlikely to remain consistent even within themselves. Due to the undefined nature of tag names, distributions are unable to fix up the mess, too.

Branden, seriously, standardizing this is your number two most important job during this design. The top one priority, of course, is to choose a syntax that is short, simple, robust, esay to read, and easy to write for both usual and unusual use cases, and - if you want to continue calling the language man(7) - backward compatible.

That's illusionary until you provide an authoritative and long-time stable list of valid tags.
Failing to do so is one among the reasons why SGML and XML turned out as hard to use in practice, and one of the reasons why HTML ultimately gave up on being XML and instead defined an exhaustive list of standard elements.

Besides, on one of my machines, i currently have almost 600 packages installed even though lots of things that are packages on Linux do *not* count as packages on OpenBSD but are part of the base system that cannot ever be uninstalled, including thing like compilers and other development tools and lots of Internet daemons like web server, mail server, routing daemons, DNS servers and much more. And yet, i still have almost 600 packages. As a rule of thumb, on average one third of packages have manual pages. So you want me to edit 200 config files and resolve conflicts in them whenever updating packages? I'm not yet convinced that's a very good plan. :-o

I freely admit that .St .At .Bx .Nx .Fx .Ox .Dx .Lb are among the least well designed parts of mdoc(7). But that design problem is completely unrelated to the fact that the list of macros (in mdoc(7) parlance) or tags (in your proposed terminology) needs to be fixed. That's *the* central purpuse of a markup language. Personally, i would go as far as calling a non-extensible markup language usually better than an extensible one, but even for an extensible markup language, the quality of the standard lexicon is the second highest priority, right after the quality of the basic syntax.

Given how utterly your .TG proposal is breaking compatibility, i wonder whether it would be wiser to give your new macroset a new name and leave the legacy man(7) language alone. That would allow cleaning up some design deficiencies of man(7) at the same time, for example:

* properly define display macros, a purpose now served in a manner the is partially quirky, partially redundant, and yet incomplete by .RS and .EX
* get rid of the font alternation macros; it is well known that even though those squat on a considerable parcel of namespace, they only solve part of their task - the infamous three-font issue; your proposal needs a solution for that anyway that still needs to be developed
* get rid of the redundancy among .TP .TQ .IP .HP
* get rid of legacy macros like .AT .DT .LP .P .SM .SB .UC that are useless in practice
* get rid of misdesigned macros like .SY

Please also consider that having a mix of semantic and presentational markup in a language is usually not the best idea. As a prominent example, after a considerable amount of handwringing that lasted for many years, HTML finally abolished presentational markup completely. For similar reasons, we somewhat discourage using .Em and .Sy in mdoc(7), except for purposes similar to <em>, <i>, <strong>, and <b> in HTML 5.

If you choose a new name, compatibility concerns vanish instantly. But of course, once you publicly announce "this is now ready for production", stability becomes important again, so the design should be well thought out from the start. Finding a major design issue or design gap after it has been in production for, say, two years would be very unfortunate.

On the other hand, if you want to continue calling it man(7), then the logical consequence is that compatibility concerns become paramount.

Yours,
Ingo

man-db doesn't ship its own autocomplete code at the moment. This is probably shipped by some other package in your distribution, although I don't know exactly what that would be (maybe bash-completion?).

Colin Watson (he/him)                              [[email protected]]

Hi Colin,

Heh, I bet it's the same as yours, as you show up as the maintainer ;)

$ apt-cache show man-db | grep Maintainer
Maintainer: Colin Watson <[email protected]>

Yeah, I use Debian (Sid).

Reported to upstream:
<https://github.com/scop/bash-completion/issues/778>

Cheers,

Alex

Alejandro Colomar
<http://www.alejandro-colomar.es/>