Bogus index in man-pages book from other projects

Hi Deri!

I've tried to build a man-pages book of the shadow project, using the Linux man-pages build system (which calls your script). The text contents look good, but the index seems to be bogus, and man-page references don't work (it results in just text).

I only see in the index the names of the pages that are aliases. None of the actual pages (their in-page sections are top-level entries in the index).

To reproduce it,

```sh
git clone -b shadow http://www.alejandro-colomar.es/src/alx/linux/man-pages/man-pages.git/
git clone https://github.com/shadow-maint/shadow
mkdir build
cd build
sudo apt-get build-dep shadow
../shadow/autogen.sh
make -j4
cd ../man-pages
make build-book MANDIR=../build/man
open .tmp/man/man-pages.pdf
```

The make build-book step results in the following warning, which may be relevant:

$ make build-book MANDIR=../build/man
MKDIR		.tmp/man/
Build		.tmp/man/man-pages.pdf
for my (...) is experimental at ./scripts/LinuxManBook/prepare.pl line 62.
pdf.tmac:chage.1:29: warning: adjusted level 3 bookmark; should be <= 1
pdf.tmac:chage.1:31: warning: adjusted level 3 bookmark; should be <= 2

Would you mind having a look at this?

Have a lovely day!
Alex

<https://www.alejandro-colomar.es/>

Hi Alex,

This is due to the .TH line not being recognised because its parameters were quoted, regex adjusted to recognise with or without quotes.

The missing intra-page references (i.e. in See Also) were not recognised because they did not use .MR or .BR, instead they used "\fBcommand\fR(n)", experimentally I have added this pattern to recognise as a potential link.

This only works on debian type systems (apt-get), so I had to use my raspberrypi5 instead of my desktop.

I tested it using MANDIR=../man-pages/ru as well, some of it is cyrillic!

Another reason .TH was not being recognised is the incorrect usage of \& before a full stop in the command, i.e. login\&.defs. This is not necessary because it will not be recognised as an end of sentence, so I removed it.

Cheers

Deri

index bb8667dd8..bfacb3648 100755
@@ -59,7 +59,7 @@ foreach my $al (`grep -E '^\\.so' $dir/man*/*`)
 	$aliases{$1}=$2;
 }
 
-foreach my ($k,$v) (%aliases)
+while (my ($k,$v)=each %aliases)
 {
 	while (exists($aliases{$v})) {
 		$v=$aliases{$v};

@@ -98,6 +98,15 @@ sub BuildPage
 
 	# If this is an alias, just add it to the outline panel.
 
+	# if new section add top level bookmark
+
+	if ($sec ne $Section) {
+		print ".nr PDFOUTLINE.FOLDLEVEL 1\n";
+		print ".pdfbookmark 1 $Sections{$sec}\n";
+		print ".nr PDFOUTLINE.FOLDLEVEL 2\n";
+		$Section=$sec;
+	}
+
 	if (exists($aliases{$bkmark})) {
 		print ".eo\n.device ps:exec [/Dest /$aliases{$bkmark} /Title ($title) /Level 2 /OUT pdfmark\n.ec\n.fl\n";
 		return;

@@ -123,7 +132,7 @@ sub BuildPage
 
 			s/\\-/-/g if /^\.[BM]R\s+/;
 
-			if (m/^\.BR\s+([-\w\\.]+)\s+\((.+?)\)(.*)/ or m/^\.MR\s+([-\w\\.]+)\s+(\w+)\s+(.*)/) {
+			if (m/^\.BR\s+([-\w\\.]+)\s+\((.+?)\)(.*)/ or m/^\.MR\s+([-\w\\.]+)\s+(\w+)\s+(.*)/ or m/^\\fB([-\w\\.]+)\\fR\((.+?)\)(.*)$/) {
 				my $bkmark="$1";
 				my $sec=$2;
 				my $after=$3;

@@ -135,7 +144,7 @@ sub BuildPage
 					my $dest=$files{"${bkmark}.$sec"}->[1];
 					$_=".pdfhref L -D \"$dest\" -A \"$after\" -- \\fI$bkmark\\fP($sec)";
 				} else {
-					$_=".IR ".substr($_,4);
+					$_=".IR $bkmark ($sec)\\c\n$after";
 				}
 			}
 

@@ -161,16 +170,9 @@ sub BuildPage
 				s/\n\n/\n/g;
 			}
 
-			if (m/^\.TH\s+([-\w\\.]+)\s+(\w+)/) {
+			s/\\&\././ if m/^.TH /;
 
-				# if new section add top level bookmark
-
-				if ($sec ne $Section) {
-					print ".nr PDFOUTLINE.FOLDLEVEL 1\n";
-					print ".pdfbookmark 1 $Sections{$sec}\n";
-					print ".nr PDFOUTLINE.FOLDLEVEL 2\n";
-					$Section=$sec;
-				}
+			if (m/^\.TH\s+"?([-\w\\.]+)"?\s+"?(\w+)"?/) {
 
 				print "$_\n";
 

Hi Deri,

Sorry, I didn't know how to do the equivalent of apt-get build-dep in other systems. :)

There seems to be some equivalent for dnf(1)-based systems:
<https://unix.stackexchange.com/questions/326047/does-dnf-have-an-equivalent-to-apts-build-dep>

Hmm, interesting thing to try! I've tried it too, and the bookmarks for the in-page sections (e.g., DESCRIPTION, or rather ОПИСАНИЕ) appear with no name (or maybe it's a locale problem in my system?). See attached PDF.

<https://www.alejandro-colomar.es/>
Looking for a remote C programming job at the moment.

[looping in groff list]

Hi Alex,

That's a known problem with groff 1.23.0 and earlier. It went less remarked-upon than it should have because it turns out there is a way to sneak character codes with the eighth bit set as-is out of the formatter into device-independent output. (I just learned about this mechanism this past week.) And since a lot of groff users were happy with the ISO 8859-1 character repertoire in their documents, they were fine with it.

It's gone unresolved longer than it should have because fixing it is challenging. If you catch up on groff mailing list traffic for January and February you will see Deri and me discussing it.

I have a solution that I think will work,[1] but it keeps growing in scope. The thing I learned last week is that the \! escape sequence can be used to smuggle character codes 129-255 decimal into grout. (See attachment.)[2] Addressing this requires surgery on a part of the formatter that tends to be used only by relative experts and there aren't unit tests of this escape sequence to assuage my fear that I don't break things, so I'll have to write them.

Regards,
Branden

[1] The main element of it is to have the device request and the \X escape sequence (the latter being an AT&T troff feature) read their parameters in copy mode.

https://savannah.gnu.org/bugs/?64484

[2] Interestingly, what GNU troff does here is compatible with DWB 3.3 troff but not Heirloom Doctools troff; but Heirloom otherwise has no problem emitting UTF-8 sequences, for instance as arguments to trout 'C' commands.

My plan is to have GNU troff reject code points 128 <= n <= 255 in arguments to the device and output requests (both GNU extensions) and in \! and \X escape sequence parameters. We don't know what character encoding an output device requires, so my proposal is to require input documents (including macro packages) to express such code points as groff Unicode special character escape sequences (that is, in the form \[u123AB]). An alternative would be to have the output device report what encoding it requires in its DESC file, and give GNU troff the responsibility of converting to that encoding when writing output. But to me that seems like an inferior solution, loading up the formatter with more character set-conversion functionality when it's increasingly a UTF-8 world anyway. The likely persistent exception is the UTF-16-oriented PDF device. Fortunately, in groff 1.23.0, Deri added support to gropdf(1) for interpretation of such escape sequences in device "specials" (device control commands; "x X" commands in trout/grout). I'm attaching another couple of examples to illustrate this.

Also, if we make the formatter strict about 7-bit-clean input in groff 1.24, that will clear the decks for moving from an assumption of Latin-1 input today to UTF-8 input in 1.25.

.\" troff | hd # or your choice of hex dumper Hello, world.
.sp
\!x X The Stupendous Yäppi will now read your mind! .sp
Bye.

.\" groff -Kutf8 -Tpdf
.nr index 0 1
.de Section

.  sp 1i
.  ft B
.  pdfbookmark 1 "\\$*"
.  ds mark!\\n+[index] \\*[PDFBOOKMARK.NAME]
.  nop \\$*
.  ft
.  sp

..
.Section "\%A naïve attempt at bookmarking"
Sed ut perspiciatis, unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam eaque ipsa, quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt, explicabo. Nemo enim ipsam voluptatem, quia voluptas sit, aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos, qui ratione voluptatem sequi nesciunt, neque porro quisquam est, qui dolorem ipsum, quia dolor sit amet consectetur adipiscivelit, sed quia non-numquam eius modi tempora incidunt, ut labore et dolore magnam aliquam quaerat voluptatem.
.bp
.Section "Another section"
Return to
.pdfhref L -D \*[mark!1] -- the first section
or
.pdfhref L -A . -D \*[mark!2] -- the last one

.\" groff -Tpdf
.\" needs groff 1.23.0 or later
.nr index 0 1
.de Section

.  sp 1i
.  ft B
.  pdfbookmark 1 "\\$*"
.  ds mark!\\n+[index] \\*[PDFBOOKMARK.NAME]
.  nop \\$*
.  ft
.  sp

..
.Section "\%A na\[u00EF]ve attempt at bookmarking"
Sed ut perspiciatis, unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam eaque ipsa, quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt, explicabo. Nemo enim ipsam voluptatem, quia voluptas sit, aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos, qui ratione voluptatem sequi nesciunt, neque porro quisquam est, qui dolorem ipsum, quia dolor sit amet consectetur adipiscivelit, sed quia non-numquam eius modi tempora incidunt, ut labore et dolore magnam aliquam quaerat voluptatem.
.bp
.Section "Another section"
Return to
.pdfhref L -D \*[mark!1] -- the first section
or
.pdfhref L -A . -D \*[mark!2] -- the last one

Unfortunately, these are not examples of best practice for creating reference bookmarks. The problem is the bookmarks are just numbered and if you have a large document it is easier to give the bookmark a mnemonic instead.

==========================================================================

.\" groff -Kutf8 -Tpdf
.de NmSection

.  sp 1i
.  ft B
.  ds pdfsecnm \\$1
.  shift
.  pdfbookmark -T \\*[pdfsecnm] 1 "\\$*"
.  nop \\$*
.  ft
.  sp

..
.NmSection Intro "\%A naïve attempt at bookmarking"
Sed ut perspiciatis, unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam eaque ipsa, quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt, explicabo. Nemo enim ipsam voluptatem, quia voluptas sit, aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos, qui ratione voluptatem sequi nesciunt, neque porro quisquam est, qui dolorem ipsum, quia dolor sit amet consectetur adipiscivelit, sed quia non-numquam eius modi tempora incidunt, ut labore et dolore magnam aliquam quaerat voluptatem.
.bp
.NmSection Another "Another section"
Return to
.pdfhref L -D Intro -- the first section
or
.pdfhref L -A . -D Another -- the last one

==========================================================================

Cheers

Deri

[looping in groff list]

Hi Deri,

Good to hear from you!

They certainly aren't! I didn't intend them as a demonstration of best practice; they were files from my "experiments" directory, where I have hundreds...uh...1,834 files exercising various aspects of groff.

My intent was to share a simple and straightforward demonstration of the point I raised, about 8-bit characters being smugglable from *roff input to device-independent output using the pdfbookmark macro as the vehicle.

I'm pleased to report that this example also works fine to demonstrate PDF bookmarking functionality in groff Git HEAD, despite a recent change I've made to pdf.tmac.

What change and why?

Let me share a further development of the foregoing example.

.\" groff -Kutf8 -Tpdf
.de NmSection

.  sp 1i
.  ft B
.  ds pdfsecnm \\$1
.  shift
.  pdfbookmark -T \\*[pdfsecnm] 1 "\\$*"
.  nop \\$*
.  ft
.  sp

..
.NmSection \%Wacky\-Intro\:duction "A naïve attempt at bookmarking" Sed ut perspiciatis, unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam eaque ipsa, quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt, explicabo. Nemo enim ipsam voluptatem, quia voluptas sit, aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos, qui ratione voluptatem sequi nesciunt, neque porro quisquam est, qui dolorem ipsum, quia dolor sit amet consectetur adipiscivelit, sed quia non-numquam eius modi tempora incidunt, ut labore et dolore magnam aliquam quaerat voluptatem.
.bp
.NmSection Another "Another section"
Return to
.pdfhref L -D \%Wacky\-Intro\:duction -- the first section or
.pdfhref L -A . -D Another -- the last one

In groff 1.22.4, formatting the foregoing with "groff -Tpdf" yields the following.

$ /usr/bin/groff -Tpdf /tmp/branden/naive-gbr2-latin1.groff >/dev/null
troff: /tmp/branden/naive-gbr2-latin1.groff:12: '\%' is not allowed in a name
troff: /tmp/branden/naive-gbr2-latin1.groff:12: bad string definition
troff: /tmp/branden/naive-gbr2-latin1.groff:12: can't translate character code 239 to special character ':i' in transparent throughput

groff 1.23.0 says similar:

$ GROFF_ENABLE_TRANSPARENCY_WARNINGS=1 ~/groff-stable/bin/groff -Tpdf /tmp/branden/naive-gbr2-latin1.groff >/dev/null
troff:/tmp/branden/naive-gbr2-latin1.groff:12: error: an escaped '%' is not allowed in an identifier
troff:/tmp/branden/naive-gbr2-latin1.groff:12: error: bad string definition
troff:/tmp/branden/naive-gbr2-latin1.groff:12: error: can't translate character code 239 to special character ':i' in transparent throughput

But now in groff Git HEAD:

$ GROFF_ENABLE_TRANSPARENCY_WARNINGS=1 ~/groff-HEAD/bin/groff -Tpdf /tmp/branden/naive-gbr2-latin1.groff >/dev/null
troff:/tmp/branden/naive-gbr2-latin1.groff:12: error: can't translate character code 239 to special character ':i' in transparent throughput

...and the hyperlinks even still work!

The tolerance for Latin-1 smuggling continues but is, however, something I plan to change as noted earlier in the thread.

And at long last I feel I understand what that incomprehensible diagnostic message needs to say. It's been a long time coming--something like six years...

Regards,
Branden

Hi Alex,

It is not locale, it is a Branden thing! If you use the deri-gropdf-ng branch of groff, which Branden offered to do the release to master for me and has sat on it for 9 months, all languages are now supported since UTF-16 is now used. It is difficult to understand his reluctance to make groff pdfs accessible to all languages, perhaps as an American he just does not want the Russians and Chinese using his lovely groff, it is as likely as the rest of the spurious arguments he has hinted, but never specified. "I feel the code is not robust.", without giving an example of how it could fail, knowing that if he did I would just make the code "more robust"!!

Anyway, this what I get if I build the Russian version.

Cheers

Deri

Hi Deri,

Thanks!

Hmmm, that one looks great.

BTW, I've seen another glitch:

In the gpasswd(1) page, at the bottom (SEE ALSO, aka, СМОТРИТЕ ТАКЖЕ), the last two man-page references don't show correctly. Neither of them have a link, and only the second shows up in bold. The source is:

	alx@debian:~/src/shadow/shadow/build/man/ru$ tail man1/gpasswd.1
	содержит защищаемую информацию о группах
	.RE
	.SH "СМОТРИТЕ ТАКЖЕ"
	.PP
	\fBnewgrp\fR(1),
	\fBgroupadd\fR(8),
	\fBgroupdel\fR(8),
	\fBgroupmod\fR(8),
	\fBgrpck\fR(8),
	\fBgroup\fR(5), \fBgshadow\fR(5)\&.

So it might be due to having two references in the same line. I suspect maybe the updated regex has some issues?

This happens in both the English and the Russian pages.

Have a lovely day!
Alex

<https://www.alejandro-colomar.es/>
Looking for a remote C programming job at the moment.

Ahh, I tought there was a group(5) page in shadow, but there isn't. And since you only parse that pattern at the begining of a line, it didn't try the second one. Let's ignore these small and rare glitches, I guess.

Cheers,
Alex

<https://www.alejandro-colomar.es/>
Looking for a remote C programming job at the moment.