Optimize script for generating LinuxManBook.pdf

Hi Deri,

I've optimized from 18.5 s down to 16.3 s the script, by splitting the pipeline with this wrapper (and slightly reducing the perl script to just print the pages to stdout). BTW, now it can be run from any directory. And every step can be debugged by just introducing

| tee /dev/tty \

wherever you want to debug. It's all pushed to master.

The PDF is now printed to stdout, to avoid hard-coding file names.

I still need to split a bit more and reduce the longest lines. How does this script look to you?

Cheers,
Alex :-)

$ cat scripts/LinuxManBook/build_linux_man_book.sh
#!/bin/sh
# Copyright 2023, Alejandro Colomar <[email protected]>
# SPDX-License-Identifier: GPL-3.0-or-later

(

	"$(dirname "$0")"/prepare_linux_man_book.pl "$1" \

	| groff -z -dPDF.EXPORT=1 -dLABEL.REFS=1 -dpaper=a4 -Tpdf -k -pet \

		-M"$(dirname "$0")" -mandoc -manmark \
		-F"$(dirname "$0")" -P-pa4 \-rC1 -rCHECKSTYLE=3 2>&1 \

	| LC_ALL=C grep '^\. *ds ';

	"$(dirname "$0")"/prepare_linux_man_book.pl "$1";

) \

| preconv \
| tbl \
| eqn -Tpdf \
| (

	troff -Tpdf -ms <"$(dirname "$0")"/LMBfront.ms;
	troff -Tpdf -M"$(dirname "$0")" -mandoc -manmark \
		-F"$(dirname "$0")" -dpaper=a4;

) \

| gropdf -F"$(dirname "$0")" -pa4;

Hi Alex,

Is there a git address I can clone to see the changes "in the round” and give it a go?

Hi Deri,

I've optimized from 18.5 s down to 16.3 s the script, by splitting the pipeline with this wrapper (and slightly reducing the perl script to just print the pages to stdout). BTW, now it can be run from any directory. And every step can be debugged by just introducing

| tee /dev/tty \

wherever you want to debug. It's all pushed to master.

The PDF is now printed to stdout, to avoid hard-coding file names.

I still need to split a bit more and reduce the longest lines. How does this script look to you?

Cheers,
Alex :-)

Hi Alex,

It looks fine, although you have to run the code in "prepare_linux_man_book.pl" twice (to avoid using a temporary file).

If you
are going to run preconv it is best to run it first - stops pic spitting out loads of warnings.

You also dropped one stage in second pass, no pic in the pipeline. This may explain part of the speedup you observed. I don't know if any of your man pages require pic but they could in the future. The changes I would advise are:-

scripts/LinuxManBook/build_linux_man_book.sh +2 -1

@@ -4,8 +4,8 @@
 
 (
        "$(dirname "$0")"/prepare_linux_man_book.pl "$1" \
-       | pic \
        | preconv \
+       | pic \
        | tbl \
        | eqn -Tpdf \
        | troff -Tpdf -dPDF.EXPORT=1 -dLABEL.REFS=1 -dpaper=a4 \

@@ -16,6 +16,7 @@
        "$(dirname "$0")"/prepare_linux_man_book.pl "$1";
 ) \
 | preconv \
+| pic \
 | tbl \
 | eqn -Tpdf \
 | (

Cheers

Deri

Quoted 2 lines

It looks fine, although you have to run the code in "prepare_linux_man_book.pl" twice (to avoid using a temporary file).

Yep. I was wondering if we could change something in the design of prepare_linux_man_book.pl so that it could be run once without needing a temporary file. Maybe if it could insert something in the pages that the latter troff(1) would process in one take, without having to put all the bookmarks at the start of the file. That would be an important simplification of the scripts, and probably also an optimization.

Quoted 9 lines · Alejandro Colomar

Quoted 2 lines

It looks fine, although you have to run the code in "prepare_linux_man_book.pl" twice (to avoid using a temporary file).

Yep. I was wondering if we could change something in the design of prepare_linux_man_book.pl so that it could be run once without needing a temporary file. Maybe if it could insert something in the pages that the latter troff(1) would process in one take, without having to put all the bookmarks at the start of the file. That would be an important simplification of the scripts, and probably also an optimization.

Hi Deri,

I have another optimization: split the sort. It reduces around 0.3 s.

(quoted diff) +0 -0

b/scripts/LinuxManBook/prepare_linux_man_book.pl index 0a79df4e5..5a4aad429 100755

scripts/LinuxManBook/prepare_linux_man_book.pl +10 -1

@@ -88,7 +88,16 @@ sub BuildBook
 {
        print ".pdfpagenumbering D . 1\n";
 
-       foreach my $fn (sort sortman glob("$dir/man*/*")) {
+       foreach my $fn (sort glob("$dir/man*")) {
+               BuildSec($fn);
+       }
+}
+
+sub BuildSec
+{
+       my $manSdir=shift;
+
+       foreach my $fn (sort sortman glob("$manSdir/*")) {
                BuildPage($fn);
        }
 }

I didn't think of this as an optimization, but rather to move code from BuildPage() into BuildSec(). However, since it doesn't block until all of the pages are sorted, it reduces the latency of the script (that's my guess).

I think moving stuff from BuildPage() to BuildSec() would both simplify and optimize, so please check it when you can. (I'm also checking it, but while I'm learning Perl with this, I'm still very limited.)

Cheers,
Alex

Hi Alex,

I have attached the latest iteration of my work, managed to knock two seconds off the current code in your git.

It no longer uses temporary files,

outputs
the pdf to stdout, can be run from any directory and runs groff once.

It replaces the complete LinuxManBook directory and the executable is now called BuildLinuxMan2.pl.

Cheers

Deri

Quoted 2 lines

It replaces the complete LinuxManBook directory and the executable is now called BuildLinuxMan2.pl.

I'd prefer if the huge groff code would go in a separate file. Would that make sense?

Hi Deri,

Quoted 5 lines · Alejandro Colomar

Quoted 2 lines

It replaces the complete LinuxManBook directory and the executable is now called BuildLinuxMan2.pl.

I'd prefer if the huge groff code would go in a separate file. Would that make sense?

I've applied a few tweaks to the script you sent before committing it. The performance is similar, and it's less of a change to the current code.

<https://www.alejandro-colomar.es/src/alx/linux/man-pages/man-pages.git/commit/?h=contrib&id=0e64299d4dd0df90ea52b3fe6777e5ebfb2484da>

I think we should be able to cut half a second or so if we add a BuildSec() function so that we don't block the first page until the entire sort is done. And it would be also more readable. Please check.