rtla: Add hwnoise tool
Subsystems (2)
v2 cover rtla: Add hwnoise tool
This series present a new tool inside rtla. It is named hwnoise.
hwnoise dispatches the osnoise tracer and displays a summary of the noise. The difference is that it runs the tracer with the OSNOISE_IRQ_DISABLE option set, thus only allowing only hardware-related noise, resulting in a simplified output. hwnoise has the same features of osnoise.
An example of the tool's output:
# rtla hwnoise -c 1-5 -T 1 -d 10m -q
Hardware-related Noise
duration: 0 00:10:00 | time is in us
CPU Period Runtime Noise % CPU Aval Max Noise Max Single HW NMI
1 #599 599000000 138 99.99997 3 3 4 74
2 #599 599000000 85 99.99998 3 3 4 75
3 #599 599000000 86 99.99998 4 3 6 75
4 #599 599000000 81 99.99998 4 4 2 75
5 #599 599000000 85 99.99998 2 2 2 75Changes from v1:
Add rtla-hwnoise to the rtla documentation index
(kernel test robot/Bagas Sanjaya)
Daniel Bristot de Oliveira (6):
rtla/osnoise: Add helper functions to manipulate osnoise/options rtla/osnoise: Add OSNOISE_IRQ_DISABLE option rtla/osnoise: Add the mode abstraction rtla/osnoise_top: Pass the params to the usage function
rtla: Add hwnoise tool
Documentation/rtla: Add hwnoise man page
Documentation/tools/rtla/index.rst | 1 + Documentation/tools/rtla/rtla-hwnoise.rst | 107 +++++++++++++++++++ tools/tracing/rtla/Makefile | 2 + tools/tracing/rtla/src/osnoise.c | 120 ++++++++++++++++++++++ tools/tracing/rtla/src/osnoise.h | 7 ++ tools/tracing/rtla/src/osnoise_top.c | 84 ++++++++++++--- tools/tracing/rtla/src/rtla.c | 4 + 7 files changed, 309 insertions(+), 16 deletions(-) create mode 100644 Documentation/tools/rtla/rtla-hwnoise.rst
Signature · 1 line
2.38.1
PATCH v2 1/6 rtla/osnoise: Add helper functions to manipulate osnoise/options
Add some helper functions to read and set the on/off osnoise/options. No usage in this patch.
In preparation for hwnoise tool.
--- tools/tracing/rtla/src/osnoise.c | 108 +++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+)
tools/tracing/rtla/src/osnoise.c +108 -0
@@ -734,6 +734,114 @@ void osnoise_put_tracing_thresh(struct osnoise_context *context) context->orig_tracing_thresh =OSNOISE_OPTION_INIT_VAL ; } +static int osnoise_options_get_option (char *option) +{ +char *options =tracefs_instance_file_read (NULL ,"osnoise/options" ,NULL ); +char no_option[128 ]; +int retval =0 ; +char *opt; + +if (!options) +return OSNOISE_OPTION_INIT_VAL ; + +/* +* Check first if the option is disabled. +*/ +snprintf (no_option,sizeof (no_option),"NO_%s" , option); + + opt =strstr (options, no_option); +if (opt) + goto out_free; + +/* +* Now that it is not disabled, if the string is there, it is +* enabled. If the string is not there, the option does not exist. +*/ + opt =strstr (options, option); +if (opt) + retval =1 ; +else + retval =OSNOISE_OPTION_INIT_VAL ; + +out_free: +free (options); +return retval; +} + +static int osnoise_options_set_option (char *option,bool onoff) +{ +char no_option[128 ]; + +if (onoff) +return tracefs_instance_file_write (NULL ,"osnoise/options" , option); + +snprintf (no_option,sizeof (no_option),"NO_%s" , option); + +return tracefs_instance_file_write (NULL ,"osnoise/options" , no_option); +} + +#define OSNOISE_OPTION (name, option_str) \ +static int osnoise_get_##name(struct osnoise_context *context) \ +{ \ + if (context->opt_##name != OSNOISE_OPTION_INIT_VAL) \ + return context->opt_##name; \ + \ + if (context->orig_opt_##name != OSNOISE_OPTION_INIT_VAL) \ + return context->orig_opt_##name; \ + \ + context->orig_opt_##name = osnoise_options_get_option(option_str); \ + \ + return context->orig_opt_##name; \ +} \ + \ +int osnoise_set_##name(struct osnoise_context *context, bool onoff) \ +{ \ + int opt_##name = osnoise_get_##name(context); \ + int retval; \ + \ + if (opt_##name == OSNOISE_OPTION_INIT_VAL) \ + return -1; \ + \ + if (opt_##name == onoff) \ + return 0; \ + \ + retval = osnoise_options_set_option(option_str, onoff); \ + if (retval < 0) \ + return -1; \ + \ + context->opt_##name = onoff; \ + \ + return 0; \ +} \ + \ +static void osnoise_restore_##name(struct osnoise_context *context) \ +{ \ + int retval; \ + \ + if (context->orig_opt_##name == OSNOISE_OPTION_INIT_VAL) \ + return; \ + \ + if (context->orig_opt_##name == context->opt_##name) \ + goto out_done; \ + \ + retval = osnoise_options_set_option(option_str, context->orig_opt_##name); \ + if (retval < 0) \ + err_msg("Could not restore original osnoise " #option_str " option\n"); \ + \ +out_done: \ + context->orig_opt_##name = OSNOISE_OPTION_INIT_VAL; \ +} \ + \ +static void osnoise_put_##name(struct osnoise_context *context) \ +{ \ + osnoise_restore_##name(context); \ + \ + if (context->orig_opt_##name == OSNOISE_OPTION_INIT_VAL) \ + return; \ + \ + context->orig_opt_##name = OSNOISE_OPTION_INIT_VAL; \ +} +/* * enable_osnoise - enable osnoise tracer in the trace_instance */
Signature · 1 line
2.38.1
Steven Rostedt
maintainerRe: [PATCH V2 1/6] rtla/osnoise: Add helper functions to manipulate osnoise/options
Quoted 4 lines · On Tue, 31 Jan 2023 17:30:02 +0100 Daniel Bristot de Oliveira…
Add some helper functions to read and set the on/off osnoise/options. No usage in this patch.
In preparation for hwnoise tool.
Honestly, I don't see why patches 1-5 isn't a single patch. It's not that big of a change, and everything in 1-5 is to do what 5 does. Breaking it up this fine grain isn't helpful in reviewing, as I found that I had to apply 1-5 and then do a diff from where I started to make sense of any of it.
-- Steve
Quoted 7 lines
Signed-off-by Daniel Bristot de Oliveira <[email protected]>Cc Daniel Bristot de Oliveira <[email protected]>Cc Steven Rostedt <[email protected]>Cc Jonathan Corbet <[email protected]>---
Re: [PATCH V2 1/6] rtla/osnoise: Add helper functions to manipulate osnoise/options
Quoted 6 lines · Steven Rostedt
Quoted 3 lines · On Tue, 31 Jan 2023 17:30:02 +0100 Daniel Bristot de Oliveira…
Add some helper functions to read and set the on/off osnoise/options. No usage in this patch.
Quoted 6 lines
Quoted 1 line
In preparation for hwnoise tool.
Honestly, I don't see why patches 1-5 isn't a single patch. It's not that big of a change, and everything in 1-5 is to do what 5 does. Breaking it up this fine grain isn't helpful in reviewing, as I found that I had to apply 1-5 and then do a diff from where I started to make sense of any of it.
Maybe what is missing is a clear:
In preparation for hwnoise tool.
IMHO, it is easier to understand by using small "logical" pieces in preparation for the "conclusion." But I see your point, and it does not hurt :-).
I will reduce the number of patches.
-- Daniel
Quoted 11 lines
-- Steve
Quoted 7 lines
Signed-off-by Daniel Bristot de Oliveira <[email protected]>Cc Daniel Bristot de Oliveira <[email protected]>Cc Steven Rostedt <[email protected]>Cc Jonathan Corbet <[email protected]>---
Steven Rostedt
maintainerRe: [PATCH V2 1/6] rtla/osnoise: Add helper functions to manipulate osnoise/options
In reply to Daniel Bristot de OliveiraQuoted 4 lines · On Tue, 31 Jan 2023 17:30:02 +0100 Daniel Bristot de Oliveira…
tools/tracing/rtla/src/osnoise.c +4
-0+#define OSNOISE_OPTION (name, option_str) \ +static int osnoise_get_##name(struct osnoise_context *context) \ +{ \ + if (context->opt_##name != OSNOISE_OPTION_INIT_VAL) \
tools/tracing/rtla/src/osnoise.c void osnoise_put_tracing_thresh(struct osnoise_context *context) · line 796
Quoted 10 lines
tools/tracing/rtla/src/osnoise.c +10
-0+ return context->opt_##name; \ + \ + if (context->orig_opt_##name != OSNOISE_OPTION_INIT_VAL) \ + return context->orig_opt_##name; \ + \ + context->orig_opt_##name = osnoise_options_get_option(option_str); \ + \ + return context->orig_opt_##name; \ +} \ + \
What you could have done is not make this into a super macro (as there's only one instance of it). And then add a patch that turns it into this macro as the first patch of a series that adds another user.
Because I don't understand why this exists when it only has one user.
-- Steve
PATCH v2 2/6 rtla/osnoise: Add OSNOISE_IRQ_DISABLE option
In reply to Daniel Bristot de OliveiraAdd OSNOISE_IRQ_DISABLE to the set of possible options in the osnoise context. Do not use it yet.
In preparation for the hwnoise tool.
--- tools/tracing/rtla/src/osnoise.c | 6 ++++++ tools/tracing/rtla/src/osnoise.h | 6 ++++++ 2 files changed, 12 insertions(+)
tools/tracing/rtla/src/osnoise.c +6 -0
@@ -842,6 +842,8 @@ static void osnoise_put_##name(struct osnoise_context *context) \ context->orig_opt_##name = OSNOISE_OPTION_INIT_VAL; \ } +OSNOISE_OPTION (irq_disable,"OSNOISE_IRQ_DISABLE" ); +/* * enable_osnoise - enable osnoise tracer in the trace_instance */
@@ -906,6 +908,9 @@ struct osnoise_context *osnoise_context_alloc(void) context->orig_tracing_thresh =OSNOISE_OPTION_INIT_VAL ; context->tracing_thresh =OSNOISE_OPTION_INIT_VAL ; + context->orig_opt_irq_disable =OSNOISE_OPTION_INIT_VAL ; + context->opt_irq_disable =OSNOISE_OPTION_INIT_VAL ; +osnoise_get_context (context);return context;
@@ -932,6 +937,7 @@ void osnoise_put_context(struct osnoise_context *context)osnoise_put_timerlat_period_us (context);osnoise_put_print_stack (context);osnoise_put_tracing_thresh (context); +osnoise_put_irq_disable (context);free (context); }
tools/tracing/rtla/src/osnoise.h +6 -0
@@ -38,6 +38,10 @@ struct osnoise_context {/* -1 as init value because 0 is disabled */ long long orig_print_stack;long long print_stack; + +/* -1 as init value because 0 is off */ +int orig_opt_irq_disable; +int opt_irq_disable; };/*
@@ -79,6 +83,8 @@ void osnoise_restore_print_stack(struct osnoise_context *context);int osnoise_set_print_stack (struct osnoise_context *context,long long print_stack); +int osnoise_set_irq_disable (struct osnoise_context *context,bool onoff); +/* * osnoise_tool - osnoise based tool definition. */
Signature · 1 line
2.38.1
PATCH v2 3/6 rtla/osnoise: Add the mode abstraction
In reply to Daniel Bristot de OliveiraIn preparation to the hwnoise tool, add the mode abstraction to the osnoise tool, so it can have multiple operation modes.
--- tools/tracing/rtla/src/osnoise_top.c | 5 +++++ 1 file changed, 5 insertions(+)
tools/tracing/rtla/src/osnoise_top.c +5 -0
@@ -14,6 +14,10 @@#include "osnoise.h" #include "utils.h" +enum osnoise_mode { +MODE_OSNOISE =0 , +}; +/* * osnoise top parameters */
@@ -32,6 +36,7 @@ struct osnoise_top_params {int set_sched;struct sched_attr sched_param;struct trace_events *events; +enum osnoise_mode mode; };struct osnoise_top_cpu {
Signature · 1 line
2.38.1
PATCH v2 4/6 rtla/osnoise_top: Pass the params to the usage function
In reply to Daniel Bristot de OliveiraIn preparation to the hwnoise tool, pass the osnoise top params to the usage function so it can be adapted according to the mode of the tool.
Also make it static, as it is only used in this file.
--- tools/tracing/rtla/src/osnoise_top.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
tools/tracing/rtla/src/osnoise_top.c +10 -10
@@ -246,7 +246,7 @@ osnoise_print_stats(struct osnoise_top_params *params, struct osnoise_tool *top)/* * osnoise_top_usage - prints osnoise top usage message */ -void osnoise_top_usage (char *usage) +static void osnoise_top_usage (struct osnoise_top_params *params,char *usage) {int i;
@@ -350,7 +350,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)case 'c' : retval =parse_cpu_list (optarg, ¶ms->monitored_cpus);if (retval) -osnoise_top_usage ("\nInvalid -c cpu list\n" ); +osnoise_top_usage (params,"\nInvalid -c cpu list\n" ); params->cpus = optarg;break ;case 'D' :
@@ -359,7 +359,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)case 'd' : params->duration =parse_seconds_duration (optarg);if (!params->duration) -osnoise_top_usage ("Invalid -D duration\n" ); +osnoise_top_usage (params,"Invalid -D duration\n" );break ;case 'e' : tevent =trace_event_alloc (optarg);
@@ -375,17 +375,17 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)break ;case 'h' :case '?' : -osnoise_top_usage (NULL ); +osnoise_top_usage (params,NULL );break ;case 'p' : params->period =get_llong_from_str (optarg);if (params->period >10000000 ) -osnoise_top_usage ("Period longer than 10 s\n" ); +osnoise_top_usage (params,"Period longer than 10 s\n" );break ;case 'P' : retval =parse_prio (optarg, ¶ms->sched_param);if (retval ==-1 ) -osnoise_top_usage ("Invalid -P priority" ); +osnoise_top_usage (params,"Invalid -P priority" ); params->set_sched =1 ;break ;case 'q' :
@@ -394,7 +394,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)case 'r' : params->runtime =get_llong_from_str (optarg);if (params->runtime <100 ) -osnoise_top_usage ("Runtime shorter than 100 us\n" ); +osnoise_top_usage (params,"Runtime shorter than 100 us\n" );break ;case 's' : params->stop_us =get_llong_from_str (optarg);
@@ -420,7 +420,7 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)exit (EXIT_FAILURE ); } }else { -osnoise_top_usage ("--trigger requires a previous -e\n" ); +osnoise_top_usage (params,"--trigger requires a previous -e\n" ); }break ;case '1' :/* filter */
@@ -431,11 +431,11 @@ struct osnoise_top_params *osnoise_top_parse_args(int argc, char **argv)exit (EXIT_FAILURE ); } }else { -osnoise_top_usage ("--filter requires a previous -e\n" ); +osnoise_top_usage (params,"--filter requires a previous -e\n" ); }break ;default : -osnoise_top_usage ("Invalid option" ); +osnoise_top_usage (params,"Invalid option" ); } }
Signature · 1 line
2.38.1
PATCH v2 5/6 rtla: Add hwnoise tool
In reply to Daniel Bristot de OliveiraThe hwnoise tool is a special mode for the osnoise top tool.
hwnoise dispatches the osnoise tracer and displays a summary of the noise. The difference is that it runs the tracer with the OSNOISE_IRQ_DISABLE option set, thus only allowing only hardware-related noise, resulting in a simplified output. hwnoise has the same features of osnoise.
An example of the tool's output:
# rtla hwnoise -c 1-5 -T 1 -d 10m -q
Hardware-related Noise
duration: 0 00:10:00 | time is in us
CPU Period Runtime Noise % CPU Aval Max Noise Max Single HW NMI
1 #599 599000000 138 99.99997 3 3 4 74
2 #599 599000000 85 99.99998 3 3 4 75
3 #599 599000000 86 99.99998 4 3 6 75
4 #599 599000000 81 99.99998 4 4 2 75
5 #599 599000000 85 99.99998 2 2 2 75--- tools/tracing/rtla/Makefile | 2 + tools/tracing/rtla/src/osnoise.c | 6 +++ tools/tracing/rtla/src/osnoise.h | 1 + tools/tracing/rtla/src/osnoise_top.c | 59 +++++++++++++++++++++++++--- tools/tracing/rtla/src/rtla.c | 4 ++ 5 files changed, 66 insertions(+), 6 deletions(-)
tools/tracing/rtla/Makefile +2 -0
@@ -119,6 +119,8 @@ install: doc_install $(STRIP) $(DESTDIR)$(BINDIR)/rtla @test ! -f $(DESTDIR)$(BINDIR)/osnoise || rm $(DESTDIR)$(BINDIR)/osnoise ln -s rtla $(DESTDIR)$(BINDIR)/osnoise + @test ! -f $(DESTDIR)$(BINDIR)/hwnoise || rm $(DESTDIR)$(BINDIR)/hwnoise + ln -s rtla $(DESTDIR)$(BINDIR)/hwnoise @test ! -f $(DESTDIR)$(BINDIR)/timerlat || rm $(DESTDIR)$(BINDIR)/timerlat ln -s rtla $(DESTDIR)$(BINDIR)/timerlat
tools/tracing/rtla/src/osnoise.c +6 -0
@@ -1072,3 +1072,9 @@ int osnoise_main(int argc, char *argv[])osnoise_usage (1 );exit (1 ); } + +int hwnoise_main (int argc,char *argv[]) +{ +osnoise_top_main (argc, argv); +exit (0 ); +}
tools/tracing/rtla/src/osnoise.h +1 -0
@@ -103,3 +103,4 @@ struct osnoise_tool *osnoise_init_trace_tool(char *tracer);int osnoise_hist_main (int argc,char *argv[]);int osnoise_top_main (int argc,char **argv);int osnoise_main (int argc,char **argv); +int hwnoise_main (int argc,char **argv);
tools/tracing/rtla/src/osnoise_top.c +53 -6
121 lines not on this page · raw message
tools/tracing/rtla/src/rtla.c +4 -0
@@ -26,6 +26,7 @@ static void rtla_usage(int err)"" ," commands:" ," osnoise - gives information about the operating system noise (osnoise)" , +" hwnoise - gives information about hardware-related noise" ," timerlat - measures the timer irq and thread latency" ,"" , NULL,
@@ -47,6 +48,9 @@ int run_command(int argc, char **argv, int start_position)if (strcmp (argv[start_position],"osnoise" ) ==0 ) {osnoise_main (argc-start_position, &argv[start_position]); goto ran; + }else if (strcmp (argv[start_position],"hwnoise" ) ==0 ) { +hwnoise_main (argc-start_position, &argv[start_position]); + goto ran; }else if (strcmp (argv[start_position],"timerlat" ) ==0 ) {timerlat_main (argc-start_position, &argv[start_position]); goto ran;
Signature · 1 line
2.38.1
PATCH v2 6/6 Documentation/rtla: Add hwnoise man page
In reply to Daniel Bristot de OliveiraAdd a man page for the new rtla hwnoise tool, mostly based in the rtla osnoise top.
--- Documentation/tools/rtla/index.rst | 1 + Documentation/tools/rtla/rtla-hwnoise.rst | 107 ++++++++++++++++++++++ 2 files changed, 108 insertions(+) create mode 100644 Documentation/tools/rtla/rtla-hwnoise.rst
Documentation/tools/rtla/index.rst +1 -0
@@ -17,6 +17,7 @@ behavior on specific hardware. rtla-timerlat rtla-timerlat-hist rtla-timerlat-top + rtla-hwnoise .. only:: subproject and html
Documentation/tools/rtla/rtla-hwnoise.rst new file +107 -0
107 lines not on this page · raw message
Signature · 1 line
2.38.1
Re: [PATCH V2 6/6] Documentation/rtla: Add hwnoise man page
Quoted 103 lines · Daniel Bristot de Oliveira
Documentation/tools/rtla/rtla-hwnoise.rst new file +97
-0@@ -0,0 +1,107 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============ +rtla-hwnoise +============ +------------------------------------------ +Detect and quantify hardware-related noise +------------------------------------------ + +:Manual section: 1 + +SYNOPSIS +======== + +**rtla hwnoise** [*OPTIONS*] + +DESCRIPTION +=========== + +**rtla hwnoise** collects the periodic summary from the *osnoise* tracer +running with *interrupts disabled*. By disabling interrupts, and the scheduling +of threads as a consequence, only non-maskable interrupts and hardware-related +noise is allowed. + +The tool also allows the configurations of the *osnoise* tracer and the +collection of the tracer output. + +OPTIONS +======= +.. include:: common_osnoise_options.rst + +.. include:: common_top_options.rst + +.. include:: common_options.rst + +EXAMPLE +======= +In the example below, the **rtla hwnoise** tool is set to run on CPUs *1-7* +on a system with 8 cores/16 threads with hyper-threading enabled. + +The tool is set to detect any noise higher than *one microsecond*, +to run for *ten minutes*, displaying a summary of the report at the +end of the session:: + + # rtla hwnoise -c 1-7 -T 1 -d 10m -q + Hardware-related Noise + duration: 0 00:10:00 | time is in us + CPU Period Runtime Noise % CPU Aval Max Noise Max Single HW NMI + 1 #599 599000000 138 99.99997 3 3 4 74 + 2 #599 599000000 85 99.99998 3 3 4 75 + 3 #599 599000000 86 99.99998 4 3 6 75 + 4 #599 599000000 81 99.99998 4 4 2 75 + 5 #599 599000000 85 99.99998 2 2 2 75 + 6 #599 599000000 76 99.99998 2 2 0 75 + 7 #599 599000000 77 99.99998 3 3 0 75 + + +The first column shows the *CPU*, and the second column shows how many +*Periods* the tool ran during the session. The *Runtime* is the time +the tool effectively runs on the CPU. The *Noise* column is the sum of +all noise that the tool observed, and the *% CPU Aval* is the relation +between the *Runtime* and *Noise*. + +The *Max Noise* column is the maximum hardware noise the tool detected in a +single period, and the *Max Single* is the maximum single noise seen. + +The *HW* and *NMI* columns show the total number of *hardware* and *NMI* noise +occurrence observed by the tool. + +For example, *CPU 3* ran *599* periods of *1 second Runtime*. The CPU received +*86 us* of noise during the entire execution, leaving *99.99997 %* of CPU time +for the application. In the worst single period, the CPU caused *4 us* of +noise to the application, but it was certainly caused by more than one single +noise, as the *Max Single* noise was of *3 us*. The CPU has *HW noise,* at a +rate of *six occurrences*/*ten minutes*. The CPU also has *NMIs*, at a higher +frequency: around *seven per second*. + +The tool should report *0* hardware-related noise in the ideal situation. +For example, by disabling hyper-threading to remove the hardware noise, +and disabling the TSC watchdog to remove the NMI (it is possible to identify +this using tracing options of **rtla hwnoise**), it was possible to reach +the ideal situation in the same hardware:: + + # rtla hwnoise -c 1-7 -T 1 -d 10m -q + Hardware-related Noise + duration: 0 00:10:00 | time is in us + CPU Period Runtime Noise % CPU Aval Max Noise Max Single HW NMI + 1 #599 599000000 0 100.00000 0 0 0 0 + 2 #599 599000000 0 100.00000 0 0 0 0 + 3 #599 599000000 0 100.00000 0 0 0 0 + 4 #599 599000000 0 100.00000 0 0 0 0 + 5 #599 599000000 0 100.00000 0 0 0 0 + 6 #599 599000000 0 100.00000 0 0 0 0 + 7 #599 599000000 0 100.00000 0 0 0 0 + +SEE ALSO +========
Documentation/tools/rtla/rtla-hwnoise.rst line 107
Quoted 10 lines
Documentation/tools/rtla/rtla-hwnoise.rst new file +10
-0+ +**rtla-osnoise**\(1) + +Osnoise tracer documentation: <https://www.kernel.org/doc/html/latest/trace/osnoise-tracer.html> + +AUTHOR +====== +Written by Daniel Bristot de Oliveira <[email protected]> + +.. include:: common_appendix.rst
LGTM, thanks!
Signature · 1 line
An old man doll... just what I always wanted! - Clara