tracing: More updates for 5.15
Subsystems (6)
Steven Rostedt
maintainercover tracing: More updates for 5.15
More tracing updates for 5.15:
- Add migrate-disable counter to tracing header
- Fix error handling in event probes
- Fix missed unlock in osnoise in error path
- Fix merge issue with tools/bootconfig
- Clean up bootconfig data when init memory is removed
- Have kernel command lines override bootconfig options
- Increase field counts for synthetic events
- Have histograms dynamic allocate event elements to save space
- Fixes in testing and documentation
Artem Bityutskiy (1):
tracing: synth events: increase max fields count
Dan Carpenter (1):
tracing: Fix some alloc_event_probe() error handling bugs
Julio Faracco (1):
bootconfig: Fix missing return check of xbc_node_compose_key function
Masami Hiramatsu (5):
init: bootconfig: Remove all bootconfig data when the init memory is removed init/bootconfig: Reorder init parameter from bootconfig and cmdline
docs: bootconfig: Add how to use bootconfig for kernel parameters tools/bootconfig: Fix tracing_on option checking in ftrace2bconf.sh tools/bootconfig: Show whole test command for each test case
Qiang.Zhang (1):
tracing/osnoise: Fix missed cpus_read_unlock() in start_per_cpu_kthreads()
Steven Rostedt (VMware) (1):
selftests/ftrace: Exclude "(fault)" in testing add/remove eprobe events
Thomas Gleixner (1):
tracing: Add migrate-disabled counter to tracing output.
Tom Zanussi (1):
tracing: Dynamically allocate the per-elt hist_elt_data array
----
Documentation/admin-guide/bootconfig.rst | 39 +++++++++++++++++++++- init/main.c | 37 ++++++++++++++------ kernel/trace/trace.c | 26 +++++++++++---- kernel/trace/trace_eprobe.c | 5 +-- kernel/trace/trace_events.c | 1 + kernel/trace/trace_events_hist.c | 14 ++++++-- kernel/trace/trace_osnoise.c | 6 ++-- kernel/trace/trace_output.c | 11 ++++-- kernel/trace/trace_synth.h | 2 +- tools/bootconfig/main.c | 4 ++- tools/bootconfig/scripts/ftrace2bconf.sh | 4 +-- tools/bootconfig/test-bootconfig.sh | 4 +-- .../ftrace/test.d/dynevent/add_remove_eprobe.tc | 2 +- 13 files changed, 119 insertions(+), 36 deletions(-)
Steven Rostedt
maintainerPATCH 1/12 tracing: Add migrate-disabled counter to tracing output.
From: Thomas Gleixner <[email protected]>
migrate_disable() forbids task migration to another CPU. It is available since v5.11 and has already users such as highmem or BPF. It is useful to observe this task state in tracing which already has other states like the preemption counter.
Instead of adding the migrate disable counter as a new entry to struct trace_entry, which would extend the whole struct by four bytes, it is squashed into the preempt-disable counter. The lower four bits represent the preemption counter, the upper four bits represent the migrate disable counter. Both counter shouldn't exceed 15 but if they do, there is a safety net which caps the value at 15.
Add the migrate-disable counter to the trace entry so it shows up in the trace. Due to the users mentioned above, it is already possible to observe it:
| bash-1108 [000] ...21 73.950578: rss_stat: mm_id=2213312838 curr=0 type=MM_ANONPAGES size=8192B | bash-1108 [000] d..31 73.951222: irq_disable: caller=flush_tlb_mm_range+0x115/0x130 parent=ptep_clear_flush+0x42/0x50 | bash-1108 [000] d..31 73.951222: tlb_flush: pages:1 reason:local mm shootdown (3)
The last value is the migrate-disable counter.
Things that popped up:
- trace_print_lat_context() does not print the migrate counter. Not sure if it should. It is used in "verbose" mode and uses 8 digits and I'm not sure ther is something processing the value.
- trace_define_common_fields() now defines a different variable. This probably breaks things. No ide what to do in order to preserve the old behaviour. Since this is used as a filter it should be split somehow to be able to match both nibbles here.
--- kernel/trace/trace.c | 26 +++++++++++++++++++------- kernel/trace/trace_events.c | 1 + kernel/trace/trace_output.c | 11 ++++++++--- 3 files changed, 28 insertions(+), 10 deletions(-)
kernel/trace/trace.c +19 -7
@@ -2603,6 +2603,15 @@ enum print_line_t trace_handle_return(struct trace_seq *s) }EXPORT_SYMBOL_GPL (trace_handle_return); +static unsigned short migration_disable_value (void ) +{ +#if defined(CONFIG_SMP ) +return current->migration_disabled; +#else +return 0 ; +#endif +} +unsigned int tracing_gen_ctx_irq_test (unsigned int irqs_status) {unsigned int trace_flags = irqs_status;
@@ -2621,7 +2630,8 @@ unsigned int tracing_gen_ctx_irq_test(unsigned int irqs_status) trace_flags |=TRACE_FLAG_NEED_RESCHED ;if (test_preempt_need_resched ()) trace_flags |=TRACE_FLAG_PREEMPT_RESCHED ; -return (trace_flags <<16 ) | (pc &0xff ); + return (trace_flags << 16) | (min_t(unsigned int, pc & 0xff, 0xf)) | + (min_t(unsigned int, migration_disable_value(), 0xf)) << 4; } struct ring_buffer_event *
@@ -4189,9 +4199,10 @@ static void print_lat_help_header(struct seq_file *m)"# | / _----=> need-resched \n" "# || / _---=> hardirq/softirq \n" "# ||| / _--=> preempt-depth \n" -"# |||| / delay \n" -"# cmd pid ||||| time | caller \n" - "# \\ / ||||| \\ | / \n"); +"# |||| / _-=> migrate-disable \n" +"# ||||| / delay \n" +"# cmd pid |||||| time | caller \n" + "# \\ / |||||| \\ | / \n"); }static void print_event_info (struct array_buffer *buf,struct seq_file *m)
@@ -4229,9 +4240,10 @@ static void print_func_help_header_irq(struct array_buffer *buf, struct seq_fileseq_printf (m,"# %.*s / _----=> need-resched\n" , prec, space);seq_printf (m,"# %.*s| / _---=> hardirq/softirq\n" , prec, space);seq_printf (m,"# %.*s|| / _--=> preempt-depth\n" , prec, space); -seq_printf (m,"# %.*s||| / delay\n" , prec, space); -seq_printf (m,"# TASK-PID %.*s CPU# |||| TIMESTAMP FUNCTION\n" , prec," TGID " ); -seq_printf (m,"# | | %.*s | |||| | |\n" , prec," | " ); +seq_printf (m,"# %.*s||| / _-=> migrate-disable\n" , prec, space); +seq_printf (m,"# %.*s|||| / delay\n" , prec, space); +seq_printf (m,"# TASK-PID %.*s CPU# ||||| TIMESTAMP FUNCTION\n" , prec," TGID " ); +seq_printf (m,"# | | %.*s | ||||| | |\n" , prec," | " ); }void
kernel/trace/trace_events.c +1 -0
@@ -181,6 +181,7 @@ static int trace_define_common_fields(void) __common_field(unsigned short, type); __common_field(unsigned char, flags); +/* Holds both preempt_count and migrate_disable */ __common_field(unsigned char, preempt_count);__common_field (int, pid);
kernel/trace/trace_output.c +8 -3
@@ -492,8 +492,13 @@ int trace_print_lat_fmt(struct trace_seq *s, struct trace_entry *entry)trace_seq_printf (s,"%c%c%c" , irqs_off, need_resched, hardsoft_irq); -if (entry->preempt_count) -trace_seq_printf (s,"%x" , entry->preempt_count); +if (entry->preempt_count &0xf ) +trace_seq_printf (s,"%x" , entry->preempt_count &0xf ); +else +trace_seq_putc (s,'.' ); + +if (entry->preempt_count &0xf0 ) +trace_seq_printf (s,"%x" , entry->preempt_count >>4 );else trace_seq_putc (s,'.' );
@@ -656,7 +661,7 @@ int trace_print_lat_context(struct trace_iterator *iter)trace_seq_printf ( s,"%16s %7d %3d %d %08x %08lx " , comm, entry->pid, iter->cpu, entry->flags, - entry->preempt_count, iter->idx); + entry->preempt_count &0xf , iter->idx); }else {lat_print_generic (s, entry, iter->cpu); }
Signature · 1 line
2.32.0
Steven Rostedt
maintainerPATCH 2/12 tracing: Fix some alloc_event_probe() error handling bugs
In reply to Steven RostedtFrom: Dan Carpenter <[email protected]>
There are two bugs in this code. First, if the kzalloc() fails it leads to a NULL dereference of "ep" on the next line. Second, if the alloc_event_probe() function returns an error then it leads to an error pointer dereference in the caller.
--- kernel/trace/trace_eprobe.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/trace/trace_eprobe.c +3 -2
@@ -151,7 +151,7 @@ static struct trace_eprobe *alloc_event_probe(const char *group, ep =kzalloc (struct_size (ep, tp.args, nargs),GFP_KERNEL );if (!ep) { -trace_event_put_ref (ep->event); +trace_event_put_ref (event); goto error; } ep->event = event;
@@ -851,7 +851,8 @@ static int __trace_eprobe_create(int argc, const char *argv[]) ret =PTR_ERR (ep);/* This must return -ENOMEM, else there is a bug */ WARN_ON_ONCE (ret != -ENOMEM ); - goto error;/* We know ep is not allocated */ + ep =NULL ; + goto error; } argc -=2 ; argv +=2 ;
Signature · 1 line
2.32.0
Steven Rostedt
maintainerPATCH 3/12 tracing/osnoise: Fix missed cpus_read_unlock() in start_per_cpu_kthreads()
In reply to Steven RostedtFrom: "Qiang.Zhang" <[email protected]>
When start_kthread() return error, the cpus_read_unlock() need to be called.
--- kernel/trace/trace_osnoise.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/trace/trace_osnoise.c +3 -3
@@ -1548,7 +1548,7 @@ static int start_kthread(unsigned int cpu)static int start_per_cpu_kthreads (struct trace_array *tr) {struct cpumask *current_mask = &save_cpumask; -int retval; +int retval =0 ;int cpu;cpus_read_lock ();
@@ -1568,13 +1568,13 @@ static int start_per_cpu_kthreads(struct trace_array *tr) retval =start_kthread (cpu);if (retval) {stop_per_cpu_kthreads (); -return retval; +break ; } }cpus_read_unlock (); -return 0 ; +return retval; }#ifdef CONFIG_HOTPLUG_CPU
Signature · 1 line
2.32.0
Steven Rostedt
maintainerPATCH 4/12 init: bootconfig: Remove all bootconfig data when the init memory is removed
In reply to Steven RostedtFrom: Masami Hiramatsu <[email protected]>
Since the bootconfig is used only in the init functions, it doesn't need to keep the data after boot. Free it when the init memory is removed.
--- init/main.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
init/main.c +12 -2
@@ -468,7 +468,12 @@ static void __init setup_boot_config(void) return; } -#else +static void __init exit_boot_config (void ) +{ +xbc_destroy_all (); +} + +#else /* !CONFIG_BOOT_CONFIG */ static void __init setup_boot_config (void ) {
@@ -481,7 +486,11 @@ static int __init warn_bootconfig(char *str)pr_warn ("WARNING: 'bootconfig' found on the kernel command line but CONFIG_BOOT_CONFIG is not set.\n" ); return 0; } -#endif + +#define exit_boot_config () do {} while (0) + +#endif /* CONFIG_BOOT_CONFIG */ +early_param ("bootconfig" , warn_bootconfig);/* Change NUL term back to "=", to make "param" the whole string. */
@@ -1493,6 +1502,7 @@ static int __ref kernel_init(void *unused)kprobe_free_init_mem ();ftrace_free_init_mem ();kgdb_free_init_mem (); +exit_boot_config ();free_initmem ();mark_readonly ();
Signature · 1 line
2.32.0
Steven Rostedt
maintainerPATCH 5/12 init/bootconfig: Reorder init parameter from bootconfig and cmdline
In reply to Steven RostedtFrom: Masami Hiramatsu <[email protected]>
Reorder the init parameters from bootconfig and kernel cmdline so that the kernel cmdline always be the last part of the parameters as below.
" -- "[bootconfig init params][cmdline init params]
This change will help us to prevent that bootconfig init params overwrite the init params which user gives in the command line.
--- init/main.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)
init/main.c +14 -9
@@ -153,10 +153,10 @@ static char *extra_init_args;#ifdef CONFIG_BOOT_CONFIG /* Is bootconfig on command line? */ static bool bootconfig_found; -static bool initargs_found; +static size_t initargs_offs;#else # define bootconfig_found false -# define initargs_found false +# define initargs_offs 0#endif static char *execute_command;
@@ -422,9 +422,9 @@ static void __init setup_boot_config(void)if (IS_ERR (err) || !bootconfig_found)return ; -/* parse_args() stops at '--' and returns an address */ +/* parse_args() stops at the next param of '--' and returns an address */ if (err) - initargs_found = true; + initargs_offs = err - tmp_cmdline;if (!data) {pr_err ("'bootconfig' found on command line, but no bootconfig found\n" );
@@ -655,16 +655,21 @@ static void __init setup_command_line(char *command_line)* Append supplemental init boot args to saved_command_line * so that user can check what command line options passed * to init. +* The order should always be +* " -- "[bootconfig init-param][cmdline init-param] */ - len =strlen (saved_command_line); -if (initargs_found) { - saved_command_line[len++] =' ' ; +if (initargs_offs) { + len = xlen + initargs_offs; +strcpy (saved_command_line + len, extra_init_args); + len += ilen -4 ;/* strlen(extra_init_args) */ +strcpy (saved_command_line + len, + boot_command_line + initargs_offs -1 ); }else { + len =strlen (saved_command_line);strcpy (saved_command_line + len," -- " ); len +=4 ; +strcpy (saved_command_line + len, extra_init_args); } - -strcpy (saved_command_line + len, extra_init_args); } }
Signature · 1 line
2.32.0
Steven Rostedt
maintainerPATCH 6/12 docs: bootconfig: Add how to use bootconfig for kernel parameters
In reply to Steven RostedtFrom: Masami Hiramatsu <[email protected]>
Add a section to describe how to use the bootconfig for specifying kernel and init parameters. This is an important section because it is the reason why this document is under the admin-guide.
--- Documentation/admin-guide/bootconfig.rst | 39 +++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)
Documentation/admin-guide/bootconfig.rst +38 -1
@@ -178,7 +178,7 @@ update the boot loader and the kernel image itself as long as the boot loader passes the correct initrd file size. If by any chance, the boot loader passes a longer size, the kernel fails to find the bootconfig data. -To do this operation, Linux kernel provides "bootconfig" command under +To do this operation, Linux kernel provides ``bootconfig`` command under tools/bootconfig, which allows admin to apply or delete the config file to/from initrd image. You can build it by the following command::
@@ -196,6 +196,43 @@ To remove the config from the image, you can use -d option as below:: Then add "bootconfig" on the normal kernel command line to tell the kernel to look for the bootconfig at the end of the initrd file. + +Kernel parameters via Boot Config +================================= + +In addition to the kernel command line, the boot config can be used for +passing the kernel parameters. All the key-value pairs under ``kernel`` +key will be passed to kernel cmdline directly. Moreover, the key-value +pairs under ``init`` will be passed to init process via the cmdline. +The parameters are concatinated with user-given kernel cmdline string +as the following order, so that the command line parameter can override +bootconfig parameters (this depends on how the subsystem handles parameters +but in general, earlier parameter will be overwritten by later one.):: + + [bootconfig params][cmdline params] -- [bootconfig init params][cmdline init params] + +Here is an example of the bootconfig file for kernel/init parameters.:: + + kernel { + root = 01234567-89ab-cdef-0123-456789abcd + } + init { + splash + } + +This will be copied into the kernel cmdline string as the following:: + + root="01234567-89ab-cdef-0123-456789abcd" -- splash + +If user gives some other command line like,:: + + ro bootconfig -- quiet + +The final kernel cmdline will be the following:: + + root="01234567-89ab-cdef-0123-456789abcd" ro bootconfig -- splash quiet + + Config File Limitation ======================
Signature · 1 line
2.32.0
Steven Rostedt
maintainerPATCH 7/12 tools/bootconfig: Fix tracing_on option checking in ftrace2bconf.sh
In reply to Steven RostedtFrom: Masami Hiramatsu <[email protected]>
Since tracing_on indicates only "1" (default) or "0", ftrace2bconf.sh only need to check the value is "0".
--- tools/bootconfig/scripts/ftrace2bconf.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/bootconfig/scripts/ftrace2bconf.sh +2 -2
@@ -239,8 +239,8 @@ instance_options() { # [instance-name] emit_kv $PREFIX.cpumask = $val fi val=`cat $INSTANCE/tracing_on` - if [ `echo $val | sed -e s/f//g`x != x ]; then - emit_kv $PREFIX.tracing_on = $val + if [ "$val" = "0" ]; then + emit_kv $PREFIX.tracing_on = 0 fi val=`cat $INSTANCE/current_tracer`
Signature · 1 line
2.32.0
Steven Rostedt
maintainerPATCH 8/12 bootconfig: Fix missing return check of xbc_node_compose_key function
In reply to Steven RostedtFrom: Julio Faracco <[email protected]>
The function xbc_show_list should handle the keys during the composition. Even the errors returned by the compose function. Instead of removing the ret variable, it should save the value and show the exact error. This missing variable is causing a compilation issue also.
--- tools/bootconfig/main.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tools/bootconfig/main.c +3 -1
@@ -111,9 +111,11 @@ static void xbc_show_list(void)char key[XBC_KEYLEN_MAX ];struct xbc_node *leaf;const char *val; +int ret;xbc_for_each_key_value (leaf, val) { -if (xbc_node_compose_key (leaf, key,XBC_KEYLEN_MAX ) <0 ) { + ret =xbc_node_compose_key (leaf, key,XBC_KEYLEN_MAX ); +if (ret <0 ) {fprintf (stderr,"Failed to compose key %d\n" , ret);break ; }
Signature · 1 line
2.32.0
Steven Rostedt
maintainerRe: [for-next][PATCH 08/12] bootconfig: Fix missing return check of xbc_node_compose_key function
Quoted 7 lines · On Wed, 08 Sep 2021 15:18:59 -0400 Steven Rostedt <[email protected]> wrote:
From: Julio Faracco <[email protected]>
The function
xbc_show_list shouldhandle the keys during the composition. Even the errors returned by the compose function. Instead of removing theretvariable, it should save the value and show the exact error. This missing variable is causing a compilation issue also.
Quoted 7 lines
Fixes e5efaeb8a8f5 ("bootconfig: Support mixing a value and subkeys under a key")Signed-off-by Julio Faracco <[email protected]>Acked-by Masami Hiramatsu <[email protected]>
Oops, a typo. That should be: [email protected] or [email protected]. There is no "vgar" ;-)
I'll fix this up.
-- Steve
Quoted 1 line
Signed-off-by Steven Rostedt (VMware) <[email protected]>
Steven Rostedt
maintainerPATCH 9/12 tools/bootconfig: Show whole test command for each test case
In reply to Steven RostedtFrom: Masami Hiramatsu <[email protected]>
Show whole test command instead of only the 3rd argument. This will clear to show what will be actually tested by each test case.
--- tools/bootconfig/test-bootconfig.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/bootconfig/test-bootconfig.sh +2 -2
@@ -26,7 +26,7 @@ trap cleanup EXIT TERM NO=1 xpass() { # pass test command - echo "test case $NO ($3)... " + echo "test case $NO ($*)... " if ! ($@ && echo "\t\t[OK]"); then echo "\t\t[NG]"; NG=$((NG + 1)) fi
@@ -34,7 +34,7 @@ xpass() { # pass test command } xfail() { # fail test command - echo "test case $NO ($3)... " + echo "test case $NO ($*)... " if ! (! $@ && echo "\t\t[OK]"); then echo "\t\t[NG]"; NG=$((NG + 1)) fi
Signature · 1 line
2.32.0
Steven Rostedt
maintainerPATCH 10/12 tracing: synth events: increase max fields count
In reply to Steven RostedtFrom: Artem Bityutskiy <[email protected]>
Sometimes it is useful to construct larger synthetic trace events. Increase 'SYNTH_FIELDS_MAX' (maximum number of fields in a synthetic event) from 32 to 64.
--- kernel/trace/trace_synth.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace_synth.h +1 -1
@@ -5,7 +5,7 @@#include "trace_dynevent.h" #define SYNTH_SYSTEM "synthetic" -#define SYNTH_FIELDS_MAX 32 +#define SYNTH_FIELDS_MAX 64#define STR_VAR_LEN_MAX MAX_FILTER_STR_VAL/* must be multiple of sizeof(u64) */
Signature · 1 line
2.32.0
Steven Rostedt
maintainerPATCH 11/12 tracing: Dynamically allocate the per-elt hist_elt_data array
In reply to Steven RostedtFrom: Tom Zanussi <[email protected]>
Setting the hist_elt_data.field_var_str[] array unconditionally to a size of SYNTH_FIELD_MAX elements wastes space unnecessarily. The actual number of elements needed can be calculated at run-time instead.
In most cases, this will save a lot of space since it's a per-elt array which isn't normally close to being full. It also allows us to increase SYNTH_FIELD_MAX without worrying about even more wastage when we do that.
--- kernel/trace/trace_events_hist.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
kernel/trace/trace_events_hist.c +12 -2
@@ -508,7 +508,8 @@ struct track_data {struct hist_elt_data {char *comm;u64 *var_ref_vals; -char *field_var_str[SYNTH_FIELDS_MAX ]; +char **field_var_str; +int n_field_var_str; };struct snapshot_context {
@@ -1401,9 +1402,11 @@ static void hist_elt_data_free(struct hist_elt_data *elt_data) {unsigned int i; -for (i =0 ; i <SYNTH_FIELDS_MAX ; i++) +for (i =0 ; i < elt_data->n_field_var_str; i++)kfree (elt_data->field_var_str[i]); +kfree (elt_data->field_var_str); +kfree (elt_data->comm);kfree (elt_data); }
@@ -1451,6 +1454,13 @@ static int hist_trigger_elt_data_alloc(struct tracing_map_elt *elt) size =STR_VAR_LEN_MAX ; + elt_data->field_var_str =kcalloc (n_str,sizeof (char *),GFP_KERNEL ); +if (!elt_data->field_var_str) { +hist_elt_data_free (elt_data); +return -EINVAL ; + } + elt_data->n_field_var_str = n_str; +for (i =0 ; i < n_str; i++) { elt_data->field_var_str[i] =kzalloc (size,GFP_KERNEL );if (!elt_data->field_var_str[i]) {
Signature · 1 line
2.32.0
Steven Rostedt
maintainerPATCH 12/12 selftests/ftrace: Exclude "(fault)" in testing add/remove eprobe events
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
The original test for adding and removing eprobes used synthetic events and retrieved the filename from the open system call at the end of the system call. This would allow it to always be loaded into the page tables when accessed.
Masami suggested that the test was too complex for just testing add and remove, so it was changed to test just adding and removing an event probe on top of the start of the open system call event. Now it is possible that the filename will not be loaded into memory at the time the eprobe is triggered, and will result in "(fault)" being displayed in the event. This causes the test to fail.
Account for "(fault)" also being one of the values of the filename field of the event probe.
--- .../selftests/ftrace/test.d/dynevent/add_remove_eprobe.tc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/ftrace/test.d/dynevent/add_remove_eprobe.tc +1 -1
@@ -22,7 +22,7 @@ ls echo 0 > events/eprobes/$EPROBE/enable content=`grep '^ *ls-' trace | grep 'file='` -nocontent=`grep '^ *ls-' trace | grep 'file=' | grep -v -e '"/' -e '"."'` || true +nocontent=`grep '^ *ls-' trace | grep 'file=' | grep -v -e '"/' -e '"."' -e '(fault)' ` || true if [ -z "$content" ]; then exit_fail
Signature · 1 line
2.32.0