Skip to contentSkip to conversation

bpftool: Add inline annotations when dumping program CFGs

PATCH Superseded Quentin Monnet 2 participants 11 messages Mar 24, 2023 – Mar 27, 2023

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover bpftool: Add inline annotations when dumping program CFGs

This set contains some improvements for bpftool's "visual" program dump option, which produces the control flow graph in a DOT format. The main objective is to add support for inline annotations on such graphs, so that we can have the C source code for the program showing up alongside the instructions, when available. The last commits also make it possible to display the line numbers or the bare opcodes in the graph, as supported by regular program dumps.

Quentin Monnet (5):
bpftool: Fix documentation about line info display for prog dumps
bpftool: Fix bug for long instructions in program CFG dumps
bpftool: Support inline annotations when dumping the CFG of a program
bpftool: Support "opcodes", "linum", "visual" simultaneously
bpftool: Support printing opcodes and source file references in CFG

 .../bpftool/Documentation/bpftool-prog.rst    | 18 ++---
 tools/bpf/bpftool/bash-completion/bpftool     | 18 +++--
 tools/bpf/bpftool/btf_dumper.c                | 51 ++++++++++++
 tools/bpf/bpftool/cfg.c                       | 29 +++----
 tools/bpf/bpftool/cfg.h                       |  5 +-
 tools/bpf/bpftool/main.h                      |  2 +
 tools/bpf/bpftool/prog.c                      | 78 ++++++++++---------
 tools/bpf/bpftool/xlated_dumper.c             | 52 ++++++++++++-
 tools/bpf/bpftool/xlated_dumper.h             |  3 +-
 9 files changed, 184 insertions(+), 72 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/5 bpftool: Fix documentation about line info display for prog dumps

The documentation states that when line_info is available when dumping a program, the source line will be displayed "by default". There is no notion of "default" here: the line is always displayed if available, there is no way currently to turn it off.

In the next sentence, the documentation states that if "linum" is used on the command line, the relevant filename, line, and column will be displayed "on top of the source line". This is incorrect, as they are currently displayed on the right side of the source line (or on top of the eBPF instruction, not the source).

This commit fixes the documentation to address these points.

---
 tools/bpf/bpftool/Documentation/bpftool-prog.rst | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
tools/bpf/bpftool/Documentation/bpftool-prog.rst +4 -6
index 14de72544995..06d1e4314406 100644
@@ -106,9 +106,8 @@ DESCRIPTION
 		  CFG in DOT format, on standard output.
 
 		  If the programs have line_info available, the source line will
-		  be displayed by default.  If **linum** is specified,
-		  the filename, line number and line column will also be
-		  displayed on top of the source line.
+		  be displayed.  If **linum** is specified, the filename, line
+		  number and line column will also be displayed.
 
 	**bpftool prog dump jited**  *PROG* [{ **file** *FILE* | **opcodes** | **linum** }]
 		  Dump jited image (host machine code) of the program.
@@ -120,9 +119,8 @@ DESCRIPTION
 		  **opcodes** controls if raw opcodes will be printed.
 
 		  If the prog has line_info available, the source line will
-		  be displayed by default.  If **linum** is specified,
-		  the filename, line number and line column will also be
-		  displayed on top of the source line.
+		  be displayed.  If **linum** is specified, the filename, line
+		  number and line column will also be displayed.
 
 	**bpftool prog pin** *PROG* *FILE*
 		  Pin program *PROG* as *FILE*.
Signature · 1 line
2.34.1
#2 lore raw

PATCH 2/5 bpftool: Fix bug for long instructions in program CFG dumps

In reply to Quentin Monnet

When dumping the control flow graphs for programs using the 16-byte long load instruction, we need to skip the second part of this instruction when looking for the next instruction to process. Otherwise, we end up printing "BUG_ld_00" from the kernel disassembler in the CFG.

Fixes efcef17a6d65 ("tools: bpftool: generate .dot graph from CFG information")
---
 tools/bpf/bpftool/xlated_dumper.c | 7 +++++++
 1 file changed, 7 insertions(+)
tools/bpf/bpftool/xlated_dumper.c +7 -0
index 6fe3134ae45d..3daa05d9bbb7 100644
@@ -372,8 +372,15 @@ void dump_xlated_for_graph(struct dump_data *dd, void *buf_start, void *buf_end,
 	struct bpf_insn *insn_start = buf_start;
 	struct bpf_insn *insn_end = buf_end;
 	struct bpf_insn *cur = insn_start;
+	bool double_insn = false;
 
 	for (; cur <= insn_end; cur++) {
+		if (double_insn) {
+			double_insn = false;
+			continue;
+		}
+		double_insn = cur->code == (BPF_LD | BPF_IMM | BPF_DW);
+
 		printf("% 4d: ", (int)(cur - insn_start + start_idx));
 		print_bpf_insn(&cbs, cur, true);
 		if (cur != insn_end)
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH bpf-next 2/5] bpftool: Fix bug for long instructions in program CFG dumps

Quoted 31 lines · Quentin Monnet

When dumping the control flow graphs for programs using the 16-byte long load instruction, we need to skip the second part of this instruction when looking for the next instruction to process. Otherwise, we end up printing "BUG_ld_00" from the kernel disassembler in the CFG.

Fixes efcef17a6d65 ("tools: bpftool: generate .dot graph from CFG information")
---
  tools/bpf/bpftool/xlated_dumper.c | 7 +++++++
  1 file changed, 7 insertions(+)
tools/bpf/bpftool/xlated_dumper.c +7 -0
index 6fe3134ae45d..3daa05d9bbb7 100644
@@ -372,8 +372,15 @@ void dump_xlated_for_graph(struct dump_data *dd,  
 void *buf_start, void *buf_end,
  	struct bpf_insn *insn_start = buf_start;
  	struct bpf_insn *insn_end = buf_end;
  	struct bpf_insn *cur = insn_start;
+	bool double_insn = false;
 
  	for (; cur <= insn_end; cur++) {
+		if (double_insn) {
+			double_insn = false;
+			continue;
+		}
+		double_insn = cur->code == (BPF_LD | BPF_IMM | BPF_DW);
+
  		printf("% 4d: ", (int)(cur - insn_start + start_idx));
  		print_bpf_insn(&cbs, cur, true);
Quoted 1 line

if (cur != insn_end)

Any reason not to do the following here instead?

	if (cur->code == (BPF_LD | BPF_IMM | BPF_DW))
		cur++;
Quoted 2 lines

--
2.34.1

#4 lore raw

Re: [PATCH bpf-next 2/5] bpftool: Fix bug for long instructions in program CFG dumps

2023-03-24 19:54 UTC-0700 ~ Stanislav Fomichev <[email protected]>

Quoted 36 lines · Quentin Monnet
Quoted 33 lines

When dumping the control flow graphs for programs using the 16-byte long load instruction, we need to skip the second part of this instruction when looking for the next instruction to process. Otherwise, we end up printing "BUG_ld_00" from the kernel disassembler in the CFG.

Fixes efcef17a6d65 ("tools: bpftool: generate .dot graph from CFG

information")

---
  tools/bpf/bpftool/xlated_dumper.c | 7 +++++++

  1 file changed, 7 insertions(+)
tools/bpf/bpftool/xlated_dumper.c +7 -0
index 6fe3134ae45d..3daa05d9bbb7 100644
@@ -372,8 +372,15 @@ void dump_xlated_for_graph(struct dump_data *dd,
 void *buf_start, void *buf_end,
       struct bpf_insn *insn_start = buf_start;
       struct bpf_insn *insn_end = buf_end;
       struct bpf_insn *cur = insn_start;
+    bool double_insn = false;
 
       for (; cur <= insn_end; cur++) {
+        if (double_insn) {
+            double_insn = false;
+            continue;
+        }
+        double_insn = cur->code == (BPF_LD | BPF_IMM | BPF_DW);
+
           printf("% 4d: ", (int)(cur - insn_start + start_idx));
           print_bpf_insn(&cbs, cur, true);
Quoted 1 line

          if (cur != insn_end)

Quoted 4 lines

Any reason not to do the following here instead?

    if (cur->code == (BPF_LD | BPF_IMM | BPF_DW))
        cur++;

Yes, I reuse double_insn in patch 5 to print the last 8 raw bytes of the instruction if "opcodes" is passed. I could make it work with your suggestion too, but would likely have to test "cur->code" a second time, I'm not sure we'd gain in readability overall. I'll keep the variable for v2.

#5 lore raw

PATCH 3/5 bpftool: Support inline annotations when dumping the CFG of a program

In reply to Quentin Monnet

We support dumping the control flow graph of loaded programs to the DOT format with bpftool, but so far this feature wouldn't display the source code lines available through BTF along with the eBPF bytecode. Let's add support for these annotations, to make it easier to read the graph.

In prog.c, we move the call to dump_xlated_cfg() in order to pass and use the full struct dump_data, instead of creating a minimal one in draw_bb_node().

We pass the pointer to this struct down to dump_xlated_for_graph() in xlated_dumper.c, where most of the logics is added. We deal with BTF mostly like we do for plain or JSON output, except that we cannot use a "nr_skip" value to skip a given number of linfo records (we don't process the BPF instructions linearly, and apart from the root of the graph we don't know how many records we should skip, so we just store the last linfo and make sure the new one we find is different before printing it).

When printing the source instructions to the label of a DOT graph node, there are a few subtleties to address. We want some special newline markers, and there are some characters that we must escape. To deal with them, we introduce a new dedicated function btf_dump_linfo_dotlabel() in btf_dumper.c. We'll reuse this function in a later commit to format the filepath, line, and column references as well.

---
 tools/bpf/bpftool/btf_dumper.c    | 34 +++++++++++++++++++++++++++++++
 tools/bpf/bpftool/cfg.c           | 23 +++++++++------------
 tools/bpf/bpftool/cfg.h           |  4 +++-
 tools/bpf/bpftool/main.h          |  2 ++
 tools/bpf/bpftool/prog.c          | 17 +++++++---------
 tools/bpf/bpftool/xlated_dumper.c | 32 ++++++++++++++++++++++++++++-
 6 files changed, 87 insertions(+), 25 deletions(-)
tools/bpf/bpftool/btf_dumper.c +34 -0
index e7f6ec3a8f35..504d7c75cc27 100644
@@ -821,3 +821,37 @@ void btf_dump_linfo_json(const struct btf *btf,
 					BPF_LINE_INFO_LINE_COL(linfo->line_col));
 	}
 }
+
+static void dotlabel_puts(const char *s)
+{
+	FILE *stream = stdout;
+
+	for (; *s; ++s) {
+		switch (*s) {
+		case '\\':
+		case '"':
+		case '{':
+		case '}':
+		case '>':
+		case '|':
+			fputc('\\', stream);
+			__fallthrough;
+		default:
+			fputc(*s, stream);
+		}
+	}
+}
+
+void btf_dump_linfo_dotlabel(const struct btf *btf,
+			     const struct bpf_line_info *linfo)
+{
+	const char *line = btf__name_by_offset(btf, linfo->line_off);
+
+	if (!line)
+		return;
+	line = ltrim(line);
+
+	printf("; ");
+	dotlabel_puts(line);
+	printf("\\l\\\n");
+}
tools/bpf/bpftool/cfg.c +10 -13
index 1951219a9af7..9fdc1f0cdd6e 100644
@@ -380,7 +380,8 @@ static void cfg_destroy(struct cfg *cfg)
 	}
 }
 
-static void draw_bb_node(struct func_node *func, struct bb_node *bb)
+static void
+draw_bb_node(struct func_node *func, struct bb_node *bb, struct dump_data *dd)
 {
 	const char *shape;
 
@@ -398,13 +399,9 @@ static void draw_bb_node(struct func_node *func, struct bb_node *bb)
 		printf("EXIT");
 	} else {
 		unsigned int start_idx;
-		struct dump_data dd = {};
-
-		printf("{");
-		kernel_syms_load(&dd);
+		printf("{\\\n");
 		start_idx = bb->head - func->start;
-		dump_xlated_for_graph(&dd, bb->head, bb->tail, start_idx);
-		kernel_syms_destroy(&dd);
+		dump_xlated_for_graph(dd, bb->head, bb->tail, start_idx);
 		printf("}");
 	}
 
@@ -430,12 +427,12 @@ static void draw_bb_succ_edges(struct func_node *func, struct bb_node *bb)
 	}
 }
 
-static void func_output_bb_def(struct func_node *func)
+static void func_output_bb_def(struct func_node *func, struct dump_data *dd)
 {
 	struct bb_node *bb;
 
 	list_for_each_entry(bb, &func->bbs, l) {
-		draw_bb_node(func, bb);
+		draw_bb_node(func, bb, dd);
 	}
 }
 
@@ -455,7 +452,7 @@ static void func_output_edges(struct func_node *func)
 	       func_idx, ENTRY_BLOCK_INDEX, func_idx, EXIT_BLOCK_INDEX);
 }
 
-static void cfg_dump(struct cfg *cfg)
+static void cfg_dump(struct cfg *cfg, struct dump_data *dd)
 {
 	struct func_node *func;
 
@@ -463,14 +460,14 @@ static void cfg_dump(struct cfg *cfg)
 	list_for_each_entry(func, &cfg->funcs, l) {
 		printf("subgraph \"cluster_%d\" {\n\tstyle=\"dashed\";\n\tcolor=\"black\";\n\tlabel=\"func_%d ()\";\n",
 		       func->idx, func->idx);
-		func_output_bb_def(func);
+		func_output_bb_def(func, dd);
 		func_output_edges(func);
 		printf("}\n");
 	}
 	printf("}\n");
 }
 
-void dump_xlated_cfg(void *buf, unsigned int len)
+void dump_xlated_cfg(struct dump_data *dd, void *buf, unsigned int len)
 {
 	struct bpf_insn *insn = buf;
 	struct cfg cfg;
@@ -479,7 +476,7 @@ void dump_xlated_cfg(void *buf, unsigned int len)
 	if (cfg_build(&cfg, insn, len))
 		return;
 
-	cfg_dump(&cfg);
+	cfg_dump(&cfg, dd);
 
 	cfg_destroy(&cfg);
 }
tools/bpf/bpftool/cfg.h +3 -1
index e144257ea6d2..909d17e6d4c2 100644
@@ -4,6 +4,8 @@
 #ifndef __BPF_TOOL_CFG_H
 #define __BPF_TOOL_CFG_H
 
-void dump_xlated_cfg(void *buf, unsigned int len);
+#include "xlated_dumper.h"
+
+void dump_xlated_cfg(struct dump_data *dd, void *buf, unsigned int len);
 
 #endif /* __BPF_TOOL_CFG_H */
tools/bpf/bpftool/main.h +2 -0
index 0ef373cef4c7..e9ee514b22d4 100644
@@ -229,6 +229,8 @@ void btf_dump_linfo_plain(const struct btf *btf,
 			  const char *prefix, bool linum);
 void btf_dump_linfo_json(const struct btf *btf,
 			 const struct bpf_line_info *linfo, bool linum);
+void btf_dump_linfo_dotlabel(const struct btf *btf,
+			     const struct bpf_line_info *linfo);
 
 struct nlattr;
 struct ifinfomsg;
tools/bpf/bpftool/prog.c +7 -10
index afbe3ec342c8..d855118f0d96 100644
@@ -840,11 +840,6 @@ prog_dump(struct bpf_prog_info *info, enum dump_mode mode,
 					      false))
 				goto exit_free;
 		}
-	} else if (visual) {
-		if (json_output)
-			jsonw_null(json_wtr);
-		else
-			dump_xlated_cfg(buf, member_len);
 	} else {
 		kernel_syms_load(&dd);
 		dd.nr_jited_ksyms = info->nr_jited_ksyms;
@@ -854,12 +849,14 @@ prog_dump(struct bpf_prog_info *info, enum dump_mode mode,
 		dd.finfo_rec_size = info->func_info_rec_size;
 		dd.prog_linfo = prog_linfo;
 
-		if (json_output)
-			dump_xlated_json(&dd, buf, member_len, opcodes,
-					 linum);
+		if (json_output && visual)
+			jsonw_null(json_wtr);
+		else if (json_output)
+			dump_xlated_json(&dd, buf, member_len, opcodes, linum);
+		else if (visual)
+			dump_xlated_cfg(&dd, buf, member_len);
 		else
-			dump_xlated_plain(&dd, buf, member_len, opcodes,
-					  linum);
+			dump_xlated_plain(&dd, buf, member_len, opcodes, linum);
 		kernel_syms_destroy(&dd);
 	}
 
tools/bpf/bpftool/xlated_dumper.c +31 -1
index 3daa05d9bbb7..5fbe94aa8589 100644
@@ -369,20 +369,50 @@ void dump_xlated_for_graph(struct dump_data *dd, void *buf_start, void *buf_end,
 		.cb_imm		= print_imm,
 		.private_data	= dd,
 	};
+	const struct bpf_prog_linfo *prog_linfo = dd->prog_linfo;
+	const struct bpf_line_info *last_linfo = NULL;
+	struct bpf_func_info *record = dd->func_info;
 	struct bpf_insn *insn_start = buf_start;
 	struct bpf_insn *insn_end = buf_end;
 	struct bpf_insn *cur = insn_start;
+	struct btf *btf = dd->btf;
 	bool double_insn = false;
+	char func_sig[1024];
 
 	for (; cur <= insn_end; cur++) {
+		unsigned int insn_off;
+
 		if (double_insn) {
 			double_insn = false;
 			continue;
 		}
 		double_insn = cur->code == (BPF_LD | BPF_IMM | BPF_DW);
 
-		printf("% 4d: ", (int)(cur - insn_start + start_idx));
+		insn_off = (unsigned int)(cur - insn_start + start_idx);
+		if (btf && record) {
+			if (record->insn_off == insn_off) {
+				btf_dumper_type_only(btf, record->type_id,
+						     func_sig,
+						     sizeof(func_sig));
+				if (func_sig[0] != '\0')
+					printf("; %s:\\l\\\n", func_sig);
+				record = (void *)record + dd->finfo_rec_size;
+			}
+		}
+
+		if (prog_linfo) {
+			const struct bpf_line_info *linfo;
+
+			linfo = bpf_prog_linfo__lfind(prog_linfo, insn_off, 0);
+			if (linfo && linfo != last_linfo) {
+				btf_dump_linfo_dotlabel(btf, linfo);
+				last_linfo = linfo;
+			}
+		}
+
+		printf("%d: ", insn_off);
 		print_bpf_insn(&cbs, cur, true);
+
 		if (cur != insn_end)
 			printf(" | ");
 	}
Signature · 1 line
2.34.1
#6 lore raw

Re: [PATCH bpf-next 3/5] bpftool: Support inline annotations when dumping the CFG of a program

Quoted 51 lines · Quentin Monnet

We support dumping the control flow graph of loaded programs to the DOT format with bpftool, but so far this feature wouldn't display the source code lines available through BTF along with the eBPF bytecode. Let's add support for these annotations, to make it easier to read the graph.

In prog.c, we move the call to dump_xlated_cfg() in order to pass and use the full struct dump_data, instead of creating a minimal one in draw_bb_node().

We pass the pointer to this struct down to dump_xlated_for_graph() in xlated_dumper.c, where most of the logics is added. We deal with BTF mostly like we do for plain or JSON output, except that we cannot use a "nr_skip" value to skip a given number of linfo records (we don't process the BPF instructions linearly, and apart from the root of the graph we don't know how many records we should skip, so we just store the last linfo and make sure the new one we find is different before printing it).

When printing the source instructions to the label of a DOT graph node, there are a few subtleties to address. We want some special newline markers, and there are some characters that we must escape. To deal with them, we introduce a new dedicated function btf_dump_linfo_dotlabel() in btf_dumper.c. We'll reuse this function in a later commit to format the filepath, line, and column references as well.

---
  tools/bpf/bpftool/btf_dumper.c    | 34 +++++++++++++++++++++++++++++++
  tools/bpf/bpftool/cfg.c           | 23 +++++++++------------
  tools/bpf/bpftool/cfg.h           |  4 +++-
  tools/bpf/bpftool/main.h          |  2 ++
  tools/bpf/bpftool/prog.c          | 17 +++++++---------
  tools/bpf/bpftool/xlated_dumper.c | 32 ++++++++++++++++++++++++++++-
  6 files changed, 87 insertions(+), 25 deletions(-)
tools/bpf/bpftool/btf_dumper.c +8 -0
index e7f6ec3a8f35..504d7c75cc27 100644
@@ -821,3 +821,37 @@ void btf_dump_linfo_json(const struct btf *btf,
  					BPF_LINE_INFO_LINE_COL(linfo->line_col));
  	}
  }
+
+static void dotlabel_puts(const char *s)
+{
+	FILE *stream = stdout;
+
+	for (; *s; ++s) {
+		switch (*s) {
+		case '\\':

tools/bpf/bpftool/btf_dumper.c void btf_dump_linfo_json(const struct btf *btf, · line 841

Quoted 10 lines
tools/bpf/bpftool/btf_dumper.c +10 -0
+		case '"':
+		case '{':
+		case '}':
+		case '>':
+		case '|':
+			fputc('\\', stream);
+			__fallthrough;
+		default:
+			fputc(*s, stream);
+		}

nit: optionally, if you're going to respin, maybe do putchar instead of fputc + stdout? (not sure why you're doing fputs)

Quoted 225 lines
tools/bpf/bpftool/btf_dumper.c +16 -0
+	}
+}
+
+void btf_dump_linfo_dotlabel(const struct btf *btf,
+			     const struct bpf_line_info *linfo)
+{
+	const char *line = btf__name_by_offset(btf, linfo->line_off);
+
+	if (!line)
+		return;
+	line = ltrim(line);
+
+	printf("; ");
+	dotlabel_puts(line);
+	printf("\\l\\\n");
+}
tools/bpf/bpftool/cfg.c +6 -9
index 1951219a9af7..9fdc1f0cdd6e 100644
@@ -380,7 +380,8 @@ static void cfg_destroy(struct cfg *cfg)
  	}
  }
 
-static void draw_bb_node(struct func_node *func, struct bb_node *bb)
+static void
+draw_bb_node(struct func_node *func, struct bb_node *bb, struct  
 dump_data *dd)
  {
  	const char *shape;
@@ -398,13 +399,9 @@ static void draw_bb_node(struct func_node *func,  
 struct bb_node *bb)
  		printf("EXIT");
  	} else {
  		unsigned int start_idx;
-		struct dump_data dd = {};
-
-		printf("{");
-		kernel_syms_load(&dd);
+		printf("{\\\n");
  		start_idx = bb->head - func->start;
-		dump_xlated_for_graph(&dd, bb->head, bb->tail, start_idx);
-		kernel_syms_destroy(&dd);
+		dump_xlated_for_graph(dd, bb->head, bb->tail, start_idx);
  		printf("}");
  	}
@@ -430,12 +427,12 @@ static void draw_bb_succ_edges(struct func_node  
 *func, struct bb_node *bb)
  	}
  }
 
-static void func_output_bb_def(struct func_node *func)
+static void func_output_bb_def(struct func_node *func, struct dump_data  
 *dd)
  {
  	struct bb_node *bb;
 
  	list_for_each_entry(bb, &func->bbs, l) {
-		draw_bb_node(func, bb);
+		draw_bb_node(func, bb, dd);
  	}
  }
tools/bpf/bpftool/cfg.c +3 -3
@@ -455,7 +452,7 @@ static void func_output_edges(struct func_node *func)
  	       func_idx, ENTRY_BLOCK_INDEX, func_idx, EXIT_BLOCK_INDEX);
  }
 
-static void cfg_dump(struct cfg *cfg)
+static void cfg_dump(struct cfg *cfg, struct dump_data *dd)
  {
  	struct func_node *func;
 
@@ -463,14 +460,14 @@ static void cfg_dump(struct cfg *cfg)
  	list_for_each_entry(func, &cfg->funcs, l) {
  		printf("subgraph \"cluster_%d\"  
 {\n\tstyle=\"dashed\";\n\tcolor=\"black\";\n\tlabel=\"func_%d ()\";\n",
  		       func->idx, func->idx);
-		func_output_bb_def(func);
+		func_output_bb_def(func, dd);
  		func_output_edges(func);
  		printf("}\n");
  	}
  	printf("}\n");
  }
 
-void dump_xlated_cfg(void *buf, unsigned int len)
+void dump_xlated_cfg(struct dump_data *dd, void *buf, unsigned int len)
  {
  	struct bpf_insn *insn = buf;

struct cfg cfg;

tools/bpf/bpftool/cfg.c +1 -1
@@ -479,7 +476,7 @@ void dump_xlated_cfg(void *buf, unsigned int len)
  	if (cfg_build(&cfg, insn, len))
  		return;
 
-	cfg_dump(&cfg);
+	cfg_dump(&cfg, dd);
 
  	cfg_destroy(&cfg);
  }
tools/bpf/bpftool/cfg.h +3 -1
index e144257ea6d2..909d17e6d4c2 100644
@@ -4,6 +4,8 @@
  #ifndef __BPF_TOOL_CFG_H
  #define __BPF_TOOL_CFG_H
 
-void dump_xlated_cfg(void *buf, unsigned int len);
+#include "xlated_dumper.h"
+
+void dump_xlated_cfg(struct dump_data *dd, void *buf, unsigned int len);
 
  #endif /* __BPF_TOOL_CFG_H */
tools/bpf/bpftool/main.h +2 -0
index 0ef373cef4c7..e9ee514b22d4 100644
@@ -229,6 +229,8 @@ void btf_dump_linfo_plain(const struct btf *btf,
  			  const char *prefix, bool linum);
  void btf_dump_linfo_json(const struct btf *btf,
  			 const struct bpf_line_info *linfo, bool linum);
+void btf_dump_linfo_dotlabel(const struct btf *btf,
+			     const struct bpf_line_info *linfo);
 
  struct nlattr;
  struct ifinfomsg;
tools/bpf/bpftool/prog.c +0 -5
index afbe3ec342c8..d855118f0d96 100644
@@ -840,11 +840,6 @@ prog_dump(struct bpf_prog_info *info, enum dump_mode  
 mode,
  					      false))
  				goto exit_free;
  		}
-	} else if (visual) {
-		if (json_output)
-			jsonw_null(json_wtr);
-		else
-			dump_xlated_cfg(buf, member_len);
  	} else {
  		kernel_syms_load(&dd);
  		dd.nr_jited_ksyms = info->nr_jited_ksyms;
tools/bpf/bpftool/prog.c +7 -5
@@ -854,12 +849,14 @@ prog_dump(struct bpf_prog_info *info, enum  
 dump_mode mode,
  		dd.finfo_rec_size = info->func_info_rec_size;
  		dd.prog_linfo = prog_linfo;
 
-		if (json_output)
-			dump_xlated_json(&dd, buf, member_len, opcodes,
-					 linum);
+		if (json_output && visual)
+			jsonw_null(json_wtr);
+		else if (json_output)
+			dump_xlated_json(&dd, buf, member_len, opcodes, linum);
+		else if (visual)
+			dump_xlated_cfg(&dd, buf, member_len);
  		else
-			dump_xlated_plain(&dd, buf, member_len, opcodes,
-					  linum);
+			dump_xlated_plain(&dd, buf, member_len, opcodes, linum);
  		kernel_syms_destroy(&dd);
  	}
tools/bpf/bpftool/xlated_dumper.c +31 -1
index 3daa05d9bbb7..5fbe94aa8589 100644
@@ -369,20 +369,50 @@ void dump_xlated_for_graph(struct dump_data *dd,  
 void *buf_start, void *buf_end,
  		.cb_imm		= print_imm,
  		.private_data	= dd,
  	};
+	const struct bpf_prog_linfo *prog_linfo = dd->prog_linfo;
+	const struct bpf_line_info *last_linfo = NULL;
+	struct bpf_func_info *record = dd->func_info;
  	struct bpf_insn *insn_start = buf_start;
  	struct bpf_insn *insn_end = buf_end;
  	struct bpf_insn *cur = insn_start;
+	struct btf *btf = dd->btf;
  	bool double_insn = false;
+	char func_sig[1024];
 
  	for (; cur <= insn_end; cur++) {
+		unsigned int insn_off;
+
  		if (double_insn) {
  			double_insn = false;
  			continue;
  		}
  		double_insn = cur->code == (BPF_LD | BPF_IMM | BPF_DW);
 
-		printf("% 4d: ", (int)(cur - insn_start + start_idx));
+		insn_off = (unsigned int)(cur - insn_start + start_idx);
+		if (btf && record) {
+			if (record->insn_off == insn_off) {
+				btf_dumper_type_only(btf, record->type_id,
+						     func_sig,
+						     sizeof(func_sig));
+				if (func_sig[0] != '\0')
+					printf("; %s:\\l\\\n", func_sig);
+				record = (void *)record + dd->finfo_rec_size;
+			}
+		}
+
+		if (prog_linfo) {
+			const struct bpf_line_info *linfo;
+
+			linfo = bpf_prog_linfo__lfind(prog_linfo, insn_off, 0);
+			if (linfo && linfo != last_linfo) {
+				btf_dump_linfo_dotlabel(btf, linfo);
+				last_linfo = linfo;
+			}
+		}
+
+		printf("%d: ", insn_off);
  		print_bpf_insn(&cbs, cur, true);
+
  		if (cur != insn_end)
  			printf(" | ");

}
--
2.34.1

#7 lore raw

Re: [PATCH bpf-next 3/5] bpftool: Support inline annotations when dumping the CFG of a program

2023-03-24 20:05 UTC-0700 ~ Stanislav Fomichev <[email protected]>

Quoted 58 lines · Quentin Monnet
Quoted 57 lines

We support dumping the control flow graph of loaded programs to the DOT format with bpftool, but so far this feature wouldn't display the source code lines available through BTF along with the eBPF bytecode. Let's add support for these annotations, to make it easier to read the graph.

In prog.c, we move the call to dump_xlated_cfg() in order to pass and use the full struct dump_data, instead of creating a minimal one in draw_bb_node().

We pass the pointer to this struct down to dump_xlated_for_graph() in xlated_dumper.c, where most of the logics is added. We deal with BTF mostly like we do for plain or JSON output, except that we cannot use a "nr_skip" value to skip a given number of linfo records (we don't process the BPF instructions linearly, and apart from the root of the graph we don't know how many records we should skip, so we just store the last linfo and make sure the new one we find is different before printing it).

When printing the source instructions to the label of a DOT graph node, there are a few subtleties to address. We want some special newline markers, and there are some characters that we must escape. To deal with them, we introduce a new dedicated function btf_dump_linfo_dotlabel() in btf_dumper.c. We'll reuse this function in a later commit to format the filepath, line, and column references as well.

---
  tools/bpf/bpftool/btf_dumper.c    | 34 +++++++++++++++++++++++++++++++ tools/bpf/bpftool/cfg.c           | 23 +++++++++------------ tools/bpf/bpftool/cfg.h           |  4 +++- tools/bpf/bpftool/main.h          |  2 ++ tools/bpf/bpftool/prog.c          | 17 +++++++--------- tools/bpf/bpftool/xlated_dumper.c | 32 ++++++++++++++++++++++++++++-

  6 files changed, 87 insertions(+), 25 deletions(-)
tools/bpf/bpftool/btf_dumper.c +13 -0
index e7f6ec3a8f35..504d7c75cc27 100644
@@ -821,3 +821,37 @@ void btf_dump_linfo_json(const struct btf *btf,
                       BPF_LINE_INFO_LINE_COL(linfo->line_col));
       }
   }
+
+static void dotlabel_puts(const char *s)
+{
+    FILE *stream = stdout;
+
+    for (; *s; ++s) {
+        switch (*s) {
+        case '\\':
+        case '"':
+        case '{':
+        case '}':
+        case '>':
+        case '|':
Quoted 8 lines

tools/bpf/bpftool/btf_dumper.c void btf_dump_linfo_json(const struct btf *btf, · line 841

Quoted 5 lines
tools/bpf/bpftool/btf_dumper.c +5 -0
+            fputc('\\', stream);
+            __fallthrough;
+        default:
+            fputc(*s, stream);
+        }

nit: optionally, if you're going to respin, maybe do putchar instead of fputc + stdout? (not sure why you're doing fputs)

Right, no particular reason, I'll switch to putchar() and respin.

Thanks for the review!
Quentin

#8 lore raw

PATCH 4/5 bpftool: Support "opcodes", "linum", "visual" simultaneously

In reply to Quentin Monnet

When dumping a program, the keywords "opcodes" (for printing the raw opcodes), "linum" (for displaying the filename, line number, column number along with the source code), and "visual" (for generating the control flow graph for translated programs) are mutually exclusive. But there's no reason why they should be. Let's make it possible to pass several of them at once. The "file FILE" option, which makes bpftool output a binary image to a file, remains incompatible with the others.

---
 .../bpftool/Documentation/bpftool-prog.rst    |  8 +--
 tools/bpf/bpftool/bash-completion/bpftool     | 18 +++---
 tools/bpf/bpftool/prog.c                      | 61 ++++++++++---------
 3 files changed, 48 insertions(+), 39 deletions(-)
tools/bpf/bpftool/Documentation/bpftool-prog.rst +4 -4
index 06d1e4314406..9443c524bb76 100644
@@ -28,8 +28,8 @@ PROG COMMANDS
 =============
 
 |	**bpftool** **prog** { **show** | **list** } [*PROG*]
-|	**bpftool** **prog dump xlated** *PROG* [{**file** *FILE* | **opcodes** | **visual** | **linum**}]
-|	**bpftool** **prog dump jited**  *PROG* [{**file** *FILE* | **opcodes** | **linum**}]
+|	**bpftool** **prog dump xlated** *PROG* [{ **file** *FILE* | [**opcodes**] [**linum**] [**visual**] }]
+|	**bpftool** **prog dump jited**  *PROG* [{ **file** *FILE* | [**opcodes**] [**linum**] }]
 |	**bpftool** **prog pin** *PROG* *FILE*
 |	**bpftool** **prog** { **load** | **loadall** } *OBJ* *PATH* [**type** *TYPE*] [**map** {**idx** *IDX* | **name** *NAME*} *MAP*] [**dev** *NAME*] [**pinmaps** *MAP_DIR*] [**autoattach**]
 |	**bpftool** **prog attach** *PROG* *ATTACH_TYPE* [*MAP*]
@@ -88,7 +88,7 @@ DESCRIPTION
 		  programs. On such kernels bpftool will automatically emit this
 		  information as well.
 
-	**bpftool prog dump xlated** *PROG* [{ **file** *FILE* | **opcodes** | **visual** | **linum** }]
+	**bpftool prog dump xlated** *PROG* [{ **file** *FILE* | [**opcodes**] [**linum**] [**visual**] }]
 		  Dump eBPF instructions of the programs from the kernel. By
 		  default, eBPF will be disassembled and printed to standard
 		  output in human-readable format. In this case, **opcodes**
@@ -109,7 +109,7 @@ DESCRIPTION
 		  be displayed.  If **linum** is specified, the filename, line
 		  number and line column will also be displayed.
 
-	**bpftool prog dump jited**  *PROG* [{ **file** *FILE* | **opcodes** | **linum** }]
+	**bpftool prog dump jited**  *PROG* [{ **file** *FILE* | [**opcodes**] [**linum**] }]
 		  Dump jited image (host machine code) of the program.
 
 		  If *FILE* is specified image will be written to a file,
tools/bpf/bpftool/bash-completion/bpftool +11 -7
index 35f26f7c1124..98eb3852e771 100644
@@ -268,7 +268,7 @@ _bpftool()
 
     # Deal with simplest keywords
     case $prev in
-        help|hex|opcodes|visual|linum)
+        help|hex)
             return 0
             ;;
         tag)
@@ -366,13 +366,17 @@ _bpftool()
                             return 0
                             ;;
                         *)
-                            _bpftool_once_attr 'file'
+                            # "file" is not compatible with other keywords here
+                            if _bpftool_search_list 'file'; then
+                                return 0
+                            fi
+                            if ! _bpftool_search_list 'linum opcodes visual'; then
+                                _bpftool_once_attr 'file'
+                            fi
+                            _bpftool_once_attr 'linum'
+                            _bpftool_once_attr 'opcodes'
                             if _bpftool_search_list 'xlated'; then
-                                COMPREPLY+=( $( compgen -W 'opcodes visual linum' -- \
-                                    "$cur" ) )
-                            else
-                                COMPREPLY+=( $( compgen -W 'opcodes linum' -- \
-                                    "$cur" ) )
+                                _bpftool_once_attr 'visual'
                             fi
                             return 0
                             ;;
tools/bpf/bpftool/prog.c +33 -28

78 lines not on this page · raw message

Signature · 1 line
2.34.1
#9 lore raw

PATCH 5/5 bpftool: Support printing opcodes and source file references in CFG

In reply to Quentin Monnet

Add support for displaying opcodes or/and file references (filepath, line and column numbers) when dumping the control flow graphs of loaded BPF programs with bpftool.

---
 tools/bpf/bpftool/btf_dumper.c    | 19 ++++++++++++++++++-
 tools/bpf/bpftool/cfg.c           | 22 ++++++++++++++--------
 tools/bpf/bpftool/cfg.h           |  3 ++-
 tools/bpf/bpftool/main.h          |  2 +-
 tools/bpf/bpftool/prog.c          |  2 +-
 tools/bpf/bpftool/xlated_dumper.c | 15 +++++++++++++--
 tools/bpf/bpftool/xlated_dumper.h |  3 ++-
 7 files changed, 51 insertions(+), 15 deletions(-)
tools/bpf/bpftool/btf_dumper.c +18 -1
index 504d7c75cc27..54cb105c1648 100644
@@ -843,7 +843,7 @@ static void dotlabel_puts(const char *s)
 }
 
 void btf_dump_linfo_dotlabel(const struct btf *btf,
-			     const struct bpf_line_info *linfo)
+			     const struct bpf_line_info *linfo, bool linum)
 {
 	const char *line = btf__name_by_offset(btf, linfo->line_off);
 
@@ -851,6 +851,23 @@ void btf_dump_linfo_dotlabel(const struct btf *btf,
 		return;
 	line = ltrim(line);
 
+	if (linum) {
+		const char *file = btf__name_by_offset(btf, linfo->file_name_off);
+
+		/* More forgiving on file because linum option is
+		 * expected to provide more info than the already
+		 * available src line.
+		 */
+		if (!file)
+			file = "";
+
+		printf("; [file:");
+		dotlabel_puts(file);
+		printf("line_num:%u line_col:%u]\\l\\\n",
+		       BPF_LINE_INFO_LINE_NUM(linfo->line_col),
+		       BPF_LINE_INFO_LINE_COL(linfo->line_col));
+	}
+
 	printf("; ");
 	dotlabel_puts(line);
 	printf("\\l\\\n");
tools/bpf/bpftool/cfg.c +14 -8
index 9fdc1f0cdd6e..eec437cca2ea 100644
@@ -381,7 +381,8 @@ static void cfg_destroy(struct cfg *cfg)
 }
 
 static void
-draw_bb_node(struct func_node *func, struct bb_node *bb, struct dump_data *dd)
+draw_bb_node(struct func_node *func, struct bb_node *bb, struct dump_data *dd,
+	     bool opcodes, bool linum)
 {
 	const char *shape;
 
@@ -401,7 +402,8 @@ draw_bb_node(struct func_node *func, struct bb_node *bb, struct dump_data *dd)
 		unsigned int start_idx;
 		printf("{\\\n");
 		start_idx = bb->head - func->start;
-		dump_xlated_for_graph(dd, bb->head, bb->tail, start_idx);
+		dump_xlated_for_graph(dd, bb->head, bb->tail, start_idx,
+				      opcodes, linum);
 		printf("}");
 	}
 
@@ -427,12 +429,14 @@ static void draw_bb_succ_edges(struct func_node *func, struct bb_node *bb)
 	}
 }
 
-static void func_output_bb_def(struct func_node *func, struct dump_data *dd)
+static void
+func_output_bb_def(struct func_node *func, struct dump_data *dd,
+		   bool opcodes, bool linum)
 {
 	struct bb_node *bb;
 
 	list_for_each_entry(bb, &func->bbs, l) {
-		draw_bb_node(func, bb, dd);
+		draw_bb_node(func, bb, dd, opcodes, linum);
 	}
 }
 
@@ -452,7 +456,8 @@ static void func_output_edges(struct func_node *func)
 	       func_idx, ENTRY_BLOCK_INDEX, func_idx, EXIT_BLOCK_INDEX);
 }
 
-static void cfg_dump(struct cfg *cfg, struct dump_data *dd)
+static void
+cfg_dump(struct cfg *cfg, struct dump_data *dd, bool opcodes, bool linum)
 {
 	struct func_node *func;
 
@@ -460,14 +465,15 @@ static void cfg_dump(struct cfg *cfg, struct dump_data *dd)
 	list_for_each_entry(func, &cfg->funcs, l) {
 		printf("subgraph \"cluster_%d\" {\n\tstyle=\"dashed\";\n\tcolor=\"black\";\n\tlabel=\"func_%d ()\";\n",
 		       func->idx, func->idx);
-		func_output_bb_def(func, dd);
+		func_output_bb_def(func, dd, opcodes, linum);
 		func_output_edges(func);
 		printf("}\n");
 	}
 	printf("}\n");
 }
 
-void dump_xlated_cfg(struct dump_data *dd, void *buf, unsigned int len)
+void dump_xlated_cfg(struct dump_data *dd, void *buf, unsigned int len,
+		     bool opcodes, bool linum)
 {
 	struct bpf_insn *insn = buf;
 	struct cfg cfg;
@@ -476,7 +482,7 @@ void dump_xlated_cfg(struct dump_data *dd, void *buf, unsigned int len)
 	if (cfg_build(&cfg, insn, len))
 		return;
 
-	cfg_dump(&cfg, dd);
+	cfg_dump(&cfg, dd, opcodes, linum);
 
 	cfg_destroy(&cfg);
 }
tools/bpf/bpftool/cfg.h +2 -1
index 909d17e6d4c2..b3793f4e1783 100644
@@ -6,6 +6,7 @@
 
 #include "xlated_dumper.h"
 
-void dump_xlated_cfg(struct dump_data *dd, void *buf, unsigned int len);
+void dump_xlated_cfg(struct dump_data *dd, void *buf, unsigned int len,
+		     bool opcodes, bool linum);
 
 #endif /* __BPF_TOOL_CFG_H */
tools/bpf/bpftool/main.h +1 -1
index e9ee514b22d4..00d11ca6d3f2 100644
@@ -230,7 +230,7 @@ void btf_dump_linfo_plain(const struct btf *btf,
 void btf_dump_linfo_json(const struct btf *btf,
 			 const struct bpf_line_info *linfo, bool linum);
 void btf_dump_linfo_dotlabel(const struct btf *btf,
-			     const struct bpf_line_info *linfo);
+			     const struct bpf_line_info *linfo, bool linum);
 
 struct nlattr;
 struct ifinfomsg;
tools/bpf/bpftool/prog.c +1 -1
index 567ac37dbd86..848f57a7d762 100644
@@ -854,7 +854,7 @@ prog_dump(struct bpf_prog_info *info, enum dump_mode mode,
 		else if (json_output)
 			dump_xlated_json(&dd, buf, member_len, opcodes, linum);
 		else if (visual)
-			dump_xlated_cfg(&dd, buf, member_len);
+			dump_xlated_cfg(&dd, buf, member_len, opcodes, linum);
 		else
 			dump_xlated_plain(&dd, buf, member_len, opcodes, linum);
 		kernel_syms_destroy(&dd);
tools/bpf/bpftool/xlated_dumper.c +13 -2
index 5fbe94aa8589..c5e03833fadf 100644
@@ -361,7 +361,8 @@ void dump_xlated_plain(struct dump_data *dd, void *buf, unsigned int len,
 }
 
 void dump_xlated_for_graph(struct dump_data *dd, void *buf_start, void *buf_end,
-			   unsigned int start_idx)
+			   unsigned int start_idx,
+			   bool opcodes, bool linum)
 {
 	const struct bpf_insn_cbs cbs = {
 		.cb_print	= print_insn_for_graph,
@@ -405,7 +406,7 @@ void dump_xlated_for_graph(struct dump_data *dd, void *buf_start, void *buf_end,
 
 			linfo = bpf_prog_linfo__lfind(prog_linfo, insn_off, 0);
 			if (linfo && linfo != last_linfo) {
-				btf_dump_linfo_dotlabel(btf, linfo);
+				btf_dump_linfo_dotlabel(btf, linfo, linum);
 				last_linfo = linfo;
 			}
 		}
@@ -413,6 +414,16 @@ void dump_xlated_for_graph(struct dump_data *dd, void *buf_start, void *buf_end,
 		printf("%d: ", insn_off);
 		print_bpf_insn(&cbs, cur, true);
 
+		if (opcodes) {
+			printf("       ");
+			fprint_hex(stdout, cur, 8, " ");
+			if (double_insn && cur <= insn_end - 1) {
+				printf(" ");
+				fprint_hex(stdout, cur + 1, 8, " ");
+			}
+			printf("\\l\\\n");
+		}
+
 		if (cur != insn_end)
 			printf(" | ");
 	}
tools/bpf/bpftool/xlated_dumper.h +2 -1
index 54847e174273..9a946377b0e6 100644
@@ -34,6 +34,7 @@ void dump_xlated_json(struct dump_data *dd, void *buf, unsigned int len,
 void dump_xlated_plain(struct dump_data *dd, void *buf, unsigned int len,
 		       bool opcodes, bool linum);
 void dump_xlated_for_graph(struct dump_data *dd, void *buf, void *buf_end,
-			   unsigned int start_index);
+			   unsigned int start_index,
+			   bool opcodes, bool linum);
 
 #endif
Signature · 1 line
2.34.1
Acked-by on the v1 series
#10 lore raw
In reply to Quentin Monnet
Quoted 8 lines · Quentin Monnet

This set contains some improvements for bpftool's "visual" program dump option, which produces the control flow graph in a DOT format. The main objective is to add support for inline annotations on such graphs, so that we can have the C source code for the program showing up alongside the instructions, when available. The last commits also make it possible to display the line numbers or the bare opcodes in the graph, as supported by regular program dumps.

Quoted 6 lines

Quentin Monnet (5):
bpftool: Fix documentation about line info display for prog dumps
bpftool: Fix bug for long instructions in program CFG dumps
bpftool: Support inline annotations when dumping the CFG of a program
bpftool: Support "opcodes", "linum", "visual" simultaneously
bpftool: Support printing opcodes and source file references in CFG

Left two small nits. Up to you on whether it's worth to respin or not.

Quoted 13 lines
  .../bpftool/Documentation/bpftool-prog.rst    | 18 ++---
  tools/bpf/bpftool/bash-completion/bpftool     | 18 +++--
  tools/bpf/bpftool/btf_dumper.c                | 51 ++++++++++++
  tools/bpf/bpftool/cfg.c                       | 29 +++----
  tools/bpf/bpftool/cfg.h                       |  5 +-
  tools/bpf/bpftool/main.h                      |  2 +
  tools/bpf/bpftool/prog.c                      | 78 ++++++++++---------
  tools/bpf/bpftool/xlated_dumper.c             | 52 ++++++++++++-
  tools/bpf/bpftool/xlated_dumper.h             |  3 +-
  9 files changed, 184 insertions(+), 72 deletions(-)

--
2.34.1