Skip to contentSkip to conversation

bpftool: introduce btf c dump sorting

PATCH Superseded Mykyta Yatsenko [email protected] 4 participants 5 messages May 13, 2024 – May 14, 2024

Subsystems (2)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Mykyta Yatsenko [email protected]

author
#0 lore raw

PATCH v3 1/1 bpftool: introduce btf c dump sorting

From: Mykyta Yatsenko <[email protected]>

Sort bpftool c dump output; aiming to simplify vmlinux.h diffing and forcing more natural type definitions ordering.

Definitions are sorted first by their BTF kind ranks, then by their base type name and by their own name.

Type ranks

Assign ranks to btf kinds (defined in function btf_type_rank) to set next order:
1. Anonymous enums/enums64
2. Named enums/enums64
3. Trivial types typedefs (ints, then floats)
4. Structs/Unions
5. Function prototypes
6. Forward declarations

Type rank is set to maximum for unnamed reference types, structs and unions to avoid emitting those types early. They will be emitted as part of the type chain starting with named type.

Lexicographical ordering

Each type is assigned a sort_name and own_name.
sort_name is the resolved name of the final base type for reference types (typedef, pointer, array etc). Sorting by sort_name allows to group typedefs of the same base type. sort_name for non-reference type is the same as own_name. own_name is a direct name of particular type, is used as final sorting step.

---
 .../bpf/bpftool/Documentation/bpftool-btf.rst |   5 +-
 tools/bpf/bpftool/bash-completion/bpftool     |   3 +
 tools/bpf/bpftool/btf.c                       | 138 +++++++++++++++++-
 3 files changed, 139 insertions(+), 7 deletions(-)
tools/bpf/bpftool/Documentation/bpftool-btf.rst +3 -2
index eaba24320fb2..65eeb3d905f0 100644
@@ -28,7 +28,7 @@ BTF COMMANDS
 | **bpftool** **btf help**
 |
 | *BTF_SRC* := { **id** *BTF_ID* | **prog** *PROG* | **map** *MAP* [{**key** | **value** | **kv** | **all**}] | **file** *FILE* }
-| *FORMAT* := { **raw** | **c** }
+| *FORMAT* := { **raw** | **c** [**unsorted**]}
 | *MAP* := { **id** *MAP_ID* | **pinned** *FILE* }
 | *PROG* := { **id** *PROG_ID* | **pinned** *FILE* | **tag** *PROG_TAG* | **name** *PROG_NAME* }
 
@@ -63,7 +63,8 @@ bpftool btf dump *BTF_SRC*
     pahole.
 
     **format** option can be used to override default (raw) output format. Raw
-    (**raw**) or C-syntax (**c**) output formats are supported.
+    (**raw**) or C-syntax (**c**) output formats are supported. (**unsorted**)
+    option can be used with (**c**) to avoid sorting the output.
 
 bpftool btf help
     Print short help message.
tools/bpf/bpftool/bash-completion/bpftool +3 -0
index 04afe2ac2228..be99d49b8714 100644
@@ -930,6 +930,9 @@ _bpftool()
                         format)
                             COMPREPLY=( $( compgen -W "c raw" -- "$cur" ) )
                             ;;
+                        c)
+                            COMPREPLY=( $( compgen -W "unsorted" -- "$cur" ) )
+                            ;;
                         *)
                             # emit extra options
                             case ${words[3]} in
tools/bpf/bpftool/btf.c +133 -5

192 lines not on this page · raw message

Signature · 1 line
2.44.0
Acked-by Tested-by on v3 1/1
#1 lore raw
Quoted 33 lines · Mykyta Yatsenko

From: Mykyta Yatsenko <[email protected]>

Sort bpftool c dump output; aiming to simplify vmlinux.h diffing and forcing more natural type definitions ordering.

Definitions are sorted first by their BTF kind ranks, then by their base type name and by their own name.

Type ranks

Assign ranks to btf kinds (defined in function btf_type_rank) to set next order:
1. Anonymous enums/enums64
2. Named enums/enums64
3. Trivial types typedefs (ints, then floats)
4. Structs/Unions
5. Function prototypes
6. Forward declarations

Type rank is set to maximum for unnamed reference types, structs and unions to avoid emitting those types early. They will be emitted as part of the type chain starting with named type.

Lexicographical ordering

Each type is assigned a sort_name and own_name.
sort_name is the resolved name of the final base type for reference types (typedef, pointer, array etc). Sorting by sort_name allows to group typedefs of the same base type. sort_name for non-reference type is the same as own_name. own_name is a direct name of particular type, is used as final sorting step.

Quoted 7 lines
---
 .../bpf/bpftool/Documentation/bpftool-btf.rst |   5 +-
 tools/bpf/bpftool/bash-completion/bpftool     |   3 +
 tools/bpf/bpftool/btf.c                       | 138 +++++++++++++++++-
 3 files changed, 139 insertions(+), 7 deletions(-)

LGTM, tried it locally and it works well. In fact, see 6.8 kernel vs latest bpf-next/master (with basically the same config) comparison. It's quite minimal and easy to use to see what changes about some of the BPF internal types.

[0] https://gist.github.com/anakryiko/8fd8ebf2aba73961ebd3cf6587de6822

[...]

#2 lore raw
Quoted 49 lines · Andrii Nakryiko
Quoted 33 lines · Mykyta Yatsenko

From: Mykyta Yatsenko <[email protected]>

Sort bpftool c dump output; aiming to simplify vmlinux.h diffing and forcing more natural type definitions ordering.

Definitions are sorted first by their BTF kind ranks, then by their base type name and by their own name.

Type ranks

Assign ranks to btf kinds (defined in function btf_type_rank) to set next order:
1. Anonymous enums/enums64
2. Named enums/enums64
3. Trivial types typedefs (ints, then floats)
4. Structs/Unions
5. Function prototypes
6. Forward declarations

Type rank is set to maximum for unnamed reference types, structs and unions to avoid emitting those types early. They will be emitted as part of the type chain starting with named type.

Lexicographical ordering

Each type is assigned a sort_name and own_name.
sort_name is the resolved name of the final base type for reference types (typedef, pointer, array etc). Sorting by sort_name allows to group typedefs of the same base type. sort_name for non-reference type is the same as own_name. own_name is a direct name of particular type, is used as final sorting step.

Quoted 7 lines
---
 .../bpf/bpftool/Documentation/bpftool-btf.rst |   5 +-
 tools/bpf/bpftool/bash-completion/bpftool     |   3 +
 tools/bpf/bpftool/btf.c                       | 138 +++++++++++++++++-
 3 files changed, 139 insertions(+), 7 deletions(-)

LGTM, tried it locally and it works well. In fact, see 6.8 kernel vs latest bpf-next/master (with basically the same config) comparison. It's quite minimal and easy to use to see what changes about some of the BPF internal types.

Quoted 4 lines

[0] https://gist.github.com/anakryiko/8fd8ebf2aba73961ebd3cf6587de6822

Just noticed:

+ ETHTOOL_A_TS_STAT_UNSPEC = 0,
+ ETHTOOL_A_TS_STAT_TX_PKTS = 1,
+ ETHTOOL_A_TS_STAT_TX_LOST = 2,
+ ETHTOOL_A_TS_STAT_TX_ERR = 3,
+ __ETHTOOL_A_TS_STAT_CNT = 4,
+ ETHTOOL_A_TS_STAT_MAX = 3,
};

I'm a bit surprised that enum values are not sorted. I'm guessing the enum names come in dwarf order and copied the same way in BTF ?
I guess it's not an issue.

#3 lore raw
Quoted 67 lines · Alexei Starovoitov
Quoted 49 lines · Andrii Nakryiko
Quoted 33 lines · Mykyta Yatsenko

From: Mykyta Yatsenko <[email protected]>

Sort bpftool c dump output; aiming to simplify vmlinux.h diffing and forcing more natural type definitions ordering.

Definitions are sorted first by their BTF kind ranks, then by their base type name and by their own name.

Type ranks

Assign ranks to btf kinds (defined in function btf_type_rank) to set next order:
1. Anonymous enums/enums64
2. Named enums/enums64
3. Trivial types typedefs (ints, then floats)
4. Structs/Unions
5. Function prototypes
6. Forward declarations

Type rank is set to maximum for unnamed reference types, structs and unions to avoid emitting those types early. They will be emitted as part of the type chain starting with named type.

Lexicographical ordering

Each type is assigned a sort_name and own_name.
sort_name is the resolved name of the final base type for reference types (typedef, pointer, array etc). Sorting by sort_name allows to group typedefs of the same base type. sort_name for non-reference type is the same as own_name. own_name is a direct name of particular type, is used as final sorting step.

Quoted 7 lines
---
 .../bpf/bpftool/Documentation/bpftool-btf.rst |   5 +-
 tools/bpf/bpftool/bash-completion/bpftool     |   3 +
 tools/bpf/bpftool/btf.c                       | 138 +++++++++++++++++-
 3 files changed, 139 insertions(+), 7 deletions(-)

LGTM, tried it locally and it works well. In fact, see 6.8 kernel vs latest bpf-next/master (with basically the same config) comparison. It's quite minimal and easy to use to see what changes about some of the BPF internal types.

Quoted 4 lines

[0] https://gist.github.com/anakryiko/8fd8ebf2aba73961ebd3cf6587de6822

Just noticed:

+ ETHTOOL_A_TS_STAT_UNSPEC = 0,
+ ETHTOOL_A_TS_STAT_TX_PKTS = 1,
+ ETHTOOL_A_TS_STAT_TX_LOST = 2,
+ ETHTOOL_A_TS_STAT_TX_ERR = 3,
+ __ETHTOOL_A_TS_STAT_CNT = 4,
+ ETHTOOL_A_TS_STAT_MAX = 3,
};

Quoted 4 lines

I'm a bit surprised that enum values are not sorted. I'm guessing the enum names come in dwarf order and copied the same way in BTF ?
I guess it's not an issue.

Yes, it's the order in which enumerators are defined (which DWARF and thus BTF preserves). I think it makes sense to keep the original order for those.

maintainer Reviewed-by on v3 1/1
#4 lore raw
In reply to Mykyta Yatsenko [email protected]

2024-05-13 20:29 UTC+0100 ~ Mykyta Yatsenko [email protected]

Quoted 44 lines

From: Mykyta Yatsenko <[email protected]>

Sort bpftool c dump output; aiming to simplify vmlinux.h diffing and forcing more natural type definitions ordering.

Definitions are sorted first by their BTF kind ranks, then by their base type name and by their own name.

Type ranks

Assign ranks to btf kinds (defined in function btf_type_rank) to set next order:
1. Anonymous enums/enums64
2. Named enums/enums64
3. Trivial types typedefs (ints, then floats)
4. Structs/Unions
5. Function prototypes
6. Forward declarations

Type rank is set to maximum for unnamed reference types, structs and unions to avoid emitting those types early. They will be emitted as part of the type chain starting with named type.

Lexicographical ordering

Each type is assigned a sort_name and own_name.
sort_name is the resolved name of the final base type for reference types (typedef, pointer, array etc). Sorting by sort_name allows to group typedefs of the same base type. sort_name for non-reference type is the same as own_name. own_name is a direct name of particular type, is used as final sorting step.

---
 .../bpf/bpftool/Documentation/bpftool-btf.rst |   5 +-
 tools/bpf/bpftool/bash-completion/bpftool     |   3 +
 tools/bpf/bpftool/btf.c                       | 138 +++++++++++++++++-
 3 files changed, 139 insertions(+), 7 deletions(-)
tools/bpf/bpftool/Documentation/bpftool-btf.rst +0 -0
index eaba24320fb2..65eeb3d905f0 100644

tools/bpf/bpftool/Documentation/bpftool-btf.rst BTF COMMANDS · line 31

Quoted 5 lines
tools/bpf/bpftool/Documentation/bpftool-btf.rst +1 -1
@@ -28,7 +28,7 @@ BTF COMMANDS
 | **bpftool** **btf help**
 |
 | *BTF_SRC* := { **id** *BTF_ID* | **prog** *PROG* | **map** *MAP* [{**key** | **value** | **kv** | **all**}] | **file** *FILE* }
-| *FORMAT* := { **raw** | **c** }
+| *FORMAT* := { **raw** | **c** [**unsorted**]}

Missing space at the end, "[**unsorted**] }"

tools/bpf/bpftool/Documentation/bpftool-btf.rst bpftool btf dump *BTF_SRC* · line 67

Quoted 10 lines
tools/bpf/bpftool/Documentation/bpftool-btf.rst +2 -1
 | *MAP* := { **id** *MAP_ID* | **pinned** *FILE* }
 | *PROG* := { **id** *PROG_ID* | **pinned** *FILE* | **tag** *PROG_TAG* | **name** *PROG_NAME* }
 
@@ -63,7 +63,8 @@ bpftool btf dump *BTF_SRC*
     pahole.
 
     **format** option can be used to override default (raw) output format. Raw
-    (**raw**) or C-syntax (**c**) output formats are supported.
+    (**raw**) or C-syntax (**c**) output formats are supported. (**unsorted**)
+    option can be used with (**c**) to avoid sorting the output.

The parenthesis are not part of the formatting here, they are used in the previous sentence to mention the keyword related to the options mentioned in the sentence. So here we could have:

"With C-style formatting, the output is sorted by default. Use the **unsorted** option to avoid sorting the output."

Quoted 21 lines
tools/bpf/bpftool/Documentation/bpftool-btf.rst +0 -0
 bpftool btf help
     Print short help message.
tools/bpf/bpftool/bash-completion/bpftool +3 -0
index 04afe2ac2228..be99d49b8714 100644
@@ -930,6 +930,9 @@ _bpftool()
                         format)
                             COMPREPLY=( $( compgen -W "c raw" -- "$cur" ) )
                             ;;
+                        c)
+                            COMPREPLY=( $( compgen -W "unsorted" -- "$cur" ) )
+                            ;;
                         *)
                             # emit extra options
                             case ${words[3]} in
tools/bpf/bpftool/btf.c +0 -0
index 91fcb75babe3..7e7071d301df 100644

[...]

tools/bpf/bpftool/btf.c static int do_help(int argc, char **argv) · line 1194

Quoted 6 lines
tools/bpf/bpftool/btf.c +1 -1
@@ -1063,7 +1191,7 @@ static int do_help(int argc, char **argv)
 		"       %1$s %2$s help\n"
 		"\n"
 		"       BTF_SRC := { id BTF_ID | prog PROG | map MAP [{key | value | kv | all}] | file FILE }\n"
-		"       FORMAT  := { raw | c }\n"
+		"       FORMAT  := { raw | c [unsorted]}\n"

Missing space as well: "[unsorted] }\n"

tools/bpf/bpftool/btf.c static int do_help(int argc, char **argv) · line 1197

Quoted 3 lines
tools/bpf/bpftool/btf.c +0 -0
 		"       " HELP_SPEC_MAP "\n"
 		"       " HELP_SPEC_PROGRAM "\n"
 		"       " HELP_SPEC_OPTIONS " |\n"

With the above addressed: