Skip to contentSkip to conversation

Netlink protocol specs

PATCH Applied Jakub Kicinski 5 participants 15 messages Jan 20, 2023 – Jan 25, 2023

Subsystems (4)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

v4 cover Netlink protocol specs

I think the Netlink proto specs are far along enough to merge. Filling in all attribute types and quirks will be an ongoing effort but we have enough to cover FOU so it's somewhat complete.

I fully intend to continue polishing the code but at the same time I'd like to start helping others base their work on the specs (e.g. DPLL) and need to start working on some new families myself.

That's the progress / motivation for merging. The RFC [1] has more of a high level blurb, plus I created a lot of documentation, I'm not going to repeat it here. There was also the talk at LPC [2].

[1] https://lore.kernel.org/all/[email protected]/
[2] https://youtu.be/9QkXIQXkaQk?t=2562
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/1

v4:
- spec improvements (patch 2)
- Python cleanup (patch 3)
- rename auto-gen files and use the right comment style

Jakub Kicinski (8):
docs: add more netlink docs (incl. spec docs)
netlink: add schemas for YAML specs
net: add basic C code generators for Netlink
netlink: add a proto specification for FOU
net: fou: regenerate the uAPI from the spec
net: fou: rename the source for linking
net: fou: use policy and operation tables generated from the spec
tools: ynl: add a completely generic client

 Documentation/core-api/index.rst              |    1 +
 Documentation/core-api/netlink.rst            |  101 +
 Documentation/netlink/genetlink-c.yaml        |  333 +++
 Documentation/netlink/genetlink-legacy.yaml   |  356 +++
 Documentation/netlink/genetlink.yaml          |  298 +++
 Documentation/netlink/specs/fou.yaml          |  128 +
 .../userspace-api/netlink/c-code-gen.rst      |  107 +
 .../netlink/genetlink-legacy.rst              |   96 +
 Documentation/userspace-api/netlink/index.rst |    5 +
 Documentation/userspace-api/netlink/specs.rst |  422 +++
 MAINTAINERS                                   |    3 +
 include/uapi/linux/fou.h                      |   54 +-
 net/ipv4/Makefile                             |    1 +
 net/ipv4/{fou.c => fou_core.c}                |   47 +-
 net/ipv4/fou_nl.c                             |   48 +
 net/ipv4/fou_nl.h                             |   25 +
 tools/net/ynl/samples/cli.py                  |   47 +
 tools/net/ynl/samples/ynl.py                  |  534 ++++
 tools/net/ynl/ynl-gen-c.py                    | 2373 +++++++++++++++++
 tools/net/ynl/ynl-regen.sh                    |   30 +
 20 files changed, 4941 insertions(+), 68 deletions(-)
 create mode 100644 Documentation/core-api/netlink.rst
 create mode 100644 Documentation/netlink/genetlink-c.yaml
 create mode 100644 Documentation/netlink/genetlink-legacy.yaml
 create mode 100644 Documentation/netlink/genetlink.yaml
 create mode 100644 Documentation/netlink/specs/fou.yaml
 create mode 100644 Documentation/userspace-api/netlink/c-code-gen.rst
 create mode 100644 Documentation/userspace-api/netlink/genetlink-legacy.rst
 create mode 100644 Documentation/userspace-api/netlink/specs.rst
 rename net/ipv4/{fou.c => fou_core.c} (94%)
 create mode 100644 net/ipv4/fou_nl.c
 create mode 100644 net/ipv4/fou_nl.h
 create mode 100755 tools/net/ynl/samples/cli.py
 create mode 100644 tools/net/ynl/samples/ynl.py
 create mode 100755 tools/net/ynl/ynl-gen-c.py
 create mode 100755 tools/net/ynl/ynl-regen.sh
Signature · 1 line
2.39.0
author maintainer
#1 lore raw

PATCH v4 1/8 docs: add more netlink docs (incl. spec docs)

Add documentation about the upcoming Netlink protocol specs.

--
v4:
- add one more sentence on consistency of dumps
v3:
- spelling from Guillaume
v2:
- plenty -> plenty of
- type: unspec -> type: unused
- ECHO edits from Guillaume

---
 Documentation/core-api/index.rst              |   1 +
 Documentation/core-api/netlink.rst            | 101 +++++
 .../userspace-api/netlink/c-code-gen.rst      | 107 +++++
 .../netlink/genetlink-legacy.rst              |  96 ++++
 Documentation/userspace-api/netlink/index.rst |   5 +
 Documentation/userspace-api/netlink/specs.rst | 422 ++++++++++++++++++
 MAINTAINERS                                   |   2 +
 7 files changed, 734 insertions(+)
 create mode 100644 Documentation/core-api/netlink.rst
 create mode 100644 Documentation/userspace-api/netlink/c-code-gen.rst
 create mode 100644 Documentation/userspace-api/netlink/genetlink-legacy.rst
 create mode 100644 Documentation/userspace-api/netlink/specs.rst
Documentation/core-api/index.rst +1 -0
index 77eb775b8b42..7a3a08d81f11 100644
@@ -127,6 +127,7 @@ Documents that don't fit elsewhere or which have yet to be categorized.
    :maxdepth: 1
 
    librs
+   netlink
 
 .. only:: subproject and html
 
Documentation/core-api/netlink.rst new file +101 -0

101 lines not on this page · raw message

Documentation/userspace-api/netlink/c-code-gen.rst new file +107 -0

107 lines not on this page · raw message

Documentation/userspace-api/netlink/genetlink-legacy.rst new file +96 -0

96 lines not on this page · raw message

Documentation/userspace-api/netlink/index.rst +5 -0
index b0c21538d97d..be250110c8f6 100644
@@ -10,3 +10,8 @@ Netlink documentation for users.
    :maxdepth: 2
 
    intro
+   specs
+   c-code-gen
+   genetlink-legacy
+
+See also :ref:`Documentation/core-api/netlink.rst <kernel_netlink>`.
Documentation/userspace-api/netlink/specs.rst new file +422 -0

422 lines not on this page · raw message

MAINTAINERS +2 -0
index f82dd8d43c2b..3842915d863d 100644
@@ -14563,8 +14563,10 @@ Q:	https://patchwork.kernel.org/project/netdevbpf/list/
 B:	mailto:[email protected]
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
+F:	Documentation/core-api/netlink.rst
 F:	Documentation/networking/
 F:	Documentation/process/maintainer-netdev.rst
+F:	Documentation/userspace-api/netlink/
 F:	include/linux/in.h
 F:	include/linux/net.h
 F:	include/linux/netdevice.h
Signature · 1 line
2.39.0
author maintainer
#2 lore raw

PATCH v4 2/8 netlink: add schemas for YAML specs

In reply to Jakub Kicinski

Add schemas for Netlink spec files. As described in the docs we have 4 "protocols" or compatibility levels, and each one comes with its own schema, but the more general / legacy schemas are superset of more modern ones: genetlink is the smallest followed by genetlink-c and genetlink-legacy. There is no schema for raw netlink, yet, I haven't found the time..

I don't know enough jsonschema to do inheritance or something but the repetition is not too bad. I hope.

---
I did not manage to get the if to work. I works like a charm at the level where the property in the condition is defined, but we also want to allow additional properties deeper in the structure depending on a value of a global property.

v4:
- fix the $id and $schema
- use "dependencies" to select only one of subset-of or name-prefix
- adjust the format of enums everywhere (spaces after/before brackets)
- add min value of integers where it makes sense
- add some $defs for lengths and unsigned int
- improve the pattern for lengths
- fix spelling

---
 Documentation/netlink/genetlink-c.yaml      | 333 ++++++++++++++++++
 Documentation/netlink/genetlink-legacy.yaml | 356 ++++++++++++++++++++
 Documentation/netlink/genetlink.yaml        | 298 ++++++++++++++++
 3 files changed, 987 insertions(+)
 create mode 100644 Documentation/netlink/genetlink-c.yaml
 create mode 100644 Documentation/netlink/genetlink-legacy.yaml
 create mode 100644 Documentation/netlink/genetlink.yaml
Documentation/netlink/genetlink-c.yaml new file +333 -0
new file mode 100644
index 000000000000..e23e3c94a932
@@ -0,0 +1,333 @@
+# SPDX-License-Identifier: GPL-2.0
+%YAML 1.2
+---
+$id: http://kernel.org/schemas/netlink/genetlink-c.yaml#
+$schema: https://json-schema.org/draft-07/schema
+
+# Common defines
+$defs:
+  uint:
+    type: integer
+    minimum: 0
+  len-or-define:
+    type: [ string, integer ]
+    pattern: ^[0-9A-Za-z_]+( - 1)?$
+    minimum: 0
+
+# Schema for specs
+title: Protocol
+description: Specification of a genetlink protocol
+type: object
+required: [ name, doc, attribute-sets, operations ]
+additionalProperties: False
+properties:
+  name:
+    description: Name of the genetlink family.
+    type: string
+  doc:
+    type: string
+  version:
+    description: Generic Netlink family version. Default is 1.
+    type: integer
+    minimum: 1
+  protocol:
+    description: Schema compatibility level. Default is "genetlink".
+    enum: [ genetlink, genetlink-c ]
+  # Start genetlink-c
+  uapi-header:
+    description: Path to the uAPI header, default is linux/${family-name}.h
+    type: string
+  c-family-name:
+    description: Name of the define for the family name.
+    type: string
+  c-version-name:
+    description: Name of the define for the verion of the family.
+    type: string
+  max-by-define:
+    description: Makes the number of attributes and commands be specified by a define, not an enum value.
+    type: boolean
+  # End genetlink-c
+
+  definitions:
+    description: List of type and constant definitions (enums, flags, defines).
+    type: array
+    items:
+      type: object
+      required: [ type, name ]
+      additionalProperties: False
+      properties:
+        name:
+          type: string
+        header:
+          description: For C-compatible languages, header which already defines this value.
+          type: string
+        type:
+          enum: [ const, enum, flags ]
+        doc:
+          type: string
+        # For const
+        value:
+          description: For const - the value.
+          type: [ string, integer ]
+        # For enum and flags
+        value-start:
+          description: For enum or flags the literal initializer for the first value.
+          type: [ string, integer ]
+        entries:
+          description: For enum or flags array of values.
+          type: array
+          items:
+            oneOf:
+              - type: string
+              - type: object
+                required: [ name ]
+                additionalProperties: False
+                properties:
+                  name:
+                    type: string
+                  value:
+                    type: integer
+                  doc:
+                    type: string
+        render-max:
+          description: Render the max members for this enum.
+          type: boolean
+        # Start genetlink-c
+        enum-name:
+          description: Name for enum, if empty no name will be used.
+          type: [ string, "null" ]
+        name-prefix:
+          description: For enum the prefix of the values, optional.
+          type: string
+        # End genetlink-c
+
+  attribute-sets:
+    description: Definition of attribute spaces for this family.
+    type: array
+    items:
+      description: Definition of a single attribute space.
+      type: object
+      required: [ name, attributes ]
+      additionalProperties: False
+      properties:
+        name:
+          description: |
+            Name used when referring to this space in other definitions, not used outside of the spec.
+          type: string
+        name-prefix:
+          description: |
+            Prefix for the C enum name of the attributes. Default family[name]-set[name]-a-
+          type: string
+        enum-name:
+          description: Name for the enum type of the attribute.
+          type: string
+        doc:
+          description: Documentation of the space.
+          type: string
+        subset-of:
+          description: |
+            Name of another space which this is a logical part of. Sub-spaces can be used to define
+            a limited group of attributes which are used in a nest.
+          type: string
+        # Start genetlink-c
+        attr-cnt-name:
+          description: The explicit name for constant holding the count of attributes (last attr + 1).
+          type: string
+        attr-max-name:
+          description: The explicit name for last member of attribute enum.
+          type: string
+        # End genetlink-c
+        attributes:
+          description: List of attributes in the space.
+          type: array
+          items:
+            type: object
+            required: [ name, type ]
+            additionalProperties: False
+            properties:
+              name:
+                type: string
+              type: &attr-type
+                enum: [ unused, pad, flag, binary, u8, u16, u32, u64, s32, s64,
+                        string, nest, array-nest, nest-type-value ]
+              doc:
+                description: Documentation of the attribute.
+                type: string
+              value:
+                description: Value for the enum item representing this attribute in the uAPI.
+                $ref: '#/$defs/uint'
+              type-value:
+                description: Name of the value extracted from the type of a nest-type-value attribute.
+                type: array
+                items:
+                  type: string
+              byte-order:
+                enum: [ little-endian, big-endian ]
+              multi-attr:
+                type: boolean
+              nested-attributes:
+                description: Name of the space (sub-space) used inside the attribute.
+                type: string
+              enum:
+                description: Name of the enum type used for the attribute.
+                type: string
+              enum-as-flags:
+                description: |
+                  Treat the enum as flags. In most cases enum is either used as flags or as values.
+                  Sometimes, however, both forms are necessary, in which case header contains the enum
+                  form while specific attributes may request to convert the values into a bitfield.
+                type: boolean
+              checks:
+                description: Kernel input validation.
+                type: object
+                additionalProperties: False
+                properties:
+                  flags-mask:
+                    description: Name of the flags constant on which to base mask (unsigned scalar types only).
+                    type: string
+                  min:
+                    description: Min value for an integer attribute.
+                    type: integer
+                  min-len:
+                    description: Min length for a binary attribute.
+                    $ref: '#/$defs/len-or-define'
+                  max-len:
+                    description: Max length for a string or a binary attribute.
+                    $ref: '#/$defs/len-or-define'
+              sub-type: *attr-type
+
+      # Make sure name-prefix does not appear in subsets (subsets inherit naming)
+      dependencies:
+        name-prefix:
+          not:
+            required: [ subset-of ]
+        subset-of:
+          not:
+            required: [ name-prefix ]
+
+  operations:
+    description: Operations supported by the protocol.
+    type: object
+    required: [ list ]
+    additionalProperties: False
+    properties:
+      enum-model:
+        description: |
+          The model of assigning values to the operations.
+          "unified" is the recommended model where all message types belong
+          to a single enum.
+          "directional" has the messages sent to the kernel and from the kernel
+          enumerated separately.
+          "notify-split" has the notifications and request-response types in
+          different enums.
+        enum: [ unified, directional, notify-split ]
+      name-prefix:
+        description: |
+          Prefix for the C enum name of the command. The name is formed by concatenating
+          the prefix with the upper case name of the command, with dashes replaced by underscores.
+        type: string
+      enum-name:
+        description: Name for the enum type with commands.
+        type: string
+      async-prefix:
+        description: Same as name-prefix but used to render notifications and events to separate enum.
+        type: string
+      async-enum:
+        description: Name for the enum type with notifications/events.
+        type: string
+      list:
+        description: List of commands
+        type: array
+        items:
+          type: object
+          additionalProperties: False
+          required: [ name, doc ]
+          properties:
+            name:
+              description: Name of the operation, also defining its C enum value in uAPI.
+              type: string
+            doc:
+              description: Documentation for the command.
+              type: string
+            value:
+              description: Value for the enum in the uAPI.
+              $ref: '#/$defs/uint'
+            attribute-set:
+              description: |
+                Attribute space from which attributes directly in the requests and replies
+                to this command are defined.
+              type: string
+            flags: &cmd_flags
+              description: Command flags.
+              type: array
+              items:
+                enum: [ admin-perm ]
+            dont-validate:
+              description: Kernel attribute validation flags.
+              type: array
+              items:
+                enum: [ strict, dump ]
+            do: &subop-type
+              description: Main command handler.
+              type: object
+              additionalProperties: False
+              properties:
+                request: &subop-attr-list
+                  description: Definition of the request message for a given command.
+                  type: object
+                  additionalProperties: False
+                  properties:
+                    attributes:
+                      description: |
+                        Names of attributes from the attribute-set (not full attribute
+                        definitions, just names).
+                      type: array
+                      items:
+                        type: string
+                reply: *subop-attr-list
+                pre:
+                  description: Hook for a function to run before the main callback (pre_doit or start).
+                  type: string
+                post:
+                  description: Hook for a function to run after the main callback (post_doit or done).
+                  type: string
+            dump: *subop-type
+            notify:
+              description: Name of the command sharing the reply type with this notification.
+              type: string
+            event:
+              type: object
+              additionalProperties: False
+              properties:
+                attributes:
+                  description: Explicit list of the attributes for the notification.
+                  type: array
+                  items:
+                    type: string
+            mcgrp:
+              description: Name of the multicast group generating given notification.
+              type: string
+  mcast-groups:
+    description: List of multicast groups.
+    type: object
+    required: [ list ]
+    additionalProperties: False
+    properties:
+      list:
+        description: List of groups.
+        type: array
+        items:
+          type: object
+          required: [ name ]
+          additionalProperties: False
+          properties:
+            name:
+              description: |
+                The name for the group, used to form the define and the value of the define.
+              type: string
+            # Start genetlink-c
+            c-define-name:
+              description: Override for the name of the define in C uAPI.
+              type: string
+            # End genetlink-c
+            flags: *cmd_flags
Documentation/netlink/genetlink-legacy.yaml new file +356 -0

356 lines not on this page · raw message

Documentation/netlink/genetlink.yaml new file +298 -0

298 lines not on this page · raw message

Signature · 1 line
2.39.0
author maintainer
#3 lore raw

PATCH v4 3/8 net: add basic C code generators for Netlink

In reply to Jakub Kicinski

Code generators to turn Netlink specs into C code. I'm definitely not proud of it.

The main generator is in Python, there's a bash script to regen all code-gen'ed files in tree after making spec changes.

--
v2: - use /* */ comments instead of //
v4:
- remove unnecessary f-strings
- use get(, default)
- improve exception for unsupported checks
- remove empty __init__
- (more) consistently use ternary / inline ifs
- use c_upper() / c_lower() everywhere
- rename c_kw -> _C_KW
- actually use /* */ comments

---
 MAINTAINERS                |    1 +
 tools/net/ynl/ynl-gen-c.py | 2373 ++++++++++++++++++++++++++++++++++++
 tools/net/ynl/ynl-regen.sh |   30 +
 3 files changed, 2404 insertions(+)
 create mode 100755 tools/net/ynl/ynl-gen-c.py
 create mode 100755 tools/net/ynl/ynl-regen.sh
MAINTAINERS +1 -0
index 3842915d863d..75a1395c8023 100644
@@ -14578,6 +14578,7 @@ F:	include/uapi/linux/netdevice.h
 F:	lib/net_utils.c
 F:	lib/random32.c
 F:	net/
+F:	tools/net/
 F:	tools/testing/selftests/net/
 
 NETWORKING [IPSEC]
tools/net/ynl/ynl-gen-c.py new file +2373 -0

2,373 lines not on this page · raw message

tools/net/ynl/ynl-regen.sh new file +30 -0
new file mode 100755
index 000000000000..43989ae48ed0
@@ -0,0 +1,30 @@
+#!/bin/bash
+# SPDX-License-Identifier: BSD-3-Clause
+
+TOOL=$(dirname $(realpath $0))/ynl-gen-c.py
+
+force=
+
+while [ ! -z "$1" ]; do
+  case "$1" in
+    -f ) force=yes; shift ;;
+    * )  echo "Unrecognized option '$1'"; exit 1 ;;
+  esac
+done
+
+KDIR=$(dirname $(dirname $(dirname $(dirname $(realpath $0)))))
+
+files=$(git grep --files-with-matches '^/\* YNL-GEN \(kernel\|uapi\)')
+for f in $files; do
+    # params:     0       1      2     3
+    #         $YAML YNL-GEN kernel $mode
+    params=( $(git grep -B1 -h '/\* YNL-GEN' $f | sed 's@/\*\(.*\)\*/@\1@') )
+
+    if [ $f -nt ${params[0]} -a -z "$force" ]; then
+	echo -e "\tSKIP $f"
+	continue
+    fi
+
+    echo -e "\tGEN ${params[2]}\t$f"
+    $TOOL --mode ${params[2]} --${params[3]} --spec $KDIR/${params[0]} -o $f
+done
Signature · 1 line
2.39.0
author maintainer
#4 lore raw

PATCH v4 4/8 netlink: add a proto specification for FOU

In reply to Jakub Kicinski

FOU has a reasonably modern Genetlink family. Add a spec.

---
 Documentation/netlink/specs/fou.yaml | 128 +++++++++++++++++++++++++++
 1 file changed, 128 insertions(+)
 create mode 100644 Documentation/netlink/specs/fou.yaml
Documentation/netlink/specs/fou.yaml new file +128 -0
new file mode 100644
index 000000000000..266c386eedf3
@@ -0,0 +1,128 @@
+name: fou
+
+protocol: genetlink-legacy
+
+doc: |
+  Foo-over-UDP.
+
+c-family-name: fou-genl-name
+c-version-name: fou-genl-version
+max-by-define: true
+kernel-policy: global
+
+definitions:
+  -
+    type: enum
+    name: encap_type
+    name-prefix: fou-encap-
+    enum-name:
+    entries: [ unspec, direct, gue ]
+
+attribute-sets:
+  -
+    name: fou
+    name-prefix: fou-attr-
+    attributes:
+      -
+        name: unspec
+        type: unused
+      -
+        name: port
+        type: u16
+        byte-order: big-endian
+      -
+        name: af
+        type: u8
+      -
+        name: ipproto
+        type: u8
+      -
+        name: type
+        type: u8
+      -
+        name: remcsum_nopartial
+        type: flag
+      -
+        name: local_v4
+        type: u32
+      -
+        name: local_v6
+        type: binary
+        checks:
+          min-len: 16
+      -
+        name: peer_v4
+        type: u32
+      -
+        name: peer_v6
+        type: binary
+        checks:
+          min-len: 16
+      -
+        name: peer_port
+        type: u16
+        byte-order: big-endian
+      -
+        name: ifindex
+        type: s32
+
+operations:
+  list:
+    -
+      name: unspec
+      doc: unused
+
+    -
+      name: add
+      doc: Add port.
+      attribute-set: fou
+
+      dont-validate: [ strict, dump ]
+      flags: [ admin-perm ]
+
+      do:
+        request: &all_attrs
+          attributes:
+            - port
+            - ipproto
+            - type
+            - remcsum_nopartial
+            - local_v4
+            - peer_v4
+            - local_v6
+            - peer_v6
+            - peer_port
+            - ifindex
+
+    -
+      name: del
+      doc: Delete port.
+      attribute-set: fou
+
+      dont-validate: [ strict, dump ]
+      flags: [ admin-perm ]
+
+      do:
+        request:  &select_attrs
+          attributes:
+          - af
+          - ifindex
+          - port
+          - peer_port
+          - local_v4
+          - peer_v4
+          - local_v6
+          - peer_v6
+
+    -
+      name: get
+      doc: Get tunnel info.
+      attribute-set: fou
+      dont-validate: [ strict, dump ]
+
+      do:
+        request: *select_attrs
+        reply: *all_attrs
+
+      dump:
+        reply: *all_attrs
Signature · 1 line
2.39.0
author maintainer
#5 lore raw

PATCH v4 5/8 net: fou: regenerate the uAPI from the spec

In reply to Jakub Kicinski

Regenerate the FOU uAPI header from the YAML spec.

The flags now come before attributes which use them, and the comments for type disappear (coders should look at the spec instead).

---
 include/uapi/linux/fou.h | 54 +++++++++++++++++++---------------------
 1 file changed, 26 insertions(+), 28 deletions(-)
include/uapi/linux/fou.h +26 -28
index 87c2c9f08803..19ebbef41a63 100644
@@ -1,32 +1,37 @@
 /* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
-/* fou.h - FOU Interface */
+/* Do not edit directly, auto-generated from: */
+/*	Documentation/netlink/specs/fou.yaml */
+/* YNL-GEN uapi header */
 
 #ifndef _UAPI_LINUX_FOU_H
 #define _UAPI_LINUX_FOU_H
 
-/* NETLINK_GENERIC related info
- */
 #define FOU_GENL_NAME		"fou"
-#define FOU_GENL_VERSION	0x1
+#define FOU_GENL_VERSION	1
 
 enum {
-	FOU_ATTR_UNSPEC,
-	FOU_ATTR_PORT,				/* u16 */
-	FOU_ATTR_AF,				/* u8 */
-	FOU_ATTR_IPPROTO,			/* u8 */
-	FOU_ATTR_TYPE,				/* u8 */
-	FOU_ATTR_REMCSUM_NOPARTIAL,		/* flag */
-	FOU_ATTR_LOCAL_V4,			/* u32 */
-	FOU_ATTR_LOCAL_V6,			/* in6_addr */
-	FOU_ATTR_PEER_V4,			/* u32 */
-	FOU_ATTR_PEER_V6,			/* in6_addr */
-	FOU_ATTR_PEER_PORT,			/* u16 */
-	FOU_ATTR_IFINDEX,			/* s32 */
-
-	__FOU_ATTR_MAX,
+	FOU_ENCAP_UNSPEC,
+	FOU_ENCAP_DIRECT,
+	FOU_ENCAP_GUE,
 };
 
-#define FOU_ATTR_MAX		(__FOU_ATTR_MAX - 1)
+enum {
+	FOU_ATTR_UNSPEC,
+	FOU_ATTR_PORT,
+	FOU_ATTR_AF,
+	FOU_ATTR_IPPROTO,
+	FOU_ATTR_TYPE,
+	FOU_ATTR_REMCSUM_NOPARTIAL,
+	FOU_ATTR_LOCAL_V4,
+	FOU_ATTR_LOCAL_V6,
+	FOU_ATTR_PEER_V4,
+	FOU_ATTR_PEER_V6,
+	FOU_ATTR_PEER_PORT,
+	FOU_ATTR_IFINDEX,
+
+	__FOU_ATTR_MAX
+};
+#define FOU_ATTR_MAX (__FOU_ATTR_MAX - 1)
 
 enum {
 	FOU_CMD_UNSPEC,
@@ -34,15 +39,8 @@ enum {
 	FOU_CMD_DEL,
 	FOU_CMD_GET,
 
-	__FOU_CMD_MAX,
+	__FOU_CMD_MAX
 };
-
-enum {
-	FOU_ENCAP_UNSPEC,
-	FOU_ENCAP_DIRECT,
-	FOU_ENCAP_GUE,
-};
-
-#define FOU_CMD_MAX	(__FOU_CMD_MAX - 1)
+#define FOU_CMD_MAX (__FOU_CMD_MAX - 1)
 
 #endif /* _UAPI_LINUX_FOU_H */
Signature · 1 line
2.39.0
#6 lore raw

Re: [PATCH net-next v4 5/8] net: fou: regenerate the uAPI from the spec

From: Jakub Kicinski <[email protected]>
Date: Fri, 20 Jan 2023 09:50:38 -0800

Quoted 5 lines

Regenerate the FOU uAPI header from the YAML spec.

The flags now come before attributes which use them, and the comments for type disappear (coders should look at the spec instead).

Sorry I missed the whole history of this topic. Wanted to ask: if we can generate these headers and even C files, why ship the generated with the source code and not generate them during building? Or it's slow and/or requires some software etc.?

Quoted 6 lines
---
 include/uapi/linux/fou.h | 54 +++++++++++++++++++---------------------
 1 file changed, 26 insertions(+), 28 deletions(-)

[...]

include/uapi/linux/fou.h enum { · line 46

Quoted 4 lines
include/uapi/linux/fou.h +1 -1
-#define FOU_CMD_MAX	(__FOU_CMD_MAX - 1)
+#define FOU_CMD_MAX (__FOU_CMD_MAX - 1)
 
 #endif /* _UAPI_LINUX_FOU_H */

Thanks,
Olek

#7 lore raw

Re: [PATCH net-next v4 5/8] net: fou: regenerate the uAPI from the spec

Quoted 5 lines · Alexander Lobakin

From: Jakub Kicinski <[email protected]>
Date: Fri, 20 Jan 2023 09:50:38 -0800

Quoted 2 lines

Regenerate the FOU uAPI header from the YAML spec.

Quoted 9 lines
Quoted 3 lines

The flags now come before attributes which use them, and the comments for type disappear (coders should look at the spec instead).

Sorry I missed the whole history of this topic. Wanted to ask: if we can generate these headers and even C files, why ship the generated with the source code and not generate them during building? Or it's slow and/or requires some software etc.?

Currently it requires python 3 (3.6+, I'd think?).

Python is currently not documented as a build requirement in Documentation/process/changes.rst afaict.

johannes

author maintainer
#8 lore raw

Re: [PATCH net-next v4 5/8] net: fou: regenerate the uAPI from the spec

Quoted 15 lines · On Tue, 24 Jan 2023 19:50:40 +0100 Johannes Berg wrote:
Quoted 5 lines · Alexander Lobakin

From: Jakub Kicinski <[email protected]>
Date: Fri, 20 Jan 2023 09:50:38 -0800

Quoted 2 lines

Regenerate the FOU uAPI header from the YAML spec.

Quoted 8 lines
Quoted 3 lines

The flags now come before attributes which use them, and the comments for type disappear (coders should look at the spec instead).

Sorry I missed the whole history of this topic. Wanted to ask: if we can generate these headers and even C files, why ship the generated with the source code and not generate them during building? Or it's slow and/or requires some software etc.?

Quoted 4 lines

Currently it requires python 3 (3.6+, I'd think?).

Python is currently not documented as a build requirement in Documentation/process/changes.rst afaict.

Yes, I wanted to avoid bundling in changes which could be controversial. Whether code is generated during build or committed is something we can revisit at any point.

#9 lore raw

Re: [PATCH net-next v4 5/8] net: fou: regenerate the uAPI from the spec

From: Jakub Kicinski <[email protected]>
Date: Tue, 24 Jan 2023 16:40:07 -0800

Quoted 16 lines · On Tue, 24 Jan 2023 19:50:40 +0100 Johannes Berg wrote:
Quoted 15 lines · Alexander Lobakin
Quoted 13 lines

From: Jakub Kicinski <[email protected]>
Date: Fri, 20 Jan 2023 09:50:38 -0800

Quoted 5 lines

Regenerate the FOU uAPI header from the YAML spec.

The flags now come before attributes which use them, and the comments for type disappear (coders should look at the spec instead).

Sorry I missed the whole history of this topic. Wanted to ask: if we can generate these headers and even C files, why ship the generated with the source code and not generate them during building? Or it's slow and/or requires some software etc.?

Quoted 8 lines
Quoted 4 lines

Currently it requires python 3 (3.6+, I'd think?).

Python is currently not documented as a build requirement in Documentation/process/changes.rst afaict.

Yes, I wanted to avoid bundling in changes which could be controversial. Whether code is generated during build or committed is something we can revisit at any point.

+, got it, thanks!

The only reason I asked is that I was hoping it would allow us to remove a couple thousand locs because why not :D
At least the atomic functions are designed the same way (the tree has the generated code committed/included), so it's totally fine.

I like what this series and the idea in general does, thanks!

Olek

author maintainer
#10 lore raw

PATCH v4 6/8 net: fou: rename the source for linking

In reply to Jakub Kicinski

We'll need to link two objects together to form the fou module. This means the source can't be called fou, the build system expects fou.o to be the combined object.

---
 net/ipv4/Makefile              | 1 +
 net/ipv4/{fou.c => fou_core.c} | 0
 2 files changed, 1 insertion(+)
 rename net/ipv4/{fou.c => fou_core.c} (100%)
net/ipv4/Makefile +1 -0
index af7d2cf490fb..fabbe46897ce 100644
@@ -26,6 +26,7 @@ obj-$(CONFIG_IP_MROUTE) += ipmr.o
 obj-$(CONFIG_IP_MROUTE_COMMON) += ipmr_base.o
 obj-$(CONFIG_NET_IPIP) += ipip.o
 gre-y := gre_demux.o
+fou-y := fou_core.o
 obj-$(CONFIG_NET_FOU) += fou.o
 obj-$(CONFIG_NET_IPGRE_DEMUX) += gre.o
 obj-$(CONFIG_NET_IPGRE) += ip_gre.o
net/ipv4/fou_core.c renamed from net/ipv4/fou.c +0 -0
similarity index 100%
rename from net/ipv4/fou.c
rename to net/ipv4/fou_core.c
Signature · 1 line
2.39.0
author maintainer
#11 lore raw

PATCH v4 7/8 net: fou: use policy and operation tables generated from the spec

In reply to Jakub Kicinski

Generate and plug in the spec-based tables.

A little bit of renaming is needed in the FOU code.

---
v4:
- use _ in the names of files instead of -
- regen with /* */ comments

---
 net/ipv4/Makefile   |  2 +-
 net/ipv4/fou_core.c | 47 +++++++-------------------------------------
 net/ipv4/fou_nl.c   | 48 +++++++++++++++++++++++++++++++++++++++++++++
 net/ipv4/fou_nl.h   | 25 +++++++++++++++++++++++
 4 files changed, 81 insertions(+), 41 deletions(-)
 create mode 100644 net/ipv4/fou_nl.c
 create mode 100644 net/ipv4/fou_nl.h
net/ipv4/Makefile +1 -1
index fabbe46897ce..880277c9fd07 100644
@@ -26,7 +26,7 @@ obj-$(CONFIG_IP_MROUTE) += ipmr.o
 obj-$(CONFIG_IP_MROUTE_COMMON) += ipmr_base.o
 obj-$(CONFIG_NET_IPIP) += ipip.o
 gre-y := gre_demux.o
-fou-y := fou_core.o
+fou-y := fou_core.o fou_nl.o
 obj-$(CONFIG_NET_FOU) += fou.o
 obj-$(CONFIG_NET_IPGRE_DEMUX) += gre.o
 obj-$(CONFIG_NET_IPGRE) += ip_gre.o
net/ipv4/fou_core.c +7 -40

94 lines not on this page · raw message

net/ipv4/fou_nl.c new file +48 -0

48 lines not on this page · raw message

net/ipv4/fou_nl.h new file +25 -0
new file mode 100644
index 000000000000..b7a68121ce6f
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause */
+/* Do not edit directly, auto-generated from: */
+/*	Documentation/netlink/specs/fou.yaml */
+/* YNL-GEN kernel header */
+
+#ifndef _LINUX_FOU_GEN_H
+#define _LINUX_FOU_GEN_H
+
+#include <net/netlink.h>
+#include <net/genetlink.h>
+
+#include <linux/fou.h>
+
+/* Global operation policy for fou */
+extern const struct nla_policy fou_nl_policy[FOU_ATTR_IFINDEX + 1];
+
+/* Ops table for fou */
+extern const struct genl_small_ops fou_nl_ops[3];
+
+int fou_nl_add_doit(struct sk_buff *skb, struct genl_info *info);
+int fou_nl_del_doit(struct sk_buff *skb, struct genl_info *info);
+int fou_nl_get_doit(struct sk_buff *skb, struct genl_info *info);
+int fou_nl_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb);
+
+#endif /* _LINUX_FOU_GEN_H */
Signature · 1 line
2.39.0
author maintainer
#12 lore raw

PATCH v4 8/8 tools: ynl: add a completely generic client

In reply to Jakub Kicinski

Add a CLI sample which can take in arbitrary request in JSON format, convert it to Netlink and do the inverse for output.

It's meant as a development tool primarily and perhaps for selftests which need to tickle netlink in a special way.

---
 tools/net/ynl/samples/cli.py |  47 +++
 tools/net/ynl/samples/ynl.py | 534 +++++++++++++++++++++++++++++++++++
 2 files changed, 581 insertions(+)
 create mode 100755 tools/net/ynl/samples/cli.py
 create mode 100644 tools/net/ynl/samples/ynl.py
tools/net/ynl/samples/cli.py new file +47 -0
new file mode 100755
index 000000000000..b27159c70710
@@ -0,0 +1,47 @@
+#!/usr/bin/env python
+# SPDX-License-Identifier: BSD-3-Clause
+
+import argparse
+import json
+import pprint
+import time
+
+from ynl import YnlFamily
+
+
+def main():
+    parser = argparse.ArgumentParser(description='YNL CLI sample')
+    parser.add_argument('--spec', dest='spec', type=str, required=True)
+    parser.add_argument('--schema', dest='schema', type=str)
+    parser.add_argument('--json', dest='json_text', type=str)
+    parser.add_argument('--do', dest='do', type=str)
+    parser.add_argument('--dump', dest='dump', type=str)
+    parser.add_argument('--sleep', dest='sleep', type=int)
+    parser.add_argument('--subscribe', dest='ntf', type=str)
+    args = parser.parse_args()
+
+    attrs = {}
+    if args.json_text:
+        attrs = json.loads(args.json_text)
+
+    ynl = YnlFamily(args.spec, args.schema)
+
+    if args.ntf:
+        ynl.ntf_subscribe(args.ntf)
+
+    if args.sleep:
+        time.sleep(args.sleep)
+
+    if args.do or args.dump:
+        method = getattr(ynl, args.do if args.do else args.dump)
+
+        reply = method(attrs, dump=bool(args.dump))
+        pprint.PrettyPrinter().pprint(reply)
+
+    if args.ntf:
+        ynl.check_ntf()
+        pprint.PrettyPrinter().pprint(ynl.async_msg_queue)
+
+
+if __name__ == "__main__":
+    main()
tools/net/ynl/samples/ynl.py new file +534 -0

534 lines not on this page · raw message

Signature · 1 line
2.39.0
maintainer
#13 lore raw
In reply to Jakub Kicinski
Quoted 9 lines · Jakub Kicinski

I think the Netlink proto specs are far along enough to merge. Filling in all attribute types and quirks will be an ongoing effort but we have enough to cover FOU so it's somewhat complete.

I fully intend to continue polishing the code but at the same time I'd like to start helping others base their work on the specs (e.g. DPLL) and need to start working on some new families myself.

Quoted 8 lines

That's the progress / motivation for merging. The RFC [1] has more of a high level blurb, plus I created a lot of documentation, I'm not going to repeat it here. There was also the talk at LPC [2].

[1] https://lore.kernel.org/all/[email protected]/
[2] https://youtu.be/9QkXIQXkaQk?t=2562
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/1

My understanding is that there is agreement on this, so merging as-is.

As possible follow-ups, I think it would be nice to drop the generated files from the source tree and instead trigger the re-generation on- demand. Additionally the yaml files could include an SPDX-License- Identifier in an initial comments, I think.

Thanks,

Paolo

p.s. I guess we have new todo item for the mptcp protocol ;)

#14 lore raw
In reply to Jakub Kicinski
This series was applied to netdev/net-next.git (master) by Paolo Abeni <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (master) by Paolo Abeni <[email protected]>:

Quoted 10 lines · On Fri, 20 Jan 2023 09:50:33 -0800 you wrote:

I think the Netlink proto specs are far along enough to merge. Filling in all attribute types and quirks will be an ongoing effort but we have enough to cover FOU so it's somewhat complete.

I fully intend to continue polishing the code but at the same time I'd like to start helping others base their work on the specs (e.g. DPLL) and need to start working on some new families myself.

[...]

Here is the summary with links:
- [net-next,v4,1/8] docs: add more netlink docs (incl. spec docs)

    https://git.kernel.org/netdev/net-next/c/9d6a65079c98
  - [net-next,v4,2/8] netlink: add schemas for YAML specs
    https://git.kernel.org/netdev/net-next/c/e616c07ca518
  - [net-next,v4,3/8] net: add basic C code generators for Netlink
    https://git.kernel.org/netdev/net-next/c/be5bea1cc0bf
  - [net-next,v4,4/8] netlink: add a proto specification for FOU
    https://git.kernel.org/netdev/net-next/c/4eb77b4ecd3c
  - [net-next,v4,5/8] net: fou: regenerate the uAPI from the spec
    https://git.kernel.org/netdev/net-next/c/3a330496baa8
  - [net-next,v4,6/8] net: fou: rename the source for linking
    https://git.kernel.org/netdev/net-next/c/08d323234d10
  - [net-next,v4,7/8] net: fou: use policy and operation tables generated from the spec
    https://git.kernel.org/netdev/net-next/c/1d562c32e439
  - [net-next,v4,8/8] tools: ynl: add a completely generic client
    https://git.kernel.org/netdev/net-next/c/e4b48ed460d3

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html