Skip to contentSkip to conversation

dtc: Add a plugin interface

RFC PATCH In review Andrei Ziureaev 2 participants 6 messages Sep 15, 2020 – Nov 19, 2020 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v4 cover dtc: Add a plugin interface

From: Andrei Ziureaev <[email protected]>

I added full documentation to the live tree API and changed the style of comments in "dtc-plugin.h" to kernel-doc.

The live tree can be kept read-only. So, the input to a plugin is some information about the tree and its outputs are printed error messages and an ack/nack return back to DTC.
===========

This is one possible implementation of a plugin interface for DTC.

A plugin interface would allow us to add checks in languages other than C. Specifically, it would allow DTC and DT Schema (which is written in Python) to be integrated.

It would also allow for better debugging of dts files. For example, DT Schema would be able to print dts source line information, without us having to add source line annotations to yaml files.

Plugins are allowed read-only access to the internal unflattened device tree using the provided API. They can print error messages and return a boolean status back to DTC.

There's a question of whether we should relicense some headers to dual BSD to be able call Python code.

Any thoughts would be much appreciated.

Thanks,
Andrei.

Changes in v4:
- full documentation for live tree API
- changed comments to kernel-doc
- simplified "Exporting Functionality" in the manual
- the docs mention that the tree is read-only

Changes in v3:
- live tree API (dt.h and dt.c)
- additional functionality in dtc.h
- plugins have to implement prototypes
- improved documentation

Changes in v2:
- improved documentation
- plugins must register with the build system
- the "validate_fn_t" hook can return a status
- define "struct reserve_info" in "dtc-plugin.h"
- specify that minor versions are compatible
- check if plugin_dir is NULL, just in case
- better variable names in register_plugin_info

Andrei Ziureaev (4):
dtc: Add marker type functionality to dtc.h
dtc: Add a live tree API
dtc: Add plugin documentation and examples
dtc: Add a plugin interface

 Documentation/manual.txt         | 154 ++++++++++++++++
 Makefile                         |  29 ++-
 Makefile.dtc                     |   1 +
 dt.c                             | 158 ++++++++++++++++
 dt.h                             | 299 +++++++++++++++++++++++++++++++
 dtc-plugin.h                     |  90 ++++++++++
 dtc.c                            | 142 ++++++++++++++-
 dtc.h                            |  57 ++++++
 plugins/example/Makefile.example |  27 +++
 plugins/example/example.c        |  33 ++++
 treesource.c                     |   8 +-
 11 files changed, 992 insertions(+), 6 deletions(-)
 create mode 100644 dt.c
 create mode 100644 dt.h
 create mode 100644 dtc-plugin.h
 create mode 100644 plugins/example/Makefile.example
 create mode 100644 plugins/example/example.c
Signature · 1 line
2.17.1
#2 lore raw

RFC PATCH v4 1/4 dtc: Add marker type functionality to dtc.h

In reply to a message not in the archive

From: Andrei Ziureaev <[email protected]>

Add an inline helper function to dtc.h that checks if there's a marker of a type at an offset.

Also, there are two useful static functions in treesource.c. Make them public, add their prototypes to dtc.h and rename one of them.

---
 dtc.h        | 11 +++++++++++
 treesource.c |  8 ++++----
 2 files changed, 15 insertions(+), 4 deletions(-)
dtc.h +11 -0
index a08f415..286d999 100644
@@ -125,6 +125,17 @@ struct data {
 	for_each_marker(m) \
 		if ((m)->type == (t))
 
+static inline bool markers_have_type_at_offset(struct marker *m, enum markertype type, int off)
+{
+	for_each_marker_of_type(m, type)
+		if (m->offset == off)
+			return true;
+
+	return false;
+}
+
+bool marker_has_data_type_information(struct marker *m);
+struct marker *next_type_marker(struct marker *m);
 size_t type_marker_length(struct marker *m);
 
 void data_free(struct data d);
treesource.c +4 -4
index 061ba8c..0785d83 100644
@@ -124,14 +124,14 @@ static void write_propval_int(FILE *f, const char *p, size_t len, size_t width)
 	}
 }
 
-static bool has_data_type_information(struct marker *m)
+bool marker_has_data_type_information(struct marker *m)
 {
 	return m->type >= TYPE_UINT8;
 }
 
-static struct marker *next_type_marker(struct marker *m)
+struct marker *next_type_marker(struct marker *m)
 {
-	while (m && !has_data_type_information(m))
+	while (m && !marker_has_data_type_information(m))
 		m = m->next;
 	return m;
 }
@@ -230,7 +230,7 @@ static void write_propval(FILE *f, struct property *prop)
 		size_t data_len = type_marker_length(m) ? : len - m->offset;
 		const char *p = &prop->val.val[m->offset];
 
-		if (has_data_type_information(m)) {
+		if (marker_has_data_type_information(m)) {
 			emit_type = m->type;
 			fprintf(f, " %s", delim_start[emit_type]);
 		} else if (m->type == LABEL)
Signature · 1 line
2.17.1
#3 lore raw

RFC PATCH v4 2/4 dtc: Add a live tree API

In reply to a message not in the archive

From: Andrei Ziureaev <[email protected]>

The purpose of this API is to completely abstract away the internal live tree implementation.

This API can traverse the tree and read its values. This is enough to validate the tree in Python, for example. Specifically, the API gives read-only access to:
- source positions (directory names, file names, line numbers)
- node names and property names
- values of properties (strings and ints)
- if the value is an int, its width and whether it's a phandle

---

Changes in v4:
- full documentation

---
 Makefile.dtc |   1 +
 dt.c         | 158 ++++++++++++++++++++++++++
 dt.h         | 305 +++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 464 insertions(+)
 create mode 100644 dt.c
 create mode 100644 dt.h
Makefile.dtc +1 -0
index 9c467b0..09b3123 100644
@@ -7,6 +7,7 @@
 DTC_SRCS = \
 	checks.c \
 	data.c \
+	dt.c \
 	dtc.c \
 	flattree.c \
 	fstree.c \
dt.c new file +158 -0

158 lines not on this page · raw message

dt.h new file +305 -0

305 lines not on this page · raw message

Signature · 1 line
2.17.1
#4 lore raw

RFC PATCH v4 3/4 dtc: Add plugin documentation and examples

In reply to a message not in the archive

From: Andrei Ziureaev <[email protected]>

Document the plugin API in Documentation/manual.txt and provide an example plugin.

---

Changes in v4:
- changed comments to kernel-doc
- simplified "Exporting Functionality"
- mention that live tree is read-only

Changes in v3:
- plugins have to implement prototypes
- added license tags to the example
- copied some definitions from dtc-plugin.h to the manual
- better wording

Changes in v2:
- better structure
- information about the data model under "5.2) Exporting Functionality"
- plugins must register with the build system
- the "validate_fn_t" hook can return a status

---
 Documentation/manual.txt         | 152 +++++++++++++++++++++++++++++++
 plugins/example/Makefile.example |  27 ++++++
 plugins/example/example.c        |  33 +++++++
 3 files changed, 212 insertions(+)
 create mode 100644 plugins/example/Makefile.example
 create mode 100644 plugins/example/example.c
Documentation/manual.txt +152 -0
index adf5ccb..f23d3a0 100644
@@ -10,6 +10,9 @@ I - "dtc", the device tree compiler
     4.1) Overview
     4.2) Properties
     4.3) Labels and References
+    5) Plugins
+    5.1) Building and Installing
+    5.2) Exporting Functionality
 
 II - The DT block format
     1) Header
@@ -115,6 +118,16 @@ Options:
     -d <dependency_filename>
 	Generate a dependency file during compilation.
 
+    -l, --plugin <plugin name>[,key[,value]]
+	Load a plugin and, optionally, pass it an argument.
+		plugin name - the name of the shared object without the extension (can contain a path)
+		key         - the name of the argument
+		value       - the value of the argument (can be omitted)
+	Example: dtc -l some-plugin,o,out.dts -l some-plugin,help -l another-plugin [...]
+
+    -L, --plugin-dir <arg>
+	The directory in which to search for plugins
+
     -q
 	Quiet: -q suppress warnings, -qq errors, -qqq all
 
@@ -272,6 +285,145 @@ And used in properties, labels may appear before or after any value:
     };
 
 
+5) Plugins
+
+Plugins are shared libraries that DTC loads at runtime using
+
+	dlopen(path, RTLD_NOW | RTLD_GLOBAL | RTLD_DEEPBIND)
+
+RTLD_NOW means they are loaded immediately (right before the parsing
+stage).
+
+RTLD_GLOBAL means their symbols can be used by other plugins.
+
+RTLD_DEEPBIND means the dynamic linker will look for symbols within the
+plugin first, before searching in the main program and other plugins.
+
+All plugins must include the "dtc-plugin.h" header.
+
+
+5.1) Building and Installing
+
+Plugins are built and installed using the command "make plugins".
+
+Suppose your plugin is called "example" and its source code is in
+"plugins/example/example.c". To register it with the build system,
+create a file "plugins/example/Makefile.example" with the following
+line as its contents:
+
+PLUGIN_LIBS += $(PLUGIN_dir)/example/example.so
+
+This means "make plugins" will try to build
+
+	plugins/example/example.so
+
+from
+
+	plugins/example/example.c
+
+It will also make a symlink
+
+	plugins/example.so
+
+for convenience. You could then call DTC like this:
+
+    ./dtc -Onull some-file.dts -l plugins/example
+
+Please, see "plugins/example/Makefile.example" for how to override the
+default behavior, add GCC flags, etc.
+
+
+5.2) Exporting Functionality
+
+- From "dtc-plugin.h":
+
+```
+/**
+ * DOC: How to Export Functionality
+ *
+ * Plugins export functionality by implementing one or more of the
+ * functions below. DTC tries to call each function exactly once for
+ * each plugin.
+ *
+ * There are typedefs for conveniently storing pointers to these
+ * functions.
+ */
+
+/**
+ * dtc_init() - Initialize the plugin.
+ * @dtc_ver: DTC's plugin API version
+ * @argc:    Length of @argv
+ * @argv:    Array of plugin arguments
+ *
+ * Called right after the plugin is loaded.
+ *
+ * Every plugin must implement this. At the very least, it should
+ * perform a version check.
+ *
+ * Return: 0 on success, or 1 on failure
+ */
+int dtc_init(struct plugin_version dtc_ver, int argc, struct plugin_arg *argv);
+typedef int (*dtc_init_fn_t)(struct plugin_version dtc_ver, int argc,
+			     struct plugin_arg *argv);
+
+/**
+ * dtc_validate() - Validate a device tree.
+ * @dti: The unflattened device tree. The header ``dt.h`` contains
+ *       functionality for read-only access to its data.
+ *
+ * Called after DTC's parsing stage, but before the output stage.
+ *
+ * Return: 1 on a fatal failure, otherwise 0
+ */
+int dtc_validate(struct dt_info *dti);
+typedef int (*dtc_validate_fn_t)(struct dt_info *dti);
+```
+
+- Other definitions from "dtc-plugin.h":
+
+```
+/**
+ * struct plugin_arg - Plugin argument.
+ * @key: A non-empty string
+ * @value: NULL or a non-empty string
+ */
+struct plugin_arg {
+	char *key;
+	char *value;
+};
+
+/**
+ * struct plugin_version - Plugin API version.
+ * @major: Incompatible changes
+ * @minor: Compatible changes, such as adding a new field to the end of
+ *         a struct
+ */
+struct plugin_version {
+	int major;
+	int minor;
+};
+
+/* The API version that the plugin is being compiled against */
+#define DTC_PLUGIN_API_VERSION ((struct plugin_version){ .major = 0, .minor = 0 })
+
+/**
+ * dtc_plugin_default_version_check() - The strictest possible version
+ * check.
+ * @dtc_ver: DTC's plugin API version
+ *
+ * Return: true on success, false on failure
+ */
+static inline bool dtc_plugin_default_version_check(struct plugin_version dtc_ver)
+{
+	struct plugin_version plugin_ver = DTC_PLUGIN_API_VERSION;
+	return dtc_ver.major == plugin_ver.major && dtc_ver.minor == plugin_ver.minor;
+}
+```
+
+Please, see an example of a "dtc_init" implementation in
+"plugins/example/example.c".
+
+
 
 II - The DT block format
 ========================
plugins/example/Makefile.example new file +27 -0
new file mode 100644
index 0000000..9dc06e1
@@ -0,0 +1,27 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Makefile.example
+#
+# This is not a complete Makefile of itself.  Instead, it is designed to
+# be easily embeddable into other systems of Makefiles.
+#
+
+# Allow "make plugins" to discover the plugin
+PLUGIN_LIBS += $(PLUGIN_dir)/example/example.$(SHAREDLIB_EXT)
+
+# Add GCC flags:
+# PLUGIN_CFLAGS_example = -some-flag
+
+# Add libraries:
+# PLUGIN_LDLIBS_example = -lsome-lib
+
+# Add files to clean:
+# PLUGIN_CLEANFILES += $(PLUGIN_dir)/example/*.tmp
+
+# Override the default rules:
+# $(PLUGIN_dir)/example/example.$(SHAREDLIB_EXT): $(PLUGIN_dir)/example/example.o
+#	@$(VECHO) LD $@
+#	...
+#
+# $(PLUGIN_dir)/example/example.o: $(PLUGIN_dir)/example/example.c
+#	@$(VECHO) CC $@
+#	...
plugins/example/example.c new file +33 -0
new file mode 100644
index 0000000..4d4e7f3
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * (C) Copyright Arm Holdings.  2020
+ */
+
+#include <stdio.h>
+#include <string.h>
+
+#include "dtc-plugin.h"
+
+#define NAME "example: "
+
+int dtc_init(struct plugin_version dtc_ver, int argc, struct plugin_arg *argv)
+{
+	if (!dtc_plugin_default_version_check(dtc_ver)) {
+		fprintf(stderr, NAME"Plugin is incompatible with this version of DTC\n");
+		return 1;
+	}
+
+	for (int i = 0; i < argc; i++) {
+		if (strcmp(argv[i].key, "h") == 0
+		 || strcmp(argv[i].key, "help") == 0) {
+			printf(NAME"This is an example plugin. It prints its arguments.\n");
+			return 1;
+		} else {
+			printf(NAME"%s: %s\n", argv[i].key,
+			       argv[i].value ? argv[i].value : "");
+		}
+	}
+
+	printf(NAME"Loaded plugin 'example'\n");
+	return 0;
+}
Signature · 1 line
2.17.1
#5 lore raw

RFC PATCH v4 4/4 dtc: Add a plugin interface

In reply to a message not in the archive

From: Andrei Ziureaev <[email protected]>

Add support for building and loading plugins.

A plugin interface makes it possible to add checks in any language. It also allows these checks to print dts source line information.

---

Changes in v4:
- changed comments to kernel-doc
- mention that live tree is read-only

Changes in v3:
- plugins have to implement prototypes
- better wording of comments and messages

Changes in v2:
- describe the data model in dtc-plugin.h
- plugins must register with the build system
- the "validate_fn_t" hook can return a status
- specify that minor versions are compatible
- check if plugin_dir is NULL, just in case
- better variable names in register_plugin_info

---
 Makefile     |  29 ++++++++++-
 dtc-plugin.h |  88 +++++++++++++++++++++++++++++++
 dtc.c        | 142 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 dtc.h        |  46 +++++++++++++++++
 4 files changed, 303 insertions(+), 2 deletions(-)
 create mode 100644 dtc-plugin.h
Makefile +28 -1
index c187d5f..e96bc6e 100644
@@ -25,6 +25,8 @@ WARNINGS = -Wall -Wpointer-arith -Wcast-qual -Wnested-externs \
 	-Wstrict-prototypes -Wmissing-prototypes -Wredundant-decls -Wshadow
 CFLAGS = -g -Os $(SHAREDLIB_CFLAGS) -Werror $(WARNINGS) $(EXTRA_CFLAGS)
 
+LDLIBS_dtc += -ldl -export-dynamic
+
 BISON = bison
 LEX = flex
 SWIG = swig
@@ -66,14 +68,17 @@ ifeq ($(HOSTOS),darwin)
 SHAREDLIB_EXT     = dylib
 SHAREDLIB_CFLAGS  = -fPIC
 SHAREDLIB_LDFLAGS = -fPIC -dynamiclib -Wl,-install_name -Wl,
+PLUGIN_ldflags = -fPIC -dynamiclib
 else ifeq ($(HOSTOS),$(filter $(HOSTOS),msys cygwin))
 SHAREDLIB_EXT     = so
 SHAREDLIB_CFLAGS  =
 SHAREDLIB_LDFLAGS = -shared -Wl,--version-script=$(LIBFDT_version) -Wl,-soname,
+PLUGIN_ldflags = -shared
 else
 SHAREDLIB_EXT     = so
 SHAREDLIB_CFLAGS  = -fPIC
 SHAREDLIB_LDFLAGS = -fPIC -shared -Wl,--version-script=$(LIBFDT_version) -Wl,-soname,
+PLUGIN_ldflags = -fPIC -shared
 endif
 
 #
@@ -187,7 +192,29 @@ ifneq ($(MAKECMDGOALS),libfdt)
 endif
 endif
 
+#
+# Rules for plugins
+#
+PLUGIN_dir = plugins
+PLUGIN_CLEANFILES += $(PLUGIN_dir)/*.$(SHAREDLIB_EXT)
+PLUGIN_CLEANFILES += $(addprefix $(PLUGIN_dir)/*/,*.o *.$(SHAREDLIB_EXT))
+
+include $(wildcard $(PLUGIN_dir)/*/Makefile.*)
+
+plugins: $(PLUGIN_LIBS)
+
+$(PLUGIN_dir)/%.$(SHAREDLIB_EXT): $(PLUGIN_dir)/%.o
+	@$(VECHO) LD $@
+	$(LINK.c) $(PLUGIN_ldflags) -o $@ $< $(PLUGIN_LDLIBS_$(notdir $*))
+	ln -sf $(patsubst $(PLUGIN_dir)/%,%,$@) $(PLUGIN_dir)/$(notdir $@)
+
+$(PLUGIN_dir)/%.o: $(PLUGIN_dir)/%.c
+	@$(VECHO) CC $@
+	$(CC) $(CPPFLAGS) $(CFLAGS) $(PLUGIN_CFLAGS_$(notdir $*)) -o $@ -c $<
 
+plugins_clean:
+	@$(VECHO) CLEAN "(plugins)"
+	rm -f $(PLUGIN_CLEANFILES)
 
 #
 # Rules for libfdt
@@ -331,7 +358,7 @@ endif
 STD_CLEANFILES = *~ *.o *.$(SHAREDLIB_EXT) *.d *.a *.i *.s core a.out vgcore.* \
 	*.tab.[ch] *.lex.c *.output
 
-clean: libfdt_clean pylibfdt_clean tests_clean
+clean: libfdt_clean pylibfdt_clean tests_clean plugins_clean
 	@$(VECHO) CLEAN
 	rm -f $(STD_CLEANFILES)
 	rm -f $(VERSION_FILE)
dtc-plugin.h new file +88 -0

88 lines not on this page · raw message

dtc.c +141 -1

201 lines not on this page · raw message

dtc.h +46 -0

56 lines not on this page · raw message

Signature · 1 line
2.17.1
#6 lore raw
In reply to a message not in the archive
Quoted 5 lines · Andrei Ziureaev

From: Andrei Ziureaev <[email protected]>

I added full documentation to the live tree API and changed the style of comments in "dtc-plugin.h" to kernel-doc.

David, more comments on this series?

Quoted 76 lines

The live tree can be kept read-only. So, the input to a plugin is some information about the tree and its outputs are printed error messages and an ack/nack return back to DTC.
===========

This is one possible implementation of a plugin interface for DTC.

A plugin interface would allow us to add checks in languages other than C. Specifically, it would allow DTC and DT Schema (which is written in Python) to be integrated.

It would also allow for better debugging of dts files. For example, DT Schema would be able to print dts source line information, without us having to add source line annotations to yaml files.

Plugins are allowed read-only access to the internal unflattened device tree using the provided API. They can print error messages and return a boolean status back to DTC.

There's a question of whether we should relicense some headers to dual BSD to be able call Python code.

Any thoughts would be much appreciated.

Thanks,
Andrei.

Changes in v4:
- full documentation for live tree API
- changed comments to kernel-doc
- simplified "Exporting Functionality" in the manual
- the docs mention that the tree is read-only

Changes in v3:
- live tree API (dt.h and dt.c)
- additional functionality in dtc.h
- plugins have to implement prototypes
- improved documentation

Changes in v2:
- improved documentation
- plugins must register with the build system
- the "validate_fn_t" hook can return a status
- define "struct reserve_info" in "dtc-plugin.h"
- specify that minor versions are compatible
- check if plugin_dir is NULL, just in case
- better variable names in register_plugin_info

Andrei Ziureaev (4):
dtc: Add marker type functionality to dtc.h
dtc: Add a live tree API
dtc: Add plugin documentation and examples
dtc: Add a plugin interface

 Documentation/manual.txt         | 154 ++++++++++++++++
 Makefile                         |  29 ++-
 Makefile.dtc                     |   1 +
 dt.c                             | 158 ++++++++++++++++
 dt.h                             | 299 +++++++++++++++++++++++++++++++
 dtc-plugin.h                     |  90 ++++++++++
 dtc.c                            | 142 ++++++++++++++-
 dtc.h                            |  57 ++++++
 plugins/example/Makefile.example |  27 +++
 plugins/example/example.c        |  33 ++++
 treesource.c                     |   8 +-
 11 files changed, 992 insertions(+), 6 deletions(-)
 create mode 100644 dt.c
 create mode 100644 dt.h
 create mode 100644 dtc-plugin.h
 create mode 100644 plugins/example/Makefile.example
 create mode 100644 plugins/example/example.c

--
2.17.1