Skip to contentSkip to conversation

bootconfig: Add value override operator

PATCH In review Masami Hiramatsu 2 participants 9 messages Jul 15, 2020 – Jul 30, 2020

Subsystems (3)
lore
Share & tools
lore

Masami Hiramatsu

author maintainer
#0 lore raw

cover bootconfig: Add value override operator

Hi,

Here is a seires of value-override operator support for bootconfig.

Currently, the bootconfig syntax supports normal assignment operator ("=") and value append operator ("+="), but there is no way to override (or update) existing value.

This patch adds the value override operator (":=") to update the existing value with another value. For example,

key.word = value1
key.word := value2, value3

In this case, the 2nd "key.word" config overwrites the 1st one. Thus this is equal to;

key.word = value2, value3

Note that it is still not allowed to override a subkey with a value, and a value with a subkey. This means the following case is not allowed.

key.word = value1
key := value2 # Error!

With this change, if the bootloader wants to change some value in the default bootconfig, it doesn't need to parse the existing bootconfig, but it can just append the new configs at the tail of the bootconfig and update the footer (size, checksum and magic word).

Thank you,

---

Masami Hiramatsu (3):
lib/bootconfig: Add override operator support tools/bootconfig: Add testcases for value override operator
Documentation: bootconfig: Add bootconfig override operator

 Documentation/admin-guide/bootconfig.rst     |   11 +++++++++
 lib/bootconfig.c                             |   33 ++++++++++++++++++--------
 tools/bootconfig/samples/bad-override.bconf  |    3 ++
 tools/bootconfig/samples/bad-override2.bconf |    3 ++
 tools/bootconfig/samples/good-override.bconf |    6 +++++
 tools/bootconfig/test-bootconfig.sh          |   13 ++++++++++
 6 files changed, 59 insertions(+), 10 deletions(-)
 create mode 100644 tools/bootconfig/samples/bad-override.bconf
 create mode 100644 tools/bootconfig/samples/bad-override2.bconf
 create mode 100644 tools/bootconfig/samples/good-override.bconf

--
Masami Hiramatsu (Linaro) <[email protected]>

author maintainer
#2 lore raw

PATCH 2/3 tools/bootconfig: Add testcases for value override operator

Add some testcases and examples for value override operator.

---
 tools/bootconfig/samples/bad-override.bconf  |    3 +++
 tools/bootconfig/samples/bad-override2.bconf |    3 +++
 tools/bootconfig/samples/good-override.bconf |    6 ++++++
 tools/bootconfig/test-bootconfig.sh          |   13 +++++++++++++
 4 files changed, 25 insertions(+)
 create mode 100644 tools/bootconfig/samples/bad-override.bconf
 create mode 100644 tools/bootconfig/samples/bad-override2.bconf
 create mode 100644 tools/bootconfig/samples/good-override.bconf
tools/bootconfig/samples/bad-override.bconf new file +3 -0
new file mode 100644
index 000000000000..fde6c561512e
@@ -0,0 +1,3 @@
+key.subkey = value
+# We can not override pre-defined subkeys with value
+key := value
tools/bootconfig/samples/bad-override2.bconf new file +3 -0
new file mode 100644
index 000000000000..688587cb023c
@@ -0,0 +1,3 @@
+key = value
+# We can not override pre-defined value with subkey
+key.subkey := value
tools/bootconfig/samples/good-override.bconf new file +6 -0
new file mode 100644
index 000000000000..7d31d5f8fbd8
@@ -0,0 +1,6 @@
+# Override the value
+key.word = 1,2,4
+key.word := 2,3
+
+# No pre-defined key
+key.new.word := "new"
tools/bootconfig/test-bootconfig.sh +13 -0
index 3c2ab9e75730..56284b98d8f0 100755
@@ -117,6 +117,19 @@ xpass grep -q "bar" $OUTFILE
 xpass grep -q "baz" $OUTFILE
 xpass grep -q "qux" $OUTFILE
 
+echo "Override same-key values"
+cat > $TEMPCONF << EOF
+key = bar, baz
+key := qux
+EOF
+echo > $INITRD
+
+xpass $BOOTCONF -a $TEMPCONF $INITRD
+$BOOTCONF $INITRD > $OUTFILE
+xfail grep -q "bar" $OUTFILE
+xfail grep -q "baz" $OUTFILE
+xpass grep -q "qux" $OUTFILE
+
 echo "Double/single quotes test"
 echo "key = '\"string\"';" > $TEMPCONF
 $BOOTCONF -a $TEMPCONF $INITRD
author maintainer
#1 lore raw

PATCH 1/3 lib/bootconfig: Add override operator support

In reply to Masami Hiramatsu

Add the value override operator (":=") support to the bootconfig.

This value override operator will be useful for the bootloaders which will only update the existing bootconfig according to the bootloader boot options.

Without this override operator, the bootloader needs to parse the existing bootconfig and update it. However, with this assignment, it can just append the updated (partial) bootconfig text at the tail of existing one without parsing it. (Of course, it must update the size, checksum and magic,
but that will be done easily)

---
 lib/bootconfig.c |   33 +++++++++++++++++++++++----------
 1 file changed, 23 insertions(+), 10 deletions(-)
lib/bootconfig.c +23 -10
index 912ef4921398..a5f701161f6b 100644
@@ -329,22 +329,30 @@ const char * __init xbc_node_find_next_key_value(struct xbc_node *root,
 
 /* XBC parse and tree build */
 
+static int __init xbc_init_node(struct xbc_node *node, char *data, u32 flag)
+{
+	unsigned long offset = data - xbc_data;
+
+	if (WARN_ON(offset >= XBC_DATA_MAX))
+		return -EINVAL;
+
+	node->data = (u16)offset | flag;
+	node->child = 0;
+	node->next = 0;
+
+	return 0;
+}
+
 static struct xbc_node * __init xbc_add_node(char *data, u32 flag)
 {
 	struct xbc_node *node;
-	unsigned long offset;
 
 	if (xbc_node_num == XBC_NODE_MAX)
 		return NULL;
 
 	node = &xbc_nodes[xbc_node_num++];
-	offset = data - xbc_data;
-	node->data = (u16)offset;
-	if (WARN_ON(offset >= XBC_DATA_MAX))
+	if (xbc_init_node(node, data, flag) < 0)
 		return NULL;
-	node->data |= flag;
-	node->child = 0;
-	node->next = 0;
 
 	return node;
 }
@@ -603,7 +611,9 @@ static int __init xbc_parse_kv(char **k, char *v, int op)
 	if (c < 0)
 		return c;
 
-	if (!xbc_add_sibling(v, XBC_VALUE))
+	if (op == ':' && child) {
+		xbc_init_node(child, v, XBC_VALUE);
+	} else if (!xbc_add_sibling(v, XBC_VALUE))
 		return -ENOMEM;
 
 	if (c == ',') {	/* Array */
@@ -787,7 +797,7 @@ int __init xbc_init(char *buf, const char **emsg, int *epos)
 
 	p = buf;
 	do {
-		q = strpbrk(p, "{}=+;\n#");
+		q = strpbrk(p, "{}=+;:\n#");
 		if (!q) {
 			p = skip_spaces(p);
 			if (*p != '\0')
@@ -798,9 +808,12 @@ int __init xbc_init(char *buf, const char **emsg, int *epos)
 		c = *q;
 		*q++ = '\0';
 		switch (c) {
+		case ':':
 		case '+':
 			if (*q++ != '=') {
-				ret = xbc_parse_error("Wrong '+' operator",
+				ret = xbc_parse_error(c == '+' ?
+						"Wrong '+' operator" :
+						"Wrong ':' operator",
 							q - 2);
 				break;
 			}
author maintainer
#3 lore raw

PATCH 3/3 Documentation: bootconfig: Add bootconfig override operator

In reply to Masami Hiramatsu

Add a sentence about bootconfig override operator (":=") to bootconfig.rst.

---
 Documentation/admin-guide/bootconfig.rst |   11 +++++++++++
 1 file changed, 11 insertions(+)
Documentation/admin-guide/bootconfig.rst +11 -0
index d6b3b77a4129..a22024f9175e 100644
@@ -71,6 +71,16 @@ For example,::
  foo = bar, baz
  foo = qux  # !ERROR! we can not re-define same key
 
+If you want to update the value, you must use the override operator
+``:=`` explicitly. For example::
+
+ foo = bar, baz
+ foo := qux
+
+then, the ``qux`` is assigned to ``foo`` key. This is useful for
+overriding the default value by adding (partial) custom bootconfigs
+without parsing the default bootconfig.
+
 If you want to append the value to existing key as an array member,
 you can use ``+=`` operator. For example::
 
@@ -84,6 +94,7 @@ For example, following config is NOT allowed.::
 
  foo = value1
  foo.bar = value2 # !ERROR! subkey "bar" and value "value1" can NOT co-exist
+ foo.bar := value2 # !ERROR! even with the override operator, this is NOT allowed.
 
 
 Comments
#4 lore raw
In reply to Masami Hiramatsu
Quoted 5 lines · On Thu, 16 Jul 2020 01:00:21 +0900 Masami Hiramatsu <[email protected]> wrote:

With this change, if the bootloader wants to change some value in the default bootconfig, it doesn't need to parse the existing bootconfig, but it can just append the new configs at the tail of the bootconfig and update the footer (size, checksum and magic word).

I wonder if we should support multiple bootconfigs instead of updating the size/checksum/magic?

So the end of the initrd would have:

[data][size/checksum/magic][more-data][size/checksum/magic]

And the kernel could do the following:

1. read the end of the initrd for bootconfig
2. If found parse the bootconfig data.
3. look at the content before the bootconfig
4. if another bootconfig exists, goto 2.

-- Steve

author maintainer
#5 lore raw
Quoted 17 lines · On Wed, 15 Jul 2020 16:45:04 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 5 lines · On Thu, 16 Jul 2020 01:00:21 +0900 Masami Hiramatsu <[email protected]> wrote:

With this change, if the bootloader wants to change some value in the default bootconfig, it doesn't need to parse the existing bootconfig, but it can just append the new configs at the tail of the bootconfig and update the footer (size, checksum and magic word).

I wonder if we should support multiple bootconfigs instead of updating the size/checksum/magic?

So the end of the initrd would have:

[data][size/checksum/magic][more-data][size/checksum/magic]

Quoted 7 lines

And the kernel could do the following:

1. read the end of the initrd for bootconfig
2. If found parse the bootconfig data.
3. look at the content before the bootconfig
4. if another bootconfig exists, goto 2.

Yeah, that is possible. But since the total size of the bootconfig is limited to 32KB (this means data + 1st footer + more-data), I would like to give a chance of sanity check to the bootloader.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
#6 lore raw
Quoted 13 lines · On Thu, 16 Jul 2020 07:38:43 +0900 Masami Hiramatsu <[email protected]> wrote:
Quoted 5 lines

So the end of the initrd would have:

[data][size/checksum/magic][more-data][size/checksum/magic]

Quoted 7 lines

And the kernel could do the following:

1. read the end of the initrd for bootconfig
2. If found parse the bootconfig data.
3. look at the content before the bootconfig
4. if another bootconfig exists, goto 2.

Quoted 3 lines

Yeah, that is possible. But since the total size of the bootconfig is limited to 32KB (this means data + 1st footer + more-data), I would like to give a chance of sanity check to the bootloader.

That's a limit of the size field, right?

The bootloader (and all tools including the kernel) could check for multiple instances, and that would even increase the size of what can be added. As each section would be 32KB max size, but there's no limit to how many you have. All tools, bootconfig, the bootloader, and the kernel can perform the checksum.

-- Steve

author maintainer
#7 lore raw

Hi Steve,

Quoted 17 lines · On Wed, 15 Jul 2020 20:02:12 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 13 lines · On Thu, 16 Jul 2020 07:38:43 +0900 Masami Hiramatsu <[email protected]> wrote:
Quoted 12 lines

So the end of the initrd would have:

[data][size/checksum/magic][more-data][size/checksum/magic]

And the kernel could do the following:

1. read the end of the initrd for bootconfig
2. If found parse the bootconfig data.
3. look at the content before the bootconfig
4. if another bootconfig exists, goto 2.

Quoted 6 lines
Quoted 3 lines

Yeah, that is possible. But since the total size of the bootconfig is limited to 32KB (this means data + 1st footer + more-data), I would like to give a chance of sanity check to the bootloader.

That's a limit of the size field, right?

If you mean the size field in the footer, no, it is u32.

To be honest, the size limitation came from the xbc_node data structure. To minimize the memory footprint, I decided to pack the data structure into 64bits with 4 fields. Each field has 16bits, and the data field needs 1 bit flag to distinguish the value and the key.
Thus the maximum number of nodes can be expanded to 64K (but it is not feasible, maybe less than 8K will be a real size), but the data field (the offset from the bootconfig start address) is 15bits = 32KB long.
Of course we can use the bitfield to tune it, but maybe current balance ( 512 node / 32KB ) is enough.

Note that if we expand the number of nodes, we need to pre-allocate the node data structure as a static data (in .bss) because parsing will be done before initializing memory management. 512 nodes means 4096B is already allocated. 8K node needs 64KB, and that will be not filled in most cases.

Quoted 5 lines

The bootloader (and all tools including the kernel) could check for multiple instances, and that would even increase the size of what can be added. As each section would be 32KB max size, but there's no limit to how many you have. All tools, bootconfig, the bootloader, and the kernel can perform the checksum.

In fact, I previously considered the multi-section, but came to the conclusion that it wasn't much benefit for both Linux and the bootloaders.

If we support multi-section, we have to mix the section nodes on a single tree for overriding values, this means the data field must have a section number (and per-section starting address pointers), or an offset from the 1st section address.

And I think it is easy for the bootloader to just concatenate the data as below, because the data is already on memory.

[data][more-data][size/checksum/magic]

To support multiple-section, the bootloader will do

0. load the bootconfig with the initrd from media
1. prepare the data
2. calculate the size and checksum of the data
3. append the data and footer right after the last footer

and to support single section,

0. load the bootconfig with the initrd from media
1. prepare the data
2. calculate the size and checksum of the data
3. increment the size and the checksum
(note that the size and checksum is already on memory)
4. append the data and footer right after the last data

Thus, I think there is no much benefit to support multiple sections.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
author maintainer
#8 lore raw

Hi Steve,

Quoted 78 lines · On Thu, 16 Jul 2020 10:27:03 +0900 Masami Hiramatsu <[email protected]> wrote:

Hi Steve,

Quoted 17 lines · On Wed, 15 Jul 2020 20:02:12 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 13 lines · On Thu, 16 Jul 2020 07:38:43 +0900 Masami Hiramatsu <[email protected]> wrote:
Quoted 12 lines

So the end of the initrd would have:

[data][size/checksum/magic][more-data][size/checksum/magic]

And the kernel could do the following:

1. read the end of the initrd for bootconfig
2. If found parse the bootconfig data.
3. look at the content before the bootconfig
4. if another bootconfig exists, goto 2.

Quoted 6 lines
Quoted 3 lines

Yeah, that is possible. But since the total size of the bootconfig is limited to 32KB (this means data + 1st footer + more-data), I would like to give a chance of sanity check to the bootloader.

That's a limit of the size field, right?

If you mean the size field in the footer, no, it is u32.

To be honest, the size limitation came from the xbc_node data structure. To minimize the memory footprint, I decided to pack the data structure into 64bits with 4 fields. Each field has 16bits, and the data field needs 1 bit flag to distinguish the value and the key.
Thus the maximum number of nodes can be expanded to 64K (but it is not feasible, maybe less than 8K will be a real size), but the data field (the offset from the bootconfig start address) is 15bits = 32KB long.
Of course we can use the bitfield to tune it, but maybe current balance ( 512 node / 32KB ) is enough.

Note that if we expand the number of nodes, we need to pre-allocate the node data structure as a static data (in .bss) because parsing will be done before initializing memory management. 512 nodes means 4096B is already allocated. 8K node needs 64KB, and that will be not filled in most cases.

Quoted 5 lines

The bootloader (and all tools including the kernel) could check for multiple instances, and that would even increase the size of what can be added. As each section would be 32KB max size, but there's no limit to how many you have. All tools, bootconfig, the bootloader, and the kernel can perform the checksum.

In fact, I previously considered the multi-section, but came to the conclusion that it wasn't much benefit for both Linux and the bootloaders.

If we support multi-section, we have to mix the section nodes on a single tree for overriding values, this means the data field must have a section number (and per-section starting address pointers), or an offset from the 1st section address.

And I think it is easy for the bootloader to just concatenate the data as below, because the data is already on memory.

[data][more-data][size/checksum/magic]

To support multiple-section, the bootloader will do

0. load the bootconfig with the initrd from media
1. prepare the data
2. calculate the size and checksum of the data
3. append the data and footer right after the last footer

and to support single section,

Quoted 8 lines

0. load the bootconfig with the initrd from media
1. prepare the data
2. calculate the size and checksum of the data
3. increment the size and the checksum
(note that the size and checksum is already on memory)
4. append the data and footer right after the last data

Thus, I think there is no much benefit to support multiple sections.

What would you think?

I guess if we have other types of data appended to the initrd as similar to the bootconfig, I think we should add the multiple section support. But only for the bootconfig, we can just update the bootconfig data as I suggested, since it keeps the code simple. It might be a chiken-egg problem...

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>