Skip to contentSkip to conversation

cli: Add a new shell

RFC PATCH In review Sean Anderson 10 participants 107 messages Jul 1, 2021 – Jul 8, 2021 Library code

lore
Share & tools
lore
#0 lore raw

cover cli: Add a new shell

Well, this has been sitting on my hard drive for too long without feedback ("Release early, release often"), so here's the first RFC. This is not ready to merge (see the "Future work" section below), but the shell is functional and at least partially tested.

The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes over on sandbox.

add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)

= Getting started

Enable CONFIG_LIL. If you would like to run tests, enable CONFIG_LIL_FULL. Note that dm_test_acpi_cmd_dump and setexpr_test_str_oper will fail. CONFIG_LIL_POOLS is currently broken (with what appears to be a double free).

For an overview of the language as a whole, refer to the original readme [1].

[1] http://runtimeterror.com/tech/lil/readme.txt

== Key patches

The following patches are particularly significant for reviewing and understanding this series:

cli: Add LIL shell
This contains the LIL shell as originally written by Kostas with some major deletions and some minor additions.
cli: lil: Wire up LIL to the rest of U-Boot
This allows you to use LIL as a shell just like Hush.
cli: lil: Document structures
This adds documentation for the major structures of LIL. It is a good place to start looking at the internals.
test: Add tests for LIL
This adds some basic integration tests and provides some examples of LIL code.
cli: lil: Add a distinct parsing step
This adds a parser separate from the interpreter. This patch is the largest original work in this series.
cli: lil: Load procs from the environment
This allows procedures to be saved and loaded like variables.

= A new shell

This series adds a new shell for U-Boot. The aim is to eventually replace Hush as the primary shell for all boards which currently use it. Hush should be replaced because it has several major problems:

- It has not had a major update in two decades, resulting in duplication of effort in finding bugs. Regarding a bug in variable setting, Wolfgang remarks

So the specific problem has (long) been fixed in upstream, and instead of adding a patch to our old version, thus cementing the broken behaviour, we should upgrade hush to recent upstream code.

-- Wolfgang Denk [2]

These lack of updates are further compounded by a significant amount of ifdef-ing in the Hush code. This makes the shell hard to read and debug. Further, the original purpose of such ifdef-ing (upgrading to a newer Hush) has never happened.

- It was designed for a preempting OS which supports pipes and processes. This fundamentally does not match the computing model of U-Boot where there is exactly one thread (and every other CPU is spinning or sleeping). Working around these design differences is a significant cause of the aformentioned ifdef-ing.

- It lacks many major features expected of even the most basic shells, such as functions and command substitution ($() syntax). This makes it difficult to script with Hush. While it is desirable to write some code in C, much code *must* be written in C because there is no way to express the logic in Hush.

I believe that U-Boot should have a shell which is more featureful, has cleaner code, and which is the same size as Hush (or less). The ergonomic advantages afforded by a new shell will make U-Boot easier to use and customize.

[2] https://lore.kernel.org/u-boot/[email protected]/

= Open questions

While the primary purpose of this series is of course to get feedback on the code I have already written, there are several decisions where I am not sure what the best course of action is.

- What should be done about 'expr'? The 'expr' command is a significant portion of the final code size. It cannot be removed outright, because it is used by several builtin functions like 'if', 'while', 'for', etc. The way I see it, there are two general approaches to take

- Rewrite expr to parse expressions and then evaluate them. The parsing could re-use several of the existing parse functions like how parse_list does. This could reduce code, as instead of many functions each with their own while/switch statements, we could have two while/switch statements (one to parse, and one to evaluate). However, this may end up increasing code size (such as when the main language had evaluation split from parsing).

- Don't parse infix expressions, and just make arithmetic operators normal functions. This would affect ergonomics a bit. For example, instead of

if {$i < 10} { ... }

one would need to write

if {< $i 10} { ... }

and instead of

if {$some_bool} { ... }

one would need to write

if {quote $some_bool} { ... }

Though, given how much setexpr is used (not much), this may not be such a big price to pay. This route is almost certain to reduce code size.

- How should LIL functions integrate with the rest of U-Boot? At the moment, lil functions and procedures exist in a completely separate world from normal commands. I would like to integrate them more closely, but I am not sure the best way to go about this. At the very minimum, each LIL builtin function needs to get its hands on the LIL interpreter somehow. I'd rather this didn't happen through gd_t or similar so that it is easier to unit test. Additionally, LIL functions expect an array of lil_values instead of strings. We could strip them out, but I worry that might start to impact performance (from all the copying).

The other half of this is adding LIL features into regular commands. The most important feature here is being able to return a string result. I took an initial crack at it [3], but I think with this series there is a stronger motivating factor (along with things like [4]).

[3] https://patchwork.ozlabs.org/project/uboot/list/?series=231377
[4] https://patchwork.ozlabs.org/project/uboot/list/?series=251013

= Future work

The series as presented today is incomplete. The following are the major issues I see with it at the moment. I would like to address all of these issues, but some of them might be postponed until after first merging this series.

- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly, there still remains a significant amount of original code which just ignores errors. In particular, I would like to ensure that the following categories of error conditions are handled:

- Running out of memory.
- Access to a nonexistant variable.
- Passing the wrong number of arguments to a function.
- Interpreting a value as the wrong type (e.g. "foo" should not have a numeric representation, instead of just being treated as 1).

- There are many deviations from TCL with no purpose. For example, the list indexing function is named "index" and not "lindex". It is perfectly fine to drop features or change semantics to reduce code size, make parsing easier, or make execution easier. But changing things for the sake of it should be avoided.

- The test suite is rather anemic compared with the amount of code this series introduces. I would like to expand it significantly. In particular, error conditions are not well tested (only the "happy path" is tested).

- While I have documented all new functions I have written, there are many existing functions which remain to be documented. In addition, there is no user documentation, which is critical in driving adoption of any new programming language. Some of this cover letter might be integrated with any documentation written.

- Some shell features such as command repetition and secondary shell prompts have not been implemented.

- Arguments to native lil functions are incompatible with U-Boot functions. For example, the command

foo bar baz

would be passed to a U-Boot command as

	{ "foo", "bar", "baz", NULL }

but would be passed to a LIL function as

	{ "bar", "baz" }

This makes it more difficult to use the same function to parse several different commands. At the moment this is solved by passing the command name in lil->env->proc, but I would like to switch to the U-Boot argument list style.

- Several existing tests break when using LIL because they expect no output on failure, but LIL produces some output notifying the user of the failure.

- Implement DISTRO_BOOT in LIL. I think this is an important proof-of-concept to show what can be done with LIL, and to determine which features should be moved to LIL_FULL.

= Why Lil?

When looking for a suitable replacement shell, I evaluated implementations using the following criteria:

- It must have a GPLv2-compatible license.
- It must be written in C, and have no major external dependencies.
- It must support bare function calls. That is, a script such as 'foo bar' should invoke the function 'foo' with the argument 'bar'. This preserves the shell-like syntax we expect.
- It must be small. The eventual target is that it compiles to around 10KiB with -Os and -ffunction-sections.
- There should be good tests. Any tests at all are good, but a functioning suite is better.
- There should be good documentation
- There should be comments in the source.
- It should be "finished" or have only slow development. This will hopefully make it easier to port changes.

Notably absent from the above list is performance. Most scripts in U-Boot will be run once on boot. As long as the time spent evaluating scripts is kept under a reasonable threshold (a fraction of the time spend initializing hardware or reading data from persistant storage), there is no need to optimize for speed.

In addition, I did not consider updating Hush from Busybox. The mismatch in computing environment expectations (as noted in the "New shell" section above) still applies. IMO, this mismatch is the biggest reason that things like functions and command substitution have been excluded from the U-Boot's Hush.

== lil

- zLib
- TCL
- Compiles to around 10k with no builtins. To 25k with builtins.
- Some tests, but not organized into a suite with expected output. Some evidence that the author ran APL, but no harness.
- Some architectural documentation. Some for each functions, but not much.
- No comments :l
- 3.5k LoC

== picol

- 2-clause BSD
- TCL
- Compiles to around 25k with no builtins. To 80k with builtins.
- Tests with suite (in-language). No evidence of fuzzing.
- No documentation :l
- No comments :l
- 5k LoC

== jimtcl

- 2-clause BSD
- TCL
- Compiles to around 95k with no builtins. To 140k with builtins. Too big...

== boron

- LGPLv3+ (so this is right out)
- REBOL
- Compiles to around 125k with no builtins. To 190k with builtins. Too big...

== libmawk

- GPLv2
- Awk
- Compiles to around 225k. Too big...

== libfawk

- 3-clause BSD
- Uses bison+yacc...
- Awk; As it turns out, this has parentheses for function calls.
- Compiles to around 24-30k. Not sure how to remove builtins.
- Test suite (in-language). No fuzzing.
- Tutorial book. No function reference.
- No comments
- Around 2-4k LoC

== MicroPython

- MIT
- Python (but included for completeness)
- Compiles to around 300k. Too big...

== mruby/c

- 3-clause BSD
- Ruby
- Compiles to around 85k without builtins and 120k with. Too big...

== eLua

- MIT
- Lua
- Build system is a royal pain (custom and written in Lua with external deps)
- Base binary is around 250KiB and I don't want to deal with reducing it

So the interesting/viable ones are
- lil
- picol
- libfawk (maybe)

I started with LIL because it was the smallest. I have found several issues with LIL along the way. Some of these are addressed in this series already, while others remain unaddressed (see the section "Future Work").

Sean Anderson (28):
Add Zlib License
cli: Add LIL shell
cli: lil: Replace strclone with strdup
cli: lil: Remove most functions by default
cli: lil: Rename some functions to be more like TCL
cli: lil: Convert some defines to enums
cli: lil: Simplify callbacks
cli: lil: Handle commands with dots
cli: lil: Use error codes
cli: lil: Add printf-style format helper for errors
cli: lil: Add several helper functions for errors
cli: lil: Check for ctrl-c
cli: lil: Wire up LIL to the rest of U-Boot
cli: lil: Document structures
cli: lil: Convert LIL_ENABLE_POOLS to Kconfig
cli: lil: Convert LIL_ENABLE_RECLIMIT to KConfig
test: Add tests for LIL
cli: lil: Remove duplicate function bodies
cli: lil: Add "symbol" structure
cli: lil: Add config to enable debug output
cli: lil: Add a distinct parsing step
env: Add a priv pointer to hwalk_r
cli: lil: Handle OOM for hm_put
cli: lil: Make proc always take 3 arguments
cli: lil: Always quote items in lil_list_to_value
cli: lil: Allocate len even when str is NULL in alloc_value_len
cli: lil: Add a function to quote values
cli: lil: Load procs from the environment

 Licenses/README   |    1 +
 Licenses/zlib.txt |   14 +
 MAINTAINERS       |    7 +
 cmd/Kconfig       |   52 +-
 cmd/nvedit.c      |    8 +-
 common/Makefile   |    4 +
 common/cli.c      |  118 +-
 common/cli_lil.c  | 4321 +++++++++++++++++++++++++++++++++++++++++++++
 env/callback.c    |    4 +-
 env/flags.c       |    4 +-
 include/cli_lil.h |  202 +++
 include/search.h  |    2 +-
 lib/hashtable.c   |    5 +-
 test/cmd/Makefile |    1 +
 test/cmd/lil.c    |  306 ++++
 15 files changed, 5021 insertions(+), 28 deletions(-)
 create mode 100644 Licenses/zlib.txt
 create mode 100644 common/cli_lil.c
 create mode 100644 include/cli_lil.h
 create mode 100644 test/cmd/lil.c
Signature · 1 line
2.32.0
#1 lore raw

RFC PATCH 1/28 Add Zlib License

This adds the Zlib License which is compatible with GPLv2 as long as its terms are met. Files originally licensed as Zlib should use the "GPL-2.0+ AND Zlib" SPDX identifier.

---

 Licenses/README   |  1 +
 Licenses/zlib.txt | 14 ++++++++++++++
 2 files changed, 15 insertions(+)
 create mode 100644 Licenses/zlib.txt
Licenses/README +1 -0
index d09f0fe2cb..d5af353520 100644
@@ -152,4 +152,5 @@ IBM PIBS (PowerPC Initialization and		IBM-pibs			ibm-pibs.txt
 ISC License					ISC		Y		isc.txt			https://spdx.org/licenses/ISC
 MIT License					MIT		Y		mit.txt			https://spdx.org/licenses/MIT.html
 SIL OPEN FONT LICENSE (OFL-1.1)			OFL-1.1		Y		OFL.txt			https://spdx.org/licenses/OFL-1.1.html
+zlib License					Zlib		Y		zlib.txt		https://spdx.org/licenses/Zlib.html
 X11 License					X11				x11.txt			https://spdx.org/licenses/X11.html
Licenses/zlib.txt new file +14 -0
new file mode 100644
index 0000000000..7ea8dde871
@@ -0,0 +1,14 @@
+This software is provided 'as-is', without any express or implied warranty. In
+no event will the authors be held liable for any damages arising from the use
+of this software.
+
+Permission is granted to anyone to use this software for any purpose, including
+commercial applications, and to alter it and redistribute it freely, subject
+to the following restrictions:
+  1. The origin of this software must not be misrepresented; you must not
+claim that you wrote the original software. If you use this software in a
+product, an acknowledgment in the product documentation would be appreciated
+but is not required.
+  2. Altered source versions must be plainly marked as such, and must not be
+misrepresented as being the original software.
+  3. This notice may not be removed or altered from any source distribution.
Signature · 1 line
2.32.0
Reviewed-by on v1 1/28
#2 lore raw

Re: [RFC PATCH 01/28] Add Zlib License

Quoted 5 lines · Sean Anderson

This adds the Zlib License which is compatible with GPLv2 as long as its terms are met. Files originally licensed as Zlib should use the "GPL-2.0+ AND Zlib" SPDX identifier.

Quoted 7 lines

---

 Licenses/README   |  1 +
 Licenses/zlib.txt | 14 ++++++++++++++
 2 files changed, 15 insertions(+)
 create mode 100644 Licenses/zlib.txt
#3 lore raw

RFC PATCH 2/28 cli: Add LIL shell

In reply to Sean Anderson

This is the LIL programming language [1] as originally written by Kostas Michalopoulos <[email protected]>. LIL is a stripped-down TCL variant. Many syntax features are very similar to shell:

=> echo Hello from $stdout
Hello from serial,vidconsole

LIL supports functions, scoped variables, and command substitution. For a short example, running the following script

	func fact {n} {
		if {$n} {
			expr {$n * [fact [expr {$n - 1}]]}
		} {
			return 1
		}
	}
	echo [fact 10]
	echo [fact 20]

would result in the output

	fact
	3628800
	2432902008176640000

For more examples, please refer to the "test: Add tests for LIL" later in this series.

I have made several mostly-mechanical modifications to the source before importing it to reduce patches to functional changes. If you are interested in what has been changed before this commit, refer to my lil git repository [2]. In addition, some functions and features have been removed from this patch as I have better determined what is necessary for U-Boot and what is not. If there is sufficient interest, I can add those changes to the repository as well.

[1] http://runtimeterror.com/tech/lil/
[2] https://github.com/Forty-Bot/lil/commits/master

---

 MAINTAINERS       |    6 +
 cmd/Kconfig       |   11 +-
 common/Makefile   |    1 +
 common/cli_lil.c  | 2991 +++++++++++++++++++++++++++++++++++++++++++++
 include/cli_lil.h |  111 ++
 5 files changed, 3119 insertions(+), 1 deletion(-)
 create mode 100644 common/cli_lil.c
 create mode 100644 include/cli_lil.h
MAINTAINERS +6 -0
index 2d267aeff2..0184de5f93 100644
@@ -766,6 +766,12 @@ S:	Maintained
 T:	git https://source.denx.de/u-boot/custodians/u-boot-i2c.git
 F:	drivers/i2c/
 
+LIL SHELL
+M:	Sean Anderson <[email protected]>
+S:	Maintained
+F:	common/cli_lil.c
+F:	include/cli_lil.h
+
 LOGGING
 M:	Simon Glass <[email protected]>
 S:	Maintained
cmd/Kconfig +10 -1
index 9e8b69258f..8bccc572af 100644
@@ -23,6 +23,15 @@ config HUSH_PARSER
 	  If disabled, you get the old, much simpler behaviour with a somewhat
 	  smaller memory footprint.
 
+config LIL
+	bool "Use LIL shell"
+	depends on CMDLINE
+	help
+	  This options enables the "Little Interpreted Language" (LIL) shell as
+	  command line interpreter, thus enabling powerful command line syntax
+	  like `proc name {args} {body}' functions or `echo [some command]`
+	  command substitution ("tcl scripts").
+
 config CMDLINE_EDITING
 	bool "Enable command line editing"
 	depends on CMDLINE
@@ -1379,7 +1388,7 @@ config CMD_ECHO
 
 config CMD_ITEST
 	bool "itest"
-	default y
+	default y if !LIL
 	help
 	  Return true/false on integer compare.
 
common/Makefile +1 -0
index 829ea5fb42..dce04b305e 100644
@@ -10,6 +10,7 @@ obj-y += main.o
 obj-y += exports.o
 obj-$(CONFIG_HASH) += hash.o
 obj-$(CONFIG_HUSH_PARSER) += cli_hush.o
+obj-$(CONFIG_LIL) += cli_lil.o
 obj-$(CONFIG_AUTOBOOT) += autoboot.o
 
 # This option is not just y/n - it can have a numeric value
common/cli_lil.c new file +2991 -0

2,958 lines not on this page · raw message

include/cli_lil.h new file +111 -0

111 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.32.0

Part of the diff is not shown here. View the raw message.

#4 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Dear Sean,

Quoted 3 lines · In message <[email protected]> you wrote:

This is the LIL programming language [1] as originally written by Kostas Michalopoulos <[email protected]>. LIL is a stripped-down TCL variant. Many syntax features are very similar to shell:

Do you have a list of the exact differencec between LIL and a standard shell?

I wonder, if we deviate from standard shell syntax anyway, we could also have a look at lua, for example?

Best regards,

Wolfgang Denk

Signature · 6 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
One essential to success is that you desire be an all-obsessing  one,
your thoughts and aims be co-ordinated, and your energy be concentra-
ted and applied without letup.
#5 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 9 lines · Wolfgang Denk

Dear Sean,

Quoted 3 lines · In message <[email protected]> you wrote:

This is the LIL programming language [1] as originally written by Kostas Michalopoulos <[email protected]>. LIL is a stripped-down TCL variant. Many syntax features are very similar to shell:

Do you have a list of the exact differencec between LIL and a standard shell?

For a partial list, see

[1] https://github.com/Forty-Bot/lil/commits/master

Quoted 2 lines

I wonder, if we deviate from standard shell syntax anyway, we could also have a look at lua, for example?

I also looked at lua (see the cover letter), but I rejected it based on size constraints (eLua was around the size of U-Boot itself).

Because of how often the shell is used to debug things, I wanted the candidate I picked to have similar syntax to the existing shell. For example,

	load mmc ${mmcdev}:${mmcpart} $loadaddr $image

is a valid command in both Hush and LIL. Compare with lua, which might express the above as

	load("mmc", mmcdev .. ":" .. mmcpart, loadaddr, image)

which I think is a much larger deviation from existing syntax.

--Sean

#6 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 8 lines · Sean Anderson
Quoted 7 lines · Wolfgang Denk

Dear Sean,

Quoted 3 lines · In message <[email protected]> you wrote:

This is the LIL programming language [1] as originally written by Kostas Michalopoulos <[email protected]>. LIL is a stripped-down TCL variant. Many syntax features are very similar to shell:

Quoted 6 lines
Quoted 2 lines

Do you have a list of the exact differencec between LIL and a standard shell?

For a partial list, see

[1] https://github.com/Forty-Bot/lil/commits/master

Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:

sh tcl

foo=bar set foo bar
echo $foo echo $foo

if [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }

foo() {				proc foo {first second} {
	echo $1 $2			echo $first $second
}				}

for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }

fact() {
	if [ $1 -eq 0 ]; then
		echo 1
	else
		echo $(($1 * $(fact $(($1 - 1)))))
	fi
}
				proc fact {n} {
					if {$n} {
						expr {$n * [fact [expr {$n - 1}]]}
					} {
						return 1
					}
				}

Hopefully this gives you a bit of a feel for the basic differences.

--Sean

Quoted 21 lines
Quoted 3 lines

I wonder, if we deviate from standard shell syntax anyway, we could also have a look at lua, for example?

I also looked at lua (see the cover letter), but I rejected it based on size constraints (eLua was around the size of U-Boot itself).

Because of how often the shell is used to debug things, I wanted the candidate I picked to have similar syntax to the existing shell. For example,

     load mmc ${mmcdev}:${mmcpart} $loadaddr $image

is a valid command in both Hush and LIL. Compare with lua, which might express the above as

     load("mmc", mmcdev .. ":" .. mmcpart, loadaddr, image)

which I think is a much larger deviation from existing syntax.

--Sean

#7 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Dear Sean,

Quoted 44 lines · In message <[email protected]> you wrote:
Quoted 4 lines

For a partial list, see

[1] https://github.com/Forty-Bot/lil/commits/master

Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:

sh tcl

foo=bar set foo bar
echo $foo echo $foo

if [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }

foo() {				proc foo {first second} {
	echo $1 $2			echo $first $second
}				}

for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }

fact() {
	if [ $1 -eq 0 ]; then
		echo 1
	else
		echo $(($1 * $(fact $(($1 - 1)))))
	fi
}
				proc fact {n} {
					if {$n} {
						expr {$n * [fact [expr {$n - 1}]]}
					} {
						return 1
					}
				}
Quoted 1 line

Hopefully this gives you a bit of a feel for the basic differences.

Well, I know TCL, and there has been a zillion of reasons to move _from_ that language many, many years ago. Intoducing this as new "shell" language in U-Boot does not look attractive to me. Actually, it gives me the creeps.

Best regards,

Wolfgang Denk

Signature · 4 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
Drawing on my fine command of language, I said nothing.
Reviewed-by
#8 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Hi,

Quoted 48 lines · Wolfgang Denk

Dear Sean,

Quoted 44 lines · In message <[email protected]> you wrote:
Quoted 4 lines

For a partial list, see

[1] https://github.com/Forty-Bot/lil/commits/master

Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:

sh tcl

foo=bar set foo bar
echo $foo echo $foo

if [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }

foo() { proc foo {first second} {
echo $1 $2 echo $first $second
} }

for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }

fact() {
      if [ $1 -eq 0 ]; then
              echo 1
      else
              echo $(($1 * $(fact $(($1 - 1)))))
      fi
}
                              proc fact {n} {
                                      if {$n} {
                                              expr {$n * [fact [expr {$n - 1}]]}
                                      } {
                                              return 1
                                      }
                              }
Quoted 6 lines
Quoted 1 line

Hopefully this gives you a bit of a feel for the basic differences.

Well, I know TCL, and there has been a zillion of reasons to move _from_ that language many, many years ago. Intoducing this as new "shell" language in U-Boot does not look attractive to me. Actually, it gives me the creeps.

I was no fan of TCL either but this does seem like a reasonable idea to me.

Obviously whoever wrote it was very concerned about compiler performance as there seem to be no comments?

Regards,
Simon

#9 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

In reply to Wolfgang Denk
Quoted 48 lines · Wolfgang Denk

Dear Sean,

Quoted 45 lines · In message <[email protected]> you wrote:
Quoted 4 lines

For a partial list, see

[1] https://github.com/Forty-Bot/lil/commits/master

Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:

sh tcl

foo=bar set foo bar
echo $foo echo $foo

if [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }

foo() {				proc foo {first second} {
	echo $1 $2			echo $first $second
}				}

for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }

fact() {
	if [ $1 -eq 0 ]; then
		echo 1
	else
		echo $(($1 * $(fact $(($1 - 1)))))
	fi
}
				proc fact {n} {
					if {$n} {
						expr {$n * [fact [expr {$n - 1}]]}
					} {
						return 1
					}
				}

Hopefully this gives you a bit of a feel for the basic differences.

Which of these things, from each column, can you do in the context of U-Boot? That's important too.

Quoted 4 lines

Well, I know TCL, and there has been a zillion of reasons to move _from_ that language many, many years ago. Intoducing this as new "shell" language in U-Boot does not look attractive to me. Actually, it gives me the creeps.

This is I think the hard question. A draw of the current shell is that it it looks and acts like bash/sh/etc, for at least basic operations. That's something that's comfortable to a large audience. That has disadvantages when people want to start doing something complex. Sean has shown that several times and he's not the only one. LIL being tcl'ish is not.

Something that has "sh" syntax but also clear to the user errors when trying to do something not supported would also be interesting to see. It seems like a lot of the frustration from users with our shell is that it's not clear where the line between "this is an sh-like shell" and "no, not like that" is.

Signature · 1 line
Tom
#10 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 15 lines · Tom Rini
Quoted 14 lines · Wolfgang Denk

Dear Sean,

Quoted 11 lines · In message <[email protected]> you wrote:
Quoted 4 lines

For a partial list, see

[1] https://github.com/Forty-Bot/lil/commits/master

Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:

sh tcl

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

foo=bar set foo bar
echo $foo echo $foo

if [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }

The left side is possible with something like

if itest 1 -gt 2; then # etc.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo() {				proc foo {first second} {
	echo $1 $2			echo $first $second
}				}

This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }

This is possible only if you already have a list of files. For example, one could do

part list mmc 0 -bootable parts
for p in $parts; do #etc

but the part command is one of the only ones which produces output in the correct format. If you want to (e.g.) dynamically construct a list you will have a much harder time.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
fact() {
	if [ $1 -eq 0 ]; then
		echo 1
	else
		echo $(($1 * $(fact $(($1 - 1)))))
	fi
}

This is technically possible with run and setexpr, but fairly cumbersome to do.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
				proc fact {n} {
					if {$n} {
						expr {$n * [fact [expr {$n - 1}]]}
					} {
						return 1
					}
				}
Quoted 4 lines
Quoted 1 line
Quoted 1 line

Hopefully this gives you a bit of a feel for the basic differences.

Which of these things, from each column, can you do in the context of U-Boot? That's important too.

See above.

--Sean

#11 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 73 lines · Sean Anderson
Quoted 15 lines · Tom Rini
Quoted 14 lines · Wolfgang Denk

Dear Sean,

Quoted 11 lines · In message <[email protected]> you wrote:
Quoted 4 lines

For a partial list, see

[1] https://github.com/Forty-Bot/lil/commits/master

Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:

sh tcl

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

foo=bar set foo bar
echo $foo echo $foo

if [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }

The left side is possible with something like

if itest 1 -gt 2; then # etc.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo() {				proc foo {first second} {
	echo $1 $2			echo $first $second
}				}

This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }

This is possible only if you already have a list of files. For example, one could do

part list mmc 0 -bootable parts
for p in $parts; do #etc

but the part command is one of the only ones which produces output in the correct format. If you want to (e.g.) dynamically construct a list you will have a much harder time.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
fact() {
	if [ $1 -eq 0 ]; then
		echo 1
	else
		echo $(($1 * $(fact $(($1 - 1)))))
	fi
}

This is technically possible with run and setexpr, but fairly cumbersome to do.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
				proc fact {n} {
					if {$n} {
						expr {$n * [fact [expr {$n - 1}]]}
					} {
						return 1
					}
				}
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

Hopefully this gives you a bit of a feel for the basic differences.

Which of these things, from each column, can you do in the context of U-Boot? That's important too.

See above.

And for clarity, on the LIL side, with a few things like needing to bring in the list of files somehow, all of those would work in U-Boot?

Signature · 1 line
Tom
#12 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 76 lines · Tom Rini
Quoted 75 lines · Sean Anderson
Quoted 15 lines · Tom Rini
Quoted 14 lines · Wolfgang Denk

Dear Sean,

Quoted 11 lines · In message <[email protected]> you wrote:
Quoted 4 lines

For a partial list, see

[1] https://github.com/Forty-Bot/lil/commits/master

Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:

sh tcl

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

foo=bar set foo bar
echo $foo echo $foo

if [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }

The left side is possible with something like

if itest 1 -gt 2; then # etc.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo() {				proc foo {first second} {
	echo $1 $2			echo $first $second
}				}

This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }

This is possible only if you already have a list of files. For example, one could do

part list mmc 0 -bootable parts
for p in $parts; do #etc

but the part command is one of the only ones which produces output in the correct format. If you want to (e.g.) dynamically construct a list you will have a much harder time.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
fact() {
	if [ $1 -eq 0 ]; then
		echo 1
	else
		echo $(($1 * $(fact $(($1 - 1)))))
	fi
}

This is technically possible with run and setexpr, but fairly cumbersome to do.

Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
				proc fact {n} {
					if {$n} {
						expr {$n * [fact [expr {$n - 1}]]}
					} {
						return 1
					}
				}

Hopefully this gives you a bit of a feel for the basic differences.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Which of these things, from each column, can you do in the context of U-Boot? That's important too.

See above.

And for clarity, on the LIL side, with a few things like needing to bring in the list of files somehow, all of those would work in U-Boot?

Correct. The only unimplemented function is "glob".

--Sean

#13 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

In reply to Sean Anderson

Dear Sean,

Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo() {				proc foo {first second} {
	echo $1 $2			echo $first $second
}				}

This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.

Again: this is not an issue with hush as is, but only with our resource-limited port of a nearly 20 year old version of it.

Updating to a current version would fix this, in an almost 100% backward compatible way.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
If you don't have time to do it right, when will you have time to  do
it over?                                                - John Wooden
#14 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 13 lines · Wolfgang Denk

Dear Sean,

Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

foo() { proc foo {first second} {
echo $1 $2 echo $first $second
} }

This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.

Quoted 5 lines

Again: this is not an issue with hush as is, but only with our resource-limited port of a nearly 20 year old version of it.

Updating to a current version would fix this, in an almost 100% backward compatible way.

Agree on this. There is another email about the LIL and how is right now maintained and if the solutions that we have already in place can be just updated, I don't find any valid reason to change it but update seems more straightforward.

Michael

Quoted 11 lines

Best regards,

Wolfgang Denk

--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] If you don't have time to do it right, when will you have time to do
it over? - John Wooden

Signature · 11 lines
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
www.amarulasolutions.com
#15 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Hi Michael,

Quoted 18 lines · Michael Nazzareno Trimarchi
Quoted 16 lines · Wolfgang Denk

Dear Sean,

Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

foo() { proc foo {first second} {
echo $1 $2 echo $first $second
} }

This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.

Again: this is not an issue with hush as is, but only with our resource-limited port of a nearly 20 year old version of it.

Quoted 7 lines
Quoted 2 lines

Updating to a current version would fix this, in an almost 100% backward compatible way.

Agree on this. There is another email about the LIL and how is right now maintained and if the solutions that we have already in place can be just updated, I don't find any valid reason to change it but update seems more straightforward.

Would you be interested in taking this on? It has been talked about for some years...

Regards,
Simon

#16 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 25 lines · Simon Glass

Hi Michael,

Quoted 21 lines · Michael Nazzareno Trimarchi
Quoted 18 lines · Wolfgang Denk

Dear Sean,

Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

foo() { proc foo {first second} {
echo $1 $2 echo $first $second
} }

This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.

Again: this is not an issue with hush as is, but only with our resource-limited port of a nearly 20 year old version of it.

Updating to a current version would fix this, in an almost 100% backward compatible way.

Quoted 7 lines
Quoted 4 lines

Agree on this. There is another email about the LIL and how is right now maintained and if the solutions that we have already in place can be just updated, I don't find any valid reason to change it but update seems more straightforward.

Would you be interested in taking this on? It has been talked about for some years...

To repeat myself, someone posting the patches to update (or more likely, replace as a new Kconfig option and I'd be fine with a "pick old or new" as a starting point) is the main gating factor on using a new version of hush.

Signature · 1 line
Tom
#17 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

In reply to Simon Glass

Hi

Quoted 31 lines · Simon Glass

Hi Michael,

Quoted 18 lines · Michael Nazzareno Trimarchi
Quoted 16 lines · Wolfgang Denk

Dear Sean,

Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

foo() { proc foo {first second} {
echo $1 $2 echo $first $second
} }

This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.

Again: this is not an issue with hush as is, but only with our resource-limited port of a nearly 20 year old version of it.

Quoted 7 lines
Quoted 2 lines

Updating to a current version would fix this, in an almost 100% backward compatible way.

Agree on this. There is another email about the LIL and how is right now maintained and if the solutions that we have already in place can be just updated, I don't find any valid reason to change it but update seems more straightforward.

Quoted 5 lines

Would you be interested in taking this on? It has been talked about for some years...

Regards,
Simon

I will ask to someone in the company to do as task. Francis can you do it?

Michael

Signature · 11 lines
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
www.amarulasolutions.com
#18 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

In reply to Wolfgang Denk
Quoted 12 lines · Wolfgang Denk

Dear Sean,

Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo() {				proc foo {first second} {
	echo $1 $2			echo $first $second
}				}

This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.

Quoted 5 lines

Again: this is not an issue with hush as is, but only with our resource-limited port of a nearly 20 year old version of it.

Updating to a current version would fix this, in an almost 100% backward compatible way.

Let us cut to the chase then. Who is going to port a modern version of hush over to U-Boot, and maintain it? If we fork and forget again, we'll be in a bad place once again in 2-3 years.

Signature · 1 line
Tom
#19 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Dear Tom,

Quoted 7 lines · In message <20210706145420.GQ9516@bill-the-cat> you wrote:
Quoted 3 lines

Updating to a current version would fix this, in an almost 100% backward compatible way.

Let us cut to the chase then. Who is going to port a modern version of hush over to U-Boot, and maintain it? If we fork and forget again, we'll be in a bad place once again in 2-3 years.

Would we really be better off if we switch to some exotic piece of code instead (I was not able to locate any user base, nor other developers), which has been reported to have poor or no error handling, and comes with an incompatible command line interface?

There is a zillion of shell scripts in the field, from non-trivial boot sequences to complex download-and-upgrade scripts. You can't really even think of breaking compatibility on such a level.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
There are no data that cannot be plotted on a straight  line  if  the
axis are chosen correctly.
#20 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 11 lines · Wolfgang Denk

Dear Tom,

Quoted 7 lines · In message <20210706145420.GQ9516@bill-the-cat> you wrote:
Quoted 3 lines

Updating to a current version would fix this, in an almost 100% backward compatible way.

Let us cut to the chase then. Who is going to port a modern version of hush over to U-Boot, and maintain it? If we fork and forget again, we'll be in a bad place once again in 2-3 years.

Quoted 8 lines

Would we really be better off if we switch to some exotic piece of code instead (I was not able to locate any user base, nor other developers), which has been reported to have poor or no error handling, and comes with an incompatible command line interface?

There is a zillion of shell scripts in the field, from non-trivial boot sequences to complex download-and-upgrade scripts. You can't really even think of breaking compatibility on such a level.

As I've said a few times in this thread, this not being an sh-style interpreter is a strike against it. And if we're going to insist on a bug-for-bug upgrade to our hush (so that all of the hugely complex existing scripts keep working) we might as well not upgrade. Frankly I suspect that down the line IF a new cli interpreter comes in to U-Boot we will have to keep the old one around as a "use this instead" option for another long number of years, so that if there are any systems with non-trivial scripts but upgrade U-Boot and don't / won't / can't re-validate their entire sequence, they can just use the old cli.

Signature · 1 line
Tom
#21 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Dear Tom,

Quoted 10 lines · In message <20210707135839.GW9516@bill-the-cat> you wrote:

As I've said a few times in this thread, this not being an sh-style interpreter is a strike against it. And if we're going to insist on a bug-for-bug upgrade to our hush (so that all of the hugely complex existing scripts keep working) we might as well not upgrade. Frankly I suspect that down the line IF a new cli interpreter comes in to U-Boot we will have to keep the old one around as a "use this instead" option for another long number of years, so that if there are any systems with non-trivial scripts but upgrade U-Boot and don't / won't / can't re-validate their entire sequence, they can just use the old cli.

Do you actually have an example where code working on our ancient port of hush would fail on the current upstream version?

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
If you can't explain it to a six year old, you  don't  understand  it
yourself.                                           - Albert Einstein
#22 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 14 lines · Wolfgang Denk

Dear Tom,

Quoted 10 lines · In message <20210707135839.GW9516@bill-the-cat> you wrote:

As I've said a few times in this thread, this not being an sh-style interpreter is a strike against it. And if we're going to insist on a bug-for-bug upgrade to our hush (so that all of the hugely complex existing scripts keep working) we might as well not upgrade. Frankly I suspect that down the line IF a new cli interpreter comes in to U-Boot we will have to keep the old one around as a "use this instead" option for another long number of years, so that if there are any systems with non-trivial scripts but upgrade U-Boot and don't / won't / can't re-validate their entire sequence, they can just use the old cli.

Quoted 2 lines

Do you actually have an example where code working on our ancient port of hush would fail on the current upstream version?

Have you validated one of those exceedingly complex boot scripts with a modern hush (and some fakery for u-boot commands) ? No. I'm just saying I expect there to be enough risk-adverse groups that just dropping our old hush entirely might not be possible right away. Of course, if all of the current in-tree complex cases Just Work, that might be a good argument against needing to keep such levels of backwards compatibility.

Signature · 1 line
Tom
#23 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Dear Tom,

Quoted 3 lines · In message <20210707141418.GZ9516@bill-the-cat> you wrote:

Have you validated one of those exceedingly complex boot scripts with a modern hush (and some fakery for u-boot commands) ? No.

No, I havent.

I also don't claim that I know all the warts and issues with our old hus, but for the ones I am aware the woraround usually splitting complex or nested expressions into a sequence of simpler steps. I cannot remember any cases where the resulting code should be incompatible to a shell without that specific bug.

And yes, I am aware of the problems with the distro_bootcmd stuff. Thisis exactly what I had in mind when I wrote: if you run into such situations you should lean back and reflect a bit. I can undrstand the intentions of all this stuff, but the implementation is a horrible mess.

Quoted 6 lines

I'm just
saying I expect there to be enough risk-adverse groups that just dropping our old hush entirely might not be possible right away. Of course, if all of the current in-tree complex cases Just Work, that might be a good argument against needing to keep such levels of backwards compatibility.

There is only one way to test this.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
The inappropriate cannot be beautiful.
             - Frank Lloyd Wright _The Future of Architecture_ (1953)
#24 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

In reply to Tom Rini

Dear Tom,

Quoted 34 lines · In message <20210705191058.GB9516@bill-the-cat> you wrote:
Quoted 34 lines
Quoted 34 lines

foo=bar set foo bar
echo $foo echo $foo

if [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }

foo() {				proc foo {first second} {
	echo $1 $2			echo $first $second
}				}

for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }

fact() {
	if [ $1 -eq 0 ]; then
		echo 1
	else
		echo $(($1 * $(fact $(($1 - 1)))))
	fi
}
				proc fact {n} {
					if {$n} {
						expr {$n * [fact [expr {$n - 1}]]}
					} {
						return 1
					}
				}
Quoted 4 lines
Quoted 1 line
Quoted 1 line

Hopefully this gives you a bit of a feel for the basic differences.

Which of these things, from each column, can you do in the context of U-Boot? That's important too.

Well, with a current version of hush we can do:

-> foo=bar
-> echo $foo
bar

-> if [ 1 -gt 2 ]; then

Quoted 4 lines

echo a
else
echo b
fi

b

-> foo() {

Quoted 2 lines

echo $1 $2
}

-> foo bar baz
bar baz

-> for file in $(ls *.c); do

Quoted 2 lines

echo $file
done

ls: cannot access '*.c': No such file or directory

-> fact() {

Quoted 6 lines

if [ $1 -eq 0 ]; then
echo 1
else

          echo $(($1 * $(fact $(($1 - 1)))))

fi
}

-> fact 4
24

Oh, in the contect of U-Boot? Well, there are of course limitations, but not because of the shell, but because of the fact that we have no concept of files, for example.

But another command interpreter will not fix this.

Quoted 6 lines

This is I think the hard question. A draw of the current shell is that it it looks and acts like bash/sh/etc, for at least basic operations. That's something that's comfortable to a large audience. That has disadvantages when people want to start doing something complex. Sean has shown that several times and he's not the only one. LIL being tcl'ish is not.

Tcl is a horror of a language for anything that is above trivial level.

Do you really think that replacing standard shell syntax with Tcl is "something that's comfortable to a large audience"? I seriously doubt that.

Quoted 5 lines

Something that has "sh" syntax but also clear to the user errors when trying to do something not supported would also be interesting to see. It seems like a lot of the frustration from users with our shell is that it's not clear where the line between "this is an sh-like shell" and "no, not like that" is.

Did you run some tests on the version of hush as comes with recent busybox releases? Which of our user's requirements does it fail to meet?

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
The first thing we do is kill all the lawyers.
(Shakespeare. II Henry VI, Act IV, scene ii)
#25 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 77 lines · Wolfgang Denk

Dear Tom,

Quoted 34 lines · In message <20210705191058.GB9516@bill-the-cat> you wrote:
Quoted 34 lines
Quoted 34 lines

foo=bar set foo bar
echo $foo echo $foo

if [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }

foo() {				proc foo {first second} {
	echo $1 $2			echo $first $second
}				}

for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }

fact() {
	if [ $1 -eq 0 ]; then
		echo 1
	else
		echo $(($1 * $(fact $(($1 - 1)))))
	fi
}
				proc fact {n} {
					if {$n} {
						expr {$n * [fact [expr {$n - 1}]]}
					} {
						return 1
					}
				}
Quoted 4 lines
Quoted 1 line
Quoted 1 line

Hopefully this gives you a bit of a feel for the basic differences.

Which of these things, from each column, can you do in the context of U-Boot? That's important too.

Well, with a current version of hush we can do:

-> foo=bar
-> echo $foo
bar

-> if [ 1 -gt 2 ]; then

Quoted 4 lines

echo a
else
echo b
fi

b

-> foo() {

Quoted 2 lines

echo $1 $2
}

-> foo bar baz
bar baz

-> for file in $(ls *.c); do

Quoted 2 lines

echo $file
done

ls: cannot access '*.c': No such file or directory

-> fact() {

Quoted 6 lines

if [ $1 -eq 0 ]; then
echo 1
else

          echo $(($1 * $(fact $(($1 - 1)))))

fi
}

-> fact 4
24

Quoted 5 lines

Oh, in the contect of U-Boot? Well, there are of course limitations, but not because of the shell, but because of the fact that we have no concept of files, for example.

But another command interpreter will not fix this.

Yes, clearly the file based examples won't work either way, as-is. I was asking for what things can be done today with the implementations we have now.

I'm pretty confident that exactly zero people have written complex U-Boot scripts and then been happy about the experience.

Quoted 9 lines
Quoted 6 lines

This is I think the hard question. A draw of the current shell is that it it looks and acts like bash/sh/etc, for at least basic operations. That's something that's comfortable to a large audience. That has disadvantages when people want to start doing something complex. Sean has shown that several times and he's not the only one. LIL being tcl'ish is not.

Tcl is a horror of a language for anything that is above trivial level.

TCL has its fans. csh has it's fans. The question isn't what's the best desktop shell or general scripting language, but what's the most useful in our environment an use cases.

Quoted 3 lines

Do you really think that replacing standard shell syntax with Tcl is "something that's comfortable to a large audience"? I seriously doubt that.

I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.

Quoted 9 lines
Quoted 5 lines

Something that has "sh" syntax but also clear to the user errors when trying to do something not supported would also be interesting to see. It seems like a lot of the frustration from users with our shell is that it's not clear where the line between "this is an sh-like shell" and "no, not like that" is.

Did you run some tests on the version of hush as comes with recent busybox releases? Which of our user's requirements does it fail to meet?

It fails to meet the requirement of having been ported to U-Boot. I'd joke that perhaps you can bootefi busybox, but a quick search says that particular trick looks to still just be for python
(https://lwn.net/Articles/641244/).

Signature · 1 line
Tom
#26 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 6 lines · Tom Rini

I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.

I think the same would happen with Tcl and LIL. LIL might look similar to Tcl (and it is inspired by it), but it doesn't have the same syntax. A big difference comes from how variables are parsed with LIL allowing more variables symbols than Tcl without requiring escaping which affects some uses like expr, so e.g. in Tcl this

    set a 10 ; set b 20 ; expr $a+$b

will give back 30 however in LIL will give back 20. This is because Tcl parses "$a+$b" as "$a" "+" "$b" whereas LIL parses it as "$a+" "$b", evaluates "$a+" as an empty string (there isn't any "a+" variable), "$b" as 20 and then just runs expr with the single argument "20".

Kostas

#27 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 14 lines · Kostas Michalopoulos
Quoted 6 lines · Tom Rini

I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.

I think the same would happen with Tcl and LIL. LIL might look similar to Tcl (and it is inspired by it), but it doesn't have the same syntax. A big difference comes from how variables are parsed with LIL allowing more variables symbols than Tcl without requiring escaping which affects some uses like expr, so e.g. in Tcl this

Quoted 7 lines
     set a 10 ; set b 20 ; expr $a+$b

will give back 30 however in LIL will give back 20. This is because Tcl parses "$a+$b" as "$a" "+" "$b" whereas LIL parses it as "$a+" "$b", evaluates "$a+" as an empty string (there isn't any "a+" variable), "$b" as 20 and then just runs expr with the single argument "20".

IMO both of these are nuts. I think the best thing to do here is to raise an error about a missing variable "a+" to force the user to rewrite this as

	set a 10 ; set b 20 ; expr ${a}+$b

removing the ambiguity. Although I would generally like to follow TCL's lead, there are several places where TCL does crazy things (e.g. comments) which I have purposely not replicated.

--Sean

#28 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

In reply to Tom Rini

Dear Tom,

Quoted 3 lines · In message <20210706154346.GT9516@bill-the-cat> you wrote:

I'm pretty confident that exactly zero people have written complex U-Boot scripts and then been happy about the experience.

I have seen many U-Boot scripts which were pretty complex, but working absolutely reliably.

Quoted 3 lines

TCL has its fans. csh has it's fans. The question isn't what's the best desktop shell or general scripting language, but what's the most useful in our environment an use cases.

Maybe you should try and do a poll of our user base which CLI they _want_? I doubt there will be any significant percentage voting for Tcl.

I know of a large number of systems which offer a shell interface on their command line, and those who don't usually use completely proprietary code. I know of very few examples where Tcl is being used.

Quoted 6 lines

I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.

Point taken. But if you think this to an end, the result is: lets write some documentation and explain the limitations of a shell in U-Boot environment, and document the warts and bugs of this (or an updated) version of hush. This should make more users happy than completely new and incompatible stuff.

Frankly, I believe when you run into problems with hush in U-Boot (even the current version) you should lean back and think about what you are doing.

U-Boot is a boot loader, and while it is powerful enough to do complex things, this is not necessarily the most clever approach. 15 years ago, I've written complex update scripts for U-Boot. This was easy enough to do, and worked perfectly. But there are so many limitations in a boot loader environment. We don't do this any more. Instead, we use an OS suitable for such tasks (Linux with SWUpdate).

And talking about problems and limitations in U-Boot... Is the CLI really our biggest concern right now? None of our users (customers) has asked for a better command interpreter - the question we hear are more like: "When will you support IPv6?", "NFS does not work with recent Linux distros, will this be fixed?", "Can I download over WiFi?", "Can I download using HTTP/HTTPS?", "How can I harden U-Boot for security-critical environments?", etc.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
Always try to do things in chronological order; it's  less  confusing
that way.
#29 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 14 lines · Wolfgang Denk

Dear Tom,

Quoted 3 lines · In message <20210706154346.GT9516@bill-the-cat> you wrote:

I'm pretty confident that exactly zero people have written complex U-Boot scripts and then been happy about the experience.

I have seen many U-Boot scripts which were pretty complex, but working absolutely reliably.

Quoted 3 lines

TCL has its fans. csh has it's fans. The question isn't what's the best desktop shell or general scripting language, but what's the most useful in our environment an use cases.

Quoted 8 lines

Maybe you should try and do a poll of our user base which CLI they _want_? I doubt there will be any significant percentage voting for Tcl.

I know of a large number of systems which offer a shell interface on their command line, and those who don't usually use completely proprietary code. I know of very few examples where Tcl is being used.

Off the top of my head, most of the tooling for FPGAs uses TCL for scripting. OpenOCD uses it too.

Quoted 28 lines
Quoted 6 lines

I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.

Point taken. But if you think this to an end, the result is: lets write some documentation and explain the limitations of a shell in U-Boot environment, and document the warts and bugs of this (or an updated) version of hush. This should make more users happy than completely new and incompatible stuff.

Frankly, I believe when you run into problems with hush in U-Boot (even the current version) you should lean back and think about what you are doing.

U-Boot is a boot loader, and while it is powerful enough to do complex things, this is not necessarily the most clever approach. 15 years ago, I've written complex update scripts for U-Boot. This was easy enough to do, and worked perfectly. But there are so many limitations in a boot loader environment. We don't do this any more. Instead, we use an OS suitable for such tasks (Linux with SWUpdate).

Quoted 7 lines

And talking about problems and limitations in U-Boot... Is the CLI really our biggest concern right now? None of our users (customers) has asked for a better command interpreter - the question we hear are more like: "When will you support IPv6?", "NFS does not work with recent Linux distros, will this be fixed?", "Can I download over WiFi?", "Can I download using HTTP/HTTPS?", "How can I harden U-Boot for security-critical environments?", etc.

I wanted a better shell, so I worked on it.

--Sean

#30 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 64 lines · Sean Anderson
Quoted 14 lines · Wolfgang Denk

Dear Tom,

Quoted 3 lines · In message <20210706154346.GT9516@bill-the-cat> you wrote:

I'm pretty confident that exactly zero people have written complex U-Boot scripts and then been happy about the experience.

I have seen many U-Boot scripts which were pretty complex, but working absolutely reliably.

Quoted 3 lines

TCL has its fans. csh has it's fans. The question isn't what's the best desktop shell or general scripting language, but what's the most useful in our environment an use cases.

Quoted 8 lines

Maybe you should try and do a poll of our user base which CLI they _want_? I doubt there will be any significant percentage voting for Tcl.

I know of a large number of systems which offer a shell interface on their command line, and those who don't usually use completely proprietary code. I know of very few examples where Tcl is being used.

Off the top of my head, most of the tooling for FPGAs uses TCL for scripting. OpenOCD uses it too.

Quoted 28 lines
Quoted 6 lines

I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.

Point taken. But if you think this to an end, the result is: lets write some documentation and explain the limitations of a shell in U-Boot environment, and document the warts and bugs of this (or an updated) version of hush. This should make more users happy than completely new and incompatible stuff.

Frankly, I believe when you run into problems with hush in U-Boot (even the current version) you should lean back and think about what you are doing.

U-Boot is a boot loader, and while it is powerful enough to do complex things, this is not necessarily the most clever approach. 15 years ago, I've written complex update scripts for U-Boot. This was easy enough to do, and worked perfectly. But there are so many limitations in a boot loader environment. We don't do this any more. Instead, we use an OS suitable for such tasks (Linux with SWUpdate).

Quoted 7 lines

And talking about problems and limitations in U-Boot... Is the CLI really our biggest concern right now? None of our users (customers) has asked for a better command interpreter - the question we hear are more like: "When will you support IPv6?", "NFS does not work with recent Linux distros, will this be fixed?", "Can I download over WiFi?", "Can I download using HTTP/HTTPS?", "How can I harden U-Boot for security-critical environments?", etc.

Quoted 1 line

I wanted a better shell, so I worked on it.

This here is also an important point, and why I'm commenting on the series. A developer sees a problem, and works on the problem. I know I don't comment on as much stuff as I should, but for wide reaching patches, I really really try to.

Signature · 1 line
Tom
#31 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

In reply to Wolfgang Denk
Quoted 9 lines · Wolfgang Denk

Dear Tom,

Quoted 3 lines · In message <20210706154346.GT9516@bill-the-cat> you wrote:

I'm pretty confident that exactly zero people have written complex U-Boot scripts and then been happy about the experience.

I have seen many U-Boot scripts which were pretty complex, but working absolutely reliably.

As have I. We have them in-tree, even. My point is not "you cannot write these". My point is "you come away very disappointed in the interpreter" after writing these. It's certainly not bash/zsh. But it's also not POSIX-sh. It's not modern hush. It's unique, slightly different from everything else people normally have easy access to and those quirks aren't documented really.

Quoted 4 lines
Quoted 3 lines

TCL has its fans. csh has it's fans. The question isn't what's the best desktop shell or general scripting language, but what's the most useful in our environment an use cases.

Quoted 8 lines

Maybe you should try and do a poll of our user base which CLI they _want_? I doubt there will be any significant percentage voting for Tcl.

I know of a large number of systems which offer a shell interface on their command line, and those who don't usually use completely proprietary code. I know of very few examples where Tcl is being used.

A good point, yes.

Quoted 7 lines
Quoted 6 lines

I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.

Quoted 5 lines

Point taken. But if you think this to an end, the result is: lets write some documentation and explain the limitations of a shell in U-Boot environment, and document the warts and bugs of this (or an updated) version of hush. This should make more users happy than completely new and incompatible stuff.

That would be much appreciated. And to reiterate, our current hush isn't going away soon, or likely entirely within the decade. So this would be something long term useful to have written.

Quoted 3 lines

Frankly, I believe when you run into problems with hush in U-Boot (even the current version) you should lean back and think about what you are doing.

Agreed.

Quoted 7 lines

U-Boot is a boot loader, and while it is powerful enough to do complex things, this is not necessarily the most clever approach. 15 years ago, I've written complex update scripts for U-Boot. This was easy enough to do, and worked perfectly. But there are so many limitations in a boot loader environment. We don't do this any more. Instead, we use an OS suitable for such tasks (Linux with SWUpdate).

Yes. Although there are some complex things that we can't push up higher (see distro_bootcmd) and update systems still tend to need something from us (or they do it in grub instead).

Signature · 1 line
Tom
#32 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

In reply to Wolfgang Denk

Dear Tom, Sean, Wolfgang and others,

here are some of my opinions for this discussion

- I agree with Wolfgang that there are far better options than a Tcl-like shell, if we want to add another language

- I also think that instead of adding another language, it is more preferable to improve the existing one. Adding a new language will cause more problems in the future:
- I think it can end up with OS distributions needing to write boot scripts in both languages, because they can't be sure which will be compiled into U-Boot
- we will certainly end up with more bugs
- userbase will fragment between the two languages

- I think we can start improving the current U-Boot's shell in ways that are incompatible with upstream Hush.

The idea back then, as I understand it, was to minimize man-hours invested into the CLI code, and so an existing shell was incorporated (with many #ifdef guards). But U-Boot has since evolved so much that it is very probable it would be more economic to simply fork from upsteam Hush, remove all the #ifdefs and start developing features we want in U-Boot. Is upstream Hush even maintained properly? What is the upstream repository? Is it
https://github.com/sheumann/hush?

- even if we decide to stay with upstream Hush and just upgrade U-Boot's Hush to upstream (since it supports functions, arithmetic with $((...)), command substitution with $(...), these are all nice features), it is IMO still better than adding a new language

- one of the points Sean mentioned with LIL is that when compiled, it's size does not exceed the size of U-Boot's Hush.

If we were to add new features into U-Boot's Hush, the code size would certainly increase.

I think we should implement these new features, and instead of adding a new language, we should work on minimizing the code size / resulting U-Boot image size. This is where U-Boot will gain most not only with it's CLI, but also everywhere else. Regarding this,
- we already have LTO
- Simon worked on dtoc so that devicetrees can be compiled into C code
- we can start playing with compression
- either we can compress the whole image for machines with enough RAM but small place for U-Boot (Nokia N900 for example has only 256 KiB space for U-Boot)
- or we can try to invent a way to decompress code when it is needed, for machines with small RAM

Marek

#33 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Hi

Quoted 20 lines · Marek Behun

Dear Tom, Sean, Wolfgang and others,

here are some of my opinions for this discussion

- I agree with Wolfgang that there are far better options than a Tcl-like shell, if we want to add another language

- I also think that instead of adding another language, it is more preferable to improve the existing one. Adding a new language will cause more problems in the future:
- I think it can end up with OS distributions needing to write boot scripts in both languages, because they can't be sure which will be compiled into U-Boot
- we will certainly end up with more bugs
- userbase will fragment between the two languages

- I think we can start improving the current U-Boot's shell in ways that are incompatible with upstream Hush.

Quoted 9 lines

The idea back then, as I understand it, was to minimize man-hours invested into the CLI code, and so an existing shell was incorporated (with many #ifdef guards). But U-Boot has since evolved so much that it is very probable it would be more economic to simply fork from upsteam Hush, remove all the #ifdefs and start developing features we want in U-Boot. Is upstream Hush even maintained properly? What is the upstream repository? Is it
https://github.com/sheumann/hush?

I think that hush is the one that is now in the busybox. I could spent ten minutes this morning and this is my short list:

- we have several define that allow it to enabled e/o disable a lot of features
- we are talking about 11K lines compared to 3K (including comment)
- we have 25-30 configuration option on hush on busybox
- in u-boot code some of the problem was solved some time ago
- as describe is 68Kb, I think this consider all the option enables
- the code is different from what we have and what is there

I don't know if options like ENABLE_HUSH_JOB and ENABLE_MMU can partially solve some of the problems described in the thread

* Sean *: You have spent more on this, can you please complete it.

Out of that. Do we have some script shell unit test in uboot?

Michael

Quoted 25 lines

- even if we decide to stay with upstream Hush and just upgrade U-Boot's Hush to upstream (since it supports functions, arithmetic with $((...)), command substitution with $(...), these are all nice features), it is IMO still better than adding a new language

- one of the points Sean mentioned with LIL is that when compiled, it's size does not exceed the size of U-Boot's Hush.

If we were to add new features into U-Boot's Hush, the code size would certainly increase.

I think we should implement these new features, and instead of adding a new language, we should work on minimizing the code size / resulting U-Boot image size. This is where U-Boot will gain most not only with it's CLI, but also everywhere else. Regarding this,
- we already have LTO
- Simon worked on dtoc so that devicetrees can be compiled into C code
- we can start playing with compression
- either we can compress the whole image for machines with enough RAM but small place for U-Boot (Nokia N900 for example has only 256 KiB space for U-Boot)
- or we can try to invent a way to decompress code when it is needed, for machines with small RAM

Marek

Signature · 11 lines
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
[email protected]
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
[email protected]
www.amarulasolutions.com
#34 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Quoted 43 lines · Michael Nazzareno Trimarchi

Hi

Quoted 20 lines · Marek Behun

Dear Tom, Sean, Wolfgang and others,

here are some of my opinions for this discussion

- I agree with Wolfgang that there are far better options than a Tcl-like shell, if we want to add another language

- I also think that instead of adding another language, it is more preferable to improve the existing one. Adding a new language will cause more problems in the future:
- I think it can end up with OS distributions needing to write boot scripts in both languages, because they can't be sure which will be compiled into U-Boot
- we will certainly end up with more bugs
- userbase will fragment between the two languages

- I think we can start improving the current U-Boot's shell in ways that are incompatible with upstream Hush.

Quoted 9 lines

The idea back then, as I understand it, was to minimize man-hours invested into the CLI code, and so an existing shell was incorporated (with many #ifdef guards). But U-Boot has since evolved so much that it is very probable it would be more economic to simply fork from upsteam Hush, remove all the #ifdefs and start developing features we want in U-Boot. Is upstream Hush even maintained properly? What is the upstream repository? Is it
https://github.com/sheumann/hush?

I think that hush is the one that is now in the busybox. I could spent ten minutes this morning and this is my short list:

- we have several define that allow it to enabled e/o disable a lot of features
- we are talking about 11K lines compared to 3K (including comment)
- we have 25-30 configuration option on hush on busybox
- in u-boot code some of the problem was solved some time ago
- as describe is 68Kb, I think this consider all the option enables
- the code is different from what we have and what is there

Quoted 6 lines

I don't know if options like ENABLE_HUSH_JOB and ENABLE_MMU can partially solve some of the problems described in the thread

* Sean *: You have spent more on this, can you please complete it.

Out of that. Do we have some script shell unit test in uboot?

To the last point, yes, we have some, but always need more, tests for how hush behaves as part of pytest.

Signature · 1 line
Tom
#35 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

In reply to Wolfgang Denk
Quoted 10 lines · Wolfgang Denk

Dear Tom,

Quoted 6 lines · In message <20210705191058.GB9516@bill-the-cat> you wrote:

This is I think the hard question. A draw of the current shell is that it it looks and acts like bash/sh/etc, for at least basic operations. That's something that's comfortable to a large audience. That has disadvantages when people want to start doing something complex. Sean has shown that several times and he's not the only one. LIL being tcl'ish is not.

Quoted 2 lines

Tcl is a horror of a language for anything that is above trivial level.

Can you please elaborate on this? You've made this claim several times, but haven't explained your reasoning for it.

--Sean

#36 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

Dear Sean,

Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 3 lines

Tcl is a horror of a language for anything that is above trivial level.

Can you please elaborate on this? You've made this claim several times, but haven't explained your reasoning for it.

A long, long time ago I I was looking for an efficient way to do regression testing for embedded systems, mainly U-Boot and Linux. I found GNU gnats to bee too difficult to handle und started writing my own test system (you can still find this ancient stuff here [1]) based on expect/Tcl. At that time, I could not find any other solution, so I had to accept Tcl as programming language for this
tool. And at the beginning it usually worked just fine. Mostly.
Coming from a C programming environment I always found Tcl ... strange. For example it's scoping rules (see [2] for an example). Things got worse when the system grew and we had to deal with strings or parameters which might contail quotes or other special characters. This required fancy quoting, which depended of the calling depth - if there was only on specific level. you might find a solution after some (cumbersome) experimenting. But if the same string was used in several places in the code, and different levels, you really lost. At a certain point all our developers refused to write new test cases for the system, because it always was the same nightmare of fighting unforeseeable language problems.

That was when we abandoned DUTS, and some time later Heiko came up with an initial implementation of tbot, soon to be joined by Harald to make a production quality test system out of it [4].

IMO, Tcl has a large number of situations where it's actual behaviours is not what you expect from it - it is good enough for simple things, but any more complex processing of arbitrary text data quickly drives you insane - things like quotes or braces and other special characters.

[1] http://www.denx.de/wiki/DUTS/DUTSDocs
[2] https://wiki.tcl-lang.org/page/Commonly+encountered+problems+in+Tcl
[3] https://wiki.tcl-lang.org/page/Why+can+I+not+place+unmatched+braces+in+Tcl+comments
[4] https://tbot.tools/

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
The number you have dialed is imaginary. Please divide by 0  and  try
again.
#37 lore raw

Re: [RFC PATCH 02/28] cli: Add LIL shell

In reply to Sean Anderson

Dear Sean,

Quoted 7 lines · In message <[email protected]> you wrote:
Quoted 3 lines

Do you have a list of the exact differencec between LIL and a standard shell?

For a partial list, see

[1] https://github.com/Forty-Bot/lil/commits/master

Hm, this list of commits is not exactly helpful, I'm afraid.

Where _exactly_ should I look?

Quoted 5 lines
Quoted 2 lines

I wonder, if we deviate from standard shell syntax anyway, we could also have a look at lua, for example?

I also looked at lua (see the cover letter), but I rejected it based on size constraints (eLua was around the size of U-Boot itself).

I have to admit that I never tried myself to build a lua based system, optimizing for minimal size - but I'm surprised by your results.

Quoted 8 lines

Because of how often the shell is used to debug things, I wanted the candidate I picked to have similar syntax to the existing shell. For example,

	load mmc ${mmcdev}:${mmcpart} $loadaddr $image

is a valid command in both Hush and LIL. Compare with lua, which might express the above as

Yes, I'm aware of this. But then, it would add another level of powerful scripting capabilities - and as it was already suggested before, _replacing_ the standard shell is only one way to use lua - another would be to use lua as a command that can be started from the shell when needed - assuming you want to pay the price in terms of size.

Best regards,

Wolfgang Denk

Signature · 6 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
They're usually so busy thinking about what  happens  next  that  the
only  time they ever find out what is happening now is when they come
to look back on it.                 - Terry Pratchett, _Wyrd Sisters_
#38 lore raw

RFC PATCH 3/28 cli: lil: Replace strclone with strdup

In reply to Sean Anderson

Apparently strdup is not portable, so LIL used its own. Use strdup.

---

 common/cli_lil.c | 27 ++++++++-------------------
 1 file changed, 8 insertions(+), 19 deletions(-)
common/cli_lil.c +8 -19
index c8c6986fe1..294d79e86e 100644
@@ -123,17 +123,6 @@ static struct lil_env **envpool;
 static size_t envpoolsize, envpoolcap;
 #endif
 
-static char *strclone(const char *s)
-{
-	size_t len = strlen(s) + 1;
-	char *ns = malloc(len);
-
-	if (!ns)
-		return NULL;
-	memcpy(ns, s, len);
-	return ns;
-}
-
 static unsigned long hm_hash(const char *key)
 {
 	unsigned long hash = 5381;
@@ -173,7 +162,7 @@ static void hm_put(struct hashmap *hm, const char *key, void *value)
 	}
 
 	cell->e = realloc(cell->e, sizeof(struct hashentry) * (cell->c + 1));
-	cell->e[cell->c].k = strclone(key);
+	cell->e[cell->c].k = strdup(key);
 	cell->e[cell->c].v = value;
 	cell->c++;
 }
@@ -606,7 +595,7 @@ static struct lil_func *add_func(struct lil *lil, const char *name)
 	}
 
 	cmd = calloc(1, sizeof(struct lil_func));
-	cmd->name = strclone(name);
+	cmd->name = strdup(name);
 
 	ncmd = realloc(lil->cmd, sizeof(struct lil_func *) * (lil->cmds + 1));
 	if (!ncmd) {
@@ -705,7 +694,7 @@ struct lil_var *lil_set_var(struct lil *lil, const char *name,
 
 	env->var = nvar;
 	nvar[env->vars] = calloc(1, sizeof(struct lil_var));
-	nvar[env->vars]->n = strclone(name);
+	nvar[env->vars]->n = strdup(name);
 	nvar[env->vars]->env = env;
 	nvar[env->vars]->v = freeval ? val : lil_clone_value(val);
 	hm_put(&env->varmap, name, nvar[env->vars]);
@@ -1207,7 +1196,7 @@ void lil_set_error(struct lil *lil, const char *msg)
 	free(lil->err_msg);
 	lil->error = ERROR_FIXHEAD;
 	lil->err_head = 0;
-	lil->err_msg = strclone(msg ? msg : "");
+	lil->err_msg = strdup(msg ? msg : "");
 }
 
 void lil_set_error_at(struct lil *lil, size_t pos, const char *msg)
@@ -1218,7 +1207,7 @@ void lil_set_error_at(struct lil *lil, size_t pos, const char *msg)
 	free(lil->err_msg);
 	lil->error = ERROR_DEFAULT;
 	lil->err_head = pos;
-	lil->err_msg = strclone(msg ? msg : "");
+	lil->err_msg = strdup(msg ? msg : "");
 }
 
 int lil_error(struct lil *lil, const char **msg, size_t *pos)
@@ -2016,7 +2005,7 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
 		hm_put(&lil->cmdmap, oldname, 0);
 		hm_put(&lil->cmdmap, newname, func);
 		free(func->name);
-		func->name = strclone(newname);
+		func->name = strdup(newname);
 	} else {
 		del_func(lil, func);
 	}
@@ -2728,7 +2717,7 @@ static struct lil_value *real_trim(const char *str, const char *chars, int left,
 		size_t len;
 		char *s;
 
-		s = strclone(str + base);
+		s = strdup(str + base);
 		len = strlen(s);
 		while (len && strchr(chars, s[len - 1]))
 			len--;
@@ -2833,7 +2822,7 @@ static struct lil_value *fnc_repstr(struct lil *lil, size_t argc,
 	if (!from[0])
 		return NULL;
 
-	src = strclone(lil_to_string(argv[0]));
+	src = strdup(lil_to_string(argv[0]));
 	srclen = strlen(src);
 	fromlen = strlen(from);
 	tolen = strlen(to);
Signature · 1 line
2.32.0
#39 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

Quoted 1 line · Sean Anderson

Apparently strdup is not portable, so LIL used its own. Use strdup.

You could reduce the churn by just making strclone "#define strclone(x) strdup(x)", but I suppose you end up modifying the upstream code so much that there's not really anything gained by that.

But that begs the question: What is the long-term plan for this? While it does seem to be an improvement compared to hush, will we ever be able to incorporate fixes&features from upstream, or will this code end up in the same situation as hush?

Have you been in contact with upstream about this project? Perhaps some of the things you do could go upstream - e.g. the conversion from an array of wrongly-typed callbacks to an "ops" struct seems to be an obvious improvement [lil_callback would have to be kept, and changed to use a switch() statement to update the right slot, but we wouldn't have to care because ld will remove it anyway if we don't add any users of it].

Rasmus

#40 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

Dear Rasmus,

Quoted 5 lines · In message <[email protected]> you wrote:

But that begs the question: What is the long-term plan for this? While it does seem to be an improvement compared to hush, will we ever be able to incorporate fixes&features from upstream, or will this code end up in the same situation as hush?

I would like to put the context of this right.

As written, "improvement compared to hush", this could be misunderstood. What you likely mean is "compared to our ancient (nearly 20 years old) version of hush".

Comparing to recent versions have probably much different results.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
Ever try. Ever fail. No matter. Try again. Fail again.  Fail  better.
                                                        -- S. Beckett
#41 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

In reply to Rasmus Villemoes
Quoted 11 lines · Rasmus Villemoes
Quoted 1 line · Sean Anderson

Apparently strdup is not portable, so LIL used its own. Use strdup.

You could reduce the churn by just making strclone "#define strclone(x) strdup(x)", but I suppose you end up modifying the upstream code so much that there's not really anything gained by that.

But that begs the question: What is the long-term plan for this? While it does seem to be an improvement compared to hush, will we ever be able to incorporate fixes&features from upstream, or will this code end up in the same situation as hush?

Well, since Hush was never updated, I don't believe LIL will be either. I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.

The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.

Quoted 7 lines

Have you been in contact with upstream about this project? Perhaps some of the things you do could go upstream - e.g. the conversion from an array of wrongly-typed callbacks to an "ops" struct seems to be an obvious improvement [lil_callback would have to be kept, and changed to use a switch() statement to update the right slot, but we wouldn't have to care because ld will remove it anyway if we don't add any users of it].

I have not. I'm not sure what changes I've made are compatible with the vision he has for LIL. He's CC'd on this series, so perhaps he can comment.

--Sean

#42 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

Quoted 18 lines · Sean Anderson
Quoted 11 lines · Rasmus Villemoes
Quoted 1 line · Sean Anderson

Apparently strdup is not portable, so LIL used its own. Use strdup.

You could reduce the churn by just making strclone "#define strclone(x) strdup(x)", but I suppose you end up modifying the upstream code so much that there's not really anything gained by that.

But that begs the question: What is the long-term plan for this? While it does seem to be an improvement compared to hush, will we ever be able to incorporate fixes&features from upstream, or will this code end up in the same situation as hush?

Well, since Hush was never updated, I don't believe LIL will be either. I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.

Quoted 4 lines

The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.

Port it and forget it was a problem, not a bug, of our hush implementation. For other things, I'm trying to keep us in sync regularly, but having less luck as some stuff has already gotten well behind and is non-trivial to resync. A digression, but for this thread I think keeping abreast of LIL releases is important. I diff'd 20190821 to 20210502 and it was (as you note in another part of the thread) copyright, license addition and I saw one line of code move around but didn't check if that applied here too or not. A monthly calendar reminder to check the site for new releases (which is what I do for Kconfiglib) should keep us in-sync, especially given the LIL project intentions.

Signature · 1 line
Tom
#43 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

In reply to Sean Anderson
Quoted 3 lines · Sean Anderson

I have not. I'm not sure what changes I've made are compatible with the vision he has for LIL. He's CC'd on this series, so perhaps he can comment.

Yeah, sadly several of the changes i've seen in the code are not backwards compatible and backwards compatibility is very important for me (this is also why the callback setup is done through a lil_callback function instead of exposing a struct: adding new callbacks in the future wont break any existing code or application that links to the library dynamically). I might break it before a properly versioned release is made at some point in the (not close) future, but that is usually unavoidable changes to fix big issues (all cases where that happened are mentioned in the site).

In terms of future changes, i do not plan making any *big* changes - and certainly anything i'll make it'll be backwards compatible (especially after a versioned release) but one thing that i need to do is to improve the interpreter's performance, which will affect its internals - two areas i might need to change are how values are represented (everything is a string -and things must always behave like that- but i may need to cache already parsed non-string values) and how the code itself -e.g. in functions- is represented.

Also i'll most likely add more functions to the library to expose and alter internal state (the Free Pascal version of LIL already has some of that), but that shouldn't affect much internally.

Beyond that it will be mainly bugfixes - like a use-after-free case with "reflect this" i just fixed and uploaded :-P

Kostas

#44 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

Quoted 8 lines · Kostas Michalopoulos
Quoted 3 lines · Sean Anderson

I have not. I'm not sure what changes I've made are compatible with the vision he has for LIL. He's CC'd on this series, so perhaps he can comment.

Yeah, sadly several of the changes i've seen in the code are not backwards compatible and backwards compatibility is very important for me

I don't intend to keep backward compatibility. The way I see it, U-Boot has never had a LIL shell before, so there is nothing to break :)

Quoted 4 lines

(this is also why the callback setup is done through a lil_callback function instead of exposing a struct: adding new callbacks in the future wont break any existing code or application that links to the library dynamically).

Well, C has a very nice ABI in some repects. As long as you only append to that struct you will not break compatibility.

Quoted 5 lines

I might break it before a properly versioned release is made at some point in the (not close) future, but that is usually unavoidable changes to fix big issues (all cases where that happened are mentioned in the site).

Quoted 8 lines

In terms of future changes, i do not plan making any *big* changes - and certainly anything i'll make it'll be backwards compatible (especially after a versioned release) but one thing that i need to do is to improve the interpreter's performance, which will affect its internals - two areas i might need to change are how values are represented (everything is a string -and things must always behave like that- but i may need to cache already parsed non-string values) and how the code itself -e.g. in functions- is represented.

One of the nice side-effects of adding a separate parser is that it will now be very easy to get some performance gains by parsing code once and then re-using that parse on subsequent evaluations. For example, functions could be parsed on first use, and cached. Loop bodies could re-used for subsequent iterations.

Quoted 6 lines

Also i'll most likely add more functions to the library to expose and alter internal state (the Free Pascal version of LIL already has some of that), but that shouldn't affect much internally.

Beyond that it will be mainly bugfixes - like a use-after-free case with "reflect this" i just fixed and uploaded :-P

As it happens, I ended up removing that because there was no longer a "rootcode" after removing the parsing step. Perhaps at some point I will add it back, but I do not see a pressing need for the same level of dynamicness that you built in originally.

--Sean

#45 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

In reply to Sean Anderson

Dear Sean,

Quoted 2 lines · In message <[email protected]> you wrote:

Well, since Hush was never updated, I don't believe LIL will be either.

Let's please be exact here: Hus has never been updated _in_U-Boot_, but it has seen a lot of changes upstream, which apparently fix all the issues that motivated you to look for a replacement.

Quoted 8 lines

I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.

The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.

This sounds like an advantage, indeed, but then you can also interpret this as betting on a dead horse...

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
Never worry about theory as long as  the  machinery  does  what  it's
supposed to do.                                      - R. A. Heinlein
#46 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

Quoted 15 lines · Wolfgang Denk

Dear Sean,

Quoted 2 lines · In message <[email protected]> you wrote:

Well, since Hush was never updated, I don't believe LIL will be either.

Let's please be exact here: Hus has never been updated _in_U-Boot_, but it has seen a lot of changes upstream, which apparently fix all the issues that motivated you to look for a replacement.

Quoted 4 lines

I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.

Quoted 7 lines
Quoted 4 lines

The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.

This sounds like an advantage, indeed, but then you can also interpret this as betting on a dead horse...

My 2c on this.

I am the maintainer of JimTcl (and I agree it is too big to be considered a candidate). LIL source code has almost zero comments, poor error checking and no test suite. I would be very hesitant to adopt it in u-boot without serious work.

I would much rather see effort put into updating hush to upstream.
My guess is that Denys would be amenable to small changes to make it easier to synchronise with busybox in the future.

Cheers,
Steve

#47 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

Quoted 21 lines · Steve Bennett
Quoted 20 lines · Wolfgang Denk

Dear Sean,

Quoted 2 lines · In message <[email protected]> you wrote:

Well, since Hush was never updated, I don't believe LIL will be either.

Let's please be exact here: Hus has never been updated _in_U-Boot_, but it has seen a lot of changes upstream, which apparently fix all the issues that motivated you to look for a replacement.

Quoted 8 lines

I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.

The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.

Quoted 7 lines
Quoted 2 lines

This sounds like an advantage, indeed, but then you can also interpret this as betting on a dead horse...

My 2c on this.

I am the maintainer of JimTcl (and I agree it is too big to be considered a candidate). LIL source code has almost zero comments, poor error checking and no test suite.

FWIW I added a (small) test suite in "[RFC PATCH 17/28] test: Add tests for LIL" based on the tests included in the LIL distibution. However, I really would like to expand upon it.

Quoted 1 line

I would be very hesitant to adopt it in u-boot without serious work.

I think around half of the "serious work" has already been done. I have worked on most of the core of LIL, and added error handling and comments. I believe that most of the remaining instances of dropping errors lie in the built-in commands.

Quoted 1 line

I would much rather see effort put into updating hush to upstream.

AIUI hush has diverged significantly from what U-Boot has. This would not be an "update" moreso than a complete port in the style of the current series.

Quoted 2 lines

My guess is that Denys would be amenable to small changes to make it easier to synchronise with busybox in the future.

I don't think sh-style shells are a good match for U-Boot's execution environment in the first place. The fundamental idea of an sh-style shell is that the output of one command can be redirected to the input (or arguments) of another command. This cannot be done (or rather would be difficult to do) in U-Boot for a few reasons

* U-Boot does not support multithreading. Existing shells tend to depend strongly on this feature of the enviromnent. Many of the changes to U-Boot's hush are solely to deal with the lack of this feature.

* Existing commands do not read from stdin, nor do they print useful information to stdout. Command output is designed for human consumption and is substantially more verbose than typical unix commands.

* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely useful when working with streams are not present in U-Boot.

And of course, this feature is currently not present in U-Boot. To get around this, commands resort to two of my least-favorite hacks: passing in the name of a environmental variable and overloading the return value. For an example of the first, consider

=> part uuid mmc 0:1 my_uuid

which will set my_uuid to the uuid of the selected partition. My issue with this is threefold: every command must add new syntax to do this, that syntax is inconsistent, and it prevents easy composition. Consider a script which wants to iterate over partitions. Instead of doing

	for p in $(part list mmc 0); do
		# ...
	done

it must instead do

part list mmc 0 partitions
for p in $partitions; do
# ...
done

which unnecessarily adds an extra step. This overhead accumulates with each command which adds something like this.

The other way to return more information is to use the return value. Consider the button command; it currently returns

0 ON, the button is pressed
1 OFF, the button is released
0 button list was shown
1 button not found
1 invalid arguments

and so there is no way to distinguish between whether the button is off, whether the button does not exist, or whether there was a problem with the button driver.

Both of these workarounds are natural consequences of using a sh-tyle shell in an environment it is not suited for. If we are going to go to the effort of porting a new language (which must be done no matter if we use Hush or some other language), we should pick one which has better support for single-threaded programming.

--Sean

#48 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

Hi,

Quoted 108 lines · Sean Anderson
Quoted 21 lines · Steve Bennett
Quoted 20 lines · Wolfgang Denk

Dear Sean,

Quoted 2 lines · In message <[email protected]> you wrote:

Well, since Hush was never updated, I don't believe LIL will be either.

Let's please be exact here: Hus has never been updated _in_U-Boot_, but it has seen a lot of changes upstream, which apparently fix all the issues that motivated you to look for a replacement.

Quoted 8 lines

I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.

The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.

Quoted 7 lines
Quoted 2 lines

This sounds like an advantage, indeed, but then you can also interpret this as betting on a dead horse...

My 2c on this.

I am the maintainer of JimTcl (and I agree it is too big to be considered a candidate). LIL source code has almost zero comments, poor error checking and no test suite.

FWIW I added a (small) test suite in "[RFC PATCH 17/28] test: Add tests for LIL" based on the tests included in the LIL distibution. However, I really would like to expand upon it.

Quoted 1 line

I would be very hesitant to adopt it in u-boot without serious work.

I think around half of the "serious work" has already been done. I have worked on most of the core of LIL, and added error handling and comments. I believe that most of the remaining instances of dropping errors lie in the built-in commands.

Quoted 1 line

I would much rather see effort put into updating hush to upstream.

AIUI hush has diverged significantly from what U-Boot has. This would not be an "update" moreso than a complete port in the style of the current series.

Quoted 2 lines

My guess is that Denys would be amenable to small changes to make it easier to synchronise with busybox in the future.

I don't think sh-style shells are a good match for U-Boot's execution environment in the first place. The fundamental idea of an sh-style shell is that the output of one command can be redirected to the input (or arguments) of another command. This cannot be done (or rather would be difficult to do) in U-Boot for a few reasons

* U-Boot does not support multithreading. Existing shells tend to depend strongly on this feature of the enviromnent. Many of the changes to U-Boot's hush are solely to deal with the lack of this feature.

* Existing commands do not read from stdin, nor do they print useful information to stdout. Command output is designed for human consumption and is substantially more verbose than typical unix commands.

* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely useful when working with streams are not present in U-Boot.

And of course, this feature is currently not present in U-Boot. To get around this, commands resort to two of my least-favorite hacks: passing in the name of a environmental variable and overloading the return value. For an example of the first, consider

=> part uuid mmc 0:1 my_uuid

which will set my_uuid to the uuid of the selected partition. My issue with this is threefold: every command must add new syntax to do this, that syntax is inconsistent, and it prevents easy composition. Consider a script which wants to iterate over partitions. Instead of doing

        for p in $(part list mmc 0); do
                # ...
        done

it must instead do

part list mmc 0 partitions
for p in $partitions; do
# ...
done

which unnecessarily adds an extra step. This overhead accumulates with each command which adds something like this.

The other way to return more information is to use the return value. Consider the button command; it currently returns

        0       ON, the button is pressed
        1       OFF, the button is released
        0       button list was shown
        1       button not found
        1       invalid arguments

and so there is no way to distinguish between whether the button is off, whether the button does not exist, or whether there was a problem with the button driver.

Quoted 5 lines

Both of these workarounds are natural consequences of using a sh-tyle shell in an environment it is not suited for. If we are going to go to the effort of porting a new language (which must be done no matter if we use Hush or some other language), we should pick one which has better support for single-threaded programming.

I think these are good points, particularly the thing about mutiltasking. In fact I think it would be better if hush could implement things like '| grep xxx' since it would be useful in U-Boot.

But in any case, adding lil does not preclude someone coming along and adaptive hush for U-Boot (and this time upstreaming the changes!). I have seen discussions about updating hush for several years and no one has done it. I took a quick look and found it was about 3x the size it used to be and was not even sure where to start in terms of adapting it for single-threaded use.

Re this patch, I think it should be sent upstream.

Regards,
Simon

#49 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

Quoted 121 lines · Simon Glass

Hi,

Quoted 108 lines · Sean Anderson
Quoted 21 lines · Steve Bennett
Quoted 20 lines · Wolfgang Denk

Dear Sean,

Quoted 2 lines · In message <[email protected]> you wrote:

Well, since Hush was never updated, I don't believe LIL will be either.

Let's please be exact here: Hus has never been updated _in_U-Boot_, but it has seen a lot of changes upstream, which apparently fix all the issues that motivated you to look for a replacement.

Quoted 8 lines

I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.

The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.

Quoted 7 lines
Quoted 2 lines

This sounds like an advantage, indeed, but then you can also interpret this as betting on a dead horse...

My 2c on this.

I am the maintainer of JimTcl (and I agree it is too big to be considered a candidate). LIL source code has almost zero comments, poor error checking and no test suite.

FWIW I added a (small) test suite in "[RFC PATCH 17/28] test: Add tests for LIL" based on the tests included in the LIL distibution. However, I really would like to expand upon it.

Quoted 1 line

I would be very hesitant to adopt it in u-boot without serious work.

I think around half of the "serious work" has already been done. I have worked on most of the core of LIL, and added error handling and comments. I believe that most of the remaining instances of dropping errors lie in the built-in commands.

Quoted 1 line

I would much rather see effort put into updating hush to upstream.

AIUI hush has diverged significantly from what U-Boot has. This would not be an "update" moreso than a complete port in the style of the current series.

Quoted 2 lines

My guess is that Denys would be amenable to small changes to make it easier to synchronise with busybox in the future.

I don't think sh-style shells are a good match for U-Boot's execution environment in the first place. The fundamental idea of an sh-style shell is that the output of one command can be redirected to the input (or arguments) of another command. This cannot be done (or rather would be difficult to do) in U-Boot for a few reasons

* U-Boot does not support multithreading. Existing shells tend to depend strongly on this feature of the enviromnent. Many of the changes to U-Boot's hush are solely to deal with the lack of this feature.

* Existing commands do not read from stdin, nor do they print useful information to stdout. Command output is designed for human consumption and is substantially more verbose than typical unix commands.

* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely useful when working with streams are not present in U-Boot.

And of course, this feature is currently not present in U-Boot. To get around this, commands resort to two of my least-favorite hacks: passing in the name of a environmental variable and overloading the return value. For an example of the first, consider

=> part uuid mmc 0:1 my_uuid

which will set my_uuid to the uuid of the selected partition. My issue with this is threefold: every command must add new syntax to do this, that syntax is inconsistent, and it prevents easy composition. Consider a script which wants to iterate over partitions. Instead of doing

         for p in $(part list mmc 0); do
                 # ...
         done

it must instead do

part list mmc 0 partitions
for p in $partitions; do
# ...
done

which unnecessarily adds an extra step. This overhead accumulates with each command which adds something like this.

The other way to return more information is to use the return value. Consider the button command; it currently returns

         0       ON, the button is pressed
         1       OFF, the button is released
         0       button list was shown
         1       button not found
         1       invalid arguments

and so there is no way to distinguish between whether the button is off, whether the button does not exist, or whether there was a problem with the button driver.

Quoted 5 lines

Both of these workarounds are natural consequences of using a sh-tyle shell in an environment it is not suited for. If we are going to go to the effort of porting a new language (which must be done no matter if we use Hush or some other language), we should pick one which has better support for single-threaded programming.

I think these are good points, particularly the thing about mutiltasking. In fact I think it would be better if hush could implement things like '| grep xxx' since it would be useful in U-Boot.

Quoted 8 lines

But in any case, adding lil does not preclude someone coming along and adaptive hush for U-Boot (and this time upstreaming the changes!). I have seen discussions about updating hush for several years and no one has done it. I took a quick look and found it was about 3x the size it used to be and was not even sure where to start in terms of adapting it for single-threaded use.

Re this patch, I think it should be sent upstream.

I believe it will not be accepted, since strdup is POSIX and not C99.

--Sean

#50 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

In reply to Sean Anderson

Dear Sean,

Quoted 4 lines · In message <[email protected]> you wrote:

AIUI hush has diverged significantly from what U-Boot has. This would not be an "update" moreso than a complete port in the style of the current series.

Agreed. However, as you write this it sounds like a big problem only. I disagree here - it is also a chance to do a few things different than with the original port.

Please keep in mind when this original port of the hush shell was
done: it was a time when many systems came with a total of 4 MB flash memory, and not only U-Boot, but also the Linux kernel and a ram-disk image or such had to fit into that. At that time 40 or 60 kB code size for just a fancy shell was immense!

Under such restrictions (and the need to complete the task with a given budget) many things were just commented out which from today's point of view would be nice to have.

This is not a problem of the code complexity or resources, but only of different requirements and different resources.

Quoted 5 lines

I don't think sh-style shells are a good match for U-Boot's execution environment in the first place. The fundamental idea of an sh-style shell is that the output of one command can be redirected to the input (or arguments) of another command. This cannot be done (or rather would be difficult to do) in U-Boot for a few reasons

There is an old saying:

The loser says: "It might be possible, but it is too difficult." The winner says: "It might be difficult, but it is possible."

Apparently you take another position here than me.

First, I disagree that pipes are the "fundamental idea" of a shell. It is a fundamental idea of the UNIX operating systems, indeed.

But please keep in mind that we are here in a boot loader, not in a full-blown OS context.

Quoted 3 lines

* U-Boot does not support multithreading. Existing shells tend to depend strongly on this feature of the enviromnent. Many of the changes to U-Boot's hush are solely to deal with the lack of this feature.

I disagree to both parts of your statement.

Multithreading (or rather, a concept of separate processes which can eventually even run in parallel) is very nice to have, but it is not mandatory in most cases. Command pipes can almost always be strictly sequentialized and thus run in a single-task environment, too. I'm not sure, but I think I even remember pipes in the "shell" of the CPM "OS" on Z80 processors a number of decades ago...

And the fact that our current version of hush did not attempt to keep these features was much more driven by strict memory footprint limitations that anything else. I claim it would have been possible, and still is.

You also don't mention (and I guess you oversee) another critical
fact: so far, U-Boot has no concept of files. The classic design principle "everything is a file" is missing even more than the concept of processes.

Quoted 4 lines

* Existing commands do not read from stdin, nor do they print useful information to stdout. Command output is designed for human consumption and is substantially more verbose than typical unix commands.

This is a statement which is correct, but it does not contain any pro or con for the discussion here.

And if we had the features, it would be easy to fix.

Quoted 2 lines

* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely useful when working with streams are not present in U-Boot.

You are talking about OS environments here. But we are a boot loader.

Also, this argument is not fair, as the suggested LIL does not provide grep, sed, awk, sort, uniq etc. functionality either, or does it?

Quoted 1 line

And of course, this feature is currently not present in U-Boot. To get

Correct, and for very good reasons.

Quoted 11 lines

which will set my_uuid to the uuid of the selected partition. My issue with this is threefold: every command must add new syntax to do this, that syntax is inconsistent, and it prevents easy composition. Consider a script which wants to iterate over partitions. Instead of doing

	for p in $(part list mmc 0); do
		# ...
	done

it must instead do

Quoted 7 lines

part list mmc 0 partitions
for p in $partitions; do
# ...
done

which unnecessarily adds an extra step. This overhead accumulates with each command which adds something like this.

Apparently you fail to understand that this "extra step" is primarily due to the fact that we are single tasking. Even if we has command substitution (like we could have with hush) the execution sequence would be serialized in exactly the same way under the hood - it's just not as directly visible.

Quoted 2 lines

Both of these workarounds are natural consequences of using a sh-tyle shell in an environment it is not suited for. If we are going to go to

No, they are not. They are much more the consequence of no strict design guidelines for commands, so everybody implements what fits his purposes best. A cleaner approach does not require any of the additional features you've asked for - it would be possible as is.

Quoted 3 lines

the effort of porting a new language (which must be done no matter if we use Hush or some other language), we should pick one which has better support for single-threaded programming.

In which way do you think "a new language" needs to be ported when switching from an old to a new version of hush? It would be still a (mostly) POSIX compatible shell, with some restrictions.

I can fully understand that you defend your proposal, but let's be fair and stick with the facts. Thanks.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
To get something done, a committee should consist  of  no  more  than
three men, two of them absent.
#51 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

Quoted 33 lines · Wolfgang Denk

Dear Sean,

Quoted 4 lines · In message <[email protected]> you wrote:

AIUI hush has diverged significantly from what U-Boot has. This would not be an "update" moreso than a complete port in the style of the current series.

Agreed. However, as you write this it sounds like a big problem only. I disagree here - it is also a chance to do a few things different than with the original port.

Please keep in mind when this original port of the hush shell was
done: it was a time when many systems came with a total of 4 MB flash memory, and not only U-Boot, but also the Linux kernel and a ram-disk image or such had to fit into that. At that time 40 or 60 kB code size for just a fancy shell was immense!

Under such restrictions (and the need to complete the task with a given budget) many things were just commented out which from today's point of view would be nice to have.

This is not a problem of the code complexity or resources, but only of different requirements and different resources.

Quoted 5 lines

I don't think sh-style shells are a good match for U-Boot's execution environment in the first place. The fundamental idea of an sh-style shell is that the output of one command can be redirected to the input (or arguments) of another command. This cannot be done (or rather would be difficult to do) in U-Boot for a few reasons

Quoted 4 lines

There is an old saying:

The loser says: "It might be possible, but it is too difficult." The winner says: "It might be difficult, but it is possible."

And yet, the winner may also say "this is too difficult to be worth it" instead of laboring inefficiently :)

Quoted 5 lines

Apparently you take another position here than me.

First, I disagree that pipes are the "fundamental idea" of a shell. It is a fundamental idea of the UNIX operating systems, indeed.

Well, I think all programming languages at their core are about transferring information from one part of the program to another. Without pipes, how do bourne-style shells communicate? Sure you can transfer data into functions using arguments, but what about the other way around? Numeric return values? Global variables? I think these are rather anemic compared to proper return values.

Quoted 7 lines

But please keep in mind that we are here in a boot loader, not in a full-blown OS context.

Quoted 3 lines

* U-Boot does not support multithreading. Existing shells tend to depend strongly on this feature of the enviromnent. Many of the changes to U-Boot's hush are solely to deal with the lack of this feature.

Quoted 8 lines

I disagree to both parts of your statement.

Multithreading (or rather, a concept of separate processes which can eventually even run in parallel) is very nice to have, but it is not mandatory in most cases. Command pipes can almost always be strictly sequentialized and thus run in a single-task environment, too. I'm not sure, but I think I even remember pipes in the "shell" of the CPM "OS" on Z80 processors a number of decades ago...

The point here is that many Hush features were written with fork. For example, to create a new scope (such as for a subshell), Hush just fork()s. When parsing, if it encounters a construct like $(), it fork()s and then modifies the map variable, (correctly) assuming that the "original" map will remain unmodified. These things are all over and make doing a port difficult; especially when they crop up as unforseen bugs. Though I suppose the real issue here is a lack of virtual memory.

Quoted 9 lines

And the fact that our current version of hush did not attempt to keep these features was much more driven by strict memory footprint limitations that anything else. I claim it would have been possible, and still is.

You also don't mention (and I guess you oversee) another critical
fact: so far, U-Boot has no concept of files. The classic design principle "everything is a file" is missing even more than the concept of processes.

IMO "everything is a file" is more of an API thing than a shell thing. E.g. the idea that you can use open/read/write/close to access any resource. But without pipes, I don't think such an abstraction is very useful.

Quoted 7 lines
Quoted 4 lines

* Existing commands do not read from stdin, nor do they print useful information to stdout. Command output is designed for human consumption and is substantially more verbose than typical unix commands.

This is a statement which is correct, but it does not contain any pro or con for the discussion here.

In order for pipes to be useful, programs should generally follow the Unix philosophy. In particular, many commands in U-Boot fall afoul of rule 2:

Quoted 4 lines

Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.

Which makes it difficult to compose them.

Quoted 1 line

And if we had the features, it would be easy to fix.

I don't know about that. I think making U-Boot commands emit output designed for machine consumption would require substantial effort.

Quoted 5 lines
Quoted 2 lines

* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely useful when working with streams are not present in U-Boot.

You are talking about OS environments here. But we are a boot loader.

These programs are just as much part of the shell as the builtin commands. They allow one to take input from one program and transform it into the appropriate input to another program. This is especially important for extracting information from programs which were not designed to interface well.

Quoted 3 lines

Also, this argument is not fair, as the suggested LIL does not provide grep, sed, awk, sort, uniq etc. functionality either, or does it?

Of course it is fair. These programs are just as much part of the *nix programming environment as the bourne shells which invoke them.

Quoted 21 lines
Quoted 2 lines

And of course, this feature is currently not present in U-Boot. To get

Correct, and for very good reasons.

Quoted 16 lines

which will set my_uuid to the uuid of the selected partition. My issue with this is threefold: every command must add new syntax to do this, that syntax is inconsistent, and it prevents easy composition. Consider a script which wants to iterate over partitions. Instead of doing

	for p in $(part list mmc 0); do
		# ...
	done

it must instead do

part list mmc 0 partitions
for p in $partitions; do
# ...
done

Quoted 8 lines
Quoted 2 lines

which unnecessarily adds an extra step. This overhead accumulates with each command which adds something like this.

Apparently you fail to understand that this "extra step" is primarily due to the fact that we are single tasking. Even if we has command substitution (like we could have with hush) the execution sequence would be serialized in exactly the same way under the hood - it's just not as directly visible.

Please don't assume what I do and do not understand ;)

My point here is that instead of having a consistent interface for passing data from a command to its caller, we instead have an inconsistent one which is different for each command.

Quoted 7 lines
Quoted 2 lines

Both of these workarounds are natural consequences of using a sh-tyle shell in an environment it is not suited for. If we are going to go to

No, they are not. They are much more the consequence of no strict design guidelines for commands, so everybody implements what fits his purposes best. A cleaner approach does not require any of the additional features you've asked for - it would be possible as is.

So you would have every command have a new parameter at the end specifying which environmental variable to put a return value in? IMO such things should be done at the language level so that commands can just call a function like set_result() and let the shell handle the rest.

Quoted 7 lines
Quoted 3 lines

the effort of porting a new language (which must be done no matter if we use Hush or some other language), we should pick one which has better support for single-threaded programming.

In which way do you think "a new language" needs to be ported when switching from an old to a new version of hush? It would be still a (mostly) POSIX compatible shell, with some restrictions.

Modern Hush is completely unrecognizable compared to what we have in U-Boot today. Any "updating" effort would be akin to going over the entire upstream codebase and porting it from scratch.

Quoted 2 lines

I can fully understand that you defend your proposal, but let's be fair and stick with the facts. Thanks.

And I'd appreciate if we could presume good faith of our peers.

--Sean

#52 lore raw

Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup

Dear Sean,

Quoted 5 lines · In message <[email protected]> you wrote:

Without pipes, how do bourne-style shells communicate? Sure you can transfer data into functions using arguments, but what about the other way around? Numeric return values? Global variables? I think these are rather anemic compared to proper return values.

Pipes are basically just a shortcut to avoid buffer space and to make scripting easier and more elegant.

For example:

	$ ps aux | grep foo | grep -v grep | awk '{print $2}' | xargs kill

can be written without use of pipes as:

	$ ps aux >/tmp/$$.1 ; \
	  grep foo </tmp/$$.1 >/tmp/$$.2 ; \
	  grep -v grep </tmp/$$.2 >/tmp/$$.1 ; \
	  awk '{print $2}' </tmp/$$.1 >/tmp/$$.2  ; \
	  xargs kill </tmp/$$.2 ; \
	  rm -f /tmp/$$.1 /tmp/$$.2

[and of course this could be done in many other ways, too, cleaner and more efficient]

The point is: Pipes are just a shortcut. You can do all the same
using files - of course with different performance and resource usage.

Quoted 3 lines

The point here is that many Hush features were written with fork. For example, to create a new scope (such as for a subshell), Hush just fork()s. When parsing, if it encounters a construct like $(), it fork()s

Yes, of course I am fully aware of this. This is one of the reasons why command substitution was excluded from the inital port of hush to U-Boot - but again: command sustitution is just a shortcut; it is simple enough to write scripts without it, and it is certainly possible to implent this feature in a U-Boot context as well.

The reason it was not done 18 years ago is 1) typical systems at that time did not have the resources (especially not enough ROM for a bigger U-Boot image), and 2) having a shell with basic scripting capabilities was sheer extravagance in a boot loader.

Yes, the situation has changed since then; today a U-Boot image is as big as a v2.2 Linux kernel image was then, and nobody cares.

Quoted 4 lines

and then modifies the map variable, (correctly) assuming that the "original" map will remain unmodified. These things are all over and make doing a port difficult; especially when they crop up as unforseen bugs. Though I suppose the real issue here is a lack of virtual memory.

Not really. At the time of the port the concern was about code size (size of the text segment).

Quoted 4 lines

IMO "everything is a file" is more of an API thing than a shell thing. E.g. the idea that you can use open/read/write/close to access any resource. But without pipes, I don't think such an abstraction is very useful.

Pipes are just a notation to allow for a convenient use of coroutines, even from a command line interface. They make you think of concurrent processing, but (on single core machines - which was all they had at the time this was invented) it was just an emulation of concurrency on a strictly sequential system.

Nothing really fancy at all, once you have the idea. And nothing that could not be emulated in a U-Boot environment, either.

The question is: how urgently do we need it?

My experience is that if anything is _really_ needed, someone will show up and implement it, probably funded by some customer who needs this feature for some product. I have yet to see any such urgent need for pipes in U-Boot.

Quoted 3 lines
Quoted 3 lines

Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats.

That's the theory, yes. In reality, things are a bit different. Look at the output format of classic Unix commands: ls, ps, df, ...

Quoted 2 lines

I don't know about that. I think making U-Boot commands emit output designed for machine consumption would require substantial effort.

Unix commands usually have NOT been designed for "machine consumption", on contrary. Even network protocols, log files atc. were intentionally designed to be humanly readable.

It's the flexibility of the tools which allows for clever usage.

In U-Boot, commands could be cleaned up for better use in pipes - assuming we had those. But there would be no need to do this ina single huge action - it could be done step by step, one command at a time when somebody needs to use it in such a way.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely useful when working with streams are not present in U-Boot.

You are talking about OS environments here. But we are a boot loader.

These programs are just as much part of the shell as the builtin commands.

Sorry, but they are definitely NOT part of the _shell_.

They are part of the standard Unix toolbox for basic text processing, but for many good reasons each of these is a separate program, and the shell knows nothing of these.

Quoted 7 lines
Quoted 3 lines

In which way do you think "a new language" needs to be ported when switching from an old to a new version of hush? It would be still a (mostly) POSIX compatible shell, with some restrictions.

Modern Hush is completely unrecognizable compared to what we have in U-Boot today. Any "updating" effort would be akin to going over the entire upstream codebase and porting it from scratch.

Agreed. But this means porting some code, which still implements the very same language (i. e. "shell"). There would be no new language in this case - just a bigger subset, less restrictions, more options, probably.

Best regards,

Wolfgang Denk

Signature · 4 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
God made machine language; all the rest is the work of man.
#53 lore raw

RFC PATCH 4/28 cli: lil: Remove most functions by default

In reply to Sean Anderson

This helps reduce the size impact of LIL to approximately that of hush. The builtin functions have been chosen to roughly match the functionality present in hush. In addition, arithmetic is removed from expr to reduce size further.

---

 cmd/Kconfig      | 10 ++++++
 common/cli_lil.c | 83 +++++++++++++++++++++++++++---------------------
 2 files changed, 57 insertions(+), 36 deletions(-)
cmd/Kconfig +10 -0
index 8bccc572af..0a7b73cb6d 100644
@@ -32,6 +32,16 @@ config LIL
 	  like `proc name {args} {body}' functions or `echo [some command]`
 	  command substitution ("tcl scripts").
 
+if LIL
+
+config LIL_FULL
+	bool "Enable all LIL features"
+	help
+	  This enables all LIL builtin functions, as well as expression support
+	  for arithmetic and bitwise operations.
+
+endif
+
 config CMDLINE_EDITING
 	bool "Enable command line editing"
 	depends on CMDLINE
common/cli_lil.c +47 -36

135 lines not on this page · raw message

Signature · 1 line
2.32.0
Reviewed-by on v1 4/28
#54 lore raw

Re: [RFC PATCH 04/28] cli: lil: Remove most functions by default

Quoted 6 lines · Sean Anderson

This helps reduce the size impact of LIL to approximately that of hush. The builtin functions have been chosen to roughly match the functionality present in hush. In addition, arithmetic is removed from expr to reduce size further.

Quoted 6 lines

---

 cmd/Kconfig      | 10 ++++++
 common/cli_lil.c | 83 +++++++++++++++++++++++++++---------------------
 2 files changed, 57 insertions(+), 36 deletions(-)
#55 lore raw

RFC PATCH 5/28 cli: lil: Rename some functions to be more like TCL

In reply to Sean Anderson

Several functions have different names than they do in TCL. To make things easier for those familiar with TCL, rename them to their TCL equivalents. At the moment, this is only done for functions not used by LIL_FULL. Some functions need more substantive work to conform them to TCL. For example, in TCL, there is a string function with a subcommand of compare, which is the same as the top-level function compare. Several functions also have no TCL equivalent. Do we need these?

TODO: do this for all functions

---

 common/cli_lil.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)
common/cli_lil.c +14 -14
index a2e5cdf35a..a6c77ee19c 100644
@@ -1829,10 +1829,10 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
 		return lil_clone_value(func->code);
 	}
 
-	if (!strcmp(type, "func-count"))
+	if (!strcmp(type, "proc-count"))
 		return lil_alloc_integer(lil->cmds);
 
-	if (!strcmp(type, "funcs")) {
+	if (!strcmp(type, "procs")) {
 		struct lil_list *funcs = lil_alloc_list();
 
 		for (i = 0; i < lil->cmds; i++)
@@ -1878,7 +1878,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
 		return r;
 	}
 
-	if (!strcmp(type, "has-func")) {
+	if (!strcmp(type, "has-proc")) {
 		const char *target;
 
 		if (argc == 1)
@@ -1948,7 +1948,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
 	return NULL;
 }
 
-static struct lil_value *fnc_func(struct lil *lil, size_t argc,
+static struct lil_value *fnc_proc(struct lil *lil, size_t argc,
 				  struct lil_value **argv)
 {
 	struct lil_value *name;
@@ -2472,8 +2472,8 @@ static struct lil_value *real_inc(struct lil *lil, const char *varname,
 	return pv;
 }
 
-static struct lil_value *fnc_inc(struct lil *lil, size_t argc,
-				 struct lil_value **argv)
+static struct lil_value *fnc_incr(struct lil *lil, size_t argc,
+				  struct lil_value **argv)
 {
 	if (argc < 1)
 		return NULL;
@@ -2482,8 +2482,8 @@ static struct lil_value *fnc_inc(struct lil *lil, size_t argc,
 			argc > 1 ? lil_to_integer(argv[1]) : 1);
 }
 
-static struct lil_value *fnc_dec(struct lil *lil, size_t argc,
-				 struct lil_value **argv)
+static struct lil_value *fnc_decr(struct lil *lil, size_t argc,
+				  struct lil_value **argv)
 {
 	if (argc < 1)
 		return NULL;
@@ -2786,8 +2786,8 @@ static struct lil_value *fnc_rtrim(struct lil *lil, size_t argc,
 	return real_trim(lil_to_string(argv[0]), chars, 0, 1);
 }
 
-static struct lil_value *fnc_strcmp(struct lil *lil, size_t argc,
-				    struct lil_value **argv)
+static struct lil_value *fnc_compare(struct lil *lil, size_t argc,
+				     struct lil_value **argv)
 {
 	if (argc < 2)
 		return NULL;
@@ -2940,18 +2940,18 @@ static struct lil_value *fnc_lmap(struct lil *lil, size_t argc,
 
 static void register_stdcmds(struct lil *lil)
 {
-	lil_register(lil, "dec", fnc_dec);
+	lil_register(lil, "decr", fnc_decr);
 	lil_register(lil, "eval", fnc_eval);
 	lil_register(lil, "expr", fnc_expr);
 	lil_register(lil, "for", fnc_for);
 	lil_register(lil, "foreach", fnc_foreach);
-	lil_register(lil, "func", fnc_func);
+	lil_register(lil, "proc", fnc_proc);
 	lil_register(lil, "if", fnc_if);
-	lil_register(lil, "inc", fnc_inc);
+	lil_register(lil, "incr", fnc_incr);
 	lil_register(lil, "local", fnc_local);
 	lil_register(lil, "return", fnc_return);
 	lil_register(lil, "set", fnc_set);
-	lil_register(lil, "strcmp", fnc_strcmp);
+	lil_register(lil, "compare", fnc_compare);
 	lil_register(lil, "try", fnc_try);
 	lil_register(lil, "while", fnc_while);
 
Signature · 1 line
2.32.0
#56 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

Hi Sean,

Quoted 9 lines · Sean Anderson

Several functions have different names than they do in TCL. To make things easier for those familiar with TCL, rename them to their TCL equivalents. At the moment, this is only done for functions not used by LIL_FULL. Some functions need more substantive work to conform them to TCL. For example, in TCL, there is a string function with a subcommand of compare, which is the same as the top-level function compare. Several functions also have no TCL equivalent. Do we need these?

Quoted 7 lines

TODO: do this for all functions

---

 common/cli_lil.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

Is your intent to create a fork of this in U-Boot? Could we not update things upstream, at least as an option, to avoid carrying these patches?

Regards,
Simon

#57 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

Quoted 14 lines · Simon Glass

Hi Sean,

Quoted 11 lines · Sean Anderson

Several functions have different names than they do in TCL. To make things easier for those familiar with TCL, rename them to their TCL equivalents. At the moment, this is only done for functions not used by LIL_FULL. Some functions need more substantive work to conform them to TCL. For example, in TCL, there is a string function with a subcommand of compare, which is the same as the top-level function compare. Several functions also have no TCL equivalent. Do we need these?

TODO: do this for all functions

Quoted 7 lines
Quoted 5 lines

---

  common/cli_lil.c | 28 ++++++++++++++--------------
  1 file changed, 14 insertions(+), 14 deletions(-)

Is your intent to create a fork of this in U-Boot?

Yes. I believe some of the major additions I have made (especially "[RFC PATCH 21/28] cli: lil: Add a distinct parsing step") would not be accepted by upstream.

Quoted 2 lines

Could we not update things upstream, at least as an option, to avoid carrying these patches?

For some of the smaller patches, that may be possible. However, I am not a fan of the major amount of ifdefs that Hush has. For something as core as a shell, I think we should be free to make changes as we see fit without worrying about how it will affect a hypothetical backport.

For this patch in particular, I believe upstream would no accept it because it would break backwards compatibility for existing LIL users. However, I view compatibility with TCL as a whole more valuble than compatibility with LIL.

--Sean

#58 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

Dear Sean,

Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 2 lines

Is your intent to create a fork of this in U-Boot?

Yes. I believe some of the major additions I have made (especially "[RFC PATCH 21/28] cli: lil: Add a distinct parsing step") would not be accepted by upstream.

Ouch...

Quoted 7 lines
Quoted 2 lines

Could we not update things upstream, at least as an option, to avoid carrying these patches?

For some of the smaller patches, that may be possible. However, I am not a fan of the major amount of ifdefs that Hush has. For something as core as a shell, I think we should be free to make changes as we see fit without worrying about how it will affect a hypothetical backport.

I'm afraind I cannot understand your thinking.

You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream, but cannot be easily fixed in U-Boot because we essentially created an unmaintained fork - and as a cure, you recommend to do the same thing again, but this time intentionally and deliberately?

If you had not apparently already invested a lot of effort into this thing I would assume you must be joking...

To me such an approach is unacceptable.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
If a train station is a place where a train stops,
                                           then what's a workstation?
#59 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

Quoted 29 lines · Wolfgang Denk

Dear Sean,

Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 2 lines

Is your intent to create a fork of this in U-Boot?

Yes. I believe some of the major additions I have made (especially "[RFC PATCH 21/28] cli: lil: Add a distinct parsing step") would not be accepted by upstream.

Ouch...

Quoted 7 lines
Quoted 2 lines

Could we not update things upstream, at least as an option, to avoid carrying these patches?

For some of the smaller patches, that may be possible. However, I am not a fan of the major amount of ifdefs that Hush has. For something as core as a shell, I think we should be free to make changes as we see fit without worrying about how it will affect a hypothetical backport.

I'm afraind I cannot understand your thinking.

You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream, but cannot be easily fixed in U-Boot because we essentially created an unmaintained fork - and as a cure, you recommend to do the same thing again, but this time intentionally and deliberately?

Quoted 4 lines

If you had not apparently already invested a lot of effort into this thing I would assume you must be joking...

To me such an approach is unacceptable.

I think I want to try and address this. While with "hush" we have something that's in heavy active development outside of U-Boot, with LIL we have something that's mature and "done". Tracking an active outside development is HARD and requires constant resync. Look at the last few LIL releases. That could be easily re-worked in to our fork if needed. I see that as a positive, not a negative.

Signature · 1 line
Tom
#60 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

Hi Tom,

Quoted 34 lines · Tom Rini
Quoted 32 lines · Wolfgang Denk

Dear Sean,

Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 2 lines

Is your intent to create a fork of this in U-Boot?

Yes. I believe some of the major additions I have made (especially "[RFC PATCH 21/28] cli: lil: Add a distinct parsing step") would not be accepted by upstream.

Ouch...

Quoted 7 lines
Quoted 2 lines

Could we not update things upstream, at least as an option, to avoid carrying these patches?

For some of the smaller patches, that may be possible. However, I am not a fan of the major amount of ifdefs that Hush has. For something as core as a shell, I think we should be free to make changes as we see fit without worrying about how it will affect a hypothetical backport.

I'm afraind I cannot understand your thinking.

You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream, but cannot be easily fixed in U-Boot because we essentially created an unmaintained fork - and as a cure, you recommend to do the same thing again, but this time intentionally and deliberately?

If you had not apparently already invested a lot of effort into this thing I would assume you must be joking...

Quoted 8 lines
Quoted 1 line

To me such an approach is unacceptable.

I think I want to try and address this. While with "hush" we have something that's in heavy active development outside of U-Boot, with LIL we have something that's mature and "done". Tracking an active outside development is HARD and requires constant resync. Look at the last few LIL releases. That could be easily re-worked in to our fork if needed. I see that as a positive, not a negative.

Yes I wondered about that, since hush has been in busybox, an active project, all these years. I think this is a good point and perhaps means that forking it should not be too much of a concern. Still, if we can send some things upstream, we should.

Regards,
Simon

#61 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

Quoted 46 lines · Simon Glass

Hi Tom,

Quoted 34 lines · Tom Rini
Quoted 32 lines · Wolfgang Denk

Dear Sean,

Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 2 lines

Is your intent to create a fork of this in U-Boot?

Yes. I believe some of the major additions I have made (especially "[RFC PATCH 21/28] cli: lil: Add a distinct parsing step") would not be accepted by upstream.

Ouch...

Quoted 7 lines
Quoted 2 lines

Could we not update things upstream, at least as an option, to avoid carrying these patches?

For some of the smaller patches, that may be possible. However, I am not a fan of the major amount of ifdefs that Hush has. For something as core as a shell, I think we should be free to make changes as we see fit without worrying about how it will affect a hypothetical backport.

I'm afraind I cannot understand your thinking.

You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream, but cannot be easily fixed in U-Boot because we essentially created an unmaintained fork - and as a cure, you recommend to do the same thing again, but this time intentionally and deliberately?

If you had not apparently already invested a lot of effort into this thing I would assume you must be joking...

Quoted 8 lines
Quoted 1 line

To me such an approach is unacceptable.

I think I want to try and address this. While with "hush" we have something that's in heavy active development outside of U-Boot, with LIL we have something that's mature and "done". Tracking an active outside development is HARD and requires constant resync. Look at the last few LIL releases. That could be easily re-worked in to our fork if needed. I see that as a positive, not a negative.

Quoted 4 lines

Yes I wondered about that, since hush has been in busybox, an active project, all these years. I think this is a good point and perhaps means that forking it should not be too much of a concern. Still, if we can send some things upstream, we should.

Well, that's part of the problem. If we re-port modern hush, who is going to keep it in sync, and how often? Yes, there will be changes we don't need, but there will be bugfixes we do as well.

Signature · 1 line
Tom
#62 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

In reply to Tom Rini

Dear Tom,

Quoted 4 lines · In message <20210705185141.GA9516@bill-the-cat> you wrote:

I think I want to try and address this. While with "hush" we have something that's in heavy active development outside of U-Boot, with LIL we have something that's mature and "done".

Mature? And still without consequent error checking? And done, i. e. this will never be fixed?

Quoted 2 lines

Tracking an active outside development is HARD and requires constant resync.

Based on that logic we should stop supporting Linux, and stop using DTs or file systems - all of these are under "active outside development".

This is a bogus argiment.

Quoted 3 lines

Look at the last few
LIL releases. That could be easily re-worked in to our fork if needed. I see that as a positive, not a negative.

OK, this is your opinion. Mine differs.

Please consider this as a full NAK from me when when you think of it as a _replacement_ (even an optional one) of the standard shell. If you like, have it added as an _additional_ command, of course fully optional and without impact on the rest of U-Boot if not intentionally selected.

Best regards,

Wolfgang Denk

Signature · 4 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
Of all the things I've lost, I miss my mind the most.
#63 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

Hi Wolfgang,

Quoted 27 lines · Wolfgang Denk

Dear Tom,

Quoted 4 lines · In message <20210705185141.GA9516@bill-the-cat> you wrote:

I think I want to try and address this. While with "hush" we have something that's in heavy active development outside of U-Boot, with LIL we have something that's mature and "done".

Mature? And still without consequent error checking? And done, i. e. this will never be fixed?

Quoted 2 lines

Tracking an active outside development is HARD and requires constant resync.

Based on that logic we should stop supporting Linux, and stop using DTs or file systems - all of these are under "active outside development".

This is a bogus argiment.

Quoted 3 lines

Look at the last few
LIL releases. That could be easily re-worked in to our fork if needed. I see that as a positive, not a negative.

Quoted 7 lines

OK, this is your opinion. Mine differs.

Please consider this as a full NAK from me when when you think of it as a _replacement_ (even an optional one) of the standard shell. If you like, have it added as an _additional_ command, of course fully optional and without impact on the rest of U-Boot if not intentionally selected.

That seems reasonable to me. There is certainly interest in these threads in reporting hush, perhaps with a goal of upstreaming the U-Boot-specific changes back to busybox. But so far no one has done it...

Regards,
Simon

#64 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

In reply to Wolfgang Denk
Quoted 10 lines · Wolfgang Denk

Dear Tom,

Quoted 4 lines · In message <20210705185141.GA9516@bill-the-cat> you wrote:

I think I want to try and address this. While with "hush" we have something that's in heavy active development outside of U-Boot, with LIL we have something that's mature and "done".

Mature? And still without consequent error checking? And done, i. e. this will never be fixed?

Intentional design by upstream, and then for the actual problem part (error checking, test suite), Sean is saying he'll fix it, and has started on it.

Quoted 8 lines
Quoted 2 lines

Tracking an active outside development is HARD and requires constant resync.

Based on that logic we should stop supporting Linux, and stop using DTs or file systems - all of these are under "active outside development".

This is a bogus argiment.

Alright, go and update our Kbuild integration to be in sync with current Linux. Or just v5.0. I'd suggest picking up the metadata_csum support patch but someone did say they'd try and update that (off-list) so that ext4 filesystem made with default options in the last 5 years (not exaggerating) work. I'm not even going to talk about the various level of out of sync our DTs files are but I do have some hope there will be more re-syncs especially as it comes to more light that the DT U-Boot uses can just be the DT the OS gets. Then tell me it's easy to keep this stuff in sync. Because we certainly don't want to just fork-and-forget again, right? Because that's why we have both crazy parse bugs as well as lack of now old-and-expected features.

OK, snark aside, I'm very serious here, any "we'll just import ..." needs to have a plan to keep it up to date, or be easy enough to do such that I can set a monthly reminder to check for and do the update. Every area where we don't do this is a set of problems waiting to get worse, as we can see with the hush shell right now as it's one of the oldest things we stopped syncing with.

Quoted 11 lines
Quoted 3 lines

Look at the last few
LIL releases. That could be easily re-worked in to our fork if needed. I see that as a positive, not a negative.

OK, this is your opinion. Mine differs.

Please consider this as a full NAK from me when when you think of it as a _replacement_ (even an optional one) of the standard shell. If you like, have it added as an _additional_ command, of course fully optional and without impact on the rest of U-Boot if not intentionally selected.

Any new cli for U-Boot won't be a default build as a command for a long while after it's merged. I'm not sure how much past that point it would further need to be to become the default cli. And I'll even repeat what I said elsewhere about how an sh syntax is a good user feature. But I really think we want a shell environment that is not actively adding new features is a good thing, for the default. Just how much stuff should we be doing or need to be doing before we hand things over to the OS?

Signature · 1 line
Tom
#65 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

Quoted 6 lines · Tom Rini
Quoted 2 lines

Mature? And still without consequent error checking? And done, i. e. this will never be fixed?

Intentional design by upstream, and then for the actual problem part (error checking, test suite), Sean is saying he'll fix it, and has started on it.

To clarify, the intentional design aspect is for the language (e.g. giving garbage to "expr"). The library does need better memory alloc checks (this is the main thing missing because i never had to use LIL on an environment without virtual memory... or where malloc would ever practically fail), though IIRC there aren't any other cases where failure could happen and isn't checked. Similarly i do plan on implementing a proper test suite (currently there is a shell script that runs all the examples in the source code - many examples were made to expose previous bugs - with two different implementations which can be used to find regressions in subsequent releases and differences between implementations, but the examples do not exhaust the code paths the parser could take) at some point - the latter is more likely to happen soon than the former.

The same applies with LIL being "done" (well, mostly done) - this is about the language, not the implementation which (as i mentioned in another email) does need improvements, especially around performance (which is also a reason i need to implement better automated tests).

Kostas

#66 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

In reply to Tom Rini

Dear Tom,

Quoted 7 lines · In message <20210706153327.GS9516@bill-the-cat> you wrote:
Quoted 3 lines

Mature? And still without consequent error checking? And done, i. e. this will never be fixed?

Intentional design by upstream, and then for the actual problem part (error checking, test suite), Sean is saying he'll fix it, and has started on it.

Seriously - any piece of software that omits error checking intentionally be design should be indented six feet downward and covered with dirt. We should not even consider looking at it.

Quoted 6 lines

OK, snark aside, I'm very serious here, any "we'll just import ..." needs to have a plan to keep it up to date, or be easy enough to do such that I can set a monthly reminder to check for and do the update. Every area where we don't do this is a set of problems waiting to get worse, as we can see with the hush shell right now as it's one of the oldest things we stopped syncing with.

Which exact _new_ problems do we see with hush right now? I can only see old ones, that have been known (and worked around) for nearly two decades.

The limitations and bugs have all been there since the beginning - the limitations actually being intentional due to the typical resource situation at that time.

Quoted 4 lines

But I
really think we want a shell environment that is not actively adding new features is a good thing, for the default. Just how much stuff should we be doing or need to be doing before we hand things over to the OS?

You are shooting yourself in the knee here.

If you think out CLI should not be adding new features, then we should just stick with our ancient hush and neither update it nor replace it with something else that adds not only new features, but breaks backward compatibility, hard.

Best regards,

Wolfgang Denk

Signature · 6 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
God runs electromagnetics by wave theory on  Monday,  Wednesday,  and
Friday,  and the Devil runs them by quantum theory on Tuesday, Thurs-
day, and Saturday.                                   -- William Bragg
#67 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

Quoted 22 lines · Wolfgang Denk

Dear Tom,

Quoted 7 lines · In message <20210706153327.GS9516@bill-the-cat> you wrote:
Quoted 3 lines

Mature? And still without consequent error checking? And done, i. e. this will never be fixed?

Intentional design by upstream, and then for the actual problem part (error checking, test suite), Sean is saying he'll fix it, and has started on it.

Seriously - any piece of software that omits error checking intentionally be design should be indented six feet downward and covered with dirt. We should not even consider looking at it.

Quoted 6 lines

OK, snark aside, I'm very serious here, any "we'll just import ..." needs to have a plan to keep it up to date, or be easy enough to do such that I can set a monthly reminder to check for and do the update. Every area where we don't do this is a set of problems waiting to get worse, as we can see with the hush shell right now as it's one of the oldest things we stopped syncing with.

Quoted 7 lines

Which exact _new_ problems do we see with hush right now? I can only see old ones, that have been known (and worked around) for nearly two decades.

The limitations and bugs have all been there since the beginning - the limitations actually being intentional due to the typical resource situation at that time.

There's all of the new features that've been suggested for our current hush, many by Sean, for which the reply has been "our hush is old, it should be updated!".

And you didn't address my point, taking code from another project requires dedicated maintenance to keep it up to date and we do have a lot of in-tree examples right now of where that lack of dedicated maintenance is a problem.

Quoted 11 lines
Quoted 4 lines

But I
really think we want a shell environment that is not actively adding new features is a good thing, for the default. Just how much stuff should we be doing or need to be doing before we hand things over to the OS?

You are shooting yourself in the knee here.

If you think out CLI should not be adding new features, then we should just stick with our ancient hush and neither update it nor replace it with something else that adds not only new features, but breaks backward compatibility, hard.

Honestly, adding new features to the CLI (which is NOT the same as adding new commands, or enhancing existing commands) is very low on my priority list. We need to get the Kconfig migration done. We need to get DM migration done (which will help with the Kconfig one by showing hardware no one cares for anymore, and reducing symbols). We need to make adding new hardware easier. We need to get DTS files in-sync more often.

There are very important developer use cases for a more flexible CLI interpreter. But the main use case is still "boot the (redundant) OS ASAP".

So, what is my priority on this series right here? Well, a developer has posted a series. They believe it's useful and addressing a long-standing problem. They want feedback. I'm trying to provide feedback as it's to a general area of the codebase.

Signature · 1 line
Tom
#68 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

In reply to Wolfgang Denk
Quoted 11 lines · Wolfgang Denk

Dear Sean,

Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 2 lines

Is your intent to create a fork of this in U-Boot?

Yes. I believe some of the major additions I have made (especially "[RFC PATCH 21/28] cli: lil: Add a distinct parsing step") would not be accepted by upstream.

Ouch...

I don't particularly mind if Kostas doesn't view these patches as good additions to upstream. We have different goals and requirements, and so not all changes will be compatible.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Could we not update things upstream, at least as an option, to avoid carrying these patches?

For some of the smaller patches, that may be possible. However, I am not a fan of the major amount of ifdefs that Hush has. For something as core as a shell, I think we should be free to make changes as we see fit without worrying about how it will affect a hypothetical backport.

Quoted 4 lines

I'm afraind I cannot understand your thinking.

You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream,

The bugs are fairly minor. The particular characteristics of Hush have not changed. These characteristics make Hush difficult to adapt to the limitations of U-Boot. When we cannot support the basic abstractions expected by Hush, the shell will necessarily change for the worse.

Quoted 1 line

but cannot be easily fixed in U-Boot

Because they are core to the design of Hush (and other bourne derived shells).

Quoted 1 line

because we essentially created an unmaintained fork

I plan to maintain this fork.

--Sean

Quoted 11 lines

- and as a cure, you recommend to do the same
thing again, but this time intentionally and deliberately? If you had not apparently already invested a lot of effort into this thing I would assume you must be joking...

To me such an approach is unacceptable.

Best regards,

Wolfgang Denk

#69 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

Dear Sean,

Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 3 lines

You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream,

The bugs are fairly minor. The particular characteristics of Hush have not changed. These characteristics make Hush difficult to adapt to the limitations of U-Boot. When we cannot support the basic abstractions expected by Hush, the shell will necessarily change for the worse.

This is not true. Just have a look what hush in a recent version of Busybox offers.

Quoted 4 lines
Quoted 1 line

but cannot be easily fixed in U-Boot

Because they are core to the design of Hush (and other bourne derived shells).

Oh, this is an interesting opinion. I doubt if a majority (or even a significant percentage) of U-Boot users share it. If you were right, there would be far less users of bash (or other "bourne derived shells"). Guess which percentage of users of UNIX operating systems is using a Tcl based command interpreter as their login shell?

Best regards,

Wolfgang Denk

Signature · 6 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
"There are three principal ways to lose money: wine, women, and engi-
neers. While the first two are more pleasant, the third is by far the
more certain."                           - Baron Rothschild, ca. 1800
#70 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

Quoted 7 lines · Wolfgang Denk

Dear Sean,

Quoted 4 lines · In message <[email protected]> you wrote:
Quoted 3 lines

You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream,

Quoted 7 lines
Quoted 4 lines

The bugs are fairly minor. The particular characteristics of Hush have not changed. These characteristics make Hush difficult to adapt to the limitations of U-Boot. When we cannot support the basic abstractions expected by Hush, the shell will necessarily change for the worse.

This is not true. Just have a look what hush in a recent version of Busybox offers.

Busybox runs in a Linux environment. Many of its features rely on the core functionality provided by Linux, which we do not provide in U-Boot. This is what makes porting features difficult.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

but cannot be easily fixed in U-Boot

Because they are core to the design of Hush (and other bourne derived shells).

Quoted 6 lines

Oh, this is an interesting opinion. I doubt if a majority (or even a significant percentage) of U-Boot users share it. If you were right, there would be far less users of bash (or other "bourne derived shells"). Guess which percentage of users of UNIX operating systems is using a Tcl based command interpreter as their login shell?

And yet, this is not the field we compete in. While bourne-style shells can take advantage of a multi-threaded environment, embedded shells tend to implement a much wider set of languages. See [1] for a survey of examples.

--Sean

[1] https://github.com/dbohdan/embedded-scripting-languages

#71 lore raw

Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL

Dear Sean,

Quoted 1 line · In message <[email protected]> you wrote:

...

Quoted 6 lines

And yet, this is not the field we compete in. While bourne-style shells can take advantage of a multi-threaded environment, embedded shells tend to implement a much wider set of languages. See [1] for a survey of examples.

[1] https://github.com/dbohdan/embedded-scripting-languages

"Embedded scripting languages" is probably not a good match for what we are discussing. I don't think this comparison was made for restricted boot loader environments, but for use in general purpose / embedded operating systems.

Hey, they even list Forth there. Maybe somebody should port OpenBoot, so we can have a Forth interpreter as new commandline language? We could implement device trees in the old, traditional way then, too :-)

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
There are bugs and then there are bugs.  And then there are bugs.
                                                    - Karl Lehenbauer
#72 lore raw

RFC PATCH 6/28 cli: lil: Convert some defines to enums

In reply to Sean Anderson

This converts some defines to enums. This allows them to be documented more easily, and helps the compiler give better errors for switch statements.

---

 common/cli_lil.c  | 44 +++++++++++++++++++++++---------------------
 include/cli_lil.h | 12 +++++++-----
 2 files changed, 30 insertions(+), 26 deletions(-)
common/cli_lil.c +23 -21
index a6c77ee19c..3d1e6181f9 100644
@@ -24,10 +24,6 @@
  * overflows and is also useful when running through an automated fuzzer like AFL */
 /*#define LIL_ENABLE_RECLIMIT 10000*/
 
-#define ERROR_NOERROR 0
-#define ERROR_DEFAULT 1
-#define ERROR_FIXHEAD 2
-
 #define CALLBACKS 8
 #define HASHMAP_CELLS 256
 #define HASHMAP_CELLMASK 0xFF
@@ -97,7 +93,11 @@ struct lil {
 	struct lil_env *rootenv;
 	struct lil_env *downenv;
 	struct lil_value *empty;
-	int error;
+	enum {
+		ERROR_NOERROR = 0,
+		ERROR_DEFAULT,
+		ERROR_FIXHEAD,
+	} error;
 	size_t err_head;
 	char *err_msg;
 	lil_callback_proc_t callback[CALLBACKS];
@@ -108,7 +108,12 @@ struct expreval {
 	const char *code;
 	size_t len, head;
 	ssize_t ival;
-	int error;
+	enum {
+		EERR_NO_ERROR = 0,
+		EERR_SYNTAX_ERROR,
+		EERR_DIVISION_BY_ZERO,
+		EERR_INVALID_EXPRESSION,
+	} error;
 };
 
 static struct lil_value *next_word(struct lil *lil);
@@ -645,7 +650,7 @@ int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc)
 }
 
 struct lil_var *lil_set_var(struct lil *lil, const char *name,
-			    struct lil_value *val, int local)
+			    struct lil_value *val, enum lil_setvar local)
 {
 	struct lil_var **nvar;
 	struct lil_env *env =
@@ -1222,11 +1227,6 @@ int lil_error(struct lil *lil, const char **msg, size_t *pos)
 	return 1;
 }
 
-#define EERR_NO_ERROR 0
-#define EERR_SYNTAX_ERROR 1
-#define EERR_DIVISION_BY_ZERO 3
-#define EERR_INVALID_EXPRESSION 4
-
 static void ee_expr(struct expreval *ee);
 
 static int ee_invalidpunct(int ch)
@@ -1673,16 +1673,18 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
 
 	ee_expr(&ee);
 	lil_free_value(code);
-	if (ee.error) {
-		switch (ee.error) {
-		case EERR_DIVISION_BY_ZERO:
-			lil_set_error(lil, "division by zero in expression");
-			break;
-		case EERR_SYNTAX_ERROR:
-			lil_set_error(lil, "expression syntax error");
-			break;
-		}
+	switch (ee.error) {
+	case EERR_DIVISION_BY_ZERO:
+		lil_set_error(lil, "division by zero in expression");
 		return NULL;
+	case EERR_SYNTAX_ERROR:
+		lil_set_error(lil, "expression syntax error");
+		return NULL;
+	case EERR_INVALID_EXPRESSION:
+		lil_set_error(lil, "invalid expression");
+		return NULL;
+	case EERR_NO_ERROR:
+		break;
 	}
 	return lil_alloc_integer(ee.ival);
 }
include/cli_lil.h +7 -5
index c72977ea5c..48735e0605 100644
@@ -13,10 +13,12 @@
 
 #define LIL_VERSION_STRING "0.1"
 
-#define LIL_SETVAR_GLOBAL 0
-#define LIL_SETVAR_LOCAL 1
-#define LIL_SETVAR_LOCAL_NEW 2
-#define LIL_SETVAR_LOCAL_ONLY 3
+enum lil_setvar {
+	LIL_SETVAR_GLOBAL = 0,
+	LIL_SETVAR_LOCAL,
+	LIL_SETVAR_LOCAL_NEW,
+	LIL_SETVAR_LOCAL_ONLY,
+};
 
 #define LIL_CALLBACK_EXIT 0
 #define LIL_CALLBACK_WRITE 1
@@ -98,7 +100,7 @@ struct lil_env *lil_push_env(struct lil *lil);
 void lil_pop_env(struct lil *lil);
 
 struct lil_var *lil_set_var(struct lil *lil, const char *name,
-			    struct lil_value *val, int local);
+			    struct lil_value *val, enum lil_setvar local);
 struct lil_value *lil_get_var(struct lil *lil, const char *name);
 struct lil_value *lil_get_var_or(struct lil *lil, const char *name,
 				 struct lil_value *defvalue);
Signature · 1 line
2.32.0
#73 lore raw

RFC PATCH 7/28 cli: lil: Simplify callbacks

In reply to Sean Anderson

We only need the getvar and setvar callbacks for now (to integrate with the environment). So remove all the other callbacks. Instead of allowing callbacks to be set at any time, just set them once in lil_new. Lastly, get rid of the typedefs and use a struct to pass in the callbacks instead.

---

 common/cli_lil.c  | 45 ++++++++++++---------------------------------
 include/cli_lil.h | 34 ++++++++--------------------------
 2 files changed, 20 insertions(+), 59 deletions(-)
common/cli_lil.c +12 -33
index 3d1e6181f9..539a4a3238 100644
@@ -24,7 +24,6 @@
  * overflows and is also useful when running through an automated fuzzer like AFL */
 /*#define LIL_ENABLE_RECLIMIT 10000*/
 
-#define CALLBACKS 8
 #define HASHMAP_CELLS 256
 #define HASHMAP_CELLMASK 0xFF
 
@@ -100,7 +99,7 @@ struct lil {
 	} error;
 	size_t err_head;
 	char *err_msg;
-	lil_callback_proc_t callback[CALLBACKS];
+	struct lil_callbacks callbacks;
 	size_t parse_depth;
 };
 
@@ -667,14 +666,10 @@ struct lil_var *lil_set_var(struct lil *lil, const char *name,
 		    var->env == lil->rootenv && var->env != env)
 			var = NULL;
 
-		if (((!var && env == lil->rootenv) ||
-		     (var && var->env == lil->rootenv)) &&
-		    lil->callback[LIL_CALLBACK_SETVAR]) {
-			lil_setvar_callback_proc_t proc =
-				(lil_setvar_callback_proc_t)
-					lil->callback[LIL_CALLBACK_SETVAR];
+		if (lil->callbacks.setvar && ((!var && env == lil->rootenv) ||
+					      (var && var->env == lil->rootenv))) {
 			struct lil_value *newval = val;
-			int r = proc(lil, name, &newval);
+			int r = lil->callbacks.setvar(lil, name, &newval);
 
 			if (r < 0) {
 				return NULL;
@@ -717,14 +712,10 @@ struct lil_value *lil_get_var_or(struct lil *lil, const char *name,
 	struct lil_var *var = lil_find_var(lil, lil->env, name);
 	struct lil_value *retval = var ? var->v : defvalue;
 
-	if (lil->callback[LIL_CALLBACK_GETVAR] &&
-	    (!var || var->env == lil->rootenv)) {
-		lil_getvar_callback_proc_t proc =
-			(lil_getvar_callback_proc_t)
-				lil->callback[LIL_CALLBACK_GETVAR];
+	if (lil->callbacks.getvar && (!var || var->env == lil->rootenv)) {
 		struct lil_value *newretval = retval;
 
-		if (proc(lil, name, &newretval))
+		if (lil->callbacks.getvar(lil, name, &newretval))
 			retval = newretval;
 	}
 	return retval;
@@ -748,10 +739,15 @@ void lil_pop_env(struct lil *lil)
 	}
 }
 
-struct lil *lil_new(void)
+struct lil *lil_new(const struct lil_callbacks *callbacks)
 {
 	struct lil *lil = calloc(1, sizeof(struct lil));
 
+	if (!lil)
+		return NULL;
+
+	if (callbacks)
+		memcpy(&lil->callbacks, callbacks, sizeof(lil->callbacks));
 	lil->rootenv = lil->env = lil_alloc_env(NULL);
 	lil->empty = alloc_value(NULL);
 	hm_init(&lil->cmdmap);
@@ -1148,15 +1144,6 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
 	}
 
 cleanup:
-	if (lil->error && lil->callback[LIL_CALLBACK_ERROR] &&
-	    lil->parse_depth == 1) {
-		lil_error_callback_proc_t proc =
-			(lil_error_callback_proc_t)
-				lil->callback[LIL_CALLBACK_ERROR];
-
-		proc(lil, lil->err_head, lil->err_msg);
-	}
-
 	if (words)
 		lil_free_list(words);
 	lil->code = save_code;
@@ -1185,14 +1172,6 @@ struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
 	return lil_parse(lil, val->d, val->l, funclevel);
 }
 
-void lil_callback(struct lil *lil, int cb, lil_callback_proc_t proc)
-{
-	if (cb < 0 || cb > CALLBACKS)
-		return;
-
-	lil->callback[cb] = proc;
-}
-
 void lil_set_error(struct lil *lil, const char *msg)
 {
 	if (lil->error)
include/cli_lil.h +8 -26
index 48735e0605..b8df94a766 100644
@@ -20,15 +20,6 @@ enum lil_setvar {
 	LIL_SETVAR_LOCAL_ONLY,
 };
 
-#define LIL_CALLBACK_EXIT 0
-#define LIL_CALLBACK_WRITE 1
-#define LIL_CALLBACK_READ 2
-#define LIL_CALLBACK_STORE 3
-#define LIL_CALLBACK_SOURCE 4
-#define LIL_CALLBACK_ERROR 5
-#define LIL_CALLBACK_SETVAR 6
-#define LIL_CALLBACK_GETVAR 7
-
 #include <stdint.h>
 #include <inttypes.h>
 
@@ -40,22 +31,15 @@ struct lil_list;
 struct lil;
 typedef struct lil_value *(*lil_func_proc_t)(struct lil *lil, size_t argc,
 					     struct lil_value **argv);
-typedef void (*lil_exit_callback_proc_t)(struct lil *lil,
-					 struct lil_value *arg);
-typedef void (*lil_write_callback_proc_t)(struct lil *lil, const char *msg);
-typedef char *(*lil_read_callback_proc_t)(struct lil *lil, const char *name);
-typedef char *(*lil_source_callback_proc_t)(struct lil *lil, const char *name);
-typedef void (*lil_store_callback_proc_t)(struct lil *lil, const char *name,
-					  const char *data);
-typedef void (*lil_error_callback_proc_t)(struct lil *lil, size_t pos,
-					  const char *msg);
-typedef int (*lil_setvar_callback_proc_t)(struct lil *lil, const char *name,
-					  struct lil_value **value);
-typedef int (*lil_getvar_callback_proc_t)(struct lil *lil, const char *name,
-					  struct lil_value **value);
-typedef void (*lil_callback_proc_t)(void);
 
-struct lil *lil_new(void);
+struct lil_callbacks {
+	int (*setvar)(struct lil *lil, const char *name,
+		      struct lil_value **value);
+	int (*getvar)(struct lil *lil, const char *name,
+		      struct lil_value **value);
+};
+
+struct lil *lil_new(const struct lil_callbacks *callbacks);
 void lil_free(struct lil *lil);
 
 int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc);
@@ -65,8 +49,6 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
 struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
 				  int funclevel);
 
-void lil_callback(struct lil *lil, int cb, lil_callback_proc_t proc);
-
 void lil_set_error(struct lil *lil, const char *msg);
 void lil_set_error_at(struct lil *lil, size_t pos, const char *msg);
 int lil_error(struct lil *lil, const char **msg, size_t *pos);
Signature · 1 line
2.32.0
#74 lore raw

RFC PATCH 8/28 cli: lil: Handle commands with dots

In reply to Sean Anderson

Some commands (such as md, mw, etc) use dot suffixes as options to the command. Therefore, we must ignore anything after a dot when finding commands.

---

 common/cli_lil.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
common/cli_lil.c +11 -1
index 539a4a3238..dd020e8452 100644
@@ -579,7 +579,17 @@ static struct lil_var *lil_find_var(struct lil *lil, struct lil_env *env,
 
 static struct lil_func *lil_find_cmd(struct lil *lil, const char *name)
 {
-	return hm_get(&lil->cmdmap, name);
+	struct lil_func *r;
+	char *dot = strchr(name, '.');
+
+	/* Some U-Boot commands have dots in their names */
+	if (dot)
+		*dot = '\0';
+	r = hm_get(&lil->cmdmap, name);
+
+	if (dot)
+		*dot = '.';
+	return r;
 }
 
 static struct lil_func *add_func(struct lil *lil, const char *name)
Signature · 1 line
2.32.0
#75 lore raw

RFC PATCH 9/28 cli: lil: Use error codes

In reply to Sean Anderson

This adds numeric error codes to be used for errors. This makes it easier for code to differentiate between different classes of errors. Some error codes are not used yet, and will be added in future commits. The position argument has been removed for a few reasons:

- It doesn't make sense for some errors, such as running out of memory
- It was often not very helpful, since it pointed at the end of the command
- Determining the location of an error is more difficult when the parsing and evaluating steps are done separately.

In addition, lil.error is changed to lil.err for consistency.

---

 common/cli_lil.c  | 92 ++++++++++++++++++-----------------------------
 include/cli_lil.h | 36 +++++++++++++++++--
 2 files changed, 68 insertions(+), 60 deletions(-)
common/cli_lil.c +35 -57
index dd020e8452..1bdf0e5265 100644
@@ -92,12 +92,7 @@ struct lil {
 	struct lil_env *rootenv;
 	struct lil_env *downenv;
 	struct lil_value *empty;
-	enum {
-		ERROR_NOERROR = 0,
-		ERROR_DEFAULT,
-		ERROR_FIXHEAD,
-	} error;
-	size_t err_head;
+	enum lil_error err;
 	char *err_msg;
 	struct lil_callbacks callbacks;
 	size_t parse_depth;
@@ -117,6 +112,7 @@ struct expreval {
 
 static struct lil_value *next_word(struct lil *lil);
 static void register_stdcmds(struct lil *lil);
+static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg);
 
 #ifdef LIL_ENABLE_POOLS
 static struct lil_value **pool;
@@ -974,7 +970,7 @@ static struct lil_list *substitute(struct lil *lil)
 	struct lil_list *words = lil_alloc_list();
 
 	lil_skip_spaces(lil);
-	while (lil->head < lil->clen && !ateol(lil) && !lil->error) {
+	while (lil->head < lil->clen && !ateol(lil) && !lil->err) {
 		struct lil_value *w = alloc_value(NULL);
 
 		do {
@@ -993,7 +989,7 @@ static struct lil_list *substitute(struct lil *lil)
 			lil_free_value(wp);
 		} while (lil->head < lil->clen &&
 			 !eolchar(lil->code[lil->head]) &&
-			 !isspace(lil->code[lil->head]) && !lil->error);
+			 !isspace(lil->code[lil->head]) && !lil->err);
 		lil_skip_spaces(lil);
 
 		lil_list_append(words, w);
@@ -1042,13 +1038,7 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
 	struct lil_value *r;
 
 	if (cmd->proc) {
-		size_t shead = lil->head;
-
 		r = cmd->proc(lil, words->c - 1, words->v + 1);
-		if (lil->error == ERROR_FIXHEAD) {
-			lil->error = ERROR_DEFAULT;
-			lil->err_head = shead;
-		}
 	} else {
 		lil_push_env(lil);
 		lil->env->func = cmd;
@@ -1102,18 +1092,18 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
 	lil->parse_depth++;
 #ifdef LIL_ENABLE_RECLIMIT
 	if (lil->parse_depth > LIL_ENABLE_RECLIMIT) {
-		lil_set_error(lil, "Too many recursive calls");
+		lil_set_error(lil, LIL_ERR_DEPTH, "Too many recursive calls");
 		goto cleanup;
 	}
 #endif
 
 	if (lil->parse_depth == 1)
-		lil->error = 0;
+		lil->err = LIL_ERR_NONE;
 
 	if (funclevel)
 		lil->env->breakrun = 0;
 
-	while (lil->head < lil->clen && !lil->error) {
+	while (lil->head < lil->clen && !lil->err) {
 		if (words)
 			lil_free_list(words);
 
@@ -1122,7 +1112,7 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
 		val = NULL;
 
 		words = substitute(lil);
-		if (!words || lil->error)
+		if (!words || lil->err)
 			goto cleanup;
 
 		if (words->c) {
@@ -1136,7 +1126,7 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
 					snprintf(msg, sizeof(msg),
 						 "unknown function %s",
 						 words->v[0]->d);
-					lil_set_error_at(lil, lil->head, msg);
+					lil_set_error(lil, LIL_ERR_NOCMD, msg);
 					goto cleanup;
 				}
 			} else {
@@ -1182,38 +1172,25 @@ struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
 	return lil_parse(lil, val->d, val->l, funclevel);
 }
 
-void lil_set_error(struct lil *lil, const char *msg)
+static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg)
 {
-	if (lil->error)
-		return;
-
+	assert(!lil->err);
 	free(lil->err_msg);
-	lil->error = ERROR_FIXHEAD;
-	lil->err_head = 0;
-	lil->err_msg = strdup(msg ? msg : "");
+	lil->err = err;
+	lil->err_msg = strdup(msg);
 }
 
-void lil_set_error_at(struct lil *lil, size_t pos, const char *msg)
+enum lil_error lil_error(struct lil *lil, const char **msg)
 {
-	if (lil->error)
-		return;
+	enum lil_error err = lil->err;
 
-	free(lil->err_msg);
-	lil->error = ERROR_DEFAULT;
-	lil->err_head = pos;
-	lil->err_msg = strdup(msg ? msg : "");
-}
-
-int lil_error(struct lil *lil, const char **msg, size_t *pos)
-{
-	if (!lil->error)
-		return 0;
+	if (!err)
+		return LIL_ERR_NONE;
 
 	*msg = lil->err_msg;
-	*pos = lil->err_head;
-	lil->error = ERROR_NOERROR;
+	lil->err = LIL_ERR_NONE;
 
-	return 1;
+	return err;
 }
 
 static void ee_expr(struct expreval *ee);
@@ -1642,7 +1619,7 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
 	struct expreval ee;
 
 	code = lil_subst_to_value(lil, code);
-	if (lil->error)
+	if (lil->err)
 		return NULL;
 
 	ee.code = lil_to_string(code);
@@ -1664,13 +1641,13 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
 	lil_free_value(code);
 	switch (ee.error) {
 	case EERR_DIVISION_BY_ZERO:
-		lil_set_error(lil, "division by zero in expression");
+		lil_set_error(lil, LIL_ERR_DOM, "division by zero");
 		return NULL;
 	case EERR_SYNTAX_ERROR:
-		lil_set_error(lil, "expression syntax error");
+		lil_set_error(lil, LIL_ERR_SYNTAX, "expression syntax");
 		return NULL;
 	case EERR_INVALID_EXPRESSION:
-		lil_set_error(lil, "invalid expression");
+		lil_set_error(lil, LIL_ERR_SYNTAX, "invalid expression");
 		return NULL;
 	case EERR_NO_ERROR:
 		break;
@@ -1994,7 +1971,7 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
 		char *msg = malloc(24 + strlen(oldname));
 
 		sprintf(msg, "unknown function '%s'", oldname);
-		lil_set_error_at(lil, lil->head, msg);
+		lil_set_error(lil, LIL_ERR_NOCMD, msg);
 		free(msg);
 		return NULL;
 	}
@@ -2397,7 +2374,7 @@ static struct lil_value *fnc_foreach(struct lil *lil, size_t argc,
 		else
 			lil_free_value(rv);
 
-		if (lil->env->breakrun || lil->error)
+		if (lil->env->breakrun || lil->err)
 			break;
 	}
 
@@ -2499,7 +2476,7 @@ static struct lil_value *fnc_if(struct lil *lil, size_t argc,
 		return NULL;
 
 	val = lil_eval_expr(lil, argv[base]);
-	if (!val || lil->error)
+	if (!val || lil->err)
 		return NULL;
 
 	v = lil_to_boolean(val);
@@ -2530,9 +2507,9 @@ static struct lil_value *fnc_while(struct lil *lil, size_t argc,
 	if (argc < (size_t)base + 2)
 		return NULL;
 
-	while (!lil->error && !lil->env->breakrun) {
+	while (!lil->err && !lil->env->breakrun) {
 		val = lil_eval_expr(lil, argv[base]);
-		if (!val || lil->error)
+		if (!val || lil->err)
 			return NULL;
 
 		v = lil_to_boolean(val);
@@ -2562,9 +2539,9 @@ static struct lil_value *fnc_for(struct lil *lil, size_t argc,
 		return NULL;
 
 	lil_free_value(lil_parse_value(lil, argv[0], 0));
-	while (!lil->error && !lil->env->breakrun) {
+	while (!lil->err && !lil->env->breakrun) {
 		val = lil_eval_expr(lil, argv[1]);
-		if (!val || lil->error)
+		if (!val || lil->err)
 			return NULL;
 
 		if (!lil_to_boolean(val)) {
@@ -2889,12 +2866,12 @@ static struct lil_value *fnc_try(struct lil *lil, size_t argc,
 	if (argc < 1)
 		return NULL;
 
-	if (lil->error)
+	if (lil->err)
 		return NULL;
 
 	r = lil_parse_value(lil, argv[0], 0);
-	if (lil->error) {
-		lil->error = ERROR_NOERROR;
+	if (lil->err) {
+		lil->err = LIL_ERR_NONE;
 		lil_free_value(r);
 
 		if (argc > 1)
@@ -2908,7 +2885,8 @@ static struct lil_value *fnc_try(struct lil *lil, size_t argc,
 static struct lil_value *fnc_error(struct lil *lil, size_t argc,
 				   struct lil_value **argv)
 {
-	lil_set_error(lil, argc > 0 ? lil_to_string(argv[0]) : NULL);
+	lil_set_error(lil, LIL_ERR_USER,
+		      argc > 0 ? lil_to_string(argv[0]) : NULL);
 	return NULL;
 }
 
include/cli_lil.h +33 -3
index b8df94a766..cdaa79fd15 100644
@@ -39,6 +39,38 @@ struct lil_callbacks {
 		      struct lil_value **value);
 };
 
+/**
+ * enum lil_error - Error codes returned by @lil_error
+ * LIL_ERR_NONE: No error
+ * LIL_ERR_OOM: Out of memory
+ * LIL_ERR_CASE: Missing case statement; generally a bug in the program or the
+ *               result of memory corruption.
+ * LIL_ERR_DEPTH: Too much recursion while parsing or evaluating
+ * LIL_ERR_SYNTAX: Unrecoverable syntax error
+ * LIL_ERR_EOF: Unexpected end of input; may be recoverable by prompting the
+ *              user for more input.
+ * LIL_ERR_INTR: Interrupted by Ctrl-C
+ * LIL_ERR_NOCMD: No such command
+ * LIL_ERR_NOVAR: No such variable
+ * LIL_ERR_ARGC: Incorrect number of arguments
+ * LIL_ERR_DOM: Numerical argument out of range (typically division by zero)
+ * LIL_ERR_USER: Error set by user
+ */
+enum lil_error {
+	LIL_ERR_NONE = 0,
+	LIL_ERR_OOM,
+	LIL_ERR_CASE,
+	LIL_ERR_DEPTH,
+	LIL_ERR_SYNTAX,
+	LIL_ERR_EOF,
+	LIL_ERR_INTR,
+	LIL_ERR_NOCMD,
+	LIL_ERR_NOVAR,
+	LIL_ERR_ARGC,
+	LIL_ERR_DOM,
+	LIL_ERR_USER,
+};
+
 struct lil *lil_new(const struct lil_callbacks *callbacks);
 void lil_free(struct lil *lil);
 
@@ -49,9 +81,7 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
 struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
 				  int funclevel);
 
-void lil_set_error(struct lil *lil, const char *msg);
-void lil_set_error_at(struct lil *lil, size_t pos, const char *msg);
-int lil_error(struct lil *lil, const char **msg, size_t *pos);
+enum lil_error lil_error(struct lil *lil, const char **msg);
 
 const char *lil_to_string(struct lil_value *val);
 ssize_t lil_to_integer(struct lil_value *val);
Signature · 1 line
2.32.0
#76 lore raw

RFC PATCH 10/28 cli: lil: Add printf-style format helper for errors

In reply to Sean Anderson

This adds a helper for dynamically-created error messages.

---

 common/cli_lil.c | 43 ++++++++++++++++++++++++++++++++-----------
 1 file changed, 32 insertions(+), 11 deletions(-)
common/cli_lil.c +32 -11
index 1bdf0e5265..f29c8065d8 100644
@@ -113,6 +113,9 @@ struct expreval {
 static struct lil_value *next_word(struct lil *lil);
 static void register_stdcmds(struct lil *lil);
 static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg);
+static void lil_set_errorf(struct lil *lil, enum lil_error err,
+			   const char *fmt, ...)
+	__attribute((format(__printf__, 3, 4)));
 
 #ifdef LIL_ENABLE_POOLS
 static struct lil_value **pool;
@@ -1121,12 +1124,9 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
 
 			if (!cmd) {
 				if (words->v[0]->l) {
-					char msg[64];
-
-					snprintf(msg, sizeof(msg),
-						 "unknown function %s",
-						 words->v[0]->d);
-					lil_set_error(lil, LIL_ERR_NOCMD, msg);
+					lil_set_errorf(lil, LIL_ERR_NOCMD,
+						       "unknown function %s",
+						       words->v[0]->d);
 					goto cleanup;
 				}
 			} else {
@@ -1180,6 +1180,30 @@ static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg)
 	lil->err_msg = strdup(msg);
 }
 
+static void lil_set_errorf(struct lil *lil, enum lil_error err,
+			   const char *fmt, ...)
+{
+	va_list args, saveargs;
+	size_t n;
+
+	assert(!lil->err);
+	free(lil->err_msg);
+	lil->err = err;
+
+	va_start(args, fmt);
+	va_copy(saveargs, args);
+	n = vsnprintf(NULL, 0, fmt, args) + 1;
+	va_end(args);
+
+	lil->err_msg = calloc(1, n);
+	if (!lil->err_msg) {
+		lil->err = LIL_ERR_OOM;
+		return;
+	}
+	vsnprintf(lil->err_msg, n, fmt, saveargs);
+	va_end(saveargs);
+}
+
 enum lil_error lil_error(struct lil *lil, const char **msg)
 {
 	enum lil_error err = lil->err;
@@ -1968,11 +1992,8 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
 	newname = lil_to_string(argv[1]);
 	func = lil_find_cmd(lil, oldname);
 	if (!func) {
-		char *msg = malloc(24 + strlen(oldname));
-
-		sprintf(msg, "unknown function '%s'", oldname);
-		lil_set_error(lil, LIL_ERR_NOCMD, msg);
-		free(msg);
+		lil_set_errorf(lil, LIL_ERR_NOCMD, "unknown function %s",
+			       oldname);
 		return NULL;
 	}
 
Signature · 1 line
2.32.0
#77 lore raw

RFC PATCH 11/28 cli: lil: Add several helper functions for errors

In reply to Sean Anderson

This adds several helper functions for common error types. This helps keep down code duplication, and also ensures that each of these errors uses the same message.

---

 common/cli_lil.c | 38 +++++++++++++++++++++++++++++++++-----
 1 file changed, 33 insertions(+), 5 deletions(-)
common/cli_lil.c +33 -5
index f29c8065d8..c19a25b2bf 100644
@@ -116,6 +116,9 @@ static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg);
 static void lil_set_errorf(struct lil *lil, enum lil_error err,
 			   const char *fmt, ...)
 	__attribute((format(__printf__, 3, 4)));
+static void lil_set_error_oom(struct lil *lil);
+static void lil_set_error_nocmd(struct lil *lil, const char *cmdname);
+static void lil_set_error_intr(struct lil *lil);
 
 #ifdef LIL_ENABLE_POOLS
 static struct lil_value **pool;
@@ -1124,9 +1127,8 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
 
 			if (!cmd) {
 				if (words->v[0]->l) {
-					lil_set_errorf(lil, LIL_ERR_NOCMD,
-						       "unknown function %s",
-						       words->v[0]->d);
+					lil_set_error_nocmd(lil,
+							    words->v[0]->d);
 					goto cleanup;
 				}
 			} else {
@@ -1204,6 +1206,33 @@ static void lil_set_errorf(struct lil *lil, enum lil_error err,
 	va_end(saveargs);
 }
 
+/*
+ * The next several functions provide helpers for throwing some formulaic
+ * errors. Their purpose is to ensure that the same wording is used everywhere
+ * the error is used. This reduces data size.
+ */
+
+static void lil_set_error_oom(struct lil *lil)
+{
+	lil_set_error(lil, LIL_ERR_OOM, NULL);
+}
+
+static void lil_set_error_argc(struct lil *lil, size_t expected)
+{
+	lil_set_errorf(lil, LIL_ERR_ARGC, "%s: expected %zu arguments",
+		       lil->env->proc ?: lil->env->func->name, expected);
+}
+
+static void lil_set_error_nocmd(struct lil *lil, const char *cmdname)
+{
+	lil_set_errorf(lil, LIL_ERR_NOCMD, "no such command %s", cmdname);
+}
+
+static void lil_set_error_intr(struct lil *lil)
+{
+	lil_set_error(lil, LIL_ERR_INTR, "interrupted");
+}
+
 enum lil_error lil_error(struct lil *lil, const char **msg)
 {
 	enum lil_error err = lil->err;
@@ -1992,8 +2021,7 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
 	newname = lil_to_string(argv[1]);
 	func = lil_find_cmd(lil, oldname);
 	if (!func) {
-		lil_set_errorf(lil, LIL_ERR_NOCMD, "unknown function %s",
-			       oldname);
+		lil_set_error_nocmd(lil, oldname);
 		return NULL;
 	}
 
Signature · 1 line
2.32.0
#78 lore raw

RFC PATCH 12/28 cli: lil: Check for ctrl-c

In reply to Sean Anderson

Check for ctrl-c in lil_parse. This works out to around every time a function or command is called. We also check at the beginning of lil_eval_expr so that constructs like

	while {1} {}

get interrupted. Since there are no non-trivial commands in that example, lil_parse never gets to its ctrlc check. However, we do need to evaluate the loop expression, so that's a good place to put a check.

---

 common/cli_lil.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
common/cli_lil.c +11 -0
index c19a25b2bf..50e314a643 100644
@@ -10,6 +10,7 @@
 
 #include <common.h>
 #include <cli_lil.h>
+#include <console.h>
 #include <ctype.h>
 #include <stdlib.h>
 #include <stdio.h>
@@ -1117,6 +1118,11 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
 			lil_free_value(val);
 		val = NULL;
 
+		if (ctrlc()) {
+			lil_set_error(lil, LIL_ERR_INTR, "interrupted");
+			goto cleanup;
+		}
+
 		words = substitute(lil);
 		if (!words || lil->err)
 			goto cleanup;
@@ -1671,6 +1677,11 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
 {
 	struct expreval ee;
 
+	if (ctrlc()) {
+		lil_set_error(lil, LIL_ERR_INTR, "interrupted");
+		return NULL;
+	}
+
 	code = lil_subst_to_value(lil, code);
 	if (lil->err)
 		return NULL;
Signature · 1 line
2.32.0
Reviewed-by on v1 12/28
#79 lore raw

Re: [RFC PATCH 12/28] cli: lil: Check for ctrl-c

Quoted 11 lines · Sean Anderson

Check for ctrl-c in lil_parse. This works out to around every time a function or command is called. We also check at the beginning of lil_eval_expr so that constructs like

        while {1} {}

get interrupted. Since there are no non-trivial commands in that example, lil_parse never gets to its ctrlc check. However, we do need to evaluate the loop expression, so that's a good place to put a check.

Quoted 5 lines

---

 common/cli_lil.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
#80 lore raw

RFC PATCH 13/28 cli: lil: Wire up LIL to the rest of U-Boot

In reply to Sean Anderson

This sets the shell to LIL when CONFIG_LIL is enabled. Repeated commands are not supporteed. Neither are partial commands a la Hush's secondary prompt. Setting and getting environmental variables is done through callbacks to assist with testing.

---

 cmd/Kconfig      | 12 +++++--
 common/cli.c     | 84 +++++++++++++++++++++++++++++++++++++++---------
 common/cli_lil.c | 32 ++++++++++++++++++
 3 files changed, 111 insertions(+), 17 deletions(-)
cmd/Kconfig +10 -2
index 0a7b73cb6d..b61a7557a9 100644
@@ -11,9 +11,14 @@ config CMDLINE
 	  Depending on the number of commands enabled, this can add
 	  substantially to the size of U-Boot.
 
+if CMDLINE
+
+choice
+	prompt "Shell"
+	default HUSH_PARSER
+
 config HUSH_PARSER
 	bool "Use hush shell"
-	depends on CMDLINE
 	help
 	  This option enables the "hush" shell (from Busybox) as command line
 	  interpreter, thus enabling powerful command line syntax like
@@ -25,13 +30,14 @@ config HUSH_PARSER
 
 config LIL
 	bool "Use LIL shell"
-	depends on CMDLINE
 	help
 	  This options enables the "Little Interpreted Language" (LIL) shell as
 	  command line interpreter, thus enabling powerful command line syntax
 	  like `proc name {args} {body}' functions or `echo [some command]`
 	  command substitution ("tcl scripts").
 
+endchoice
+
 if LIL
 
 config LIL_FULL
@@ -42,6 +48,8 @@ config LIL_FULL
 
 endif
 
+endif
+
 config CMDLINE_EDITING
 	bool "Enable command line editing"
 	depends on CMDLINE
common/cli.c +69 -15

149 lines not on this page · raw message

common/cli_lil.c +32 -0
index 50e314a643..66ee62bf33 100644
@@ -11,6 +11,7 @@
 #include <common.h>
 #include <cli_lil.h>
 #include <console.h>
+#include <command.h>
 #include <ctype.h>
 #include <stdlib.h>
 #include <stdio.h>
@@ -59,6 +60,7 @@ struct lil_var {
 struct lil_env {
 	struct lil_env *parent;
 	struct lil_func *func;
+	const char *proc;
 	struct lil_var **var;
 	size_t vars;
 	struct hashmap varmap;
@@ -1045,7 +1047,9 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
 	struct lil_value *r;
 
 	if (cmd->proc) {
+		lil->env->proc = words->v[0]->d;
 		r = cmd->proc(lil, words->c - 1, words->v + 1);
+		lil->env->proc = NULL;
 	} else {
 		lil_push_env(lil);
 		lil->env->func = cmd;
@@ -2967,8 +2971,33 @@ static struct lil_value *fnc_lmap(struct lil *lil, size_t argc,
 	return NULL;
 }
 
+static struct lil_value *fnc_builtin(struct lil *lil, size_t argc,
+				     struct lil_value **lil_argv)
+{
+	int err, repeatable;
+	size_t i;
+	/*
+	 * We need space for the function name, and the last argv must be NULL
+	 */
+	char **argv = calloc(sizeof(char *), argc + 2);
+
+	argv[0] = (char *)lil->env->proc;
+	for (i = 0; i < argc; i++)
+		argv[i + 1] = (char *)lil_to_string(lil_argv[i]);
+
+	err = cmd_process(0, argc + 1, argv, &repeatable, NULL);
+	if (err)
+		lil_set_errorf(lil, LIL_ERR_USER, "%s failed", argv[0]);
+	free(argv);
+
+	return 0;
+}
+
 static void register_stdcmds(struct lil *lil)
 {
+	struct cmd_tbl *cmdtp, *start = ll_entry_start(struct cmd_tbl, cmd);
+	const int len = ll_entry_count(struct cmd_tbl, cmd);
+
 	lil_register(lil, "decr", fnc_decr);
 	lil_register(lil, "eval", fnc_eval);
 	lil_register(lil, "expr", fnc_expr);
@@ -2984,6 +3013,9 @@ static void register_stdcmds(struct lil *lil)
 	lil_register(lil, "try", fnc_try);
 	lil_register(lil, "while", fnc_while);
 
+	for (cmdtp = start; cmdtp != start + len; cmdtp++)
+		lil_register(lil, cmdtp->name, fnc_builtin);
+
 	if (IS_ENABLED(CONFIG_LIL_FULL)) {
 		lil_register(lil, "append", fnc_append);
 		lil_register(lil, "char", fnc_char);
Signature · 1 line
2.32.0
#81 lore raw

Re: [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot

Quoted 4 lines · Sean Anderson

This sets the shell to LIL when CONFIG_LIL is enabled. Repeated commands are not supporteed. Neither are partial commands a la Hush's secondary prompt. Setting and getting environmental variables is done through callbacks to assist with testing.

This all looks very interesting, thanks for doing this!

Re this patch, is there some way LIL and HUSH could coexist, with HUSH then being the initial shell, and one could then enter a "lil shell" with the command "lil", just as I can start a busybox shell from bash by saying "busybox sh". That could make it a bit easier for playing around with initially.

I have no idea how hard that is to do with Kconfig. Perhaps make LIL selectable by itself, then have a hidden symbol LIL_SHELL which is "default y if LIL && !HUSH_PARSER" or something like that.

Rasmus

#82 lore raw

Re: [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot

Quoted 14 lines · Rasmus Villemoes
Quoted 4 lines · Sean Anderson

This sets the shell to LIL when CONFIG_LIL is enabled. Repeated commands are not supporteed. Neither are partial commands a la Hush's secondary prompt. Setting and getting environmental variables is done through callbacks to assist with testing.

This all looks very interesting, thanks for doing this!

Re this patch, is there some way LIL and HUSH could coexist, with HUSH then being the initial shell, and one could then enter a "lil shell" with the command "lil", just as I can start a busybox shell from bash by saying "busybox sh". That could make it a bit easier for playing around with initially.

Quoted 3 lines

I have no idea how hard that is to do with Kconfig. Perhaps make LIL selectable by itself, then have a hidden symbol LIL_SHELL which is "default y if LIL && !HUSH_PARSER" or something like that.

Yeah, I wanted to do this anyway for unit testing at least.

--Sean

#83 lore raw

Re: [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot

In reply to Sean Anderson

Hi Sean,

Quoted 6 lines · Sean Anderson

This sets the shell to LIL when CONFIG_LIL is enabled. Repeated commands are not supporteed. Neither are partial commands a la Hush's secondary prompt. Setting and getting environmental variables is done through callbacks to assist with testing.

Quoted 7 lines

---

 cmd/Kconfig      | 12 +++++--
 common/cli.c     | 84 +++++++++++++++++++++++++++++++++++++++---------
 common/cli_lil.c | 32 ++++++++++++++++++
 3 files changed, 111 insertions(+), 17 deletions(-)

Can you try using if() instead of #ifdef in here?

Regards,
Simon

#84 lore raw

RFC PATCH 14/28 cli: lil: Document structures

In reply to Sean Anderson

This documents the major structures of LIL.

---

 common/cli_lil.c  | 100 ++++++++++++++++++++++++++++++++++++++++++++++
 include/cli_lil.h |  57 ++++++++++++++++++++++++++
 2 files changed, 157 insertions(+)
common/cli_lil.c +100 -0
index 66ee62bf33..7fbae1964a 100644
@@ -29,20 +29,44 @@
 #define HASHMAP_CELLS 256
 #define HASHMAP_CELLMASK 0xFF
 
+/**
+ * struct hashentry - An entry in a cell
+ * @k: The key
+ * @v: The value
+ */
 struct hashentry {
 	char *k;
 	void *v;
 };
 
+/**
+ * struct hashcell - A list of entries with the same hash
+ * @e: An array of entries
+ * @c: The capacity of this cell
+ */
 struct hashcell {
 	struct hashentry *e;
 	size_t c;
 };
 
+/**
+ * struct hashmap - An array-based hash map
+ * @cell: An array of cells
+ *
+ * This hash map uses separate chaining with an array for each cell. We cannot
+ * use the existing hash map functions (hsearch_r et al.) because they assume
+ * that the value is a string.
+ */
 struct hashmap {
 	struct hashcell cell[HASHMAP_CELLS];
 };
 
+/**
+ * struct lil_value - A string and its length
+ * @l: The length of the value.
+ * @c: The capacity of this value (maximum of @l before requiring reallocation).
+ * @d: The contents of the value, as a nul-terminated string.
+ */
 struct lil_value {
 	size_t l;
 #ifdef LIL_ENABLE_POOLS
@@ -51,12 +75,34 @@ struct lil_value {
 	char *d;
 };
 
+/**
+ * struct lil_var - A variable
+ * @n: The name of this variable
+ * @env: The environment containing this variable
+ * @v: The value of this variable
+ */
 struct lil_var {
 	char *n;
 	struct lil_env *env;
 	struct lil_value *v;
 };
 
+/**
+ * struct lil_env - A function call's execution environment
+ * @parent: The parent of this environment. This is %NULL for the root
+ *          environment.
+ * @func: The currently-executing function
+ * @proc: The name of the currently-executing built-in procedure
+ * @var: A list of the variables in this environment
+ * @vars: The number of variables in @var
+ * @varmap: A hash map mapping variable names to pointers to variables in @var
+ * @retval: The value set by "return" or "result"
+ * @retval_set: Whether @retval has been set
+ * @breakrun: Whether to immediately break out the current run of execution.
+ *            This is set by "return"
+ *
+ * Variables are inherited from @parent.
+ */
 struct lil_env {
 	struct lil_env *parent;
 	struct lil_func *func;
@@ -69,12 +115,28 @@ struct lil_env {
 	int breakrun;
 };
 
+/**
+ * struct list - A list of values
+ * @v: A list of pointers to &struct lil_value
+ * @c: The number of values in this list
+ * @cap: The space allocated for @v
+ */
 struct lil_list {
 	struct lil_value **v;
 	size_t c;
 	size_t cap;
 };
 
+/**
+ * struct lil_func - A function which may be evaluated with a list of arguments
+ * @name: The name of the function
+ * @code: A value containing LIL code to be evaluated
+ * @argnames: The names of variables to assign to the passed-in arguments
+ * @proc: A C function to use to evaluate this function
+ *
+ * @code and @argnames are only used to evaluate the function if @proc is %NULL.
+ * If @argnames is empty, then the arguments are assigned to a variable "args".
+ */
 struct lil_func {
 	char *name;
 	struct lil_value *code;
@@ -82,6 +144,31 @@ struct lil_func {
 	lil_func_proc_t proc;
 };
 
+/**
+ * struct lil - The current state of the interpreter
+ * @code: The code which is being interpreted
+ * @rootcode: The top-level code (e.g. the code for the initial call to
+ *            lil_parse())
+ * @clen: The length of @code
+ * @head: The first uninterpreted part of this code, as an index of @code
+ * @ignoreeol: Whether to treat newlines as whitespace or command terminators
+ * @cmd: A list of the current commands
+ * @cmds: The number of commands in @cmd
+ * @cmdmap: A hash map mapping command names to pointers to commands in @cmd
+ * @env: The current environment to evaluate commands in
+ * @rootenv: The top-level "root" environment
+ * @downenv: The original environment after a call to "topeval" or "upeval"
+ * @empty: An empty value, allocated once
+ * @ERROR_NOERROR: There is no error.
+ * @ERROR_DEFAULT: There was an error.
+ * @ERROR_FIXHEAD: There was an error, but @err_head needs to be fixed.
+ * @ERROR_UNBALANCED: An opening quote or bracket lacks a balancing closing
+ *                    quote or bracket.
+ * @error: The current error status
+ * @err_head: The offset in @code which caused the @error
+ * @err_msg: An optional string describing the current @error
+ * @parse_depth: The depth of recursive function calls
+ */
 struct lil {
 	const char *code; /* need save on parse */
 	const char *rootcode;
@@ -101,6 +188,19 @@ struct lil {
 	size_t parse_depth;
 };
 
+/**
+ * struct expreval - A (mathematical) expression being evaluated
+ * @code: The complete code for this expression
+ * @len: The length of @code
+ * @head: The first unevaluated part of this expression, as an index of @code
+ * @ival: The integer value of this expression
+ * @EERR_NO_ERROR: There is no error
+ * @EERR_SYNTAX_ERROR: Syntax error. For now this is just mismatched
+ *                     parentheses.
+ * @EERR_DIVISION_BY_ZERO: Attempted division by zero
+ * @EERR_INVALID_EXPRESSION: A non-number was present
+ * @error: The error of this expression (if any)
+ */
 struct expreval {
 	const char *code;
 	size_t len, head;
include/cli_lil.h +57 -0

76 lines not on this page · raw message

Signature · 1 line
2.32.0
#85 lore raw

RFC PATCH 15/28 cli: lil: Convert LIL_ENABLE_POOLS to Kconfig

In reply to Sean Anderson

This adds a Kconfig option to enable memory pools for some commonly-created and destroyed LIL structures. There is still some significant code duplication, so perhaps these functions can be refactored further to all use common pool functions.

---

 cmd/Kconfig      |   7 ++
 common/cli_lil.c | 275 +++++++++++++++++++++++------------------------
 2 files changed, 143 insertions(+), 139 deletions(-)
cmd/Kconfig +7 -0
index b61a7557a9..28a387b380 100644
@@ -46,6 +46,13 @@ config LIL_FULL
 	  This enables all LIL builtin functions, as well as expression support
 	  for arithmetic and bitwise operations.
 
+config LIL_POOLS
+	bool "Use memory pools for LIL structures"
+	help
+	  Enable pools for reusing values, lists, and environments. This will
+	  use more memory but will cause considerably less memory fragmentation
+	  and improve the script execution performance.
+
 endif
 
 endif
common/cli_lil.c +136 -139

400 lines not on this page · raw message

Signature · 1 line
2.32.0
#86 lore raw

RFC PATCH 16/28 cli: lil: Convert LIL_ENABLE_RECLIMIT to KConfig

In reply to Sean Anderson

This adds a configuration option to set the recursion limit. I've set it to a (conservative) 1000 by default. In addition, there is an option to turn it off for a very minor space savings and performance increase.

---
Do we need this? Perhaps it should default to 0.

 cmd/Kconfig      |  8 ++++++++
 common/cli_lil.c | 10 ++--------
 2 files changed, 10 insertions(+), 8 deletions(-)
cmd/Kconfig +8 -0
index 28a387b380..7c8962cfc2 100644
@@ -53,6 +53,14 @@ config LIL_POOLS
 	  use more memory but will cause considerably less memory fragmentation
 	  and improve the script execution performance.
 
+config LIL_RECLIMIT
+	int "LIL function recursion limit"
+	default 1000
+	help
+	  Enable limiting recursive calls to lil_parse - this can be used to
+	  avoid call stack overflows and is also useful when running through an
+	  automated fuzzer like AFL. Set to 0 to disable the recursion limit.
+
 endif
 
 endif
common/cli_lil.c +2 -8
index 750a085f63..6c05531441 100644
@@ -17,10 +17,6 @@
 #include <stdio.h>
 #include <string.h>
 
-/* Enable limiting recursive calls to lil_parse - this can be used to avoid call stack
- * overflows and is also useful when running through an automated fuzzer like AFL */
-/*#define LIL_ENABLE_RECLIMIT 10000*/
-
 #define HASHMAP_CELLS 256
 #define HASHMAP_CELLMASK 0xFF
 
@@ -1198,12 +1194,10 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
 
 	lil_skip_spaces(lil);
 	lil->parse_depth++;
-#ifdef LIL_ENABLE_RECLIMIT
-	if (lil->parse_depth > LIL_ENABLE_RECLIMIT) {
-		lil_set_error(lil, LIL_ERR_DEPTH, "Too many recursive calls");
+	if (CONFIG_LIL_RECLIMIT && lil->parse_depth > CONFIG_LIL_RECLIMIT) {
+		lil_set_error(lil, LIL_ERR_DEPTH, "recursion limit reached");
 		goto cleanup;
 	}
-#endif
 
 	if (lil->parse_depth == 1)
 		lil->err = LIL_ERR_NONE;
Signature · 1 line
2.32.0
#87 lore raw

RFC PATCH 17/28 test: Add tests for LIL

In reply to Sean Anderson

This tests several aspects of the parser. These tests are primarily adapted from the *.lil code examples included with upstream LIL. These tests should probably get their own category, especially if I add additional styles of tests.

---
Yes, I know checkpatch complains about the quoted string being split, but that warning is intended for user-visible strings.

 MAINTAINERS       |   1 +
 test/cmd/Makefile |   1 +
 test/cmd/lil.c    | 339 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 341 insertions(+)
 create mode 100644 test/cmd/lil.c
MAINTAINERS +1 -0
index 0184de5f93..3bf460127b 100644
@@ -771,6 +771,7 @@ M:	Sean Anderson <[email protected]>
 S:	Maintained
 F:	common/cli_lil.c
 F:	include/cli_lil.h
+F:	test/cmd/lil.c
 
 LOGGING
 M:	Simon Glass <[email protected]>
test/cmd/Makefile +1 -0
index 2cfe43a6bd..4f7440cb44 100644
@@ -5,6 +5,7 @@
 ifdef CONFIG_HUSH_PARSER
 obj-$(CONFIG_CONSOLE_RECORD) += test_echo.o
 endif
+obj-$(CONFIG_LIL_FULL) += lil.o
 obj-y += mem.o
 obj-$(CONFIG_CMD_ADDRMAP) += addrmap.o
 obj-$(CONFIG_CMD_MEM_SEARCH) += mem_search.o
test/cmd/lil.c new file +339 -0

339 lines not on this page · raw message

Signature · 1 line
2.32.0
Reviewed-by on v1 17/28
#88 lore raw

Re: [RFC PATCH 17/28] test: Add tests for LIL

Quoted 11 lines · Sean Anderson

This tests several aspects of the parser. These tests are primarily adapted from the *.lil code examples included with upstream LIL. These tests should probably get their own category, especially if I add additional styles of tests.

---
Yes, I know checkpatch complains about the quoted string being split, but that warning is intended for user-visible strings.

Quoted 6 lines
 MAINTAINERS       |   1 +
 test/cmd/Makefile |   1 +
 test/cmd/lil.c    | 339 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 341 insertions(+)
 create mode 100644 test/cmd/lil.c
#89 lore raw

RFC PATCH 18/28 cli: lil: Remove duplicate function bodies

In reply to Sean Anderson

lil_append_val is just lil_append_string with the string and length taken from a struct lil_value. Use lil_append_stringh_len to implement both. Do the same for lil_clone_value.

---

 common/cli_lil.c | 51 ++----------------------------------------------
 1 file changed, 2 insertions(+), 49 deletions(-)
common/cli_lil.c +2 -49
index 6c05531441..5875fbd46b 100644
@@ -369,39 +369,7 @@ static struct lil_value *alloc_value(const char *str)
 
 struct lil_value *lil_clone_value(struct lil_value *src)
 {
-	struct lil_value *val;
-
-	if (!src)
-		return NULL;
-
-	if (IS_ENABLED(CONFIG_LIL_POOLS))
-		val = alloc_from_pool();
-	else
-		val = calloc(1, sizeof(struct lil_value));
-	if (!val)
-		return NULL;
-
-	val->l = src->l;
-	if (src->l) {
-		if (IS_ENABLED(CONFIG_LIL_POOLS)) {
-			ensure_capacity(val, val->l + 1);
-		} else {
-			val->d = malloc(val->l + 1);
-			if (!val->d) {
-				free(val);
-				return NULL;
-			}
-		}
-		memcpy(val->d, src->d, val->l + 1);
-	} else {
-		if (IS_ENABLED(CONFIG_LIL_POOLS)) {
-			ensure_capacity(val, 1);
-			val->d[0] = '\0';
-		} else {
-			val->d = NULL;
-		}
-	}
-	return val;
+	return alloc_value_len(src->d, src->l);
 }
 
 int lil_append_char(struct lil_value *val, char ch)
@@ -450,22 +418,7 @@ int lil_append_string(struct lil_value *val, const char *s)
 
 int lil_append_val(struct lil_value *val, struct lil_value *v)
 {
-	if (!v || !v->l)
-		return 1;
-
-	if (IS_ENABLED(CONFIG_LIL_POOLS)) {
-		ensure_capacity(val, val->l + v->l + 1);
-		memcpy(val->d + val->l, v->d, v->l + 1);
-	} else {
-		char *new = realloc(val->d, val->l + v->l + 1);
-
-		if (!new)
-			return 0;
-		memcpy(new + val->l, v->d, v->l + 1);
-		val->d = new;
-	}
-	val->l += v->l;
-	return 1;
+	return lil_append_string_len(val, v->d, v->l);
 }
 
 void lil_free_value(struct lil_value *val)
Signature · 1 line
2.32.0
#90 lore raw

RFC PATCH 19/28 cli: lil: Add "symbol" structure

In reply to Sean Anderson

We need a generic structure to hold symbols parsed by the parser. We would also like to re-use existing code as much as possible without rewriting everything. To do this, we hijack the allocators for lil_list and lil_value and have them allocate enough space for a lil_symbol.

While we're at it, we can make lil_list hold lil_symbols instead of lil_values. To keep all the old users sane, we just cast back to lil_value before retrieving the value (with an assert to make sure we're not sending back something else). Unfortunately, many functions were accessing the list vector directly, so convert them.

This commit also fixes pools not behaving properly when running out of memory. This should likely be refactored in the future so everything uses one set of allocator/free routines to avoid code duplication.

---

 common/cli_lil.c  | 317 +++++++++++++++++++++++++++++-----------------
 include/cli_lil.h |   8 +-
 2 files changed, 206 insertions(+), 119 deletions(-)
common/cli_lil.c +202 -115
index 5875fbd46b..06fd37c383 100644
@@ -106,18 +106,45 @@ struct lil_env {
 	int breakrun;
 };
 
+struct lil_symbol;
+
 /**
- * struct list - A list of values
- * @v: A list of pointers to &struct lil_value
- * @c: The number of values in this list
+ * struct list - A list of symbols
+ * @v: A list of pointers to symbols
+ * @c: The number of symbols in this list
  * @cap: The space allocated for @v
  */
 struct lil_list {
-	struct lil_value **v;
+	struct lil_symbol **v;
 	size_t c;
 	size_t cap;
 };
 
+/**
+ * struct lil_symbol - A symbol parsed by the parser
+ * @LIL_SYMBOL_VALUE: A plain old string and length
+ * @LIL_SYMBOL_VARIABLE: A name of a variable to be substituted
+ * @LIL_SYMBOL_LIST: A list of symbols
+ * @value: A literal value or name of variable
+ * @list: A list of commands in the script
+ * @word: Another word to be evaluated
+ * @type: The type of word
+ */
+struct lil_symbol {
+	union {
+		struct lil_value value;
+		struct lil_symbol *symbol;
+		struct lil_list list;
+	};
+	enum {
+		LIL_SYMBOL_VALUE = 0,
+		LIL_SYMBOL_LIST,
+		LIL_SYMBOL_VARIABLE,
+		LIL_SYMBOL_COMMAND,
+		LIL_SYMBOL_SCRIPT,
+	} type;
+};
+
 /**
  * struct lil_func - A function which may be evaluated with a list of arguments
  * @name: The name of the function
@@ -287,24 +314,50 @@ static int hm_has(struct hashmap *hm, const char *key)
 	return 0;
 }
 
-#ifdef CONFIG_LIL_POOLS
+static void lil_free_symbol(struct lil_symbol *sym)
+{
+	switch (sym->type) {
+	case LIL_SYMBOL_VALUE:
+		lil_free_value(&sym->value);
+		return;
+	case LIL_SYMBOL_VARIABLE:
+		lil_free_symbol(sym->symbol);
+		free(sym);
+		return;
+	case LIL_SYMBOL_LIST:
+	case LIL_SYMBOL_COMMAND:
+	case LIL_SYMBOL_SCRIPT:
+		lil_free_list(&sym->list);
+		return;
+	}
+	log_debug("unknown type %d\n", sym->type);
+	assert(0);
+}
+
+#if IS_ENABLED(CONFIG_LIL_POOLS)
 static struct lil_value *alloc_from_pool(void)
 {
-	if (poolsize > 0) {
-		poolsize--;
-		return pool[poolsize];
-	} else {
-		struct lil_value *val = calloc(1, sizeof(struct lil_value));
-
-		return val;
-	}
+	if (poolsize > 0)
+		return pool[--poolsize];
+	else
+		return calloc(1, sizeof(struct lil_symbol));
 }
 
 static void release_to_pool(struct lil_value *val)
 {
 	if (poolsize == poolcap) {
-		poolcap = poolcap ? (poolcap + poolcap / 2) : 64;
-		pool = realloc(pool, sizeof(struct lil_value *) * poolcap);
+		size_t npoolcap = poolcap ? (poolcap + poolcap / 2) : 64;
+		struct lil_value **npool =
+			realloc(pool, sizeof(struct lil_symbol *) * npoolcap);
+
+		if (!npool) {
+			free(val->d);
+			free(val);
+			return;
+		}
+
+		poolcap = npoolcap;
+		pool = npool;
 	}
 	pool[poolsize++] = val;
 }
@@ -319,45 +372,48 @@ static void ensure_capacity(struct lil_value *val, size_t cap)
 #else
 static struct lil_value *alloc_from_pool(void)
 {
-	return NULL;
+	return calloc(1, sizeof(struct lil_symbol));
 }
 
-static void release_to_pool(struct lil_value *val) { }
-static void ensure_capacity(struct lil_value *val, size_t cap) { }
+static void release_to_pool(struct lil_value *val)
+{
+	free(val->d);
+	free(val);
+}
+static void ensure_capacity(struct lil_value *val, size_t cap)
+{
+	val->d = realloc(val->d, cap);
+}
 #endif
 
+static struct lil_symbol *value_to_symbol(struct lil_value *val)
+{
+	return container_of(val, struct lil_symbol, value);
+}
+
 static struct lil_value *alloc_value_len(const char *str, size_t len)
 {
 	struct lil_value *val;
 
-	if (IS_ENABLED(CONFIG_LIL_POOLS))
-		val = alloc_from_pool();
-	else
-		val = calloc(1, sizeof(struct lil_value));
+	val = alloc_from_pool();
 	if (!val)
 		return NULL;
+	value_to_symbol(val)->type = LIL_SYMBOL_VALUE;
 
 	if (str) {
 		val->l = len;
-		if (IS_ENABLED(CONFIG_LIL_POOLS)) {
-			ensure_capacity(val, len + 1);
-		} else {
-			val->d = malloc(len + 1);
-			if (!val->d) {
-				free(val);
-				return NULL;
-			}
+		ensure_capacity(val, len + 1);
+		if (!val->d) {
+			release_to_pool(val);
+			return NULL;
 		}
 		memcpy(val->d, str, len);
 		val->d[len] = 0;
 	} else {
 		val->l = 0;
-		if (IS_ENABLED(CONFIG_LIL_POOLS)) {
-			ensure_capacity(val, 1);
+		ensure_capacity(val, 1);
+		if (val->d)
 			val->d[0] = '\0';
-		} else {
-			val->d = NULL;
-		}
 	}
 	return val;
 }
@@ -372,53 +428,41 @@ struct lil_value *lil_clone_value(struct lil_value *src)
 	return alloc_value_len(src->d, src->l);
 }
 
-int lil_append_char(struct lil_value *val, char ch)
+enum lil_error lil_append_char(struct lil_value *val, char ch)
 {
-	if (IS_ENABLED(CONFIG_LIL_POOLS)) {
-		ensure_capacity(val, val->l + 2);
-		val->d[val->l++] = ch;
-		val->d[val->l] = '\0';
-	} else {
-		char *new = realloc(val->d, val->l + 2);
+	ensure_capacity(val, val->l + 2);
+	if (!val->d)
+		return LIL_ERR_OOM;
 
-		if (!new)
-			return 0;
-
-		new[val->l++] = ch;
-		new[val->l] = 0;
-		val->d = new;
-	}
-	return 1;
+	val->d[val->l++] = ch;
+	val->d[val->l] = '\0';
+	return LIL_ERR_NONE;
 }
 
-int lil_append_string_len(struct lil_value *val, const char *s, size_t len)
+static enum lil_error lil_append_string_len(struct lil_value *val,
+					    const char *s, size_t len)
 {
 	if (!s || !s[0])
-		return 1;
+		return LIL_ERR_NONE;
 
-	if (IS_ENABLED(CONFIG_LIL_POOLS)) {
-		ensure_capacity(val, val->l + len + 1);
-		memcpy(val->d + val->l, s, len + 1);
-	} else {
-		char *new = realloc(val->d, val->l + len + 1);
-
-		if (!new)
-			return 0;
-		memcpy(new + val->l, s, len + 1);
-		val->d = new;
-	}
+	ensure_capacity(val, val->l + len + 1);
+	if (!val->d)
+		return LIL_ERR_OOM;
+	memcpy(val->d + val->l, s, len + 1);
 	val->l += len;
-	return 1;
+	return LIL_ERR_NONE;
 }
 
-int lil_append_string(struct lil_value *val, const char *s)
+enum lil_error lil_append_string(struct lil_value *val, const char *s)
 {
 	return lil_append_string_len(val, s, strlen(s));
 }
 
-int lil_append_val(struct lil_value *val, struct lil_value *v)
+enum lil_error lil_append_val(struct lil_value *val, struct lil_value *v)
 {
-	return lil_append_string_len(val, v->d, v->l);
+	if (v)
+		return lil_append_string_len(val, v->d, v->l);
+	return LIL_ERR_NONE;
 }
 
 void lil_free_value(struct lil_value *val)
@@ -426,23 +470,28 @@ void lil_free_value(struct lil_value *val)
 	if (!val)
 		return;
 
-	if (IS_ENABLED(CONFIG_LIL_POOLS)) {
-		release_to_pool(val);
-	} else {
-		free(val->d);
-		free(val);
-	}
+	release_to_pool(val);
+}
+
+static struct lil_symbol *list_to_symbol(struct lil_list *list)
+{
+	return container_of(list, struct lil_symbol, list);
 }
 
 struct lil_list *lil_alloc_list(void)
 {
 	struct lil_list *list;
 
-	if (IS_ENABLED(CONFIG_LIL_POOLS) && listpoolsize > 0)
-		return listpool[--listpoolsize];
+	if (IS_ENABLED(CONFIG_LIL_POOLS) && listpoolsize > 0) {
+		list = listpool[--listpoolsize];
+	} else {
+		list = calloc(1, sizeof(struct lil_symbol));
+		if (!list)
+			return list;
+		list->v = NULL;
+	}
 
-	list = calloc(1, sizeof(struct lil_list));
-	list->v = NULL;
+	list_to_symbol(list)->type = LIL_SYMBOL_LIST;
 	return list;
 }
 
@@ -454,49 +503,61 @@ void lil_free_list(struct lil_list *list)
 		return;
 
 	for (i = 0; i < list->c; i++)
-		lil_free_value(list->v[i]);
+		lil_free_symbol(list->v[i]);
 
 	if (IS_ENABLED(CONFIG_LIL_POOLS)) {
 		list->c = 0;
 		if (listpoolsize == listpoolcap) {
+			int ncap;
+			struct lil_list **npool;
+
 			if (listpoolcap)
-				listpoolcap += listpoolcap / 2;
+				ncap += listpoolcap / 2;
 			else
-				listpoolcap = 32;
-			listpool = realloc(listpool,
-					   sizeof(*listpool) * listpoolcap);
+				ncap = 32;
+			npool = realloc(listpool, sizeof(*npool) * ncap);
+			if (!npool)
+				goto free;
+
+			listpoolcap = ncap;
+			listpool = npool;
 		}
 		listpool[listpoolsize++] = list;
 	} else {
+free:
 		free(list->v);
 		free(list);
 	}
 }
 
-void lil_list_append(struct lil_list *list, struct lil_value *val)
+int lil_list_append(struct lil_list *list, void *item)
 {
 	if (list->c == list->cap) {
 		size_t cap = list->cap ? (list->cap + list->cap / 2) : 32;
-		struct lil_value **nv =
-			realloc(list->v, sizeof(struct lil_value *) * cap);
+		struct lil_symbol **nv = realloc(list->v, sizeof(void *) * cap);
 
 		if (!nv)
-			return;
+			return -ENOMEM;
 
 		list->cap = cap;
 		list->v = nv;
 	}
-	list->v[list->c++] = val;
+	list->v[list->c++] = item;
+	return 0;
 }
 
-size_t lil_list_size(struct lil_list *list)
+static struct lil_symbol *lil_list_gets(struct lil_list *list, size_t index)
 {
-	return list->c;
+	assert(index < list->c);
+	return list->v[index];
 }
 
 struct lil_value *lil_list_get(struct lil_list *list, size_t index)
 {
-	return index >= list->c ? NULL : list->v[index];
+	struct lil_symbol *sym = lil_list_gets(list, index);
+
+	assert(sym->type == LIL_SYMBOL_VALUE);
+	return &sym->value;
 }
 
 static int needs_escape(const char *str)
@@ -519,25 +580,26 @@ struct lil_value *lil_list_to_value(struct lil_list *list, int do_escape)
 	size_t i, j;
 
 	for (i = 0; i < list->c; i++) {
+		struct lil_value *item = lil_list_get(list, i);
 		int escape =
-			do_escape ? needs_escape(lil_to_string(list->v[i])) : 0;
+			do_escape ? needs_escape(lil_to_string(item)) : 0;
 
 		if (i)
 			lil_append_char(val, ' ');
 
 		if (escape) {
 			lil_append_char(val, '{');
-			for (j = 0; j < list->v[i]->l; j++) {
-				if (list->v[i]->d[j] == '{')
+			for (j = 0; j < item->l; j++) {
+				if (item->d[j] == '{')
 					lil_append_string(val, "}\"\\o\"{");
-				else if (list->v[i]->d[j] == '}')
+				else if (item->d[j] == '}')
 					lil_append_string(val, "}\"\\c\"{");
 				else
-					lil_append_char(val, list->v[i]->d[j]);
+					lil_append_char(val, item->d[j]);
 			}
 			lil_append_char(val, '}');
 		} else {
-			lil_append_val(val, list->v[i]);
+			lil_append_val(val, item);
 		}
 	}
 	return val;
@@ -588,15 +650,23 @@ void lil_free_env(struct lil_env *env)
 		free(env->var);
 
 		if (envpoolsize == envpoolcap) {
+			int ncap;
+			struct lil_env **npool;
+
 			if (envpoolcap)
 				envpoolcap += envpoolcap / 2;
 			else
 				envpoolcap = 64;
-			envpool = realloc(envpool,
-					  sizeof(*envpool) * envpoolcap);
+			npool = realloc(envpool, sizeof(*npool) * ncap);
+			if (!npool)
+				goto free;
+
+			envpoolcap = ncap;
+			envpool = npool;
 		}
 		envpool[envpoolsize++] = env;
 	} else {
+free:
 		hm_destroy(&env->varmap);
 		for (i = 0; i < env->vars; i++) {
 			free(env->var[i]->n);
@@ -660,6 +730,8 @@ static struct lil_func *add_func(struct lil *lil, const char *name)
 	}
 
 	cmd = calloc(1, sizeof(struct lil_func));
+	if (!cmd)
+		return NULL;
 	cmd->name = strdup(name);
 
 	ncmd = realloc(lil->cmd, sizeof(struct lil_func *) * (lil->cmds + 1));
@@ -1093,15 +1165,17 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
 	struct lil_value *r;
 
 	if (cmd->proc) {
-		lil->env->proc = words->v[0]->d;
-		r = cmd->proc(lil, words->c - 1, words->v + 1);
+		lil->env->proc = lil_to_string(lil_list_get(words, 0));
+		r = cmd->proc(lil, words->c - 1,
+			      (struct lil_value **)words->v + 1);
 		lil->env->proc = NULL;
 	} else {
 		lil_push_env(lil);
 		lil->env->func = cmd;
 
 		if (cmd->argnames->c == 1 &&
-		    !strcmp(lil_to_string(cmd->argnames->v[0]), "args")) {
+		    !strcmp(lil_to_string(lil_list_get(cmd->argnames, 0)),
+						       "args")) {
 			struct lil_value *args = lil_list_to_value(words, 1);
 
 			lil_set_var(lil, "args", args, LIL_SETVAR_LOCAL_NEW);
@@ -1111,14 +1185,15 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
 
 			for (i = 0; i < cmd->argnames->c; i++) {
 				struct lil_value *val;
+				struct lil_value *name =
+					    lil_list_get(cmd->argnames, i);
 
 				if (i < words->c - 1)
-					val = words->v[i + 1];
+					val = lil_list_get(words, i + 1);
 				else
 					val = lil->empty;
 
-				lil_set_var(lil,
-					    lil_to_string(cmd->argnames->v[i]),
+				lil_set_var(lil, lil_to_string(name),
 					    val, LIL_SETVAR_LOCAL_NEW);
 			}
 		}
@@ -1176,13 +1251,13 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
 			goto cleanup;
 
 		if (words->c) {
-			struct lil_func *cmd =
-				lil_find_cmd(lil, lil_to_string(words->v[0]));
+			const char *cmdname =
+				lil_to_string(lil_list_get(words, 0));
+			struct lil_func *cmd = lil_find_cmd(lil, cmdname);
 
 			if (!cmd) {
-				if (words->v[0]->l) {
-					lil_set_error_nocmd(lil,
-							    words->v[0]->d);
+				if (cmdname[0]) {
+					lil_set_error_nocmd(lil, cmdname);
 					goto cleanup;
 				}
 			} else {
@@ -2042,6 +2117,9 @@ static struct lil_value *fnc_proc(struct lil *lil, size_t argc,
 		name = lil_clone_value(argv[0]);
 		fargs = lil_subst_to_list(lil, argv[1]);
 		cmd = add_func(lil, lil_to_string(argv[0]));
+		if (!cmd)
+			return NULL;
+
 		cmd->argnames = fargs;
 		cmd->code = lil_clone_value(argv[2]);
 	} else {
@@ -2052,11 +2130,17 @@ static struct lil_value *fnc_proc(struct lil *lil, size_t argc,
 			fargs = lil_subst_to_list(lil, tmp);
 			lil_free_value(tmp);
 			cmd = add_func(lil, lil_to_string(name));
+			if (!cmd)
+				return NULL;
+
 			cmd->argnames = fargs;
 			cmd->code = lil_clone_value(argv[0]);
 		} else {
 			fargs = lil_subst_to_list(lil, argv[0]);
 			cmd = add_func(lil, lil_to_string(name));
+			if (!cmd)
+				return NULL;
+
 			cmd->argnames = fargs;
 			cmd->code = lil_clone_value(argv[1]);
 		}
@@ -2278,7 +2362,7 @@ static struct lil_value *fnc_index(struct lil *lil, size_t argc,
 	if (index >= list->c)
 		r = NULL;
 	else
-		r = lil_clone_value(list->v[index]);
+		r = lil_clone_value(lil_list_get(list, index));
 	lil_free_list(list);
 	return r;
 }
@@ -2295,7 +2379,7 @@ static struct lil_value *fnc_indexof(struct lil *lil, size_t argc,
 
 	list = lil_subst_to_list(lil, argv[0]);
 	for (index = 0; index < list->c; index++) {
-		if (!strcmp(lil_to_string(list->v[index]),
+		if (!strcmp(lil_to_string(lil_list_get(list, index)),
 			    lil_to_string(argv[1]))) {
 			r = lil_alloc_integer(index);
 			break;
@@ -2364,7 +2448,7 @@ static struct lil_value *fnc_slice(struct lil *lil, size_t argc,
 
 	slice = lil_alloc_list();
 	for (i = (size_t)from; i < (size_t)to; i++)
-		lil_list_append(slice, lil_clone_value(list->v[i]));
+		lil_list_append(slice, lil_clone_value(lil_list_get(list, i)));
 	lil_free_list(list);
 
 	r = lil_list_to_value(slice, 1);
@@ -2395,10 +2479,12 @@ static struct lil_value *fnc_filter(struct lil *lil, size_t argc,
 	list = lil_subst_to_list(lil, argv[base]);
 	filtered = lil_alloc_list();
 	for (i = 0; i < list->c && !lil->env->breakrun; i++) {
-		lil_set_var(lil, varname, list->v[i], LIL_SETVAR_LOCAL_ONLY);
+		lil_set_var(lil, varname, lil_list_get(list, i),
+			    LIL_SETVAR_LOCAL_ONLY);
 		r = lil_eval_expr(lil, argv[base + 1]);
 		if (lil_to_boolean(r))
-			lil_list_append(filtered, lil_clone_value(list->v[i]));
+			lil_list_append(filtered,
+					lil_clone_value(lil_list_get(list, i)));
 		lil_free_value(r);
 	}
 	lil_free_list(list);
@@ -2475,7 +2561,8 @@ static struct lil_value *fnc_foreach(struct lil *lil, size_t argc,
 	for (i = 0; i < list->c; i++) {
 		struct lil_value *rv;
 
-		lil_set_var(lil, varname, list->v[i], LIL_SETVAR_LOCAL_ONLY);
+		lil_set_var(lil, varname, lil_list_get(list, i),
+			    LIL_SETVAR_LOCAL_ONLY);
 		rv = lil_parse_value(lil, argv[codeidx], 0);
 		if (rv->l)
 			lil_list_append(rlist, rv);
include/cli_lil.h +4 -4
index 91e79c12f4..40c822401e 100644
@@ -149,13 +149,13 @@ struct lil_value *lil_alloc_integer(ssize_t num);
 void lil_free_value(struct lil_value *val);
 
 struct lil_value *lil_clone_value(struct lil_value *src);
-int lil_append_char(struct lil_value *val, char ch);
-int lil_append_string(struct lil_value *val, const char *s);
-int lil_append_val(struct lil_value *val, struct lil_value *v);
+enum lil_error lil_append_char(struct lil_value *val, char ch);
+enum lil_error lil_append_string(struct lil_value *val, const char *s);
+enum lil_error lil_append_val(struct lil_value *val, struct lil_value *v);
 
 struct lil_list *lil_alloc_list(void);
 void lil_free_list(struct lil_list *list);
-void lil_list_append(struct lil_list *list, struct lil_value *val);
+int lil_list_append(struct lil_list *list, void *item);
 size_t lil_list_size(struct lil_list *list);
 struct lil_value *lil_list_get(struct lil_list *list, size_t index);
 struct lil_value *lil_list_to_value(struct lil_list *list, int do_escape);
Signature · 1 line
2.32.0
#91 lore raw

RFC PATCH 20/28 cli: lil: Add config to enable debug output

In reply to Sean Anderson

This provides an easy way to enable assertions and debug messages. It will also be used to enable tracing features in future patches.

---

 cmd/Kconfig     | 6 ++++++
 common/Makefile | 3 +++
 2 files changed, 9 insertions(+)
cmd/Kconfig +6 -0
index 7c8962cfc2..bba72bbdc2 100644
@@ -61,6 +61,12 @@ config LIL_RECLIMIT
 	  avoid call stack overflows and is also useful when running through an
 	  automated fuzzer like AFL. Set to 0 to disable the recursion limit.
 
+config LIL_DEBUG
+	bool "Enable LIL debugging"
+	help
+	  This enables debug prints, assertions, and other tracing features in
+	  LIL. If you are not working on LIL, say 'n' here.
+
 endif
 
 endif
common/Makefile +3 -0
index dce04b305e..558e1932fe 100644
@@ -11,6 +11,9 @@ obj-y += exports.o
 obj-$(CONFIG_HASH) += hash.o
 obj-$(CONFIG_HUSH_PARSER) += cli_hush.o
 obj-$(CONFIG_LIL) += cli_lil.o
+ifneq ($(CONFIG_LIL_DEBUG),)
+CFLAGS_cli_lil.o += -DDEBUG
+endif
 obj-$(CONFIG_AUTOBOOT) += autoboot.o
 
 # This option is not just y/n - it can have a numeric value
Signature · 1 line
2.32.0
#92 lore raw

RFC PATCH 21/28 cli: lil: Add a distinct parsing step

In reply to Sean Anderson

This adds a parser to LIL (as separate from the interpreter). This is necessary to detect syntax errors before evaluating anything. Before this, running a script like

	echo some message; echo syntax error}

would result in "some message" being printed before the error was discovered. This is not only rather surprising, but also makes things like Hush's secondary prompt impossible to implement. In addition, the original parser would accept almost any input, and silently return NULL if it encountered problems. This made it difficult to determine if a command had been mis-parsed, since an empty command would just evaluate to "".

The grammar is not the same as LIL originally. Several ideas have been taken from TCL proper as well. In order to simplify the parser, it has been rewritten to be LL(1), except for line continuations which are LL(2). In particular, multi-line comments and command/variable subtitutions partially through unquoted words (e.g. a$b) have been removed. Some other characters such as unescaped, unmatched }s are now syntax errors. On the other hand, some things such as escaped characters in unquoted words have been added back (as seen in TCL). Unlike TCL, comments may be placed almost anywhere. The exact grammar is subject to change, but I have tried to make it as sane as I can get it.

The grammar has been documented in (extended) EBNF. The names of the nonterminals are the same as are used in the dodekalogue [1]. Each nonterminal foo has a function parse_foo() which recognizes it.

[1] https://www.tcl.tk/man/tcl8.6/TclCmd/Tcl.htm

---

 cmd/Kconfig       |    4 +-
 common/cli.c      |    2 +-
 common/cli_lil.c  | 1880 ++++++++++++++++++++++++++++++++++++---------
 include/cli_lil.h |   11 +-
 test/cmd/lil.c    |   73 +-
 5 files changed, 1527 insertions(+), 443 deletions(-)
cmd/Kconfig +2 -2
index bba72bbdc2..7ff8e4a7e5 100644
@@ -43,8 +43,8 @@ if LIL
 config LIL_FULL
 	bool "Enable all LIL features"
 	help
-	  This enables all LIL builtin functions, as well as expression support
-	  for arithmetic and bitwise operations.
+	  This enables all LIL builtin functions, expression support for
+	  arithmetic and bitwise operations, and expanded error messages.
 
 config LIL_POOLS
 	bool "Use memory pools for LIL structures"
common/cli.c +1 -1
index ad5d76d563..391fee0ec7 100644
@@ -49,7 +49,7 @@ static const struct lil_callbacks env_callbacks = {
 static int lil_run(const char *cmd)
 {
 	int err;
-	struct lil_value *result = lil_parse(lil, cmd, 0, 0);
+	struct lil_value *result = lil_parse_eval(lil, cmd, 0, true);
 	const char *err_msg, *strres = lil_to_string(result);
 
 	/* The result may be very big, so use puts */
common/cli_lil.c +1498 -382

2,043 lines not on this page · raw message

include/cli_lil.h +7 -4
index 40c822401e..290329372a 100644
@@ -133,10 +133,13 @@ void lil_free(struct lil *lil);
 
 int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc);
 
-struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,
-			    int funclevel);
+struct lil_list *lil_parse(struct lil *lil, const char *code, size_t codelen);
+struct lil_value *lil_eval(struct lil *lil, struct lil_list *script,
+			   bool new_frame);
+struct lil_value *lil_parse_eval(struct lil *lil, const char *code,
+				 size_t codelen, bool new_frame);
 struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,
-				  int funclevel);
+				  bool new_frame);
 
 enum lil_error lil_error(struct lil *lil, const char **msg);
 
@@ -158,7 +161,7 @@ void lil_free_list(struct lil_list *list);
 int lil_list_append(struct lil_list *list, void *item);
 size_t lil_list_size(struct lil_list *list);
 struct lil_value *lil_list_get(struct lil_list *list, size_t index);
-struct lil_value *lil_list_to_value(struct lil_list *list, int do_escape);
+struct lil_value *lil_list_to_value(struct lil_list *list, bool do_escape);
 
 struct lil_list *lil_subst_to_list(struct lil *lil, struct lil_value *code);
 struct lil_value *lil_subst_to_value(struct lil *lil, struct lil_value *code);
test/cmd/lil.c +19 -54

143 lines not on this page · raw message

Signature · 1 line
2.32.0
#93 lore raw

RFC PATCH 22/28 env: Add a priv pointer to hwalk_r

In reply to Sean Anderson

This allows callers of hwalk_r to pass data to their callback. This mirrors e.g. env_attr_walk.

---

 cmd/nvedit.c     | 8 ++++----
 env/callback.c   | 4 ++--
 env/flags.c      | 4 ++--
 include/search.h | 2 +-
 lib/hashtable.c  | 5 +++--
 5 files changed, 12 insertions(+), 11 deletions(-)
cmd/nvedit.c +4 -4
index d14ba10cef..b855e502c0 100644
@@ -481,7 +481,7 @@ static int print_static_binding(const char *var_name, const char *callback_name,
 	return 0;
 }
 
-static int print_active_callback(struct env_entry *entry)
+static int print_active_callback(struct env_entry *entry, void *priv)
 {
 	struct env_clbk_tbl *clbkp;
 	int i;
@@ -544,7 +544,7 @@ int do_env_callback(struct cmd_tbl *cmdtp, int flag, int argc,
 	puts("Active callback bindings:\n");
 	printf("\t%-20s %-20s\n", "Variable Name", "Callback Name");
 	printf("\t%-20s %-20s\n", "-------------", "-------------");
-	hwalk_r(&env_htab, print_active_callback);
+	hwalk_r(&env_htab, print_active_callback, NULL);
 	return 0;
 }
 #endif
@@ -563,7 +563,7 @@ static int print_static_flags(const char *var_name, const char *flags,
 	return 0;
 }
 
-static int print_active_flags(struct env_entry *entry)
+static int print_active_flags(struct env_entry *entry, void *priv)
 {
 	enum env_flags_vartype type;
 	enum env_flags_varaccess access;
@@ -617,7 +617,7 @@ int do_env_flags(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[])
 		"Variable Access");
 	printf("\t%-20s %-20s %-20s\n", "-------------", "-------------",
 		"---------------");
-	hwalk_r(&env_htab, print_active_flags);
+	hwalk_r(&env_htab, print_active_flags, NULL);
 	return 0;
 }
 #endif
env/callback.c +2 -2
index 638a02b28f..47075acb92 100644
@@ -83,7 +83,7 @@ void env_callback_init(struct env_entry *var_entry)
  * Called on each existing env var prior to the blanket update since removing
  * a callback association should remove its callback.
  */
-static int clear_callback(struct env_entry *entry)
+static int clear_callback(struct env_entry *entry, void *priv)
 {
 	entry->callback = NULL;
 
@@ -127,7 +127,7 @@ static int on_callbacks(const char *name, const char *value, enum env_op op,
 	int flags)
 {
 	/* remove all callbacks */
-	hwalk_r(&env_htab, clear_callback);
+	hwalk_r(&env_htab, clear_callback, NULL);
 
 	/* configure any static callback bindings */
 	env_attr_walk(ENV_CALLBACK_LIST_STATIC, set_callback, NULL);
env/flags.c +2 -2
index e3e833c433..f2e36e3dd3 100644
@@ -468,7 +468,7 @@ void env_flags_init(struct env_entry *var_entry)
  * Called on each existing env var prior to the blanket update since removing
  * a flag in the flag list should remove its flags.
  */
-static int clear_flags(struct env_entry *entry)
+static int clear_flags(struct env_entry *entry, void *priv)
 {
 	entry->flags = 0;
 
@@ -503,7 +503,7 @@ static int on_flags(const char *name, const char *value, enum env_op op,
 	int flags)
 {
 	/* remove all flags */
-	hwalk_r(&env_htab, clear_flags);
+	hwalk_r(&env_htab, clear_flags, NULL);
 
 	/* configure any static flags */
 	env_attr_walk(ENV_FLAGS_LIST_STATIC, set_flags, NULL);
include/search.h +1 -1
index d0bb44388e..4a0828fb8d 100644
@@ -105,7 +105,7 @@ int himport_r(struct hsearch_data *htab, const char *env, size_t size,
 
 /* Walk the whole table calling the callback on each element */
 int hwalk_r(struct hsearch_data *htab,
-	    int (*callback)(struct env_entry *entry));
+	    int (*callback)(struct env_entry *entry, void *priv), void *priv);
 
 /* Flags for himport_r(), hexport_r(), hdelete_r(), and hsearch_r() */
 #define H_NOCLEAR	(1 << 0) /* do not clear hash table before importing */
lib/hashtable.c +3 -2
index ff5ff72639..425a880222 100644
@@ -998,14 +998,15 @@ end:
  * Walk all of the entries in the hash, calling the callback for each one.
  * this allows some generic operation to be performed on each element.
  */
-int hwalk_r(struct hsearch_data *htab, int (*callback)(struct env_entry *entry))
+int hwalk_r(struct hsearch_data *htab,
+	    int (*callback)(struct env_entry *entry, void *priv), void *priv)
 {
 	int i;
 	int retval;
 
 	for (i = 1; i <= htab->size; ++i) {
 		if (htab->table[i].used > 0) {
-			retval = callback(&htab->table[i].entry);
+			retval = callback(&htab->table[i].entry, priv);
 			if (retval)
 				return retval;
 		}
Signature · 1 line
2.32.0
#95 lore raw

RFC PATCH 23/28 cli: lil: Handle OOM for hm_put

In reply to Sean Anderson

hm_put allocates memory, and this can fail. Instead of failing silently, return an error code. This also fixes up callers to handle this error.

---

 common/cli_lil.c | 47 ++++++++++++++++++++++++++++++++---------------
 1 file changed, 32 insertions(+), 15 deletions(-)
common/cli_lil.c +32 -15
index 2ed96ebc2d..7ec73675f3 100644
@@ -326,22 +326,30 @@ static void hm_destroy(struct hashmap *hm)
 	}
 }
 
-static void hm_put(struct hashmap *hm, const char *key, void *value)
+static enum lil_error hm_put(struct hashmap *hm, const char *key, void *value)
 {
 	struct hashcell *cell = hm->cell + (hm_hash(key) & HASHMAP_CELLMASK);
+	struct hashentry *newe;
 	size_t i;
 
 	for (i = 0; i < cell->c; i++) {
 		if (!strcmp(key, cell->e[i].k)) {
 			cell->e[i].v = value;
-			return;
+			return LIL_ERR_NONE;
 		}
 	}
 
-	cell->e = realloc(cell->e, sizeof(struct hashentry) * (cell->c + 1));
-	cell->e[cell->c].k = strdup(key);
-	cell->e[cell->c].v = value;
+	newe = realloc(cell->e, sizeof(struct hashentry) * (cell->c + 1));
+	if (!newe)
+		return LIL_ERR_OOM;
+	cell->e = newe;
+
+	newe[cell->c].k = strdup(key);
+	if (!newe[cell->c].k)
+		return LIL_ERR_OOM;
+	newe[cell->c].v = value;
 	cell->c++;
+	return LIL_ERR_NONE;
 }
 
 static void *hm_get(struct hashmap *hm, const char *key)
@@ -738,19 +746,24 @@ static struct lil_func *add_func(struct lil *lil, const char *name)
 
 	cmd = calloc(1, sizeof(struct lil_func));
 	if (!cmd)
-		return NULL;
+		goto oom;
 	cmd->name = strdup(name);
 
 	ncmd = realloc(lil->cmd, sizeof(struct lil_func *) * (lil->cmds + 1));
-	if (!ncmd) {
-		free(cmd);
-		return NULL;
-	}
-
+	if (!ncmd)
+		goto oom;
 	lil->cmd = ncmd;
+
 	ncmd[lil->cmds++] = cmd;
-	hm_put(&lil->cmdmap, name, cmd);
+	if (hm_put(&lil->cmdmap, name, cmd))
+		goto oom;
+
 	return cmd;
+
+oom:
+	free(cmd);
+	lil_set_error_oom(lil);
+	return NULL;
 }
 
 static void del_func(struct lil *lil, struct lil_func *cmd)
@@ -766,7 +779,11 @@ static void del_func(struct lil *lil, struct lil_func *cmd)
 	if (index == lil->cmds)
 		return;
 
-	hm_put(&lil->cmdmap, cmd->name, 0);
+	/*
+	 * The only way this fails is if we don't find the command; this
+	 * means our caller wants to delete a command which doesn't exist
+	 */
+	assert(!hm_put(&lil->cmdmap, cmd->name, NULL));
 	if (cmd->argnames)
 		lil_free_list(cmd->argnames);
 
@@ -783,9 +800,9 @@ int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc)
 	struct lil_func *cmd = add_func(lil, name);
 
 	if (!cmd)
-		return 0;
+		return LIL_ERR_OOM;
 	cmd->proc = proc;
-	return 1;
+	return LIL_ERR_NONE;
 }
 
 struct lil_var *lil_set_var(struct lil *lil, const char *name,
Signature · 1 line
2.32.0
#96 lore raw

RFC PATCH 24/28 cli: lil: Make proc always take 3 arguments

In reply to Sean Anderson

This rewrites proc to always take 3 arguments. It also adds proper error handling. TCL does not allow for anonymous functions to be created with proc. Allowing for a variable number of arguments makes the code much more complex when adding error handling.

Since fnc_proc was the last user of lil_unused_name (other than fnc_unusedname), remove it.

---

 common/cli_lil.c | 103 ++++++++++++++++-------------------------------
 test/cmd/lil.c   |   6 ++-
 2 files changed, 38 insertions(+), 71 deletions(-)
common/cli_lil.c +34 -69
index 7ec73675f3..1c7c340bda 100644
@@ -2985,32 +2985,6 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)
 	return lil_alloc_integer(ee.ival);
 }
 
-struct lil_value *lil_unused_name(struct lil *lil, const char *part)
-{
-	char *name = malloc(strlen(part) + 64);
-	struct lil_value *val;
-	size_t i;
-
-	for (i = 0; i < (size_t)-1; i++) {
-		sprintf(name, "!!un!%s!%09u!nu!!", part, (unsigned int)i);
-		if (lil_find_cmd(lil, name))
-			continue;
-
-		if (lil_find_var(lil, lil->env, name))
-			continue;
-
-		val = lil_alloc_string(name);
-		free(name);
-		return val;
-	}
-	return NULL;
-}
-
-struct lil_value *lil_arg(struct lil_value **argv, size_t index)
-{
-	return argv ? argv[index] : NULL;
-}
-
 const char *lil_to_string(struct lil_value *val)
 {
 	return (val && val->l) ? val->d : "";
@@ -3237,47 +3211,46 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
 static struct lil_value *fnc_proc(struct lil *lil, size_t argc,
 				  struct lil_value **argv)
 {
-	struct lil_value *name;
 	struct lil_func *cmd;
-	struct lil_list *fargs;
+	struct lil_list *args;
+	struct lil_value *name, *code;
 
-	if (argc < 1)
+	if (argc != 3) {
+		lil_set_error_argc(lil, 3);
 		return NULL;
-
-	if (argc >= 3) {
-		name = lil_clone_value(argv[0]);
-		fargs = lil_subst_to_list(lil, argv[1]);
-		cmd = add_func(lil, lil_to_string(argv[0]));
-		if (!cmd)
-			return NULL;
-
-		cmd->argnames = fargs;
-		cmd->code = lil_clone_value(argv[2]);
-	} else {
-		name = lil_unused_name(lil, "anonymous-function");
-		if (argc < 2) {
-			struct lil_value *tmp = lil_alloc_string("args");
-
-			fargs = lil_subst_to_list(lil, tmp);
-			lil_free_value(tmp);
-			cmd = add_func(lil, lil_to_string(name));
-			if (!cmd)
-				return NULL;
-
-			cmd->argnames = fargs;
-			cmd->code = lil_clone_value(argv[0]);
-		} else {
-			fargs = lil_subst_to_list(lil, argv[0]);
-			cmd = add_func(lil, lil_to_string(name));
-			if (!cmd)
-				return NULL;
-
-			cmd->argnames = fargs;
-			cmd->code = lil_clone_value(argv[1]);
-		}
 	}
 
+	name = lil_clone_value(argv[0]);
+	if (!name) {
+		lil_set_error_oom(lil);
+		return NULL;
+	}
+
+	args = lil_subst_to_list(lil, argv[1]);
+	if (!args)
+		goto err_args;
+
+	code = lil_clone_value(argv[2]);
+	if (!code) {
+		lil_set_error_oom(lil);
+		goto err_code;
+	}
+
+	cmd = add_func(lil, lil_to_string(name));
+	if (!cmd)
+		goto err_func;
+	cmd->argnames = args;
+	cmd->code = code;
+
 	return name;
+
+err_func:
+	lil_free_value(code);
+err_code:
+	lil_free_list(args);
+err_args:
+	lil_free_value(name);
+	return NULL;
 }
 
 static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
@@ -3312,13 +3285,6 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
 	return r;
 }
 
-static struct lil_value *fnc_unusedname(struct lil *lil, size_t argc,
-					struct lil_value **argv)
-{
-	return lil_unused_name(lil, argc > 0 ? lil_to_string(argv[0]) :
-						     "unusedname");
-}
-
 static struct lil_value *fnc_quote(struct lil *lil, size_t argc,
 				   struct lil_value **argv)
 {
@@ -4310,7 +4276,6 @@ static void register_stdcmds(struct lil *lil)
 		lil_register(lil, "substr", fnc_substr);
 		lil_register(lil, "topeval", fnc_topeval);
 		lil_register(lil, "trim", fnc_trim);
-		lil_register(lil, "unusedname", fnc_unusedname);
 		lil_register(lil, "upeval", fnc_upeval);
 	}
 }
test/cmd/lil.c +4 -2
index fb33fa83a6..58bc6ee842 100644
@@ -149,9 +149,11 @@ static const struct {
 			"[list eh??]"
 		"];"
 		"asserteq_list [lapply $list length] [list 9 10 4];"
-		"asserteq_list [lapply $list [proc {a} {"
+		"proc firstword {a} {"
 			"return [index [split $a] 0]"
-		"}]] [list {bad's} {good's} eh??]"
+		"};"
+		"asserteq_list [lapply $list firstword] "
+			"[list {bad's} {good's} eh??]"
 	},
 	{"lists",
 		"set l [list foo bar baz bad];"
Signature · 1 line
2.32.0
#97 lore raw

RFC PATCH 25/28 cli: lil: Always quote items in lil_list_to_value

In reply to Sean Anderson

This function took an argument do_quote which determined whether or not to quote an item. All callers set it to true, so remove it and always quote.

---

 common/cli_lil.c  | 34 ++++++++++++++--------------------
 include/cli_lil.h |  2 +-
 2 files changed, 15 insertions(+), 21 deletions(-)
common/cli_lil.c +14 -20
index 1c7c340bda..153c34791b 100644
@@ -2092,7 +2092,7 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,
 		if (cmd->argnames->c == 1 &&
 		    !strcmp(lil_to_string(lil_list_get(cmd->argnames, 0)),
 						       "args")) {
-			struct lil_value *args = lil_list_to_value(words, 1);
+			struct lil_value *args = lil_list_to_value(words);
 
 			lil_set_var(lil, "args", args, LIL_SETVAR_LOCAL_NEW);
 			lil_free_value(args);
@@ -2389,7 +2389,7 @@ braces:
 	return needs;
 }
 
-struct lil_value *lil_list_to_value(struct lil_list *list, bool do_escape)
+struct lil_value *lil_list_to_value(struct lil_list *list)
 {
 	struct lil_value *val = alloc_value(NULL);
 	size_t i, j;
@@ -2397,17 +2397,11 @@ struct lil_value *lil_list_to_value(struct lil_list *list, bool do_escape)
 	for (i = 0; i < list->c; i++) {
 		char q;
 		struct lil_value *item = lil_list_get(list, i);
-		enum needs needs;
-
-		if (do_escape)
-			needs = item_needs(lil_to_string(item), item->l);
-		else
-			needs = NEEDS_NOTHING;
 
 		if (i)
 			lil_append_char(val, ' ');
 
-		switch (needs) {
+		switch (item_needs(lil_to_string(item), item->l)) {
 		case NEEDS_NOTHING:
 			if (lil_append_val(val, item))
 				goto err;
@@ -3087,7 +3081,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
 		func = lil_find_cmd(lil, lil_to_string(argv[1]));
 		if (!func || !func->argnames)
 			return NULL;
-		return lil_list_to_value(func->argnames, 1);
+		return lil_list_to_value(func->argnames);
 	}
 
 	if (!strcmp(type, "body")) {
@@ -3111,7 +3105,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
 			lil_list_append(funcs,
 					lil_alloc_string(lil->cmd[i]->name));
 
-		r = lil_list_to_value(funcs, 1);
+		r = lil_list_to_value(funcs);
 		lil_free_list(funcs);
 		return r;
 	}
@@ -3130,7 +3124,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
 			env = env->parent;
 		}
 
-		r = lil_list_to_value(vars, 1);
+		r = lil_list_to_value(vars);
 		lil_free_list(vars);
 		return r;
 	}
@@ -3145,7 +3139,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,
 			lil_list_append(vars, var);
 		}
 
-		r = lil_list_to_value(vars, 1);
+		r = lil_list_to_value(vars);
 		lil_free_list(vars);
 		return r;
 	}
@@ -3513,7 +3507,7 @@ static struct lil_value *fnc_append(struct lil *lil, size_t argc,
 	for (i = base; i < argc; i++)
 		lil_list_append(list, lil_clone_value(argv[i]));
 
-	r = lil_list_to_value(list, 1);
+	r = lil_list_to_value(list);
 	lil_free_list(list);
 	lil_set_var(lil, varname, r, access);
 	return r;
@@ -3548,7 +3542,7 @@ static struct lil_value *fnc_slice(struct lil *lil, size_t argc,
 		lil_list_append(slice, lil_clone_value(lil_list_get(list, i)));
 	lil_free_list(list);
 
-	r = lil_list_to_value(slice, 1);
+	r = lil_list_to_value(slice);
 	lil_free_list(slice);
 	return r;
 }
@@ -3586,7 +3580,7 @@ static struct lil_value *fnc_filter(struct lil *lil, size_t argc,
 	}
 	lil_free_list(list);
 
-	r = lil_list_to_value(filtered, 1);
+	r = lil_list_to_value(filtered);
 	lil_free_list(filtered);
 	return r;
 }
@@ -3601,7 +3595,7 @@ static struct lil_value *fnc_list(struct lil *lil, size_t argc,
 	for (i = 0; i < argc; i++)
 		lil_list_append(list, lil_clone_value(argv[i]));
 
-	r = lil_list_to_value(list, 1);
+	r = lil_list_to_value(list);
 	lil_free_list(list);
 	return r;
 }
@@ -3628,7 +3622,7 @@ static struct lil_value *fnc_concat(struct lil *lil, size_t argc,
 	r = lil_alloc_string("");
 	for (i = 0; i < argc; i++) {
 		list = lil_subst_to_list(lil, argv[i]);
-		tmp = lil_list_to_value(list, 1);
+		tmp = lil_list_to_value(list);
 		lil_free_list(list);
 		lil_append_val(r, tmp);
 		lil_free_value(tmp);
@@ -3672,7 +3666,7 @@ static struct lil_value *fnc_foreach(struct lil *lil, size_t argc,
 			break;
 	}
 
-	r = lil_list_to_value(rlist, 1);
+	r = lil_list_to_value(rlist);
 	lil_free_list(list);
 	lil_free_list(rlist);
 	return r;
@@ -4147,7 +4141,7 @@ static struct lil_value *fnc_split(struct lil *lil, size_t argc,
 	}
 
 	lil_list_append(list, val);
-	val = lil_list_to_value(list, 1);
+	val = lil_list_to_value(list);
 	lil_free_list(list);
 	return val;
 }
include/cli_lil.h +1 -1
index 290329372a..6fbc270f1b 100644
@@ -161,7 +161,7 @@ void lil_free_list(struct lil_list *list);
 int lil_list_append(struct lil_list *list, void *item);
 size_t lil_list_size(struct lil_list *list);
 struct lil_value *lil_list_get(struct lil_list *list, size_t index);
-struct lil_value *lil_list_to_value(struct lil_list *list, bool do_escape);
+struct lil_value *lil_list_to_value(struct lil_list *list);
 
 struct lil_list *lil_subst_to_list(struct lil *lil, struct lil_value *code);
 struct lil_value *lil_subst_to_value(struct lil *lil, struct lil_value *code);
Signature · 1 line
2.32.0
#98 lore raw

RFC PATCH 26/28 cli: lil: Allocate len even when str is NULL in alloc_value_len

In reply to Sean Anderson

This allows us to reserve some space ahead of time, avoiding another alloc/copy/free.

---

 common/cli_lil.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
common/cli_lil.c +6 -8
index 153c34791b..42659920b5 100644
@@ -459,21 +459,19 @@ static struct lil_value *alloc_value_len(const char *str, size_t len)
 	if (!val)
 		return NULL;
 	value_to_symbol(val)->type = LIL_SYMBOL_VALUE;
+	ensure_capacity(val, len + 1);
+	if (!val->d) {
+		release_to_pool(val);
+		return NULL;
+	}
 
 	if (str) {
 		val->l = len;
-		ensure_capacity(val, len + 1);
-		if (!val->d) {
-			release_to_pool(val);
-			return NULL;
-		}
 		memcpy(val->d, str, len);
 		val->d[len] = 0;
 	} else {
 		val->l = 0;
-		ensure_capacity(val, 1);
-		if (val->d)
-			val->d[0] = '\0';
+		val->d[0] = '\0';
 	}
 	return val;
 }
Signature · 1 line
2.32.0
#99 lore raw

RFC PATCH 27/28 cli: lil: Add a function to quote values

In reply to Sean Anderson

This allows us to convert lil_values into a form which can be re-parsed. We were already doing this for lists, so we just have to expose the inner loop.

---

 common/cli_lil.c | 89 ++++++++++++++++++++++++++++--------------------
 1 file changed, 52 insertions(+), 37 deletions(-)
common/cli_lil.c +52 -37
index 42659920b5..2a8600ffb6 100644
@@ -2387,49 +2387,64 @@ braces:
 	return needs;
 }
 
+static enum lil_error do_quote(struct lil_value *val, struct lil_value *item)
+{
+	char q;
+	size_t i;
+
+	switch (item_needs(lil_to_string(item), item->l)) {
+	case NEEDS_NOTHING:
+		return lil_append_val(val, item);
+	case NEEDS_BRACES:
+		return lil_append_char(val, '{') ?:
+			lil_append_val(val, item) ?:
+			lil_append_char(val, '}');
+	case NEEDS_DOUBLE:
+		q = '"';
+		goto quote;
+	case NEEDS_SINGLE:
+		q = '\'';
+quote:
+		if (lil_append_char(val, q))
+			return LIL_ERR_OOM;
+
+		for (i = 0; i < item->l; i++) {
+			char c = item->d[i];
+
+			if (c == '\\' || c == q)
+				if (lil_append_char(val, '\\'))
+					return LIL_ERR_OOM;
+			if (lil_append_char(val, c))
+				return LIL_ERR_OOM;
+		}
+		if (lil_append_char(val, q))
+			return LIL_ERR_OOM;
+	}
+	return LIL_ERR_NONE;
+}
+
+static struct lil_value *lil_quote_value(struct lil_value *val)
+{
+	struct lil_value *r = alloc_value(NULL);
+
+	if (do_quote(r, val)) {
+		lil_free_value(r);
+		return NULL;
+	}
+	return r;
+}
+
 struct lil_value *lil_list_to_value(struct lil_list *list)
 {
 	struct lil_value *val = alloc_value(NULL);
-	size_t i, j;
+	size_t i;
 
 	for (i = 0; i < list->c; i++) {
-		char q;
-		struct lil_value *item = lil_list_get(list, i);
+		if (i && lil_append_char(val, ' '))
+			goto err;
 
-		if (i)
-			lil_append_char(val, ' ');
-
-		switch (item_needs(lil_to_string(item), item->l)) {
-		case NEEDS_NOTHING:
-			if (lil_append_val(val, item))
-				goto err;
-			continue;
-		case NEEDS_BRACES:
-			if (lil_append_char(val, '{') ||
-			    lil_append_val(val, item) ||
-			    lil_append_char(val, '}'))
-				goto err;
-			continue;
-		case NEEDS_DOUBLE:
-			q = '"';
-			goto quote;
-		case NEEDS_SINGLE:
-			q = '\'';
-quote:
-			if (lil_append_char(val, q))
-				goto err;
-			for (j = 0; j < item->l; j++) {
-				char c = item->d[j];
-
-				if (c == '\\' || c == q)
-					if (lil_append_char(val, '\\'))
-						goto err;
-				if (lil_append_char(val, c))
-					goto err;
-			}
-			if (lil_append_char(val, q))
-				goto err;
-		}
+		if (do_quote(val, lil_list_get(list, i)))
+			goto err;
 	}
 	return val;
 
Signature · 1 line
2.32.0
#100 lore raw

RFC PATCH 28/28 cli: lil: Load procs from the environment

In reply to Sean Anderson

When we start up the LIL interpreter, go through every variable and see if it looks like a new procedure. If it does, try and parse it. For the return trip, every time that we create a new procedure, create a new global variable containing that procedure.

The end result of this is that procedures should now be saved to the environment in the same way that variables are. So you can do

	=> proc foo {args} { ... }
	=> env save

and foo will be there after you reboot.

---

 common/cli.c      | 54 ++++++++++++++++++++++++++++-------
 common/cli_lil.c  | 71 ++++++++++++++++++++++++++++++++++-------------
 include/cli_lil.h | 17 ++++++++++++
 3 files changed, 113 insertions(+), 29 deletions(-)
common/cli.c +44 -10
index 391fee0ec7..c71f75e684 100644
@@ -16,9 +16,11 @@
 #include <command.h>
 #include <console.h>
 #include <env.h>
+#include <env_internal.h>
 #include <fdtdec.h>
 #include <hang.h>
 #include <malloc.h>
+#include <search.h>
 #include <asm/global_data.h>
 
 DECLARE_GLOBAL_DATA_PTR;
@@ -26,6 +28,21 @@ DECLARE_GLOBAL_DATA_PTR;
 #ifdef CONFIG_LIL
 static struct lil *lil;
 
+static enum lil_error print_lil_err(struct lil *lil)
+{
+	enum lil_error ret;
+	const char *err_msg;
+
+	ret = lil_error(lil, &err_msg);
+	if (ret) {
+		if (err_msg)
+			printf("error: %s\n", err_msg);
+		else
+			printf("error: %d\n", ret);
+	}
+	return ret;
+}
+
 static int env_setvar(struct lil *lil, const char *name,
 		      struct lil_value **value)
 {
@@ -41,16 +58,40 @@ static int env_getvar(struct lil *lil, const char *name,
 	return 1;
 }
 
+static int env_register_proc(struct env_entry *entry, void *priv)
+{
+	struct lil *lil = priv;
+	struct lil_value *name;
+	const char *name_str, prefix[] = "proc";
+
+	/* Skip variables which are obviously not procedures */
+	if (strncmp(entry->data, prefix, sizeof(prefix) - 1))
+		return 0;
+
+	name = lil_parse_eval(lil, entry->data, 0, true);
+	name_str = lil_to_string(name);
+	if (strcmp(entry->key, name_str))
+		log_debug("proc %s created by variable %s\n",
+			  name_str, entry->key);
+	lil_free_value(name);
+	return print_lil_err(lil);
+}
+
+static int env_initprocs(struct lil *lil)
+{
+	return hwalk_r(&env_htab, env_register_proc, lil);
+}
+
 static const struct lil_callbacks env_callbacks = {
 	.setvar = env_setvar,
 	.getvar = env_getvar,
+	.initprocs = env_initprocs,
 };
 
 static int lil_run(const char *cmd)
 {
-	int err;
 	struct lil_value *result = lil_parse_eval(lil, cmd, 0, true);
-	const char *err_msg, *strres = lil_to_string(result);
+	const char *strres = lil_to_string(result);
 
 	/* The result may be very big, so use puts */
 	if (strres && strres[0]) {
@@ -59,14 +100,7 @@ static int lil_run(const char *cmd)
 	}
 	lil_free_value(result);
 
-	err = lil_error(lil, &err_msg);
-	if (err) {
-		if (err_msg)
-			printf("error: %s\n", err_msg);
-		else
-			printf("error: %d\n", err);
-	}
-	return !!err;
+	return !!print_lil_err(lil);
 }
 #endif
 
common/cli_lil.c +52 -19

107 lines not on this page · raw message

include/cli_lil.h +17 -0
index 6fbc270f1b..47a2eeefc9 100644
@@ -94,6 +94,23 @@ struct lil_callbacks {
 	 */
 	int (*getvar)(struct lil *lil, const char *name,
 		      struct lil_value **value);
+
+	/**
+	 * @initprocs: Called when procedures should be created
+	 *
+	 * @lil: The LIL interpreter
+	 *
+	 * This is called once in lil_new() when user functions should be added.
+	 * When this is called, the only function registered is "proc". You can
+	 * register functions by calling lil_parse_eval() with code which
+	 * initializes a procedure.
+	 *
+	 * In practice, this is expected to be used to initialize procedures
+	 * from environmental variables.
+	 *
+	 * @Return: 0 if ok, or non-zero on error.
+	 */
+	int (*initprocs)(struct lil *lil);
 };
 
 /**
Signature · 1 line
2.32.0
#101 lore raw
In reply to Sean Anderson
Quoted 7 lines · Sean Anderson

Well, this has been sitting on my hard drive for too long without feedback ("Release early, release often"), so here's the first RFC. This is not ready to merge (see the "Future work" section below), but the shell is functional and at least partially tested.

The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes over on sandbox.

A good goal, but perhaps slightly too strict?

Quoted 66 lines

add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)

= Getting started

Enable CONFIG_LIL. If you would like to run tests, enable CONFIG_LIL_FULL. Note that dm_test_acpi_cmd_dump and setexpr_test_str_oper will fail. CONFIG_LIL_POOLS is currently broken (with what appears to be a double free).

For an overview of the language as a whole, refer to the original readme [1].

[1] http://runtimeterror.com/tech/lil/readme.txt

== Key patches

The following patches are particularly significant for reviewing and understanding this series:

cli: Add LIL shell
This contains the LIL shell as originally written by Kostas with some major deletions and some minor additions.
cli: lil: Wire up LIL to the rest of U-Boot
This allows you to use LIL as a shell just like Hush.
cli: lil: Document structures
This adds documentation for the major structures of LIL. It is a good place to start looking at the internals.
test: Add tests for LIL
This adds some basic integration tests and provides some examples of LIL code.
cli: lil: Add a distinct parsing step
This adds a parser separate from the interpreter. This patch is the largest original work in this series.
cli: lil: Load procs from the environment
This allows procedures to be saved and loaded like variables.

= A new shell

This series adds a new shell for U-Boot. The aim is to eventually replace Hush as the primary shell for all boards which currently use it. Hush should be replaced because it has several major problems:

- It has not had a major update in two decades, resulting in duplication of effort in finding bugs. Regarding a bug in variable setting, Wolfgang remarks

So the specific problem has (long) been fixed in upstream, and instead of adding a patch to our old version, thus cementing the broken behaviour, we should upgrade hush to recent upstream code.

-- Wolfgang Denk [2]

These lack of updates are further compounded by a significant amount of ifdef-ing in the Hush code. This makes the shell hard to read and debug. Further, the original purpose of such ifdef-ing (upgrading to a newer Hush) has never happened.

- It was designed for a preempting OS which supports pipes and processes. This fundamentally does not match the computing model of U-Boot where there is exactly one thread (and every other CPU is spinning or sleeping). Working around these design differences is a significant cause of the aformentioned ifdef-ing.

- It lacks many major features expected of even the most basic shells, such as functions and command substitution ($() syntax). This makes it difficult to script with Hush. While it is desirable to write some code in C, much code *must* be written in C because there is no way to express the logic in Hush.

Quoted 5 lines

I believe that U-Boot should have a shell which is more featureful, has cleaner code, and which is the same size as Hush (or less). The ergonomic advantages afforded by a new shell will make U-Boot easier to use and customize.

[2] https://lore.kernel.org/u-boot/[email protected]/

First, great! Thanks for doing this. A new shell really is the only viable path forward here, and I appreciate you taking the time to evaluate several and implement one.

Quoted 29 lines

= Open questions

While the primary purpose of this series is of course to get feedback on the code I have already written, there are several decisions where I am not sure what the best course of action is.

- What should be done about 'expr'? The 'expr' command is a significant portion of the final code size. It cannot be removed outright, because it is used by several builtin functions like 'if', 'while', 'for', etc. The way I see it, there are two general approaches to take

- Rewrite expr to parse expressions and then evaluate them. The parsing could re-use several of the existing parse functions like how parse_list does. This could reduce code, as instead of many functions each with their own while/switch statements, we could have two while/switch statements (one to parse, and one to evaluate). However, this may end up increasing code size (such as when the main language had evaluation split from parsing).

- Don't parse infix expressions, and just make arithmetic operators normal functions. This would affect ergonomics a bit. For example, instead of

if {$i < 10} { ... }

one would need to write

if {< $i 10} { ... }

and instead of

Quoted 8 lines

if {$some_bool} { ... }

one would need to write

if {quote $some_bool} { ... }

Though, given how much setexpr is used (not much), this may not be such a big price to pay. This route is almost certain to reduce code size.

So, this is a question because we have cmd/setexpr.c that provides "expr" today? Or because this is a likely place to reclaim some of that 800 byte growth?

Quoted 9 lines

- How should LIL functions integrate with the rest of U-Boot? At the moment, lil functions and procedures exist in a completely separate world from normal commands. I would like to integrate them more closely, but I am not sure the best way to go about this. At the very minimum, each LIL builtin function needs to get its hands on the LIL interpreter somehow. I'd rather this didn't happen through gd_t or similar so that it is easier to unit test. Additionally, LIL functions expect an array of lil_values instead of strings. We could strip them out, but I worry that might start to impact performance (from all the copying).

I might be missing something here. But, given that whenever we have C code run-around and generate a string to then pass to the interpreter to run, someone asks why we don't just make API calls directly, perhaps the answer is that we don't need to?

Quoted 77 lines

The other half of this is adding LIL features into regular commands. The most important feature here is being able to return a string result. I took an initial crack at it [3], but I think with this series there is a stronger motivating factor (along with things like [4]).

[3] https://patchwork.ozlabs.org/project/uboot/list/?series=231377
[4] https://patchwork.ozlabs.org/project/uboot/list/?series=251013

= Future work

The series as presented today is incomplete. The following are the major issues I see with it at the moment. I would like to address all of these issues, but some of them might be postponed until after first merging this series.

- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly, there still remains a significant amount of original code which just ignores errors. In particular, I would like to ensure that the following categories of error conditions are handled:

- Running out of memory.
- Access to a nonexistant variable.
- Passing the wrong number of arguments to a function.
- Interpreting a value as the wrong type (e.g. "foo" should not have a numeric representation, instead of just being treated as 1).

- There are many deviations from TCL with no purpose. For example, the list indexing function is named "index" and not "lindex". It is perfectly fine to drop features or change semantics to reduce code size, make parsing easier, or make execution easier. But changing things for the sake of it should be avoided.

- The test suite is rather anemic compared with the amount of code this series introduces. I would like to expand it significantly. In particular, error conditions are not well tested (only the "happy path" is tested).

- While I have documented all new functions I have written, there are many existing functions which remain to be documented. In addition, there is no user documentation, which is critical in driving adoption of any new programming language. Some of this cover letter might be integrated with any documentation written.

- Some shell features such as command repetition and secondary shell prompts have not been implemented.

- Arguments to native lil functions are incompatible with U-Boot functions. For example, the command

foo bar baz

would be passed to a U-Boot command as

	{ "foo", "bar", "baz", NULL }

but would be passed to a LIL function as

	{ "bar", "baz" }

This makes it more difficult to use the same function to parse several different commands. At the moment this is solved by passing the command name in lil->env->proc, but I would like to switch to the U-Boot argument list style.

- Several existing tests break when using LIL because they expect no output on failure, but LIL produces some output notifying the user of the failure.

- Implement DISTRO_BOOT in LIL. I think this is an important proof-of-concept to show what can be done with LIL, and to determine which features should be moved to LIL_FULL.

= Why Lil?

When looking for a suitable replacement shell, I evaluated implementations using the following criteria:

Quoted 13 lines

- It must have a GPLv2-compatible license.
- It must be written in C, and have no major external dependencies.
- It must support bare function calls. That is, a script such as 'foo bar' should invoke the function 'foo' with the argument 'bar'. This preserves the shell-like syntax we expect.
- It must be small. The eventual target is that it compiles to around 10KiB with -Os and -ffunction-sections.
- There should be good tests. Any tests at all are good, but a functioning suite is better.
- There should be good documentation
- There should be comments in the source.
- It should be "finished" or have only slow development. This will hopefully make it easier to port changes.

On this last point, I believe this is based on lil20190821 and current is now lil20210502. With a quick diff between them, I can see that the changes there are small enough that while you've introduced a number of changes here, it would be a very easy update.

Quoted 79 lines

Notably absent from the above list is performance. Most scripts in U-Boot will be run once on boot. As long as the time spent evaluating scripts is kept under a reasonable threshold (a fraction of the time spend initializing hardware or reading data from persistant storage), there is no need to optimize for speed.

In addition, I did not consider updating Hush from Busybox. The mismatch in computing environment expectations (as noted in the "New shell" section above) still applies. IMO, this mismatch is the biggest reason that things like functions and command substitution have been excluded from the U-Boot's Hush.

== lil

- zLib
- TCL
- Compiles to around 10k with no builtins. To 25k with builtins.
- Some tests, but not organized into a suite with expected output. Some evidence that the author ran APL, but no harness.
- Some architectural documentation. Some for each functions, but not much.
- No comments :l
- 3.5k LoC

== picol

- 2-clause BSD
- TCL
- Compiles to around 25k with no builtins. To 80k with builtins.
- Tests with suite (in-language). No evidence of fuzzing.
- No documentation :l
- No comments :l
- 5k LoC

== jimtcl

- 2-clause BSD
- TCL
- Compiles to around 95k with no builtins. To 140k with builtins. Too big...

== boron

- LGPLv3+ (so this is right out)
- REBOL
- Compiles to around 125k with no builtins. To 190k with builtins. Too big...

== libmawk

- GPLv2
- Awk
- Compiles to around 225k. Too big...

== libfawk

- 3-clause BSD
- Uses bison+yacc...
- Awk; As it turns out, this has parentheses for function calls.
- Compiles to around 24-30k. Not sure how to remove builtins.
- Test suite (in-language). No fuzzing.
- Tutorial book. No function reference.
- No comments
- Around 2-4k LoC

== MicroPython

- MIT
- Python (but included for completeness)
- Compiles to around 300k. Too big...

== mruby/c

- 3-clause BSD
- Ruby
- Compiles to around 85k without builtins and 120k with. Too big...

== eLua

- MIT
- Lua
- Build system is a royal pain (custom and written in Lua with external deps)
- Base binary is around 250KiB and I don't want to deal with reducing it

Quoted 8 lines

So the interesting/viable ones are
- lil
- picol
- libfawk (maybe)

I started with LIL because it was the smallest. I have found several issues with LIL along the way. Some of these are addressed in this series already, while others remain unaddressed (see the section "Future Work").

Thanks for the evaluations, of these, lil does make the most sense.

Signature · 1 line
Tom
#102 lore raw

Dear Tom,

Quoted 4 lines · In message <20210701202155.GQ9516@bill-the-cat> you wrote:

First, great! Thanks for doing this. A new shell really is the only viable path forward here, and I appreciate you taking the time to evaluate several and implement one.

I disagree that a new shell is the _only_ way forward.

AFAICT, all the raised concerns have long been fixed in upstream versions of hush; see for example [1]:

...
//config: hush is a small shell. It handles the normal flow control
//config: constructs such as if/then/elif/else/fi, for/in/do/done, while loops,
//config: case/esac. Redirections, here documents, $((arithmetic))
//config: and functions are supported.
//config:

[1] https://git.busybox.net/busybox/tree/shell/hush.c#n98

My gut feeling is that updating to a recent version of hush is the most efficent _backward_compatible_ way.

And if we drop that requirement, we might even take a bigger step and move to lua - which would allow for a complete new level of script based extensions.

Quoted 6 lines
Quoted 6 lines

- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly, there still remains a significant amount of original code which just ignores errors. In particular, I would like to ensure that the following categories of error conditions are handled:

This is something that scares me like hell. This in a shell? For me this is close to a killing point.

Quoted 5 lines
Quoted 5 lines

- Running out of memory.
- Access to a nonexistant variable.
- Passing the wrong number of arguments to a function.
- Interpreting a value as the wrong type (e.g. "foo" should not have a numeric representation, instead of just being treated as 1).

Who says so?

Bash says:

	-> printf "%d\n" foo
	-bash: printf: foo: invalid number
	0

So it is _not_ 1 ...

Quoted 5 lines
Quoted 5 lines

- There are many deviations from TCL with no purpose. For example, the list indexing function is named "index" and not "lindex". It is perfectly fine to drop features or change semantics to reduce code size, make parsing easier, or make execution easier. But changing things for the sake of it should be avoided.

It's not a standard POSIX shell, it's not TCL (ick!), ... it's something new, incompatible...

Quoted 1 line

Thanks for the evaluations, of these, lil does make the most sense.

You mean, adding a complete new, incompatible and non-standard shell is a better approach than updating to a recent version of hush?

What makes you think so?

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
Ernest asks Frank how long he has been working for the company.
        "Ever since they threatened to fire me."
#103 lore raw
Quoted 13 lines · Wolfgang Denk

Dear Tom,

Quoted 4 lines · In message <20210701202155.GQ9516@bill-the-cat> you wrote:

First, great! Thanks for doing this. A new shell really is the only viable path forward here, and I appreciate you taking the time to evaluate several and implement one.

I disagree that a new shell is the _only_ way forward.

AFAICT, all the raised concerns have long been fixed in upstream versions of hush; see for example [1]:

Quoted 8 lines

...
//config: hush is a small shell. It handles the normal flow control
//config: constructs such as if/then/elif/else/fi, for/in/do/done, while loops,
//config: case/esac. Redirections, here documents, $((arithmetic))
//config: and functions are supported.
//config:

[1] https://git.busybox.net/busybox/tree/shell/hush.c#n98

In fact, the code for most of this is present but ifdef'd out. The real issue is that the implementation of much of the above relies on things like fork() which we can't provide.

Quoted 6 lines

My gut feeling is that updating to a recent version of hush is the most efficent _backward_compatible_ way.

And if we drop that requirement, we might even take a bigger step and move to lua - which would allow for a complete new level of script based extensions.

I'm not aware of any Lua implementations which meet the size requirements for U-Boot. Any port would need to be either written from scratch or have some serious modification, not unlike what I've done here.

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines

- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly, there still remains a significant amount of original code which just ignores errors. In particular, I would like to ensure that the following categories of error conditions are handled:

This is something that scares me like hell. This in a shell? For me this is close to a killing point.

Yes, it was for me as well. But it is not something so obvious unless you are looking for it. I believe I have addressed this issue in much of the core code (parser and interpreter). But the builtin commands must be gone through and converted.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

- Running out of memory.
- Access to a nonexistant variable.
- Passing the wrong number of arguments to a function.
- Interpreting a value as the wrong type (e.g. "foo" should not have a numeric representation, instead of just being treated as 1).

Who says so?

The current LIL code.

Quoted 8 lines

Bash says:

	-> printf "%d\n" foo
	-bash: printf: foo: invalid number
	0

So it is _not_ 1 ...

And I would like to replicate this behavior.

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines

- There are many deviations from TCL with no purpose. For example, the list indexing function is named "index" and not "lindex". It is perfectly fine to drop features or change semantics to reduce code size, make parsing easier, or make execution easier. But changing things for the sake of it should be avoided.

It's not a standard POSIX shell, it's not TCL (ick!), ... it's something new, incompatible...

And Hush isn't POSIX either :)

But as noted above, I would like to hew much closer to TCL than LIL has traditionally done.

Quoted 4 lines
Quoted 1 line

Thanks for the evaluations, of these, lil does make the most sense.

You mean, adding a complete new, incompatible and non-standard shell is a better approach than updating to a recent version of hush?

Correct.

Quoted 1 line

What makes you think so?

Hush relies heavily on its posix environment. Porting it has, and will, require substantial modification. IMO the work necessary will be around the same or more for Hush as for any other language.

--Sean

#104 lore raw
In reply to Tom Rini
Quoted 11 lines · Tom Rini
Quoted 7 lines · Sean Anderson

Well, this has been sitting on my hard drive for too long without feedback ("Release early, release often"), so here's the first RFC. This is not ready to merge (see the "Future work" section below), but the shell is functional and at least partially tested.

The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes over on sandbox.

A good goal, but perhaps slightly too strict?

Perhaps. But I think getting in the ballpark will significantly help drive adoption. I want to make it as easy as possible for maintainers to enable LIL and start using it.

Quoted 110 lines
Quoted 67 lines

add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)

= Getting started

Enable CONFIG_LIL. If you would like to run tests, enable CONFIG_LIL_FULL. Note that dm_test_acpi_cmd_dump and setexpr_test_str_oper will fail. CONFIG_LIL_POOLS is currently broken (with what appears to be a double free).

For an overview of the language as a whole, refer to the original readme [1].

[1] http://runtimeterror.com/tech/lil/readme.txt

== Key patches

The following patches are particularly significant for reviewing and understanding this series:

cli: Add LIL shell
This contains the LIL shell as originally written by Kostas with some major deletions and some minor additions.
cli: lil: Wire up LIL to the rest of U-Boot
This allows you to use LIL as a shell just like Hush.
cli: lil: Document structures
This adds documentation for the major structures of LIL. It is a good place to start looking at the internals.
test: Add tests for LIL
This adds some basic integration tests and provides some examples of LIL code.
cli: lil: Add a distinct parsing step
This adds a parser separate from the interpreter. This patch is the largest original work in this series.
cli: lil: Load procs from the environment
This allows procedures to be saved and loaded like variables.

= A new shell

This series adds a new shell for U-Boot. The aim is to eventually replace Hush as the primary shell for all boards which currently use it. Hush should be replaced because it has several major problems:

- It has not had a major update in two decades, resulting in duplication of effort in finding bugs. Regarding a bug in variable setting, Wolfgang remarks

So the specific problem has (long) been fixed in upstream, and instead of adding a patch to our old version, thus cementing the broken behaviour, we should upgrade hush to recent upstream code.

-- Wolfgang Denk [2]

These lack of updates are further compounded by a significant amount of ifdef-ing in the Hush code. This makes the shell hard to read and debug. Further, the original purpose of such ifdef-ing (upgrading to a newer Hush) has never happened.

- It was designed for a preempting OS which supports pipes and processes. This fundamentally does not match the computing model of U-Boot where there is exactly one thread (and every other CPU is spinning or sleeping). Working around these design differences is a significant cause of the aformentioned ifdef-ing.

- It lacks many major features expected of even the most basic shells, such as functions and command substitution ($() syntax). This makes it difficult to script with Hush. While it is desirable to write some code in C, much code *must* be written in C because there is no way to express the logic in Hush.

Quoted 5 lines

I believe that U-Boot should have a shell which is more featureful, has cleaner code, and which is the same size as Hush (or less). The ergonomic advantages afforded by a new shell will make U-Boot easier to use and customize.

[2] https://lore.kernel.org/u-boot/[email protected]/

First, great! Thanks for doing this. A new shell really is the only viable path forward here, and I appreciate you taking the time to evaluate several and implement one.

Quoted 33 lines

= Open questions

While the primary purpose of this series is of course to get feedback on the code I have already written, there are several decisions where I am not sure what the best course of action is.

- What should be done about 'expr'? The 'expr' command is a significant portion of the final code size. It cannot be removed outright, because it is used by several builtin functions like 'if', 'while', 'for', etc. The way I see it, there are two general approaches to take

- Rewrite expr to parse expressions and then evaluate them. The parsing could re-use several of the existing parse functions like how parse_list does. This could reduce code, as instead of many functions each with their own while/switch statements, we could have two while/switch statements (one to parse, and one to evaluate). However, this may end up increasing code size (such as when the main language had evaluation split from parsing).

- Don't parse infix expressions, and just make arithmetic operators normal functions. This would affect ergonomics a bit. For example, instead of

if {$i < 10} { ... }

one would need to write

if {< $i 10} { ... }

and instead of

if {$some_bool} { ... }

one would need to write

Quoted 8 lines
Quoted 4 lines

if {quote $some_bool} { ... }

Though, given how much setexpr is used (not much), this may not be such a big price to pay. This route is almost certain to reduce code size.

So, this is a question because we have cmd/setexpr.c that provides "expr" today? Or because this is a likely place to reclaim some of that 800 byte growth?

The latter. setexpr cannot be used because it does not return a result, and instead sets a (global) variable. The expression parsing functionality is core to LIL and used in many builtin commands (such as if above), and really needs to return a lil_value.

Quoted 11 lines
Quoted 10 lines

- How should LIL functions integrate with the rest of U-Boot? At the moment, lil functions and procedures exist in a completely separate world from normal commands. I would like to integrate them more closely, but I am not sure the best way to go about this. At the very minimum, each LIL builtin function needs to get its hands on the LIL interpreter somehow. I'd rather this didn't happen through gd_t or similar so that it is easier to unit test. Additionally, LIL functions expect an array of lil_values instead of strings. We could strip them out, but I worry that might start to impact performance (from all the copying).

Quoted 4 lines

I might be missing something here. But, given that whenever we have C code run-around and generate a string to then pass to the interpreter to run, someone asks why we don't just make API calls directly, perhaps the answer is that we don't need to?

err, the issue here is that the signature for regular commands is rougly

	int cmd(..., int argc, char **argv, ...)

and the signature for LIL commands is

struct lil_value *cmd(struct lil *lil, size_t argc, struct lil_value **argv)

where lil_value is

	struct lil_value {
		size_t l;
		char *d;
	};

so while regular commands can be reimplemented as LIL commands (just create a new argv containing the strings directly), it is more difficult to go the other way. I bring this up because I think having two separate ways to write a command is not the best way to do things going forward.

Quoted 91 lines
Quoted 77 lines

The other half of this is adding LIL features into regular commands. The most important feature here is being able to return a string result. I took an initial crack at it [3], but I think with this series there is a stronger motivating factor (along with things like [4]).

[3] https://patchwork.ozlabs.org/project/uboot/list/?series=231377
[4] https://patchwork.ozlabs.org/project/uboot/list/?series=251013

= Future work

The series as presented today is incomplete. The following are the major issues I see with it at the moment. I would like to address all of these issues, but some of them might be postponed until after first merging this series.

- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly, there still remains a significant amount of original code which just ignores errors. In particular, I would like to ensure that the following categories of error conditions are handled:

- Running out of memory.
- Access to a nonexistant variable.
- Passing the wrong number of arguments to a function.
- Interpreting a value as the wrong type (e.g. "foo" should not have a numeric representation, instead of just being treated as 1).

- There are many deviations from TCL with no purpose. For example, the list indexing function is named "index" and not "lindex". It is perfectly fine to drop features or change semantics to reduce code size, make parsing easier, or make execution easier. But changing things for the sake of it should be avoided.

- The test suite is rather anemic compared with the amount of code this series introduces. I would like to expand it significantly. In particular, error conditions are not well tested (only the "happy path" is tested).

- While I have documented all new functions I have written, there are many existing functions which remain to be documented. In addition, there is no user documentation, which is critical in driving adoption of any new programming language. Some of this cover letter might be integrated with any documentation written.

- Some shell features such as command repetition and secondary shell prompts have not been implemented.

- Arguments to native lil functions are incompatible with U-Boot functions. For example, the command

foo bar baz

would be passed to a U-Boot command as

	{ "foo", "bar", "baz", NULL }

but would be passed to a LIL function as

	{ "bar", "baz" }

This makes it more difficult to use the same function to parse several different commands. At the moment this is solved by passing the command name in lil->env->proc, but I would like to switch to the U-Boot argument list style.

- Several existing tests break when using LIL because they expect no output on failure, but LIL produces some output notifying the user of the failure.

- Implement DISTRO_BOOT in LIL. I think this is an important proof-of-concept to show what can be done with LIL, and to determine which features should be moved to LIL_FULL.

= Why Lil?

When looking for a suitable replacement shell, I evaluated implementations using the following criteria:

Quoted 13 lines

- It must have a GPLv2-compatible license.
- It must be written in C, and have no major external dependencies.
- It must support bare function calls. That is, a script such as 'foo bar' should invoke the function 'foo' with the argument 'bar'. This preserves the shell-like syntax we expect.
- It must be small. The eventual target is that it compiles to around 10KiB with -Os and -ffunction-sections.
- There should be good tests. Any tests at all are good, but a functioning suite is better.
- There should be good documentation
- There should be comments in the source.
- It should be "finished" or have only slow development. This will hopefully make it easier to port changes.

Quoted 4 lines

On this last point, I believe this is based on lil20190821 and current is now lil20210502. With a quick diff between them, I can see that the changes there are small enough that while you've introduced a number of changes here, it would be a very easy update.

From what I understand, the only changes are updated copyrights and the addition of a license file to cover the tests.

--Sean

#105 lore raw
In reply to Sean Anderson

Hello Sean,

Quoted 9 lines · Sean Anderson

Well, this has been sitting on my hard drive for too long without feedback ("Release early, release often"), so here's the first RFC. This is not ready to merge (see the "Future work" section below), but the shell is functional and at least partially tested.

The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes over on sandbox.

add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)

Thanks for this comparison. Here just my thoughts, but I have currently to low time to follow here completly, so sorry if I miss stuff already discussed.

From my side of view we can add a second shell beside hush, and as you already mentioned it may is even possible to start LIL from hush. This would be cool.

But I disagree in the following points:

- I do not see why LIL should *replace* hush. This may break a lot of current boards. Also I see no need to do this, as if you do complex stuff in u-boot shell it may is the wrong way and you have to think about... and I see no big discussions here about BUGS in our current hush implementation ... may it makes sense to bring them up and fix?

- Is LIL really so good code, that it is done? Or is there only no community which develop further?

- Do we really want code, which does no error checking? Commentless code...

Quoted 3 lines · you

- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly,

Which really let me think, nobody is interested in this code ... this sounds like crap... and this brings me to the next point

- Do we not have the same problem as with hush, if we change LIL code?

Do you plan to upstream your changes?

So please do not understand me wrong, I see you invested much time here, but I see LIL not in the short term as a replacement for hush.

I would accept to add LIL as an option and we will see, how good it works.

And if we have in lets say 2-3 years more boards which use LIL instead of hush *and* also active maintained LIL code, we may can think of getting rid of hush... and may have a "hush compatibility" layer...

I have no idea of current hush mainline code and how many work it will be to update U-Boot to a current hush version. But may this would be make more sense as hush is in active development. Also may to discuss with hush maintainers to upstream U-Boot changes... this would make in my eyes more sense.

Quoted 98 lines

= Why Lil?

When looking for a suitable replacement shell, I evaluated implementations using the following criteria:

- It must have a GPLv2-compatible license.
- It must be written in C, and have no major external dependencies.
- It must support bare function calls. That is, a script such as 'foo bar' should invoke the function 'foo' with the argument 'bar'. This preserves the shell-like syntax we expect.
- It must be small. The eventual target is that it compiles to around 10KiB with -Os and -ffunction-sections.
- There should be good tests. Any tests at all are good, but a functioning suite is better.
- There should be good documentation
- There should be comments in the source.
- It should be "finished" or have only slow development. This will hopefully make it easier to port changes.

Notably absent from the above list is performance. Most scripts in U-Boot will be run once on boot. As long as the time spent evaluating scripts is kept under a reasonable threshold (a fraction of the time spend initializing hardware or reading data from persistant storage), there is no need to optimize for speed.

In addition, I did not consider updating Hush from Busybox. The mismatch in computing environment expectations (as noted in the "New shell" section above) still applies. IMO, this mismatch is the biggest reason that things like functions and command substitution have been excluded from the U-Boot's Hush.

== lil

- zLib
- TCL
- Compiles to around 10k with no builtins. To 25k with builtins.
- Some tests, but not organized into a suite with expected output. Some evidence that the author ran APL, but no harness.
- Some architectural documentation. Some for each functions, but not much.
- No comments :l
- 3.5k LoC

== picol

- 2-clause BSD
- TCL
- Compiles to around 25k with no builtins. To 80k with builtins.
- Tests with suite (in-language). No evidence of fuzzing.
- No documentation :l
- No comments :l
- 5k LoC

== jimtcl

- 2-clause BSD
- TCL
- Compiles to around 95k with no builtins. To 140k with builtins. Too big...

== boron

- LGPLv3+ (so this is right out)
- REBOL
- Compiles to around 125k with no builtins. To 190k with builtins. Too big...

== libmawk

- GPLv2
- Awk
- Compiles to around 225k. Too big...

== libfawk

- 3-clause BSD
- Uses bison+yacc...
- Awk; As it turns out, this has parentheses for function calls.
- Compiles to around 24-30k. Not sure how to remove builtins.
- Test suite (in-language). No fuzzing.
- Tutorial book. No function reference.
- No comments
- Around 2-4k LoC

== MicroPython

- MIT
- Python (but included for completeness)
- Compiles to around 300k. Too big...

== mruby/c

- 3-clause BSD
- Ruby
- Compiles to around 85k without builtins and 120k with. Too big...

== eLua

- MIT
- Lua
- Build system is a royal pain (custom and written in Lua with external deps)
- Base binary is around 250KiB and I don't want to deal with reducing it

Quoted 4 lines

So the interesting/viable ones are
- lil
- picol
- libfawk (maybe)

Very nice overview, thanks!

bye,
Heiko

Signature · 3 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-52   Fax: +49-8142-66989-80   Email: [email protected]
#106 lore raw
Quoted 17 lines · Heiko Schocher

Hello Sean,

Quoted 9 lines · Sean Anderson

Well, this has been sitting on my hard drive for too long without feedback ("Release early, release often"), so here's the first RFC. This is not ready to merge (see the "Future work" section below), but the shell is functional and at least partially tested.

The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes over on sandbox.

add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)

Thanks for this comparison. Here just my thoughts, but I have currently to low time to follow here completly, so sorry if I miss stuff already discussed.

Quoted 7 lines

From my side of view we can add a second shell beside hush, and as you
already mentioned it may is even possible to start LIL from hush. This would be cool.

But I disagree in the following points:

- I do not see why LIL should *replace* hush.

This is the eventual goal. This series just adds an option at build time (CONFIG_LIL) which uses LIL as the shell instead of Hush. But anyone who writes a series like this must be convinced that it is better than what already exists. So I write that my goal is to replace hush, even if I think that it will take a long while.

Quoted 5 lines

This may break a lot of current boards. Also I see no need to do this, as if you do complex stuff in u-boot shell it may is the wrong way and you have to think about... and I see no big discussions here about BUGS in our current hush implementation ... may it makes sense to bring them up and fix?

I have been told not to work on the current shell; that it is too old and there is no point in improving it. Since we will be starting afresh anyway, and I do not think that the Hush model is particularly good, I chose to work from a different base.

Quoted 2 lines

- Is LIL really so good code, that it is done? Or is there only no community which develop further?

AFAIK there is not too much of a LIL community. The code quality is something I found out after working on it for a while. From a cursory inspection, it was around the same as all the other shells I looked at (no comments, etc). FWIW the current Hush shell does not really handle errors either. For example, on (x)malloc failure, we print and hang. This could easily be done for LIL, but it is not so nice when your language has recursion. So I have tried to add error checking where possible. And of course, Hush ignores missing variables as well.

Quoted 11 lines

- Do we really want code, which does no error checking? Commentless code...

Quoted 3 lines · you

- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly,

Which really let me think, nobody is interested in this code ... this sounds like crap... and this brings me to the next point

- Do we not have the same problem as with hush, if we change LIL code?

I think the primary problem with Hush was that we expected to backport updates at all. I think such a core UI feature should not have so many ifdefs in it when no backport was ever made. It is difficult to modify Hush because not only do you have to understand how U-Boot uses some feature, you also have to understand how Busybox uses some feature. And Hush of course has less comments as LIL does (after all patches in this series are applied).

Quoted 1 line

Do you plan to upstream your changes?

I don't view it as especially important. I view LIL as a starting point which can be modified to suit U-Boot. It may end up that each individual component is changed in some way, but starting with a working shell has been very helpful for development.

Quoted 4 lines

So please do not understand me wrong, I see you invested much time here, but I see LIL not in the short term as a replacement for hush.

I would accept to add LIL as an option and we will see, how good it works.

That is what I intend to do.

Quoted 8 lines

And if we have in lets say 2-3 years more boards which use LIL instead of hush *and* also active maintained LIL code, we may can think of getting rid of hush... and may have a "hush compatibility" layer...

I have no idea of current hush mainline code and how many work it will be to update U-Boot to a current hush version. But may this would be make more sense as hush is in active development. Also may to discuss with hush maintainers to upstream U-Boot changes... this would make in my eyes more sense.

AFAICT most of the U-Boot changes are removal of features, workarounds for unimplemented functionality (e.g. anywhere Hush does fork/exec), and some small additions like command repetition which cannot be upstreamed because it would break any semblance of POSIX compatibility.

--Sean

Quoted 108 lines
Quoted 98 lines

= Why Lil?

When looking for a suitable replacement shell, I evaluated implementations using the following criteria:

- It must have a GPLv2-compatible license.
- It must be written in C, and have no major external dependencies.
- It must support bare function calls. That is, a script such as 'foo bar' should invoke the function 'foo' with the argument 'bar'. This preserves the shell-like syntax we expect.
- It must be small. The eventual target is that it compiles to around 10KiB with -Os and -ffunction-sections.
- There should be good tests. Any tests at all are good, but a functioning suite is better.
- There should be good documentation
- There should be comments in the source.
- It should be "finished" or have only slow development. This will hopefully make it easier to port changes.

Notably absent from the above list is performance. Most scripts in U-Boot will be run once on boot. As long as the time spent evaluating scripts is kept under a reasonable threshold (a fraction of the time spend initializing hardware or reading data from persistant storage), there is no need to optimize for speed.

In addition, I did not consider updating Hush from Busybox. The mismatch in computing environment expectations (as noted in the "New shell" section above) still applies. IMO, this mismatch is the biggest reason that things like functions and command substitution have been excluded from the U-Boot's Hush.

== lil

- zLib
- TCL
- Compiles to around 10k with no builtins. To 25k with builtins.
- Some tests, but not organized into a suite with expected output. Some evidence that the author ran APL, but no harness.
- Some architectural documentation. Some for each functions, but not much.
- No comments :l
- 3.5k LoC

== picol

- 2-clause BSD
- TCL
- Compiles to around 25k with no builtins. To 80k with builtins.
- Tests with suite (in-language). No evidence of fuzzing.
- No documentation :l
- No comments :l
- 5k LoC

== jimtcl

- 2-clause BSD
- TCL
- Compiles to around 95k with no builtins. To 140k with builtins. Too big...

== boron

- LGPLv3+ (so this is right out)
- REBOL
- Compiles to around 125k with no builtins. To 190k with builtins. Too big...

== libmawk

- GPLv2
- Awk
- Compiles to around 225k. Too big...

== libfawk

- 3-clause BSD
- Uses bison+yacc...
- Awk; As it turns out, this has parentheses for function calls.
- Compiles to around 24-30k. Not sure how to remove builtins.
- Test suite (in-language). No fuzzing.
- Tutorial book. No function reference.
- No comments
- Around 2-4k LoC

== MicroPython

- MIT
- Python (but included for completeness)
- Compiles to around 300k. Too big...

== mruby/c

- 3-clause BSD
- Ruby
- Compiles to around 85k without builtins and 120k with. Too big...

== eLua

- MIT
- Lua
- Build system is a royal pain (custom and written in Lua with external deps)
- Base binary is around 250KiB and I don't want to deal with reducing it

Quoted 4 lines

So the interesting/viable ones are
- lil
- picol
- libfawk (maybe)

Very nice overview, thanks!

bye,
Heiko