cli: Add a new shell
cover cli: Add a new shell
Well, this has been sitting on my hard drive for too long without feedback ("Release early, release often"), so here's the first RFC. This is not ready to merge (see the "Future work" section below), but the shell is functional and at least partially tested.
The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes over on sandbox.
add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)
= Getting started
Enable CONFIG_LIL. If you would like to run tests, enable CONFIG_LIL_FULL. Note that dm_test_acpi_cmd_dump and setexpr_test_str_oper will fail. CONFIG_LIL_POOLS is currently broken (with what appears to be a double free).
For an overview of the language as a whole, refer to the original readme [1].
[1] http://runtimeterror.com/tech/lil/readme.txt
== Key patches
The following patches are particularly significant for reviewing and understanding this series:
cli: Add LIL shell
This contains the LIL shell as originally written by Kostas with some major deletions and some minor additions.
cli: lil: Wire up LIL to the rest of U-Boot
This allows you to use LIL as a shell just like Hush.
cli: lil: Document structures
This adds documentation for the major structures of LIL. It is a good place to start looking at the internals.
test: Add tests for LIL
This adds some basic integration tests and provides some examples of LIL code.
cli: lil: Add a distinct parsing step
This adds a parser separate from the interpreter. This patch is the largest original work in this series.
cli: lil: Load procs from the environment
This allows procedures to be saved and loaded like variables.
= A new shell
This series adds a new shell for U-Boot. The aim is to eventually replace Hush as the primary shell for all boards which currently use it. Hush should be replaced because it has several major problems:
- It has not had a major update in two decades, resulting in duplication of effort in finding bugs. Regarding a bug in variable setting, Wolfgang remarks
So the specific problem has (long) been fixed in upstream, and instead of adding a patch to our old version, thus cementing the broken behaviour, we should upgrade hush to recent upstream code.
-- Wolfgang Denk [2]
These lack of updates are further compounded by a significant amount of ifdef-ing in the Hush code. This makes the shell hard to read and debug. Further, the original purpose of such ifdef-ing (upgrading to a newer Hush) has never happened.
- It was designed for a preempting OS which supports pipes and processes. This fundamentally does not match the computing model of U-Boot where there is exactly one thread (and every other CPU is spinning or sleeping). Working around these design differences is a significant cause of the aformentioned ifdef-ing.
- It lacks many major features expected of even the most basic shells, such as functions and command substitution ($() syntax). This makes it difficult to script with Hush. While it is desirable to write some code in C, much code *must* be written in C because there is no way to express the logic in Hush.
I believe that U-Boot should have a shell which is more featureful, has cleaner code, and which is the same size as Hush (or less). The ergonomic advantages afforded by a new shell will make U-Boot easier to use and customize.
[2] https://lore.kernel.org/u-boot/[email protected]/
= Open questions
While the primary purpose of this series is of course to get feedback on the code I have already written, there are several decisions where I am not sure what the best course of action is.
- What should be done about 'expr'? The 'expr' command is a significant portion of the final code size. It cannot be removed outright, because it is used by several builtin functions like 'if', 'while', 'for', etc. The way I see it, there are two general approaches to take
- Rewrite expr to parse expressions and then evaluate them. The parsing could re-use several of the existing parse functions like how parse_list does. This could reduce code, as instead of many functions each with their own while/switch statements, we could have two while/switch statements (one to parse, and one to evaluate). However, this may end up increasing code size (such as when the main language had evaluation split from parsing).
- Don't parse infix expressions, and just make arithmetic operators normal functions. This would affect ergonomics a bit. For example, instead of
if {$i < 10} { ... }
one would need to write
if {< $i 10} { ... }
and instead of
if {$some_bool} { ... }
one would need to write
if {quote $some_bool} { ... }
Though, given how much setexpr is used (not much), this may not be such a big price to pay. This route is almost certain to reduce code size.
- How should LIL functions integrate with the rest of U-Boot? At the moment, lil functions and procedures exist in a completely separate world from normal commands. I would like to integrate them more closely, but I am not sure the best way to go about this. At the very minimum, each LIL builtin function needs to get its hands on the LIL interpreter somehow. I'd rather this didn't happen through gd_t or similar so that it is easier to unit test. Additionally, LIL functions expect an array of lil_values instead of strings. We could strip them out, but I worry that might start to impact performance (from all the copying).
The other half of this is adding LIL features into regular commands. The most important feature here is being able to return a string result. I took an initial crack at it [3], but I think with this series there is a stronger motivating factor (along with things like [4]).
[3] https://patchwork.ozlabs.org/project/uboot/list/?series=231377
[4] https://patchwork.ozlabs.org/project/uboot/list/?series=251013
= Future work
The series as presented today is incomplete. The following are the major issues I see with it at the moment. I would like to address all of these issues, but some of them might be postponed until after first merging this series.
- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly, there still remains a significant amount of original code which just ignores errors. In particular, I would like to ensure that the following categories of error conditions are handled:
- Running out of memory.
- Access to a nonexistant variable.
- Passing the wrong number of arguments to a function.
- Interpreting a value as the wrong type (e.g. "foo" should not have a numeric representation, instead of just being treated as 1).
- There are many deviations from TCL with no purpose. For example, the list indexing function is named "index" and not "lindex". It is perfectly fine to drop features or change semantics to reduce code size, make parsing easier, or make execution easier. But changing things for the sake of it should be avoided.
- The test suite is rather anemic compared with the amount of code this series introduces. I would like to expand it significantly. In particular, error conditions are not well tested (only the "happy path" is tested).
- While I have documented all new functions I have written, there are many existing functions which remain to be documented. In addition, there is no user documentation, which is critical in driving adoption of any new programming language. Some of this cover letter might be integrated with any documentation written.
- Some shell features such as command repetition and secondary shell prompts have not been implemented.
- Arguments to native lil functions are incompatible with U-Boot functions. For example, the command
foo bar baz
would be passed to a U-Boot command as
{ "foo", "bar", "baz", NULL }but would be passed to a LIL function as
{ "bar", "baz" } This makes it more difficult to use the same function to parse several different commands. At the moment this is solved by passing the command name in lil->env->proc, but I would like to switch to the U-Boot argument list style.
- Several existing tests break when using LIL because they expect no output on failure, but LIL produces some output notifying the user of the failure.
- Implement DISTRO_BOOT in LIL. I think this is an important proof-of-concept to show what can be done with LIL, and to determine which features should be moved to LIL_FULL.
= Why Lil?
When looking for a suitable replacement shell, I evaluated implementations using the following criteria:
- It must have a GPLv2-compatible license.
- It must be written in C, and have no major external dependencies.
- It must support bare function calls. That is, a script such as 'foo bar' should invoke the function 'foo' with the argument 'bar'. This preserves the shell-like syntax we expect.
- It must be small. The eventual target is that it compiles to around 10KiB with -Os and -ffunction-sections.
- There should be good tests. Any tests at all are good, but a functioning suite is better.
- There should be good documentation
- There should be comments in the source.
- It should be "finished" or have only slow development. This will hopefully make it easier to port changes.
Notably absent from the above list is performance. Most scripts in U-Boot will be run once on boot. As long as the time spent evaluating scripts is kept under a reasonable threshold (a fraction of the time spend initializing hardware or reading data from persistant storage), there is no need to optimize for speed.
In addition, I did not consider updating Hush from Busybox. The mismatch in computing environment expectations (as noted in the "New shell" section above) still applies. IMO, this mismatch is the biggest reason that things like functions and command substitution have been excluded from the U-Boot's Hush.
== lil
- zLib
- TCL
- Compiles to around 10k with no builtins. To 25k with builtins.
- Some tests, but not organized into a suite with expected output. Some evidence that the author ran APL, but no harness.
- Some architectural documentation. Some for each functions, but not much.
- No comments :l
- 3.5k LoC
== picol
- 2-clause BSD
- TCL
- Compiles to around 25k with no builtins. To 80k with builtins.
- Tests with suite (in-language). No evidence of fuzzing.
- No documentation :l
- No comments :l
- 5k LoC
== jimtcl
- 2-clause BSD
- TCL
- Compiles to around 95k with no builtins. To 140k with builtins. Too big...
== boron
- LGPLv3+ (so this is right out)
- REBOL
- Compiles to around 125k with no builtins. To 190k with builtins. Too big...
== libmawk
- GPLv2
- Awk
- Compiles to around 225k. Too big...
== libfawk
- 3-clause BSD
- Uses bison+yacc...
- Awk; As it turns out, this has parentheses for function calls.
- Compiles to around 24-30k. Not sure how to remove builtins.
- Test suite (in-language). No fuzzing.
- Tutorial book. No function reference.
- No comments
- Around 2-4k LoC
== MicroPython
- MIT
- Python (but included for completeness)
- Compiles to around 300k. Too big...
== mruby/c
- 3-clause BSD
- Ruby
- Compiles to around 85k without builtins and 120k with. Too big...
== eLua
- MIT
- Lua
- Build system is a royal pain (custom and written in Lua with external deps)
- Base binary is around 250KiB and I don't want to deal with reducing it
So the interesting/viable ones are
- lil
- picol
- libfawk (maybe)
I started with LIL because it was the smallest. I have found several issues with LIL along the way. Some of these are addressed in this series already, while others remain unaddressed (see the section "Future Work").
Sean Anderson (28):
Add Zlib License
cli: Add LIL shell
cli: lil: Replace strclone with strdup
cli: lil: Remove most functions by default
cli: lil: Rename some functions to be more like TCL
cli: lil: Convert some defines to enums
cli: lil: Simplify callbacks
cli: lil: Handle commands with dots
cli: lil: Use error codes
cli: lil: Add printf-style format helper for errors
cli: lil: Add several helper functions for errors
cli: lil: Check for ctrl-c
cli: lil: Wire up LIL to the rest of U-Boot
cli: lil: Document structures
cli: lil: Convert LIL_ENABLE_POOLS to Kconfig
cli: lil: Convert LIL_ENABLE_RECLIMIT to KConfig
test: Add tests for LIL
cli: lil: Remove duplicate function bodies
cli: lil: Add "symbol" structure
cli: lil: Add config to enable debug output
cli: lil: Add a distinct parsing step
env: Add a priv pointer to hwalk_r
cli: lil: Handle OOM for hm_put
cli: lil: Make proc always take 3 arguments
cli: lil: Always quote items in lil_list_to_value
cli: lil: Allocate len even when str is NULL in alloc_value_len
cli: lil: Add a function to quote values
cli: lil: Load procs from the environment
Licenses/README | 1 + Licenses/zlib.txt | 14 + MAINTAINERS | 7 + cmd/Kconfig | 52 +- cmd/nvedit.c | 8 +- common/Makefile | 4 + common/cli.c | 118 +- common/cli_lil.c | 4321 +++++++++++++++++++++++++++++++++++++++++++++ env/callback.c | 4 +- env/flags.c | 4 +- include/cli_lil.h | 202 +++ include/search.h | 2 +- lib/hashtable.c | 5 +- test/cmd/Makefile | 1 + test/cmd/lil.c | 306 ++++ 15 files changed, 5021 insertions(+), 28 deletions(-) create mode 100644 Licenses/zlib.txt create mode 100644 common/cli_lil.c create mode 100644 include/cli_lil.h create mode 100644 test/cmd/lil.c
Signature · 1 line
2.32.0
RFC PATCH 1/28 Add Zlib License
This adds the Zlib License which is compatible with GPLv2 as long as its terms are met. Files originally licensed as Zlib should use the "GPL-2.0+ AND Zlib" SPDX identifier.
---
Licenses/README | 1 + Licenses/zlib.txt | 14 ++++++++++++++ 2 files changed, 15 insertions(+) create mode 100644 Licenses/zlib.txt
Licenses/README +1 -0
@@ -152,4 +152,5 @@ IBM PIBS (PowerPC Initialization and IBM-pibs ibm-pibs.txt ISC License ISC Y isc.txt https://spdx.org/licenses/ISC MIT License MIT Y mit.txt https://spdx.org/licenses/MIT.html SIL OPEN FONT LICENSE (OFL-1.1) OFL-1.1 Y OFL.txt https://spdx.org/licenses/OFL-1.1.html +zlib License Zlib Y zlib.txt https://spdx.org/licenses/Zlib.html X11 License X11 x11.txt https://spdx.org/licenses/X11.html
Licenses/zlib.txt new file +14 -0
@@ -0,0 +1,14 @@ +This software is provided 'as-is', without any express or implied warranty. In +no event will the authors be held liable for any damages arising from the use +of this software. + +Permission is granted to anyone to use this software for any purpose, including +commercial applications, and to alter it and redistribute it freely, subject +to the following restrictions: + 1. The origin of this software must not be misrepresented; you must not +claim that you wrote the original software. If you use this software in a +product, an acknowledgment in the product documentation would be appreciated +but is not required. + 2. Altered source versions must be plainly marked as such, and must not be +misrepresented as being the original software. + 3. This notice may not be removed or altered from any source distribution.
Signature · 1 line
2.32.0
Re: [RFC PATCH 01/28] Add Zlib License
Quoted 5 lines · Sean Anderson
This adds the Zlib License which is compatible with GPLv2 as long as its terms are met. Files originally licensed as Zlib should use the "GPL-2.0+ AND Zlib" SPDX identifier.
Quoted 7 lines
Signed-off-by Sean Anderson <[email protected]>---
Licenses/README | 1 + Licenses/zlib.txt | 14 ++++++++++++++ 2 files changed, 15 insertions(+) create mode 100644 Licenses/zlib.txt
RFC PATCH 2/28 cli: Add LIL shell
In reply to Sean AndersonThis is the LIL programming language [1] as originally written by Kostas Michalopoulos <[email protected]>. LIL is a stripped-down TCL variant. Many syntax features are very similar to shell:
=> echo Hello from $stdout
Hello from serial,vidconsole
LIL supports functions, scoped variables, and command substitution. For a short example, running the following script
func fact {n} {
if {$n} {
expr {$n * [fact [expr {$n - 1}]]}
} {
return 1
}
}
echo [fact 10]
echo [fact 20]would result in the output
fact 3628800 2432902008176640000
For more examples, please refer to the "test: Add tests for LIL" later in this series.
I have made several mostly-mechanical modifications to the source before importing it to reduce patches to functional changes. If you are interested in what has been changed before this commit, refer to my lil git repository [2]. In addition, some functions and features have been removed from this patch as I have better determined what is necessary for U-Boot and what is not. If there is sufficient interest, I can add those changes to the repository as well.
[1] http://runtimeterror.com/tech/lil/
[2] https://github.com/Forty-Bot/lil/commits/master
---
MAINTAINERS | 6 + cmd/Kconfig | 11 +- common/Makefile | 1 + common/cli_lil.c | 2991 +++++++++++++++++++++++++++++++++++++++++++++ include/cli_lil.h | 111 ++ 5 files changed, 3119 insertions(+), 1 deletion(-) create mode 100644 common/cli_lil.c create mode 100644 include/cli_lil.h
MAINTAINERS +6 -0
@@ -766,6 +766,12 @@ S: Maintained T: git https://source.denx.de/u-boot/custodians/u-boot-i2c.git F: drivers/i2c/ +LIL SHELL +M: Sean Anderson <[email protected]> +S: Maintained +F: common/cli_lil.c +F: include/cli_lil.h + LOGGING M: Simon Glass <[email protected]> S: Maintained
cmd/Kconfig +10 -1
@@ -23,6 +23,15 @@ config HUSH_PARSER If disabled, you get the old, much simpler behaviour with a somewhat smaller memory footprint. +config LIL + bool "Use LIL shell" + depends on CMDLINE + help + This options enables the "Little Interpreted Language" (LIL) shell as + command line interpreter, thus enabling powerful command line syntax + like `proc name {args} {body}' functions or `echo [some command]` + command substitution ("tcl scripts"). + config CMDLINE_EDITING bool "Enable command line editing" depends on CMDLINE
@@ -1379,7 +1388,7 @@ config CMD_ECHO config CMD_ITEST bool "itest" - default y + default y if !LIL help Return true/false on integer compare.
common/Makefile +1 -0
@@ -10,6 +10,7 @@ obj-y += main.o obj-y += exports.o obj-$(CONFIG_HASH) += hash.o obj-$(CONFIG_HUSH_PARSER) += cli_hush.o +obj-$(CONFIG_LIL) += cli_lil.o obj-$(CONFIG_AUTOBOOT) += autoboot.o # This option is not just y/n - it can have a numeric value
common/cli_lil.c new file +2991 -0
2,958 lines not on this page · raw message
include/cli_lil.h new file +111 -0
111 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.32.0
Part of the diff is not shown here. View the raw message.
Re: [RFC PATCH 02/28] cli: Add LIL shell
Dear Sean,
Quoted 3 lines · In message <[email protected]> you wrote:
This is the LIL programming language [1] as originally written by Kostas Michalopoulos <[email protected]>. LIL is a stripped-down TCL variant. Many syntax features are very similar to shell:
Do you have a list of the exact differencec between LIL and a standard shell?
I wonder, if we deviate from standard shell syntax anyway, we could also have a look at lua, for example?
Best regards,
Wolfgang Denk
Signature · 6 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] One essential to success is that you desire be an all-obsessing one, your thoughts and aims be co-ordinated, and your energy be concentra- ted and applied without letup.
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 9 lines · Wolfgang Denk
Dear Sean,
Quoted 3 lines · In message <[email protected]> you wrote:
This is the LIL programming language [1] as originally written by Kostas Michalopoulos <[email protected]>. LIL is a stripped-down TCL variant. Many syntax features are very similar to shell:
Do you have a list of the exact differencec between LIL and a standard shell?
For a partial list, see
[1] https://github.com/Forty-Bot/lil/commits/master
Quoted 2 lines
I wonder, if we deviate from standard shell syntax anyway, we could also have a look at lua, for example?
I also looked at lua (see the cover letter), but I rejected it based on size constraints (eLua was around the size of U-Boot itself).
Because of how often the shell is used to debug things, I wanted the candidate I picked to have similar syntax to the existing shell. For example,
load mmc ${mmcdev}:${mmcpart} $loadaddr $imageis a valid command in both Hush and LIL. Compare with lua, which might express the above as
load("mmc", mmcdev .. ":" .. mmcpart, loadaddr, image)which I think is a much larger deviation from existing syntax.
--Sean
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 8 lines · Sean Anderson
Quoted 7 lines · Wolfgang Denk
Dear Sean,
Quoted 3 lines · In message <[email protected]> you wrote:
This is the LIL programming language [1] as originally written by Kostas Michalopoulos <[email protected]>. LIL is a stripped-down TCL variant. Many syntax features are very similar to shell:
Quoted 6 lines
Quoted 2 lines
Do you have a list of the exact differencec between LIL and a standard shell?
For a partial list, see
Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:
sh tcl
foo=bar set foo bar
echo $foo echo $foo
if [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }
foo() { proc foo {first second} {
echo $1 $2 echo $first $second
} }for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }
fact() {
if [ $1 -eq 0 ]; then
echo 1
else
echo $(($1 * $(fact $(($1 - 1)))))
fi
} proc fact {n} {
if {$n} {
expr {$n * [fact [expr {$n - 1}]]}
} {
return 1
}
}Hopefully this gives you a bit of a feel for the basic differences.
--Sean
Quoted 21 lines
Quoted 3 lines
I wonder, if we deviate from standard shell syntax anyway, we could also have a look at lua, for example?
I also looked at lua (see the cover letter), but I rejected it based on size constraints (eLua was around the size of U-Boot itself).
Because of how often the shell is used to debug things, I wanted the candidate I picked to have similar syntax to the existing shell. For example,
load mmc ${mmcdev}:${mmcpart} $loadaddr $imageis a valid command in both Hush and LIL. Compare with lua, which might express the above as
load("mmc", mmcdev .. ":" .. mmcpart, loadaddr, image)which I think is a much larger deviation from existing syntax.
--Sean
Re: [RFC PATCH 02/28] cli: Add LIL shell
Dear Sean,
Quoted 44 lines · In message <[email protected]> you wrote:
Quoted 4 lines
For a partial list, see
Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:
sh tcl
foo=bar set foo bar
echo $foo echo $fooif [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }foo() { proc foo {first second} { echo $1 $2 echo $first $second } }for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }fact() { if [ $1 -eq 0 ]; then echo 1 else echo $(($1 * $(fact $(($1 - 1))))) fi }proc fact {n} { if {$n} { expr {$n * [fact [expr {$n - 1}]]} } { return 1 } }
Quoted 1 line
Hopefully this gives you a bit of a feel for the basic differences.
Well, I know TCL, and there has been a zillion of reasons to move _from_ that language many, many years ago. Intoducing this as new "shell" language in U-Boot does not look attractive to me. Actually, it gives me the creeps.
Best regards,
Wolfgang Denk
Signature · 4 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] Drawing on my fine command of language, I said nothing.
Simon Glass
Reviewed-byRe: [RFC PATCH 02/28] cli: Add LIL shell
Hi,
Quoted 48 lines · Wolfgang Denk
Dear Sean,
Quoted 44 lines · In message <[email protected]> you wrote:
Quoted 4 lines
For a partial list, see
Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:
sh tcl
foo=bar set foo bar
echo $foo echo $fooif [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }
foo(){ proc foo {first second} {
echo $1 $2 echo $first $second
} }for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }fact() { if [ $1 -eq 0 ]; then echo 1 else echo $(($1 * $(fact $(($1 - 1))))) fi }proc fact {n} { if {$n} { expr {$n * [fact [expr {$n - 1}]]} } { return 1 } }
Quoted 6 lines
Quoted 1 line
Hopefully this gives you a bit of a feel for the basic differences.
Well, I know TCL, and there has been a zillion of reasons to move _from_ that language many, many years ago. Intoducing this as new "shell" language in U-Boot does not look attractive to me. Actually, it gives me the creeps.
I was no fan of TCL either but this does seem like a reasonable idea to me.
Obviously whoever wrote it was very concerned about compiler performance as there seem to be no comments?
Regards,
Simon
Re: [RFC PATCH 02/28] cli: Add LIL shell
In reply to Wolfgang DenkQuoted 48 lines · Wolfgang Denk
Dear Sean,
Quoted 45 lines · In message <[email protected]> you wrote:
Quoted 4 lines
For a partial list, see
Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:
sh tcl
foo=bar set foo bar
echo $foo echo $fooif [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }foo() { proc foo {first second} { echo $1 $2 echo $first $second } }for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }fact() { if [ $1 -eq 0 ]; then echo 1 else echo $(($1 * $(fact $(($1 - 1))))) fi }proc fact {n} { if {$n} { expr {$n * [fact [expr {$n - 1}]]} } { return 1 } }Hopefully this gives you a bit of a feel for the basic differences.
Which of these things, from each column, can you do in the context of U-Boot? That's important too.
Quoted 4 lines
Well, I know TCL, and there has been a zillion of reasons to move _from_ that language many, many years ago. Intoducing this as new "shell" language in U-Boot does not look attractive to me. Actually, it gives me the creeps.
This is I think the hard question. A draw of the current shell is that it it looks and acts like bash/sh/etc, for at least basic operations. That's something that's comfortable to a large audience. That has disadvantages when people want to start doing something complex. Sean has shown that several times and he's not the only one. LIL being tcl'ish is not.
Something that has "sh" syntax but also clear to the user errors when trying to do something not supported would also be interesting to see. It seems like a lot of the frustration from users with our shell is that it's not clear where the line between "this is an sh-like shell" and "no, not like that" is.
Signature · 1 line
Tom
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 15 lines · Tom Rini
Quoted 14 lines · Wolfgang Denk
Dear Sean,
Quoted 11 lines · In message <[email protected]> you wrote:
Quoted 4 lines
For a partial list, see
Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:
sh tcl
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
foo=bar set foo bar
echo $foo echo $fooif [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }
The left side is possible with something like
if itest 1 -gt 2; then # etc.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo() { proc foo {first second} { echo $1 $2 echo $first $second } }
This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }
This is possible only if you already have a list of files. For example, one could do
part list mmc 0 -bootable parts
for p in $parts; do #etc
but the part command is one of the only ones which produces output in the correct format. If you want to (e.g.) dynamically construct a list you will have a much harder time.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
fact() { if [ $1 -eq 0 ]; then echo 1 else echo $(($1 * $(fact $(($1 - 1))))) fi }
This is technically possible with run and setexpr, but fairly cumbersome to do.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
proc fact {n} { if {$n} { expr {$n * [fact [expr {$n - 1}]]} } { return 1 } }
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Hopefully this gives you a bit of a feel for the basic differences.
Which of these things, from each column, can you do in the context of U-Boot? That's important too.
See above.
--Sean
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 73 lines · Sean Anderson
Quoted 15 lines · Tom Rini
Quoted 14 lines · Wolfgang Denk
Dear Sean,
Quoted 11 lines · In message <[email protected]> you wrote:
Quoted 4 lines
For a partial list, see
Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:
sh tcl
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
foo=bar set foo bar
echo $foo echo $fooif [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }The left side is possible with something like
if itest 1 -gt 2; then # etc.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo() { proc foo {first second} { echo $1 $2 echo $first $second } }This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }This is possible only if you already have a list of files. For example, one could do
part list mmc 0 -bootable parts
for p in $parts; do #etcbut the part command is one of the only ones which produces output in the correct format. If you want to (e.g.) dynamically construct a list you will have a much harder time.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
fact() { if [ $1 -eq 0 ]; then echo 1 else echo $(($1 * $(fact $(($1 - 1))))) fi }This is technically possible with run and setexpr, but fairly cumbersome to do.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
proc fact {n} { if {$n} { expr {$n * [fact [expr {$n - 1}]]} } { return 1 } }
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Hopefully this gives you a bit of a feel for the basic differences.
Which of these things, from each column, can you do in the context of U-Boot? That's important too.
See above.
And for clarity, on the LIL side, with a few things like needing to bring in the list of files somehow, all of those would work in U-Boot?
Signature · 1 line
Tom
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 76 lines · Tom Rini
Quoted 75 lines · Sean Anderson
Quoted 15 lines · Tom Rini
Quoted 14 lines · Wolfgang Denk
Dear Sean,
Quoted 11 lines · In message <[email protected]> you wrote:
Quoted 4 lines
For a partial list, see
Whoops, looks like I completely misread what you were asking here. I don't have an exhaustive list of differences, but here are some similar things expressed in both languages:
sh tcl
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
foo=bar set foo bar
echo $foo echo $fooif [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }The left side is possible with something like
if itest 1 -gt 2; then # etc.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo() { proc foo {first second} { echo $1 $2 echo $first $second } }This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }This is possible only if you already have a list of files. For example, one could do
part list mmc 0 -bootable parts
for p in $parts; do #etcbut the part command is one of the only ones which produces output in the correct format. If you want to (e.g.) dynamically construct a list you will have a much harder time.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
fact() { if [ $1 -eq 0 ]; then echo 1 else echo $(($1 * $(fact $(($1 - 1))))) fi }This is technically possible with run and setexpr, but fairly cumbersome to do.
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
proc fact {n} { if {$n} { expr {$n * [fact [expr {$n - 1}]]} } { return 1 } }Hopefully this gives you a bit of a feel for the basic differences.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Which of these things, from each column, can you do in the context of U-Boot? That's important too.
See above.
And for clarity, on the LIL side, with a few things like needing to bring in the list of files somehow, all of those would work in U-Boot?
Correct. The only unimplemented function is "glob".
--Sean
Re: [RFC PATCH 02/28] cli: Add LIL shell
In reply to Sean AndersonDear Sean,
Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo() { proc foo {first second} { echo $1 $2 echo $first $second } }This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.
Again: this is not an issue with hush as is, but only with our resource-limited port of a nearly 20 year old version of it.
Updating to a current version would fix this, in an almost 100% backward compatible way.
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] If you don't have time to do it right, when will you have time to do it over? - John Wooden
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 13 lines · Wolfgang Denk
Dear Sean,
Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo(){ proc foo {first second} {
echo $1 $2 echo $first $second
} }This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.
Quoted 5 lines
Again: this is not an issue with hush as is, but only with our resource-limited port of a nearly 20 year old version of it.
Updating to a current version would fix this, in an almost 100% backward compatible way.
Agree on this. There is another email about the LIL and how is right now maintained and if the solutions that we have already in place can be just updated, I don't find any valid reason to change it but update seems more straightforward.
Michael
Quoted 11 lines
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] If you don't have time to do it right, when will you have time to do
it over? - John Wooden
Signature · 11 lines
Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 [email protected] __________________________________ Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 [email protected] www.amarulasolutions.com
Re: [RFC PATCH 02/28] cli: Add LIL shell
Hi Michael,
Quoted 18 lines · Michael Nazzareno Trimarchi
Quoted 16 lines · Wolfgang Denk
Dear Sean,
Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo(){ proc foo {first second} {
echo $1 $2 echo $first $second
} }This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.
Again: this is not an issue with hush as is, but only with our resource-limited port of a nearly 20 year old version of it.
Quoted 7 lines
Quoted 2 lines
Updating to a current version would fix this, in an almost 100% backward compatible way.
Agree on this. There is another email about the LIL and how is right now maintained and if the solutions that we have already in place can be just updated, I don't find any valid reason to change it but update seems more straightforward.
Would you be interested in taking this on? It has been talked about for some years...
Regards,
Simon
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 25 lines · Simon Glass
Hi Michael,
Quoted 21 lines · Michael Nazzareno Trimarchi
Quoted 18 lines · Wolfgang Denk
Dear Sean,
Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo(){ proc foo {first second} {
echo $1 $2 echo $first $second
} }This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.
Again: this is not an issue with hush as is, but only with our resource-limited port of a nearly 20 year old version of it.
Updating to a current version would fix this, in an almost 100% backward compatible way.
Quoted 7 lines
Quoted 4 lines
Agree on this. There is another email about the LIL and how is right now maintained and if the solutions that we have already in place can be just updated, I don't find any valid reason to change it but update seems more straightforward.
Would you be interested in taking this on? It has been talked about for some years...
To repeat myself, someone posting the patches to update (or more likely, replace as a new Kconfig option and I'd be fine with a "pick old or new" as a starting point) is the main gating factor on using a new version of hush.
Signature · 1 line
Tom
Re: [RFC PATCH 02/28] cli: Add LIL shell
In reply to Simon GlassHi
Quoted 31 lines · Simon Glass
Hi Michael,
Quoted 18 lines · Michael Nazzareno Trimarchi
Quoted 16 lines · Wolfgang Denk
Dear Sean,
Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo(){ proc foo {first second} {
echo $1 $2 echo $first $second
} }This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.
Again: this is not an issue with hush as is, but only with our resource-limited port of a nearly 20 year old version of it.
Quoted 7 lines
Quoted 2 lines
Updating to a current version would fix this, in an almost 100% backward compatible way.
Agree on this. There is another email about the LIL and how is right now maintained and if the solutions that we have already in place can be just updated, I don't find any valid reason to change it but update seems more straightforward.
Quoted 5 lines
Would you be interested in taking this on? It has been talked about for some years...
Regards,
Simon
I will ask to someone in the company to do as task. Francis can you do it?
Michael
Signature · 11 lines
Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 [email protected] __________________________________ Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 [email protected] www.amarulasolutions.com
Re: [RFC PATCH 02/28] cli: Add LIL shell
In reply to Wolfgang DenkQuoted 12 lines · Wolfgang Denk
Dear Sean,
Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
foo() { proc foo {first second} { echo $1 $2 echo $first $second } }This is not possible. We only have eval (run) as of today. I view adding functions as one of the most important usability improvements we can make.
Quoted 5 lines
Again: this is not an issue with hush as is, but only with our resource-limited port of a nearly 20 year old version of it.
Updating to a current version would fix this, in an almost 100% backward compatible way.
Let us cut to the chase then. Who is going to port a modern version of hush over to U-Boot, and maintain it? If we fork and forget again, we'll be in a bad place once again in 2-3 years.
Signature · 1 line
Tom
Re: [RFC PATCH 02/28] cli: Add LIL shell
Dear Tom,
Quoted 7 lines · In message <20210706145420.GQ9516@bill-the-cat> you wrote:
Quoted 3 lines
Updating to a current version would fix this, in an almost 100% backward compatible way.
Let us cut to the chase then. Who is going to port a modern version of hush over to U-Boot, and maintain it? If we fork and forget again, we'll be in a bad place once again in 2-3 years.
Would we really be better off if we switch to some exotic piece of code instead (I was not able to locate any user base, nor other developers), which has been reported to have poor or no error handling, and comes with an incompatible command line interface?
There is a zillion of shell scripts in the field, from non-trivial boot sequences to complex download-and-upgrade scripts. You can't really even think of breaking compatibility on such a level.
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] There are no data that cannot be plotted on a straight line if the axis are chosen correctly.
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 11 lines · Wolfgang Denk
Dear Tom,
Quoted 7 lines · In message <20210706145420.GQ9516@bill-the-cat> you wrote:
Quoted 3 lines
Updating to a current version would fix this, in an almost 100% backward compatible way.
Let us cut to the chase then. Who is going to port a modern version of hush over to U-Boot, and maintain it? If we fork and forget again, we'll be in a bad place once again in 2-3 years.
Quoted 8 lines
Would we really be better off if we switch to some exotic piece of code instead (I was not able to locate any user base, nor other developers), which has been reported to have poor or no error handling, and comes with an incompatible command line interface?
There is a zillion of shell scripts in the field, from non-trivial boot sequences to complex download-and-upgrade scripts. You can't really even think of breaking compatibility on such a level.
As I've said a few times in this thread, this not being an sh-style interpreter is a strike against it. And if we're going to insist on a bug-for-bug upgrade to our hush (so that all of the hugely complex existing scripts keep working) we might as well not upgrade. Frankly I suspect that down the line IF a new cli interpreter comes in to U-Boot we will have to keep the old one around as a "use this instead" option for another long number of years, so that if there are any systems with non-trivial scripts but upgrade U-Boot and don't / won't / can't re-validate their entire sequence, they can just use the old cli.
Signature · 1 line
Tom
Re: [RFC PATCH 02/28] cli: Add LIL shell
Dear Tom,
Quoted 10 lines · In message <20210707135839.GW9516@bill-the-cat> you wrote:
As I've said a few times in this thread, this not being an sh-style interpreter is a strike against it. And if we're going to insist on a bug-for-bug upgrade to our hush (so that all of the hugely complex existing scripts keep working) we might as well not upgrade. Frankly I suspect that down the line IF a new cli interpreter comes in to U-Boot we will have to keep the old one around as a "use this instead" option for another long number of years, so that if there are any systems with non-trivial scripts but upgrade U-Boot and don't / won't / can't re-validate their entire sequence, they can just use the old cli.
Do you actually have an example where code working on our ancient port of hush would fail on the current upstream version?
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] If you can't explain it to a six year old, you don't understand it yourself. - Albert Einstein
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 14 lines · Wolfgang Denk
Dear Tom,
Quoted 10 lines · In message <20210707135839.GW9516@bill-the-cat> you wrote:
As I've said a few times in this thread, this not being an sh-style interpreter is a strike against it. And if we're going to insist on a bug-for-bug upgrade to our hush (so that all of the hugely complex existing scripts keep working) we might as well not upgrade. Frankly I suspect that down the line IF a new cli interpreter comes in to U-Boot we will have to keep the old one around as a "use this instead" option for another long number of years, so that if there are any systems with non-trivial scripts but upgrade U-Boot and don't / won't / can't re-validate their entire sequence, they can just use the old cli.
Quoted 2 lines
Do you actually have an example where code working on our ancient port of hush would fail on the current upstream version?
Have you validated one of those exceedingly complex boot scripts with a modern hush (and some fakery for u-boot commands) ? No. I'm just saying I expect there to be enough risk-adverse groups that just dropping our old hush entirely might not be possible right away. Of course, if all of the current in-tree complex cases Just Work, that might be a good argument against needing to keep such levels of backwards compatibility.
Signature · 1 line
Tom
Re: [RFC PATCH 02/28] cli: Add LIL shell
Dear Tom,
Quoted 3 lines · In message <20210707141418.GZ9516@bill-the-cat> you wrote:
Have you validated one of those exceedingly complex boot scripts with a modern hush (and some fakery for u-boot commands) ? No.
No, I havent.
I also don't claim that I know all the warts and issues with our old hus, but for the ones I am aware the woraround usually splitting complex or nested expressions into a sequence of simpler steps. I cannot remember any cases where the resulting code should be incompatible to a shell without that specific bug.
And yes, I am aware of the problems with the distro_bootcmd stuff. Thisis exactly what I had in mind when I wrote: if you run into such situations you should lean back and reflect a bit. I can undrstand the intentions of all this stuff, but the implementation is a horrible mess.
Quoted 6 lines
I'm just
saying I expect there to be enough risk-adverse groups that just dropping our old hush entirely might not be possible right away. Of course, if all of the current in-tree complex cases Just Work, that might be a good argument against needing to keep such levels of backwards compatibility.
There is only one way to test this.
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] The inappropriate cannot be beautiful. - Frank Lloyd Wright _The Future of Architecture_ (1953)
Re: [RFC PATCH 02/28] cli: Add LIL shell
In reply to Tom RiniDear Tom,
Quoted 34 lines · In message <20210705191058.GB9516@bill-the-cat> you wrote:
Quoted 34 lines
Quoted 34 lines
foo=bar set foo bar
echo $foo echo $fooif [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }foo() { proc foo {first second} { echo $1 $2 echo $first $second } }for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }fact() { if [ $1 -eq 0 ]; then echo 1 else echo $(($1 * $(fact $(($1 - 1))))) fi }proc fact {n} { if {$n} { expr {$n * [fact [expr {$n - 1}]]} } { return 1 } }
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Hopefully this gives you a bit of a feel for the basic differences.
Which of these things, from each column, can you do in the context of U-Boot? That's important too.
Well, with a current version of hush we can do:
-> foo=bar
-> echo $foo
bar
-> if [ 1 -gt 2 ]; then
Quoted 4 lines
echo a
else
echo b
fi
b
-> foo() {
Quoted 2 lines
echo $1 $2
}
-> foo bar baz
bar baz
-> for file in $(ls *.c); do
Quoted 2 lines
echo $file
done
ls: cannot access '*.c': No such file or directory
-> fact() {
Quoted 6 lines
if [ $1 -eq 0 ]; then
echo 1
elseecho $(($1 * $(fact $(($1 - 1)))))fi
}
-> fact 4
24
Oh, in the contect of U-Boot? Well, there are of course limitations, but not because of the shell, but because of the fact that we have no concept of files, for example.
But another command interpreter will not fix this.
Quoted 6 lines
This is I think the hard question. A draw of the current shell is that it it looks and acts like bash/sh/etc, for at least basic operations. That's something that's comfortable to a large audience. That has disadvantages when people want to start doing something complex. Sean has shown that several times and he's not the only one. LIL being tcl'ish is not.
Tcl is a horror of a language for anything that is above trivial level.
Do you really think that replacing standard shell syntax with Tcl is "something that's comfortable to a large audience"? I seriously doubt that.
Quoted 5 lines
Something that has "sh" syntax but also clear to the user errors when trying to do something not supported would also be interesting to see. It seems like a lot of the frustration from users with our shell is that it's not clear where the line between "this is an sh-like shell" and "no, not like that" is.
Did you run some tests on the version of hush as comes with recent busybox releases? Which of our user's requirements does it fail to meet?
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] The first thing we do is kill all the lawyers. (Shakespeare. II Henry VI, Act IV, scene ii)
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 77 lines · Wolfgang Denk
Dear Tom,
Quoted 34 lines · In message <20210705191058.GB9516@bill-the-cat> you wrote:
Quoted 34 lines
Quoted 34 lines
foo=bar set foo bar
echo $foo echo $fooif [ 1 -gt 2 ]; then if {1 > 2} {
echo a echo a
else } {
echo b echo b
fi }foo() { proc foo {first second} { echo $1 $2 echo $first $second } }for file in $(ls *.c); do foreach file [glob *.c] {
echo $file echo $file
done }fact() { if [ $1 -eq 0 ]; then echo 1 else echo $(($1 * $(fact $(($1 - 1))))) fi }proc fact {n} { if {$n} { expr {$n * [fact [expr {$n - 1}]]} } { return 1 } }Quoted 4 lines
Quoted 1 line
Quoted 1 line
Hopefully this gives you a bit of a feel for the basic differences.
Which of these things, from each column, can you do in the context of U-Boot? That's important too.
Well, with a current version of hush we can do:
-> foo=bar
-> echo $foo
bar-> if [ 1 -gt 2 ]; then
Quoted 4 lines
echo a
else
echo b
fib
->
foo(){Quoted 2 lines
echo $1 $2
}-> foo bar baz
bar baz-> for file in $(ls *.c); do
Quoted 2 lines
echo $file
donels: cannot access '*.c': No such file or directory
->
fact(){Quoted 6 lines
if [ $1 -eq 0 ]; then
echo 1
elseecho $(($1 * $(fact $(($1 - 1)))))fi
}-> fact 4
24
Quoted 5 lines
Oh, in the contect of U-Boot? Well, there are of course limitations, but not because of the shell, but because of the fact that we have no concept of files, for example.
But another command interpreter will not fix this.
Yes, clearly the file based examples won't work either way, as-is. I was asking for what things can be done today with the implementations we have now.
I'm pretty confident that exactly zero people have written complex U-Boot scripts and then been happy about the experience.
Quoted 9 lines
Quoted 6 lines
This is I think the hard question. A draw of the current shell is that it it looks and acts like bash/sh/etc, for at least basic operations. That's something that's comfortable to a large audience. That has disadvantages when people want to start doing something complex. Sean has shown that several times and he's not the only one. LIL being tcl'ish is not.
Tcl is a horror of a language for anything that is above trivial level.
TCL has its fans. csh has it's fans. The question isn't what's the best desktop shell or general scripting language, but what's the most useful in our environment an use cases.
Quoted 3 lines
Do you really think that replacing standard shell syntax with Tcl is "something that's comfortable to a large audience"? I seriously doubt that.
I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.
Quoted 9 lines
Quoted 5 lines
Something that has "sh" syntax but also clear to the user errors when trying to do something not supported would also be interesting to see. It seems like a lot of the frustration from users with our shell is that it's not clear where the line between "this is an sh-like shell" and "no, not like that" is.
Did you run some tests on the version of hush as comes with recent busybox releases? Which of our user's requirements does it fail to meet?
It fails to meet the requirement of having been ported to U-Boot. I'd joke that perhaps you can bootefi busybox, but a quick search says that particular trick looks to still just be for python
(https://lwn.net/Articles/641244/).
Signature · 1 line
Tom
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 6 lines · Tom Rini
I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.
I think the same would happen with Tcl and LIL. LIL might look similar to Tcl (and it is inspired by it), but it doesn't have the same syntax. A big difference comes from how variables are parsed with LIL allowing more variables symbols than Tcl without requiring escaping which affects some uses like expr, so e.g. in Tcl this
set a 10 ; set b 20 ; expr $a+$b
will give back 30 however in LIL will give back 20. This is because Tcl parses "$a+$b" as "$a" "+" "$b" whereas LIL parses it as "$a+" "$b", evaluates "$a+" as an empty string (there isn't any "a+" variable), "$b" as 20 and then just runs expr with the single argument "20".
Kostas
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 14 lines · Kostas Michalopoulos
Quoted 6 lines · Tom Rini
I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.
I think the same would happen with Tcl and LIL. LIL might look similar to Tcl (and it is inspired by it), but it doesn't have the same syntax. A big difference comes from how variables are parsed with LIL allowing more variables symbols than Tcl without requiring escaping which affects some uses like expr, so e.g. in Tcl this
Quoted 7 lines
set a 10 ; set b 20 ; expr $a+$bwill give back 30 however in LIL will give back 20. This is because Tcl parses "$a+$b" as "$a" "+" "$b" whereas LIL parses it as "$a+" "$b", evaluates "$a+" as an empty string (there isn't any "a+" variable), "$b" as 20 and then just runs expr with the single argument "20".
IMO both of these are nuts. I think the best thing to do here is to raise an error about a missing variable "a+" to force the user to rewrite this as
set a 10 ; set b 20 ; expr ${a}+$bremoving the ambiguity. Although I would generally like to follow TCL's lead, there are several places where TCL does crazy things (e.g. comments) which I have purposely not replicated.
--Sean
Re: [RFC PATCH 02/28] cli: Add LIL shell
In reply to Tom RiniDear Tom,
Quoted 3 lines · In message <20210706154346.GT9516@bill-the-cat> you wrote:
I'm pretty confident that exactly zero people have written complex U-Boot scripts and then been happy about the experience.
I have seen many U-Boot scripts which were pretty complex, but working absolutely reliably.
Quoted 3 lines
TCL has its fans. csh has it's fans. The question isn't what's the best desktop shell or general scripting language, but what's the most useful in our environment an use cases.
Maybe you should try and do a poll of our user base which CLI they _want_? I doubt there will be any significant percentage voting for Tcl.
I know of a large number of systems which offer a shell interface on their command line, and those who don't usually use completely proprietary code. I know of very few examples where Tcl is being used.
Quoted 6 lines
I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.
Point taken. But if you think this to an end, the result is: lets write some documentation and explain the limitations of a shell in U-Boot environment, and document the warts and bugs of this (or an updated) version of hush. This should make more users happy than completely new and incompatible stuff.
Frankly, I believe when you run into problems with hush in U-Boot (even the current version) you should lean back and think about what you are doing.
U-Boot is a boot loader, and while it is powerful enough to do complex things, this is not necessarily the most clever approach. 15 years ago, I've written complex update scripts for U-Boot. This was easy enough to do, and worked perfectly. But there are so many limitations in a boot loader environment. We don't do this any more. Instead, we use an OS suitable for such tasks (Linux with SWUpdate).
And talking about problems and limitations in U-Boot... Is the CLI really our biggest concern right now? None of our users (customers) has asked for a better command interpreter - the question we hear are more like: "When will you support IPv6?", "NFS does not work with recent Linux distros, will this be fixed?", "Can I download over WiFi?", "Can I download using HTTP/HTTPS?", "How can I harden U-Boot for security-critical environments?", etc.
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] Always try to do things in chronological order; it's less confusing that way.
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 14 lines · Wolfgang Denk
Dear Tom,
Quoted 3 lines · In message <20210706154346.GT9516@bill-the-cat> you wrote:
I'm pretty confident that exactly zero people have written complex U-Boot scripts and then been happy about the experience.
I have seen many U-Boot scripts which were pretty complex, but working absolutely reliably.
Quoted 3 lines
TCL has its fans. csh has it's fans. The question isn't what's the best desktop shell or general scripting language, but what's the most useful in our environment an use cases.
Quoted 8 lines
Maybe you should try and do a poll of our user base which CLI they _want_? I doubt there will be any significant percentage voting for Tcl.
I know of a large number of systems which offer a shell interface on their command line, and those who don't usually use completely proprietary code. I know of very few examples where Tcl is being used.
Off the top of my head, most of the tooling for FPGAs uses TCL for scripting. OpenOCD uses it too.
Quoted 28 lines
Quoted 6 lines
I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.
Point taken. But if you think this to an end, the result is: lets write some documentation and explain the limitations of a shell in U-Boot environment, and document the warts and bugs of this (or an updated) version of hush. This should make more users happy than completely new and incompatible stuff.
Frankly, I believe when you run into problems with hush in U-Boot (even the current version) you should lean back and think about what you are doing.
U-Boot is a boot loader, and while it is powerful enough to do complex things, this is not necessarily the most clever approach. 15 years ago, I've written complex update scripts for U-Boot. This was easy enough to do, and worked perfectly. But there are so many limitations in a boot loader environment. We don't do this any more. Instead, we use an OS suitable for such tasks (Linux with SWUpdate).
Quoted 7 lines
And talking about problems and limitations in U-Boot... Is the CLI really our biggest concern right now? None of our users (customers) has asked for a better command interpreter - the question we hear are more like: "When will you support IPv6?", "NFS does not work with recent Linux distros, will this be fixed?", "Can I download over WiFi?", "Can I download using HTTP/HTTPS?", "How can I harden U-Boot for security-critical environments?", etc.
I wanted a better shell, so I worked on it.
--Sean
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 64 lines · Sean Anderson
Quoted 14 lines · Wolfgang Denk
Dear Tom,
Quoted 3 lines · In message <20210706154346.GT9516@bill-the-cat> you wrote:
I'm pretty confident that exactly zero people have written complex U-Boot scripts and then been happy about the experience.
I have seen many U-Boot scripts which were pretty complex, but working absolutely reliably.
Quoted 3 lines
TCL has its fans. csh has it's fans. The question isn't what's the best desktop shell or general scripting language, but what's the most useful in our environment an use cases.
Quoted 8 lines
Maybe you should try and do a poll of our user base which CLI they _want_? I doubt there will be any significant percentage voting for Tcl.
I know of a large number of systems which offer a shell interface on their command line, and those who don't usually use completely proprietary code. I know of very few examples where Tcl is being used.
Off the top of my head, most of the tooling for FPGAs uses TCL for scripting. OpenOCD uses it too.
Quoted 28 lines
Quoted 6 lines
I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.
Point taken. But if you think this to an end, the result is: lets write some documentation and explain the limitations of a shell in U-Boot environment, and document the warts and bugs of this (or an updated) version of hush. This should make more users happy than completely new and incompatible stuff.
Frankly, I believe when you run into problems with hush in U-Boot (even the current version) you should lean back and think about what you are doing.
U-Boot is a boot loader, and while it is powerful enough to do complex things, this is not necessarily the most clever approach. 15 years ago, I've written complex update scripts for U-Boot. This was easy enough to do, and worked perfectly. But there are so many limitations in a boot loader environment. We don't do this any more. Instead, we use an OS suitable for such tasks (Linux with SWUpdate).
Quoted 7 lines
And talking about problems and limitations in U-Boot... Is the CLI really our biggest concern right now? None of our users (customers) has asked for a better command interpreter - the question we hear are more like: "When will you support IPv6?", "NFS does not work with recent Linux distros, will this be fixed?", "Can I download over WiFi?", "Can I download using HTTP/HTTPS?", "How can I harden U-Boot for security-critical environments?", etc.
Quoted 1 line
I wanted a better shell, so I worked on it.
This here is also an important point, and why I'm commenting on the series. A developer sees a problem, and works on the problem. I know I don't comment on as much stuff as I should, but for wide reaching patches, I really really try to.
Signature · 1 line
Tom
Re: [RFC PATCH 02/28] cli: Add LIL shell
In reply to Wolfgang DenkQuoted 9 lines · Wolfgang Denk
Dear Tom,
Quoted 3 lines · In message <20210706154346.GT9516@bill-the-cat> you wrote:
I'm pretty confident that exactly zero people have written complex U-Boot scripts and then been happy about the experience.
I have seen many U-Boot scripts which were pretty complex, but working absolutely reliably.
As have I. We have them in-tree, even. My point is not "you cannot write these". My point is "you come away very disappointed in the interpreter" after writing these. It's certainly not bash/zsh. But it's also not POSIX-sh. It's not modern hush. It's unique, slightly different from everything else people normally have easy access to and those quirks aren't documented really.
Quoted 4 lines
Quoted 3 lines
TCL has its fans. csh has it's fans. The question isn't what's the best desktop shell or general scripting language, but what's the most useful in our environment an use cases.
Quoted 8 lines
Maybe you should try and do a poll of our user base which CLI they _want_? I doubt there will be any significant percentage voting for Tcl.
I know of a large number of systems which offer a shell interface on their command line, and those who don't usually use completely proprietary code. I know of very few examples where Tcl is being used.
A good point, yes.
Quoted 7 lines
Quoted 6 lines
I don't know if it's right either. But drawing on my comment just now and above about complex boot scripts, I also don't know if "it's sh but quirky and incomplete, WHY DOESN'T THIS WORK RIGHT" is better than "It's TCL? I don't know that, let me hit stackoverflow and do a little reading" as would be the common experience. Especially if we document up-front what the quirks we have are.
Quoted 5 lines
Point taken. But if you think this to an end, the result is: lets write some documentation and explain the limitations of a shell in U-Boot environment, and document the warts and bugs of this (or an updated) version of hush. This should make more users happy than completely new and incompatible stuff.
That would be much appreciated. And to reiterate, our current hush isn't going away soon, or likely entirely within the decade. So this would be something long term useful to have written.
Quoted 3 lines
Frankly, I believe when you run into problems with hush in U-Boot (even the current version) you should lean back and think about what you are doing.
Agreed.
Quoted 7 lines
U-Boot is a boot loader, and while it is powerful enough to do complex things, this is not necessarily the most clever approach. 15 years ago, I've written complex update scripts for U-Boot. This was easy enough to do, and worked perfectly. But there are so many limitations in a boot loader environment. We don't do this any more. Instead, we use an OS suitable for such tasks (Linux with SWUpdate).
Yes. Although there are some complex things that we can't push up higher (see distro_bootcmd) and update systems still tend to need something from us (or they do it in grub instead).
Signature · 1 line
Tom
Re: [RFC PATCH 02/28] cli: Add LIL shell
In reply to Wolfgang DenkDear Tom, Sean, Wolfgang and others,
here are some of my opinions for this discussion
- I agree with Wolfgang that there are far better options than a Tcl-like shell, if we want to add another language
- I also think that instead of adding another language, it is more preferable to improve the existing one. Adding a new language will cause more problems in the future:
- I think it can end up with OS distributions needing to write boot scripts in both languages, because they can't be sure which will be compiled into U-Boot
- we will certainly end up with more bugs
- userbase will fragment between the two languages
- I think we can start improving the current U-Boot's shell in ways that are incompatible with upstream Hush.
The idea back then, as I understand it, was to minimize man-hours invested into the CLI code, and so an existing shell was incorporated (with many #ifdef guards). But U-Boot has since evolved so much that it is very probable it would be more economic to simply fork from upsteam Hush, remove all the #ifdefs and start developing features we want in U-Boot. Is upstream Hush even maintained properly? What is the upstream repository? Is it
https://github.com/sheumann/hush?
- even if we decide to stay with upstream Hush and just upgrade U-Boot's Hush to upstream (since it supports functions, arithmetic with $((...)), command substitution with $(...), these are all nice features), it is IMO still better than adding a new language
- one of the points Sean mentioned with LIL is that when compiled, it's size does not exceed the size of U-Boot's Hush.
If we were to add new features into U-Boot's Hush, the code size would certainly increase.
I think we should implement these new features, and instead of adding a new language, we should work on minimizing the code size / resulting U-Boot image size. This is where U-Boot will gain most not only with it's CLI, but also everywhere else. Regarding this,
- we already have LTO
- Simon worked on dtoc so that devicetrees can be compiled into C code
- we can start playing with compression
- either we can compress the whole image for machines with enough RAM but small place for U-Boot (Nokia N900 for example has only 256 KiB space for U-Boot)
- or we can try to invent a way to decompress code when it is needed, for machines with small RAM
Marek
Re: [RFC PATCH 02/28] cli: Add LIL shell
Hi
Quoted 20 lines · Marek Behun
Dear Tom, Sean, Wolfgang and others,
here are some of my opinions for this discussion
- I agree with Wolfgang that there are far better options than a Tcl-like shell, if we want to add another language
- I also think that instead of adding another language, it is more preferable to improve the existing one. Adding a new language will cause more problems in the future:
- I think it can end up with OS distributions needing to write boot scripts in both languages, because they can't be sure which will be compiled into U-Boot
- we will certainly end up with more bugs
- userbase will fragment between the two languages- I think we can start improving the current U-Boot's shell in ways that are incompatible with upstream Hush.
Quoted 9 lines
The idea back then, as I understand it, was to minimize man-hours invested into the CLI code, and so an existing shell was incorporated (with many #ifdef guards). But U-Boot has since evolved so much that it is very probable it would be more economic to simply fork from upsteam Hush, remove all the #ifdefs and start developing features we want in U-Boot. Is upstream Hush even maintained properly? What is the upstream repository? Is it
https://github.com/sheumann/hush?
I think that hush is the one that is now in the busybox. I could spent ten minutes this morning and this is my short list:
- we have several define that allow it to enabled e/o disable a lot of features
- we are talking about 11K lines compared to 3K (including comment)
- we have 25-30 configuration option on hush on busybox
- in u-boot code some of the problem was solved some time ago
- as describe is 68Kb, I think this consider all the option enables
- the code is different from what we have and what is there
I don't know if options like ENABLE_HUSH_JOB and ENABLE_MMU can partially solve some of the problems described in the thread
* Sean *: You have spent more on this, can you please complete it.
Out of that. Do we have some script shell unit test in uboot?
Michael
Quoted 25 lines
- even if we decide to stay with upstream Hush and just upgrade U-Boot's Hush to upstream (since it supports functions, arithmetic with $((...)), command substitution with $(...), these are all nice features), it is IMO still better than adding a new language
- one of the points Sean mentioned with LIL is that when compiled, it's size does not exceed the size of U-Boot's Hush.
If we were to add new features into U-Boot's Hush, the code size would certainly increase.
I think we should implement these new features, and instead of adding a new language, we should work on minimizing the code size / resulting U-Boot image size. This is where U-Boot will gain most not only with it's CLI, but also everywhere else. Regarding this,
- we already have LTO
- Simon worked on dtoc so that devicetrees can be compiled into C code
- we can start playing with compression
- either we can compress the whole image for machines with enough RAM but small place for U-Boot (Nokia N900 for example has only 256 KiB space for U-Boot)
- or we can try to invent a way to decompress code when it is needed, for machines with small RAMMarek
Signature · 11 lines
Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 [email protected] __________________________________ Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 [email protected] www.amarulasolutions.com
Re: [RFC PATCH 02/28] cli: Add LIL shell
Quoted 43 lines · Michael Nazzareno Trimarchi
Hi
Quoted 20 lines · Marek Behun
Dear Tom, Sean, Wolfgang and others,
here are some of my opinions for this discussion
- I agree with Wolfgang that there are far better options than a Tcl-like shell, if we want to add another language
- I also think that instead of adding another language, it is more preferable to improve the existing one. Adding a new language will cause more problems in the future:
- I think it can end up with OS distributions needing to write boot scripts in both languages, because they can't be sure which will be compiled into U-Boot
- we will certainly end up with more bugs
- userbase will fragment between the two languages- I think we can start improving the current U-Boot's shell in ways that are incompatible with upstream Hush.
Quoted 9 lines
The idea back then, as I understand it, was to minimize man-hours invested into the CLI code, and so an existing shell was incorporated (with many #ifdef guards). But U-Boot has since evolved so much that it is very probable it would be more economic to simply fork from upsteam Hush, remove all the #ifdefs and start developing features we want in U-Boot. Is upstream Hush even maintained properly? What is the upstream repository? Is it
https://github.com/sheumann/hush?I think that hush is the one that is now in the busybox. I could spent ten minutes this morning and this is my short list:
- we have several define that allow it to enabled e/o disable a lot of features
- we are talking about 11K lines compared to 3K (including comment)
- we have 25-30 configuration option on hush on busybox
- in u-boot code some of the problem was solved some time ago
- as describe is 68Kb, I think this consider all the option enables
- the code is different from what we have and what is there
Quoted 6 lines
I don't know if options like
ENABLE_HUSH_JOBandENABLE_MMUcan partially solve some of the problems described in the thread* Sean *: You have spent more on this, can you please complete it.
Out of that. Do we have some script shell unit test in uboot?
To the last point, yes, we have some, but always need more, tests for how hush behaves as part of pytest.
Signature · 1 line
Tom
Re: [RFC PATCH 02/28] cli: Add LIL shell
In reply to Wolfgang DenkQuoted 10 lines · Wolfgang Denk
Dear Tom,
Quoted 6 lines · In message <20210705191058.GB9516@bill-the-cat> you wrote:
This is I think the hard question. A draw of the current shell is that it it looks and acts like bash/sh/etc, for at least basic operations. That's something that's comfortable to a large audience. That has disadvantages when people want to start doing something complex. Sean has shown that several times and he's not the only one. LIL being tcl'ish is not.
Quoted 2 lines
Tcl is a horror of a language for anything that is above trivial level.
Can you please elaborate on this? You've made this claim several times, but haven't explained your reasoning for it.
--Sean
Re: [RFC PATCH 02/28] cli: Add LIL shell
Dear Sean,
Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 3 lines
Tcl is a horror of a language for anything that is above trivial level.
Can you please elaborate on this? You've made this claim several times, but haven't explained your reasoning for it.
A long, long time ago I I was looking for an efficient way to do regression testing for embedded systems, mainly U-Boot and Linux. I found GNU gnats to bee too difficult to handle und started writing my own test system (you can still find this ancient stuff here [1]) based on expect/Tcl. At that time, I could not find any other solution, so I had to accept Tcl as programming language for this
tool. And at the beginning it usually worked just fine. Mostly.
Coming from a C programming environment I always found Tcl ... strange. For example it's scoping rules (see [2] for an example). Things got worse when the system grew and we had to deal with strings or parameters which might contail quotes or other special characters. This required fancy quoting, which depended of the calling depth - if there was only on specific level. you might find a solution after some (cumbersome) experimenting. But if the same string was used in several places in the code, and different levels, you really lost. At a certain point all our developers refused to write new test cases for the system, because it always was the same nightmare of fighting unforeseeable language problems.
That was when we abandoned DUTS, and some time later Heiko came up with an initial implementation of tbot, soon to be joined by Harald to make a production quality test system out of it [4].
IMO, Tcl has a large number of situations where it's actual behaviours is not what you expect from it - it is good enough for simple things, but any more complex processing of arbitrary text data quickly drives you insane - things like quotes or braces and other special characters.
[1] http://www.denx.de/wiki/DUTS/DUTSDocs
[2] https://wiki.tcl-lang.org/page/Commonly+encountered+problems+in+Tcl
[3] https://wiki.tcl-lang.org/page/Why+can+I+not+place+unmatched+braces+in+Tcl+comments
[4] https://tbot.tools/
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] The number you have dialed is imaginary. Please divide by 0 and try again.
Re: [RFC PATCH 02/28] cli: Add LIL shell
In reply to Sean AndersonDear Sean,
Quoted 7 lines · In message <[email protected]> you wrote:
Quoted 3 lines
Do you have a list of the exact differencec between LIL and a standard shell?
For a partial list, see
Hm, this list of commits is not exactly helpful, I'm afraid.
Where _exactly_ should I look?
Quoted 5 lines
Quoted 2 lines
I wonder, if we deviate from standard shell syntax anyway, we could also have a look at lua, for example?
I also looked at lua (see the cover letter), but I rejected it based on size constraints (eLua was around the size of U-Boot itself).
I have to admit that I never tried myself to build a lua based system, optimizing for minimal size - but I'm surprised by your results.
Quoted 8 lines
Because of how often the shell is used to debug things, I wanted the candidate I picked to have similar syntax to the existing shell. For example,
load mmc ${mmcdev}:${mmcpart} $loadaddr $imageis a valid command in both Hush and LIL. Compare with lua, which might express the above as
Yes, I'm aware of this. But then, it would add another level of powerful scripting capabilities - and as it was already suggested before, _replacing_ the standard shell is only one way to use lua - another would be to use lua as a command that can be started from the shell when needed - assuming you want to pay the price in terms of size.
Best regards,
Wolfgang Denk
Signature · 6 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] They're usually so busy thinking about what happens next that the only time they ever find out what is happening now is when they come to look back on it. - Terry Pratchett, _Wyrd Sisters_
RFC PATCH 3/28 cli: lil: Replace strclone with strdup
In reply to Sean AndersonApparently strdup is not portable, so LIL used its own. Use strdup.
---
common/cli_lil.c | 27 ++++++++------------------- 1 file changed, 8 insertions(+), 19 deletions(-)
common/cli_lil.c +8 -19
@@ -123,17 +123,6 @@ static struct lil_env **envpool;static size_t envpoolsize, envpoolcap;#endif -static char *strclone (const char *s) -{ -size_t len =strlen (s) +1 ; -char *ns =malloc (len); - -if (!ns) -return NULL ; -memcpy (ns, s, len); -return ns; -} -static unsigned long hm_hash (const char *key) {unsigned long hash =5381 ;
@@ -173,7 +162,7 @@ static void hm_put(struct hashmap *hm, const char *key, void *value) } cell->e =realloc (cell->e,sizeof (struct hashentry ) * (cell->c +1 )); - cell->e[cell->c].k =strclone (key); + cell->e[cell->c].k =strdup (key); cell->e[cell->c].v = value; cell->c++; }
@@ -606,7 +595,7 @@ static struct lil_func *add_func(struct lil *lil, const char *name) } cmd =calloc (1 ,sizeof (struct lil_func )); - cmd->name =strclone (name); + cmd->name =strdup (name); ncmd =realloc (lil->cmd,sizeof (struct lil_func *) * (lil->cmds +1 ));if (!ncmd) {
@@ -705,7 +694,7 @@ struct lil_var *lil_set_var(struct lil *lil, const char *name, env->var = nvar; nvar[env->vars] =calloc (1 ,sizeof (struct lil_var )); - nvar[env->vars]->n =strclone (name); + nvar[env->vars]->n =strdup (name); nvar[env->vars]->env = env; nvar[env->vars]->v = freeval ? val :lil_clone_value (val);hm_put (&env->varmap, name, nvar[env->vars]);
@@ -1207,7 +1196,7 @@ void lil_set_error(struct lil *lil, const char *msg)free (lil->err_msg); lil->error =ERROR_FIXHEAD ; lil->err_head =0 ; - lil->err_msg =strclone (msg ? msg :"" ); + lil->err_msg =strdup (msg ? msg :"" ); }void lil_set_error_at (struct lil *lil,size_t pos,const char *msg)
@@ -1218,7 +1207,7 @@ void lil_set_error_at(struct lil *lil, size_t pos, const char *msg)free (lil->err_msg); lil->error =ERROR_DEFAULT ; lil->err_head = pos; - lil->err_msg =strclone (msg ? msg :"" ); + lil->err_msg =strdup (msg ? msg :"" ); }int lil_error (struct lil *lil,const char **msg,size_t *pos)
@@ -2016,7 +2005,7 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,hm_put (&lil->cmdmap, oldname,0 );hm_put (&lil->cmdmap, newname, func);free (func->name); - func->name =strclone (newname); + func->name =strdup (newname); }else {del_func (lil, func); }
@@ -2728,7 +2717,7 @@ static struct lil_value *real_trim(const char *str, const char *chars, int left,size_t len;char *s; - s =strclone (str + base); + s =strdup (str + base); len =strlen (s);while (len &&strchr (chars, s[len -1 ])) len--;
@@ -2833,7 +2822,7 @@ static struct lil_value *fnc_repstr(struct lil *lil, size_t argc,if (!from[0 ])return NULL ; - src =strclone (lil_to_string (argv[0 ])); + src =strdup (lil_to_string (argv[0 ])); srclen =strlen (src); fromlen =strlen (from); tolen =strlen (to);
Signature · 1 line
2.32.0
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
Quoted 1 line · Sean Anderson
Apparently strdup is not portable, so LIL used its own. Use strdup.
You could reduce the churn by just making strclone "#define strclone(x) strdup(x)", but I suppose you end up modifying the upstream code so much that there's not really anything gained by that.
But that begs the question: What is the long-term plan for this? While it does seem to be an improvement compared to hush, will we ever be able to incorporate fixes&features from upstream, or will this code end up in the same situation as hush?
Have you been in contact with upstream about this project? Perhaps some of the things you do could go upstream - e.g. the conversion from an array of wrongly-typed callbacks to an "ops" struct seems to be an obvious improvement [lil_callback would have to be kept, and changed to use a switch() statement to update the right slot, but we wouldn't have to care because ld will remove it anyway if we don't add any users of it].
Rasmus
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
Dear Rasmus,
Quoted 5 lines · In message <[email protected]> you wrote:
But that begs the question: What is the long-term plan for this? While it does seem to be an improvement compared to hush, will we ever be able to incorporate fixes&features from upstream, or will this code end up in the same situation as hush?
I would like to put the context of this right.
As written, "improvement compared to hush", this could be misunderstood. What you likely mean is "compared to our ancient (nearly 20 years old) version of hush".
Comparing to recent versions have probably much different results.
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] Ever try. Ever fail. No matter. Try again. Fail again. Fail better. -- S. Beckett
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
In reply to Rasmus VillemoesQuoted 11 lines · Rasmus Villemoes
Quoted 1 line · Sean Anderson
Apparently strdup is not portable, so LIL used its own. Use strdup.
You could reduce the churn by just making strclone "#define strclone(x) strdup(x)", but I suppose you end up modifying the upstream code so much that there's not really anything gained by that.
But that begs the question: What is the long-term plan for this? While it does seem to be an improvement compared to hush, will we ever be able to incorporate fixes&features from upstream, or will this code end up in the same situation as hush?
Well, since Hush was never updated, I don't believe LIL will be either. I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.
The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.
Quoted 7 lines
Have you been in contact with upstream about this project? Perhaps some of the things you do could go upstream - e.g. the conversion from an array of wrongly-typed callbacks to an "ops" struct seems to be an obvious improvement [
lil_callbackwould have to be kept, and changed to use aswitch()statement to update the right slot, but we wouldn't have to care because ld will remove it anyway if we don't add any users of it].
I have not. I'm not sure what changes I've made are compatible with the vision he has for LIL. He's CC'd on this series, so perhaps he can comment.
--Sean
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
Quoted 18 lines · Sean Anderson
Quoted 11 lines · Rasmus Villemoes
Quoted 1 line · Sean Anderson
Apparently strdup is not portable, so LIL used its own. Use strdup.
You could reduce the churn by just making strclone "#define strclone(x) strdup(x)", but I suppose you end up modifying the upstream code so much that there's not really anything gained by that.
But that begs the question: What is the long-term plan for this? While it does seem to be an improvement compared to hush, will we ever be able to incorporate fixes&features from upstream, or will this code end up in the same situation as hush?
Well, since Hush was never updated, I don't believe LIL will be either. I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.
Quoted 4 lines
The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.
Port it and forget it was a problem, not a bug, of our hush implementation. For other things, I'm trying to keep us in sync regularly, but having less luck as some stuff has already gotten well behind and is non-trivial to resync. A digression, but for this thread I think keeping abreast of LIL releases is important. I diff'd 20190821 to 20210502 and it was (as you note in another part of the thread) copyright, license addition and I saw one line of code move around but didn't check if that applied here too or not. A monthly calendar reminder to check the site for new releases (which is what I do for Kconfiglib) should keep us in-sync, especially given the LIL project intentions.
Signature · 1 line
Tom
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
In reply to Sean AndersonQuoted 3 lines · Sean Anderson
I have not. I'm not sure what changes I've made are compatible with the vision he has for LIL. He's CC'd on this series, so perhaps he can comment.
Yeah, sadly several of the changes i've seen in the code are not backwards compatible and backwards compatibility is very important for me (this is also why the callback setup is done through a lil_callback function instead of exposing a struct: adding new callbacks in the future wont break any existing code or application that links to the library dynamically). I might break it before a properly versioned release is made at some point in the (not close) future, but that is usually unavoidable changes to fix big issues (all cases where that happened are mentioned in the site).
In terms of future changes, i do not plan making any *big* changes - and certainly anything i'll make it'll be backwards compatible (especially after a versioned release) but one thing that i need to do is to improve the interpreter's performance, which will affect its internals - two areas i might need to change are how values are represented (everything is a string -and things must always behave like that- but i may need to cache already parsed non-string values) and how the code itself -e.g. in functions- is represented.
Also i'll most likely add more functions to the library to expose and alter internal state (the Free Pascal version of LIL already has some of that), but that shouldn't affect much internally.
Beyond that it will be mainly bugfixes - like a use-after-free case with "reflect this" i just fixed and uploaded :-P
Kostas
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
Quoted 8 lines · Kostas Michalopoulos
Quoted 3 lines · Sean Anderson
I have not. I'm not sure what changes I've made are compatible with the vision he has for LIL. He's CC'd on this series, so perhaps he can comment.
Yeah, sadly several of the changes i've seen in the code are not backwards compatible and backwards compatibility is very important for me
I don't intend to keep backward compatibility. The way I see it, U-Boot has never had a LIL shell before, so there is nothing to break :)
Quoted 4 lines
(this is also why the callback setup is done through a
lil_callbackfunction instead of exposing a struct: adding new callbacks in the future wont break any existing code or application that links to the library dynamically).
Well, C has a very nice ABI in some repects. As long as you only append to that struct you will not break compatibility.
Quoted 5 lines
I might break it before a properly versioned release is made at some point in the (not close) future, but that is usually unavoidable changes to fix big issues (all cases where that happened are mentioned in the site).
Quoted 8 lines
In terms of future changes, i do not plan making any *big* changes - and certainly anything i'll make it'll be backwards compatible (especially after a versioned release) but one thing that i need to do is to improve the interpreter's performance, which will affect its internals - two areas i might need to change are how values are represented (everything is a string -and things must always behave like that- but i may need to cache already parsed non-string values) and how the code itself -e.g. in functions- is represented.
One of the nice side-effects of adding a separate parser is that it will now be very easy to get some performance gains by parsing code once and then re-using that parse on subsequent evaluations. For example, functions could be parsed on first use, and cached. Loop bodies could re-used for subsequent iterations.
Quoted 6 lines
Also i'll most likely add more functions to the library to expose and alter internal state (the Free Pascal version of LIL already has some of that), but that shouldn't affect much internally.
Beyond that it will be mainly bugfixes - like a use-after-free case with "reflect this" i just fixed and uploaded :-P
As it happens, I ended up removing that because there was no longer a "rootcode" after removing the parsing step. Perhaps at some point I will add it back, but I do not see a pressing need for the same level of dynamicness that you built in originally.
--Sean
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
In reply to Sean AndersonDear Sean,
Quoted 2 lines · In message <[email protected]> you wrote:
Well, since Hush was never updated, I don't believe LIL will be either.
Let's please be exact here: Hus has never been updated _in_U-Boot_, but it has seen a lot of changes upstream, which apparently fix all the issues that motivated you to look for a replacement.
Quoted 8 lines
I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.
The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.
This sounds like an advantage, indeed, but then you can also interpret this as betting on a dead horse...
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] Never worry about theory as long as the machinery does what it's supposed to do. - R. A. Heinlein
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
Quoted 15 lines · Wolfgang Denk
Dear Sean,
Quoted 2 lines · In message <[email protected]> you wrote:
Well, since Hush was never updated, I don't believe LIL will be either.
Let's please be exact here: Hus has never been updated _in_U-Boot_, but it has seen a lot of changes upstream, which apparently fix all the issues that motivated you to look for a replacement.
Quoted 4 lines
I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.
Quoted 7 lines
Quoted 4 lines
The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.
This sounds like an advantage, indeed, but then you can also interpret this as betting on a dead horse...
My 2c on this.
I am the maintainer of JimTcl (and I agree it is too big to be considered a candidate). LIL source code has almost zero comments, poor error checking and no test suite. I would be very hesitant to adopt it in u-boot without serious work.
I would much rather see effort put into updating hush to upstream.
My guess is that Denys would be amenable to small changes to make it easier to synchronise with busybox in the future.
Cheers,
Steve
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
Quoted 21 lines · Steve Bennett
Quoted 20 lines · Wolfgang Denk
Dear Sean,
Quoted 2 lines · In message <[email protected]> you wrote:
Well, since Hush was never updated, I don't believe LIL will be either.
Let's please be exact here: Hus has never been updated _in_U-Boot_, but it has seen a lot of changes upstream, which apparently fix all the issues that motivated you to look for a replacement.
Quoted 8 lines
I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.
The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.
Quoted 7 lines
Quoted 2 lines
This sounds like an advantage, indeed, but then you can also interpret this as betting on a dead horse...
My 2c on this.
I am the maintainer of JimTcl (and I agree it is too big to be considered a candidate). LIL source code has almost zero comments, poor error checking and no test suite.
FWIW I added a (small) test suite in "[RFC PATCH 17/28] test: Add tests for LIL" based on the tests included in the LIL distibution. However, I really would like to expand upon it.
Quoted 1 line
I would be very hesitant to adopt it in u-boot without serious work.
I think around half of the "serious work" has already been done. I have worked on most of the core of LIL, and added error handling and comments. I believe that most of the remaining instances of dropping errors lie in the built-in commands.
Quoted 1 line
I would much rather see effort put into updating hush to upstream.
AIUI hush has diverged significantly from what U-Boot has. This would not be an "update" moreso than a complete port in the style of the current series.
Quoted 2 lines
My guess is that Denys would be amenable to small changes to make it easier to synchronise with busybox in the future.
I don't think sh-style shells are a good match for U-Boot's execution environment in the first place. The fundamental idea of an sh-style shell is that the output of one command can be redirected to the input (or arguments) of another command. This cannot be done (or rather would be difficult to do) in U-Boot for a few reasons
* U-Boot does not support multithreading. Existing shells tend to depend strongly on this feature of the enviromnent. Many of the changes to U-Boot's hush are solely to deal with the lack of this feature.
* Existing commands do not read from stdin, nor do they print useful information to stdout. Command output is designed for human consumption and is substantially more verbose than typical unix commands.
* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely useful when working with streams are not present in U-Boot.
And of course, this feature is currently not present in U-Boot. To get around this, commands resort to two of my least-favorite hacks: passing in the name of a environmental variable and overloading the return value. For an example of the first, consider
=> part uuid mmc 0:1 my_uuid
which will set my_uuid to the uuid of the selected partition. My issue with this is threefold: every command must add new syntax to do this, that syntax is inconsistent, and it prevents easy composition. Consider a script which wants to iterate over partitions. Instead of doing
for p in $(part list mmc 0); do # ... done
it must instead do
part list mmc 0 partitions
for p in $partitions; do
# ...
done
which unnecessarily adds an extra step. This overhead accumulates with each command which adds something like this.
The other way to return more information is to use the return value. Consider the button command; it currently returns
0 ON, the button is pressed
1 OFF, the button is released
0 button list was shown
1 button not found
1 invalid arguments
and so there is no way to distinguish between whether the button is off, whether the button does not exist, or whether there was a problem with the button driver.
Both of these workarounds are natural consequences of using a sh-tyle shell in an environment it is not suited for. If we are going to go to the effort of porting a new language (which must be done no matter if we use Hush or some other language), we should pick one which has better support for single-threaded programming.
--Sean
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
Hi,
Quoted 108 lines · Sean Anderson
Quoted 21 lines · Steve Bennett
Quoted 20 lines · Wolfgang Denk
Dear Sean,
Quoted 2 lines · In message <[email protected]> you wrote:
Well, since Hush was never updated, I don't believe LIL will be either.
Let's please be exact here: Hus has never been updated _in_U-Boot_, but it has seen a lot of changes upstream, which apparently fix all the issues that motivated you to look for a replacement.
Quoted 8 lines
I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.
The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.
Quoted 7 lines
Quoted 2 lines
This sounds like an advantage, indeed, but then you can also interpret this as betting on a dead horse...
My 2c on this.
I am the maintainer of JimTcl (and I agree it is too big to be considered a candidate). LIL source code has almost zero comments, poor error checking and no test suite.
FWIW I added a (small) test suite in "[RFC PATCH 17/28] test: Add tests for LIL" based on the tests included in the LIL distibution. However, I really would like to expand upon it.
Quoted 1 line
I would be very hesitant to adopt it in u-boot without serious work.
I think around half of the "serious work" has already been done. I have worked on most of the core of LIL, and added error handling and comments. I believe that most of the remaining instances of dropping errors lie in the built-in commands.
Quoted 1 line
I would much rather see effort put into updating hush to upstream.
AIUI hush has diverged significantly from what U-Boot has. This would not be an "update" moreso than a complete port in the style of the current series.
Quoted 2 lines
My guess is that Denys would be amenable to small changes to make it easier to synchronise with busybox in the future.
I don't think sh-style shells are a good match for U-Boot's execution environment in the first place. The fundamental idea of an sh-style shell is that the output of one command can be redirected to the input (or arguments) of another command. This cannot be done (or rather would be difficult to do) in U-Boot for a few reasons
* U-Boot does not support multithreading. Existing shells tend to depend strongly on this feature of the enviromnent. Many of the changes to U-Boot's hush are solely to deal with the lack of this feature.
* Existing commands do not read from stdin, nor do they print useful information to stdout. Command output is designed for human consumption and is substantially more verbose than typical unix commands.
* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely useful when working with streams are not present in U-Boot.
And of course, this feature is currently not present in U-Boot. To get around this, commands resort to two of my least-favorite hacks: passing in the name of a environmental variable and overloading the return value. For an example of the first, consider
=> part uuid mmc 0:1
my_uuidwhich will set
my_uuidto the uuid of the selected partition. My issue with this is threefold: every command must add new syntax to do this, that syntax is inconsistent, and it prevents easy composition. Consider a script which wants to iterate over partitions. Instead of doingfor p in $(part list mmc 0); do # ... doneit must instead do
part list mmc 0 partitions
for p in $partitions; do
# ...
donewhich unnecessarily adds an extra step. This overhead accumulates with each command which adds something like this.
The other way to return more information is to use the return value. Consider the button command; it currently returns
0 ON, the button is pressed 1 OFF, the button is released 0 button list was shown 1 button not found 1 invalid argumentsand so there is no way to distinguish between whether the button is off, whether the button does not exist, or whether there was a problem with the button driver.
Quoted 5 lines
Both of these workarounds are natural consequences of using a sh-tyle shell in an environment it is not suited for. If we are going to go to the effort of porting a new language (which must be done no matter if we use Hush or some other language), we should pick one which has better support for single-threaded programming.
I think these are good points, particularly the thing about mutiltasking. In fact I think it would be better if hush could implement things like '| grep xxx' since it would be useful in U-Boot.
But in any case, adding lil does not preclude someone coming along and adaptive hush for U-Boot (and this time upstreaming the changes!). I have seen discussions about updating hush for several years and no one has done it. I took a quick look and found it was about 3x the size it used to be and was not even sure where to start in terms of adapting it for single-threaded use.
Re this patch, I think it should be sent upstream.
Regards,
Simon
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
Quoted 121 lines · Simon Glass
Hi,
Quoted 108 lines · Sean Anderson
Quoted 21 lines · Steve Bennett
Quoted 20 lines · Wolfgang Denk
Dear Sean,
Quoted 2 lines · In message <[email protected]> you wrote:
Well, since Hush was never updated, I don't believe LIL will be either.
Let's please be exact here: Hus has never been updated _in_U-Boot_, but it has seen a lot of changes upstream, which apparently fix all the issues that motivated you to look for a replacement.
Quoted 8 lines
I think reducing the amount of ifdefs makes the code substantially easier to maintain. My intention is to just use LIL as a starting point which can be modified as needed to better suit U-Boot.
The other half of this is that LIL is not particularly actively developed. I believe the author sees his work as essentially feature-complete, so I expect no major features which we might like to backport.
Quoted 7 lines
Quoted 2 lines
This sounds like an advantage, indeed, but then you can also interpret this as betting on a dead horse...
My 2c on this.
I am the maintainer of JimTcl (and I agree it is too big to be considered a candidate). LIL source code has almost zero comments, poor error checking and no test suite.
FWIW I added a (small) test suite in "[RFC PATCH 17/28] test: Add tests for LIL" based on the tests included in the LIL distibution. However, I really would like to expand upon it.
Quoted 1 line
I would be very hesitant to adopt it in u-boot without serious work.
I think around half of the "serious work" has already been done. I have worked on most of the core of LIL, and added error handling and comments. I believe that most of the remaining instances of dropping errors lie in the built-in commands.
Quoted 1 line
I would much rather see effort put into updating hush to upstream.
AIUI hush has diverged significantly from what U-Boot has. This would not be an "update" moreso than a complete port in the style of the current series.
Quoted 2 lines
My guess is that Denys would be amenable to small changes to make it easier to synchronise with busybox in the future.
I don't think sh-style shells are a good match for U-Boot's execution environment in the first place. The fundamental idea of an sh-style shell is that the output of one command can be redirected to the input (or arguments) of another command. This cannot be done (or rather would be difficult to do) in U-Boot for a few reasons
* U-Boot does not support multithreading. Existing shells tend to depend strongly on this feature of the enviromnent. Many of the changes to U-Boot's hush are solely to deal with the lack of this feature.
* Existing commands do not read from stdin, nor do they print useful information to stdout. Command output is designed for human consumption and is substantially more verbose than typical unix commands.
* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely useful when working with streams are not present in U-Boot.
And of course, this feature is currently not present in U-Boot. To get around this, commands resort to two of my least-favorite hacks: passing in the name of a environmental variable and overloading the return value. For an example of the first, consider
=> part uuid mmc 0:1
my_uuidwhich will set
my_uuidto the uuid of the selected partition. My issue with this is threefold: every command must add new syntax to do this, that syntax is inconsistent, and it prevents easy composition. Consider a script which wants to iterate over partitions. Instead of doingfor p in $(part list mmc 0); do # ... doneit must instead do
part list mmc 0 partitions
for p in $partitions; do
# ...
donewhich unnecessarily adds an extra step. This overhead accumulates with each command which adds something like this.
The other way to return more information is to use the return value. Consider the button command; it currently returns
0 ON, the button is pressed 1 OFF, the button is released 0 button list was shown 1 button not found 1 invalid argumentsand so there is no way to distinguish between whether the button is off, whether the button does not exist, or whether there was a problem with the button driver.
Quoted 5 lines
Both of these workarounds are natural consequences of using a sh-tyle shell in an environment it is not suited for. If we are going to go to the effort of porting a new language (which must be done no matter if we use Hush or some other language), we should pick one which has better support for single-threaded programming.
I think these are good points, particularly the thing about mutiltasking. In fact I think it would be better if hush could implement things like '| grep xxx' since it would be useful in U-Boot.
Quoted 8 lines
But in any case, adding lil does not preclude someone coming along and adaptive hush for U-Boot (and this time upstreaming the changes!). I have seen discussions about updating hush for several years and no one has done it. I took a quick look and found it was about 3x the size it used to be and was not even sure where to start in terms of adapting it for single-threaded use.
Re this patch, I think it should be sent upstream.
I believe it will not be accepted, since strdup is POSIX and not C99.
--Sean
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
In reply to Sean AndersonDear Sean,
Quoted 4 lines · In message <[email protected]> you wrote:
AIUI hush has diverged significantly from what U-Boot has. This would not be an "update" moreso than a complete port in the style of the current series.
Agreed. However, as you write this it sounds like a big problem only. I disagree here - it is also a chance to do a few things different than with the original port.
Please keep in mind when this original port of the hush shell was
done: it was a time when many systems came with a total of 4 MB flash memory, and not only U-Boot, but also the Linux kernel and a ram-disk image or such had to fit into that. At that time 40 or 60 kB code size for just a fancy shell was immense!
Under such restrictions (and the need to complete the task with a given budget) many things were just commented out which from today's point of view would be nice to have.
This is not a problem of the code complexity or resources, but only of different requirements and different resources.
Quoted 5 lines
I don't think sh-style shells are a good match for U-Boot's execution environment in the first place. The fundamental idea of an sh-style shell is that the output of one command can be redirected to the input (or arguments) of another command. This cannot be done (or rather would be difficult to do) in U-Boot for a few reasons
There is an old saying:
The loser says: "It might be possible, but it is too difficult." The winner says: "It might be difficult, but it is possible."
Apparently you take another position here than me.
First, I disagree that pipes are the "fundamental idea" of a shell. It is a fundamental idea of the UNIX operating systems, indeed.
But please keep in mind that we are here in a boot loader, not in a full-blown OS context.
Quoted 3 lines
* U-Boot does not support multithreading. Existing shells tend to depend strongly on this feature of the enviromnent. Many of the changes to U-Boot's hush are solely to deal with the lack of this feature.
I disagree to both parts of your statement.
Multithreading (or rather, a concept of separate processes which can eventually even run in parallel) is very nice to have, but it is not mandatory in most cases. Command pipes can almost always be strictly sequentialized and thus run in a single-task environment, too. I'm not sure, but I think I even remember pipes in the "shell" of the CPM "OS" on Z80 processors a number of decades ago...
And the fact that our current version of hush did not attempt to keep these features was much more driven by strict memory footprint limitations that anything else. I claim it would have been possible, and still is.
You also don't mention (and I guess you oversee) another critical
fact: so far, U-Boot has no concept of files. The classic design principle "everything is a file" is missing even more than the concept of processes.
Quoted 4 lines
* Existing commands do not read from stdin, nor do they print useful information to stdout. Command output is designed for human consumption and is substantially more verbose than typical unix commands.
This is a statement which is correct, but it does not contain any pro or con for the discussion here.
And if we had the features, it would be easy to fix.
Quoted 2 lines
* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely useful when working with streams are not present in U-Boot.
You are talking about OS environments here. But we are a boot loader.
Also, this argument is not fair, as the suggested LIL does not provide grep, sed, awk, sort, uniq etc. functionality either, or does it?
Quoted 1 line
And of course, this feature is currently not present in U-Boot. To get
Correct, and for very good reasons.
Quoted 11 lines
which will set
my_uuidto the uuid of the selected partition. My issue with this is threefold: every command must add new syntax to do this, that syntax is inconsistent, and it prevents easy composition. Consider a script which wants to iterate over partitions. Instead of doingfor p in $(part list mmc 0); do # ... doneit must instead do
Quoted 7 lines
part list mmc 0 partitions
for p in $partitions; do
# ...
donewhich unnecessarily adds an extra step. This overhead accumulates with each command which adds something like this.
Apparently you fail to understand that this "extra step" is primarily due to the fact that we are single tasking. Even if we has command substitution (like we could have with hush) the execution sequence would be serialized in exactly the same way under the hood - it's just not as directly visible.
Quoted 2 lines
Both of these workarounds are natural consequences of using a sh-tyle shell in an environment it is not suited for. If we are going to go to
No, they are not. They are much more the consequence of no strict design guidelines for commands, so everybody implements what fits his purposes best. A cleaner approach does not require any of the additional features you've asked for - it would be possible as is.
Quoted 3 lines
the effort of porting a new language (which must be done no matter if we use Hush or some other language), we should pick one which has better support for single-threaded programming.
In which way do you think "a new language" needs to be ported when switching from an old to a new version of hush? It would be still a (mostly) POSIX compatible shell, with some restrictions.
I can fully understand that you defend your proposal, but let's be fair and stick with the facts. Thanks.
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] To get something done, a committee should consist of no more than three men, two of them absent.
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
Quoted 33 lines · Wolfgang Denk
Dear Sean,
Quoted 4 lines · In message <[email protected]> you wrote:
AIUI hush has diverged significantly from what U-Boot has. This would not be an "update" moreso than a complete port in the style of the current series.
Agreed. However, as you write this it sounds like a big problem only. I disagree here - it is also a chance to do a few things different than with the original port.
Please keep in mind when this original port of the hush shell was
done: it was a time when many systems came with a total of 4 MB flash memory, and not only U-Boot, but also the Linux kernel and a ram-disk image or such had to fit into that. At that time 40 or 60 kB code size for just a fancy shell was immense!Under such restrictions (and the need to complete the task with a given budget) many things were just commented out which from today's point of view would be nice to have.
This is not a problem of the code complexity or resources, but only of different requirements and different resources.
Quoted 5 lines
I don't think sh-style shells are a good match for U-Boot's execution environment in the first place. The fundamental idea of an sh-style shell is that the output of one command can be redirected to the input (or arguments) of another command. This cannot be done (or rather would be difficult to do) in U-Boot for a few reasons
Quoted 4 lines
There is an old saying:
The loser says: "It might be possible, but it is too difficult." The winner says: "It might be difficult, but it is possible."
And yet, the winner may also say "this is too difficult to be worth it" instead of laboring inefficiently :)
Quoted 5 lines
Apparently you take another position here than me.
First, I disagree that pipes are the "fundamental idea" of a shell. It is a fundamental idea of the UNIX operating systems, indeed.
Well, I think all programming languages at their core are about transferring information from one part of the program to another. Without pipes, how do bourne-style shells communicate? Sure you can transfer data into functions using arguments, but what about the other way around? Numeric return values? Global variables? I think these are rather anemic compared to proper return values.
Quoted 7 lines
But please keep in mind that we are here in a boot loader, not in a full-blown OS context.
Quoted 3 lines
* U-Boot does not support multithreading. Existing shells tend to depend strongly on this feature of the enviromnent. Many of the changes to U-Boot's hush are solely to deal with the lack of this feature.
Quoted 8 lines
I disagree to both parts of your statement.
Multithreading (or rather, a concept of separate processes which can eventually even run in parallel) is very nice to have, but it is not mandatory in most cases. Command pipes can almost always be strictly sequentialized and thus run in a single-task environment, too. I'm not sure, but I think I even remember pipes in the "shell" of the CPM "OS" on Z80 processors a number of decades ago...
The point here is that many Hush features were written with fork. For example, to create a new scope (such as for a subshell), Hush just fork()s. When parsing, if it encounters a construct like $(), it fork()s and then modifies the map variable, (correctly) assuming that the "original" map will remain unmodified. These things are all over and make doing a port difficult; especially when they crop up as unforseen bugs. Though I suppose the real issue here is a lack of virtual memory.
Quoted 9 lines
And the fact that our current version of hush did not attempt to keep these features was much more driven by strict memory footprint limitations that anything else. I claim it would have been possible, and still is.
You also don't mention (and I guess you oversee) another critical
fact: so far, U-Boot has no concept of files. The classic design principle "everything is a file" is missing even more than the concept of processes.
IMO "everything is a file" is more of an API thing than a shell thing. E.g. the idea that you can use open/read/write/close to access any resource. But without pipes, I don't think such an abstraction is very useful.
Quoted 7 lines
Quoted 4 lines
* Existing commands do not read from stdin, nor do they print useful information to stdout. Command output is designed for human consumption and is substantially more verbose than typical unix commands.
This is a statement which is correct, but it does not contain any pro or con for the discussion here.
In order for pipes to be useful, programs should generally follow the Unix philosophy. In particular, many commands in U-Boot fall afoul of rule 2:
Quoted 4 lines
Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.
Which makes it difficult to compose them.
Quoted 1 line
And if we had the features, it would be easy to fix.
I don't know about that. I think making U-Boot commands emit output designed for machine consumption would require substantial effort.
Quoted 5 lines
Quoted 2 lines
* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely useful when working with streams are not present in U-Boot.
You are talking about OS environments here. But we are a boot loader.
These programs are just as much part of the shell as the builtin commands. They allow one to take input from one program and transform it into the appropriate input to another program. This is especially important for extracting information from programs which were not designed to interface well.
Quoted 3 lines
Also, this argument is not fair, as the suggested LIL does not provide grep, sed, awk, sort, uniq etc. functionality either, or does it?
Of course it is fair. These programs are just as much part of the *nix programming environment as the bourne shells which invoke them.
Quoted 21 lines
Quoted 2 lines
And of course, this feature is currently not present in U-Boot. To get
Correct, and for very good reasons.
Quoted 16 lines
which will set
my_uuidto the uuid of the selected partition. My issue with this is threefold: every command must add new syntax to do this, that syntax is inconsistent, and it prevents easy composition. Consider a script which wants to iterate over partitions. Instead of doingfor p in $(part list mmc 0); do # ... doneit must instead do
part list mmc 0 partitions
for p in $partitions; do
# ...
done
Quoted 8 lines
Quoted 2 lines
which unnecessarily adds an extra step. This overhead accumulates with each command which adds something like this.
Apparently you fail to understand that this "extra step" is primarily due to the fact that we are single tasking. Even if we has command substitution (like we could have with hush) the execution sequence would be serialized in exactly the same way under the hood - it's just not as directly visible.
Please don't assume what I do and do not understand ;)
My point here is that instead of having a consistent interface for passing data from a command to its caller, we instead have an inconsistent one which is different for each command.
Quoted 7 lines
Quoted 2 lines
Both of these workarounds are natural consequences of using a sh-tyle shell in an environment it is not suited for. If we are going to go to
No, they are not. They are much more the consequence of no strict design guidelines for commands, so everybody implements what fits his purposes best. A cleaner approach does not require any of the additional features you've asked for - it would be possible as is.
So you would have every command have a new parameter at the end specifying which environmental variable to put a return value in? IMO such things should be done at the language level so that commands can just call a function like set_result() and let the shell handle the rest.
Quoted 7 lines
Quoted 3 lines
the effort of porting a new language (which must be done no matter if we use Hush or some other language), we should pick one which has better support for single-threaded programming.
In which way do you think "a new language" needs to be ported when switching from an old to a new version of hush? It would be still a (mostly) POSIX compatible shell, with some restrictions.
Modern Hush is completely unrecognizable compared to what we have in U-Boot today. Any "updating" effort would be akin to going over the entire upstream codebase and porting it from scratch.
Quoted 2 lines
I can fully understand that you defend your proposal, but let's be fair and stick with the facts. Thanks.
And I'd appreciate if we could presume good faith of our peers.
--Sean
Re: [RFC PATCH 03/28] cli: lil: Replace strclone with strdup
Dear Sean,
Quoted 5 lines · In message <[email protected]> you wrote:
Without pipes, how do bourne-style shells communicate? Sure you can transfer data into functions using arguments, but what about the other way around? Numeric return values? Global variables? I think these are rather anemic compared to proper return values.
Pipes are basically just a shortcut to avoid buffer space and to make scripting easier and more elegant.
For example:
$ ps aux | grep foo | grep -v grep | awk '{print $2}' | xargs killcan be written without use of pipes as:
$ ps aux >/tmp/$$.1 ; \
grep foo </tmp/$$.1 >/tmp/$$.2 ; \
grep -v grep </tmp/$$.2 >/tmp/$$.1 ; \
awk '{print $2}' </tmp/$$.1 >/tmp/$$.2 ; \
xargs kill </tmp/$$.2 ; \
rm -f /tmp/$$.1 /tmp/$$.2[and of course this could be done in many other ways, too, cleaner and more efficient]
The point is: Pipes are just a shortcut. You can do all the same
using files - of course with different performance and resource usage.
Quoted 3 lines
The point here is that many Hush features were written with fork. For example, to create a new scope (such as for a subshell), Hush just fork()s. When parsing, if it encounters a construct like $(), it fork()s
Yes, of course I am fully aware of this. This is one of the reasons why command substitution was excluded from the inital port of hush to U-Boot - but again: command sustitution is just a shortcut; it is simple enough to write scripts without it, and it is certainly possible to implent this feature in a U-Boot context as well.
The reason it was not done 18 years ago is 1) typical systems at that time did not have the resources (especially not enough ROM for a bigger U-Boot image), and 2) having a shell with basic scripting capabilities was sheer extravagance in a boot loader.
Yes, the situation has changed since then; today a U-Boot image is as big as a v2.2 Linux kernel image was then, and nobody cares.
Quoted 4 lines
and then modifies the map variable, (correctly) assuming that the "original" map will remain unmodified. These things are all over and make doing a port difficult; especially when they crop up as unforseen bugs. Though I suppose the real issue here is a lack of virtual memory.
Not really. At the time of the port the concern was about code size (size of the text segment).
Quoted 4 lines
IMO "everything is a file" is more of an API thing than a shell thing. E.g. the idea that you can use open/read/write/close to access any resource. But without pipes, I don't think such an abstraction is very useful.
Pipes are just a notation to allow for a convenient use of coroutines, even from a command line interface. They make you think of concurrent processing, but (on single core machines - which was all they had at the time this was invented) it was just an emulation of concurrency on a strictly sequential system.
Nothing really fancy at all, once you have the idea. And nothing that could not be emulated in a U-Boot environment, either.
The question is: how urgently do we need it?
My experience is that if anything is _really_ needed, someone will show up and implement it, probably funded by some customer who needs this feature for some product. I have yet to see any such urgent need for pipes in U-Boot.
Quoted 3 lines
Quoted 3 lines
Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats.
That's the theory, yes. In reality, things are a bit different. Look at the output format of classic Unix commands: ls, ps, df, ...
Quoted 2 lines
I don't know about that. I think making U-Boot commands emit output designed for machine consumption would require substantial effort.
Unix commands usually have NOT been designed for "machine consumption", on contrary. Even network protocols, log files atc. were intentionally designed to be humanly readable.
It's the flexibility of the tools which allows for clever usage.
In U-Boot, commands could be cleaned up for better use in pipes - assuming we had those. But there would be no need to do this ina single huge action - it could be done step by step, one command at a time when somebody needs to use it in such a way.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
* Tools such as grep, cut, tr, sed, sort, uniq, etc. which are extremely useful when working with streams are not present in U-Boot.
You are talking about OS environments here. But we are a boot loader.
These programs are just as much part of the shell as the builtin commands.
Sorry, but they are definitely NOT part of the _shell_.
They are part of the standard Unix toolbox for basic text processing, but for many good reasons each of these is a separate program, and the shell knows nothing of these.
Quoted 7 lines
Quoted 3 lines
In which way do you think "a new language" needs to be ported when switching from an old to a new version of hush? It would be still a (mostly) POSIX compatible shell, with some restrictions.
Modern Hush is completely unrecognizable compared to what we have in U-Boot today. Any "updating" effort would be akin to going over the entire upstream codebase and porting it from scratch.
Agreed. But this means porting some code, which still implements the very same language (i. e. "shell"). There would be no new language in this case - just a bigger subset, less restrictions, more options, probably.
Best regards,
Wolfgang Denk
Signature · 4 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] God made machine language; all the rest is the work of man.
RFC PATCH 4/28 cli: lil: Remove most functions by default
In reply to Sean AndersonThis helps reduce the size impact of LIL to approximately that of hush. The builtin functions have been chosen to roughly match the functionality present in hush. In addition, arithmetic is removed from expr to reduce size further.
---
cmd/Kconfig | 10 ++++++ common/cli_lil.c | 83 +++++++++++++++++++++++++++--------------------- 2 files changed, 57 insertions(+), 36 deletions(-)
cmd/Kconfig +10 -0
@@ -32,6 +32,16 @@ config LIL like `proc name {args} {body}' functions or `echo [some command]` command substitution ("tcl scripts"). +if LIL + +config LIL_FULL + bool "Enable all LIL features" + help + This enables all LIL builtin functions, as well as expression support + for arithmetic and bitwise operations. + +endif + config CMDLINE_EDITING bool "Enable command line editing" depends on CMDLINE
common/cli_lil.c +47 -36
135 lines not on this page · raw message
Signature · 1 line
2.32.0
Re: [RFC PATCH 04/28] cli: lil: Remove most functions by default
Quoted 6 lines · Sean Anderson
This helps reduce the size impact of LIL to approximately that of hush. The builtin functions have been chosen to roughly match the functionality present in hush. In addition, arithmetic is removed from expr to reduce size further.
Quoted 6 lines
Signed-off-by Sean Anderson <[email protected]>---
cmd/Kconfig | 10 ++++++ common/cli_lil.c | 83 +++++++++++++++++++++++++++--------------------- 2 files changed, 57 insertions(+), 36 deletions(-)
RFC PATCH 5/28 cli: lil: Rename some functions to be more like TCL
In reply to Sean AndersonSeveral functions have different names than they do in TCL. To make things easier for those familiar with TCL, rename them to their TCL equivalents. At the moment, this is only done for functions not used by LIL_FULL. Some functions need more substantive work to conform them to TCL. For example, in TCL, there is a string function with a subcommand of compare, which is the same as the top-level function compare. Several functions also have no TCL equivalent. Do we need these?
TODO: do this for all functions
---
common/cli_lil.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
common/cli_lil.c +14 -14
@@ -1829,10 +1829,10 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,return lil_clone_value (func->code); } -if (!strcmp (type,"func-count" )) +if (!strcmp (type,"proc-count" ))return lil_alloc_integer (lil->cmds); -if (!strcmp (type,"funcs" )) { +if (!strcmp (type,"procs" )) {struct lil_list *funcs =lil_alloc_list ();for (i =0 ; i < lil->cmds; i++)
@@ -1878,7 +1878,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,return r; } -if (!strcmp (type,"has-func" )) { +if (!strcmp (type,"has-proc" )) {const char *target;if (argc ==1 )
@@ -1948,7 +1948,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,return NULL ; } -static struct lil_value *fnc_func (struct lil *lil,size_t argc, +static struct lil_value *fnc_proc (struct lil *lil,size_t argc,struct lil_value **argv) {struct lil_value *name;
@@ -2472,8 +2472,8 @@ static struct lil_value *real_inc(struct lil *lil, const char *varname,return pv; } -static struct lil_value *fnc_inc (struct lil *lil,size_t argc, -struct lil_value **argv) +static struct lil_value *fnc_incr (struct lil *lil,size_t argc, +struct lil_value **argv) {if (argc <1 )return NULL ;
@@ -2482,8 +2482,8 @@ static struct lil_value *fnc_inc(struct lil *lil, size_t argc, argc > 1 ? lil_to_integer(argv[1]) : 1); } -static struct lil_value *fnc_dec (struct lil *lil,size_t argc, -struct lil_value **argv) +static struct lil_value *fnc_decr (struct lil *lil,size_t argc, +struct lil_value **argv) {if (argc <1 )return NULL ;
@@ -2786,8 +2786,8 @@ static struct lil_value *fnc_rtrim(struct lil *lil, size_t argc,return real_trim (lil_to_string (argv[0 ]), chars,0 ,1 ); } -static struct lil_value *fnc_strcmp (struct lil *lil,size_t argc, -struct lil_value **argv) +static struct lil_value *fnc_compare (struct lil *lil,size_t argc, +struct lil_value **argv) {if (argc <2 )return NULL ;
@@ -2940,18 +2940,18 @@ static struct lil_value *fnc_lmap(struct lil *lil, size_t argc,static void register_stdcmds (struct lil *lil) { -lil_register (lil,"dec" , fnc_dec); +lil_register (lil,"decr" , fnc_decr);lil_register (lil,"eval" , fnc_eval);lil_register (lil,"expr" , fnc_expr);lil_register (lil,"for" , fnc_for);lil_register (lil,"foreach" , fnc_foreach); -lil_register (lil,"func" , fnc_func); +lil_register (lil,"proc" , fnc_proc);lil_register (lil,"if" , fnc_if); -lil_register (lil,"inc" , fnc_inc); +lil_register (lil,"incr" , fnc_incr);lil_register (lil,"local" , fnc_local);lil_register (lil,"return" , fnc_return);lil_register (lil,"set" , fnc_set); -lil_register (lil,"strcmp" , fnc_strcmp); +lil_register (lil,"compare" , fnc_compare);lil_register (lil,"try" , fnc_try);lil_register (lil,"while" , fnc_while);
Signature · 1 line
2.32.0
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
Hi Sean,
Quoted 9 lines · Sean Anderson
Several functions have different names than they do in TCL. To make things easier for those familiar with TCL, rename them to their TCL equivalents. At the moment, this is only done for functions not used by
LIL_FULL. Some functions need more substantive work to conform them to TCL. For example, in TCL, there is astringfunction with a subcommand ofcompare, which is the same as the top-level functioncompare. Several functions also have no TCL equivalent. Do we need these?
Quoted 7 lines
TODO: do this for all functions
Signed-off-by Sean Anderson <[email protected]>---
common/cli_lil.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
Is your intent to create a fork of this in U-Boot? Could we not update things upstream, at least as an option, to avoid carrying these patches?
Regards,
Simon
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
Quoted 14 lines · Simon Glass
Hi Sean,
Quoted 11 lines · Sean Anderson
Several functions have different names than they do in TCL. To make things easier for those familiar with TCL, rename them to their TCL equivalents. At the moment, this is only done for functions not used by
LIL_FULL. Some functions need more substantive work to conform them to TCL. For example, in TCL, there is astringfunction with a subcommand ofcompare, which is the same as the top-level functioncompare. Several functions also have no TCL equivalent. Do we need these?TODO: do this for all functions
Quoted 7 lines
Quoted 5 lines
Signed-off-by Sean Anderson <[email protected]>---
common/cli_lil.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)Is your intent to create a fork of this in U-Boot?
Yes. I believe some of the major additions I have made (especially "[RFC PATCH 21/28] cli: lil: Add a distinct parsing step") would not be accepted by upstream.
Quoted 2 lines
Could we not update things upstream, at least as an option, to avoid carrying these patches?
For some of the smaller patches, that may be possible. However, I am not a fan of the major amount of ifdefs that Hush has. For something as core as a shell, I think we should be free to make changes as we see fit without worrying about how it will affect a hypothetical backport.
For this patch in particular, I believe upstream would no accept it because it would break backwards compatibility for existing LIL users. However, I view compatibility with TCL as a whole more valuble than compatibility with LIL.
--Sean
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
Dear Sean,
Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 2 lines
Is your intent to create a fork of this in U-Boot?
Yes. I believe some of the major additions I have made (especially "[RFC PATCH 21/28] cli: lil: Add a distinct parsing step") would not be accepted by upstream.
Ouch...
Quoted 7 lines
Quoted 2 lines
Could we not update things upstream, at least as an option, to avoid carrying these patches?
For some of the smaller patches, that may be possible. However, I am not a fan of the major amount of ifdefs that Hush has. For something as core as a shell, I think we should be free to make changes as we see fit without worrying about how it will affect a hypothetical backport.
I'm afraind I cannot understand your thinking.
You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream, but cannot be easily fixed in U-Boot because we essentially created an unmaintained fork - and as a cure, you recommend to do the same thing again, but this time intentionally and deliberately?
If you had not apparently already invested a lot of effort into this thing I would assume you must be joking...
To me such an approach is unacceptable.
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] If a train station is a place where a train stops, then what's a workstation?
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
Quoted 29 lines · Wolfgang Denk
Dear Sean,
Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 2 lines
Is your intent to create a fork of this in U-Boot?
Yes. I believe some of the major additions I have made (especially "[RFC PATCH 21/28] cli: lil: Add a distinct parsing step") would not be accepted by upstream.
Ouch...
Quoted 7 lines
Quoted 2 lines
Could we not update things upstream, at least as an option, to avoid carrying these patches?
For some of the smaller patches, that may be possible. However, I am not a fan of the major amount of ifdefs that Hush has. For something as core as a shell, I think we should be free to make changes as we see fit without worrying about how it will affect a hypothetical backport.
I'm afraind I cannot understand your thinking.
You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream, but cannot be easily fixed in U-Boot because we essentially created an unmaintained fork - and as a cure, you recommend to do the same thing again, but this time intentionally and deliberately?
Quoted 4 lines
If you had not apparently already invested a lot of effort into this thing I would assume you must be joking...
To me such an approach is unacceptable.
I think I want to try and address this. While with "hush" we have something that's in heavy active development outside of U-Boot, with LIL we have something that's mature and "done". Tracking an active outside development is HARD and requires constant resync. Look at the last few LIL releases. That could be easily re-worked in to our fork if needed. I see that as a positive, not a negative.
Signature · 1 line
Tom
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
Hi Tom,
Quoted 34 lines · Tom Rini
Quoted 32 lines · Wolfgang Denk
Dear Sean,
Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 2 lines
Is your intent to create a fork of this in U-Boot?
Yes. I believe some of the major additions I have made (especially "[RFC PATCH 21/28] cli: lil: Add a distinct parsing step") would not be accepted by upstream.
Ouch...
Quoted 7 lines
Quoted 2 lines
Could we not update things upstream, at least as an option, to avoid carrying these patches?
For some of the smaller patches, that may be possible. However, I am not a fan of the major amount of ifdefs that Hush has. For something as core as a shell, I think we should be free to make changes as we see fit without worrying about how it will affect a hypothetical backport.
I'm afraind I cannot understand your thinking.
You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream, but cannot be easily fixed in U-Boot because we essentially created an unmaintained fork - and as a cure, you recommend to do the same thing again, but this time intentionally and deliberately?
If you had not apparently already invested a lot of effort into this thing I would assume you must be joking...
Quoted 8 lines
Quoted 1 line
To me such an approach is unacceptable.
I think I want to try and address this. While with "hush" we have something that's in heavy active development outside of U-Boot, with LIL we have something that's mature and "done". Tracking an active outside development is HARD and requires constant resync. Look at the last few LIL releases. That could be easily re-worked in to our fork if needed. I see that as a positive, not a negative.
Yes I wondered about that, since hush has been in busybox, an active project, all these years. I think this is a good point and perhaps means that forking it should not be too much of a concern. Still, if we can send some things upstream, we should.
Regards,
Simon
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
Quoted 46 lines · Simon Glass
Hi Tom,
Quoted 34 lines · Tom Rini
Quoted 32 lines · Wolfgang Denk
Dear Sean,
Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 2 lines
Is your intent to create a fork of this in U-Boot?
Yes. I believe some of the major additions I have made (especially "[RFC PATCH 21/28] cli: lil: Add a distinct parsing step") would not be accepted by upstream.
Ouch...
Quoted 7 lines
Quoted 2 lines
Could we not update things upstream, at least as an option, to avoid carrying these patches?
For some of the smaller patches, that may be possible. However, I am not a fan of the major amount of ifdefs that Hush has. For something as core as a shell, I think we should be free to make changes as we see fit without worrying about how it will affect a hypothetical backport.
I'm afraind I cannot understand your thinking.
You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream, but cannot be easily fixed in U-Boot because we essentially created an unmaintained fork - and as a cure, you recommend to do the same thing again, but this time intentionally and deliberately?
If you had not apparently already invested a lot of effort into this thing I would assume you must be joking...
Quoted 8 lines
Quoted 1 line
To me such an approach is unacceptable.
I think I want to try and address this. While with "hush" we have something that's in heavy active development outside of U-Boot, with LIL we have something that's mature and "done". Tracking an active outside development is HARD and requires constant resync. Look at the last few LIL releases. That could be easily re-worked in to our fork if needed. I see that as a positive, not a negative.
Quoted 4 lines
Yes I wondered about that, since hush has been in busybox, an active project, all these years. I think this is a good point and perhaps means that forking it should not be too much of a concern. Still, if we can send some things upstream, we should.
Well, that's part of the problem. If we re-port modern hush, who is going to keep it in sync, and how often? Yes, there will be changes we don't need, but there will be bugfixes we do as well.
Signature · 1 line
Tom
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
In reply to Tom RiniDear Tom,
Quoted 4 lines · In message <20210705185141.GA9516@bill-the-cat> you wrote:
I think I want to try and address this. While with "hush" we have something that's in heavy active development outside of U-Boot, with LIL we have something that's mature and "done".
Mature? And still without consequent error checking? And done, i. e. this will never be fixed?
Quoted 2 lines
Tracking an active outside development is HARD and requires constant resync.
Based on that logic we should stop supporting Linux, and stop using DTs or file systems - all of these are under "active outside development".
This is a bogus argiment.
Quoted 3 lines
Look at the last few
LIL releases. That could be easily re-worked in to our fork if needed. I see that as a positive, not a negative.
OK, this is your opinion. Mine differs.
Please consider this as a full NAK from me when when you think of it as a _replacement_ (even an optional one) of the standard shell. If you like, have it added as an _additional_ command, of course fully optional and without impact on the rest of U-Boot if not intentionally selected.
Best regards,
Wolfgang Denk
Signature · 4 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] Of all the things I've lost, I miss my mind the most.
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
Hi Wolfgang,
Quoted 27 lines · Wolfgang Denk
Dear Tom,
Quoted 4 lines · In message <20210705185141.GA9516@bill-the-cat> you wrote:
I think I want to try and address this. While with "hush" we have something that's in heavy active development outside of U-Boot, with LIL we have something that's mature and "done".
Mature? And still without consequent error checking? And done, i. e. this will never be fixed?
Quoted 2 lines
Tracking an active outside development is HARD and requires constant resync.
Based on that logic we should stop supporting Linux, and stop using DTs or file systems - all of these are under "active outside development".
This is a bogus argiment.
Quoted 3 lines
Look at the last few
LIL releases. That could be easily re-worked in to our fork if needed. I see that as a positive, not a negative.
Quoted 7 lines
OK, this is your opinion. Mine differs.
Please consider this as a full NAK from me when when you think of it as a _replacement_ (even an optional one) of the standard shell. If you like, have it added as an _additional_ command, of course fully optional and without impact on the rest of U-Boot if not intentionally selected.
That seems reasonable to me. There is certainly interest in these threads in reporting hush, perhaps with a goal of upstreaming the U-Boot-specific changes back to busybox. But so far no one has done it...
Regards,
Simon
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
In reply to Wolfgang DenkQuoted 10 lines · Wolfgang Denk
Dear Tom,
Quoted 4 lines · In message <20210705185141.GA9516@bill-the-cat> you wrote:
I think I want to try and address this. While with "hush" we have something that's in heavy active development outside of U-Boot, with LIL we have something that's mature and "done".
Mature? And still without consequent error checking? And done, i. e. this will never be fixed?
Intentional design by upstream, and then for the actual problem part (error checking, test suite), Sean is saying he'll fix it, and has started on it.
Quoted 8 lines
Quoted 2 lines
Tracking an active outside development is HARD and requires constant resync.
Based on that logic we should stop supporting Linux, and stop using DTs or file systems - all of these are under "active outside development".
This is a bogus argiment.
Alright, go and update our Kbuild integration to be in sync with current Linux. Or just v5.0. I'd suggest picking up the metadata_csum support patch but someone did say they'd try and update that (off-list) so that ext4 filesystem made with default options in the last 5 years (not exaggerating) work. I'm not even going to talk about the various level of out of sync our DTs files are but I do have some hope there will be more re-syncs especially as it comes to more light that the DT U-Boot uses can just be the DT the OS gets. Then tell me it's easy to keep this stuff in sync. Because we certainly don't want to just fork-and-forget again, right? Because that's why we have both crazy parse bugs as well as lack of now old-and-expected features.
OK, snark aside, I'm very serious here, any "we'll just import ..." needs to have a plan to keep it up to date, or be easy enough to do such that I can set a monthly reminder to check for and do the update. Every area where we don't do this is a set of problems waiting to get worse, as we can see with the hush shell right now as it's one of the oldest things we stopped syncing with.
Quoted 11 lines
Quoted 3 lines
Look at the last few
LIL releases. That could be easily re-worked in to our fork if needed. I see that as a positive, not a negative.OK, this is your opinion. Mine differs.
Please consider this as a full NAK from me when when you think of it as a _replacement_ (even an optional one) of the standard shell. If you like, have it added as an _additional_ command, of course fully optional and without impact on the rest of U-Boot if not intentionally selected.
Any new cli for U-Boot won't be a default build as a command for a long while after it's merged. I'm not sure how much past that point it would further need to be to become the default cli. And I'll even repeat what I said elsewhere about how an sh syntax is a good user feature. But I really think we want a shell environment that is not actively adding new features is a good thing, for the default. Just how much stuff should we be doing or need to be doing before we hand things over to the OS?
Signature · 1 line
Tom
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
Quoted 6 lines · Tom Rini
Quoted 2 lines
Mature? And still without consequent error checking? And done, i. e. this will never be fixed?
Intentional design by upstream, and then for the actual problem part (error checking, test suite), Sean is saying he'll fix it, and has started on it.
To clarify, the intentional design aspect is for the language (e.g. giving garbage to "expr"). The library does need better memory alloc checks (this is the main thing missing because i never had to use LIL on an environment without virtual memory... or where malloc would ever practically fail), though IIRC there aren't any other cases where failure could happen and isn't checked. Similarly i do plan on implementing a proper test suite (currently there is a shell script that runs all the examples in the source code - many examples were made to expose previous bugs - with two different implementations which can be used to find regressions in subsequent releases and differences between implementations, but the examples do not exhaust the code paths the parser could take) at some point - the latter is more likely to happen soon than the former.
The same applies with LIL being "done" (well, mostly done) - this is about the language, not the implementation which (as i mentioned in another email) does need improvements, especially around performance (which is also a reason i need to implement better automated tests).
Kostas
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
In reply to Tom RiniDear Tom,
Quoted 7 lines · In message <20210706153327.GS9516@bill-the-cat> you wrote:
Quoted 3 lines
Mature? And still without consequent error checking? And done, i. e. this will never be fixed?
Intentional design by upstream, and then for the actual problem part (error checking, test suite), Sean is saying he'll fix it, and has started on it.
Seriously - any piece of software that omits error checking intentionally be design should be indented six feet downward and covered with dirt. We should not even consider looking at it.
Quoted 6 lines
OK, snark aside, I'm very serious here, any "we'll just import ..." needs to have a plan to keep it up to date, or be easy enough to do such that I can set a monthly reminder to check for and do the update. Every area where we don't do this is a set of problems waiting to get worse, as we can see with the hush shell right now as it's one of the oldest things we stopped syncing with.
Which exact _new_ problems do we see with hush right now? I can only see old ones, that have been known (and worked around) for nearly two decades.
The limitations and bugs have all been there since the beginning - the limitations actually being intentional due to the typical resource situation at that time.
Quoted 4 lines
But I
really think we want a shell environment that is not actively adding new features is a good thing, for the default. Just how much stuff should we be doing or need to be doing before we hand things over to the OS?
You are shooting yourself in the knee here.
If you think out CLI should not be adding new features, then we should just stick with our ancient hush and neither update it nor replace it with something else that adds not only new features, but breaks backward compatibility, hard.
Best regards,
Wolfgang Denk
Signature · 6 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] God runs electromagnetics by wave theory on Monday, Wednesday, and Friday, and the Devil runs them by quantum theory on Tuesday, Thurs- day, and Saturday. -- William Bragg
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
Quoted 22 lines · Wolfgang Denk
Dear Tom,
Quoted 7 lines · In message <20210706153327.GS9516@bill-the-cat> you wrote:
Quoted 3 lines
Mature? And still without consequent error checking? And done, i. e. this will never be fixed?
Intentional design by upstream, and then for the actual problem part (error checking, test suite), Sean is saying he'll fix it, and has started on it.
Seriously - any piece of software that omits error checking intentionally be design should be indented six feet downward and covered with dirt. We should not even consider looking at it.
Quoted 6 lines
OK, snark aside, I'm very serious here, any "we'll just import ..." needs to have a plan to keep it up to date, or be easy enough to do such that I can set a monthly reminder to check for and do the update. Every area where we don't do this is a set of problems waiting to get worse, as we can see with the hush shell right now as it's one of the oldest things we stopped syncing with.
Quoted 7 lines
Which exact _new_ problems do we see with hush right now? I can only see old ones, that have been known (and worked around) for nearly two decades.
The limitations and bugs have all been there since the beginning - the limitations actually being intentional due to the typical resource situation at that time.
There's all of the new features that've been suggested for our current hush, many by Sean, for which the reply has been "our hush is old, it should be updated!".
And you didn't address my point, taking code from another project requires dedicated maintenance to keep it up to date and we do have a lot of in-tree examples right now of where that lack of dedicated maintenance is a problem.
Quoted 11 lines
Quoted 4 lines
But I
really think we want a shell environment that is not actively adding new features is a good thing, for the default. Just how much stuff should we be doing or need to be doing before we hand things over to the OS?You are shooting yourself in the knee here.
If you think out CLI should not be adding new features, then we should just stick with our ancient hush and neither update it nor replace it with something else that adds not only new features, but breaks backward compatibility, hard.
Honestly, adding new features to the CLI (which is NOT the same as adding new commands, or enhancing existing commands) is very low on my priority list. We need to get the Kconfig migration done. We need to get DM migration done (which will help with the Kconfig one by showing hardware no one cares for anymore, and reducing symbols). We need to make adding new hardware easier. We need to get DTS files in-sync more often.
There are very important developer use cases for a more flexible CLI interpreter. But the main use case is still "boot the (redundant) OS ASAP".
So, what is my priority on this series right here? Well, a developer has posted a series. They believe it's useful and addressing a long-standing problem. They want feedback. I'm trying to provide feedback as it's to a general area of the codebase.
Signature · 1 line
Tom
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
In reply to Wolfgang DenkQuoted 11 lines · Wolfgang Denk
Dear Sean,
Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 2 lines
Is your intent to create a fork of this in U-Boot?
Yes. I believe some of the major additions I have made (especially "[RFC PATCH 21/28] cli: lil: Add a distinct parsing step") would not be accepted by upstream.
Ouch...
I don't particularly mind if Kostas doesn't view these patches as good additions to upstream. We have different goals and requirements, and so not all changes will be compatible.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Could we not update things upstream, at least as an option, to avoid carrying these patches?
For some of the smaller patches, that may be possible. However, I am not a fan of the major amount of ifdefs that Hush has. For something as core as a shell, I think we should be free to make changes as we see fit without worrying about how it will affect a hypothetical backport.
Quoted 4 lines
I'm afraind I cannot understand your thinking.
You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream,
The bugs are fairly minor. The particular characteristics of Hush have not changed. These characteristics make Hush difficult to adapt to the limitations of U-Boot. When we cannot support the basic abstractions expected by Hush, the shell will necessarily change for the worse.
Quoted 1 line
but cannot be easily fixed in U-Boot
Because they are core to the design of Hush (and other bourne derived shells).
Quoted 1 line
because we essentially created an unmaintained fork
I plan to maintain this fork.
--Sean
Quoted 11 lines
- and as a cure, you recommend to do the same
thing again, but this time intentionally and deliberately? If you had not apparently already invested a lot of effort into this thing I would assume you must be joking...To me such an approach is unacceptable.
Best regards,
Wolfgang Denk
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
Dear Sean,
Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 3 lines
You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream,
The bugs are fairly minor. The particular characteristics of Hush have not changed. These characteristics make Hush difficult to adapt to the limitations of U-Boot. When we cannot support the basic abstractions expected by Hush, the shell will necessarily change for the worse.
This is not true. Just have a look what hush in a recent version of Busybox offers.
Quoted 4 lines
Quoted 1 line
but cannot be easily fixed in U-Boot
Because they are core to the design of Hush (and other bourne derived shells).
Oh, this is an interesting opinion. I doubt if a majority (or even a significant percentage) of U-Boot users share it. If you were right, there would be far less users of bash (or other "bourne derived shells"). Guess which percentage of users of UNIX operating systems is using a Tcl based command interpreter as their login shell?
Best regards,
Wolfgang Denk
Signature · 6 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] "There are three principal ways to lose money: wine, women, and engi- neers. While the first two are more pleasant, the third is by far the more certain." - Baron Rothschild, ca. 1800
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
Quoted 7 lines · Wolfgang Denk
Dear Sean,
Quoted 4 lines · In message <[email protected]> you wrote:
Quoted 3 lines
You complain that the existing port of hus has a number of severe limitations or bugs which have long been fixed upstream,
Quoted 7 lines
Quoted 4 lines
The bugs are fairly minor. The particular characteristics of Hush have not changed. These characteristics make Hush difficult to adapt to the limitations of U-Boot. When we cannot support the basic abstractions expected by Hush, the shell will necessarily change for the worse.
This is not true. Just have a look what hush in a recent version of Busybox offers.
Busybox runs in a Linux environment. Many of its features rely on the core functionality provided by Linux, which we do not provide in U-Boot. This is what makes porting features difficult.
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
but cannot be easily fixed in U-Boot
Because they are core to the design of Hush (and other bourne derived shells).
Quoted 6 lines
Oh, this is an interesting opinion. I doubt if a majority (or even a significant percentage) of U-Boot users share it. If you were right, there would be far less users of bash (or other "bourne derived shells"). Guess which percentage of users of UNIX operating systems is using a Tcl based command interpreter as their login shell?
And yet, this is not the field we compete in. While bourne-style shells can take advantage of a multi-threaded environment, embedded shells tend to implement a much wider set of languages. See [1] for a survey of examples.
--Sean
Re: [RFC PATCH 05/28] cli: lil: Rename some functions to be more like TCL
Dear Sean,
Quoted 1 line · In message <[email protected]> you wrote:
...
Quoted 6 lines
And yet, this is not the field we compete in. While bourne-style shells can take advantage of a multi-threaded environment, embedded shells tend to implement a much wider set of languages. See [1] for a survey of examples.
"Embedded scripting languages" is probably not a good match for what we are discussing. I don't think this comparison was made for restricted boot loader environments, but for use in general purpose / embedded operating systems.
Hey, they even list Forth there. Maybe somebody should port OpenBoot, so we can have a Forth interpreter as new commandline language? We could implement device trees in the old, traditional way then, too :-)
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] There are bugs and then there are bugs. And then there are bugs. - Karl Lehenbauer
RFC PATCH 6/28 cli: lil: Convert some defines to enums
In reply to Sean AndersonThis converts some defines to enums. This allows them to be documented more easily, and helps the compiler give better errors for switch statements.
---
common/cli_lil.c | 44 +++++++++++++++++++++++--------------------- include/cli_lil.h | 12 +++++++----- 2 files changed, 30 insertions(+), 26 deletions(-)
common/cli_lil.c +23 -21
@@ -24,10 +24,6 @@* overflows and is also useful when running through an automated fuzzer like AFL */ /*#define LIL_ENABLE_RECLIMIT 10000*/ -#define ERROR_NOERROR 0 -#define ERROR_DEFAULT 1 -#define ERROR_FIXHEAD 2 -#define CALLBACKS 8#define HASHMAP_CELLS 256#define HASHMAP_CELLMASK 0xFF
@@ -97,7 +93,11 @@ struct lil {struct lil_env *rootenv;struct lil_env *downenv;struct lil_value *empty; -int error; +enum { +ERROR_NOERROR =0 , +ERROR_DEFAULT , +ERROR_FIXHEAD , + } error;size_t err_head;char *err_msg;lil_callback_proc_t callback[CALLBACKS ];
@@ -108,7 +108,12 @@ struct expreval {const char *code;size_t len, head;ssize_t ival; -int error; +enum { +EERR_NO_ERROR =0 , +EERR_SYNTAX_ERROR , +EERR_DIVISION_BY_ZERO , +EERR_INVALID_EXPRESSION , + } error; };static struct lil_value *next_word (struct lil *lil);
@@ -645,7 +650,7 @@ int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc) }struct lil_var *lil_set_var (struct lil *lil,const char *name, -struct lil_value *val,int local) +struct lil_value *val,enum lil_setvar local) {struct lil_var **nvar; struct lil_env *env =
@@ -1222,11 +1227,6 @@ int lil_error(struct lil *lil, const char **msg, size_t *pos)return 1 ; } -#define EERR_NO_ERROR 0 -#define EERR_SYNTAX_ERROR 1 -#define EERR_DIVISION_BY_ZERO 3 -#define EERR_INVALID_EXPRESSION 4 -static void ee_expr (struct expreval *ee);static int ee_invalidpunct (int ch)
@@ -1673,16 +1673,18 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)ee_expr (&ee);lil_free_value (code); -if (ee.error) { -switch (ee.error) { -case EERR_DIVISION_BY_ZERO : -lil_set_error (lil,"division by zero in expression" ); -break ; -case EERR_SYNTAX_ERROR : -lil_set_error (lil,"expression syntax error" ); -break ; - } +switch (ee.error) { +case EERR_DIVISION_BY_ZERO : +lil_set_error (lil,"division by zero in expression" );return NULL ; +case EERR_SYNTAX_ERROR : +lil_set_error (lil,"expression syntax error" ); +return NULL ; +case EERR_INVALID_EXPRESSION : +lil_set_error (lil,"invalid expression" ); +return NULL ; +case EERR_NO_ERROR : +break ; }return lil_alloc_integer (ee.ival); }
include/cli_lil.h +7 -5
@@ -13,10 +13,12 @@#define LIL_VERSION_STRING "0.1" -#define LIL_SETVAR_GLOBAL 0 -#define LIL_SETVAR_LOCAL 1 -#define LIL_SETVAR_LOCAL_NEW 2 -#define LIL_SETVAR_LOCAL_ONLY 3 +enum lil_setvar { +LIL_SETVAR_GLOBAL =0 , +LIL_SETVAR_LOCAL , +LIL_SETVAR_LOCAL_NEW , +LIL_SETVAR_LOCAL_ONLY , +};#define LIL_CALLBACK_EXIT 0#define LIL_CALLBACK_WRITE 1
@@ -98,7 +100,7 @@ struct lil_env *lil_push_env(struct lil *lil);void lil_pop_env (struct lil *lil);struct lil_var *lil_set_var (struct lil *lil,const char *name, -struct lil_value *val,int local); +struct lil_value *val,enum lil_setvar local);struct lil_value *lil_get_var (struct lil *lil,const char *name);struct lil_value *lil_get_var_or (struct lil *lil,const char *name,struct lil_value *defvalue);
Signature · 1 line
2.32.0
RFC PATCH 7/28 cli: lil: Simplify callbacks
In reply to Sean AndersonWe only need the getvar and setvar callbacks for now (to integrate with the environment). So remove all the other callbacks. Instead of allowing callbacks to be set at any time, just set them once in lil_new. Lastly, get rid of the typedefs and use a struct to pass in the callbacks instead.
---
common/cli_lil.c | 45 ++++++++++++--------------------------------- include/cli_lil.h | 34 ++++++++-------------------------- 2 files changed, 20 insertions(+), 59 deletions(-)
common/cli_lil.c +12 -33
@@ -24,7 +24,6 @@* overflows and is also useful when running through an automated fuzzer like AFL */ /*#define LIL_ENABLE_RECLIMIT 10000*/ -#define CALLBACKS 8#define HASHMAP_CELLS 256#define HASHMAP_CELLMASK 0xFF
@@ -100,7 +99,7 @@ struct lil { } error;size_t err_head;char *err_msg; -lil_callback_proc_t callback[CALLBACKS ]; +struct lil_callbacks callbacks;size_t parse_depth; };
@@ -667,14 +666,10 @@ struct lil_var *lil_set_var(struct lil *lil, const char *name, var->env == lil->rootenv && var->env != env) var =NULL ; -if (((!var && env == lil->rootenv) || - (var && var->env == lil->rootenv)) && - lil->callback[LIL_CALLBACK_SETVAR ]) { -lil_setvar_callback_proc_t proc = - (lil_setvar_callback_proc_t ) - lil->callback[LIL_CALLBACK_SETVAR ]; +if (lil->callbacks.setvar && ((!var && env == lil->rootenv) || + (var && var->env == lil->rootenv))) {struct lil_value *newval = val; -int r =proc (lil, name, &newval); +int r = lil->callbacks.setvar (lil, name, &newval);if (r <0 ) {return NULL ;
@@ -717,14 +712,10 @@ struct lil_value *lil_get_var_or(struct lil *lil, const char *name,struct lil_var *var =lil_find_var (lil, lil->env, name);struct lil_value *retval = var ? var->v : defvalue; -if (lil->callback[LIL_CALLBACK_GETVAR ] && - (!var || var->env == lil->rootenv)) { -lil_getvar_callback_proc_t proc = - (lil_getvar_callback_proc_t ) - lil->callback[LIL_CALLBACK_GETVAR ]; +if (lil->callbacks.getvar && (!var || var->env == lil->rootenv)) {struct lil_value *newretval = retval; -if (proc (lil, name, &newretval)) +if (lil->callbacks.getvar (lil, name, &newretval)) retval = newretval; }return retval;
@@ -748,10 +739,15 @@ void lil_pop_env(struct lil *lil) } } -struct lil *lil_new (void ) +struct lil *lil_new (const struct lil_callbacks *callbacks) {struct lil *lil =calloc (1 ,sizeof (struct lil )); +if (!lil) +return NULL ; + +if (callbacks) +memcpy (&lil->callbacks, callbacks,sizeof (lil->callbacks)); lil->rootenv = lil->env =lil_alloc_env (NULL ); lil->empty =alloc_value (NULL );hm_init (&lil->cmdmap);
@@ -1148,15 +1144,6 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen, } cleanup: -if (lil->error && lil->callback[LIL_CALLBACK_ERROR ] && - lil->parse_depth ==1 ) { -lil_error_callback_proc_t proc = - (lil_error_callback_proc_t ) - lil->callback[LIL_CALLBACK_ERROR ]; - -proc (lil, lil->err_head, lil->err_msg); - } -if (words)lil_free_list (words); lil->code = save_code;
@@ -1185,14 +1172,6 @@ struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,return lil_parse (lil, val->d, val->l, funclevel); } -void lil_callback (struct lil *lil,int cb,lil_callback_proc_t proc) -{ -if (cb <0 || cb >CALLBACKS ) -return ; - - lil->callback[cb] = proc; -} -void lil_set_error (struct lil *lil,const char *msg) {if (lil->error)
include/cli_lil.h +8 -26
@@ -20,15 +20,6 @@ enum lil_setvar { LIL_SETVAR_LOCAL_ONLY, }; -#define LIL_CALLBACK_EXIT 0 -#define LIL_CALLBACK_WRITE 1 -#define LIL_CALLBACK_READ 2 -#define LIL_CALLBACK_STORE 3 -#define LIL_CALLBACK_SOURCE 4 -#define LIL_CALLBACK_ERROR 5 -#define LIL_CALLBACK_SETVAR 6 -#define LIL_CALLBACK_GETVAR 7 -#include <stdint.h> #include <inttypes.h>
@@ -40,22 +31,15 @@ struct lil_list;struct lil ;typedef struct lil_value *(*lil_func_proc_t )(struct lil *lil,size_t argc,struct lil_value **argv); -typedef void (*lil_exit_callback_proc_t )(struct lil *lil, -struct lil_value *arg); -typedef void (*lil_write_callback_proc_t )(struct lil *lil,const char *msg); -typedef char *(*lil_read_callback_proc_t )(struct lil *lil,const char *name); -typedef char *(*lil_source_callback_proc_t )(struct lil *lil,const char *name); -typedef void (*lil_store_callback_proc_t )(struct lil *lil,const char *name, -const char *data); -typedef void (*lil_error_callback_proc_t )(struct lil *lil,size_t pos, -const char *msg); -typedef int (*lil_setvar_callback_proc_t )(struct lil *lil,const char *name, -struct lil_value **value); -typedef int (*lil_getvar_callback_proc_t )(struct lil *lil,const char *name, -struct lil_value **value); -typedef void (*lil_callback_proc_t )(void ); -struct lil *lil_new (void ); +struct lil_callbacks { +int (*setvar)(struct lil *lil,const char *name, +struct lil_value **value); +int (*getvar)(struct lil *lil,const char *name, +struct lil_value **value); +}; + +struct lil *lil_new (const struct lil_callbacks *callbacks);void lil_free (struct lil *lil);int lil_register (struct lil *lil,const char *name,lil_func_proc_t proc);
@@ -65,8 +49,6 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,struct lil_value *lil_parse_value (struct lil *lil,struct lil_value *val,int funclevel); -void lil_callback (struct lil *lil,int cb,lil_callback_proc_t proc); -void lil_set_error (struct lil *lil,const char *msg);void lil_set_error_at (struct lil *lil,size_t pos,const char *msg);int lil_error (struct lil *lil,const char **msg,size_t *pos);
Signature · 1 line
2.32.0
RFC PATCH 8/28 cli: lil: Handle commands with dots
In reply to Sean AndersonSome commands (such as md, mw, etc) use dot suffixes as options to the command. Therefore, we must ignore anything after a dot when finding commands.
---
common/cli_lil.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
common/cli_lil.c +11 -1
@@ -579,7 +579,17 @@ static struct lil_var *lil_find_var(struct lil *lil, struct lil_env *env,static struct lil_func *lil_find_cmd (struct lil *lil,const char *name) { -return hm_get (&lil->cmdmap, name); +struct lil_func *r; +char *dot =strchr (name,'.' ); + +/* Some U-Boot commands have dots in their names */ +if (dot) + *dot ='\0' ; + r =hm_get (&lil->cmdmap, name); + +if (dot) + *dot ='.' ; +return r; }static struct lil_func *add_func (struct lil *lil,const char *name)
Signature · 1 line
2.32.0
RFC PATCH 9/28 cli: lil: Use error codes
In reply to Sean AndersonThis adds numeric error codes to be used for errors. This makes it easier for code to differentiate between different classes of errors. Some error codes are not used yet, and will be added in future commits. The position argument has been removed for a few reasons:
- It doesn't make sense for some errors, such as running out of memory
- It was often not very helpful, since it pointed at the end of the command
- Determining the location of an error is more difficult when the parsing and evaluating steps are done separately.
In addition, lil.error is changed to lil.err for consistency.
---
common/cli_lil.c | 92 ++++++++++++++++++----------------------------- include/cli_lil.h | 36 +++++++++++++++++-- 2 files changed, 68 insertions(+), 60 deletions(-)
common/cli_lil.c +35 -57
@@ -92,12 +92,7 @@ struct lil {struct lil_env *rootenv;struct lil_env *downenv;struct lil_value *empty; -enum { -ERROR_NOERROR =0 , -ERROR_DEFAULT , -ERROR_FIXHEAD , - } error; -size_t err_head; +enum lil_error err;char *err_msg;struct lil_callbacks callbacks;size_t parse_depth;
@@ -117,6 +112,7 @@ struct expreval {static struct lil_value *next_word (struct lil *lil);static void register_stdcmds (struct lil *lil); +static void lil_set_error (struct lil *lil,enum lil_error err,const char *msg);#ifdef LIL_ENABLE_POOLS static struct lil_value **pool;
@@ -974,7 +970,7 @@ static struct lil_list *substitute(struct lil *lil)struct lil_list *words =lil_alloc_list ();lil_skip_spaces (lil); -while (lil->head < lil->clen && !ateol (lil) && !lil->error) { +while (lil->head < lil->clen && !ateol (lil) && !lil->err) {struct lil_value *w =alloc_value (NULL ); do {
@@ -993,7 +989,7 @@ static struct lil_list *substitute(struct lil *lil)lil_free_value (wp); }while (lil->head < lil->clen && !eolchar (lil->code[lil->head]) && - !isspace (lil->code[lil->head]) && !lil->error); + !isspace (lil->code[lil->head]) && !lil->err);lil_skip_spaces (lil);lil_list_append (words, w);
@@ -1042,13 +1038,7 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,struct lil_value *r;if (cmd->proc) { -size_t shead = lil->head; - r = cmd->proc (lil, words->c -1 , words->v +1 ); -if (lil->error ==ERROR_FIXHEAD ) { - lil->error =ERROR_DEFAULT ; - lil->err_head = shead; - } }else {lil_push_env (lil); lil->env->func = cmd;
@@ -1102,18 +1092,18 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen, lil->parse_depth++;#ifdef LIL_ENABLE_RECLIMIT if (lil->parse_depth >LIL_ENABLE_RECLIMIT ) { -lil_set_error (lil,"Too many recursive calls" ); +lil_set_error (lil,LIL_ERR_DEPTH ,"Too many recursive calls" ); goto cleanup; }#endif if (lil->parse_depth ==1 ) - lil->error =0 ; + lil->err =LIL_ERR_NONE ;if (funclevel) lil->env->breakrun =0 ; -while (lil->head < lil->clen && !lil->error) { +while (lil->head < lil->clen && !lil->err) {if (words)lil_free_list (words);
@@ -1122,7 +1112,7 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen, val =NULL ; words =substitute (lil); -if (!words || lil->error) +if (!words || lil->err) goto cleanup;if (words->c) {
@@ -1136,7 +1126,7 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,snprintf (msg,sizeof (msg),"unknown function %s" , words->v[0 ]->d); -lil_set_error_at (lil, lil->head, msg); +lil_set_error (lil,LIL_ERR_NOCMD , msg); goto cleanup; } }else {
@@ -1182,38 +1172,25 @@ struct lil_value *lil_parse_value(struct lil *lil, struct lil_value *val,return lil_parse (lil, val->d, val->l, funclevel); } -void lil_set_error (struct lil *lil,const char *msg) +static void lil_set_error (struct lil *lil,enum lil_error err,const char *msg) { -if (lil->error) -return ; - +assert (!lil->err);free (lil->err_msg); - lil->error =ERROR_FIXHEAD ; - lil->err_head =0 ; - lil->err_msg =strdup (msg ? msg :"" ); + lil->err = err; + lil->err_msg =strdup (msg); } -void lil_set_error_at (struct lil *lil,size_t pos,const char *msg) +enum lil_error lil_error (struct lil *lil,const char **msg) { -if (lil->error) -return ; +enum lil_error err = lil->err; -free (lil->err_msg); - lil->error =ERROR_DEFAULT ; - lil->err_head = pos; - lil->err_msg =strdup (msg ? msg :"" ); -} - -int lil_error (struct lil *lil,const char **msg,size_t *pos) -{ -if (!lil->error) -return 0 ; +if (!err) +return LIL_ERR_NONE ; *msg = lil->err_msg; - *pos = lil->err_head; - lil->error =ERROR_NOERROR ; + lil->err =LIL_ERR_NONE ; -return 1 ; +return err; }static void ee_expr (struct expreval *ee);
@@ -1642,7 +1619,7 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)struct expreval ee; code =lil_subst_to_value (lil, code); -if (lil->error) +if (lil->err)return NULL ; ee.code =lil_to_string (code);
@@ -1664,13 +1641,13 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)lil_free_value (code);switch (ee.error) {case EERR_DIVISION_BY_ZERO : -lil_set_error (lil,"division by zero in expression" ); +lil_set_error (lil,LIL_ERR_DOM ,"division by zero" );return NULL ;case EERR_SYNTAX_ERROR : -lil_set_error (lil,"expression syntax error" ); +lil_set_error (lil,LIL_ERR_SYNTAX ,"expression syntax" );return NULL ;case EERR_INVALID_EXPRESSION : -lil_set_error (lil,"invalid expression" ); +lil_set_error (lil,LIL_ERR_SYNTAX ,"invalid expression" );return NULL ;case EERR_NO_ERROR :break ;
@@ -1994,7 +1971,7 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,char *msg =malloc (24 +strlen (oldname));sprintf (msg,"unknown function '%s'" , oldname); -lil_set_error_at (lil, lil->head, msg); +lil_set_error (lil,LIL_ERR_NOCMD , msg);free (msg);return NULL ; }
@@ -2397,7 +2374,7 @@ static struct lil_value *fnc_foreach(struct lil *lil, size_t argc,else lil_free_value (rv); -if (lil->env->breakrun || lil->error) +if (lil->env->breakrun || lil->err)break ; }
@@ -2499,7 +2476,7 @@ static struct lil_value *fnc_if(struct lil *lil, size_t argc,return NULL ; val =lil_eval_expr (lil, argv[base]); -if (!val || lil->error) +if (!val || lil->err)return NULL ; v =lil_to_boolean (val);
@@ -2530,9 +2507,9 @@ static struct lil_value *fnc_while(struct lil *lil, size_t argc,if (argc < (size_t )base +2 )return NULL ; -while (!lil->error && !lil->env->breakrun) { +while (!lil->err && !lil->env->breakrun) { val =lil_eval_expr (lil, argv[base]); -if (!val || lil->error) +if (!val || lil->err)return NULL ; v =lil_to_boolean (val);
@@ -2562,9 +2539,9 @@ static struct lil_value *fnc_for(struct lil *lil, size_t argc,return NULL ;lil_free_value (lil_parse_value (lil, argv[0 ],0 )); -while (!lil->error && !lil->env->breakrun) { +while (!lil->err && !lil->env->breakrun) { val =lil_eval_expr (lil, argv[1 ]); -if (!val || lil->error) +if (!val || lil->err)return NULL ;if (!lil_to_boolean (val)) {
@@ -2889,12 +2866,12 @@ static struct lil_value *fnc_try(struct lil *lil, size_t argc,if (argc <1 )return NULL ; -if (lil->error) +if (lil->err)return NULL ; r =lil_parse_value (lil, argv[0 ],0 ); -if (lil->error) { - lil->error =ERROR_NOERROR ; +if (lil->err) { + lil->err =LIL_ERR_NONE ;lil_free_value (r);if (argc >1 )
@@ -2908,7 +2885,8 @@ static struct lil_value *fnc_try(struct lil *lil, size_t argc,static struct lil_value *fnc_error (struct lil *lil,size_t argc,struct lil_value **argv) { -lil_set_error (lil, argc >0 ?lil_to_string (argv[0 ]) :NULL ); +lil_set_error (lil,LIL_ERR_USER , + argc >0 ?lil_to_string (argv[0 ]) :NULL );return NULL ; }
include/cli_lil.h +33 -3
@@ -39,6 +39,38 @@ struct lil_callbacks { struct lil_value **value); }; +/** +* enum lil_error - Error codes returned by @lil_error +* LIL_ERR_NONE: No error +* LIL_ERR_OOM: Out of memory +* LIL_ERR_CASE: Missing case statement; generally a bug in the program or the +* result of memory corruption. +* LIL_ERR_DEPTH: Too much recursion while parsing or evaluating +* LIL_ERR_SYNTAX: Unrecoverable syntax error +* LIL_ERR_EOF: Unexpected end of input; may be recoverable by prompting the +* user for more input. +* LIL_ERR_INTR: Interrupted by Ctrl-C +* LIL_ERR_NOCMD: No such command +* LIL_ERR_NOVAR: No such variable +* LIL_ERR_ARGC: Incorrect number of arguments +* LIL_ERR_DOM: Numerical argument out of range (typically division by zero) +* LIL_ERR_USER: Error set by user +*/ +enum lil_error { +LIL_ERR_NONE =0 , +LIL_ERR_OOM , +LIL_ERR_CASE , +LIL_ERR_DEPTH , +LIL_ERR_SYNTAX , +LIL_ERR_EOF , +LIL_ERR_INTR , +LIL_ERR_NOCMD , +LIL_ERR_NOVAR , +LIL_ERR_ARGC , +LIL_ERR_DOM , +LIL_ERR_USER , +}; +struct lil *lil_new (const struct lil_callbacks *callbacks);void lil_free (struct lil *lil);
@@ -49,9 +81,7 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,struct lil_value *lil_parse_value (struct lil *lil,struct lil_value *val,int funclevel); -void lil_set_error (struct lil *lil,const char *msg); -void lil_set_error_at (struct lil *lil,size_t pos,const char *msg); -int lil_error (struct lil *lil,const char **msg,size_t *pos); +enum lil_error lil_error (struct lil *lil,const char **msg);const char *lil_to_string (struct lil_value *val);ssize_t lil_to_integer (struct lil_value *val);
Signature · 1 line
2.32.0
RFC PATCH 10/28 cli: lil: Add printf-style format helper for errors
In reply to Sean AndersonThis adds a helper for dynamically-created error messages.
---
common/cli_lil.c | 43 ++++++++++++++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 11 deletions(-)
common/cli_lil.c +32 -11
@@ -113,6 +113,9 @@ struct expreval {static struct lil_value *next_word (struct lil *lil);static void register_stdcmds (struct lil *lil);static void lil_set_error (struct lil *lil,enum lil_error err,const char *msg); +static void lil_set_errorf (struct lil *lil,enum lil_error err, +const char *fmt, ...) + __attribute((format (__printf__,3 ,4 )));#ifdef LIL_ENABLE_POOLS static struct lil_value **pool;
@@ -1121,12 +1124,9 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,if (!cmd) {if (words->v[0 ]->l) { -char msg[64 ]; - -snprintf (msg,sizeof (msg), -"unknown function %s" , - words->v[0 ]->d); -lil_set_error (lil,LIL_ERR_NOCMD , msg); +lil_set_errorf (lil,LIL_ERR_NOCMD , +"unknown function %s" , + words->v[0 ]->d); goto cleanup; } }else {
@@ -1180,6 +1180,30 @@ static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg) lil->err_msg =strdup (msg); } +static void lil_set_errorf (struct lil *lil,enum lil_error err, +const char *fmt, ...) +{ +va_list args, saveargs; +size_t n; + +assert (!lil->err); +free (lil->err_msg); + lil->err = err; + +va_start (args, fmt); +va_copy (saveargs, args); + n =vsnprintf (NULL ,0 , fmt, args) +1 ; +va_end (args); + + lil->err_msg =calloc (1 , n); +if (!lil->err_msg) { + lil->err =LIL_ERR_OOM ; +return ; + } +vsnprintf (lil->err_msg, n, fmt, saveargs); +va_end (saveargs); +} +enum lil_error lil_error (struct lil *lil,const char **msg) {enum lil_error err = lil->err;
@@ -1968,11 +1992,8 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc, newname =lil_to_string (argv[1 ]); func =lil_find_cmd (lil, oldname);if (!func) { -char *msg =malloc (24 +strlen (oldname)); - -sprintf (msg,"unknown function '%s'" , oldname); -lil_set_error (lil,LIL_ERR_NOCMD , msg); -free (msg); +lil_set_errorf (lil,LIL_ERR_NOCMD ,"unknown function %s" , + oldname);return NULL ; }
Signature · 1 line
2.32.0
RFC PATCH 11/28 cli: lil: Add several helper functions for errors
In reply to Sean AndersonThis adds several helper functions for common error types. This helps keep down code duplication, and also ensures that each of these errors uses the same message.
---
common/cli_lil.c | 38 +++++++++++++++++++++++++++++++++----- 1 file changed, 33 insertions(+), 5 deletions(-)
common/cli_lil.c +33 -5
@@ -116,6 +116,9 @@ static void lil_set_error(struct lil *lil, enum lil_error err, const char *msg);static void lil_set_errorf (struct lil *lil,enum lil_error err,const char *fmt, ...) __attribute((format (__printf__,3 ,4 ))); +static void lil_set_error_oom (struct lil *lil); +static void lil_set_error_nocmd (struct lil *lil,const char *cmdname); +static void lil_set_error_intr (struct lil *lil);#ifdef LIL_ENABLE_POOLS static struct lil_value **pool;
@@ -1124,9 +1127,8 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,if (!cmd) {if (words->v[0 ]->l) { -lil_set_errorf (lil,LIL_ERR_NOCMD , -"unknown function %s" , - words->v[0 ]->d); +lil_set_error_nocmd (lil, + words->v[0 ]->d); goto cleanup; } }else {
@@ -1204,6 +1206,33 @@ static void lil_set_errorf(struct lil *lil, enum lil_error err,va_end (saveargs); } +/* +* The next several functions provide helpers for throwing some formulaic +* errors. Their purpose is to ensure that the same wording is used everywhere +* the error is used. This reduces data size. +*/ + +static void lil_set_error_oom (struct lil *lil) +{ +lil_set_error (lil,LIL_ERR_OOM ,NULL ); +} + +static void lil_set_error_argc (struct lil *lil,size_t expected) +{ +lil_set_errorf (lil,LIL_ERR_ARGC ,"%s: expected %zu arguments" , + lil->env->proc ?: lil->env->func->name, expected); +} + +static void lil_set_error_nocmd (struct lil *lil,const char *cmdname) +{ +lil_set_errorf (lil,LIL_ERR_NOCMD ,"no such command %s" , cmdname); +} + +static void lil_set_error_intr (struct lil *lil) +{ +lil_set_error (lil,LIL_ERR_INTR ,"interrupted" ); +} +enum lil_error lil_error (struct lil *lil,const char **msg) {enum lil_error err = lil->err;
@@ -1992,8 +2021,7 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc, newname =lil_to_string (argv[1 ]); func =lil_find_cmd (lil, oldname);if (!func) { -lil_set_errorf (lil,LIL_ERR_NOCMD ,"unknown function %s" , - oldname); +lil_set_error_nocmd (lil, oldname);return NULL ; }
Signature · 1 line
2.32.0
RFC PATCH 12/28 cli: lil: Check for ctrl-c
In reply to Sean AndersonCheck for ctrl-c in lil_parse. This works out to around every time a function or command is called. We also check at the beginning of lil_eval_expr so that constructs like
while {1} {}get interrupted. Since there are no non-trivial commands in that example, lil_parse never gets to its ctrlc check. However, we do need to evaluate the loop expression, so that's a good place to put a check.
---
common/cli_lil.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
common/cli_lil.c +11 -0
@@ -10,6 +10,7 @@#include <common.h> #include <cli_lil.h> +#include <console.h> #include <ctype.h> #include <stdlib.h> #include <stdio.h>
@@ -1117,6 +1118,11 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,lil_free_value (val); val =NULL ; +if (ctrlc ()) { +lil_set_error (lil,LIL_ERR_INTR ,"interrupted" ); + goto cleanup; + } + words =substitute (lil);if (!words || lil->err) goto cleanup;
@@ -1671,6 +1677,11 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code) {struct expreval ee; +if (ctrlc ()) { +lil_set_error (lil,LIL_ERR_INTR ,"interrupted" ); +return NULL ; + } + code =lil_subst_to_value (lil, code);if (lil->err)return NULL ;
Signature · 1 line
2.32.0
Re: [RFC PATCH 12/28] cli: lil: Check for ctrl-c
Quoted 11 lines · Sean Anderson
Check for ctrl-c in
lil_parse. This works out to around every time a function or command is called. We also check at the beginning oflil_eval_exprso that constructs likewhile {1} {}get interrupted. Since there are no non-trivial commands in that example,
lil_parsenever gets to its ctrlc check. However, we do need to evaluate the loop expression, so that's a good place to put a check.
Quoted 5 lines
Signed-off-by Sean Anderson <[email protected]>---
common/cli_lil.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
RFC PATCH 13/28 cli: lil: Wire up LIL to the rest of U-Boot
In reply to Sean AndersonThis sets the shell to LIL when CONFIG_LIL is enabled. Repeated commands are not supporteed. Neither are partial commands a la Hush's secondary prompt. Setting and getting environmental variables is done through callbacks to assist with testing.
---
cmd/Kconfig | 12 +++++-- common/cli.c | 84 +++++++++++++++++++++++++++++++++++++++--------- common/cli_lil.c | 32 ++++++++++++++++++ 3 files changed, 111 insertions(+), 17 deletions(-)
cmd/Kconfig +10 -2
@@ -11,9 +11,14 @@ config CMDLINE Depending on the number of commands enabled, this can add substantially to the size of U-Boot. +if CMDLINE + +choice + prompt "Shell" + default HUSH_PARSER + config HUSH_PARSER bool "Use hush shell" - depends on CMDLINE help This option enables the "hush" shell (from Busybox) as command line interpreter, thus enabling powerful command line syntax like
@@ -25,13 +30,14 @@ config HUSH_PARSER config LIL bool "Use LIL shell" - depends on CMDLINE help This options enables the "Little Interpreted Language" (LIL) shell as command line interpreter, thus enabling powerful command line syntax like `proc name {args} {body}' functions or `echo [some command]` command substitution ("tcl scripts"). +endchoice + if LIL config LIL_FULL
@@ -42,6 +48,8 @@ config LIL_FULL endif +endif + config CMDLINE_EDITING bool "Enable command line editing" depends on CMDLINE
common/cli.c +69 -15
149 lines not on this page · raw message
common/cli_lil.c +32 -0
@@ -11,6 +11,7 @@#include <common.h> #include <cli_lil.h> #include <console.h> +#include <command.h> #include <ctype.h> #include <stdlib.h> #include <stdio.h>
@@ -59,6 +60,7 @@ struct lil_var {struct lil_env {struct lil_env *parent;struct lil_func *func; +const char *proc;struct lil_var **var;size_t vars;struct hashmap varmap;
@@ -1045,7 +1047,9 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,struct lil_value *r;if (cmd->proc) { + lil->env->proc = words->v[0 ]->d; r = cmd->proc (lil, words->c -1 , words->v +1 ); + lil->env->proc =NULL ; }else {lil_push_env (lil); lil->env->func = cmd;
@@ -2967,8 +2971,33 @@ static struct lil_value *fnc_lmap(struct lil *lil, size_t argc,return NULL ; } +static struct lil_value *fnc_builtin (struct lil *lil,size_t argc, +struct lil_value **lil_argv) +{ +int err, repeatable; +size_t i; +/* +* We need space for the function name, and the last argv must be NULL +*/ +char **argv =calloc (sizeof (char *), argc +2 ); + + argv[0 ] = (char *)lil->env->proc; +for (i =0 ; i < argc; i++) + argv[i +1 ] = (char *)lil_to_string (lil_argv[i]); + + err =cmd_process (0 , argc +1 , argv, &repeatable,NULL ); +if (err) +lil_set_errorf (lil,LIL_ERR_USER ,"%s failed" , argv[0 ]); +free (argv); + +return 0 ; +} +static void register_stdcmds (struct lil *lil) { + struct cmd_tbl *cmdtp, *start = ll_entry_start(struct cmd_tbl, cmd); + const int len = ll_entry_count(struct cmd_tbl, cmd); +lil_register (lil,"decr" , fnc_decr);lil_register (lil,"eval" , fnc_eval);lil_register (lil,"expr" , fnc_expr);
@@ -2984,6 +3013,9 @@ static void register_stdcmds(struct lil *lil)lil_register (lil,"try" , fnc_try);lil_register (lil,"while" , fnc_while); +for (cmdtp = start; cmdtp != start + len; cmdtp++) +lil_register (lil, cmdtp->name, fnc_builtin); +if (IS_ENABLED (CONFIG_LIL_FULL )) {lil_register (lil,"append" , fnc_append);lil_register (lil,"char" , fnc_char);
Signature · 1 line
2.32.0
Re: [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot
Quoted 4 lines · Sean Anderson
This sets the shell to LIL when
CONFIG_LILis enabled. Repeated commands are not supporteed. Neither are partial commands a la Hush's secondary prompt. Setting and getting environmental variables is done through callbacks to assist with testing.
This all looks very interesting, thanks for doing this!
Re this patch, is there some way LIL and HUSH could coexist, with HUSH then being the initial shell, and one could then enter a "lil shell" with the command "lil", just as I can start a busybox shell from bash by saying "busybox sh". That could make it a bit easier for playing around with initially.
I have no idea how hard that is to do with Kconfig. Perhaps make LIL selectable by itself, then have a hidden symbol LIL_SHELL which is "default y if LIL && !HUSH_PARSER" or something like that.
Rasmus
Re: [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot
Quoted 14 lines · Rasmus Villemoes
Quoted 4 lines · Sean Anderson
This sets the shell to LIL when
CONFIG_LILis enabled. Repeated commands are not supporteed. Neither are partial commands a la Hush's secondary prompt. Setting and getting environmental variables is done through callbacks to assist with testing.This all looks very interesting, thanks for doing this!
Re this patch, is there some way LIL and HUSH could coexist, with HUSH then being the initial shell, and one could then enter a "lil shell" with the command "lil", just as I can start a busybox shell from bash by saying "busybox sh". That could make it a bit easier for playing around with initially.
Quoted 3 lines
I have no idea how hard that is to do with
Kconfig. Perhaps make LIL selectable by itself, then have a hidden symbolLIL_SHELLwhich is "default y if LIL && !HUSH_PARSER" or something like that.
Yeah, I wanted to do this anyway for unit testing at least.
--Sean
Re: [RFC PATCH 13/28] cli: lil: Wire up LIL to the rest of U-Boot
In reply to Sean AndersonHi Sean,
Quoted 6 lines · Sean Anderson
This sets the shell to LIL when
CONFIG_LILis enabled. Repeated commands are not supporteed. Neither are partial commands a la Hush's secondary prompt. Setting and getting environmental variables is done through callbacks to assist with testing.
Quoted 7 lines
Signed-off-by Sean Anderson <[email protected]>---
cmd/Kconfig | 12 +++++-- common/cli.c | 84 +++++++++++++++++++++++++++++++++++++++--------- common/cli_lil.c | 32 ++++++++++++++++++ 3 files changed, 111 insertions(+), 17 deletions(-)
Can you try using if() instead of #ifdef in here?
Regards,
Simon
RFC PATCH 14/28 cli: lil: Document structures
In reply to Sean AndersonThis documents the major structures of LIL.
---
common/cli_lil.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++ include/cli_lil.h | 57 ++++++++++++++++++++++++++ 2 files changed, 157 insertions(+)
common/cli_lil.c +100 -0
@@ -29,20 +29,44 @@#define HASHMAP_CELLS 256#define HASHMAP_CELLMASK 0xFF +/** +* struct hashentry - An entry in a cell +* @k: The key +* @v: The value +*/ struct hashentry {char *k;void *v; }; +/** +* struct hashcell - A list of entries with the same hash +* @e: An array of entries +* @c: The capacity of this cell +*/ struct hashcell {struct hashentry *e;size_t c; }; +/** +* struct hashmap - An array-based hash map +* @cell: An array of cells +* +* This hash map uses separate chaining with an array for each cell. We cannot +* use the existing hash map functions (hsearch_r et al.) because they assume +* that the value is a string. +*/ struct hashmap {struct hashcell cell[HASHMAP_CELLS ]; }; +/** +* struct lil_value - A string and its length +* @l: The length of the value. +* @c: The capacity of this value (maximum of @l before requiring reallocation). +* @d: The contents of the value, as a nul-terminated string. +*/ struct lil_value {size_t l;#ifdef LIL_ENABLE_POOLS
@@ -51,12 +75,34 @@ struct lil_value {char *d; }; +/** +* struct lil_var - A variable +* @n: The name of this variable +* @env: The environment containing this variable +* @v: The value of this variable +*/ struct lil_var {char *n;struct lil_env *env;struct lil_value *v; }; +/** +* struct lil_env - A function call's execution environment +* @parent: The parent of this environment. This is %NULL for the root +* environment. +* @func: The currently-executing function +* @proc: The name of the currently-executing built-in procedure +* @var: A list of the variables in this environment +* @vars: The number of variables in @var +* @varmap: A hash map mapping variable names to pointers to variables in @var +* @retval: The value set by "return" or "result" +* @retval_set: Whether @retval has been set +* @breakrun: Whether to immediately break out the current run of execution. +* This is set by "return" +* +* Variables are inherited from @parent. +*/ struct lil_env {struct lil_env *parent;struct lil_func *func;
@@ -69,12 +115,28 @@ struct lil_env {int breakrun; }; +/** +* struct list - A list of values +* @v: A list of pointers to &struct lil_value +* @c: The number of values in this list +* @cap: The space allocated for @v +*/ struct lil_list {struct lil_value **v;size_t c;size_t cap; }; +/** +* struct lil_func - A function which may be evaluated with a list of arguments +* @name: The name of the function +* @code: A value containing LIL code to be evaluated +* @argnames: The names of variables to assign to the passed-in arguments +* @proc: A C function to use to evaluate this function +* +* @code and @argnames are only used to evaluate the function if @proc is %NULL. +* If @argnames is empty, then the arguments are assigned to a variable "args". +*/ struct lil_func {char *name;struct lil_value *code;
@@ -82,6 +144,31 @@ struct lil_func {lil_func_proc_t proc; }; +/** +* struct lil - The current state of the interpreter +* @code: The code which is being interpreted +* @rootcode: The top-level code (e.g. the code for the initial call to +* lil_parse()) +* @clen: The length of @code +* @head: The first uninterpreted part of this code, as an index of @code +* @ignoreeol: Whether to treat newlines as whitespace or command terminators +* @cmd: A list of the current commands +* @cmds: The number of commands in @cmd +* @cmdmap: A hash map mapping command names to pointers to commands in @cmd +* @env: The current environment to evaluate commands in +* @rootenv: The top-level "root" environment +* @downenv: The original environment after a call to "topeval" or "upeval" +* @empty: An empty value, allocated once +* @ERROR_NOERROR: There is no error. +* @ERROR_DEFAULT: There was an error. +* @ERROR_FIXHEAD: There was an error, but @err_head needs to be fixed. +* @ERROR_UNBALANCED: An opening quote or bracket lacks a balancing closing +* quote or bracket. +* @error: The current error status +* @err_head: The offset in @code which caused the @error +* @err_msg: An optional string describing the current @error +* @parse_depth: The depth of recursive function calls +*/ struct lil {const char *code;/* need save on parse */ const char *rootcode;
@@ -101,6 +188,19 @@ struct lil {size_t parse_depth; }; +/** +* struct expreval - A (mathematical) expression being evaluated +* @code: The complete code for this expression +* @len: The length of @code +* @head: The first unevaluated part of this expression, as an index of @code +* @ival: The integer value of this expression +* @EERR_NO_ERROR: There is no error +* @EERR_SYNTAX_ERROR: Syntax error. For now this is just mismatched +* parentheses. +* @EERR_DIVISION_BY_ZERO: Attempted division by zero +* @EERR_INVALID_EXPRESSION: A non-number was present +* @error: The error of this expression (if any) +*/ struct expreval {const char *code;size_t len, head;
include/cli_lil.h +57 -0
76 lines not on this page · raw message
Signature · 1 line
2.32.0
RFC PATCH 15/28 cli: lil: Convert LIL_ENABLE_POOLS to Kconfig
In reply to Sean AndersonThis adds a Kconfig option to enable memory pools for some commonly-created and destroyed LIL structures. There is still some significant code duplication, so perhaps these functions can be refactored further to all use common pool functions.
---
cmd/Kconfig | 7 ++ common/cli_lil.c | 275 +++++++++++++++++++++++------------------------ 2 files changed, 143 insertions(+), 139 deletions(-)
cmd/Kconfig +7 -0
@@ -46,6 +46,13 @@ config LIL_FULL This enables all LIL builtin functions, as well as expression support for arithmetic and bitwise operations. +config LIL_POOLS + bool "Use memory pools for LIL structures" + help + Enable pools for reusing values, lists, and environments. This will + use more memory but will cause considerably less memory fragmentation + and improve the script execution performance. + endif endif
common/cli_lil.c +136 -139
400 lines not on this page · raw message
Signature · 1 line
2.32.0
RFC PATCH 16/28 cli: lil: Convert LIL_ENABLE_RECLIMIT to KConfig
In reply to Sean AndersonThis adds a configuration option to set the recursion limit. I've set it to a (conservative) 1000 by default. In addition, there is an option to turn it off for a very minor space savings and performance increase.
---
Do we need this? Perhaps it should default to 0.
cmd/Kconfig | 8 ++++++++ common/cli_lil.c | 10 ++-------- 2 files changed, 10 insertions(+), 8 deletions(-)
cmd/Kconfig +8 -0
@@ -53,6 +53,14 @@ config LIL_POOLS use more memory but will cause considerably less memory fragmentation and improve the script execution performance. +config LIL_RECLIMIT + int "LIL function recursion limit" + default 1000 + help + Enable limiting recursive calls to lil_parse - this can be used to + avoid call stack overflows and is also useful when running through an + automated fuzzer like AFL. Set to 0 to disable the recursion limit. + endif endif
common/cli_lil.c +2 -8
@@ -17,10 +17,6 @@#include <stdio.h> #include <string.h> -/* Enable limiting recursive calls to lil_parse - this can be used to avoid call stack -* overflows and is also useful when running through an automated fuzzer like AFL */ -/*#define LIL_ENABLE_RECLIMIT 10000*/ -#define HASHMAP_CELLS 256#define HASHMAP_CELLMASK 0xFF
@@ -1198,12 +1194,10 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen,lil_skip_spaces (lil); lil->parse_depth++; -#ifdef LIL_ENABLE_RECLIMIT -if (lil->parse_depth >LIL_ENABLE_RECLIMIT ) { -lil_set_error (lil,LIL_ERR_DEPTH ,"Too many recursive calls" ); +if (CONFIG_LIL_RECLIMIT && lil->parse_depth >CONFIG_LIL_RECLIMIT ) { +lil_set_error (lil,LIL_ERR_DEPTH ,"recursion limit reached" ); goto cleanup; } -#endif if (lil->parse_depth ==1 ) lil->err =LIL_ERR_NONE ;
Signature · 1 line
2.32.0
RFC PATCH 17/28 test: Add tests for LIL
In reply to Sean AndersonThis tests several aspects of the parser. These tests are primarily adapted from the *.lil code examples included with upstream LIL. These tests should probably get their own category, especially if I add additional styles of tests.
---
Yes, I know checkpatch complains about the quoted string being split, but that warning is intended for user-visible strings.
MAINTAINERS | 1 + test/cmd/Makefile | 1 + test/cmd/lil.c | 339 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 341 insertions(+) create mode 100644 test/cmd/lil.c
MAINTAINERS +1 -0
@@ -771,6 +771,7 @@ M: Sean Anderson <[email protected]> S: Maintained F: common/cli_lil.c F: include/cli_lil.h +F: test/cmd/lil.c LOGGING M: Simon Glass <[email protected]>
test/cmd/Makefile +1 -0
@@ -5,6 +5,7 @@ ifdef CONFIG_HUSH_PARSER obj-$(CONFIG_CONSOLE_RECORD) += test_echo.o endif +obj-$(CONFIG_LIL_FULL) += lil.o obj-y += mem.o obj-$(CONFIG_CMD_ADDRMAP) += addrmap.o obj-$(CONFIG_CMD_MEM_SEARCH) += mem_search.o
test/cmd/lil.c new file +339 -0
339 lines not on this page · raw message
Signature · 1 line
2.32.0
Re: [RFC PATCH 17/28] test: Add tests for LIL
Quoted 11 lines · Sean Anderson
This tests several aspects of the parser. These tests are primarily adapted from the *.lil code examples included with upstream LIL. These tests should probably get their own category, especially if I add additional styles of tests.
Signed-off-by Sean Anderson <[email protected]>---
Yes, I know checkpatch complains about the quoted string being split, but that warning is intended for user-visible strings.
Quoted 6 lines
MAINTAINERS | 1 + test/cmd/Makefile | 1 + test/cmd/lil.c | 339 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 341 insertions(+) create mode 100644 test/cmd/lil.c
RFC PATCH 18/28 cli: lil: Remove duplicate function bodies
In reply to Sean Andersonlil_append_val is just lil_append_string with the string and length taken from a struct lil_value. Use lil_append_stringh_len to implement both. Do the same for lil_clone_value.
---
common/cli_lil.c | 51 ++---------------------------------------------- 1 file changed, 2 insertions(+), 49 deletions(-)
common/cli_lil.c +2 -49
@@ -369,39 +369,7 @@ static struct lil_value *alloc_value(const char *str)struct lil_value *lil_clone_value (struct lil_value *src) { -struct lil_value *val; - -if (!src) -return NULL ; - -if (IS_ENABLED (CONFIG_LIL_POOLS )) - val =alloc_from_pool (); -else - val =calloc (1 ,sizeof (struct lil_value )); -if (!val) -return NULL ; - - val->l = src->l; -if (src->l) { -if (IS_ENABLED (CONFIG_LIL_POOLS )) { -ensure_capacity (val, val->l +1 ); - }else { - val->d =malloc (val->l +1 ); -if (!val->d) { -free (val); -return NULL ; - } - } -memcpy (val->d, src->d, val->l +1 ); - }else { -if (IS_ENABLED (CONFIG_LIL_POOLS )) { -ensure_capacity (val,1 ); - val->d[0 ] ='\0' ; - }else { - val->d =NULL ; - } - } -return val; +return alloc_value_len (src->d, src->l); }int lil_append_char (struct lil_value *val,char ch)
@@ -450,22 +418,7 @@ int lil_append_string(struct lil_value *val, const char *s)int lil_append_val (struct lil_value *val,struct lil_value *v) { -if (!v || !v->l) -return 1 ; - -if (IS_ENABLED (CONFIG_LIL_POOLS )) { -ensure_capacity (val, val->l + v->l +1 ); -memcpy (val->d + val->l, v->d, v->l +1 ); - }else { -char *new =realloc (val->d, val->l + v->l +1 ); - -if (!new) -return 0 ; -memcpy (new + val->l, v->d, v->l +1 ); - val->d = new; - } - val->l += v->l; -return 1 ; +return lil_append_string_len (val, v->d, v->l); }void lil_free_value (struct lil_value *val)
Signature · 1 line
2.32.0
RFC PATCH 19/28 cli: lil: Add "symbol" structure
In reply to Sean AndersonWe need a generic structure to hold symbols parsed by the parser. We would also like to re-use existing code as much as possible without rewriting everything. To do this, we hijack the allocators for lil_list and lil_value and have them allocate enough space for a lil_symbol.
While we're at it, we can make lil_list hold lil_symbols instead of lil_values. To keep all the old users sane, we just cast back to lil_value before retrieving the value (with an assert to make sure we're not sending back something else). Unfortunately, many functions were accessing the list vector directly, so convert them.
This commit also fixes pools not behaving properly when running out of memory. This should likely be refactored in the future so everything uses one set of allocator/free routines to avoid code duplication.
---
common/cli_lil.c | 317 +++++++++++++++++++++++++++++----------------- include/cli_lil.h | 8 +- 2 files changed, 206 insertions(+), 119 deletions(-)
common/cli_lil.c +202 -115
@@ -106,18 +106,45 @@ struct lil_env {int breakrun; }; +struct lil_symbol ; +/** -* struct list - A list of values -* @v: A list of pointers to &struct lil_value -* @c: The number of values in this list +* struct list - A list of symbols +* @v: A list of pointers to symbols +* @c: The number of symbols in this list * @cap: The space allocated for @v */ struct lil_list { -struct lil_value **v; +struct lil_symbol **v;size_t c;size_t cap; }; +/** +* struct lil_symbol - A symbol parsed by the parser +* @LIL_SYMBOL_VALUE: A plain old string and length +* @LIL_SYMBOL_VARIABLE: A name of a variable to be substituted +* @LIL_SYMBOL_LIST: A list of symbols +* @value: A literal value or name of variable +* @list: A list of commands in the script +* @word: Another word to be evaluated +* @type: The type of word +*/ +struct lil_symbol { +union { +struct lil_value value; +struct lil_symbol *symbol; +struct lil_list list; + }; +enum { +LIL_SYMBOL_VALUE =0 , +LIL_SYMBOL_LIST , +LIL_SYMBOL_VARIABLE , +LIL_SYMBOL_COMMAND , +LIL_SYMBOL_SCRIPT , + } type; +}; +/** * struct lil_func - A function which may be evaluated with a list of arguments * @name: The name of the function
@@ -287,24 +314,50 @@ static int hm_has(struct hashmap *hm, const char *key)return 0 ; } -#ifdef CONFIG_LIL_POOLS +static void lil_free_symbol (struct lil_symbol *sym) +{ +switch (sym->type) { +case LIL_SYMBOL_VALUE : +lil_free_value (&sym->value); +return ; +case LIL_SYMBOL_VARIABLE : +lil_free_symbol (sym->symbol); +free (sym); +return ; +case LIL_SYMBOL_LIST : +case LIL_SYMBOL_COMMAND : +case LIL_SYMBOL_SCRIPT : +lil_free_list (&sym->list); +return ; + } +log_debug ("unknown type %d\n" , sym->type); +assert (0 ); +} + +#if IS_ENABLED (CONFIG_LIL_POOLS )static struct lil_value *alloc_from_pool (void ) { -if (poolsize >0 ) { - poolsize--; -return pool[poolsize]; - }else { -struct lil_value *val =calloc (1 ,sizeof (struct lil_value )); - -return val; - } +if (poolsize >0 ) +return pool[--poolsize]; +else +return calloc (1 ,sizeof (struct lil_symbol )); }static void release_to_pool (struct lil_value *val) {if (poolsize == poolcap) { - poolcap = poolcap ? (poolcap + poolcap /2 ) :64 ; - pool =realloc (pool,sizeof (struct lil_value *) * poolcap); +size_t npoolcap = poolcap ? (poolcap + poolcap /2 ) :64 ; +struct lil_value **npool = +realloc (pool,sizeof (struct lil_symbol *) * npoolcap); + +if (!npool) { +free (val->d); +free (val); +return ; + } + + poolcap = npoolcap; + pool = npool; } pool[poolsize++] = val; }
@@ -319,45 +372,48 @@ static void ensure_capacity(struct lil_value *val, size_t cap)#else static struct lil_value *alloc_from_pool (void ) { -return NULL ; +return calloc (1 ,sizeof (struct lil_symbol )); } -static void release_to_pool (struct lil_value *val) { } -static void ensure_capacity (struct lil_value *val,size_t cap) { } +static void release_to_pool (struct lil_value *val) +{ +free (val->d); +free (val); +} +static void ensure_capacity (struct lil_value *val,size_t cap) +{ + val->d =realloc (val->d, cap); +}#endif +static struct lil_symbol *value_to_symbol (struct lil_value *val) +{ + return container_of(val, struct lil_symbol, value); +} +static struct lil_value *alloc_value_len (const char *str,size_t len) {struct lil_value *val; -if (IS_ENABLED (CONFIG_LIL_POOLS )) - val =alloc_from_pool (); -else - val =calloc (1 ,sizeof (struct lil_value )); + val =alloc_from_pool ();if (!val)return NULL ; +value_to_symbol (val)->type =LIL_SYMBOL_VALUE ;if (str) { val->l = len; -if (IS_ENABLED (CONFIG_LIL_POOLS )) { -ensure_capacity (val, len +1 ); - }else { - val->d =malloc (len +1 ); -if (!val->d) { -free (val); -return NULL ; - } +ensure_capacity (val, len +1 ); +if (!val->d) { +release_to_pool (val); +return NULL ; }memcpy (val->d, str, len); val->d[len] =0 ; }else { val->l =0 ; -if (IS_ENABLED (CONFIG_LIL_POOLS )) { -ensure_capacity (val,1 ); +ensure_capacity (val,1 ); +if (val->d) val->d[0 ] ='\0' ; - }else { - val->d =NULL ; - } }return val; }
@@ -372,53 +428,41 @@ struct lil_value *lil_clone_value(struct lil_value *src)return alloc_value_len (src->d, src->l); } -int lil_append_char (struct lil_value *val,char ch) +enum lil_error lil_append_char (struct lil_value *val,char ch) { -if (IS_ENABLED (CONFIG_LIL_POOLS )) { -ensure_capacity (val, val->l +2 ); - val->d[val->l++] = ch; - val->d[val->l] ='\0' ; - }else { -char *new =realloc (val->d, val->l +2 ); +ensure_capacity (val, val->l +2 ); +if (!val->d) +return LIL_ERR_OOM ; -if (!new) -return 0 ; - - new[val->l++] = ch; - new[val->l] =0 ; - val->d = new; - } -return 1 ; + val->d[val->l++] = ch; + val->d[val->l] ='\0' ; +return LIL_ERR_NONE ; } -int lil_append_string_len (struct lil_value *val,const char *s,size_t len) +static enum lil_error lil_append_string_len (struct lil_value *val, +const char *s,size_t len) {if (!s || !s[0 ]) -return 1 ; +return LIL_ERR_NONE ; -if (IS_ENABLED (CONFIG_LIL_POOLS )) { -ensure_capacity (val, val->l + len +1 ); -memcpy (val->d + val->l, s, len +1 ); - }else { -char *new =realloc (val->d, val->l + len +1 ); - -if (!new) -return 0 ; -memcpy (new + val->l, s, len +1 ); - val->d = new; - } +ensure_capacity (val, val->l + len +1 ); +if (!val->d) +return LIL_ERR_OOM ; +memcpy (val->d + val->l, s, len +1 ); val->l += len; -return 1 ; +return LIL_ERR_NONE ; } -int lil_append_string (struct lil_value *val,const char *s) +enum lil_error lil_append_string (struct lil_value *val,const char *s) {return lil_append_string_len (val, s,strlen (s)); } -int lil_append_val (struct lil_value *val,struct lil_value *v) +enum lil_error lil_append_val (struct lil_value *val,struct lil_value *v) { -return lil_append_string_len (val, v->d, v->l); +if (v) +return lil_append_string_len (val, v->d, v->l); +return LIL_ERR_NONE ; }void lil_free_value (struct lil_value *val)
@@ -426,23 +470,28 @@ void lil_free_value(struct lil_value *val)if (!val)return ; -if (IS_ENABLED (CONFIG_LIL_POOLS )) { -release_to_pool (val); - }else { -free (val->d); -free (val); - } +release_to_pool (val); +} + +static struct lil_symbol *list_to_symbol (struct lil_list *list) +{ + return container_of(list, struct lil_symbol, list); }struct lil_list *lil_alloc_list (void ) {struct lil_list *list; -if (IS_ENABLED (CONFIG_LIL_POOLS ) && listpoolsize >0 ) -return listpool[--listpoolsize]; +if (IS_ENABLED (CONFIG_LIL_POOLS ) && listpoolsize >0 ) { + list = listpool[--listpoolsize]; + }else { + list =calloc (1 ,sizeof (struct lil_symbol )); +if (!list) +return list; + list->v =NULL ; + } - list =calloc (1 ,sizeof (struct lil_list )); - list->v =NULL ; +list_to_symbol (list)->type =LIL_SYMBOL_LIST ;return list; }
@@ -454,49 +503,61 @@ void lil_free_list(struct lil_list *list)return ;for (i =0 ; i < list->c; i++) -lil_free_value (list->v[i]); +lil_free_symbol (list->v[i]);if (IS_ENABLED (CONFIG_LIL_POOLS )) { list->c =0 ;if (listpoolsize == listpoolcap) { +int ncap; +struct lil_list **npool; +if (listpoolcap) - listpoolcap += listpoolcap /2 ; + ncap += listpoolcap /2 ;else - listpoolcap =32 ; - listpool =realloc (listpool, -sizeof (*listpool) * listpoolcap); + ncap =32 ; + npool =realloc (listpool,sizeof (*npool) * ncap); +if (!npool) + goto free; + + listpoolcap = ncap; + listpool = npool; } listpool[listpoolsize++] = list; }else { +free:free (list->v);free (list); } } -void lil_list_append (struct lil_list *list,struct lil_value *val) +int lil_list_append (struct lil_list *list,void *item) {if (list->c == list->cap) {size_t cap = list->cap ? (list->cap + list->cap /2 ) :32 ; -struct lil_value **nv = -realloc (list->v,sizeof (struct lil_value *) * cap); +struct lil_symbol **nv =realloc (list->v,sizeof (void *) * cap);if (!nv) -return ; +return -ENOMEM ; list->cap = cap; list->v = nv; } - list->v[list->c++] = val; + list->v[list->c++] = item; +return 0 ; } -size_t lil_list_size (struct lil_list *list) +static struct lil_symbol *lil_list_gets (struct lil_list *list,size_t index) { -return list->c; +assert (index < list->c); +return list->v[index]; }struct lil_value *lil_list_get (struct lil_list *list,size_t index) { -return index >= list->c ?NULL : list->v[index]; +struct lil_symbol *sym =lil_list_gets (list, index); + +assert (sym->type ==LIL_SYMBOL_VALUE ); +return &sym->value; }static int needs_escape (const char *str)
@@ -519,25 +580,26 @@ struct lil_value *lil_list_to_value(struct lil_list *list, int do_escape)size_t i, j;for (i =0 ; i < list->c; i++) { +struct lil_value *item =lil_list_get (list, i);int escape = - do_escape ?needs_escape (lil_to_string (list->v[i])) :0 ; + do_escape ?needs_escape (lil_to_string (item)) :0 ;if (i)lil_append_char (val,' ' );if (escape) {lil_append_char (val,'{' ); -for (j =0 ; j < list->v[i]->l; j++) { -if (list->v[i]->d[j] =='{' ) +for (j =0 ; j < item->l; j++) { +if (item->d[j] =='{' )lil_append_string (val,"}\"\\o\"{" ); -else if (list->v[i]->d[j] =='}' ) +else if (item->d[j] =='}' )lil_append_string (val,"}\"\\c\"{" );else -lil_append_char (val, list->v[i]->d[j]); +lil_append_char (val, item->d[j]); }lil_append_char (val,'}' ); }else { -lil_append_val (val, list->v[i]); +lil_append_val (val, item); } }return val;
@@ -588,15 +650,23 @@ void lil_free_env(struct lil_env *env)free (env->var);if (envpoolsize == envpoolcap) { +int ncap; +struct lil_env **npool; +if (envpoolcap) envpoolcap += envpoolcap /2 ;else envpoolcap =64 ; - envpool =realloc (envpool, -sizeof (*envpool) * envpoolcap); + npool =realloc (envpool,sizeof (*npool) * ncap); +if (!npool) + goto free; + + envpoolcap = ncap; + envpool = npool; } envpool[envpoolsize++] = env; }else { +free:hm_destroy (&env->varmap);for (i =0 ; i < env->vars; i++) {free (env->var[i]->n);
@@ -660,6 +730,8 @@ static struct lil_func *add_func(struct lil *lil, const char *name) } cmd =calloc (1 ,sizeof (struct lil_func )); +if (!cmd) +return NULL ; cmd->name =strdup (name); ncmd =realloc (lil->cmd,sizeof (struct lil_func *) * (lil->cmds +1 ));
@@ -1093,15 +1165,17 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,struct lil_value *r;if (cmd->proc) { - lil->env->proc = words->v[0 ]->d; - r = cmd->proc (lil, words->c -1 , words->v +1 ); + lil->env->proc =lil_to_string (lil_list_get (words,0 )); + r = cmd->proc (lil, words->c -1 , + (struct lil_value **)words->v +1 ); lil->env->proc =NULL ; }else {lil_push_env (lil); lil->env->func = cmd;if (cmd->argnames->c ==1 && - !strcmp (lil_to_string (cmd->argnames->v[0 ]),"args" )) { + !strcmp (lil_to_string (lil_list_get (cmd->argnames,0 )), +"args" )) {struct lil_value *args =lil_list_to_value (words,1 );lil_set_var (lil,"args" , args,LIL_SETVAR_LOCAL_NEW );
@@ -1111,14 +1185,15 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,for (i =0 ; i < cmd->argnames->c; i++) {struct lil_value *val; +struct lil_value *name = +lil_list_get (cmd->argnames, i);if (i < words->c -1 ) - val = words->v[i +1 ]; + val =lil_list_get (words, i +1 );else val = lil->empty; -lil_set_var (lil, -lil_to_string (cmd->argnames->v[i]), +lil_set_var (lil,lil_to_string (name), val,LIL_SETVAR_LOCAL_NEW ); } }
@@ -1176,13 +1251,13 @@ struct lil_value *lil_parse(struct lil *lil, const char *code, size_t codelen, goto cleanup;if (words->c) { -struct lil_func *cmd = -lil_find_cmd (lil,lil_to_string (words->v[0 ])); +const char *cmdname = +lil_to_string (lil_list_get (words,0 )); +struct lil_func *cmd =lil_find_cmd (lil, cmdname);if (!cmd) { -if (words->v[0 ]->l) { -lil_set_error_nocmd (lil, - words->v[0 ]->d); +if (cmdname[0 ]) { +lil_set_error_nocmd (lil, cmdname); goto cleanup; } }else {
@@ -2042,6 +2117,9 @@ static struct lil_value *fnc_proc(struct lil *lil, size_t argc, name =lil_clone_value (argv[0 ]); fargs =lil_subst_to_list (lil, argv[1 ]); cmd =add_func (lil,lil_to_string (argv[0 ])); +if (!cmd) +return NULL ; + cmd->argnames = fargs; cmd->code =lil_clone_value (argv[2 ]); }else {
@@ -2052,11 +2130,17 @@ static struct lil_value *fnc_proc(struct lil *lil, size_t argc, fargs =lil_subst_to_list (lil, tmp);lil_free_value (tmp); cmd =add_func (lil,lil_to_string (name)); +if (!cmd) +return NULL ; + cmd->argnames = fargs; cmd->code =lil_clone_value (argv[0 ]); }else { fargs =lil_subst_to_list (lil, argv[0 ]); cmd =add_func (lil,lil_to_string (name)); +if (!cmd) +return NULL ; + cmd->argnames = fargs; cmd->code =lil_clone_value (argv[1 ]); }
@@ -2278,7 +2362,7 @@ static struct lil_value *fnc_index(struct lil *lil, size_t argc,if (index >= list->c) r =NULL ;else - r =lil_clone_value (list->v[index]); + r =lil_clone_value (lil_list_get (list, index));lil_free_list (list);return r; }
@@ -2295,7 +2379,7 @@ static struct lil_value *fnc_indexof(struct lil *lil, size_t argc, list =lil_subst_to_list (lil, argv[0 ]);for (index =0 ; index < list->c; index++) { -if (!strcmp (lil_to_string (list->v[index]), +if (!strcmp (lil_to_string (lil_list_get (list, index)),lil_to_string (argv[1 ]))) { r =lil_alloc_integer (index);break ;
@@ -2364,7 +2448,7 @@ static struct lil_value *fnc_slice(struct lil *lil, size_t argc, slice =lil_alloc_list ();for (i = (size_t )from; i < (size_t )to; i++) -lil_list_append (slice,lil_clone_value (list->v[i])); +lil_list_append (slice,lil_clone_value (lil_list_get (list, i)));lil_free_list (list); r =lil_list_to_value (slice,1 );
@@ -2395,10 +2479,12 @@ static struct lil_value *fnc_filter(struct lil *lil, size_t argc, list =lil_subst_to_list (lil, argv[base]); filtered =lil_alloc_list ();for (i =0 ; i < list->c && !lil->env->breakrun; i++) { -lil_set_var (lil, varname, list->v[i],LIL_SETVAR_LOCAL_ONLY ); +lil_set_var (lil, varname,lil_list_get (list, i), +LIL_SETVAR_LOCAL_ONLY ); r =lil_eval_expr (lil, argv[base +1 ]);if (lil_to_boolean (r)) -lil_list_append (filtered,lil_clone_value (list->v[i])); +lil_list_append (filtered, +lil_clone_value (lil_list_get (list, i)));lil_free_value (r); }lil_free_list (list);
@@ -2475,7 +2561,8 @@ static struct lil_value *fnc_foreach(struct lil *lil, size_t argc,for (i =0 ; i < list->c; i++) {struct lil_value *rv; -lil_set_var (lil, varname, list->v[i],LIL_SETVAR_LOCAL_ONLY ); +lil_set_var (lil, varname,lil_list_get (list, i), +LIL_SETVAR_LOCAL_ONLY ); rv =lil_parse_value (lil, argv[codeidx],0 );if (rv->l)lil_list_append (rlist, rv);
include/cli_lil.h +4 -4
@@ -149,13 +149,13 @@ struct lil_value *lil_alloc_integer(ssize_t num);void lil_free_value (struct lil_value *val);struct lil_value *lil_clone_value (struct lil_value *src); -int lil_append_char (struct lil_value *val,char ch); -int lil_append_string (struct lil_value *val,const char *s); -int lil_append_val (struct lil_value *val,struct lil_value *v); +enum lil_error lil_append_char (struct lil_value *val,char ch); +enum lil_error lil_append_string (struct lil_value *val,const char *s); +enum lil_error lil_append_val (struct lil_value *val,struct lil_value *v);struct lil_list *lil_alloc_list (void );void lil_free_list (struct lil_list *list); -void lil_list_append (struct lil_list *list,struct lil_value *val); +int lil_list_append (struct lil_list *list,void *item);size_t lil_list_size (struct lil_list *list);struct lil_value *lil_list_get (struct lil_list *list,size_t index);struct lil_value *lil_list_to_value (struct lil_list *list,int do_escape);
Signature · 1 line
2.32.0
RFC PATCH 20/28 cli: lil: Add config to enable debug output
In reply to Sean AndersonThis provides an easy way to enable assertions and debug messages. It will also be used to enable tracing features in future patches.
---
cmd/Kconfig | 6 ++++++ common/Makefile | 3 +++ 2 files changed, 9 insertions(+)
cmd/Kconfig +6 -0
@@ -61,6 +61,12 @@ config LIL_RECLIMIT avoid call stack overflows and is also useful when running through an automated fuzzer like AFL. Set to 0 to disable the recursion limit. +config LIL_DEBUG + bool "Enable LIL debugging" + help + This enables debug prints, assertions, and other tracing features in + LIL. If you are not working on LIL, say 'n' here. + endif endif
common/Makefile +3 -0
@@ -11,6 +11,9 @@ obj-y += exports.o obj-$(CONFIG_HASH) += hash.o obj-$(CONFIG_HUSH_PARSER) += cli_hush.o obj-$(CONFIG_LIL) += cli_lil.o +ifneq ($(CONFIG_LIL_DEBUG),) +CFLAGS_cli_lil.o += -DDEBUG +endif obj-$(CONFIG_AUTOBOOT) += autoboot.o # This option is not just y/n - it can have a numeric value
Signature · 1 line
2.32.0
RFC PATCH 21/28 cli: lil: Add a distinct parsing step
In reply to Sean AndersonThis adds a parser to LIL (as separate from the interpreter). This is necessary to detect syntax errors before evaluating anything. Before this, running a script like
echo some message; echo syntax error}
would result in "some message" being printed before the error was discovered. This is not only rather surprising, but also makes things like Hush's secondary prompt impossible to implement. In addition, the original parser would accept almost any input, and silently return NULL if it encountered problems. This made it difficult to determine if a command had been mis-parsed, since an empty command would just evaluate to "".
The grammar is not the same as LIL originally. Several ideas have been taken from TCL proper as well. In order to simplify the parser, it has been rewritten to be LL(1), except for line continuations which are LL(2). In particular, multi-line comments and command/variable subtitutions partially through unquoted words (e.g. a$b) have been removed. Some other characters such as unescaped, unmatched }s are now syntax errors. On the other hand, some things such as escaped characters in unquoted words have been added back (as seen in TCL). Unlike TCL, comments may be placed almost anywhere. The exact grammar is subject to change, but I have tried to make it as sane as I can get it.
The grammar has been documented in (extended) EBNF. The names of the nonterminals are the same as are used in the dodekalogue [1]. Each nonterminal foo has a function parse_foo() which recognizes it.
[1] https://www.tcl.tk/man/tcl8.6/TclCmd/Tcl.htm
---
cmd/Kconfig | 4 +- common/cli.c | 2 +- common/cli_lil.c | 1880 ++++++++++++++++++++++++++++++++++++--------- include/cli_lil.h | 11 +- test/cmd/lil.c | 73 +- 5 files changed, 1527 insertions(+), 443 deletions(-)
cmd/Kconfig +2 -2
@@ -43,8 +43,8 @@ if LIL config LIL_FULL bool "Enable all LIL features" help - This enables all LIL builtin functions, as well as expression support - for arithmetic and bitwise operations. + This enables all LIL builtin functions, expression support for + arithmetic and bitwise operations, and expanded error messages. config LIL_POOLS bool "Use memory pools for LIL structures"
common/cli.c +1 -1
@@ -49,7 +49,7 @@ static const struct lil_callbacks env_callbacks = {static int lil_run (const char *cmd) {int err; -struct lil_value *result =lil_parse (lil, cmd,0 ,0 ); +struct lil_value *result =lil_parse_eval (lil, cmd,0 , true);const char *err_msg, *strres =lil_to_string (result);/* The result may be very big, so use puts */
common/cli_lil.c +1498 -382
2,043 lines not on this page · raw message
include/cli_lil.h +7 -4
@@ -133,10 +133,13 @@ void lil_free(struct lil *lil);int lil_register (struct lil *lil,const char *name,lil_func_proc_t proc); -struct lil_value *lil_parse (struct lil *lil,const char *code,size_t codelen, -int funclevel); +struct lil_list *lil_parse (struct lil *lil,const char *code,size_t codelen); +struct lil_value *lil_eval (struct lil *lil,struct lil_list *script, +bool new_frame); +struct lil_value *lil_parse_eval (struct lil *lil,const char *code, +size_t codelen,bool new_frame);struct lil_value *lil_parse_value (struct lil *lil,struct lil_value *val, -int funclevel); +bool new_frame);enum lil_error lil_error (struct lil *lil,const char **msg);
@@ -158,7 +161,7 @@ void lil_free_list(struct lil_list *list);int lil_list_append (struct lil_list *list,void *item);size_t lil_list_size (struct lil_list *list);struct lil_value *lil_list_get (struct lil_list *list,size_t index); -struct lil_value *lil_list_to_value (struct lil_list *list,int do_escape); +struct lil_value *lil_list_to_value (struct lil_list *list,bool do_escape);struct lil_list *lil_subst_to_list (struct lil *lil,struct lil_value *code);struct lil_value *lil_subst_to_value (struct lil *lil,struct lil_value *code);
test/cmd/lil.c +19 -54
143 lines not on this page · raw message
Signature · 1 line
2.32.0
RFC PATCH 22/28 env: Add a priv pointer to hwalk_r
In reply to Sean AndersonThis allows callers of hwalk_r to pass data to their callback. This mirrors e.g. env_attr_walk.
---
cmd/nvedit.c | 8 ++++---- env/callback.c | 4 ++-- env/flags.c | 4 ++-- include/search.h | 2 +- lib/hashtable.c | 5 +++-- 5 files changed, 12 insertions(+), 11 deletions(-)
cmd/nvedit.c +4 -4
@@ -481,7 +481,7 @@ static int print_static_binding(const char *var_name, const char *callback_name,return 0 ; } -static int print_active_callback (struct env_entry *entry) +static int print_active_callback (struct env_entry *entry,void *priv) {struct env_clbk_tbl *clbkp;int i;
@@ -544,7 +544,7 @@ int do_env_callback(struct cmd_tbl *cmdtp, int flag, int argc,puts ("Active callback bindings:\n" );printf ("\t%-20s %-20s\n" ,"Variable Name" ,"Callback Name" );printf ("\t%-20s %-20s\n" ,"-------------" ,"-------------" ); -hwalk_r (&env_htab, print_active_callback); +hwalk_r (&env_htab, print_active_callback,NULL ); return 0; }#endif
@@ -563,7 +563,7 @@ static int print_static_flags(const char *var_name, const char *flags,return 0 ; } -static int print_active_flags (struct env_entry *entry) +static int print_active_flags (struct env_entry *entry,void *priv) {enum env_flags_vartype type;enum env_flags_varaccess access;
@@ -617,7 +617,7 @@ int do_env_flags(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) "Variable Access");printf ("\t%-20s %-20s %-20s\n" ,"-------------" ,"-------------" ,"---------------" ); -hwalk_r (&env_htab, print_active_flags); +hwalk_r (&env_htab, print_active_flags,NULL ); return 0; }#endif
env/callback.c +2 -2
@@ -83,7 +83,7 @@ void env_callback_init(struct env_entry *var_entry)* Called on each existing env var prior to the blanket update since removing * a callback association should remove its callback. */ -static int clear_callback (struct env_entry *entry) +static int clear_callback (struct env_entry *entry,void *priv) { entry->callback =NULL ;
@@ -127,7 +127,7 @@ static int on_callbacks(const char *name, const char *value, enum env_op op, int flags) {/* remove all callbacks */ -hwalk_r (&env_htab, clear_callback); +hwalk_r (&env_htab, clear_callback,NULL );/* configure any static callback bindings */ env_attr_walk (ENV_CALLBACK_LIST_STATIC , set_callback,NULL );
env/flags.c +2 -2
@@ -468,7 +468,7 @@ void env_flags_init(struct env_entry *var_entry)* Called on each existing env var prior to the blanket update since removing * a flag in the flag list should remove its flags. */ -static int clear_flags (struct env_entry *entry) +static int clear_flags (struct env_entry *entry,void *priv) { entry->flags =0 ;
@@ -503,7 +503,7 @@ static int on_flags(const char *name, const char *value, enum env_op op, int flags) {/* remove all flags */ -hwalk_r (&env_htab, clear_flags); +hwalk_r (&env_htab, clear_flags,NULL );/* configure any static flags */ env_attr_walk (ENV_FLAGS_LIST_STATIC , set_flags,NULL );
include/search.h +1 -1
@@ -105,7 +105,7 @@ int himport_r(struct hsearch_data *htab, const char *env, size_t size,/* Walk the whole table calling the callback on each element */ int hwalk_r (struct hsearch_data *htab, -int (*callback)(struct env_entry *entry)); +int (*callback)(struct env_entry *entry,void *priv),void *priv);/* Flags for himport_r(), hexport_r(), hdelete_r(), and hsearch_r() */ #define H_NOCLEAR (1 << 0)/* do not clear hash table before importing */
lib/hashtable.c +3 -2
@@ -998,14 +998,15 @@ end:* Walk all of the entries in the hash, calling the callback for each one. * this allows some generic operation to be performed on each element. */ -int hwalk_r (struct hsearch_data *htab,int (*callback)(struct env_entry *entry)) +int hwalk_r (struct hsearch_data *htab, +int (*callback)(struct env_entry *entry,void *priv),void *priv) {int i;int retval;for (i =1 ; i <= htab->size; ++i) {if (htab->table[i].used >0 ) { - retval =callback (&htab->table[i].entry); + retval =callback (&htab->table[i].entry, priv);if (retval)return retval; }
Signature · 1 line
2.32.0
Re: [RFC PATCH 22/28] env: Add a priv pointer to hwalk_r
Quoted 4 lines · Sean Anderson
This allows callers of
hwalk_rto pass data to their callback. This mirrors e.g.env_attr_walk.Signed-off-by Sean Anderson <[email protected]>
Signature · 1 line
Tom
RFC PATCH 23/28 cli: lil: Handle OOM for hm_put
In reply to Sean Andersonhm_put allocates memory, and this can fail. Instead of failing silently, return an error code. This also fixes up callers to handle this error.
---
common/cli_lil.c | 47 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 32 insertions(+), 15 deletions(-)
common/cli_lil.c +32 -15
@@ -326,22 +326,30 @@ static void hm_destroy(struct hashmap *hm) } } -static void hm_put (struct hashmap *hm,const char *key,void *value) +static enum lil_error hm_put (struct hashmap *hm,const char *key,void *value) {struct hashcell *cell = hm->cell + (hm_hash (key) &HASHMAP_CELLMASK ); +struct hashentry *newe;size_t i;for (i =0 ; i < cell->c; i++) {if (!strcmp (key, cell->e[i].k)) { cell->e[i].v = value; -return ; +return LIL_ERR_NONE ; } } - cell->e =realloc (cell->e,sizeof (struct hashentry ) * (cell->c +1 )); - cell->e[cell->c].k =strdup (key); - cell->e[cell->c].v = value; + newe =realloc (cell->e,sizeof (struct hashentry ) * (cell->c +1 )); +if (!newe) +return LIL_ERR_OOM ; + cell->e = newe; + + newe[cell->c].k =strdup (key); +if (!newe[cell->c].k) +return LIL_ERR_OOM ; + newe[cell->c].v = value; cell->c++; +return LIL_ERR_NONE ; }static void *hm_get (struct hashmap *hm,const char *key)
@@ -738,19 +746,24 @@ static struct lil_func *add_func(struct lil *lil, const char *name) cmd =calloc (1 ,sizeof (struct lil_func ));if (!cmd) -return NULL ; + goto oom; cmd->name =strdup (name); ncmd =realloc (lil->cmd,sizeof (struct lil_func *) * (lil->cmds +1 )); -if (!ncmd) { -free (cmd); -return NULL ; - } - +if (!ncmd) + goto oom; lil->cmd = ncmd; + ncmd[lil->cmds++] = cmd; -hm_put (&lil->cmdmap, name, cmd); +if (hm_put (&lil->cmdmap, name, cmd)) + goto oom; +return cmd; + +oom: +free (cmd); +lil_set_error_oom (lil); +return NULL ; }static void del_func (struct lil *lil,struct lil_func *cmd)
@@ -766,7 +779,11 @@ static void del_func(struct lil *lil, struct lil_func *cmd)if (index == lil->cmds)return ; -hm_put (&lil->cmdmap, cmd->name,0 ); +/* +* The only way this fails is if we don't find the command; this +* means our caller wants to delete a command which doesn't exist +*/ +assert (!hm_put (&lil->cmdmap, cmd->name,NULL ));if (cmd->argnames)lil_free_list (cmd->argnames);
@@ -783,9 +800,9 @@ int lil_register(struct lil *lil, const char *name, lil_func_proc_t proc)struct lil_func *cmd =add_func (lil, name);if (!cmd) -return 0 ; +return LIL_ERR_OOM ; cmd->proc = proc; -return 1 ; +return LIL_ERR_NONE ; } struct lil_var *lil_set_var(struct lil *lil, const char *name,
Signature · 1 line
2.32.0
RFC PATCH 24/28 cli: lil: Make proc always take 3 arguments
In reply to Sean AndersonThis rewrites proc to always take 3 arguments. It also adds proper error handling. TCL does not allow for anonymous functions to be created with proc. Allowing for a variable number of arguments makes the code much more complex when adding error handling.
Since fnc_proc was the last user of lil_unused_name (other than fnc_unusedname), remove it.
---
common/cli_lil.c | 103 ++++++++++++++++------------------------------- test/cmd/lil.c | 6 ++- 2 files changed, 38 insertions(+), 71 deletions(-)
common/cli_lil.c +34 -69
@@ -2985,32 +2985,6 @@ struct lil_value *lil_eval_expr(struct lil *lil, struct lil_value *code)return lil_alloc_integer (ee.ival); } -struct lil_value *lil_unused_name (struct lil *lil,const char *part) -{ -char *name =malloc (strlen (part) +64 ); -struct lil_value *val; -size_t i; - -for (i =0 ; i < (size_t )-1 ; i++) { -sprintf (name,"!!un!%s!%09u!nu!!" , part, (unsigned int )i); -if (lil_find_cmd (lil, name)) -continue ; - -if (lil_find_var (lil, lil->env, name)) -continue ; - - val =lil_alloc_string (name); -free (name); -return val; - } -return NULL ; -} - -struct lil_value *lil_arg (struct lil_value **argv,size_t index) -{ -return argv ? argv[index] :NULL ; -} -const char *lil_to_string (struct lil_value *val) {return (val && val->l) ? val->d :"" ;
@@ -3237,47 +3211,46 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,static struct lil_value *fnc_proc (struct lil *lil,size_t argc,struct lil_value **argv) { -struct lil_value *name;struct lil_func *cmd; -struct lil_list *fargs; +struct lil_list *args; +struct lil_value *name, *code; -if (argc <1 ) +if (argc !=3 ) { +lil_set_error_argc (lil,3 );return NULL ; - -if (argc >=3 ) { - name =lil_clone_value (argv[0 ]); - fargs =lil_subst_to_list (lil, argv[1 ]); - cmd =add_func (lil,lil_to_string (argv[0 ])); -if (!cmd) -return NULL ; - - cmd->argnames = fargs; - cmd->code =lil_clone_value (argv[2 ]); - }else { - name =lil_unused_name (lil,"anonymous-function" ); -if (argc <2 ) { -struct lil_value *tmp =lil_alloc_string ("args" ); - - fargs =lil_subst_to_list (lil, tmp); -lil_free_value (tmp); - cmd =add_func (lil,lil_to_string (name)); -if (!cmd) -return NULL ; - - cmd->argnames = fargs; - cmd->code =lil_clone_value (argv[0 ]); - }else { - fargs =lil_subst_to_list (lil, argv[0 ]); - cmd =add_func (lil,lil_to_string (name)); -if (!cmd) -return NULL ; - - cmd->argnames = fargs; - cmd->code =lil_clone_value (argv[1 ]); - } } + name =lil_clone_value (argv[0 ]); +if (!name) { +lil_set_error_oom (lil); +return NULL ; + } + + args =lil_subst_to_list (lil, argv[1 ]); +if (!args) + goto err_args; + + code =lil_clone_value (argv[2 ]); +if (!code) { +lil_set_error_oom (lil); + goto err_code; + } + + cmd =add_func (lil,lil_to_string (name)); +if (!cmd) + goto err_func; + cmd->argnames = args; + cmd->code = code; +return name; + +err_func: +lil_free_value (code); +err_code: +lil_free_list (args); +err_args: +lil_free_value (name); +return NULL ; } static struct lil_value *fnc_rename(struct lil *lil, size_t argc,
@@ -3312,13 +3285,6 @@ static struct lil_value *fnc_rename(struct lil *lil, size_t argc,return r; } -static struct lil_value *fnc_unusedname (struct lil *lil,size_t argc, -struct lil_value **argv) -{ -return lil_unused_name (lil, argc >0 ?lil_to_string (argv[0 ]) : -"unusedname" ); -} -static struct lil_value *fnc_quote (struct lil *lil,size_t argc,struct lil_value **argv) {
@@ -4310,7 +4276,6 @@ static void register_stdcmds(struct lil *lil)lil_register (lil,"substr" , fnc_substr);lil_register (lil,"topeval" , fnc_topeval);lil_register (lil,"trim" , fnc_trim); -lil_register (lil,"unusedname" , fnc_unusedname);lil_register (lil,"upeval" , fnc_upeval); } }
test/cmd/lil.c +4 -2
@@ -149,9 +149,11 @@ static const struct {"[list eh??]" "];" "asserteq_list [lapply $list length] [list 9 10 4];" -"asserteq_list [lapply $list [proc {a} {" +"proc firstword {a} {" "return [index [split $a] 0]" -"}]] [list {bad's} {good's} eh??]" +"};" +"asserteq_list [lapply $list firstword] " +"[list {bad's} {good's} eh??]" }, {"lists","set l [list foo bar baz bad];"
Signature · 1 line
2.32.0
RFC PATCH 25/28 cli: lil: Always quote items in lil_list_to_value
In reply to Sean AndersonThis function took an argument do_quote which determined whether or not to quote an item. All callers set it to true, so remove it and always quote.
---
common/cli_lil.c | 34 ++++++++++++++-------------------- include/cli_lil.h | 2 +- 2 files changed, 15 insertions(+), 21 deletions(-)
common/cli_lil.c +14 -20
@@ -2092,7 +2092,7 @@ static struct lil_value *run_cmd(struct lil *lil, struct lil_func *cmd,if (cmd->argnames->c ==1 && !strcmp (lil_to_string (lil_list_get (cmd->argnames,0 )),"args" )) { -struct lil_value *args =lil_list_to_value (words,1 ); +struct lil_value *args =lil_list_to_value (words);lil_set_var (lil,"args" , args,LIL_SETVAR_LOCAL_NEW );lil_free_value (args);
@@ -2389,7 +2389,7 @@ braces:return needs; } -struct lil_value *lil_list_to_value (struct lil_list *list,bool do_escape) +struct lil_value *lil_list_to_value (struct lil_list *list) {struct lil_value *val =alloc_value (NULL );size_t i, j;
@@ -2397,17 +2397,11 @@ struct lil_value *lil_list_to_value(struct lil_list *list, bool do_escape)for (i =0 ; i < list->c; i++) {char q;struct lil_value *item =lil_list_get (list, i); -enum needs needs; - -if (do_escape) - needs =item_needs (lil_to_string (item), item->l); -else - needs =NEEDS_NOTHING ;if (i)lil_append_char (val,' ' ); -switch (needs) { +switch (item_needs (lil_to_string (item), item->l)) {case NEEDS_NOTHING :if (lil_append_val (val, item)) goto err;
@@ -3087,7 +3081,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc, func =lil_find_cmd (lil,lil_to_string (argv[1 ]));if (!func || !func->argnames)return NULL ; -return lil_list_to_value (func->argnames,1 ); +return lil_list_to_value (func->argnames); }if (!strcmp (type,"body" )) {
@@ -3111,7 +3105,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,lil_list_append (funcs,lil_alloc_string (lil->cmd[i]->name)); - r =lil_list_to_value (funcs,1 ); + r =lil_list_to_value (funcs);lil_free_list (funcs);return r; }
@@ -3130,7 +3124,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc, env = env->parent; } - r =lil_list_to_value (vars,1 ); + r =lil_list_to_value (vars);lil_free_list (vars);return r; }
@@ -3145,7 +3139,7 @@ static struct lil_value *fnc_reflect(struct lil *lil, size_t argc,lil_list_append (vars, var); } - r =lil_list_to_value (vars,1 ); + r =lil_list_to_value (vars);lil_free_list (vars);return r; }
@@ -3513,7 +3507,7 @@ static struct lil_value *fnc_append(struct lil *lil, size_t argc,for (i = base; i < argc; i++)lil_list_append (list,lil_clone_value (argv[i])); - r =lil_list_to_value (list,1 ); + r =lil_list_to_value (list);lil_free_list (list);lil_set_var (lil, varname, r, access);return r;
@@ -3548,7 +3542,7 @@ static struct lil_value *fnc_slice(struct lil *lil, size_t argc,lil_list_append (slice,lil_clone_value (lil_list_get (list, i)));lil_free_list (list); - r =lil_list_to_value (slice,1 ); + r =lil_list_to_value (slice);lil_free_list (slice);return r; }
@@ -3586,7 +3580,7 @@ static struct lil_value *fnc_filter(struct lil *lil, size_t argc, }lil_free_list (list); - r =lil_list_to_value (filtered,1 ); + r =lil_list_to_value (filtered);lil_free_list (filtered);return r; }
@@ -3601,7 +3595,7 @@ static struct lil_value *fnc_list(struct lil *lil, size_t argc,for (i =0 ; i < argc; i++)lil_list_append (list,lil_clone_value (argv[i])); - r =lil_list_to_value (list,1 ); + r =lil_list_to_value (list);lil_free_list (list);return r; }
@@ -3628,7 +3622,7 @@ static struct lil_value *fnc_concat(struct lil *lil, size_t argc, r =lil_alloc_string ("" );for (i =0 ; i < argc; i++) { list =lil_subst_to_list (lil, argv[i]); - tmp =lil_list_to_value (list,1 ); + tmp =lil_list_to_value (list);lil_free_list (list);lil_append_val (r, tmp);lil_free_value (tmp);
@@ -3672,7 +3666,7 @@ static struct lil_value *fnc_foreach(struct lil *lil, size_t argc,break ; } - r =lil_list_to_value (rlist,1 ); + r =lil_list_to_value (rlist);lil_free_list (list);lil_free_list (rlist);return r;
@@ -4147,7 +4141,7 @@ static struct lil_value *fnc_split(struct lil *lil, size_t argc, }lil_list_append (list, val); - val =lil_list_to_value (list,1 ); + val =lil_list_to_value (list);lil_free_list (list);return val; }
include/cli_lil.h +1 -1
@@ -161,7 +161,7 @@ void lil_free_list(struct lil_list *list);int lil_list_append (struct lil_list *list,void *item);size_t lil_list_size (struct lil_list *list);struct lil_value *lil_list_get (struct lil_list *list,size_t index); -struct lil_value *lil_list_to_value (struct lil_list *list,bool do_escape); +struct lil_value *lil_list_to_value (struct lil_list *list);struct lil_list *lil_subst_to_list (struct lil *lil,struct lil_value *code);struct lil_value *lil_subst_to_value (struct lil *lil,struct lil_value *code);
Signature · 1 line
2.32.0
RFC PATCH 26/28 cli: lil: Allocate len even when str is NULL in alloc_value_len
In reply to Sean AndersonThis allows us to reserve some space ahead of time, avoiding another alloc/copy/free.
---
common/cli_lil.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
common/cli_lil.c +6 -8
@@ -459,21 +459,19 @@ static struct lil_value *alloc_value_len(const char *str, size_t len)if (!val)return NULL ;value_to_symbol (val)->type =LIL_SYMBOL_VALUE ; +ensure_capacity (val, len +1 ); +if (!val->d) { +release_to_pool (val); +return NULL ; + }if (str) { val->l = len; -ensure_capacity (val, len +1 ); -if (!val->d) { -release_to_pool (val); -return NULL ; - }memcpy (val->d, str, len); val->d[len] =0 ; }else { val->l =0 ; -ensure_capacity (val,1 ); -if (val->d) - val->d[0 ] ='\0' ; + val->d[0 ] ='\0' ; }return val; }
Signature · 1 line
2.32.0
RFC PATCH 27/28 cli: lil: Add a function to quote values
In reply to Sean AndersonThis allows us to convert lil_values into a form which can be re-parsed. We were already doing this for lists, so we just have to expose the inner loop.
---
common/cli_lil.c | 89 ++++++++++++++++++++++++++++-------------------- 1 file changed, 52 insertions(+), 37 deletions(-)
common/cli_lil.c +52 -37
@@ -2387,49 +2387,64 @@ braces:return needs; } +static enum lil_error do_quote (struct lil_value *val,struct lil_value *item) +{ +char q; +size_t i; + +switch (item_needs (lil_to_string (item), item->l)) { +case NEEDS_NOTHING : +return lil_append_val (val, item); +case NEEDS_BRACES : +return lil_append_char (val,'{' ) ?: +lil_append_val (val, item) ?: +lil_append_char (val,'}' ); +case NEEDS_DOUBLE : + q ='"' ; + goto quote; +case NEEDS_SINGLE : + q ='\'' ; +quote: +if (lil_append_char (val, q)) +return LIL_ERR_OOM ; + +for (i =0 ; i < item->l; i++) { +char c = item->d[i]; + +if (c =='\\' || c == q) +if (lil_append_char (val,'\\' )) +return LIL_ERR_OOM ; +if (lil_append_char (val, c)) +return LIL_ERR_OOM ; + } +if (lil_append_char (val, q)) +return LIL_ERR_OOM ; + } +return LIL_ERR_NONE ; +} + +static struct lil_value *lil_quote_value (struct lil_value *val) +{ +struct lil_value *r =alloc_value (NULL ); + +if (do_quote (r, val)) { +lil_free_value (r); +return NULL ; + } +return r; +} +struct lil_value *lil_list_to_value (struct lil_list *list) {struct lil_value *val =alloc_value (NULL ); -size_t i, j; +size_t i;for (i =0 ; i < list->c; i++) { -char q; -struct lil_value *item =lil_list_get (list, i); +if (i &&lil_append_char (val,' ' )) + goto err; -if (i) -lil_append_char (val,' ' ); - -switch (item_needs (lil_to_string (item), item->l)) { -case NEEDS_NOTHING : -if (lil_append_val (val, item)) - goto err; -continue ; -case NEEDS_BRACES : -if (lil_append_char (val,'{' ) || -lil_append_val (val, item) || -lil_append_char (val,'}' )) - goto err; -continue ; -case NEEDS_DOUBLE : - q ='"' ; - goto quote; -case NEEDS_SINGLE : - q ='\'' ; -quote: -if (lil_append_char (val, q)) - goto err; -for (j =0 ; j < item->l; j++) { -char c = item->d[j]; - -if (c =='\\' || c == q) -if (lil_append_char (val,'\\' )) - goto err; -if (lil_append_char (val, c)) - goto err; - } -if (lil_append_char (val, q)) - goto err; - } +if (do_quote (val,lil_list_get (list, i))) + goto err; }return val;
Signature · 1 line
2.32.0
RFC PATCH 28/28 cli: lil: Load procs from the environment
In reply to Sean AndersonWhen we start up the LIL interpreter, go through every variable and see if it looks like a new procedure. If it does, try and parse it. For the return trip, every time that we create a new procedure, create a new global variable containing that procedure.
The end result of this is that procedures should now be saved to the environment in the same way that variables are. So you can do
=> proc foo {args} { ... }
=> env saveand foo will be there after you reboot.
---
common/cli.c | 54 ++++++++++++++++++++++++++++------- common/cli_lil.c | 71 ++++++++++++++++++++++++++++++++++------------- include/cli_lil.h | 17 ++++++++++++ 3 files changed, 113 insertions(+), 29 deletions(-)
common/cli.c +44 -10
@@ -16,9 +16,11 @@#include <command.h> #include <console.h> #include <env.h> +#include <env_internal.h> #include <fdtdec.h> #include <hang.h> #include <malloc.h> +#include <search.h> #include <asm/global_data.h> DECLARE_GLOBAL_DATA_PTR ;
@@ -26,6 +28,21 @@ DECLARE_GLOBAL_DATA_PTR;#ifdef CONFIG_LIL static struct lil *lil; +static enum lil_error print_lil_err (struct lil *lil) +{ +enum lil_error ret; +const char *err_msg; + + ret =lil_error (lil, &err_msg); +if (ret) { +if (err_msg) +printf ("error: %s\n" , err_msg); +else +printf ("error: %d\n" , ret); + } +return ret; +} +static int env_setvar (struct lil *lil,const char *name,struct lil_value **value) {
@@ -41,16 +58,40 @@ static int env_getvar(struct lil *lil, const char *name,return 1 ; } +static int env_register_proc (struct env_entry *entry,void *priv) +{ +struct lil *lil = priv; +struct lil_value *name; +const char *name_str, prefix[] ="proc" ; + +/* Skip variables which are obviously not procedures */ +if (strncmp (entry->data, prefix,sizeof (prefix) -1 )) +return 0 ; + + name =lil_parse_eval (lil, entry->data,0 , true); + name_str =lil_to_string (name); +if (strcmp (entry->key, name_str)) +log_debug ("proc %s created by variable %s\n" , + name_str, entry->key); +lil_free_value (name); +return print_lil_err (lil); +} + +static int env_initprocs (struct lil *lil) +{ +return hwalk_r (&env_htab, env_register_proc, lil); +} +static const struct lil_callbacks env_callbacks = { .setvar = env_setvar, .getvar = env_getvar, + .initprocs = env_initprocs, };static int lil_run (const char *cmd) { -int err;struct lil_value *result =lil_parse_eval (lil, cmd,0 , true); -const char *err_msg, *strres =lil_to_string (result); +const char *strres =lil_to_string (result);/* The result may be very big, so use puts */ if (strres && strres[0 ]) {
@@ -59,14 +100,7 @@ static int lil_run(const char *cmd) }lil_free_value (result); - err =lil_error (lil, &err_msg); -if (err) { -if (err_msg) -printf ("error: %s\n" , err_msg); -else -printf ("error: %d\n" , err); - } -return !!err; +return !!print_lil_err (lil); }#endif
common/cli_lil.c +52 -19
107 lines not on this page · raw message
include/cli_lil.h +17 -0
@@ -94,6 +94,23 @@ struct lil_callbacks {*/ int (*getvar)(struct lil *lil,const char *name,struct lil_value **value); + +/** +* @initprocs: Called when procedures should be created +* +* @lil: The LIL interpreter +* +* This is called once in lil_new() when user functions should be added. +* When this is called, the only function registered is "proc". You can +* register functions by calling lil_parse_eval() with code which +* initializes a procedure. +* +* In practice, this is expected to be used to initialize procedures +* from environmental variables. +* +* @Return: 0 if ok, or non-zero on error. +*/ +int (*initprocs)(struct lil *lil); };/**
Signature · 1 line
2.32.0
Quoted 7 lines · Sean Anderson
Well, this has been sitting on my hard drive for too long without feedback ("Release early, release often"), so here's the first RFC. This is not ready to merge (see the "Future work" section below), but the shell is functional and at least partially tested.
The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes over on sandbox.
A good goal, but perhaps slightly too strict?
Quoted 66 lines
add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)
= Getting started
Enable
CONFIG_LIL. If you would like to run tests, enableCONFIG_LIL_FULL. Note thatdm_test_acpi_cmd_dumpandsetexpr_test_str_operwill fail.CONFIG_LIL_POOLSis currently broken (with what appears to be a double free).For an overview of the language as a whole, refer to the original readme [1].
[1] http://runtimeterror.com/tech/lil/readme.txt
== Key patches
The following patches are particularly significant for reviewing and understanding this series:
cli: Add LIL shell
This contains the LIL shell as originally written by Kostas with some major deletions and some minor additions.
cli: lil: Wire up LIL to the rest of U-Boot
This allows you to use LIL as a shell just like Hush.
cli: lil: Document structures
This adds documentation for the major structures of LIL. It is a good place to start looking at the internals.
test: Add tests for LIL
This adds some basic integration tests and provides some examples of LIL code.
cli: lil: Add a distinct parsing step
This adds a parser separate from the interpreter. This patch is the largest original work in this series.
cli: lil: Load procs from the environment
This allows procedures to be saved and loaded like variables.= A new shell
This series adds a new shell for U-Boot. The aim is to eventually replace Hush as the primary shell for all boards which currently use it. Hush should be replaced because it has several major problems:
- It has not had a major update in two decades, resulting in duplication of effort in finding bugs. Regarding a bug in variable setting, Wolfgang remarks
So the specific problem has (long) been fixed in upstream, and instead of adding a patch to our old version, thus cementing the broken behaviour, we should upgrade hush to recent upstream code.
-- Wolfgang Denk [2]
These lack of updates are further compounded by a significant amount of ifdef-ing in the Hush code. This makes the shell hard to read and debug. Further, the original purpose of such ifdef-ing (upgrading to a newer Hush) has never happened.
- It was designed for a preempting OS which supports pipes and processes. This fundamentally does not match the computing model of U-Boot where there is exactly one thread (and every other CPU is spinning or sleeping). Working around these design differences is a significant cause of the aformentioned ifdef-ing.
- It lacks many major features expected of even the most basic shells, such as functions and command substitution ($() syntax). This makes it difficult to script with Hush. While it is desirable to write some code in C, much code *must* be written in C because there is no way to express the logic in Hush.
Quoted 5 lines
I believe that U-Boot should have a shell which is more featureful, has cleaner code, and which is the same size as Hush (or less). The ergonomic advantages afforded by a new shell will make U-Boot easier to use and customize.
First, great! Thanks for doing this. A new shell really is the only viable path forward here, and I appreciate you taking the time to evaluate several and implement one.
Quoted 29 lines
= Open questions
While the primary purpose of this series is of course to get feedback on the code I have already written, there are several decisions where I am not sure what the best course of action is.
- What should be done about 'expr'? The 'expr' command is a significant portion of the final code size. It cannot be removed outright, because it is used by several builtin functions like 'if', 'while', 'for', etc. The way I see it, there are two general approaches to take
- Rewrite expr to parse expressions and then evaluate them. The parsing could re-use several of the existing parse functions like how
parse_listdoes. This could reduce code, as instead of many functions each with their own while/switch statements, we could have two while/switch statements (one to parse, and one to evaluate). However, this may end up increasing code size (such as when the main language had evaluation split from parsing).- Don't parse infix expressions, and just make arithmetic operators normal functions. This would affect ergonomics a bit. For example, instead of
if {$i < 10} { ... }
one would need to write
if {< $i 10} { ... }
and instead of
Quoted 8 lines
if {$some_bool} { ... }
one would need to write
if {quote $some_bool} { ... }
Though, given how much setexpr is used (not much), this may not be such a big price to pay. This route is almost certain to reduce code size.
So, this is a question because we have cmd/setexpr.c that provides "expr" today? Or because this is a likely place to reclaim some of that 800 byte growth?
Quoted 9 lines
- How should LIL functions integrate with the rest of U-Boot? At the moment, lil functions and procedures exist in a completely separate world from normal commands. I would like to integrate them more closely, but I am not sure the best way to go about this. At the very minimum, each LIL builtin function needs to get its hands on the LIL interpreter somehow. I'd rather this didn't happen through
gd_tor similar so that it is easier to unit test. Additionally, LIL functions expect an array oflil_valuesinstead of strings. We could strip them out, but I worry that might start to impact performance (from all the copying).
I might be missing something here. But, given that whenever we have C code run-around and generate a string to then pass to the interpreter to run, someone asks why we don't just make API calls directly, perhaps the answer is that we don't need to?
Quoted 77 lines
The other half of this is adding LIL features into regular commands. The most important feature here is being able to return a string result. I took an initial crack at it [3], but I think with this series there is a stronger motivating factor (along with things like [4]).
[3] https://patchwork.ozlabs.org/project/uboot/list/?series=231377
[4] https://patchwork.ozlabs.org/project/uboot/list/?series=251013= Future work
The series as presented today is incomplete. The following are the major issues I see with it at the moment. I would like to address all of these issues, but some of them might be postponed until after first merging this series.
- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly, there still remains a significant amount of original code which just ignores errors. In particular, I would like to ensure that the following categories of error conditions are handled:
- Running out of memory.
- Access to a nonexistant variable.
- Passing the wrong number of arguments to a function.
- Interpreting a value as the wrong type (e.g. "foo" should not have a numeric representation, instead of just being treated as 1).- There are many deviations from TCL with no purpose. For example, the list indexing function is named "index" and not "lindex". It is perfectly fine to drop features or change semantics to reduce code size, make parsing easier, or make execution easier. But changing things for the sake of it should be avoided.
- The test suite is rather anemic compared with the amount of code this series introduces. I would like to expand it significantly. In particular, error conditions are not well tested (only the "happy path" is tested).
- While I have documented all new functions I have written, there are many existing functions which remain to be documented. In addition, there is no user documentation, which is critical in driving adoption of any new programming language. Some of this cover letter might be integrated with any documentation written.
- Some shell features such as command repetition and secondary shell prompts have not been implemented.
- Arguments to native lil functions are incompatible with U-Boot functions. For example, the command
foo bar baz
would be passed to a U-Boot command as
{ "foo", "bar", "baz", NULL }but would be passed to a LIL function as
{ "bar", "baz" }This makes it more difficult to use the same function to parse several different commands. At the moment this is solved by passing the command name in
lil->env->proc, but I would like to switch to the U-Boot argument list style.- Several existing tests break when using LIL because they expect no output on failure, but LIL produces some output notifying the user of the failure.
- Implement
DISTRO_BOOTin LIL. I think this is an important proof-of-concept to show what can be done with LIL, and to determine which features should be moved toLIL_FULL.= Why Lil?
When looking for a suitable replacement shell, I evaluated implementations using the following criteria:
Quoted 13 lines
- It must have a GPLv2-compatible license.
- It must be written in C, and have no major external dependencies.
- It must support bare function calls. That is, a script such as 'foo bar' should invoke the function 'foo' with the argument 'bar'. This preserves the shell-like syntax we expect.
- It must be small. The eventual target is that it compiles to around 10KiB with -Os and -ffunction-sections.
- There should be good tests. Any tests at all are good, but a functioning suite is better.
- There should be good documentation
- There should be comments in the source.
- It should be "finished" or have only slow development. This will hopefully make it easier to port changes.
On this last point, I believe this is based on lil20190821 and current is now lil20210502. With a quick diff between them, I can see that the changes there are small enough that while you've introduced a number of changes here, it would be a very easy update.
Quoted 79 lines
Notably absent from the above list is performance. Most scripts in U-Boot will be run once on boot. As long as the time spent evaluating scripts is kept under a reasonable threshold (a fraction of the time spend initializing hardware or reading data from persistant storage), there is no need to optimize for speed.
In addition, I did not consider updating Hush from Busybox. The mismatch in computing environment expectations (as noted in the "New shell" section above) still applies. IMO, this mismatch is the biggest reason that things like functions and command substitution have been excluded from the U-Boot's Hush.
== lil
- zLib
- TCL
- Compiles to around 10k with no builtins. To 25k with builtins.
- Some tests, but not organized into a suite with expected output. Some evidence that the author ran APL, but no harness.
- Some architectural documentation. Some for each functions, but not much.
- No comments :l
- 3.5k LoC== picol
- 2-clause BSD
- TCL
- Compiles to around 25k with no builtins. To 80k with builtins.
- Tests with suite (in-language). No evidence of fuzzing.
- No documentation :l
- No comments :l
- 5k LoC== jimtcl
- 2-clause BSD
- TCL
- Compiles to around 95k with no builtins. To 140k with builtins. Too big...== boron
- LGPLv3+ (so this is right out)
- REBOL
- Compiles to around 125k with no builtins. To 190k with builtins. Too big...== libmawk
- GPLv2
- Awk
- Compiles to around 225k. Too big...== libfawk
- 3-clause BSD
- Uses bison+yacc...
- Awk; As it turns out, this has parentheses for function calls.
- Compiles to around 24-30k. Not sure how to remove builtins.
- Test suite (in-language). No fuzzing.
- Tutorial book. No function reference.
- No comments
- Around 2-4k LoC== MicroPython
- MIT
- Python (but included for completeness)
- Compiles to around 300k. Too big...== mruby/c
- 3-clause BSD
- Ruby
- Compiles to around 85k without builtins and 120k with. Too big...== eLua
- MIT
- Lua
- Build system is a royal pain (custom and written in Lua with external deps)
- Base binary is around 250KiB and I don't want to deal with reducing it
Quoted 8 lines
So the interesting/viable ones are
- lil
- picol
- libfawk (maybe)I started with LIL because it was the smallest. I have found several issues with LIL along the way. Some of these are addressed in this series already, while others remain unaddressed (see the section "Future Work").
Thanks for the evaluations, of these, lil does make the most sense.
Signature · 1 line
Tom
Dear Tom,
Quoted 4 lines · In message <20210701202155.GQ9516@bill-the-cat> you wrote:
First, great! Thanks for doing this. A new shell really is the only viable path forward here, and I appreciate you taking the time to evaluate several and implement one.
I disagree that a new shell is the _only_ way forward.
AFAICT, all the raised concerns have long been fixed in upstream versions of hush; see for example [1]:
...
//config: hush is a small shell. It handles the normal flow control
//config: constructs such as if/then/elif/else/fi, for/in/do/done, while loops,
//config: case/esac. Redirections, here documents, $((arithmetic))
//config: and functions are supported.
//config:
[1] https://git.busybox.net/busybox/tree/shell/hush.c#n98
My gut feeling is that updating to a recent version of hush is the most efficent _backward_compatible_ way.
And if we drop that requirement, we might even take a bigger step and move to lua - which would allow for a complete new level of script based extensions.
Quoted 6 lines
Quoted 6 lines
- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly, there still remains a significant amount of original code which just ignores errors. In particular, I would like to ensure that the following categories of error conditions are handled:
This is something that scares me like hell. This in a shell? For me this is close to a killing point.
Quoted 5 lines
Quoted 5 lines
- Running out of memory.
- Access to a nonexistant variable.
- Passing the wrong number of arguments to a function.
- Interpreting a value as the wrong type (e.g. "foo" should not have a numeric representation, instead of just being treated as 1).
Who says so?
Bash says:
-> printf "%d\n" foo -bash: printf: foo: invalid number 0
So it is _not_ 1 ...
Quoted 5 lines
Quoted 5 lines
- There are many deviations from TCL with no purpose. For example, the list indexing function is named "index" and not "lindex". It is perfectly fine to drop features or change semantics to reduce code size, make parsing easier, or make execution easier. But changing things for the sake of it should be avoided.
It's not a standard POSIX shell, it's not TCL (ick!), ... it's something new, incompatible...
Quoted 1 line
Thanks for the evaluations, of these, lil does make the most sense.
You mean, adding a complete new, incompatible and non-standard shell is a better approach than updating to a recent version of hush?
What makes you think so?
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] Ernest asks Frank how long he has been working for the company. "Ever since they threatened to fire me."
Quoted 13 lines · Wolfgang Denk
Dear Tom,
Quoted 4 lines · In message <20210701202155.GQ9516@bill-the-cat> you wrote:
First, great! Thanks for doing this. A new shell really is the only viable path forward here, and I appreciate you taking the time to evaluate several and implement one.
I disagree that a new shell is the _only_ way forward.
AFAICT, all the raised concerns have long been fixed in upstream versions of hush; see for example [1]:
Quoted 8 lines
...
//config: hush is a small shell. It handles the normal flow control
//config: constructs such as if/then/elif/else/fi, for/in/do/done, while loops,
//config: case/esac. Redirections, here documents, $((arithmetic))
//config: and functions are supported.
//config:
In fact, the code for most of this is present but ifdef'd out. The real issue is that the implementation of much of the above relies on things like fork() which we can't provide.
Quoted 6 lines
My gut feeling is that updating to a recent version of hush is the most efficent _backward_compatible_ way.
And if we drop that requirement, we might even take a bigger step and move to lua - which would allow for a complete new level of script based extensions.
I'm not aware of any Lua implementations which meet the size requirements for U-Boot. Any port would need to be either written from scratch or have some serious modification, not unlike what I've done here.
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly, there still remains a significant amount of original code which just ignores errors. In particular, I would like to ensure that the following categories of error conditions are handled:
This is something that scares me like hell. This in a shell? For me this is close to a killing point.
Yes, it was for me as well. But it is not something so obvious unless you are looking for it. I believe I have addressed this issue in much of the core code (parser and interpreter). But the builtin commands must be gone through and converted.
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
- Running out of memory.
- Access to a nonexistant variable.
- Passing the wrong number of arguments to a function.
- Interpreting a value as the wrong type (e.g. "foo" should not have a numeric representation, instead of just being treated as 1).Who says so?
The current LIL code.
Quoted 8 lines
Bash says:
-> printf "%d\n" foo -bash: printf: foo: invalid number 0So it is _not_ 1 ...
And I would like to replicate this behavior.
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
- There are many deviations from TCL with no purpose. For example, the list indexing function is named "index" and not "lindex". It is perfectly fine to drop features or change semantics to reduce code size, make parsing easier, or make execution easier. But changing things for the sake of it should be avoided.
It's not a standard POSIX shell, it's not TCL (ick!), ... it's something new, incompatible...
And Hush isn't POSIX either :)
But as noted above, I would like to hew much closer to TCL than LIL has traditionally done.
Quoted 4 lines
Quoted 1 line
Thanks for the evaluations, of these, lil does make the most sense.
You mean, adding a complete new, incompatible and non-standard shell is a better approach than updating to a recent version of hush?
Correct.
Quoted 1 line
What makes you think so?
Hush relies heavily on its posix environment. Porting it has, and will, require substantial modification. IMO the work necessary will be around the same or more for Hush as for any other language.
--Sean
Quoted 11 lines · Tom Rini
Quoted 7 lines · Sean Anderson
Well, this has been sitting on my hard drive for too long without feedback ("Release early, release often"), so here's the first RFC. This is not ready to merge (see the "Future work" section below), but the shell is functional and at least partially tested.
The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes over on sandbox.
A good goal, but perhaps slightly too strict?
Perhaps. But I think getting in the ballpark will significantly help drive adoption. I want to make it as easy as possible for maintainers to enable LIL and start using it.
Quoted 110 lines
Quoted 67 lines
add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)
= Getting started
Enable
CONFIG_LIL. If you would like to run tests, enableCONFIG_LIL_FULL. Note thatdm_test_acpi_cmd_dumpandsetexpr_test_str_operwill fail.CONFIG_LIL_POOLSis currently broken (with what appears to be a double free).For an overview of the language as a whole, refer to the original readme [1].
[1] http://runtimeterror.com/tech/lil/readme.txt
== Key patches
The following patches are particularly significant for reviewing and understanding this series:
cli: Add LIL shell
This contains the LIL shell as originally written by Kostas with some major deletions and some minor additions.
cli: lil: Wire up LIL to the rest of U-Boot
This allows you to use LIL as a shell just like Hush.
cli: lil: Document structures
This adds documentation for the major structures of LIL. It is a good place to start looking at the internals.
test: Add tests for LIL
This adds some basic integration tests and provides some examples of LIL code.
cli: lil: Add a distinct parsing step
This adds a parser separate from the interpreter. This patch is the largest original work in this series.
cli: lil: Load procs from the environment
This allows procedures to be saved and loaded like variables.= A new shell
This series adds a new shell for U-Boot. The aim is to eventually replace Hush as the primary shell for all boards which currently use it. Hush should be replaced because it has several major problems:
- It has not had a major update in two decades, resulting in duplication of effort in finding bugs. Regarding a bug in variable setting, Wolfgang remarks
So the specific problem has (long) been fixed in upstream, and instead of adding a patch to our old version, thus cementing the broken behaviour, we should upgrade hush to recent upstream code.
-- Wolfgang Denk [2]
These lack of updates are further compounded by a significant amount of ifdef-ing in the Hush code. This makes the shell hard to read and debug. Further, the original purpose of such ifdef-ing (upgrading to a newer Hush) has never happened.
- It was designed for a preempting OS which supports pipes and processes. This fundamentally does not match the computing model of U-Boot where there is exactly one thread (and every other CPU is spinning or sleeping). Working around these design differences is a significant cause of the aformentioned ifdef-ing.
- It lacks many major features expected of even the most basic shells, such as functions and command substitution ($() syntax). This makes it difficult to script with Hush. While it is desirable to write some code in C, much code *must* be written in C because there is no way to express the logic in Hush.
Quoted 5 lines
I believe that U-Boot should have a shell which is more featureful, has cleaner code, and which is the same size as Hush (or less). The ergonomic advantages afforded by a new shell will make U-Boot easier to use and customize.
First, great! Thanks for doing this. A new shell really is the only viable path forward here, and I appreciate you taking the time to evaluate several and implement one.
Quoted 33 lines
= Open questions
While the primary purpose of this series is of course to get feedback on the code I have already written, there are several decisions where I am not sure what the best course of action is.
- What should be done about 'expr'? The 'expr' command is a significant portion of the final code size. It cannot be removed outright, because it is used by several builtin functions like 'if', 'while', 'for', etc. The way I see it, there are two general approaches to take
- Rewrite expr to parse expressions and then evaluate them. The parsing could re-use several of the existing parse functions like how
parse_listdoes. This could reduce code, as instead of many functions each with their own while/switch statements, we could have two while/switch statements (one to parse, and one to evaluate). However, this may end up increasing code size (such as when the main language had evaluation split from parsing).- Don't parse infix expressions, and just make arithmetic operators normal functions. This would affect ergonomics a bit. For example, instead of
if {$i < 10} { ... }
one would need to write
if {< $i 10} { ... }
and instead of
if {$some_bool} { ... }
one would need to write
Quoted 8 lines
Quoted 4 lines
if {quote $some_bool} { ... }
Though, given how much setexpr is used (not much), this may not be such a big price to pay. This route is almost certain to reduce code size.
So, this is a question because we have
cmd/setexpr.cthat provides "expr" today? Or because this is a likely place to reclaim some of that 800 byte growth?
The latter. setexpr cannot be used because it does not return a result, and instead sets a (global) variable. The expression parsing functionality is core to LIL and used in many builtin commands (such as if above), and really needs to return a lil_value.
Quoted 11 lines
Quoted 10 lines
- How should LIL functions integrate with the rest of U-Boot? At the moment, lil functions and procedures exist in a completely separate world from normal commands. I would like to integrate them more closely, but I am not sure the best way to go about this. At the very minimum, each LIL builtin function needs to get its hands on the LIL interpreter somehow. I'd rather this didn't happen through
gd_tor similar so that it is easier to unit test. Additionally, LIL functions expect an array oflil_valuesinstead of strings. We could strip them out, but I worry that might start to impact performance (from all the copying).
Quoted 4 lines
I might be missing something here. But, given that whenever we have C code run-around and generate a string to then pass to the interpreter to run, someone asks why we don't just make API calls directly, perhaps the answer is that we don't need to?
err, the issue here is that the signature for regular commands is rougly
int cmd(..., int argc, char **argv, ...)
and the signature for LIL commands is
struct lil_value *cmd(struct lil *lil, size_t argc, struct lil_value **argv)
where lil_value is
struct lil_value {
size_t l;
char *d;
};so while regular commands can be reimplemented as LIL commands (just create a new argv containing the strings directly), it is more difficult to go the other way. I bring this up because I think having two separate ways to write a command is not the best way to do things going forward.
Quoted 91 lines
Quoted 77 lines
The other half of this is adding LIL features into regular commands. The most important feature here is being able to return a string result. I took an initial crack at it [3], but I think with this series there is a stronger motivating factor (along with things like [4]).
[3] https://patchwork.ozlabs.org/project/uboot/list/?series=231377
[4] https://patchwork.ozlabs.org/project/uboot/list/?series=251013= Future work
The series as presented today is incomplete. The following are the major issues I see with it at the moment. I would like to address all of these issues, but some of them might be postponed until after first merging this series.
- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly, there still remains a significant amount of original code which just ignores errors. In particular, I would like to ensure that the following categories of error conditions are handled:
- Running out of memory.
- Access to a nonexistant variable.
- Passing the wrong number of arguments to a function.
- Interpreting a value as the wrong type (e.g. "foo" should not have a numeric representation, instead of just being treated as 1).- There are many deviations from TCL with no purpose. For example, the list indexing function is named "index" and not "lindex". It is perfectly fine to drop features or change semantics to reduce code size, make parsing easier, or make execution easier. But changing things for the sake of it should be avoided.
- The test suite is rather anemic compared with the amount of code this series introduces. I would like to expand it significantly. In particular, error conditions are not well tested (only the "happy path" is tested).
- While I have documented all new functions I have written, there are many existing functions which remain to be documented. In addition, there is no user documentation, which is critical in driving adoption of any new programming language. Some of this cover letter might be integrated with any documentation written.
- Some shell features such as command repetition and secondary shell prompts have not been implemented.
- Arguments to native lil functions are incompatible with U-Boot functions. For example, the command
foo bar baz
would be passed to a U-Boot command as
{ "foo", "bar", "baz", NULL }but would be passed to a LIL function as
{ "bar", "baz" }This makes it more difficult to use the same function to parse several different commands. At the moment this is solved by passing the command name in
lil->env->proc, but I would like to switch to the U-Boot argument list style.- Several existing tests break when using LIL because they expect no output on failure, but LIL produces some output notifying the user of the failure.
- Implement
DISTRO_BOOTin LIL. I think this is an important proof-of-concept to show what can be done with LIL, and to determine which features should be moved toLIL_FULL.= Why Lil?
When looking for a suitable replacement shell, I evaluated implementations using the following criteria:
Quoted 13 lines
- It must have a GPLv2-compatible license.
- It must be written in C, and have no major external dependencies.
- It must support bare function calls. That is, a script such as 'foo bar' should invoke the function 'foo' with the argument 'bar'. This preserves the shell-like syntax we expect.
- It must be small. The eventual target is that it compiles to around 10KiB with -Os and -ffunction-sections.
- There should be good tests. Any tests at all are good, but a functioning suite is better.
- There should be good documentation
- There should be comments in the source.
- It should be "finished" or have only slow development. This will hopefully make it easier to port changes.
Quoted 4 lines
On this last point, I believe this is based on lil20190821 and current is now lil20210502. With a quick diff between them, I can see that the changes there are small enough that while you've introduced a number of changes here, it would be a very easy update.
From what I understand, the only changes are updated copyrights and the addition of a license file to cover the tests.
--Sean
Hello Sean,
Quoted 9 lines · Sean Anderson
Well, this has been sitting on my hard drive for too long without feedback ("Release early, release often"), so here's the first RFC. This is not ready to merge (see the "Future work" section below), but the shell is functional and at least partially tested.
The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes over on sandbox.
add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)
Thanks for this comparison. Here just my thoughts, but I have currently to low time to follow here completly, so sorry if I miss stuff already discussed.
From my side of view we can add a second shell beside hush, and as you already mentioned it may is even possible to start LIL from hush. This would be cool.
But I disagree in the following points:
- I do not see why LIL should *replace* hush. This may break a lot of current boards. Also I see no need to do this, as if you do complex stuff in u-boot shell it may is the wrong way and you have to think about... and I see no big discussions here about BUGS in our current hush implementation ... may it makes sense to bring them up and fix?
- Is LIL really so good code, that it is done? Or is there only no community which develop further?
- Do we really want code, which does no error checking? Commentless code...
Quoted 3 lines · you
- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly,
Which really let me think, nobody is interested in this code ... this sounds like crap... and this brings me to the next point
- Do we not have the same problem as with hush, if we change LIL code?
Do you plan to upstream your changes?
So please do not understand me wrong, I see you invested much time here, but I see LIL not in the short term as a replacement for hush.
I would accept to add LIL as an option and we will see, how good it works.
And if we have in lets say 2-3 years more boards which use LIL instead of hush *and* also active maintained LIL code, we may can think of getting rid of hush... and may have a "hush compatibility" layer...
I have no idea of current hush mainline code and how many work it will be to update U-Boot to a current hush version. But may this would be make more sense as hush is in active development. Also may to discuss with hush maintainers to upstream U-Boot changes... this would make in my eyes more sense.
Quoted 98 lines
= Why Lil?
When looking for a suitable replacement shell, I evaluated implementations using the following criteria:
- It must have a GPLv2-compatible license.
- It must be written in C, and have no major external dependencies.
- It must support bare function calls. That is, a script such as 'foo bar' should invoke the function 'foo' with the argument 'bar'. This preserves the shell-like syntax we expect.
- It must be small. The eventual target is that it compiles to around 10KiB with -Os and -ffunction-sections.
- There should be good tests. Any tests at all are good, but a functioning suite is better.
- There should be good documentation
- There should be comments in the source.
- It should be "finished" or have only slow development. This will hopefully make it easier to port changes.Notably absent from the above list is performance. Most scripts in U-Boot will be run once on boot. As long as the time spent evaluating scripts is kept under a reasonable threshold (a fraction of the time spend initializing hardware or reading data from persistant storage), there is no need to optimize for speed.
In addition, I did not consider updating Hush from Busybox. The mismatch in computing environment expectations (as noted in the "New shell" section above) still applies. IMO, this mismatch is the biggest reason that things like functions and command substitution have been excluded from the U-Boot's Hush.
== lil
- zLib
- TCL
- Compiles to around 10k with no builtins. To 25k with builtins.
- Some tests, but not organized into a suite with expected output. Some evidence that the author ran APL, but no harness.
- Some architectural documentation. Some for each functions, but not much.
- No comments :l
- 3.5k LoC== picol
- 2-clause BSD
- TCL
- Compiles to around 25k with no builtins. To 80k with builtins.
- Tests with suite (in-language). No evidence of fuzzing.
- No documentation :l
- No comments :l
- 5k LoC== jimtcl
- 2-clause BSD
- TCL
- Compiles to around 95k with no builtins. To 140k with builtins. Too big...== boron
- LGPLv3+ (so this is right out)
- REBOL
- Compiles to around 125k with no builtins. To 190k with builtins. Too big...== libmawk
- GPLv2
- Awk
- Compiles to around 225k. Too big...== libfawk
- 3-clause BSD
- Uses bison+yacc...
- Awk; As it turns out, this has parentheses for function calls.
- Compiles to around 24-30k. Not sure how to remove builtins.
- Test suite (in-language). No fuzzing.
- Tutorial book. No function reference.
- No comments
- Around 2-4k LoC== MicroPython
- MIT
- Python (but included for completeness)
- Compiles to around 300k. Too big...== mruby/c
- 3-clause BSD
- Ruby
- Compiles to around 85k without builtins and 120k with. Too big...== eLua
- MIT
- Lua
- Build system is a royal pain (custom and written in Lua with external deps)
- Base binary is around 250KiB and I don't want to deal with reducing it
Quoted 4 lines
So the interesting/viable ones are
- lil
- picol
- libfawk (maybe)
Very nice overview, thanks!
bye,
Heiko
Signature · 3 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: +49-8142-66989-52 Fax: +49-8142-66989-80 Email: [email protected]
Quoted 17 lines · Heiko Schocher
Hello Sean,
Quoted 9 lines · Sean Anderson
Well, this has been sitting on my hard drive for too long without feedback ("Release early, release often"), so here's the first RFC. This is not ready to merge (see the "Future work" section below), but the shell is functional and at least partially tested.
The goal is to have 0 bytes gained over Hush. Currently we are around 800 bytes over on sandbox.
add/remove: 90/54 grow/shrink: 3/7 up/down: 12834/-12042 (792)
Thanks for this comparison. Here just my thoughts, but I have currently to low time to follow here completly, so sorry if I miss stuff already discussed.
Quoted 7 lines
From my side of view we can add a second shell beside hush, and as you
already mentioned it may is even possible to start LIL from hush. This would be cool.But I disagree in the following points:
- I do not see why LIL should *replace* hush.
This is the eventual goal. This series just adds an option at build time (CONFIG_LIL) which uses LIL as the shell instead of Hush. But anyone who writes a series like this must be convinced that it is better than what already exists. So I write that my goal is to replace hush, even if I think that it will take a long while.
Quoted 5 lines
This may break a lot of current boards. Also I see no need to do this, as if you do complex stuff in u-boot shell it may is the wrong way and you have to think about... and I see no big discussions here about BUGS in our current hush implementation ... may it makes sense to bring them up and fix?
I have been told not to work on the current shell; that it is too old and there is no point in improving it. Since we will be starting afresh anyway, and I do not think that the Hush model is particularly good, I chose to work from a different base.
Quoted 2 lines
- Is LIL really so good code, that it is done? Or is there only no community which develop further?
AFAIK there is not too much of a LIL community. The code quality is something I found out after working on it for a while. From a cursory inspection, it was around the same as all the other shells I looked at (no comments, etc). FWIW the current Hush shell does not really handle errors either. For example, on (x)malloc failure, we print and hang. This could easily be done for LIL, but it is not so nice when your language has recursion. So I have tried to add error checking where possible. And of course, Hush ignores missing variables as well.
Quoted 11 lines
- Do we really want code, which does no error checking? Commentless code...
Quoted 3 lines · you
- There is a serious error handling problem. Most original LIL code never checked errors. In almost every case, errors were silently ignored, even malloc failures! While I have designed new code to handle errors properly,
Which really let me think, nobody is interested in this code ... this sounds like crap... and this brings me to the next point
- Do we not have the same problem as with hush, if we change LIL code?
I think the primary problem with Hush was that we expected to backport updates at all. I think such a core UI feature should not have so many ifdefs in it when no backport was ever made. It is difficult to modify Hush because not only do you have to understand how U-Boot uses some feature, you also have to understand how Busybox uses some feature. And Hush of course has less comments as LIL does (after all patches in this series are applied).
Quoted 1 line
Do you plan to upstream your changes?
I don't view it as especially important. I view LIL as a starting point which can be modified to suit U-Boot. It may end up that each individual component is changed in some way, but starting with a working shell has been very helpful for development.
Quoted 4 lines
So please do not understand me wrong, I see you invested much time here, but I see LIL not in the short term as a replacement for hush.
I would accept to add LIL as an option and we will see, how good it works.
That is what I intend to do.
Quoted 8 lines
And if we have in lets say 2-3 years more boards which use LIL instead of hush *and* also active maintained LIL code, we may can think of getting rid of hush... and may have a "hush compatibility" layer...
I have no idea of current hush mainline code and how many work it will be to update U-Boot to a current hush version. But may this would be make more sense as hush is in active development. Also may to discuss with hush maintainers to upstream U-Boot changes... this would make in my eyes more sense.
AFAICT most of the U-Boot changes are removal of features, workarounds for unimplemented functionality (e.g. anywhere Hush does fork/exec), and some small additions like command repetition which cannot be upstreamed because it would break any semblance of POSIX compatibility.
--Sean
Quoted 108 lines
Quoted 98 lines
= Why Lil?
When looking for a suitable replacement shell, I evaluated implementations using the following criteria:
- It must have a GPLv2-compatible license.
- It must be written in C, and have no major external dependencies.
- It must support bare function calls. That is, a script such as 'foo bar' should invoke the function 'foo' with the argument 'bar'. This preserves the shell-like syntax we expect.
- It must be small. The eventual target is that it compiles to around 10KiB with -Os and -ffunction-sections.
- There should be good tests. Any tests at all are good, but a functioning suite is better.
- There should be good documentation
- There should be comments in the source.
- It should be "finished" or have only slow development. This will hopefully make it easier to port changes.Notably absent from the above list is performance. Most scripts in U-Boot will be run once on boot. As long as the time spent evaluating scripts is kept under a reasonable threshold (a fraction of the time spend initializing hardware or reading data from persistant storage), there is no need to optimize for speed.
In addition, I did not consider updating Hush from Busybox. The mismatch in computing environment expectations (as noted in the "New shell" section above) still applies. IMO, this mismatch is the biggest reason that things like functions and command substitution have been excluded from the U-Boot's Hush.
== lil
- zLib
- TCL
- Compiles to around 10k with no builtins. To 25k with builtins.
- Some tests, but not organized into a suite with expected output. Some evidence that the author ran APL, but no harness.
- Some architectural documentation. Some for each functions, but not much.
- No comments :l
- 3.5k LoC== picol
- 2-clause BSD
- TCL
- Compiles to around 25k with no builtins. To 80k with builtins.
- Tests with suite (in-language). No evidence of fuzzing.
- No documentation :l
- No comments :l
- 5k LoC== jimtcl
- 2-clause BSD
- TCL
- Compiles to around 95k with no builtins. To 140k with builtins. Too big...== boron
- LGPLv3+ (so this is right out)
- REBOL
- Compiles to around 125k with no builtins. To 190k with builtins. Too big...== libmawk
- GPLv2
- Awk
- Compiles to around 225k. Too big...== libfawk
- 3-clause BSD
- Uses bison+yacc...
- Awk; As it turns out, this has parentheses for function calls.
- Compiles to around 24-30k. Not sure how to remove builtins.
- Test suite (in-language). No fuzzing.
- Tutorial book. No function reference.
- No comments
- Around 2-4k LoC== MicroPython
- MIT
- Python (but included for completeness)
- Compiles to around 300k. Too big...== mruby/c
- 3-clause BSD
- Ruby
- Compiles to around 85k without builtins and 120k with. Too big...== eLua
- MIT
- Lua
- Build system is a royal pain (custom and written in Lua with external deps)
- Base binary is around 250KiB and I don't want to deal with reducing itQuoted 4 lines
So the interesting/viable ones are
- lil
- picol
- libfawk (maybe)Very nice overview, thanks!
bye,
Heiko