< "$@" doesn't expand properly?
hello everyone,
I wrote a simple "inplace" script to be able to
work with commands that have no -i options.
The simplest way to do it (I thought) was
#!/bin/sh
set -eux
< "$@" | sponge "$1"
so inplace counter awk '{$0+=2}1' would be expanded as
< 'counter' 'awk' '{$0+=2}1' | sponge 'counter'
but there is an error message
./inplace: 3: cannot open counter awk {$0+=2}1: No such file
so it seems that the expansion leads to
< 'counter awk {$0+=2}1' | sponge 'counter'
is this an expected behavior?
regards
Signature · 6 lines
Marc Chantreux Pôle CESAR (Calcul et services avancés à la recherche) Université de Strasbourg 14 rue René Descartes, BP 80010, 67084 STRASBOURG CEDEX 03.68.85.60.79
Quoted 14 lines · Marc Chantreux
The simplest way to do it (I thought) was
#!/bin/sh
set -eux
< "$@" | sponge "$1"so inplace counter awk '{$0+=2}1' would be expanded as
< 'counter' 'awk' '{$0+=2}1' | sponge 'counter'
but there is an error message
Quoted 7 lines
./inplace: 3: cannot open counter awk {$0+=2}1: No such file
so it seems that the expansion leads to
< 'counter awk {$0+=2}1' | sponge 'counter'
is this an expected behavior?
POSIX leaves this unspecified. Section 2.7 [1] says that field splitting is not performed on the word following a redirection operator, and section 2.5.2 [2] says that the expansion of "$@" is unspecified in that context.
Almost no shell works the way you are expecting, except for zsh in native mode (which I suspect is where you first tried this syntax):
% cat /tmp/x.sh
< "$@"
% ./bash /tmp/x.sh a b
/tmp/x.sh: line 1: "$@": ambiguous redirect
% dash /tmp/x.sh a b
/tmp/x.sh: 1: cannot open a b: No such file
% ksh /tmp/x.sh a b
/tmp/x.sh[1]: a b: cannot open [No such file or directory] % mksh /tmp/x.sh a b
/tmp/x.sh[1]: can't open a b: No such file or directory % oksh /tmp/x.sh a b
/tmp/x.sh[1]: cannot open a b: No such file or directory % yash /tmp/x.sh a b
yash: redirection: cannot open file `a b': No such file or directory % zsh /tmp/x.sh a b
/tmp/x.sh:1: no such file or directory: a
% zsh --emulate sh /tmp/x.sh a b
/tmp/x.sh:1: no such file or directory: a b
[1] https://pubs.opengroup.org/onlinepubs/9799919799.2024edition/utilities/V3_chap02.html#tag_19_07
[2] https://pubs.opengroup.org/onlinepubs/9799919799.2024edition/utilities/V3_chap02.html#tag_19_05_02
Signature · 1 line
vq
Hi Lawrence,
Quoted 4 lines
POSIX leaves this unspecified. Section 2.7 [1] says that field splitting is not performed on the word following a redirection operator, and section 2.5.2 [2] says that the expansion of "$@" is unspecified in that context.
wow ... thank you so much for digging this. this is the second time this year where oppose POSIX standard against consistency and ease of use.
I can understand why because most of the scripts I learned with were postfixing the redirections so word splitting doesn't matter at this point but prefixing the < is really a matter of easing the readability and maintainance:
< A B | C | D > E
is much better for your brain than
B < A | C | D > E
Quoted 2 lines
Almost no shell works the way you are expecting, except for zsh in native mode (which I suspect is where you first tried this syntax)
Not at all. Actually dash is not only the more lightweight shell I know but also the one which behave as close as what I expect from a POSIX shell so it became my first choice when it comes to scripting.
No shell behave the way I would like (for example, none but zsh allow prefixed redirections to compound expressions) to but the closest are by far rc, dash and zsh.
Quoted 7 lines
% ./bash
/tmp/x.sha b
% dash/tmp/x.sha b
% ksh/tmp/x.sha b
% mksh/tmp/x.sha b
% oksh/tmp/x.sha b
% yash/tmp/x.sha b
% zsh/tmp/x.sha b
I see we have the same shells in the scope. I just dropped bash from my radar because its competitor (heavy but feature oriented) is way better so I keep telling people to use zsh instead.
when in comes to the ratio (needed_features / kb), dash is gold.
Quoted 2 lines
[1] https://pubs.opengroup.org/onlinepubs/9799919799.2024edition/utilities/V3_chap02.html#tag_19_07
[2] https://pubs.opengroup.org/onlinepubs/9799919799.2024edition/utilities/V3_chap02.html#tag_19_05_02
As the behavior is undefined, of course, I would like to propose to behave the most helpful way when it makes sense because
* the other usecases are already doable:
* if the expected behavior is "$*", so use "$*"
* what's the point of A < "$@" ? use "$1" instead
* it's a matter of consistency to me. why "$@" can't behave as "$@" ?
Again, thank you so much for such a precise and useful answer. regards.
Signature · 6 lines
Marc Chantreux Pôle CESAR (Calcul et services avancés à la recherche) Université de Strasbourg 14 rue René Descartes, BP 80010, 67084 STRASBOURG CEDEX 03.68.85.60.79
Hi,
Quoted 1 line · Marc Chantreux
* it's a matter of consistency to me. why "$@" can't behave as "$@" ?
The word that follows a redirection operator is one of the special contexts where operations that split words into multiple fields are disabled for it. Variable assignment is another. This is why
foo='multiple words'
bar=$foo
echo $foo > $bar
writes the string "multiple words" to a file named "multiple words". The second line does not run a "words" command (as 'bar=multiple words' would suggest), and the third line does not create a file called "multiple" (as 'echo multiple words > multiple words' would suggest).
It is this principle that dash follows when it expands "$@" as "$*" in these contexts. I agree with you that if users want that, they can just write "$*" instead, but I do think that when users do write "$@" anyway, treating it as "$*" is the least questionable interpretation.
Cheers,
Harald van Dijk
Quoted 10 lines · Harald van Dijk
Quoted 1 line · Marc Chantreux
* it's a matter of consistency to me. why "$@" can't behave as "$@" ?
The word that follows a redirection operator is one of the special contexts where operations that split words into multiple fields are disabled for it. Variable assignment is another. This is why
foo='multiple words'
bar=$foo
echo $foo > $bar
wow ... another thing I ignored after more than 25 years of shell scripting.
"case $foo" is another example I discovered recently (on this mailing list AFAIR). The person who answered me talked about a "scalar context".
Quoted 4 lines
It is this principle that dash follows when it expands "$@" as "$*" in these contexts. I agree with you that if users want that, they can just write "$*" instead, but I do think that when users do write "$@" anyway, treating it as "$*" is the least questionable interpretation.
Well ... I see a lot of people who just recomends to quote (and even brace) everything (so $foo becomes "${foo}" in their coding style) just because they fear expansions. I try to explain that expansions are good and word splitting is useful but you have to be carreful but the more we have exceptions to a simples rules, the more people will try to avoid errors and go on quoting, losing the benefits of expansions. They finally claim that "we should rewrite in python instead because shell behavior is hard to understand".
However un fully understand than when a behavior is still used, implementors try to make it consistent. Thank you so much for your explainations.
regards.
Signature · 6 lines
Marc Chantreux Pôle CESAR (Calcul et services avancés à la recherche) Université de Strasbourg 14 rue René Descartes, BP 80010, 67084 STRASBOURG CEDEX 03.68.85.60.79
Quoted 7 lines · Marc Chantreux
wow ... thank you so much for digging this. this is the second time this year where oppose POSIX standard against consistency and ease of use.
I can understand why because most of the scripts I learned with were postfixing the redirections so word splitting doesn't matter at this point
Sure it does. Redirections can be infixed, so these are all valid:
<file cmd arg
cmd <file arg
cmd arg <file
Someone who agrees with your argument should logically expect this to "work":
set file arg cmd < "$@"
Quoted 4 lines
Quoted 2 lines
Almost no shell works the way you are expecting, except for zsh in native mode (which I suspect is where you first tried this syntax)
Not at all.
Ah, well. I guessed so because I know from the zsh mailing lists that you're a big zsh fan.
Quoted 7 lines
As the behavior is undefined, of course, I would like to propose to behave the most helpful way when it makes sense because
* the other usecases are already doable:
* if the expected behavior is "$*", so use "$*"
* what's the point of A < "$@" ? use "$1" instead
* it's a matter of consistency to me. why "$@" can't behave as "$@" ?
One could easily make the same "consistency" argument from the perspective of the redirection operator. POSIX disallows this expansion from producing multiple fields...
var='foo bar'
cmd < $var
...so why should this one produce multiple fields?
set foo bar cmd < "$@"
In any case, the existing behavior conforms to POSIX and is far more portable, so it is not likely to change.
Signature · 1 line
vq
Hi Lawrence,
Quoted 3 lines · Lawrence Velázquez
Sure it does. Redirections can be infixed, so these are all valid:
<file cmd arg
my prefered one by far
Quoted 1 line
cmd arg <file
The one i kept using for years just because examples use it while isn't now my most hated by far. a good demonstration about reading the man pages even if you think you know the tool.
Quoted 7 lines
cmd <file arg
Someone who agrees with your argument should logically expect this to "work":
set file arg cmd < "$@"
I have to admit I forgot this one because I considered it confusing and useless but thanks to your example, I realize today is a convenient way to place the data source in "$1" instead of the top of stack and that could be useful in dash scripting because where is no simple/rubust way to get this top (the equivalent of $argv[-1] in zsh). But as "$@" in not expanded after <, this usecase is wasted as well.
fool() {
local source="$1"
echo "content of $source:"
cat < "$@"
}fool mind -n
content of mind:
sh: 4: cannot open mind -n: No such file
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Almost no shell works the way you are expecting, except for zsh in native mode (which I suspect is where you first tried this syntax)
Not at all.
Ah, well. I guessed so because I know from the zsh mailing lists that you're a big zsh fan.
I'm also fan of rc and long time user of mksh, I also used tcsh at the end of the last millenium and tested any other ones (including exotic ones like zoidberg and elvish).
I once (~ 10 years ago?) wrote an email to Tom Duff to thank him for rc ask him why there was no advanced variable expansions in rc (like ${foo#bar}, ${foo?oops} and so on). He answered me that nobody were using them back then. He also wrote something that has a deep impact of the way I think about shell scripting:
«rc means "run commands" and should not do much more»
from this day, I started to bet on external commands much more than shell internals and realized my scripts where faster and more reliable.
As I didn't need all the zsh features anyore (and rc is sadly too unknown to be used when you want to share your scripts), dash became my almost only shebang (also almost all my scripts are working in a bare busybox chroot, including CGIs).
I keep using zsh for interactive stuff because there is no competitor on this part of the game thanks to
* "alternate syntax" for control statements
* advanced features like extended globs, list filtering, filename modifiers
* incredible zle widgets
Quoted 12 lines
Quoted 7 lines
As the behavior is undefined, of course, I would like to propose to behave the most helpful way when it makes sense because
* the other usecases are already doable:
* if the expected behavior is "$*", so use "$*"
* what's the point of A < "$@" ? use "$1" instead
* it's a matter of consistency to me. why "$@" can't behave as "$@" ?One could easily make the same "consistency" argument from the perspective of the redirection operator. POSIX disallows this expansion from producing multiple fields...
Quoted 6 lines
var='foo bar'
cmd < $var...so why should this one produce multiple fields?
set foo bar cmd < "$@"
from the POSIX perspective, it indeed shouldn't.
Quoted 2 lines
In any case, the existing behavior conforms to POSIX and is far more portable, so it is not likely to change.
That's what I learnt from you (again: thank you so much) and made me so sad: consistency on POSIX compliance avoid consistency around useful behaviors. On the other hand, I totally support the idea that at least one shell must be POSIX compatible flaw by flaw.
there is no perfect shell in this world :)
regards
Signature · 6 lines
Marc Chantreux Pôle CESAR (Calcul et services avancés à la recherche) Université de Strasbourg 14 rue René Descartes, BP 80010, 67084 STRASBOURG CEDEX 03.68.85.60.79