The read builtin erroneously consumes null bytes
Hi,
Consider the following test case. It demonstrates NUL being consumed by the read builtin, rather than by cat(1) as it should have been.
$ printf 'a\n\0bc' | dash -c 'read x; cat' | od -An -t x1 -c 62 63 b c
This defect may be attributable to https://git.kernel.org/pub/scm/utils/dash/dash.git/tree/src/input.c?h=v0.5.13.4#n411.
Incidentally, it results in tests #5510, #5558 and #6020 failing for git v2.54.0.
Signature · 1 line
Kerin Millar
Quoted 5 lines · Kerin Millar
Consider the following test case. It demonstrates NUL being consumed by the read builtin, rather than by cat(1) as it should have been.
$ printf 'a\n\0bc' | dash -c 'read x; cat' | od -An -t x1 -c 62 63 b c
POSIX doesn't require a read _builtin_, but for the read _utility_, it says:
Quoted 10 lines
STDIN
If the -d delim option is not specified, or if it is specified and delim is not the null string, the standard input shall contain zero or more bytes (which need not form valid characters) and shall not contain any null bytes.
If the -d delim option is specified and delim is the null string, the standard input shall contain zero or more bytes (which need not form valid characters).
Signature · 6 lines
https://pubs.opengroup.org/onlinepubs/9799919799/utilities/read.html#tag_20_100 dash's `read` builtin doesn't have an `-d delim` option, and it seems to me that in such a context, it's reasonable to regard `-d delim` as "not specified", such that stdin isn't allowed to contain any null bytes. Alexis.
Quoted 9 lines · Alexis
Quoted 5 lines · Kerin Millar
Consider the following test case. It demonstrates NUL being consumed by the read builtin, rather than by cat(1) as it should have been.
$ printf 'a\n\0bc' | dash -c 'read x; cat' | od -An -t x1 -c 62 63 b cPOSIX doesn't require a
read_builtin_,
It is not possible to implement the read utility as an external utility because the read utility is required to modify shell variables, so effectively it is pretty much required to be a builtin.
Quoted 9 lines
but for the
read_utility_,
it says:Quoted 6 lines
STDIN
If the -d delim option is not specified, or if it is specified and delim is not the null string, the standard input shall contain zero or more bytes (which need not form valid characters) and shall not contain any null bytes.
The input to 'read' does not contain any null bytes. The input to 'read' is just 'a' followed by a newline and 'read' is supposed to stop reading after that.
Cheers,
Harald van Dijk
Am 20.06.2026 um 04:51 schrieb Harald van Dijk:
Quoted 7 lines · Alexis
Quoted 2 lines
POSIX doesn't require a
read_builtin_,It is not possible to implement the read utility as an external utility because the read utility is required to modify shell variables,
Offtopic, but what a nice memory: One of Ken Thompson's aha-moments (original: https://www.tuhs.org/pipermail/tuhs/2019-October/019167.html)
---- forwarded ----
i was writing the small utilities for the first
pdp-11 unix. (rm ls date ....)
so, cd was next.
% pwd/usr/ken
% cd /tmp
% pwd/usr/ken
Aha!
Quoted 11 lines · Kerin Millar
Hi,
Consider the following test case. It demonstrates NUL being consumed by the read builtin, rather than by cat(1) as it should have been.
$ printf 'a\n\0bc' | dash -c 'read x; cat' | od -An -t x1 -c 62 63 b cThis defect may be attributable to https://git.kernel.org/pub/scm/utils/dash/dash.git/tree/src/input.c?h=v0.5.13.4#n411.
Incidentally, it results in tests #5510, #5558 and #6020 failing for git v2.54.0.
Is this a regression? If so please bisect it to the commit that introduced it.
Thanks,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Quoted 9 lines · Herbert Xu
Quoted 8 lines · Kerin Millar
Hi,
Consider the following test case. It demonstrates NUL being consumed by the read builtin, rather than by cat(1) as it should have been.
$ printf 'a\n\0bc' | dash -c 'read x; cat' | od -An -t x1 -c 62 63 b c
Quoted 6 lines
Quoted 3 lines
This defect may be attributable to https://git.kernel.org/pub/scm/utils/dash/dash.git/tree/src/input.c?h=v0.5.13.4#n411.
Incidentally, it results in tests #5510, #5558 and #6020 failing for git v2.54.0.
Is this a regression? If so please bisect it to the commit that introduced it.
Aye, that it be and no mistake.
--
Kerin Millar
PATCH 1/1 input: Fix overeager NUL deletion in SMALL mode
Quoted 4 lines · Kerin Millar
Aye, that it be and no mistake.
OK this patch should fix the problem:
---8<---
NUL characters should not be removed from input lines that are yet to be processed because they could become the input to the next executed utility.
Fix this by moving the NUL deletion into pgetc when history support is off (IS_DEFINED_SMALL).
Also fold __pgetc into pgetc since the only other caller of it is preadbuffer and that logic can also be moved up.
Finally add a missing signed char cast for the unget characters.
src/input.c +37 -33
@@ -217,26 +217,6 @@ static void freestrings(struct strpush *sp) } -static int __pgetc (void ) -{ -int c; - -if (parsefile->unget) { -long unget = -(long )(unsigned )parsefile->unget--; - -return parsefile->nextc[unget]; - } - -if (parsefile->nleft >0 ) { - parsefile->nleft--; - c = (signed char )*parsefile->nextc++; - }else - c =preadbuffer (); - -return c; -} - -/* * Read a character from the script, returning PEOF on end of file. * Nul characters in the input are silently discarded.
@@ -245,11 +225,39 @@ static int __pgetc(void)int __attribute__((noinline))pgetc (void ) {struct strpush *sp = parsefile->spfree; +int c;if (unlikely (sp))freestrings (sp); -return __pgetc (); +again: +if (parsefile->unget) { +long unget = -(long )(unsigned )parsefile->unget--; + +return (signed char )parsefile->nextc[unget]; + } + +nextc: +if (likely (parsefile->nleft >0 )) { + parsefile->nleft--; + c = (signed char )*parsefile->nextc++; + }else if (unlikely (parsefile->strpush)) { +popstring (); +/* The freestrings call must be delayed til the next +* pgetc call for PEOA to work properly. +*/ + goto again; + }else + c =preadbuffer (); + +/* delete nul characters */ +if (IS_DEFINED_SMALL &&unlikely (!c)) { + parsefile->nextc =memmove (parsefile->nextc -1 , + parsefile->nextc, parsefile->nleft); + goto nextc; + } + +return c; }int pgetc_eoa (void )
@@ -374,10 +382,6 @@ static int preadbuffer(void)int more;char *q; -if (unlikely (parsefile->strpush)) { -popstring (); -return __pgetc (); - }if (parsefile->eof &2 ) { eof: parsefile->eof =3 ;
@@ -408,6 +412,12 @@ again: } } +if (IS_DEFINED_SMALL ) { + q += more; + more =0 ; + goto done; + } +/* delete nul characters */ for (;;) {int c;
@@ -422,9 +432,6 @@ again: q++; -if (IS_DEFINED_SMALL ) - goto check; -switch (c) {case '\n' : goto done;
@@ -439,11 +446,8 @@ again: } check: -if (more <=0 ) { -if (!IS_DEFINED_SMALL ) - goto again; -break ; - } +if (more <=0 ) + goto again; } done:input_set_lleft (parsefile, more);
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Re: [PATCH] input: Fix overeager NUL deletion in SMALL mode
Quoted 7 lines · Herbert Xu
Quoted 4 lines · Kerin Millar
Aye, that it be and no mistake.
OK this patch should fix the problem:
I can confirm that, with the patch applied, dash is able to pass the git test suite, along with a slew of tests conducted by myself.
--
Kerin Millar