Request for Help - Too many perl arguments as of 2.43.0-rc0
I can no longer test git as the number of arguments (and size) sent to perl from the make infrastructure is too big for my platform. Is it possible to use xargs instead?
Thanks,
Randall
--
Brief whoami: NonStop&UNIX developer since approximately UNIX(421664400)
NonStop(211288444200000000)
-- In real life, I talk too much.
Quoted 3 lines · On Fri, Nov 3, 2023 at 3:03 PM <[email protected]> wrote:
I can no longer test git as the number of arguments (and size) sent to perl from the make infrastructure is too big for my platform. Is it possible to use xargs instead?
Presumably you're talking about the invocation of chainlint.pl? Considering that that's quite an unimportant part of make test for those just running the test suite (as opposed to those authoring new tests), the simplest workaround is to disable chainlint when you run the tests by setting environment variable GIT_TEST_EXT_CHAIN_LINT to 0.
Quoted 10 lines · Eric Sunshine
Quoted 3 lines · On Fri, Nov 3, 2023 at 3:03 PM <[email protected]> wrote:
I can no longer test git as the number of arguments (and size) sent to perl from the make infrastructure is too big for my platform. Is it possible to use xargs instead?
Presumably you're talking about the invocation of chainlint.pl? Considering that that's quite an unimportant part of
make testfor those just running the test suite (as opposed to those authoring new tests), the simplest workaround is to disable chainlint when you run the tests by setting environment variableGIT_TEST_EXT_CHAIN_LINTto 0.
I tried both ways to specify this variable with no success:
/home/randall/git: make -C t/ all SHELL=/usr/coreutils/bin/bash GIT_TEST_EXT_CHAIN_LINT=0
make: Entering directory '/home/randall/git/t'
rm -f -r 'test-results'/usr/coreutils/bin/bash: /usr/bin/perl: Argument list too long Makefile:125: recipe for target 'test-lint-shell-syntax' failed
make: *** [test-lint-shell-syntax] Error 126
make: Leaving directory '/home/randall/git/t'/home/randall/git: GIT_TEST_EXT_CHAIN_LINT=0 make -C t/ all SHELL=/usr/coreutils/bin/bash
make: Entering directory '/home/randall/git/t'
rm -f -r 'test-results'/usr/coreutils/bin/bash: /usr/bin/perl: Argument list too long Makefile:125: recipe for target 'test-lint-shell-syntax' failed
make: *** [test-lint-shell-syntax] Error 126
make: Leaving directory '/home/randall/git/t'
Quoted 12 lines · On Fri, Nov 3, 2023 at 5:31 PM <[email protected]> wrote:
Quoted 10 lines · Eric Sunshine
Quoted 3 lines · On Fri, Nov 3, 2023 at 3:03 PM <[email protected]> wrote:
I can no longer test git as the number of arguments (and size) sent to perl from the make infrastructure is too big for my platform. Is it possible to use xargs instead?
Presumably you're talking about the invocation of chainlint.pl? Considering that that's quite an unimportant part of
make testfor those just running the test suite (as opposed to those authoring new tests), the simplest workaround is to disable chainlint when you run the tests by setting environment variableGIT_TEST_EXT_CHAIN_LINTto 0.
Quoted 4 lines
I tried both ways to specify this variable with no success:
/usr/coreutils/bin/bash:/usr/bin/perl: Argument list too long Makefile:125: recipe for target 'test-lint-shell-syntax' failed
This is a separate issue from chainlint (though, it too will likely suffer the same problem). In this case, it's trying to run the check-non-portable-shell.pl script and the command-line is too long. Unfortunately, unlike chainlint for which there is a knob to disble it, there is no direct knob to disable test-lint-shell-syntax. You may be able to skip all the shell-script linting by setting TEST_LINT to an empty value.
Quoted 18 lines · Eric Sunshine
Quoted 12 lines · On Fri, Nov 3, 2023 at 5:31 PM <[email protected]> wrote:
Quoted 10 lines · Eric Sunshine
Quoted 3 lines · On Fri, Nov 3, 2023 at 3:03 PM <[email protected]> wrote:
I can no longer test git as the number of arguments (and size) sent to perl from the make infrastructure is too big for my platform. Is it possible to use xargs instead?
Presumably you're talking about the invocation of chainlint.pl? Considering that that's quite an unimportant part of
make testfor those just running the test suite (as opposed to those authoring new tests), the simplest workaround is to disable chainlint when you run the tests by setting environment variableGIT_TEST_EXT_CHAIN_LINTto 0.Quoted 4 lines
I tried both ways to specify this variable with no success:
/usr/coreutils/bin/bash:/usr/bin/perl: Argument list too long Makefile:125: recipe for target 'test-lint-shell-syntax' failed
Quoted 6 lines
This is a separate issue from chainlint (though, it too will likely suffer the same problem). In this case, it's trying to run the check-non-portable-shell.pl script and the command-line is too long.
Unfortunately, unlike chainlint for which there is a knob to disble it, there is no direct knob to disabletest-lint-shell-syntax. You may be able to skip all the shell-script linting by settingTEST_LINTto an empty value.
That worked. I wonder whether it might be useful to set up a knob for this.
Quoted 5 lines · On Fri, Nov 3, 2023 at 6:04 PM <[email protected]> wrote:
Quoted 4 lines · Eric Sunshine
Quoted 2 lines · On Fri, Nov 3, 2023 at 5:31 PM <[email protected]> wrote:
/usr/coreutils/bin/bash:/usr/bin/perl: Argument list too long Makefile:125: recipe for target 'test-lint-shell-syntax' failed
Quoted 8 lines
Quoted 6 lines
This is a separate issue from chainlint (though, it too will likely suffer the same problem). In this case, it's trying to run the check-non-portable-shell.pl script and the command-line is too long.
Unfortunately, unlike chainlint for which there is a knob to disble it, there is no direct knob to disabletest-lint-shell-syntax. You may be able to skip all the shell-script linting by settingTEST_LINTto an empty value.That worked. I wonder whether it might be useful to set up a knob for this.
I don't see an urgent need for it. Unlike the actual tests themselves run by make test which may catch platform-specific problems in Git itself, the purpose of the "linting" checks is not to catch platform-specific problems, but rather to help test authors by identifying mistakes in the tests which might make them fragile. So, disabling linting on a particular platform isn't going to cause make test to miss some important Git problem specific to that platform.
Also, in the longer term, as you suggested, xargs is likely a more fruitful solution.
Quoted 2 lines · Eric Sunshine
Also, in the longer term, as you suggested,
xargsis likely a more fruitful solution.
Hmph, the list of our test scripts exceed command line limit? That sounds a bit nasty, as we somehow need to prepare a pipe and feed them into it, in order to drive xargs downstream of the pipe.
Ideally if there were a GNUMake function that slices a list into sublists of "reasonable" lengths, we could use it to directly drive N invocations of check-non-portable-shell script instead of xargs, but I didn't find one. Here is I came up with, using foreach that is "slice the list into many sublists of 1 element", but it made me feel dirty.
t/Makefile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
t/Makefile +2 -1
@@ -122,7 +122,8 @@ test-lint-executable: echo >&2 "non-executable tests:" $$bad; exit 1; } test-lint-shell-syntax: - @'$(PERL_PATH_SQ)' check-non-portable-shell.pl $(T) $(THELPERS) $(TPERF) + @{ $(foreach t,$(T) $(THELPERS) $(TPERF),echo "$t";) } | \ + xargs '$(PERL_PATH_SQ)' check-non-portable-shell.pl test-lint-filenames: @# We do *not* pass a glob to ls-files but use grep instead, to catch
Quoted 8 lines · Junio C Hamano
Quoted 2 lines · Eric Sunshine
Also, in the longer term, as you suggested,
xargsis likely a more fruitful solution.Hmph, the list of our test scripts exceed command line limit? That sounds a bit nasty, as we somehow need to prepare a pipe and feed them into it, in order to drive xargs downstream of the pipe.
Quoted 6 lines
Ideally if there were a GNUMake function that slices a list into sublists of "reasonable" lengths, we could use it to directly drive N invocations of check-non-portable-shell script instead of xargs, but I didn't find one. Here is I came up with, using foreach that is "slice the list into many sublists of 1 element", but it made me feel dirty.
Indeed, that's ugly. I hadn't even put any thought into it since there doesn't seem to be a pressing need for it.
In the long run, Ævar's idea of having make notice which, if any, test scripts have changed, and only perform linting on an as-needed basis may be the way to go[1,2,3], thus only passing a single script as argument to each of the linters.
[1]: https://lore.kernel.org/git/[email protected]/
[2]: https://lore.kernel.org/git/[email protected]/
[3]: https://github.com/avar/git/commits/avar/t-Makefile-break-T-to-file-association
Quoted 7 lines · Eric Sunshine
I don't see an urgent need for it. Unlike the actual tests themselves run by
make testwhich may catch platform-specific problems in Git itself, the purpose of the "linting" checks is not to catch platform-specific problems, but rather to help test authors by identifying mistakes in the tests which might make them fragile. So, disabling linting on a particular platform isn't going to causemake testto miss some important Git problem specific to that platform.
Hmm. With compilation, we split the audience of "developers" vs "people who just want to build the program", and we crank up the number and severity of warning checks for the former. We could do the same here for tests. I.e., turn off test linting by default and re-enable it for DEVELOPER=1.
OTOH, this is the first time I think I've seen the linting cause a problem (whereas unexpected compile warnings are much more likely, as we are depending on the system compiler's behavior).
So consider it an idle thought for discussion, and not necessarily a proposal. ;)
-Peff
Quoted 15 lines · Jeff King
Quoted 7 lines · Eric Sunshine
I don't see an urgent need for it. Unlike the actual tests themselves run by
make testwhich may catch platform-specific problems in Git itself, the purpose of the "linting" checks is not to catch platform-specific problems, but rather to help test authors by identifying mistakes in the tests which might make them fragile. So, disabling linting on a particular platform isn't going to causemake testto miss some important Git problem specific to that platform.Hmm. With compilation, we split the audience of "developers" vs "people who just want to build the program", and we crank up the number and severity of warning checks for the former. We could do the same here for tests. I.e., turn off test linting by default and re-enable it for DEVELOPER=1.
Quoted 5 lines
OTOH, this is the first time I think I've seen the linting cause a problem (whereas unexpected compile warnings are much more likely, as we are depending on the system compiler's behavior).
So consider it an idle thought for discussion, and not necessarily a proposal. ;)
In my case, I am building git for users of the platform (a.k.a. mostly a packager), although I'm hoping to be more involved in contributions soon. This involves build + full test under multiple situations on the NonStop platform. --Randall
Quoted 9 lines · Jeff King
Quoted 7 lines · Eric Sunshine
I don't see an urgent need for it. Unlike the actual tests themselves run by
make testwhich may catch platform-specific problems in Git itself, the purpose of the "linting" checks is not to catch platform-specific problems, but rather to help test authors by identifying mistakes in the tests which might make them fragile. So, disabling linting on a particular platform isn't going to causemake testto miss some important Git problem specific to that platform.
Quoted 5 lines
Hmm. With compilation, we split the audience of "developers" vs "people who just want to build the program", and we crank up the number and severity of warning checks for the former. We could do the same here for tests. I.e., turn off test linting by default and re-enable it for DEVELOPER=1.
My knee-jerk reaction is that this would move us in the wrong direction since it is probable that most drive-by contributors won't have DEVELOPER=1 set, yet they are the ones who are likely to benefit most from test script linting (which is not to say that it doesn't help seasoned contributors, as well).
Quoted 3 lines
OTOH, this is the first time I think I've seen the linting cause a problem (whereas unexpected compile warnings are much more likely, as we are depending on the system compiler's behavior).
There have been a few other times when the linting scripts have needed a tweak or two to work properly on some less-well-represented platform, but they were minor issues[1,2,3,4].
Quoted 2 lines
So consider it an idle thought for discussion, and not necessarily a proposal. ;)
As noted in my response to Junio[5], in the long run, we may want to go with Ævar's idea of having make track changes to the test scripts, thus only run linting on an as-needed basis.
[1]: a3c4c8841c (tests: use shorter labels in chainlint.sed for AIX
sed, 2018-08-24)
[2]: 2d9ded8acc (tests: fix comment syntax in chainlint.sed for AIX
sed, 2018-08-24)
[3]: b3b753b104 (Fit to Plan 9's ANSI/POSIX compatibility layer, 2020-09-10)
[4]: 1f51b77f4f (chainlint.pl: fix /proc/cpuinfo regexp, 2022-11-22)
[5]: https://lore.kernel.org/git/CAPig+cSC8m5a8PhMw_eJbswwNB-VgBt+n56HSTLLabV9_+y--g@mail.gmail.com/
Quoted 11 lines · Eric Sunshine
Quoted 5 lines
Hmm. With compilation, we split the audience of "developers" vs "people who just want to build the program", and we crank up the number and severity of warning checks for the former. We could do the same here for tests. I.e., turn off test linting by default and re-enable it for DEVELOPER=1.
My knee-jerk reaction is that this would move us in the wrong direction since it is probable that most drive-by contributors won't have DEVELOPER=1 set, yet they are the ones who are likely to benefit most from test script linting (which is not to say that it doesn't help seasoned contributors, as well).
Yeah, that's a good point. If the linting is not causing frequent headaches (and I don't think it is), then we are better to leave it on by default.
-Peff