forking and threading
cover forking and threading
Forking and threading is a difficult thing to get right due to potential deadlocks which can occur if one thread holds a lock while another forks. The resulting process will still have the lock in a locked state with no hope of it ever being released (since forking doesn't replicate the threads as well).
The aim of this series is to push the allocation done between fork/exec to before the call to 'fork()' so that calls to malloc can't deadlock in a forked child process.
Most standard implementations of malloc (e.g. glibc) do the appropriate thing and register fork handlers in order ensure that locks are in a usable state after forking. Unfortunately other implementations don't do this so to account for this lets just avoid calls to functions which may require locking.
As far as I understand the only instance of threading and forking which exists in the current code base is 'git grep --recurse-submodules', and the standard builds against glibc shouldn't exhibit any of this deadlocking.
Brandon Williams (5):
run-command: convert sane_execvp to sane_execvpe
run-command: prepare argv before forking
run-command: allocate child_err before forking
run-command: prepare child environment before forking
run-command: add note about forking and threading
cache.h | 3 +- exec_cmd.c | 2 +- run-command.c | 151 +++++++++++++++++++++++++++++++++++++++++++++------------- 3 files changed, 119 insertions(+), 37 deletions(-)
Signature · 1 line
2.12.2.715.g7642488e1d-goog
PATCH 1/5 run-command: convert sane_execvp to sane_execvpe
Convert 'sane_execvp()' to 'sane_execvpe()' which optionally takes a pointer to an array of 'char *' which should be used as the environment for the process being exec'd. If no environment is provided (by passing NULL instead) then the already existing environment, as stored in 'environ', will be used.
--- cache.h | 3 +-- exec_cmd.c | 2 +- run-command.c | 15 ++++++++++----- 3 files changed, 12 insertions(+), 8 deletions(-)
cache.h +1 -2
@@ -2185,8 +2185,7 @@ int checkout_fast_forward(const unsigned char *from,const unsigned char *to, int overwrite_ignore); - -int sane_execvp (const char *file,char *const argv[]); +int sane_execvpe (const char *file,char *const argv[],char *const envp[]);/* * A struct to encapsulate the concept of whether a file has changed
exec_cmd.c +1 -1
@@ -118,7 +118,7 @@ int execv_git_cmd(const char **argv) {trace_argv_printf (nargv.argv,"trace: exec:" );/* execvp() can only ever return if it fails */ -sane_execvp ("git" , (char **)nargv.argv); +sane_execvpe ("git" , (char **)nargv.argv,NULL );trace_printf ("trace: exec failed: %s\n" ,strerror (errno));
run-command.c +10 -5
@@ -168,10 +168,15 @@ static int exists_in_PATH(const char *file)return r !=NULL ; } -int sane_execvp (const char *file,char *const argv[]) +int sane_execvpe (const char *file,char *const argv[],char *const envp[]) { -if (!execvp (file, argv)) -return 0 ;/* cannot happen ;-) */ +if (envp) { +if (!execvpe (file, argv, envp)) +return 0 ;/* cannot happen ;-) */ + }else { +if (!execvp (file, argv)) +return 0 ;/* cannot happen ;-) */ + }/* * When a command can't be found because one of the directories
@@ -226,7 +231,7 @@ static int execv_shell_cmd(const char **argv)struct argv_array nargv =ARGV_ARRAY_INIT ;prepare_shell_cmd (&nargv, argv);trace_argv_printf (nargv.argv,"trace: exec:" ); -sane_execvp (nargv.argv[0 ], (char **)nargv.argv); +sane_execvpe (nargv.argv[0 ], (char **)nargv.argv,NULL );argv_array_clear (&nargv);return -1 ; }
@@ -442,7 +447,7 @@ int start_command(struct child_process *cmd)else if (cmd->use_shell)execv_shell_cmd (cmd->argv);else -sane_execvp (cmd->argv[0 ], (char *const *) cmd->argv); +sane_execvpe (cmd->argv[0 ], (char *const *) cmd->argv,NULL );if (errno ==ENOENT ) {if (!cmd->silent_exec_failure) error("cannot run %s: %s", cmd->argv[0],
Signature · 1 line
2.12.2.715.g7642488e1d-goog
Re: [PATCH 1/5] run-command: convert sane_execvp to sane_execvpe
Quoted 5 lines · Brandon Williams
Convert '
sane_execvp()' to 'sane_execvpe()' which optionally takes a pointer to an array of 'char *' which should be used as the environment for the process being exec'd. If no environment is provided (by passing NULL instead) then the already existing environment, as stored in 'environ', will be used.
Turns out we probably can't use execvpe since it isn't portable. From some of the other discussion it makes more sense to just move to using execve instead.
Quoted 79 lines
Signed-off-by Brandon Williams <[email protected]>--- cache.h | 3 +-- exec_cmd.c | 2 +- run-command.c | 15 ++++++++++----- 3 files changed, 12 insertions(+), 8 deletions(-)cache.h +1
-2@@ -2185,8 +2185,7 @@ int checkout_fast_forward(const unsigned char *from,const unsigned char *to, int overwrite_ignore); - -int sane_execvp (const char *file,char *const argv[]); +int sane_execvpe (const char *file,char *const argv[],char *const envp[]);/* * A struct to encapsulate the concept of whether a file has changed exec_cmd.c +1
-1@@ -118,7 +118,7 @@ int execv_git_cmd(const char **argv) {trace_argv_printf (nargv.argv,"trace: exec:" );/* execvp() can only ever return if it fails */ -sane_execvp ("git" , (char **)nargv.argv); +sane_execvpe ("git" , (char **)nargv.argv,NULL );trace_printf ("trace: exec failed: %s\n" ,strerror (errno));run-command.c +10
-5@@ -168,10 +168,15 @@ static int exists_in_PATH(const char *file)return r !=NULL ; } -int sane_execvp (const char *file,char *const argv[]) +int sane_execvpe (const char *file,char *const argv[],char *const envp[]) { -if (!execvp (file, argv)) -return 0 ;/* cannot happen ;-) */ +if (envp) { +if (!execvpe (file, argv, envp)) +return 0 ;/* cannot happen ;-) */ + }else { +if (!execvp (file, argv)) +return 0 ;/* cannot happen ;-) */ + }/* * When a command can't be found because one of the directories @@ -226,7 +231,7 @@ static int execv_shell_cmd(const char **argv)struct argv_array nargv =ARGV_ARRAY_INIT ;prepare_shell_cmd (&nargv, argv);trace_argv_printf (nargv.argv,"trace: exec:" ); -sane_execvp (nargv.argv[0 ], (char **)nargv.argv); +sane_execvpe (nargv.argv[0 ], (char **)nargv.argv,NULL );argv_array_clear (&nargv);return -1 ; }@@ -442,7 +447,7 @@ int start_command(struct child_process *cmd)else if (cmd->use_shell)execv_shell_cmd (cmd->argv);else -sane_execvp (cmd->argv[0 ], (char *const *) cmd->argv); +sane_execvpe (cmd->argv[0 ], (char *const *) cmd->argv,NULL );if (errno ==ENOENT ) {if (!cmd->silent_exec_failure) error("cannot run %s: %s", cmd->argv[0],Signature · 1 line
2.12.2.715.g7642488e1d-goog
Signature · 1 line
Brandon Williams
PATCH 2/5 run-command: prepare argv before forking
In reply to Brandon WilliamsIn order to avoid allocation between 'fork()' and 'exec()' the argv array used in the exec call is prepared prior to forking the process.
--- run-command.c | 34 ++++++++++++++++------------------ 1 file changed, 16 insertions(+), 18 deletions(-)
run-command.c +16 -18
@@ -226,18 +226,6 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv) }#ifndef GIT_WINDOWS_NATIVE -static int execv_shell_cmd (const char **argv) -{ -struct argv_array nargv =ARGV_ARRAY_INIT ; -prepare_shell_cmd (&nargv, argv); -trace_argv_printf (nargv.argv,"trace: exec:" ); -sane_execvpe (nargv.argv[0 ], (char **)nargv.argv,NULL ); -argv_array_clear (&nargv); -return -1 ; -} -#endif - -#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ;static void notify_parent (void )
@@ -377,9 +365,20 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +struct argv_array argv =ARGV_ARRAY_INIT ; +if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; +if (cmd->git_cmd) { +argv_array_push (&argv,"git" ); +argv_array_pushv (&argv, cmd->argv); + }else if (cmd->use_shell) { +prepare_shell_cmd (&argv, cmd->argv); + }else { +argv_array_pushv (&argv, cmd->argv); + } + cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) {
@@ -442,12 +441,9 @@ int start_command(struct child_process *cmd)unsetenv (*cmd->env); } } -if (cmd->git_cmd) -execv_git_cmd (cmd->argv); -else if (cmd->use_shell) -execv_shell_cmd (cmd->argv); -else -sane_execvpe (cmd->argv[0 ], (char *const *) cmd->argv,NULL ); + +sane_execvpe (argv.argv[0 ], (char *const *) argv.argv,NULL ); +if (errno ==ENOENT ) {if (!cmd->silent_exec_failure) error("cannot run %s: %s", cmd->argv[0],
@@ -480,6 +476,8 @@ int start_command(struct child_process *cmd) cmd->pid = -1; }close (notify_pipe[0 ]); + +argv_array_clear (&argv); }#else {
Signature · 1 line
2.12.2.715.g7642488e1d-goog
PATCH 3/5 run-command: allocate child_err before forking
In reply to Brandon WilliamsIn order to avoid allocation between 'fork()' and 'exec()' open the stream used for the child's error handeling prior to forking.
--- run-command.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
run-command.c +11 -5
@@ -365,11 +365,18 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +FILE *child_err =NULL ;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; +if (cmd->no_stderr || need_err) { +int child_err_fd =dup (2 ); +set_cloexec (child_err_fd); + child_err =fdopen (child_err_fd,"w" ); + } +if (cmd->git_cmd) {argv_array_push (&argv,"git" );argv_array_pushv (&argv, cmd->argv);
@@ -387,11 +394,8 @@ int start_command(struct child_process *cmd)* before redirecting the process's stderr so that all die() * in subsequent call paths use the parent's stderr. */ -if (cmd->no_stderr || need_err) { -int child_err =dup (2 ); -set_cloexec (child_err); -set_error_handle (fdopen (child_err,"w" )); - } +if (child_err) +set_error_handle (child_err);close (notify_pipe[0 ]);set_cloexec (notify_pipe[1 ]);
@@ -477,6 +481,8 @@ int start_command(struct child_process *cmd) }close (notify_pipe[0 ]); +if (child_err) +fclose (child_err);argv_array_clear (&argv); }#else
Signature · 1 line
2.12.2.715.g7642488e1d-goog
PATCH 4/5 run-command: prepare child environment before forking
In reply to Brandon WilliamsIn order to avoid allocation between 'fork()' and 'exec()' prepare the environment to be used in the child process prior to forking.
--- run-command.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 75 insertions(+), 9 deletions(-)
run-command.c +75 -9
@@ -291,6 +291,75 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal)return code; } +static int env_isequal (const char *e1,const char *e2) +{ +for (;;) { +char c1 = *e1++; +char c2 = *e2++; + c1 = (c1 =='=' ) ?'\0' :tolower (c1); + c2 = (c2 =='=' ) ?'\0' :tolower (c2); + +if (c1 != c2) +return 0 ; +if (c1 =='\0' ) +return 1 ; + } +} + +static int searchenv (char **env,const char *name) +{ +int pos =0 ; + +for (; env[pos]; pos++) +if (env_isequal (env[pos], name)) +break ; + +return pos; +} + +static int do_putenv (char **env,int env_nr,const char *name) +{ +int pos =searchenv (env, name); + +if (strchr (name,'=' )) { +/* ('key=value'), insert of replace entry */ +if (pos >= env_nr) + env_nr++; + env[pos] = (char *) name; + }else if (pos < env_nr) { +/* otherwise ('key') remove existing entry */ + env_nr--; +memmove (&env[pos], &env[pos +1 ], + (env_nr - pos) *sizeof (char *)); + env[env_nr] =NULL ; + } + +return env_nr; +} + +static char **prep_childenv (const char *const *deltaenv) +{ +char **childenv; +int childenv_nr =0 , childenv_alloc =0 ; +int i; + +for (i =0 ; environ[i]; i++) + childenv_nr++; +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_alloc++; +/* Add one for the NULL termination */ + childenv_alloc += childenv_nr +1 ; + + childenv =xcalloc (childenv_alloc,sizeof (char *)); +memcpy (childenv, environ, childenv_nr *sizeof (char *)); + +/* merge in deltaenv */ +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_nr =do_putenv (childenv, childenv_nr, deltaenv[i]); + +return childenv; +} +int start_command (struct child_process *cmd) {int need_in, need_out, need_err;
@@ -365,12 +434,15 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +char **childenv;FILE *child_err =NULL ;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; + childenv =prep_childenv (cmd->env); +if (cmd->no_stderr || need_err) {int child_err_fd =dup (2 );set_cloexec (child_err_fd);
@@ -437,16 +509,9 @@ int start_command(struct child_process *cmd)if (cmd->dir &&chdir (cmd->dir))die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir); -if (cmd->env) { -for (; *cmd->env; cmd->env++) { -if (strchr (*cmd->env,'=' )) -putenv ((char *)*cmd->env); -else -unsetenv (*cmd->env); - } - } -sane_execvpe (argv.argv[0 ], (char *const *) argv.argv,NULL ); +sane_execvpe (argv.argv[0 ], (char *const *) argv.argv, + (char *const *) childenv);if (errno ==ENOENT ) {if (!cmd->silent_exec_failure)
@@ -483,6 +548,7 @@ int start_command(struct child_process *cmd)if (child_err)fclose (child_err); +free (childenv); argv_array_clear(&argv); }#else
Signature · 1 line
2.12.2.715.g7642488e1d-goog
Re: [PATCH 4/5] run-command: prepare child environment before forking
Quoted 2 lines · Brandon Williams
In order to avoid allocation between '
fork()' and 'exec()' prepare the environment to be used in the child process prior to forking.
If using something like posix_spawn(), this would be needed anyway, so I'll review it.
[...]
Quoted 1 line
+++
b/run-command.c
[...]
Quoted 12 lines
run-command.c +12
-0+static char **prep_childenv (const char *const *deltaenv) +{ +char **childenv; +int childenv_nr =0 , childenv_alloc =0 ; +int i; + +for (i =0 ; environ[i]; i++) + childenv_nr++; +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_alloc++; +/* Add one for the NULL termination */ + childenv_alloc += childenv_nr +1 ;
run-command.c static int wait_or_whine(pid_t pid, const char *argv0, int in_signal) · line 361
Quoted 10 lines
run-command.c +10
-0+ + childenv =xcalloc (childenv_alloc,sizeof (char *)); +memcpy (childenv, environ, childenv_nr *sizeof (char *)); + +/* merge in deltaenv */ +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_nr =do_putenv (childenv, childenv_nr, deltaenv[i]); + +return childenv; +}
This potentially copies around most of 'environ' several times as it adjusts for each deltaenv item. Can it be simplified? E.g.
struct argv_array result = ARGV_ARRAY_INIT;
struct string_list mods = STRING_LIST_INIT_DUP;
struct strbuf key = STRBUF_INIT;
const char **p;
for (p = cmd_env; *p; p++) {
const char *equals = strchr(*p, '=');
if (equals) {
strbuf_reset(&key);
strbuf_add(&key, *p, equals - *p);
string_list_append(&mods, key.buf)->util = *p;
} else {
string_list_append(&mods, *p);
}
}
string_list_sort(&mods);
for (p = environ; *p; p++) {
struct string_list_item *item;
const char *equals = strchr(*p, '=');
if (!equals)
continue;
strbuf_reset(&key);
strbuf_add(&key, *p, equals - *p);
item = string_list_lookup(&mods, key.buf);
if (!item) /* no change */
argv_array_push(&result, *p);
else if (!item->util) /* unsetenv */
; /* skip */
else /* setenv */
argv_array_push(&result, item->util);
}
strbuf_release(&key);
string_list_clear(&mods);
return argv_array_detach(&result);If the string_list API provided a lookup function taking a buffer and length as argument, the environ loop could be simplified further to use *p instead of a copy.
Thanks and hope that helps,
Jonathan
Re: [PATCH 4/5] run-command: prepare child environment before forking
Quoted 28 lines · Jonathan Nieder
Quoted 2 lines · Brandon Williams
In order to avoid allocation between '
fork()' and 'exec()' prepare the environment to be used in the child process prior to forking.If using something like
posix_spawn(), this would be needed anyway, so I'll review it.[...]
Quoted 1 line
+++
b/run-command.c[...]
Quoted 17 lines
run-command.c +17
-0+static char **prep_childenv (const char *const *deltaenv) +{ +char **childenv; +int childenv_nr =0 , childenv_alloc =0 ; +int i; + +for (i =0 ; environ[i]; i++) + childenv_nr++; +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_alloc++; +/* Add one for the NULL termination */ + childenv_alloc += childenv_nr +1 ; + + childenv =xcalloc (childenv_alloc,sizeof (char *)); +memcpy (childenv, environ, childenv_nr *sizeof (char *)); + +/* merge in deltaenv */
Quoted 8 lines
run-command.c static int wait_or_whine(pid_t pid, const char *argv0, int in_signal) · line 361
Quoted 5 lines
run-command.c +5
-0+for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_nr =do_putenv (childenv, childenv_nr, deltaenv[i]); + +return childenv; +}This potentially copies around most of 'environ' several times as it adjusts for each deltaenv item. Can it be simplified? E.g.
The only time it copies anything is that first memcpy which makes a duplicate of the environ, and then when an item is being removed in do_putenv it will move the back of the array to fill in for the entry being removed.
Quoted 30 lines
struct argv_array result = ARGV_ARRAY_INIT;
struct string_list mods = STRING_LIST_INIT_DUP;
struct strbuf key = STRBUF_INIT;
const char **p;
for (p = cmd_env; *p; p++) {
const char *equals = strchr(*p, '=');
if (equals) {
strbuf_reset(&key);
strbuf_add(&key, *p, equals - *p);
string_list_append(&mods, key.buf)->util = *p;
} else {
string_list_append(&mods, *p);
}
}
string_list_sort(&mods);
for (p = environ; *p; p++) {
struct string_list_item *item;
const char *equals = strchr(*p, '=');
if (!equals)
continue;
strbuf_reset(&key);
strbuf_add(&key, *p, equals - *p);
item = string_list_lookup(&mods, key.buf);
if (!item) /* no change */
argv_array_push(&result, *p);
else if (!item->util) /* unsetenv */Quoted 8 lines
; /* skip */ else /* setenv */ argv_array_push(&result, item->util); } strbuf_release(&key); string_list_clear(&mods); return argv_array_detach(&result);
This is probably still incomplete as I don't see how this accounts for entries in 'cmd_env' which are being added to the environment and not just replacing existing ones.
Signature · 1 line
Brandon Williams
Re: [PATCH 4/5] run-command: prepare child environment before forking
Quoted 36 lines · Brandon Williams
Quoted 34 lines · Jonathan Nieder
struct argv_array result = ARGV_ARRAY_INIT; struct string_list mods = STRING_LIST_INIT_DUP; struct strbuf key = STRBUF_INIT; const char **p; for (p = cmd_env; *p; p++) { const char *equals = strchr(*p, '='); if (equals) { strbuf_reset(&key); strbuf_add(&key, *p, equals - *p); string_list_append(&mods, key.buf)->util = *p; } else { string_list_append(&mods, *p); } } string_list_sort(&mods); for (p = environ; *p; p++) { struct string_list_item *item; const char *equals = strchr(*p, '='); if (!equals) continue; strbuf_reset(&key); strbuf_add(&key, *p, equals - *p); item = string_list_lookup(&mods, key.buf); if (!item) /* no change */ argv_array_push(&result, *p); else if (!item->util) /* unsetenv */ ; /* skip */ else /* setenv */ argv_array_push(&result, item->util); }
Quoted 7 lines
Quoted 3 lines
strbuf_release(&key); string_list_clear(&mods); return argv_array_detach(&result);This is probably still incomplete as I don't see how this accounts for entries in '
cmd_env' which are being added to the environment and not just replacing existing ones.
Yes, that's true. This sample code is incomplete since it doesn't handle those.
Jonathan
PATCH 5/5 run-command: add note about forking and threading
In reply to Brandon WilliamsAllocation was pushed before forking in order to avoid potential deadlocking when forking while multiple threads are running. This deadlocking is possible when a thread (other than the one forking) has acquired a lock and didn't get around to releasing it before the fork. This leaves the lock in a locked state in the resulting process with no hope of it ever being released.
--- run-command.c | 8 ++++++++ 1 file changed, 8 insertions(+)
run-command.c +8 -0
@@ -458,6 +458,14 @@ int start_command(struct child_process *cmd)argv_array_pushv (&argv, cmd->argv); } +/* +* NOTE: In order to prevent deadlocking when using threads special +* care should be taken with the function calls made in between the +* fork() and exec() calls. No calls should be made to functions which +* require acquiring a lock (e.g. malloc) as the lock could have been +* held by another thread at the time of forking, causing the lock to +* never be released in the child process. +*/ cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) {
Signature · 1 line
2.12.2.715.g7642488e1d-goog
Re: [PATCH 5/5] run-command: add note about forking and threading
Hi,
Quoted 5 lines · Brandon Williams
run-command.c +0
-0@@ -458,6 +458,14 @@ int start_command(struct child_process *cmd)argv_array_pushv (&argv, cmd->argv); }
run-command.c int start_command(struct child_process *cmd) · line 469
Quoted 10 lines
run-command.c +8
-0+/* +* NOTE: In order to prevent deadlocking when using threads special +* care should be taken with the function calls made in between the +* fork() and exec() calls. No calls should be made to functions which +* require acquiring a lock (e.g. malloc) as the lock could have been +* held by another thread at the time of forking, causing the lock to +* never be released in the child process. +*/ cmd->pid =fork ();
Why can't git use e.g. posix_spawn to avoid this?
fork()-ing in a threaded context is very painful for maintainability. Any library function you are using could start taking a lock, and then you have a deadlock. So you have to make use of a very small whitelisted list of library functions for this to work.
The function calls you have to audit are not only between fork() and exec() in the normal control flow. You have to worry about signal handlers, too.
By comparison, posix_spawn() takes care of this all. I really strongly do not want to move in the direction of more fork() in a multi-threaded context. I'd rather turn off threads in the submodule case of grep altogether, but that shouldn't be needed:
* it is possible to launch a command without fork+exec
* if that's not suitable, then using fork introduces parallelism that interacts much better with fork+exec than threads do
I really don't want to go down this path. I've said that a few times. I'm saying it again now.
My two cents,
Jonathan
Re: [PATCH 5/5] run-command: add note about forking and threading
Quoted 10 lines · Jonathan Nieder
Hi,
Quoted 6 lines · Brandon Williams
run-command.c +0
-0@@ -458,6 +458,14 @@ int start_command(struct child_process *cmd)argv_array_pushv (&argv, cmd->argv); }
Quoted 11 lines
run-command.c int start_command(struct child_process *cmd) · line 469
Quoted 9 lines
run-command.c +8
-0+/* +* NOTE: In order to prevent deadlocking when using threads special +* care should be taken with the function calls made in between the +* fork() and exec() calls. No calls should be made to functions which +* require acquiring a lock (e.g. malloc) as the lock could have been +* held by another thread at the time of forking, causing the lock to +* never be released in the child process. +*/ cmd->pid =fork ();Why can't git use e.g.
posix_spawnto avoid this?
posix_spawn does not support chdir, and it seems we run non-git commands so no using "git -C" for those.
Quoted 4 lines
fork()-ing in a threaded context is very painful for maintainability. Any library function you are using could start taking a lock, and then you have a deadlock. So you have to make use of a very small whitelisted list of library functions for this to work.
Completely agreed.
On the other hand, I believe we should make run-command vfork-compatible (and Brandon's series is a big (but incomplete) step in the (IMHO) right direction); as anything which is vfork-safe would also be safe in the presence of threads+(plain) fork. With vfork; the two processes share heap until execve.
I posted some notes about it last year:
https://public-inbox.org/git/[email protected]/
Quoted 3 lines
The function calls you have to audit are not only between
fork()andexec()in the normal control flow. You have to worry about signal handlers, too.
Yes, all that auditing is necessary for vfork; too, but totally doable. The mainline Ruby implementation has been using vfork for spawning subprocesses for several years, now; and I think the ruby-core developers (myself included) have fixed all the problems with it; even in multi-threaded code which calls malloc.
Re: [PATCH 5/5] run-command: add note about forking and threading
Quoted 6 lines · Eric Wong
Quoted 1 line · Jonathan Nieder
Why can't git use e.g.
posix_spawnto avoid this?
posix_spawndoes not support chdir, and it seems we run non-git commands so no using "git -C" for those.
On the other hand, a number of the non-git commands we run are in a shell. At the cost of a wasted shell process, other commands can be spawned using posix_spawn by passing the chosen directory and command to
sh -c 'cd "$1" && shift && exec "$@"' -
[...]
Quoted 3 lines
I posted some notes about it last year:
Thanks for these notes.
Sincerely,
Jonathan
Re: [PATCH 5/5] run-command: add note about forking and threading
In reply to Eric WongQuoted 5 lines · Eric Wong
Quoted 1 line · Jonathan Nieder
Why can't git use e.g.
posix_spawnto avoid this?
posix_spawndoes not support chdir, and it seems we run non-git commands so no using "git -C" for those.
This is actually the biggest reason why I didn't go down that route from the start. I didn't want to dig through each and every user of run-command and verify that removing chdir wouldn't break them (or add in some other way to do it).
Quoted 7 lines
Quoted 5 lines
fork()-ing in a threaded context is very painful for maintainability. Any library function you are using could start taking a lock, and then you have a deadlock. So you have to make use of a very small whitelisted list of library functions for this to work.
Completely agreed.
Yes it is difficult to get right, but it seems very doable to just do all of the heavy-lifting prior to fork/exec making it easier to just use async-safe between the fork/exec in the child.
Quoted 6 lines
On the other hand, I believe we should make run-command vfork-compatible (and Brandon's series is a big (but incomplete) step in the (IMHO) right direction); as anything which is vfork-safe would also be safe in the presence of threads+(plain) fork. With vfork; the two processes share heap until execve.
I haven't looked to much into vfork, one of the benefits of vfork is that it is slightly more preferment than vanilla fork correct? What are some of the other benefits of using vfork over fork?
Quoted 14 lines
I posted some notes about it last year:
https://public-inbox.org/git/[email protected]/
Quoted 3 lines
The function calls you have to audit are not only between
fork()andexec()in the normal control flow. You have to worry about signal handlers, too.Yes, all that auditing is necessary for vfork; too, but totally doable. The mainline Ruby implementation has been using vfork for spawning subprocesses for several years, now; and I think the ruby-core developers (myself included) have fixed all the problems with it; even in multi-threaded code which calls malloc.
Signature · 1 line
Brandon Williams
Re: [PATCH 5/5] run-command: add note about forking and threading
Quoted 10 lines · Brandon Williams
Quoted 5 lines · Eric Wong
On the other hand, I believe we should make run-command vfork-compatible (and Brandon's series is a big (but incomplete) step in the (IMHO) right direction); as anything which is vfork-safe would also be safe in the presence of threads+(plain) fork. With vfork; the two processes share heap until execve.
I haven't looked to much into vfork, one of the benefits of vfork is that it is slightly more preferment than vanilla fork correct? What are some of the other benefits of using vfork over fork?
Yes, mainly performance and perhaps portability... Last I checked (over a decade ago); uCLinux without MMU could not fork processes; only vfork.
[PATCH 6/5] run-command: avoid potential dangers in forked child
In reply to Brandon WilliamsHi Brandon, this series tickles an old itch of mine, so I started working off of it. I'm only somewhat concerned with the path resolution in execvp(e) pontentially calling malloc on some libcs; but I suppose that's a separate patch for another time.
Only lightly-tested at the moment, but things seem to work...
------8<-----
Subject: [PATCH] run-command: avoid potential dangers in forked child
All of our standard error handling paths have the potential to call malloc or take stdio locks; so we must avoid them inside the forked child.
Instead, the child only writes an 8 byte struct atomically to the parent through the notification pipe to propagate an error. All user-visible error reporting happens from the parent; even avoiding functions like atexit(3) and exit(3).
Finally, we block signals and disable pthreads cancellation to avoid nasty surprises from other threads and signal handlers firing inside the child.
This prepares us for eventual use of vfork, where the child and parent share heap until the child calls execve or _exit.
The only somewhat questionable part I see left is the PATH searching in execvpe; which could be performed in the parent (taking into account chdir usage).
--- run-command.c | 273 +++++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 196 insertions(+), 77 deletions(-)
run-command.c +196 -77
@@ -117,55 +117,35 @@ static inline void close_pair(int fd[2])close (fd[1 ]); } -#ifndef GIT_WINDOWS_NATIVE -static inline void dup_devnull (int to) -{ -int fd =open ("/dev/null" ,O_RDWR ); -if (fd <0 ) -die_errno (_ ("open /dev/null failed" )); -if (dup2 (fd, to) <0 ) -die_errno (_ ("dup2(%d,%d) failed" ), fd, to); -close (fd); -} -#endif - -static char *locate_in_PATH (const char *file) +static int exists_in_PATH (const char *file) {const char *p =getenv ("PATH" ); -struct strbuf buf =STRBUF_INIT ; +char buf[PATH_MAX ];if (!p || !*p) -return NULL ; +return 0 ;while (1 ) {const char *end =strchrnul (p,':' ); - -strbuf_reset (&buf); +char *dst = buf;/* POSIX specifies an empty entry as the current directory. */ if (end != p) { -strbuf_add (&buf, p, end - p); -strbuf_addch (&buf,'/' ); +memcpy (dst, p, end - p); + dst += end - p; + *dst++ ='/' ; } -strbuf_addstr (&buf, file); +strcpy (dst, file); -if (!access (buf.buf,F_OK )) -return strbuf_detach (&buf,NULL ); +if (!access (buf,F_OK )) +return 1 ;if (!*end)break ; p = end +1 ; } -strbuf_release (&buf); -return NULL ; -} - -static int exists_in_PATH (const char *file) -{ -char *r =locate_in_PATH (file); -free (r); -return r !=NULL ; +return 0 ; }int sane_execvpe (const char *file,char *const argv[],char *const envp[])
@@ -227,16 +207,145 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ; +enum child_errcode { +CHILD_ERR_NULL_STDIN , +CHILD_ERR_NULL_STDOUT , +CHILD_ERR_NULL_STDERR , +CHILD_ERR_CHDIR , +CHILD_ERR_SIGPROCMASK , +CHILD_ERR_ENOENT , +CHILD_ERR_ENOENT_SILENT , +CHILD_ERR_ERRNO , +}; + +struct child_err { +enum child_errcode err; +int syserr;/* errno */ +}; -static void notify_parent (void ) +static void child_die (enum child_errcode err) { -/* -* execvp failed. If possible, we'd like to let start_command -* know, so failures like ENOENT can be handled right away; but -* otherwise, finish_command will still report the error. -*/ -xwrite (child_notifier,"" ,1 ); +struct child_err buf; + + buf.err = err; + buf.syserr = errno; + +/* write(2) on buf smaller than PIPE_BUF (min 512) is atomic: */ +xwrite (child_notifier, &buf,sizeof (buf)); +_exit (1 ); +} + +/* +* parent will make it look like the child spewed a fatal error and died +* this is needed to prevent changes to t0061. +*/ +static void fake_fatal (const char *err,va_list params) +{ +vreportf ("fatal: " , err, params); } + +static void child_error_fn (const char *err,va_list params) +{ +const char msg[] ="error() should not be called in child\n" ; \ +xwrite (2 , msg,sizeof (msg) -1 ); \ +} + +static void child_warn_fn (const char *err,va_list params) +{ +const char msg[] ="warn() should not be called in child\n" ; \ +xwrite (2 , msg,sizeof (msg) -1 ); \ +} + +static void NORETURN child_die_fn(const char *err, va_list params) +{ +const char msg[] ="die() should not be called in child\n" ; \ +xwrite (2 , msg,sizeof (msg) -1 ); \ +_exit (2 ); +} + +/* this runs in the parent process */ +static void child_err_spew (struct child_process *cmd,struct child_err *cerr) +{ +static void (*old_errfn)(const char *err,va_list params); + + old_errfn =get_error_routine (); +set_error_routine (fake_fatal); + errno = cerr->syserr; + +switch (cerr->err) { +case CHILD_ERR_NULL_STDIN : +error_errno ("error redirecting stdin to /dev/null" ); +break ; +case CHILD_ERR_NULL_STDOUT : +error_errno ("error redirecting stdout to /dev/null" ); +break ; +case CHILD_ERR_NULL_STDERR : +error_errno ("error redirecting stderr to /dev/null" ); +break ; +case CHILD_ERR_CHDIR : +error_errno ("exec '%s': cd to '%s' failed" , + cmd->argv[0 ], cmd->dir); +break ; +case CHILD_ERR_SIGPROCMASK : +error_errno ("sigprocmask failed restoring signals" ); +case CHILD_ERR_ENOENT : +error_errno ("cannot run %s" , cmd->argv[0 ]); +case CHILD_ERR_ENOENT_SILENT : +break ; +case CHILD_ERR_ERRNO : +error_errno ("cannot exec '%s'" , cmd->argv[0 ]); + } +set_error_routine (old_errfn); +} + +struct atfork_state { +#ifndef NO_PTHREADS +int cs; +#endif +sigset_t old; +}; + +#ifndef NO_PTHREADS +static void bug_die (int err,const char *msg) +{ +if (err) { + errno = err; +die_errno ("BUG: %s" , msg); + } +} +#endif + +static void atfork_prepare (struct atfork_state *as) +{ +sigset_t all; + +if (sigfillset (&all)) +die_errno ("sigfillset" ); +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &all, &as->old)) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_sigmask (SIG_SETMASK , &all, &as->old), +"blocking all signals" ); +bug_die (pthread_setcancelstate (PTHREAD_CANCEL_DISABLE , &as->cs), +"disabling cancellation" ); +#endif +fflush (NULL ); +} + +static void atfork_parent (struct atfork_state *as) +{ +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &af->old,NULL )) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_setcancelstate (as->cs,NULL ), +"re-enabling cancellation" ); +bug_die (pthread_sigmask (SIG_SETMASK , &as->old,NULL ), +"restoring signal mask" ); +#endif +} +#endif static inline void set_cloexec (int fd)
@@ -274,13 +383,6 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal) code +=128 ; }else if (WIFEXITED (status)) { code =WEXITSTATUS (status); -/* -* Convert special exit code when execvp failed. -*/ -if (code ==127 ) { - code =-1 ; - failed_errno =ENOENT ; - } }else {error ("waitpid is confused (%s)" , argv0); }
@@ -435,18 +537,21 @@ int start_command(struct child_process *cmd) {int notify_pipe[2 ];char **childenv; -FILE *child_err =NULL ;struct argv_array argv =ARGV_ARRAY_INIT ; +int null_fd =-1 ; +struct child_err cerr; +struct atfork_state as;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; childenv =prep_childenv (cmd->env); -if (cmd->no_stderr || need_err) { -int child_err_fd =dup (2 ); -set_cloexec (child_err_fd); - child_err =fdopen (child_err_fd,"w" ); +if (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) { + null_fd =open ("/dev/null" ,O_RDWR |O_CLOEXEC |O_NONBLOCK ); +if (null_fd <0 ) +die_errno (_ ("open /dev/null failed" )); +set_cloexec (null_fd); }if (cmd->git_cmd) {
@@ -466,25 +571,28 @@ int start_command(struct child_process *cmd)* held by another thread at the time of forking, causing the lock to * never be released in the child process. */ +atfork_prepare (&as); cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) { +int sig; +/* -* Redirect the channel to write syscall error messages to -* before redirecting the process's stderr so that all die() -* in subsequent call paths use the parent's stderr. +* make sure the default routines do not get called, +* they can take stdio locks and malloc: */ -if (child_err) -set_error_handle (child_err); +set_die_routine (child_die_fn); +set_error_routine (child_error_fn); +set_warn_routine (child_warn_fn);close (notify_pipe[0 ]);set_cloexec (notify_pipe[1 ]); child_notifier = notify_pipe[1 ]; -atexit (notify_parent); -if (cmd->no_stdin) -dup_devnull (0 ); -else if (need_in) { +if (cmd->no_stdin) { +if (dup2 (null_fd,0 ) <0 ) +child_die (CHILD_ERR_NULL_STDIN ); + }else if (need_in) {dup2 (fdin[0 ],0 );close_pair (fdin); }else if (cmd->in) {
@@ -492,9 +600,10 @@ int start_command(struct child_process *cmd)close (cmd->in); } -if (cmd->no_stderr) -dup_devnull (2 ); -else if (need_err) { +if (cmd->no_stderr) { +if (dup2 (null_fd,2 ) <0 ) +child_die (CHILD_ERR_NULL_STDERR ); + }else if (need_err) {dup2 (fderr[1 ],2 );close_pair (fderr); }else if (cmd->err >1 ) {
@@ -502,9 +611,10 @@ int start_command(struct child_process *cmd)close (cmd->err); } -if (cmd->no_stdout) -dup_devnull (1 ); -else if (cmd->stdout_to_stderr) +if (cmd->no_stdout) { +if (dup2 (null_fd,1 ) <0 ) +child_die (CHILD_ERR_NULL_STDOUT ); + }else if (cmd->stdout_to_stderr)dup2 (2 ,1 );else if (need_out) {dup2 (fdout[1 ],1 );
@@ -515,21 +625,29 @@ int start_command(struct child_process *cmd) }if (cmd->dir &&chdir (cmd->dir)) -die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], - cmd->dir); +child_die (CHILD_ERR_CHDIR ); + +/* +* restore default signal handlers here, in case +* we catch a signal right before sane_execvpe below +*/ +for (sig =1 ; sig <NSIG ; sig++) + (void )signal (sig,SIG_DFL ); + +if (sigprocmask (SIG_SETMASK , &as.old,NULL ) !=0 ) +child_die (CHILD_ERR_SIGPROCMASK );sane_execvpe (argv.argv[0 ], (char *const *) argv.argv, (char *const *) childenv); -if (errno ==ENOENT ) { -if (!cmd->silent_exec_failure) -error ("cannot run %s: %s" , cmd->argv[0 ], -strerror (ENOENT )); -exit (127 ); +if (cmd->silent_exec_failure) +child_die (CHILD_ERR_ENOENT_SILENT ); +child_die (CHILD_ERR_ENOENT ); }else { -die_errno ("cannot exec '%s'" , cmd->argv[0 ]); +child_die (CHILD_ERR_ERRNO ); } } +atfork_parent (&as);if (cmd->pid <0 )error_errno ("cannot fork() for %s" , cmd->argv[0 ]);else if (cmd->clean_on_exit)
@@ -538,24 +656,25 @@ int start_command(struct child_process *cmd)/* * Wait for child's execvp. If the execvp succeeds (or if fork() * failed), EOF is seen immediately by the parent. Otherwise, the -* child process sends a single byte. +* child process sends a child_err struct. * Note that use of this infrastructure is completely advisory, * therefore, we keep error checks minimal. */ close (notify_pipe[1 ]); -if (read (notify_pipe[0 ], ¬ify_pipe[1 ],1 ) ==1 ) { +if (null_fd >=0 ) +close (null_fd); +if (xread (notify_pipe[0 ], &cerr,sizeof (cerr)) ==sizeof (cerr)) {/* * At this point we know that fork() succeeded, but execvp() * failed. Errors have been reported to our stderr. */ wait_or_whine (cmd->pid, cmd->argv[0 ],0 ); +child_err_spew (cmd, &cerr); failed_errno = errno; cmd->pid =-1 ; }close (notify_pipe[0 ]); -if (child_err) -fclose (child_err);free (childenv);argv_array_clear (&argv); }
Signature · 1 line
EW
Re: [PATCH 6/5] run-command: avoid potential dangers in forked child
Quoted 7 lines · Eric Wong
Hi Brandon, this series tickles an old itch of mine, so I started working off of it. I'm only somewhat concerned with the path resolution in execvp(e) pontentially calling malloc on some libcs; but I suppose that's a separate patch for another time.
Only lightly-tested at the moment, but things seem to work...
Thanks Eric! I'll spend some time looking at this patch later today. As for the path resolution in execvp(e), I guess we could completely avoid that if we did the path resolution ourselves, prior to forking, and then just use execv(e) since it shouldn't have any calls to malloc in them correct?
Signature · 1 line
Brandon Williams
Re: [PATCH 6/5] run-command: avoid potential dangers in forked child
Quoted 7 lines · Brandon Williams
Quoted 6 lines · Eric Wong
Hi Brandon, this series tickles an old itch of mine, so I started working off of it. I'm only somewhat concerned with the path resolution in execvp(e) pontentially calling malloc on some libcs; but I suppose that's a separate patch for another time.
Quoted 7 lines
Quoted 1 line
Only lightly-tested at the moment, but things seem to work...
Thanks Eric! I'll spend some time looking at this patch later today. As for the path resolution in execvp(e), I guess we could completely avoid that if we did the path resolution ourselves, prior to forking, and then just use execv(e) since it shouldn't have any calls to malloc in them correct?
Yeah. I spent some time looking at it last night, but emulating the existing ENOENT / EACCESS / ENOTDIR mapping made my head hurt.
And I'm not sure if I introduced any off-by-one errors in exists_in_PATH when removing strbuf usage; string manipulation
in plain C scares me :x Since memcpy/strcpy/getenv in there
are not specified as async-signal safe, they could theoretically take locks and cause breakage inside a child.
I also wonder if there's a way to annotate internal functions as async-signal safe (and thus vfork-child safe) besides sprinkling comments in certain functions like xwrite.
Re: [PATCH 6/5] run-command: avoid potential dangers in forked child
Quoted 20 lines · Eric Wong
Quoted 7 lines · Brandon Williams
Quoted 6 lines · Eric Wong
Hi Brandon, this series tickles an old itch of mine, so I started working off of it. I'm only somewhat concerned with the path resolution in execvp(e) pontentially calling malloc on some libcs; but I suppose that's a separate patch for another time.
Quoted 7 lines
Quoted 1 line
Only lightly-tested at the moment, but things seem to work...
Thanks Eric! I'll spend some time looking at this patch later today. As for the path resolution in execvp(e), I guess we could completely avoid that if we did the path resolution ourselves, prior to forking, and then just use execv(e) since it shouldn't have any calls to malloc in them correct?
Yeah. I spent some time looking at it last night, but emulating the existing ENOENT / EACCESS / ENOTDIR mapping made my head hurt.
Quoted 5 lines
And I'm not sure if I introduced any off-by-one errors in
exists_in_PATHwhen removing strbuf usage; string manipulation
in plain C scares me :x Since memcpy/strcpy/getenv in there
are not specified as async-signal safe, they could theoretically take locks and cause breakage inside a child.
Well if we move away from the (p) variant of exec, and do that resolution ourselves, then we can use the existing exists_in_PATH/locate_in_PATH logic (using strbufs!) and avoid needing to do that string manipulation in plain C. Of course we would then need to adjust some of the errno handling (e.g. if it doesn't exist in the path, we would then know that prior to forking so really no need to handle that in the child anymore).
Signature · 1 line
Brandon Williams
Quoted 3 lines · Brandon Williams
As far as I understand the only instance of threading and forking which exists in the current code base is 'git grep --recurse-submodules', and the standard builds against glibc shouldn't exhibit any of this deadlocking.
I don't think we consider builds against glibc to be the only standard way to build git. So I do think we need to fix this (and not, e.g., to modify our build instructions to require use of a malloc implementation that registers an atfork handler that unlocks all of its locks). Thanks for your work on that.
Jonathan Tan had an idea about how to side-step the issue: what if "grep" forks an appropriate set of child processes before creating any threads and then communicates with those children using pipes? Because no threads have been spawned yet in the children, the child processes can use ordinary run_command. When run_command finishes, the child is still available to launch another command using run_command.
Thoughts?
Jonathan
Quoted 8 lines · Jonathan Nieder
Brandon Williams wrote:
Jonathan Tan had an idea about how to side-step the issue: what if "grep" forks an appropriate set of child processes before creating any threads and then communicates with those children using pipes? Because no threads have been spawned yet in the children, the child processes can use ordinaryrun_command. Whenrun_commandfinishes, the child is still available to launch another command usingrun_command.
While that would be one way to solve the issue, I think that doing that would require more work refactoring and make grep's code path more complex than making the adjustments in run-command itself.
Signature · 1 line
Brandon Williams
v2 cover forking and threading
In reply to Brandon Williamsv2 does a bit of restructuring based on comments from reviewers. I took the patch by Eric and broke it up and tweaked it a bit to flow better with v2. I left out the part of Eric's patch which did signal manipulation as I wasn't experienced enough to know what it was doing or why it was necessary. Though I believe the code is structured in such a way that Eric could make another patch on top of this series with just the signal changes.
I switched to using 'execve' instead of 'execvpe' because 'execvpe' isn't a portable call and doesn't work on systems like macOS. This means that the path resolution needs to be done by hand before forking (which there already existed a function to do just that).
From what I can see, there are now no calls in the child process (after fork and before exec/_exit) which are not Async-Signal-Safe. This means that fork/exec in a threaded context should work without deadlock and we could potentially move to using vfork instead of fork, though I'll let others more experienced make that decision.
Brandon Williams (6):
t5550: use write_script to generate post-update hook
run-command: prepare command before forking
run-command: prepare child environment before forking
run-command: don't die in child when duping /dev/null
run-command: eliminate calls to error handling functions in child
run-command: add note about forking and threading
run-command.c | 291 ++++++++++++++++++++++++++++++++++----------- t/t5550-http-fetch-dumb.sh | 5 +- 2 files changed, 223 insertions(+), 73 deletions(-)
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v2 1/6 t5550: use write_script to generate post-update hook
The post-update hooks created in t5550-http-fetch-dumb.sh is missing the "!#/bin/sh" line which can cause issues with portability. Instead create the hook using the 'write_script' function which includes the proper "#!" line.
--- t/t5550-http-fetch-dumb.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
t/t5550-http-fetch-dumb.sh +3 -2
@@ -20,8 +20,9 @@ test_expect_success 'create http-accessible bare repository with loose objects' (cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" && git config core.bare true && mkdir -p hooks && - echo "exec git update-server-info" >hooks/post-update && - chmod +x hooks/post-update && + write_script "hooks/post-update" <<-\EOF && + exec git update-server-info + EOF hooks/post-update ) && git remote add public "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
Signature · 1 line
2.12.2.762.g0e3151a226-goog
Re: [PATCH v2 1/6] t5550: use write_script to generate post-update hook
Hi,
Quoted 9 lines · Brandon Williams
The post-update hooks created in
t5550-http-fetch-dumb.shis missing the "!#/bin/sh" line which can cause issues with portability. Instead create the hook using the 'write_script' function which includes the proper "#!" line.Signed-off-by Brandon Williams <[email protected]>--- t/t5550-http-fetch-dumb.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
This would allow later patches to regress a previously supported behavior.
I agree that it's silly to test that behavior as a side-effect of this unrelated test, but I don't think we want to lose the test coverage.
Thanks for tracking this down and hope that helps, Jonathan
Re: [PATCH v2 1/6] t5550: use write_script to generate post-update hook
Quoted 11 lines · Jonathan Nieder
Quoted 4 lines · Brandon Williams
The post-update hooks created in
t5550-http-fetch-dumb.shis missing the "!#/bin/sh" line which can cause issues with portability. Instead create the hook using the 'write_script' function which includes the proper "#!" line.This would allow later patches to regress a previously supported behavior.
I agree that it's silly to test that behavior as a side-effect of this unrelated test, but I don't think we want to lose the test coverage.
I was about to write something similar about this regression. The new execve-using code should handle ENOEXEC as execvpe does and probably a new test for it needs to be written.
Re: [PATCH v2 1/6] t5550: use write_script to generate post-update hook
Quoted 10 lines · Eric Wong
Quoted 9 lines · Jonathan Nieder
Quoted 4 lines · Brandon Williams
The post-update hooks created in
t5550-http-fetch-dumb.shis missing the "!#/bin/sh" line which can cause issues with portability. Instead create the hook using the 'write_script' function which includes the proper "#!" line.This would allow later patches to regress a previously supported behavior.
Quoted 6 lines
Quoted 2 lines
I agree that it's silly to test that behavior as a side-effect of this unrelated test, but I don't think we want to lose the test coverage.
I was about to write something similar about this regression. The new execve-using code should handle ENOEXEC as execvpe does and probably a new test for it needs to be written.
Would it be enough to upon seeing a failed exec call and ENOEXEC to retry a single time, invoking the shell to attempt to interpret the command?
Signature · 1 line
Brandon Williams
Re: [PATCH v2 1/6] t5550: use write_script to generate post-update hook
Quoted 14 lines · Brandon Williams
Quoted 13 lines · Eric Wong
Quoted 11 lines · Jonathan Nieder
Quoted 4 lines · Brandon Williams
The post-update hooks created in
t5550-http-fetch-dumb.shis missing the "!#/bin/sh" line which can cause issues with portability. Instead create the hook using the 'write_script' function which includes the proper "#!" line.This would allow later patches to regress a previously supported behavior.
I agree that it's silly to test that behavior as a side-effect of this unrelated test, but I don't think we want to lose the test coverage.
Quoted 7 lines
Quoted 3 lines
I was about to write something similar about this regression. The new execve-using code should handle ENOEXEC as execvpe does and probably a new test for it needs to be written.
Would it be enough to upon seeing a failed exec call and ENOEXEC to retry a single time, invoking the shell to attempt to interpret the command?
Yes, that's exactly what glibc does.
PATCH v2 2/6 run-command: prepare command before forking
In reply to Brandon WilliamsIn order to avoid allocation between 'fork()' and 'exec()' the argv array used in the exec call is prepared prior to forking the process.
In addition to this, the function used to exec is changed from 'execvp()' to 'execv()' as the (p) variant of exec has the potential to call malloc during the path resolution it performs. Instead we simply do the path resolution ourselves during the preparation stage prior to forking.
--- run-command.c | 60 +++++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 40 insertions(+), 20 deletions(-)
run-command.c +40 -20
@@ -221,18 +221,6 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv) }#ifndef GIT_WINDOWS_NATIVE -static int execv_shell_cmd (const char **argv) -{ -struct argv_array nargv =ARGV_ARRAY_INIT ; -prepare_shell_cmd (&nargv, argv); -trace_argv_printf (nargv.argv,"trace: exec:" ); -sane_execvp (nargv.argv[0 ], (char **)nargv.argv); -argv_array_clear (&nargv); -return -1 ; -} -#endif - -#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ;static void notify_parent (void )
@@ -244,6 +232,35 @@ static void notify_parent(void)*/ xwrite (child_notifier,"" ,1 ); } + +static void prepare_cmd (struct argv_array *out,const struct child_process *cmd) +{ +if (!cmd->argv[0 ]) +die ("BUG: command is empty" ); + +if (cmd->git_cmd) { +argv_array_push (out,"git" ); +argv_array_pushv (out, cmd->argv); + }else if (cmd->use_shell) { +prepare_shell_cmd (out, cmd->argv); + }else { +argv_array_pushv (out, cmd->argv); + } + +/* +* If there are no '/' characters in the command then perform a path +* lookup and use the resolved path as the command to exec. If there +* are no '/' characters or if the command wasn't found in the path, +* have exec attempt to invoke the command directly. +*/ +if (!strchr (out->argv[0 ],'/' )) { +char *program =locate_in_PATH (out->argv[0 ]); +if (program) { +free ((char *)out->argv[0 ]); + out->argv[0 ] = program; + } + } +}#endif static inline void set_cloexec (int fd)
@@ -372,9 +389,13 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +struct argv_array argv =ARGV_ARRAY_INIT ; +if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; +prepare_cmd (&argv, cmd); + cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) {
@@ -437,12 +458,9 @@ int start_command(struct child_process *cmd)unsetenv (*cmd->env); } } -if (cmd->git_cmd) -execv_git_cmd (cmd->argv); -else if (cmd->use_shell) -execv_shell_cmd (cmd->argv); -else -sane_execvp (cmd->argv[0 ], (char *const *) cmd->argv); + +execv (argv.argv[0 ], (char *const *) argv.argv); +if (errno ==ENOENT ) {if (!cmd->silent_exec_failure) error("cannot run %s: %s", cmd->argv[0],
@@ -458,7 +476,7 @@ int start_command(struct child_process *cmd)mark_child_for_cleanup (cmd->pid, cmd);/* -* Wait for child's execvp. If the execvp succeeds (or if fork() +* Wait for child's exec. If the exec succeeds (or if fork() * failed), EOF is seen immediately by the parent. Otherwise, the * child process sends a single byte. * Note that use of this infrastructure is completely advisory,
@@ -467,7 +485,7 @@ int start_command(struct child_process *cmd)close (notify_pipe[1 ]);if (read (notify_pipe[0 ], ¬ify_pipe[1 ],1 ) ==1 ) {/* -* At this point we know that fork() succeeded, but execvp() +* At this point we know that fork() succeeded, but exec() * failed. Errors have been reported to our stderr. */ wait_or_whine (cmd->pid, cmd->argv[0 ],0 );
@@ -475,6 +493,8 @@ int start_command(struct child_process *cmd) cmd->pid = -1; }close (notify_pipe[0 ]); + +argv_array_clear (&argv); }#else {
Signature · 1 line
2.12.2.762.g0e3151a226-goog
Re: [PATCH v2 2/6] run-command: prepare command before forking
Hi,
Quoted 2 lines · Brandon Williams
In order to avoid allocation between '
fork()' and 'exec()' the argv array used in the exec call is prepared prior to forking the process.
nit: s/(the argv array.*) is prepared/prepare \1/
Git's commit messages are in the imperative mood, as if they are ordering the code or the computer to do something.
More importantly, the commit message is a good place to explain some of the motivation behind the patch so that people can understand what the patch is for by reading it without having to dig into mailing list archives and get the discussion there.
E.g. this could say
- that grep tests are intermittently failing in configurations using some versions of tcmalloc
- that the cause is interaction between fork and threads: malloc holds a lock that those versions of tcmalloc doesn't release in a pthread_atfork handler
- that according to [1] we need to only call async-signal-safe operations between fork and exec. Using malloc to build the argv array isn't async-signal-safe
[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html
Quoted 3 lines
In addition to this, the function used to exec is changed from '
execvp()' to 'execv()' as the (p) variant of exec has the potential to call malloc during the path resolution it performs.
*puzzled* is execvp actually allowed to call malloc?
Could this part go in a separate patch? That would make it easier to review.
[...]
Quoted 1 line
+++
b/run-command.c
[...]
run-command.c static void notify_parent(void) · line 262
Quoted 13 lines
run-command.c +13
-0+/* +* If there are no '/' characters in the command then perform a path +* lookup and use the resolved path as the command to exec. If there +* are no '/' characters or if the command wasn't found in the path, +* have exec attempt to invoke the command directly. +*/ +if (!strchr (out->argv[0 ],'/' )) { +char *program =locate_in_PATH (out->argv[0 ]); +if (program) { +free ((char *)out->argv[0 ]); + out->argv[0 ] = program; + } + }
This does one half of what execvp does but leaves out the other half.
http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html
explains:
There are two distinct ways in which the contents of the process image file may cause the execution to fail, distinguished by the setting of errno to either [ENOEXEC] or [EINVAL] (see the ERRORS section). In the cases where the other members of the exec family of functions would fail and set errno to [ENOEXEC], the execlp() and execvp() functions shall execute a command interpreter and the environment of the executed command shall be as if the process invoked the sh utility using execl() as follows:
execl(<shell path>, arg0, file, arg1, ..., (char *)0);
I think this is what the first patch in the series was about. Do we want to drop that support?
I think we need to keep it, since it is easy for authors of e.g. credential helpers to accidentally rely on it.
[...]
run-command.c int start_command(struct child_process *cmd) · line 462
Quoted 12 lines
run-command.c +2
-6@@ -437,12 +458,9 @@ int start_command(struct child_process *cmd)unsetenv (*cmd->env); } } -if (cmd->git_cmd) -execv_git_cmd (cmd->argv); -else if (cmd->use_shell) -execv_shell_cmd (cmd->argv); -else -sane_execvp (cmd->argv[0 ], (char *const *) cmd->argv); + +execv (argv.argv[0 ], (char *const *) argv.argv);
What happens in the case sane_execvp was trying to handle? Does prepare_cmd need error handling for when the command isn't found?
Sorry this got so fussy.
Thanks and hope that helps,
Jonathan
Re: [PATCH v2 2/6] run-command: prepare command before forking
Quoted 30 lines · Jonathan Nieder
Hi,
Quoted 2 lines · Brandon Williams
In order to avoid allocation between '
fork()' and 'exec()' the argv array used in the exec call is prepared prior to forking the process.nit: s/(the argv array.*) is prepared/prepare \1/
Git's commit messages are in the imperative mood, as if they are ordering the code or the computer to do something.
More importantly, the commit message is a good place to explain some of the motivation behind the patch so that people can understand what the patch is for by reading it without having to dig into mailing list archives and get the discussion there.
E.g. this could say
- that grep tests are intermittently failing in configurations using some versions of tcmalloc
- that the cause is interaction between fork and threads: malloc holds a lock that those versions of tcmalloc doesn't release in a
pthread_atforkhandler- that according to [1] we need to only call async-signal-safe operations between fork and exec. Using malloc to build the argv array isn't async-signal-safe
Quoted 7 lines
[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html
Quoted 3 lines
In addition to this, the function used to exec is changed from '
execvp()' to 'execv()' as the (p) variant of exec has the potential to call malloc during the path resolution it performs.*puzzled* is execvp actually allowed to call malloc?
It could possible as it isn't async-signal-safe.
Quoted 3 lines
Could this part go in a separate patch? That would make it easier to review.
I'll break this conversion out to a different patch.
Quoted 60 lines
[...]
Quoted 1 line
+++
b/run-command.c[...]
run-command.c static void notify_parent(void) · line 262
Quoted 13 lines
run-command.c +13
-0+/* +* If there are no '/' characters in the command then perform a path +* lookup and use the resolved path as the command to exec. If there +* are no '/' characters or if the command wasn't found in the path, +* have exec attempt to invoke the command directly. +*/ +if (!strchr (out->argv[0 ],'/' )) { +char *program =locate_in_PATH (out->argv[0 ]); +if (program) { +free ((char *)out->argv[0 ]); + out->argv[0 ] = program; + } + }This does one half of what execvp does but leaves out the other half.
http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html
explains:There are two distinct ways in which the contents of the process image file may cause the execution to fail, distinguished by the setting of errno to either [ENOEXEC] or [EINVAL] (see the ERRORS section). In the cases where the other members of the exec family of functions would fail and set errno to [ENOEXEC], the
execlp()andexecvp()functions shall execute a command interpreter and the environment of the executed command shall be as if the process invoked the sh utility usingexecl()as follows:execl(<shell path>, arg0, file, arg1, ..., (char *)0);I think this is what the first patch in the series was about. Do we want to drop that support?
I think we need to keep it, since it is easy for authors of e.g. credential helpers to accidentally rely on it.
[...]
run-command.c int start_command(struct child_process *cmd) · line 462
Quoted 12 lines
run-command.c +2
-6@@ -437,12 +458,9 @@ int start_command(struct child_process *cmd)unsetenv (*cmd->env); } } -if (cmd->git_cmd) -execv_git_cmd (cmd->argv); -else if (cmd->use_shell) -execv_shell_cmd (cmd->argv); -else -sane_execvp (cmd->argv[0 ], (char *const *) cmd->argv); + +execv (argv.argv[0 ], (char *const *) argv.argv);What happens in the case
sane_execvpwas trying to handle? Doesprepare_cmdneed error handling for when the command isn't found?Sorry this got so fussy.
Thanks and hope that helps,
Jonathan
Signature · 1 line
Brandon Williams
PATCH v2 3/6 run-command: prepare child environment before forking
In reply to Brandon WilliamsIn order to avoid allocation between 'fork()' and 'exec()' prepare the environment to be used in the child process prior to forking.
Switch to using 'execve()' so that the construct child environment can used in the exec'd process.
--- run-command.c | 83 ++++++++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 74 insertions(+), 9 deletions(-)
run-command.c +74 -9
@@ -261,6 +261,75 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) } } } + +static int env_isequal (const char *e1,const char *e2) +{ +for (;;) { +char c1 = *e1++; +char c2 = *e2++; + c1 = (c1 =='=' ) ?'\0' :tolower (c1); + c2 = (c2 =='=' ) ?'\0' :tolower (c2); + +if (c1 != c2) +return 0 ; +if (c1 =='\0' ) +return 1 ; + } +} + +static int searchenv (char **env,const char *name) +{ +int pos =0 ; + +for (; env[pos]; pos++) +if (env_isequal (env[pos], name)) +break ; + +return pos; +} + +static int do_putenv (char **env,int env_nr,const char *name) +{ +int pos =searchenv (env, name); + +if (strchr (name,'=' )) { +/* ('key=value'), insert of replace entry */ +if (pos >= env_nr) + env_nr++; + env[pos] = (char *) name; + }else if (pos < env_nr) { +/* otherwise ('key') remove existing entry */ + env_nr--; +memmove (&env[pos], &env[pos +1 ], + (env_nr - pos) *sizeof (char *)); + env[env_nr] =NULL ; + } + +return env_nr; +} + +static char **prep_childenv (const char *const *deltaenv) +{ +char **childenv; +int childenv_nr =0 , childenv_alloc =0 ; +int i; + +for (i =0 ; environ[i]; i++) + childenv_nr++; +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_alloc++; +/* Add one for the NULL termination */ + childenv_alloc += childenv_nr +1 ; + + childenv =xcalloc (childenv_alloc,sizeof (char *)); +memcpy (childenv, environ, childenv_nr *sizeof (char *)); + +/* merge in deltaenv */ +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_nr =do_putenv (childenv, childenv_nr, deltaenv[i]); + +return childenv; +}#endif static inline void set_cloexec (int fd)
@@ -389,12 +458,14 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;prepare_cmd (&argv, cmd); + childenv =prep_childenv (cmd->env); cmd->pid =fork (); failed_errno = errno;
@@ -450,16 +521,9 @@ int start_command(struct child_process *cmd)if (cmd->dir &&chdir (cmd->dir))die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir); -if (cmd->env) { -for (; *cmd->env; cmd->env++) { -if (strchr (*cmd->env,'=' )) -putenv ((char *)*cmd->env); -else -unsetenv (*cmd->env); - } - } -execv (argv.argv[0 ], (char *const *) argv.argv); +execve (argv.argv[0 ], (char *const *) argv.argv, + (char *const *) childenv);if (errno ==ENOENT ) {if (!cmd->silent_exec_failure)
@@ -495,6 +559,7 @@ int start_command(struct child_process *cmd)close (notify_pipe[0 ]);argv_array_clear (&argv); + free(childenv); }#else {
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v2 4/6 run-command: don't die in child when duping /dev/null
In reply to Brandon Williams--- run-command.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-)
run-command.c +13 -15
@@ -117,18 +117,6 @@ static inline void close_pair(int fd[2])close (fd[1 ]); } -#ifndef GIT_WINDOWS_NATIVE -static inline void dup_devnull (int to) -{ -int fd =open ("/dev/null" ,O_RDWR ); -if (fd <0 ) -die_errno (_ ("open /dev/null failed" )); -if (dup2 (fd, to) <0 ) -die_errno (_ ("dup2(%d,%d) failed" ), fd, to); -close (fd); -} -#endif -static char *locate_in_PATH (const char *file) {const char *p =getenv ("PATH" );
@@ -458,12 +446,20 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +int null_fd =-1 ;char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; +if (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) { + null_fd =open ("/dev/null" ,O_RDWR |O_CLOEXEC |O_NONBLOCK ); +if (null_fd <0 ) +die_errno (_ ("open /dev/null failed" )); +set_cloexec (null_fd); + } +prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env);
@@ -487,7 +483,7 @@ int start_command(struct child_process *cmd)atexit (notify_parent);if (cmd->no_stdin) -dup_devnull (0 ); +dup2 (null_fd,0 );else if (need_in) {dup2 (fdin[0 ],0 );close_pair (fdin);
@@ -497,7 +493,7 @@ int start_command(struct child_process *cmd) }if (cmd->no_stderr) -dup_devnull (2 ); +dup2 (null_fd,2 );else if (need_err) {dup2 (fderr[1 ],2 );close_pair (fderr);
@@ -507,7 +503,7 @@ int start_command(struct child_process *cmd) }if (cmd->no_stdout) -dup_devnull (1 ); +dup2 (null_fd,1 );else if (cmd->stdout_to_stderr)dup2 (2 ,1 );else if (need_out) {
@@ -558,6 +554,8 @@ int start_command(struct child_process *cmd) }close (notify_pipe[0 ]); +if (null_fd >0 ) +close (null_fd);argv_array_clear (&argv);free (childenv); }
Signature · 1 line
2.12.2.762.g0e3151a226-goog
Re: [PATCH v2 4/6] run-command: don't die in child when duping /dev/null
run-command.c int start_command(struct child_process *cmd) · line 486
Quoted 6 lines · Brandon Williams
run-command.c +1
-1@@ -487,7 +483,7 @@ int start_command(struct child_process *cmd)atexit (notify_parent);if (cmd->no_stdin) -dup_devnull (0 ); +dup2 (null_fd,0 );
I prefer we keep error checking for dup2 failures, and also add more error checking for unchecked dup2 calls. Can be a separate patch, I suppose.
Ditto for other dup2 changes
run-command.c int start_command(struct child_process *cmd) · line 558
Quoted 6 lines
run-command.c +2
-0@@ -558,6 +554,8 @@ int start_command(struct child_process *cmd) }close (notify_pipe[0 ]); +if (null_fd >0 ) +close (null_fd);
I would prefer:
if (null_fd >= 0)
here, even if we currently do not release stdin.
Quoted 2 lines
run-command.c +0
-0argv_array_clear (&argv);free (childenv);
Re: [PATCH v2 4/6] run-command: don't die in child when duping /dev/null
Quoted 12 lines · Eric Wong
run-command.c int start_command(struct child_process *cmd) · line 486
Quoted 6 lines · Brandon Williams
run-command.c +1
-1@@ -487,7 +483,7 @@ int start_command(struct child_process *cmd)atexit (notify_parent);if (cmd->no_stdin) -dup_devnull (0 ); +dup2 (null_fd,0 );I prefer we keep error checking for dup2 failures, and also add more error checking for unchecked dup2 calls. Can be a separate patch, I suppose.
Quoted 1 line
Ditto for other dup2 changes
I simply figured that since we weren't doing the checks on the other dup2 calls that I would keep it consistent. But if we wanted to add in more error checking then we can can in another patch.
Quoted 5 lines
Quoted 5 lines
run-command.c +0
-0@@ -558,6 +554,8 @@ int start_command(struct child_process *cmd) }close (notify_pipe[0 ]);
Quoted 8 lines
run-command.c int start_command(struct child_process *cmd) · line 558
Quoted 2 lines
run-command.c +2
-0+if (null_fd >0 ) +close (null_fd);I would prefer:
if (null_fd >= 0)here, even if we currently do not release stdin.
K will do.
Quoted 3 lines
Quoted 3 lines
run-command.c +0
-0argv_array_clear (&argv);free (childenv);
Signature · 1 line
Brandon Williams
PATCH v2 5/6 run-command: eliminate calls to error handling functions in child
In reply to Brandon WilliamsAll of our standard error handling paths have the potential to call malloc or take stdio locks; so we must avoid them inside the forked child.
Instead, the child only writes an 8 byte struct atomically to the parent through the notification pipe to propagate an error. All user-visible error reporting happens from the parent; even avoiding functions like atexit(3) and exit(3).
--- run-command.c | 121 ++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 89 insertions(+), 32 deletions(-)
run-command.c +89 -32
@@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ; -static void notify_parent (void ) +enum child_errcode { +CHILD_ERR_CHDIR , +CHILD_ERR_ENOENT , +CHILD_ERR_SILENT , +CHILD_ERR_ERRNO , +}; + +struct child_err { +enum child_errcode err; +int syserr;/* errno */ +}; + +static void child_die (enum child_errcode err) { -/* -* execvp failed. If possible, we'd like to let start_command -* know, so failures like ENOENT can be handled right away; but -* otherwise, finish_command will still report the error. -*/ -xwrite (child_notifier,"" ,1 ); +struct child_err buf; + + buf.err = err; + buf.syserr = errno; + +/* write(2) on buf smaller than PIPE_BUF (min 512) is atomic: */ +xwrite (child_notifier, &buf,sizeof (buf)); +_exit (1 ); +} + +/* +* parent will make it look like the child spewed a fatal error and died +* this is needed to prevent changes to t0061. +*/ +static void fake_fatal (const char *err,va_list params) +{ +vreportf ("fatal: " , err, params); +} + +static void child_error_fn (const char *err,va_list params) +{ +const char msg[] ="error() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +} + +static void child_warn_fn (const char *err,va_list params) +{ +const char msg[] ="warn() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +} + +static void NORETURN child_die_fn(const char *err, va_list params) +{ +const char msg[] ="die() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +_exit (2 ); +} + +/* this runs in the parent process */ +static void child_err_spew (struct child_process *cmd,struct child_err *cerr) +{ +static void (*old_errfn)(const char *err,va_list params); + + old_errfn =get_error_routine (); +set_error_routine (fake_fatal); + errno = cerr->syserr; + +switch (cerr->err) { +case CHILD_ERR_CHDIR : +error_errno ("exec '%s': cd to '%s' failed" , + cmd->argv[0 ], cmd->dir); +break ; +case CHILD_ERR_ENOENT : +error_errno ("cannot run %s" , cmd->argv[0 ]); +break ; +case CHILD_ERR_SILENT : +break ; +case CHILD_ERR_ERRNO : +error_errno ("cannot exec '%s'" , cmd->argv[0 ]); +break ; + } +set_error_routine (old_errfn); }static void prepare_cmd (struct argv_array *out,const struct child_process *cmd)
@@ -355,13 +423,6 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal) code +=128 ; }else if (WIFEXITED (status)) { code =WEXITSTATUS (status); -/* -* Convert special exit code when execvp failed. -*/ -if (code ==127 ) { - code =-1 ; - failed_errno =ENOENT ; - } }else {error ("waitpid is confused (%s)" , argv0); }
@@ -449,6 +510,7 @@ int start_command(struct child_process *cmd)int null_fd =-1 ;char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ; +struct child_err cerr;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;
@@ -467,20 +529,16 @@ int start_command(struct child_process *cmd) failed_errno = errno;if (!cmd->pid) {/* -* Redirect the channel to write syscall error messages to -* before redirecting the process's stderr so that all die() -* in subsequent call paths use the parent's stderr. +* Ensure the default die/error/warn routines do not get +* called, they can take stdio locks and malloc. */ -if (cmd->no_stderr || need_err) { -int child_err =dup (2 ); -set_cloexec (child_err); -set_error_handle (fdopen (child_err,"w" )); - } +set_die_routine (child_die_fn); +set_error_routine (child_error_fn); +set_warn_routine (child_warn_fn);close (notify_pipe[0 ]);set_cloexec (notify_pipe[1 ]); child_notifier = notify_pipe[1 ]; -atexit (notify_parent);if (cmd->no_stdin)dup2 (null_fd,0 );
@@ -515,19 +573,17 @@ int start_command(struct child_process *cmd) }if (cmd->dir &&chdir (cmd->dir)) -die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], - cmd->dir); +child_die (CHILD_ERR_CHDIR );execve (argv.argv[0 ], (char *const *) argv.argv, (char *const *) childenv);if (errno ==ENOENT ) { -if (!cmd->silent_exec_failure) -error ("cannot run %s: %s" , cmd->argv[0 ], -strerror (ENOENT )); -exit (127 ); +if (cmd->silent_exec_failure) +child_die (CHILD_ERR_SILENT ); +child_die (CHILD_ERR_ENOENT ); }else { -die_errno ("cannot exec '%s'" , cmd->argv[0 ]); +child_die (CHILD_ERR_ERRNO ); } }if (cmd->pid <0 )
@@ -538,17 +594,18 @@ int start_command(struct child_process *cmd)/* * Wait for child's exec. If the exec succeeds (or if fork() * failed), EOF is seen immediately by the parent. Otherwise, the -* child process sends a single byte. +* child process sends a child_err struct. * Note that use of this infrastructure is completely advisory, * therefore, we keep error checks minimal. */ close (notify_pipe[1 ]); -if (read (notify_pipe[0 ], ¬ify_pipe[1 ],1 ) ==1 ) { +if (xread (notify_pipe[0 ], &cerr,sizeof (cerr)) ==sizeof (cerr)) {/* * At this point we know that fork() succeeded, but exec() * failed. Errors have been reported to our stderr. */ wait_or_whine (cmd->pid, cmd->argv[0 ],0 ); +child_err_spew (cmd, &cerr); failed_errno = errno; cmd->pid =-1 ; }
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v2 6/6 run-command: add note about forking and threading
In reply to Brandon WilliamsAll non-Async-Signal-Safe functions (e.g. malloc and die) were removed between 'fork' and 'exec' in start_command in order to avoid potential deadlocking when forking while multiple threads are running. This deadlocking is possible when a thread (other than the one forking) has acquired a lock and didn't get around to releasing it before the fork. This leaves the lock in a locked state in the resulting process with no hope of it ever being released.
Add a note describing this potential pitfall before the call to 'fork()' so people working in this section of the code know to only use Async-Signal-Safe functions in the child process.
--- run-command.c | 9 +++++++++ 1 file changed, 9 insertions(+)
run-command.c +9 -0
@@ -525,6 +525,15 @@ int start_command(struct child_process *cmd)prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env); +/* +* NOTE: In order to prevent deadlocking when using threads special +* care should be taken with the function calls made in between the +* fork() and exec() calls. No calls should be made to functions which +* require acquiring a lock (e.g. malloc) as the lock could have been +* held by another thread at the time of forking, causing the lock to +* never be released in the child process. This means only +* Async-Signal-Safe functions are permitted in the child. +*/ cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) {
Signature · 1 line
2.12.2.762.g0e3151a226-goog
Quoted 3 lines · Brandon Williams
From what I can see, there are now no calls in the child process (after fork and before exec/_exit) which are not Async-Signal-Safe. This means that fork/exec in a threaded context should work without deadlock
I don't see why the former implies the latter. Can you explain further?
You already know my opinions about fork+threads by now. I continue to think this is heading in a direction of decreased maintainability that I dread.
That's not to say that this is wasted work. I would prefer an approach like the following:
1. First, make grep work without running fork() from threads. Jonathan Tan's approach would be one way to do this. Another way would be to simply disable threads in --recurse-submodules mode.
This would be the first thing to do because it would make tests reliable again, without having to wait for deeper changes.
2. Then, teaching run_command to prepare the environment and do $PATH lookup before forking. This might make it possible for run_command to use vfork or might not.
3. Teaching run_command to delegate chdir to the child, using -C for git commands and 'cd' for shell commands, and using a shell where necessary where it didn't before.
4. Switching run_command to use posix_spawn on platforms where it is available, which would make it possible to use in a threaded context on those platforms.
Thoughts?
Jonathan
Quoted 6 lines · Jonathan Nieder
Quoted 3 lines · Brandon Williams
From what I can see, there are now no calls in the child process (after fork and before exec/_exit) which are not Async-Signal-Safe. This means that fork/exec in a threaded context should work without deadlock
Quoted 6 lines
I don't see why the former implies the latter. Can you explain further?
You already know my opinions about fork+threads by now. I continue to think this is heading in a direction of decreased maintainability that I dread.
I disagree here. No one thought it was a bad idea back when I was implementing grep to fork while running with threads. Id rather fix the problem in run_command so that we don't ever have to worry about this again, especially for contributors who are unaware of this issue while threading.
Quoted 10 lines
That's not to say that this is wasted work. I would prefer an approach like the following:
1. First, make grep work without running
fork()from threads. Jonathan Tan's approach would be one way to do this. Another way would be to simply disable threads in --recurse-submodules mode.This would be the first thing to do because it would make tests reliable again, without having to wait for deeper changes.
I'm not much of a fan of Jonathan Tan's suggestion, id rather just fix the problem at its root instead of adding in an additional hack. If this series crashes and burns then yes, lets just shut off threading in grep with --recurse-submodules is uses, otherwise this series will fix that case.
Quoted 5 lines
2. Then, teaching
run_commandto prepare the environment and do $PATH lookup before forking. This might make it possible forrun_commandto use vfork or might not.
Quoted 7 lines
3. Teaching
run_commandto delegate chdir to the child, using -C for git commands and 'cd' for shell commands, and using a shell where necessary where it didn't before.4. Switching
run_commandto useposix_spawnon platforms where it is available, which would make it possible to use in a threaded context on those platforms.
After this series it should be easy enough to convert to posix_spawn if someone wants to follow up with a patch to do that.
Signature · 1 line
Brandon Williams
[PATCH 7/6] run-command: block signals between fork and execve
In reply to Brandon WilliamsQuoted 6 lines · Brandon Williams
v2 does a bit of restructuring based on comments from reviewers. I took the patch by Eric and broke it up and tweaked it a bit to flow better with v2. I left out the part of Eric's patch which did signal manipulation as I wasn't experienced enough to know what it was doing or why it was necessary. Though I believe the code is structured in such a way that Eric could make another patch on top of this series with just the signal changes.
Yeah, I think a separate commit message might be necessary to explain the signal changes.
-------8<-----
Subject: [PATCH] run-command: block signals between fork and execve
Signal handlers of the parent firing in the forked child may have unintended side effects. Rather than auditing every signal handler we have and will ever have, block signals while forking and restore default signal handlers in the child before execve.
Restoring default signal handlers is required because execve does not unblock signals, it only restores default signal handlers. So we must restore them with sigprocmask before execve, leaving a window when signal handlers we control can fire in the child. Continue ignoring ignored signals, but reset the rest to defaults.
Similarly, disable pthread cancellation to future-proof our code in case we start using cancellation; as cancellation is implemented with signals in glibc.
---
Changes from my original in <20170411070534.GA10552@whir>:
- fixed typo in NO_PTHREADS code
- dropped fflush(NULL) before fork, consider us screwed anyways if the child uses stdio
- respect SIG_IGN in child; that seems to be the prevailing wisdom from reading https://ewontfix.com/7/ and process.c in ruby (git clone https://github.com/ruby/ruby.git)
run-command.c | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+)
run-command.c +69 -0
@@ -213,6 +213,7 @@ static int child_notifier = -1;enum child_errcode {CHILD_ERR_CHDIR , +CHILD_ERR_SIGPROCMASK ,CHILD_ERR_ENOENT ,CHILD_ERR_SILENT ,CHILD_ERR_ERRNO ,
@@ -277,6 +278,8 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)error_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir);break ; +case CHILD_ERR_SIGPROCMASK : +error_errno ("sigprocmask failed restoring signals" );case CHILD_ERR_ENOENT :error_errno ("cannot run %s" , cmd->argv[0 ]);break ;
@@ -388,6 +391,53 @@ static char **prep_childenv(const char *const *deltaenv) }#endif +struct atfork_state { +#ifndef NO_PTHREADS +int cs; +#endif +sigset_t old; +}; + +#ifndef NO_PTHREADS +static void bug_die (int err,const char *msg) +{ +if (err) { + errno = err; +die_errno ("BUG: %s" , msg); + } +} +#endif + +static void atfork_prepare (struct atfork_state *as) +{ +sigset_t all; + +if (sigfillset (&all)) +die_errno ("sigfillset" ); +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &all, &as->old)) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_sigmask (SIG_SETMASK , &all, &as->old), +"blocking all signals" ); +bug_die (pthread_setcancelstate (PTHREAD_CANCEL_DISABLE , &as->cs), +"disabling cancellation" ); +#endif +} + +static void atfork_parent (struct atfork_state *as) +{ +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &as->old,NULL )) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_setcancelstate (as->cs,NULL ), +"re-enabling cancellation" ); +bug_die (pthread_sigmask (SIG_SETMASK , &as->old,NULL ), +"restoring signal mask" ); +#endif +} +static inline void set_cloexec (int fd) {int flags =fcntl (fd,F_GETFD );
@@ -511,6 +561,7 @@ int start_command(struct child_process *cmd)char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;struct child_err cerr; +struct atfork_state as;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;
@@ -524,6 +575,7 @@ int start_command(struct child_process *cmd)prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env); +atfork_prepare (&as);/* * NOTE: In order to prevent deadlocking when using threads special
@@ -537,6 +589,7 @@ int start_command(struct child_process *cmd) cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) { +int sig;/* * Ensure the default die/error/warn routines do not get * called, they can take stdio locks and malloc.
@@ -584,6 +637,21 @@ int start_command(struct child_process *cmd)if (cmd->dir &&chdir (cmd->dir))child_die (CHILD_ERR_CHDIR ); +/* +* restore default signal handlers here, in case +* we catch a signal right before execve below +*/ +for (sig =1 ; sig <NSIG ; sig++) { +sighandler_t old =signal (sig,SIG_DFL ); + +/* ignored signals get reset to SIG_DFL on execve */ +if (old ==SIG_IGN ) +signal (sig,SIG_IGN ); + } + +if (sigprocmask (SIG_SETMASK , &as.old,NULL ) !=0 ) +child_die (CHILD_ERR_SIGPROCMASK ); +execve (argv.argv[0 ], (char *const *) argv.argv, (char *const *) childenv);
@@ -595,6 +663,7 @@ int start_command(struct child_process *cmd)child_die (CHILD_ERR_ERRNO ); } } +atfork_parent (&as);if (cmd->pid <0 )error_errno ("cannot fork() for %s" , cmd->argv[0 ]);else if (cmd->clean_on_exit)
Signature · 1 line
EW
Re: [PATCH 7/6] run-command: block signals between fork and execve
Quoted 10 lines · Eric Wong
Quoted 6 lines · Brandon Williams
v2 does a bit of restructuring based on comments from reviewers. I took the patch by Eric and broke it up and tweaked it a bit to flow better with v2. I left out the part of Eric's patch which did signal manipulation as I wasn't experienced enough to know what it was doing or why it was necessary. Though I believe the code is structured in such a way that Eric could make another patch on top of this series with just the signal changes.
Yeah, I think a separate commit message might be necessary to explain the signal changes.
Perfect! I'll carry the changes along in the reroll.
Quoted 167 lines
-------8<-----
Subject: [PATCH] run-command: block signals between fork and execveSignal handlers of the parent firing in the forked child may have unintended side effects. Rather than auditing every signal handler we have and will ever have, block signals while forking and restore default signal handlers in the child before execve.
Restoring default signal handlers is required because execve does not unblock signals, it only restores default signal handlers. So we must restore them with sigprocmask before execve, leaving a window when signal handlers we control can fire in the child. Continue ignoring ignored signals, but reset the rest to defaults.
Similarly, disable pthread cancellation to future-proof our code in case we start using cancellation; as cancellation is implemented with signals in glibc.
Signed-off-by Eric Wong <[email protected]>---
Changes from my original in <20170411070534.GA10552@whir>:- fixed typo in
NO_PTHREADScode- dropped fflush(NULL) before fork, consider us screwed anyways if the child uses stdio
- respect
SIG_IGNin child; that seems to be the prevailing wisdom from reading https://ewontfix.com/7/ andprocess.cin ruby (git clone https://github.com/ruby/ruby.git)run-command.c | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+)run-command.c +69
-0@@ -213,6 +213,7 @@ static int child_notifier = -1;enum child_errcode {CHILD_ERR_CHDIR , +CHILD_ERR_SIGPROCMASK ,CHILD_ERR_ENOENT ,CHILD_ERR_SILENT ,CHILD_ERR_ERRNO ,@@ -277,6 +278,8 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)error_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir);break ; +case CHILD_ERR_SIGPROCMASK : +error_errno ("sigprocmask failed restoring signals" );case CHILD_ERR_ENOENT :error_errno ("cannot run %s" , cmd->argv[0 ]);break ;@@ -388,6 +391,53 @@ static char **prep_childenv(const char *const *deltaenv) }#endif +struct atfork_state { +#ifndef NO_PTHREADS +int cs; +#endif +sigset_t old; +}; + +#ifndef NO_PTHREADS +static void bug_die (int err,const char *msg) +{ +if (err) { + errno = err; +die_errno ("BUG: %s" , msg); + } +} +#endif + +static void atfork_prepare (struct atfork_state *as) +{ +sigset_t all; + +if (sigfillset (&all)) +die_errno ("sigfillset" ); +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &all, &as->old)) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_sigmask (SIG_SETMASK , &all, &as->old), +"blocking all signals" ); +bug_die (pthread_setcancelstate (PTHREAD_CANCEL_DISABLE , &as->cs), +"disabling cancellation" ); +#endif +} + +static void atfork_parent (struct atfork_state *as) +{ +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &as->old,NULL )) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_setcancelstate (as->cs,NULL ), +"re-enabling cancellation" ); +bug_die (pthread_sigmask (SIG_SETMASK , &as->old,NULL ), +"restoring signal mask" ); +#endif +} +static inline void set_cloexec (int fd) {int flags =fcntl (fd,F_GETFD );@@ -511,6 +561,7 @@ int start_command(struct child_process *cmd)char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;struct child_err cerr; +struct atfork_state as;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;@@ -524,6 +575,7 @@ int start_command(struct child_process *cmd)prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env); +atfork_prepare (&as);/* * NOTE: In order to prevent deadlocking when using threads special @@ -537,6 +589,7 @@ int start_command(struct child_process *cmd) cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) { +int sig;/* * Ensure the default die/error/warn routines do not get * called, they can take stdio locks and malloc. @@ -584,6 +637,21 @@ int start_command(struct child_process *cmd)if (cmd->dir &&chdir (cmd->dir))child_die (CHILD_ERR_CHDIR ); +/* +* restore default signal handlers here, in case +* we catch a signal right before execve below +*/ +for (sig =1 ; sig <NSIG ; sig++) { +sighandler_t old =signal (sig,SIG_DFL ); + +/* ignored signals get reset to SIG_DFL on execve */ +if (old ==SIG_IGN ) +signal (sig,SIG_IGN ); + } + +if (sigprocmask (SIG_SETMASK , &as.old,NULL ) !=0 ) +child_die (CHILD_ERR_SIGPROCMASK ); +execve (argv.argv[0 ], (char *const *) argv.argv, (char *const *) childenv);@@ -595,6 +663,7 @@ int start_command(struct child_process *cmd)child_die (CHILD_ERR_ERRNO ); } } +atfork_parent (&as);if (cmd->pid <0 )error_errno ("cannot fork() for %s" , cmd->argv[0 ]);else if (cmd->clean_on_exit)Signature · 1 line
EW
Signature · 1 line
Brandon Williams
Re: [PATCH 7/6] run-command: block signals between fork and execve
In reply to Eric Wongrun-command.c static void child_err_spew(struct child_process *cmd, struct child_err *cerr) · line 282
Quoted 6 lines · Eric Wong
run-command.c +2
-0@@ -277,6 +278,8 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)error_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir);break ; +case CHILD_ERR_SIGPROCMASK : +error_errno ("sigprocmask failed restoring signals" );
missing a break statement here I'll add it in, in the re-roll.
Quoted 3 lines
run-command.c +0
-0case CHILD_ERR_ENOENT :error_errno ("cannot run %s" , cmd->argv[0 ]);break ;
Signature · 1 line
Brandon Williams
Re: [PATCH 7/6] run-command: block signals between fork and execve
Quoted 9 lines · Brandon Williams
run-command.c static void child_err_spew(struct child_process *cmd, struct child_err *cerr) · line 282
Quoted 6 lines · Eric Wong
run-command.c +2
-0@@ -277,6 +278,8 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)error_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir);break ; +case CHILD_ERR_SIGPROCMASK : +error_errno ("sigprocmask failed restoring signals" );missing a break statement here I'll add it in, in the re-roll.
Good catch, thanks!
Quoted 3 lines
Quoted 3 lines
run-command.c +0
-0case CHILD_ERR_ENOENT :error_errno ("cannot run %s" , cmd->argv[0 ]);break ;
Re: [PATCH 7/6] run-command: block signals between fork and execve
Quoted 11 lines · Eric Wong
Quoted 9 lines · Brandon Williams
run-command.c static void child_err_spew(struct child_process *cmd, struct child_err *cerr) · line 282
Quoted 6 lines · Eric Wong
run-command.c +2
-0@@ -277,6 +278,8 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)error_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir);break ; +case CHILD_ERR_SIGPROCMASK : +error_errno ("sigprocmask failed restoring signals" );missing a break statement here I'll add it in, in the re-roll.
Quoted 1 line
Good catch, thanks!
Actually, I now wonder if that should be die_errno instead. sigprocmask failures (EFAULT/EINVAL) would only be due to programmer error.
In one of my minor projects(*), I do something like this:
# define CHECK(type, expect, expr) do { \
type checkvar = (expr); \
assert(checkvar == (expect) && "BUG" && __FILE__ && __LINE__); \
} while (0)CHECK(int, 0, sigfillset(&fullset)); CHECK(int, 0, sigemptyset(&emptyset)); CHECK(int, 0, pthread_sigmask(SIG_SETMASK, &fullset, NULL));
Dunno if it's considered good style or not, here.
(*) git clone git://bogomips.org/cmogstored
v3 cover forking and threading
In reply to Brandon WilliamsChanges in v3:
* More error handling for dup2/close calls in the child
* Added a test in t0061 to test for regressions in run_command's ability to interpret scripts without a "#!" line
* In the event execve fails with ENOEXEC, attempt to exec one more time by invoking the shell to interpret the command
Brandon Williams (9):
t5550: use write_script to generate post-update hook
t0061: run_command executes scripts without a #! line
run-command: prepare command before forking
run-command: use the async-signal-safe execv instead of execvp
run-command: prepare child environment before forking
run-command: don't die in child when duping /dev/null
run-command: eliminate calls to error handling functions in child
run-command: handle dup2 and close errors in child
run-command: add note about forking and threading
Eric Wong (1):
run-command: block signals between fork and execve
run-command.c | 426 ++++++++++++++++++++++++++++++++++++--------- t/t0061-run-command.sh | 11 ++ t/t5550-http-fetch-dumb.sh | 5 +- 3 files changed, 357 insertions(+), 85 deletions(-)
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v3 1/10 t5550: use write_script to generate post-update hook
The post-update hooks created in t5550-http-fetch-dumb.sh is missing the "!#/bin/sh" line which can cause issues with portability. Instead create the hook using the 'write_script' function which includes the proper "#!" line.
--- t/t5550-http-fetch-dumb.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
t/t5550-http-fetch-dumb.sh +3 -2
@@ -20,8 +20,9 @@ test_expect_success 'create http-accessible bare repository with loose objects' (cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" && git config core.bare true && mkdir -p hooks && - echo "exec git update-server-info" >hooks/post-update && - chmod +x hooks/post-update && + write_script "hooks/post-update" <<-\EOF && + exec git update-server-info + EOF hooks/post-update ) && git remote add public "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v3 2/10 t0061: run_command executes scripts without a #! line
In reply to Brandon WilliamsAdd a test to 't0061-run-command.sh' to ensure that run_command can continue to execute scripts which don't include a '#!' line.
--- t/t0061-run-command.sh | 11 +++++++++++ 1 file changed, 11 insertions(+)
t/t0061-run-command.sh +11 -0
@@ -26,6 +26,17 @@ test_expect_success 'run_command can run a command' ' test_cmp empty err ' +test_expect_success 'run_command can run a script without a #! line' ' + cat >hello <<-\EOF && + cat hello-script + EOF + chmod +x hello && + test-run-command run-command ./hello >actual 2>err && + + test_cmp hello-script actual && + test_cmp empty err +' + test_expect_success POSIXPERM 'run_command reports EACCES' ' cat hello-script >hello.sh && chmod -x hello.sh &&
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v3 3/10 run-command: prepare command before forking
In reply to Brandon WilliamsAccording to [1] we need to only call async-signal-safe operations between fork and exec. Using malloc to build the argv array isn't async-signal-safe.
In order to avoid allocation between 'fork()' and 'exec()' prepare the argv array used in the exec call prior to forking the process.
[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html
--- run-command.c | 46 ++++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-)
run-command.c +26 -20
@@ -221,18 +221,6 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv) }#ifndef GIT_WINDOWS_NATIVE -static int execv_shell_cmd (const char **argv) -{ -struct argv_array nargv =ARGV_ARRAY_INIT ; -prepare_shell_cmd (&nargv, argv); -trace_argv_printf (nargv.argv,"trace: exec:" ); -sane_execvp (nargv.argv[0 ], (char **)nargv.argv); -argv_array_clear (&nargv); -return -1 ; -} -#endif - -#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ;static void notify_parent (void )
@@ -244,6 +232,21 @@ static void notify_parent(void)*/ xwrite(child_notifier, "", 1); } + +static void prepare_cmd (struct argv_array *out,const struct child_process *cmd) +{ +if (!cmd->argv[0 ]) +die ("BUG: command is empty" ); + +if (cmd->git_cmd) { +argv_array_push (out,"git" ); +argv_array_pushv (out, cmd->argv); + }else if (cmd->use_shell) { +prepare_shell_cmd (out, cmd->argv); + }else { +argv_array_pushv (out, cmd->argv); + } +}#endif static inline void set_cloexec (int fd)
@@ -372,9 +375,13 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +struct argv_array argv =ARGV_ARRAY_INIT ; +if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; +prepare_cmd (&argv, cmd); + cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) {
@@ -437,12 +444,9 @@ int start_command(struct child_process *cmd)unsetenv (*cmd->env); } } -if (cmd->git_cmd) -execv_git_cmd (cmd->argv); -else if (cmd->use_shell) -execv_shell_cmd (cmd->argv); -else -sane_execvp (cmd->argv[0 ], (char *const *) cmd->argv); + +sane_execvp (argv.argv[0 ], (char *const *) argv.argv); +if (errno ==ENOENT ) {if (!cmd->silent_exec_failure) error("cannot run %s: %s", cmd->argv[0],
@@ -458,7 +462,7 @@ int start_command(struct child_process *cmd)mark_child_for_cleanup (cmd->pid, cmd);/* -* Wait for child's execvp. If the execvp succeeds (or if fork() +* Wait for child's exec. If the exec succeeds (or if fork() * failed), EOF is seen immediately by the parent. Otherwise, the * child process sends a single byte. * Note that use of this infrastructure is completely advisory,
@@ -467,7 +471,7 @@ int start_command(struct child_process *cmd)close (notify_pipe[1 ]);if (read (notify_pipe[0 ], ¬ify_pipe[1 ],1 ) ==1 ) {/* -* At this point we know that fork() succeeded, but execvp() +* At this point we know that fork() succeeded, but exec() * failed. Errors have been reported to our stderr. */ wait_or_whine (cmd->pid, cmd->argv[0 ],0 );
@@ -475,6 +479,8 @@ int start_command(struct child_process *cmd) cmd->pid = -1; }close (notify_pipe[0 ]); + +argv_array_clear (&argv); }#else {
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v3 4/10 run-command: use the async-signal-safe execv instead of execvp
In reply to Brandon WilliamsConvert the function used to exec from 'execvp()' to 'execv()' as the (p) variant of exec isn't async-signal-safe and has the potential to call malloc during the path resolution it performs. Instead we simply do the path resolution ourselves during the preparation stage prior to forking. There also don't exist any portable (p) variants which also take in an environment to use in the exec'd process. This allows easy migration to using 'execve()' in a future patch.
Also, as noted in [1], in the event of an ENOEXEC the (p) variants of exec will attempt to execute the command by interpreting it with the 'sh' utility. To maintain this functionality, if 'execv()' fails with ENOEXEC, start_command will atempt to execute the command by interpreting it with 'sh'.
[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html
--- run-command.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
run-command.c +29 -1
@@ -238,6 +238,12 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)if (!cmd->argv[0 ])die ("BUG: command is empty" ); +/* +* Add SHELL_PATH so in the event exec fails with ENOEXEC we can +* attempt to interpret the command with 'sh'. +*/ +argv_array_push (out,SHELL_PATH ); +if (cmd->git_cmd) {argv_array_push (out,"git" );argv_array_pushv (out, cmd->argv);
@@ -246,6 +252,20 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) } else {argv_array_pushv (out, cmd->argv); } + +/* +* If there are no '/' characters in the command then perform a path +* lookup and use the resolved path as the command to exec. If there +* are no '/' characters or if the command wasn't found in the path, +* have exec attempt to invoke the command directly. +*/ +if (!strchr (out->argv[1 ],'/' )) { +char *program =locate_in_PATH (out->argv[1 ]); +if (program) { +free ((char *)out->argv[1 ]); + out->argv[1 ] = program; + } + } }#endif
@@ -445,7 +465,15 @@ int start_command(struct child_process *cmd) } } -sane_execvp (argv.argv[0 ], (char *const *) argv.argv); +/* +* Attempt to exec using the command and arguments starting at +* argv.argv[1]. argv.argv[0] contains SHELL_PATH which will +* be used in the event exec failed with ENOEXEC at which point +* we will try to interpret the command using 'sh'. +*/ +execv (argv.argv[1 ], (char *const *) argv.argv +1 ); +if (errno ==ENOEXEC ) +execv (argv.argv[0 ], (char *const *) argv.argv);if (errno ==ENOENT ) {if (!cmd->silent_exec_failure)
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v3 5/10 run-command: prepare child environment before forking
In reply to Brandon WilliamsIn order to avoid allocation between 'fork()' and 'exec()' prepare the environment to be used in the child process prior to forking.
Switch to using 'execve()' so that the construct child environment can used in the exec'd process.
--- run-command.c | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 76 insertions(+), 10 deletions(-)
run-command.c +76 -10
@@ -267,6 +267,75 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) } } } + +static int env_isequal (const char *e1,const char *e2) +{ +for (;;) { +char c1 = *e1++; +char c2 = *e2++; + c1 = (c1 =='=' ) ?'\0' :tolower (c1); + c2 = (c2 =='=' ) ?'\0' :tolower (c2); + +if (c1 != c2) +return 0 ; +if (c1 =='\0' ) +return 1 ; + } +} + +static int searchenv (char **env,const char *name) +{ +int pos =0 ; + +for (; env[pos]; pos++) +if (env_isequal (env[pos], name)) +break ; + +return pos; +} + +static int do_putenv (char **env,int env_nr,const char *name) +{ +int pos =searchenv (env, name); + +if (strchr (name,'=' )) { +/* ('key=value'), insert of replace entry */ +if (pos >= env_nr) + env_nr++; + env[pos] = (char *) name; + }else if (pos < env_nr) { +/* otherwise ('key') remove existing entry */ + env_nr--; +memmove (&env[pos], &env[pos +1 ], + (env_nr - pos) *sizeof (char *)); + env[env_nr] =NULL ; + } + +return env_nr; +} + +static char **prep_childenv (const char *const *deltaenv) +{ +char **childenv; +int childenv_nr =0 , childenv_alloc =0 ; +int i; + +for (i =0 ; environ[i]; i++) + childenv_nr++; +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_alloc++; +/* Add one for the NULL termination */ + childenv_alloc += childenv_nr +1 ; + + childenv =xcalloc (childenv_alloc,sizeof (char *)); +memcpy (childenv, environ, childenv_nr *sizeof (char *)); + +/* merge in deltaenv */ +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_nr =do_putenv (childenv, childenv_nr, deltaenv[i]); + +return childenv; +}#endif static inline void set_cloexec (int fd)
@@ -395,12 +464,14 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;prepare_cmd (&argv, cmd); + childenv =prep_childenv (cmd->env); cmd->pid =fork (); failed_errno = errno;
@@ -456,14 +527,6 @@ int start_command(struct child_process *cmd)if (cmd->dir &&chdir (cmd->dir))die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir); -if (cmd->env) { -for (; *cmd->env; cmd->env++) { -if (strchr (*cmd->env,'=' )) -putenv ((char *)*cmd->env); -else -unsetenv (*cmd->env); - } - }/* * Attempt to exec using the command and arguments starting at
@@ -471,9 +534,11 @@ int start_command(struct child_process *cmd)* be used in the event exec failed with ENOEXEC at which point * we will try to interpret the command using 'sh'. */ -execv (argv.argv[1 ], (char *const *) argv.argv +1 ); +execve (argv.argv[1 ], (char *const *) argv.argv +1 , + (char *const *) childenv);if (errno ==ENOEXEC ) -execv (argv.argv[0 ], (char *const *) argv.argv); +execve (argv.argv[0 ], (char *const *) argv.argv, + (char *const *) childenv);if (errno ==ENOENT ) {if (!cmd->silent_exec_failure)
@@ -509,6 +574,7 @@ int start_command(struct child_process *cmd)close (notify_pipe[0 ]);argv_array_clear (&argv); + free(childenv); }#else {
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v3 6/10 run-command: don't die in child when duping /dev/null
In reply to Brandon Williams--- run-command.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-)
run-command.c +13 -15
@@ -117,18 +117,6 @@ static inline void close_pair(int fd[2])close (fd[1 ]); } -#ifndef GIT_WINDOWS_NATIVE -static inline void dup_devnull (int to) -{ -int fd =open ("/dev/null" ,O_RDWR ); -if (fd <0 ) -die_errno (_ ("open /dev/null failed" )); -if (dup2 (fd, to) <0 ) -die_errno (_ ("dup2(%d,%d) failed" ), fd, to); -close (fd); -} -#endif -static char *locate_in_PATH (const char *file) {const char *p =getenv ("PATH" );
@@ -464,12 +452,20 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +int null_fd =-1 ;char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; +if (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) { + null_fd =open ("/dev/null" ,O_RDWR |O_CLOEXEC |O_NONBLOCK ); +if (null_fd <0 ) +die_errno (_ ("open /dev/null failed" )); +set_cloexec (null_fd); + } +prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env);
@@ -493,7 +489,7 @@ int start_command(struct child_process *cmd)atexit (notify_parent);if (cmd->no_stdin) -dup_devnull (0 ); +dup2 (null_fd,0 );else if (need_in) {dup2 (fdin[0 ],0 );close_pair (fdin);
@@ -503,7 +499,7 @@ int start_command(struct child_process *cmd) }if (cmd->no_stderr) -dup_devnull (2 ); +dup2 (null_fd,2 );else if (need_err) {dup2 (fderr[1 ],2 );close_pair (fderr);
@@ -513,7 +509,7 @@ int start_command(struct child_process *cmd) }if (cmd->no_stdout) -dup_devnull (1 ); +dup2 (null_fd,1 );else if (cmd->stdout_to_stderr)dup2 (2 ,1 );else if (need_out) {
@@ -573,6 +569,8 @@ int start_command(struct child_process *cmd) }close (notify_pipe[0 ]); +if (null_fd >=0 ) +close (null_fd);argv_array_clear (&argv);free (childenv); }
Signature · 1 line
2.12.2.762.g0e3151a226-goog
Re: [PATCH v3 06/10] run-command: don't die in child when duping /dev/null
run-command.c int start_command(struct child_process *cmd) · line 463
Quoted 2 lines · Brandon Williams
run-command.c +2
-0+if (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) { + null_fd =open ("/dev/null" ,O_RDWR |O_CLOEXEC |O_NONBLOCK );
O_NONBLOCK? This was in my original patch, too :x Wow, I wonder what I was smoking that day...
Re: [PATCH v3 06/10] run-command: don't die in child when duping /dev/null
Quoted 6 lines · Eric Wong
run-command.c int start_command(struct child_process *cmd) · line 463
Quoted 2 lines · Brandon Williams
run-command.c +2
-0+if (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) { + null_fd =open ("/dev/null" ,O_RDWR |O_CLOEXEC |O_NONBLOCK );
O_NONBLOCK? This was in my original patch, too :x Wow, I wonder what I was smoking that day...
And I apparently wasn't thinking enough to catch that! I'll fix that.
Signature · 1 line
Brandon Williams
PATCH v3 7/10 run-command: eliminate calls to error handling functions in child
In reply to Brandon WilliamsAll of our standard error handling paths have the potential to call malloc or take stdio locks; so we must avoid them inside the forked child.
Instead, the child only writes an 8 byte struct atomically to the parent through the notification pipe to propagate an error. All user-visible error reporting happens from the parent; even avoiding functions like atexit(3) and exit(3).
--- run-command.c | 121 ++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 89 insertions(+), 32 deletions(-)
run-command.c +89 -32
@@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ; -static void notify_parent (void ) +enum child_errcode { +CHILD_ERR_CHDIR , +CHILD_ERR_ENOENT , +CHILD_ERR_SILENT , +CHILD_ERR_ERRNO , +}; + +struct child_err { +enum child_errcode err; +int syserr;/* errno */ +}; + +static void child_die (enum child_errcode err) { -/* -* execvp failed. If possible, we'd like to let start_command -* know, so failures like ENOENT can be handled right away; but -* otherwise, finish_command will still report the error. -*/ -xwrite (child_notifier,"" ,1 ); +struct child_err buf; + + buf.err = err; + buf.syserr = errno; + +/* write(2) on buf smaller than PIPE_BUF (min 512) is atomic: */ +xwrite (child_notifier, &buf,sizeof (buf)); +_exit (1 ); +} + +/* +* parent will make it look like the child spewed a fatal error and died +* this is needed to prevent changes to t0061. +*/ +static void fake_fatal (const char *err,va_list params) +{ +vreportf ("fatal: " , err, params); +} + +static void child_error_fn (const char *err,va_list params) +{ +const char msg[] ="error() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +} + +static void child_warn_fn (const char *err,va_list params) +{ +const char msg[] ="warn() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +} + +static void NORETURN child_die_fn(const char *err, va_list params) +{ +const char msg[] ="die() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +_exit (2 ); +} + +/* this runs in the parent process */ +static void child_err_spew (struct child_process *cmd,struct child_err *cerr) +{ +static void (*old_errfn)(const char *err,va_list params); + + old_errfn =get_error_routine (); +set_error_routine (fake_fatal); + errno = cerr->syserr; + +switch (cerr->err) { +case CHILD_ERR_CHDIR : +error_errno ("exec '%s': cd to '%s' failed" , + cmd->argv[0 ], cmd->dir); +break ; +case CHILD_ERR_ENOENT : +error_errno ("cannot run %s" , cmd->argv[0 ]); +break ; +case CHILD_ERR_SILENT : +break ; +case CHILD_ERR_ERRNO : +error_errno ("cannot exec '%s'" , cmd->argv[0 ]); +break ; + } +set_error_routine (old_errfn); }static void prepare_cmd (struct argv_array *out,const struct child_process *cmd)
@@ -361,13 +429,6 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal) code +=128 ; }else if (WIFEXITED (status)) { code =WEXITSTATUS (status); -/* -* Convert special exit code when execvp failed. -*/ -if (code ==127 ) { - code =-1 ; - failed_errno =ENOENT ; - } }else {error ("waitpid is confused (%s)" , argv0); }
@@ -455,6 +516,7 @@ int start_command(struct child_process *cmd)int null_fd =-1 ;char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ; +struct child_err cerr;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;
@@ -473,20 +535,16 @@ int start_command(struct child_process *cmd) failed_errno = errno;if (!cmd->pid) {/* -* Redirect the channel to write syscall error messages to -* before redirecting the process's stderr so that all die() -* in subsequent call paths use the parent's stderr. +* Ensure the default die/error/warn routines do not get +* called, they can take stdio locks and malloc. */ -if (cmd->no_stderr || need_err) { -int child_err =dup (2 ); -set_cloexec (child_err); -set_error_handle (fdopen (child_err,"w" )); - } +set_die_routine (child_die_fn); +set_error_routine (child_error_fn); +set_warn_routine (child_warn_fn);close (notify_pipe[0 ]);set_cloexec (notify_pipe[1 ]); child_notifier = notify_pipe[1 ]; -atexit (notify_parent);if (cmd->no_stdin)dup2 (null_fd,0 );
@@ -521,8 +579,7 @@ int start_command(struct child_process *cmd) }if (cmd->dir &&chdir (cmd->dir)) -die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], - cmd->dir); +child_die (CHILD_ERR_CHDIR );/* * Attempt to exec using the command and arguments starting at
@@ -537,12 +594,11 @@ int start_command(struct child_process *cmd) (char *const *) childenv);if (errno ==ENOENT ) { -if (!cmd->silent_exec_failure) -error ("cannot run %s: %s" , cmd->argv[0 ], -strerror (ENOENT )); -exit (127 ); +if (cmd->silent_exec_failure) +child_die (CHILD_ERR_SILENT ); +child_die (CHILD_ERR_ENOENT ); }else { -die_errno ("cannot exec '%s'" , cmd->argv[0 ]); +child_die (CHILD_ERR_ERRNO ); } }if (cmd->pid <0 )
@@ -553,17 +609,18 @@ int start_command(struct child_process *cmd)/* * Wait for child's exec. If the exec succeeds (or if fork() * failed), EOF is seen immediately by the parent. Otherwise, the -* child process sends a single byte. +* child process sends a child_err struct. * Note that use of this infrastructure is completely advisory, * therefore, we keep error checks minimal. */ close (notify_pipe[1 ]); -if (read (notify_pipe[0 ], ¬ify_pipe[1 ],1 ) ==1 ) { +if (xread (notify_pipe[0 ], &cerr,sizeof (cerr)) ==sizeof (cerr)) {/* * At this point we know that fork() succeeded, but exec() * failed. Errors have been reported to our stderr. */ wait_or_whine (cmd->pid, cmd->argv[0 ],0 ); +child_err_spew (cmd, &cerr); failed_errno = errno; cmd->pid =-1 ; }
Signature · 1 line
2.12.2.762.g0e3151a226-goog
Re: [PATCH v3 07/10] run-command: eliminate calls to error handling functions in child
run-command.c static const char **prepare_shell_cmd(struct argv_array *out, const char **argv) · line 219
Quoted 12 lines · Brandon Williams
+++
b/run-command.crun-command.c +6
-1@@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ; -static void notify_parent (void ) +enum child_errcode { +CHILD_ERR_CHDIR , +CHILD_ERR_ENOENT , +CHILD_ERR_SILENT , +CHILD_ERR_ERRNO , +};
I realize I introduced this in my original, but trailing commas on the last enum value might not be portable. Checking other enum usages in our tree suggests we omit the last comma.
Re: [PATCH v3 07/10] run-command: eliminate calls to error handling functions in child
Quoted 6 lines · Eric Wong
Quoted 5 lines · Brandon Williams
+++
b/run-command.crun-command.c +0
-0@@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ;
Quoted 11 lines
run-command.c static const char **prepare_shell_cmd(struct argv_array *out, const char **argv) · line 219
Quoted 7 lines
run-command.c +6
-1-static void notify_parent (void ) +enum child_errcode { +CHILD_ERR_CHDIR , +CHILD_ERR_ENOENT , +CHILD_ERR_SILENT , +CHILD_ERR_ERRNO , +};I realize I introduced this in my original, but trailing commas on the last enum value might not be portable. Checking other enum usages in our tree suggests we omit the last comma.
While I realize its not portal, I think there are other places that do this same thing. I think it means we can move to a newer standard of C!
In all seriousness though I'll drop the trailing comma.
Signature · 1 line
Brandon Williams
PATCH v3 8/10 run-command: handle dup2 and close errors in child
In reply to Brandon Williams--- run-command.c | 58 ++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 42 insertions(+), 16 deletions(-)
run-command.c +42 -16
@@ -213,6 +213,8 @@ static int child_notifier = -1;enum child_errcode {CHILD_ERR_CHDIR , +CHILD_ERR_DUP2 , +CHILD_ERR_CLOSE ,CHILD_ERR_ENOENT ,CHILD_ERR_SILENT ,CHILD_ERR_ERRNO ,
@@ -235,6 +237,24 @@ static void child_die(enum child_errcode err)_exit (1 ); } +static void child_dup2 (int fd,int to) +{ +if (dup2 (fd, to) <0 ) +child_die (CHILD_ERR_DUP2 ); +} + +static void child_close (int fd) +{ +if (close (fd)) +child_die (CHILD_ERR_CLOSE ); +} + +static void child_close_pair (int fd[2 ]) +{ +child_close (fd[0 ]); +child_close (fd[1 ]); +} +/* * parent will make it look like the child spewed a fatal error and died * this is needed to prevent changes to t0061.
@@ -277,6 +297,12 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)error_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir);break ; +case CHILD_ERR_DUP2 : +error_errno ("dup2() in child failed" ); +break ; +case CHILD_ERR_CLOSE : +error_errno ("close() in child failed" ); +break ;case CHILD_ERR_ENOENT :error_errno ("cannot run %s" , cmd->argv[0 ]);break ;
@@ -547,35 +573,35 @@ int start_command(struct child_process *cmd) child_notifier = notify_pipe[1 ];if (cmd->no_stdin) -dup2 (null_fd,0 ); +child_dup2 (null_fd,0 );else if (need_in) { -dup2 (fdin[0 ],0 ); -close_pair (fdin); +child_dup2 (fdin[0 ],0 ); +child_close_pair (fdin); }else if (cmd->in) { -dup2 (cmd->in,0 ); -close (cmd->in); +child_dup2 (cmd->in,0 ); +child_close (cmd->in); }if (cmd->no_stderr) -dup2 (null_fd,2 ); +child_dup2 (null_fd,2 );else if (need_err) { -dup2 (fderr[1 ],2 ); -close_pair (fderr); +child_dup2 (fderr[1 ],2 ); +child_close_pair (fderr); }else if (cmd->err >1 ) { -dup2 (cmd->err,2 ); -close (cmd->err); +child_dup2 (cmd->err,2 ); +child_close (cmd->err); }if (cmd->no_stdout) -dup2 (null_fd,1 ); +child_dup2 (null_fd,1 );else if (cmd->stdout_to_stderr) -dup2 (2 ,1 ); +child_dup2 (2 ,1 );else if (need_out) { -dup2 (fdout[1 ],1 ); -close_pair (fdout); +child_dup2 (fdout[1 ],1 ); +child_close_pair (fdout); }else if (cmd->out >1 ) { -dup2 (cmd->out,1 ); -close (cmd->out); +child_dup2 (cmd->out,1 ); +child_close (cmd->out); }if (cmd->dir &&chdir (cmd->dir))
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v3 9/10 run-command: add note about forking and threading
In reply to Brandon WilliamsAll non-Async-Signal-Safe functions (e.g. malloc and die) were removed between 'fork' and 'exec' in start_command in order to avoid potential deadlocking when forking while multiple threads are running. This deadlocking is possible when a thread (other than the one forking) has acquired a lock and didn't get around to releasing it before the fork. This leaves the lock in a locked state in the resulting process with no hope of it ever being released.
Add a note describing this potential pitfall before the call to 'fork()' so people working in this section of the code know to only use Async-Signal-Safe functions in the child process.
--- run-command.c | 9 +++++++++ 1 file changed, 9 insertions(+)
run-command.c +9 -0
@@ -557,6 +557,15 @@ int start_command(struct child_process *cmd)prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env); +/* +* NOTE: In order to prevent deadlocking when using threads special +* care should be taken with the function calls made in between the +* fork() and exec() calls. No calls should be made to functions which +* require acquiring a lock (e.g. malloc) as the lock could have been +* held by another thread at the time of forking, causing the lock to +* never be released in the child process. This means only +* Async-Signal-Safe functions are permitted in the child. +*/ cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) {
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v3 10/10 run-command: block signals between fork and execve
In reply to Brandon WilliamsFrom: Eric Wong <[email protected]>
Signal handlers of the parent firing in the forked child may have unintended side effects. Rather than auditing every signal handler we have and will ever have, block signals while forking and restore default signal handlers in the child before execve.
Restoring default signal handlers is required because execve does not unblock signals, it only restores default signal handlers. So we must restore them with sigprocmask before execve, leaving a window when signal handlers we control can fire in the child. Continue ignoring ignored signals, but reset the rest to defaults.
Similarly, disable pthread cancellation to future-proof our code in case we start using cancellation; as cancellation is implemented with signals in glibc.
--- run-command.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+)
run-command.c +70 -0
@@ -215,6 +215,7 @@ enum child_errcode {CHILD_ERR_CHDIR ,CHILD_ERR_DUP2 ,CHILD_ERR_CLOSE , +CHILD_ERR_SIGPROCMASK ,CHILD_ERR_ENOENT ,CHILD_ERR_SILENT , CHILD_ERR_ERRNO,
@@ -303,6 +304,9 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)case CHILD_ERR_CLOSE :error_errno ("close() in child failed" );break ; +case CHILD_ERR_SIGPROCMASK : +error_errno ("sigprocmask failed restoring signals" ); +break ;case CHILD_ERR_ENOENT :error_errno ("cannot run %s" , cmd->argv[0 ]);break ;
@@ -420,6 +424,53 @@ static char **prep_childenv(const char *const *deltaenv) }#endif +struct atfork_state { +#ifndef NO_PTHREADS +int cs; +#endif +sigset_t old; +}; + +#ifndef NO_PTHREADS +static void bug_die (int err,const char *msg) +{ +if (err) { + errno = err; +die_errno ("BUG: %s" , msg); + } +} +#endif + +static void atfork_prepare (struct atfork_state *as) +{ +sigset_t all; + +if (sigfillset (&all)) +die_errno ("sigfillset" ); +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &all, &as->old)) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_sigmask (SIG_SETMASK , &all, &as->old), +"blocking all signals" ); +bug_die (pthread_setcancelstate (PTHREAD_CANCEL_DISABLE , &as->cs), +"disabling cancellation" ); +#endif +} + +static void atfork_parent (struct atfork_state *as) +{ +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &as->old,NULL )) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_setcancelstate (as->cs,NULL ), +"re-enabling cancellation" ); +bug_die (pthread_sigmask (SIG_SETMASK , &as->old,NULL ), +"restoring signal mask" ); +#endif +} +static inline void set_cloexec (int fd) {int flags =fcntl (fd,F_GETFD );
@@ -543,6 +594,7 @@ int start_command(struct child_process *cmd)char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;struct child_err cerr; +struct atfork_state as;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;
@@ -556,6 +608,7 @@ int start_command(struct child_process *cmd)prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env); +atfork_prepare (&as);/* * NOTE: In order to prevent deadlocking when using threads special
@@ -569,6 +622,7 @@ int start_command(struct child_process *cmd) cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) { +int sig;/* * Ensure the default die/error/warn routines do not get * called, they can take stdio locks and malloc.
@@ -617,6 +671,21 @@ int start_command(struct child_process *cmd)child_die (CHILD_ERR_CHDIR );/* +* restore default signal handlers here, in case +* we catch a signal right before execve below +*/ +for (sig =1 ; sig <NSIG ; sig++) { +sighandler_t old =signal (sig,SIG_DFL ); + +/* ignored signals get reset to SIG_DFL on execve */ +if (old ==SIG_IGN ) +signal (sig,SIG_IGN ); + } + +if (sigprocmask (SIG_SETMASK , &as.old,NULL ) !=0 ) +child_die (CHILD_ERR_SIGPROCMASK ); + +/* * Attempt to exec using the command and arguments starting at * argv.argv[1]. argv.argv[0] contains SHELL_PATH which will * be used in the event exec failed with ENOEXEC at which point
@@ -636,6 +705,7 @@ int start_command(struct child_process *cmd)child_die (CHILD_ERR_ERRNO ); } } +atfork_parent (&as);if (cmd->pid <0 )error_errno ("cannot fork() for %s" , cmd->argv[0 ]);else if (cmd->clean_on_exit)
Signature · 1 line
2.12.2.762.g0e3151a226-goog
Re: [PATCH v3 10/10] run-command: block signals between fork and execve
run-command.c int start_command(struct child_process *cmd) · line 678
Quoted 6 lines · Brandon Williams
run-command.c +5
-0/* +* restore default signal handlers here, in case +* we catch a signal right before execve below +*/ +for (sig =1 ; sig <NSIG ; sig++) { +sighandler_t old =signal (sig,SIG_DFL );
So sighandler_t doesn't work on macOS. Is there a more portable lib that needs to be included for this to work?
Quoted 9 lines
run-command.c +9
-0+ +/* ignored signals get reset to SIG_DFL on execve */ +if (old ==SIG_IGN ) +signal (sig,SIG_IGN ); + } + +if (sigprocmask (SIG_SETMASK , &as.old,NULL ) !=0 ) +child_die (CHILD_ERR_SIGPROCMASK ); +
Signature · 1 line
Brandon Williams
Re: [PATCH v3 10/10] run-command: block signals between fork and execve
Quoted 10 lines · Brandon Williams
Review
Quoted 6 lines · Brandon Williams
(quoted diff) +5
-0/* + * restore default signal handlers here, in case + * we catch a signal right before execve below + */ + for (sig = 1; sig < NSIG; sig++) { + sighandler_t old = signal(sig, SIG_DFL);So
sighandler_tdoesn't work on macOS. Is there a more portable lib that needs to be included for this to work?
Oops, maybe this works (only tested on GNU/Linux):
run-command.c +1 -1
@@ -675,7 +675,7 @@ int start_command(struct child_process *cmd)* we catch a signal right before execve below */ for (sig =1 ; sig <NSIG ; sig++) { -sighandler_t old =signal (sig,SIG_DFL ); +void (*old)(int ) =signal (sig,SIG_DFL );/* ignored signals get reset to SIG_DFL on execve */ if (old ==SIG_IGN )
Otherwise, maybe just casting to 'void *' is OK:
void *old = (void *)signal(sig, SIG_DFL); if (old == (void *)SIG_IGN) ...
void *old = signal(sig, SIG_DFL);
v4 cover forking and threading
In reply to Brandon Williamsv4 fixes a few minor compatability issues:
* add 'extern' reference for 'environ'
* small portability change with the signal handeling
* remove trailing ',' in enum
* null_fd not opened with O_NONBLOCK
Brandon Williams (9):
t5550: use write_script to generate post-update hook
t0061: run_command executes scripts without a #! line
run-command: prepare command before forking
run-command: use the async-signal-safe execv instead of execvp
run-command: prepare child environment before forking
run-command: don't die in child when duping /dev/null
run-command: eliminate calls to error handling functions in child
run-command: handle dup2 and close errors in child
run-command: add note about forking and threading
Eric Wong (1):
run-command: block signals between fork and execve
run-command.c | 425 ++++++++++++++++++++++++++++++++++++--------- t/t0061-run-command.sh | 11 ++ t/t5550-http-fetch-dumb.sh | 5 +- 3 files changed, 356 insertions(+), 85 deletions(-)
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v4 1/10 t5550: use write_script to generate post-update hook
The post-update hooks created in t5550-http-fetch-dumb.sh is missing the "!#/bin/sh" line which can cause issues with portability. Instead create the hook using the 'write_script' function which includes the proper "#!" line.
--- t/t5550-http-fetch-dumb.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
t/t5550-http-fetch-dumb.sh +3 -2
@@ -20,8 +20,9 @@ test_expect_success 'create http-accessible bare repository with loose objects' (cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" && git config core.bare true && mkdir -p hooks && - echo "exec git update-server-info" >hooks/post-update && - chmod +x hooks/post-update && + write_script "hooks/post-update" <<-\EOF && + exec git update-server-info + EOF hooks/post-update ) && git remote add public "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v4 2/10 t0061: run_command executes scripts without a #! line
In reply to Brandon WilliamsAdd a test to 't0061-run-command.sh' to ensure that run_command can continue to execute scripts which don't include a '#!' line.
--- t/t0061-run-command.sh | 11 +++++++++++ 1 file changed, 11 insertions(+)
t/t0061-run-command.sh +11 -0
@@ -26,6 +26,17 @@ test_expect_success 'run_command can run a command' ' test_cmp empty err ' +test_expect_success 'run_command can run a script without a #! line' ' + cat >hello <<-\EOF && + cat hello-script + EOF + chmod +x hello && + test-run-command run-command ./hello >actual 2>err && + + test_cmp hello-script actual && + test_cmp empty err +' + test_expect_success POSIXPERM 'run_command reports EACCES' ' cat hello-script >hello.sh && chmod -x hello.sh &&
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v4 3/10 run-command: prepare command before forking
In reply to Brandon WilliamsAccording to [1] we need to only call async-signal-safe operations between fork and exec. Using malloc to build the argv array isn't async-signal-safe.
In order to avoid allocation between 'fork()' and 'exec()' prepare the argv array used in the exec call prior to forking the process.
[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html
--- run-command.c | 46 ++++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-)
run-command.c +26 -20
@@ -221,18 +221,6 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv) }#ifndef GIT_WINDOWS_NATIVE -static int execv_shell_cmd (const char **argv) -{ -struct argv_array nargv =ARGV_ARRAY_INIT ; -prepare_shell_cmd (&nargv, argv); -trace_argv_printf (nargv.argv,"trace: exec:" ); -sane_execvp (nargv.argv[0 ], (char **)nargv.argv); -argv_array_clear (&nargv); -return -1 ; -} -#endif - -#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ;static void notify_parent (void )
@@ -244,6 +232,21 @@ static void notify_parent(void)*/ xwrite(child_notifier, "", 1); } + +static void prepare_cmd (struct argv_array *out,const struct child_process *cmd) +{ +if (!cmd->argv[0 ]) +die ("BUG: command is empty" ); + +if (cmd->git_cmd) { +argv_array_push (out,"git" ); +argv_array_pushv (out, cmd->argv); + }else if (cmd->use_shell) { +prepare_shell_cmd (out, cmd->argv); + }else { +argv_array_pushv (out, cmd->argv); + } +}#endif static inline void set_cloexec (int fd)
@@ -372,9 +375,13 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +struct argv_array argv =ARGV_ARRAY_INIT ; +if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; +prepare_cmd (&argv, cmd); + cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) {
@@ -437,12 +444,9 @@ int start_command(struct child_process *cmd)unsetenv (*cmd->env); } } -if (cmd->git_cmd) -execv_git_cmd (cmd->argv); -else if (cmd->use_shell) -execv_shell_cmd (cmd->argv); -else -sane_execvp (cmd->argv[0 ], (char *const *) cmd->argv); + +sane_execvp (argv.argv[0 ], (char *const *) argv.argv); +if (errno ==ENOENT ) {if (!cmd->silent_exec_failure) error("cannot run %s: %s", cmd->argv[0],
@@ -458,7 +462,7 @@ int start_command(struct child_process *cmd)mark_child_for_cleanup (cmd->pid, cmd);/* -* Wait for child's execvp. If the execvp succeeds (or if fork() +* Wait for child's exec. If the exec succeeds (or if fork() * failed), EOF is seen immediately by the parent. Otherwise, the * child process sends a single byte. * Note that use of this infrastructure is completely advisory,
@@ -467,7 +471,7 @@ int start_command(struct child_process *cmd)close (notify_pipe[1 ]);if (read (notify_pipe[0 ], ¬ify_pipe[1 ],1 ) ==1 ) {/* -* At this point we know that fork() succeeded, but execvp() +* At this point we know that fork() succeeded, but exec() * failed. Errors have been reported to our stderr. */ wait_or_whine (cmd->pid, cmd->argv[0 ],0 );
@@ -475,6 +479,8 @@ int start_command(struct child_process *cmd) cmd->pid = -1; }close (notify_pipe[0 ]); + +argv_array_clear (&argv); }#else {
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v4 4/10 run-command: use the async-signal-safe execv instead of execvp
In reply to Brandon WilliamsConvert the function used to exec from 'execvp()' to 'execv()' as the (p) variant of exec isn't async-signal-safe and has the potential to call malloc during the path resolution it performs. Instead we simply do the path resolution ourselves during the preparation stage prior to forking. There also don't exist any portable (p) variants which also take in an environment to use in the exec'd process. This allows easy migration to using 'execve()' in a future patch.
Also, as noted in [1], in the event of an ENOEXEC the (p) variants of exec will attempt to execute the command by interpreting it with the 'sh' utility. To maintain this functionality, if 'execv()' fails with ENOEXEC, start_command will atempt to execute the command by interpreting it with 'sh'.
[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html
--- run-command.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
run-command.c +29 -1
@@ -238,6 +238,12 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)if (!cmd->argv[0 ])die ("BUG: command is empty" ); +/* +* Add SHELL_PATH so in the event exec fails with ENOEXEC we can +* attempt to interpret the command with 'sh'. +*/ +argv_array_push (out,SHELL_PATH ); +if (cmd->git_cmd) {argv_array_push (out,"git" );argv_array_pushv (out, cmd->argv);
@@ -246,6 +252,20 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) } else {argv_array_pushv (out, cmd->argv); } + +/* +* If there are no '/' characters in the command then perform a path +* lookup and use the resolved path as the command to exec. If there +* are no '/' characters or if the command wasn't found in the path, +* have exec attempt to invoke the command directly. +*/ +if (!strchr (out->argv[1 ],'/' )) { +char *program =locate_in_PATH (out->argv[1 ]); +if (program) { +free ((char *)out->argv[1 ]); + out->argv[1 ] = program; + } + } }#endif
@@ -445,7 +465,15 @@ int start_command(struct child_process *cmd) } } -sane_execvp (argv.argv[0 ], (char *const *) argv.argv); +/* +* Attempt to exec using the command and arguments starting at +* argv.argv[1]. argv.argv[0] contains SHELL_PATH which will +* be used in the event exec failed with ENOEXEC at which point +* we will try to interpret the command using 'sh'. +*/ +execv (argv.argv[1 ], (char *const *) argv.argv +1 ); +if (errno ==ENOEXEC ) +execv (argv.argv[0 ], (char *const *) argv.argv);if (errno ==ENOENT ) {if (!cmd->silent_exec_failure)
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v4 5/10 run-command: prepare child environment before forking
In reply to Brandon WilliamsIn order to avoid allocation between 'fork()' and 'exec()' prepare the environment to be used in the child process prior to forking.
Switch to using 'execve()' so that the construct child environment can used in the exec'd process.
--- run-command.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 77 insertions(+), 10 deletions(-)
run-command.c +77 -10
@@ -267,6 +267,76 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) } } } + +static int env_isequal (const char *e1,const char *e2) +{ +for (;;) { +char c1 = *e1++; +char c2 = *e2++; + c1 = (c1 =='=' ) ?'\0' :tolower (c1); + c2 = (c2 =='=' ) ?'\0' :tolower (c2); + +if (c1 != c2) +return 0 ; +if (c1 =='\0' ) +return 1 ; + } +} + +static int searchenv (char **env,const char *name) +{ +int pos =0 ; + +for (; env[pos]; pos++) +if (env_isequal (env[pos], name)) +break ; + +return pos; +} + +static int do_putenv (char **env,int env_nr,const char *name) +{ +int pos =searchenv (env, name); + +if (strchr (name,'=' )) { +/* ('key=value'), insert of replace entry */ +if (pos >= env_nr) + env_nr++; + env[pos] = (char *) name; + }else if (pos < env_nr) { +/* otherwise ('key') remove existing entry */ + env_nr--; +memmove (&env[pos], &env[pos +1 ], + (env_nr - pos) *sizeof (char *)); + env[env_nr] =NULL ; + } + +return env_nr; +} + +static char **prep_childenv (const char *const *deltaenv) +{ +extern char **environ; +char **childenv; +int childenv_nr =0 , childenv_alloc =0 ; +int i; + +for (i =0 ; environ[i]; i++) + childenv_nr++; +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_alloc++; +/* Add one for the NULL termination */ + childenv_alloc += childenv_nr +1 ; + + childenv =xcalloc (childenv_alloc,sizeof (char *)); +memcpy (childenv, environ, childenv_nr *sizeof (char *)); + +/* merge in deltaenv */ +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_nr =do_putenv (childenv, childenv_nr, deltaenv[i]); + +return childenv; +}#endif static inline void set_cloexec (int fd)
@@ -395,12 +465,14 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;prepare_cmd (&argv, cmd); + childenv =prep_childenv (cmd->env); cmd->pid =fork (); failed_errno = errno;
@@ -456,14 +528,6 @@ int start_command(struct child_process *cmd)if (cmd->dir &&chdir (cmd->dir))die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir); -if (cmd->env) { -for (; *cmd->env; cmd->env++) { -if (strchr (*cmd->env,'=' )) -putenv ((char *)*cmd->env); -else -unsetenv (*cmd->env); - } - }/* * Attempt to exec using the command and arguments starting at
@@ -471,9 +535,11 @@ int start_command(struct child_process *cmd)* be used in the event exec failed with ENOEXEC at which point * we will try to interpret the command using 'sh'. */ -execv (argv.argv[1 ], (char *const *) argv.argv +1 ); +execve (argv.argv[1 ], (char *const *) argv.argv +1 , + (char *const *) childenv);if (errno ==ENOEXEC ) -execv (argv.argv[0 ], (char *const *) argv.argv); +execve (argv.argv[0 ], (char *const *) argv.argv, + (char *const *) childenv);if (errno ==ENOENT ) {if (!cmd->silent_exec_failure)
@@ -509,6 +575,7 @@ int start_command(struct child_process *cmd)close (notify_pipe[0 ]);argv_array_clear (&argv); + free(childenv); }#else {
Signature · 1 line
2.12.2.762.g0e3151a226-goog
Re: [PATCH v4 05/10] run-command: prepare child environment before forking
+Cc Karsten for comments below...
Quoted 17 lines · Brandon Williams
In order to avoid allocation between '
fork()' and 'exec()' prepare the environment to be used in the child process prior to forking.Switch to using '
execve()' so that the construct child environment can used in the exec'd process.Signed-off-by Brandon Williams <[email protected]>--- run-command.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 77 insertions(+), 10 deletions(-)run-command.c +0
-0@@ -267,6 +267,76 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) }
run-command.c static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) · line 277
Quoted 10 lines
run-command.c +8
-0} } + +static int env_isequal (const char *e1,const char *e2) +{ +for (;;) { +char c1 = *e1++; +char c2 = *e2++; + c1 = (c1 =='=' ) ?'\0' :tolower (c1); + c2 = (c2 =='=' ) ?'\0' :tolower (c2);
Dealing with C strings scares me so maybe I'm misreading;
but: why is this comparison case-insensitive?
Reading on...
Quoted 8 lines
run-command.c +8
-0+ +if (c1 != c2) +return 0 ; +if (c1 =='\0' ) +return 1 ; + } +} +
run-command.c static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) · line 295
Quoted 10 lines
run-command.c +10
-0+static int searchenv (char **env,const char *name) +{ +int pos =0 ; + +for (; env[pos]; pos++) +if (env_isequal (env[pos], name)) +break ; + +return pos; +}
So this scans through every string in env looking for something that matches 'name' followed by a '=', and return the position of that in env so do_putenv below can use it:
run-command.c static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) · line 302
Quoted 7 lines
run-command.c +7
-0+ +static int do_putenv (char **env,int env_nr,const char *name) +{ +int pos =searchenv (env, name); + +if (strchr (name,'=' )) { +/* ('key=value'), insert of replace entry */
"insert or replace"
run-command.c static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) · line 305
Quoted 3 lines
run-command.c +3
-0+if (pos >= env_nr) + env_nr++; + env[pos] = (char *) name;
OK, this sets an entry...
run-command.c static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) · line 311
Quoted 6 lines
run-command.c +6
-0+ }else if (pos < env_nr) { +/* otherwise ('key') remove existing entry */ + env_nr--; +memmove (&env[pos], &env[pos +1 ], + (env_nr - pos) *sizeof (char *)); + env[env_nr] =NULL ;
And this clobbers it, freeing up a slot for future sets and tells the caller by returning env_nr below:
run-command.c static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) · line 315
Quoted 4 lines
run-command.c +4
-0+ } + +return env_nr; +}
So now the caller below will know where there's a free slot to place the next environment variable:
Quoted 13 lines
run-command.c +13
-0+static char **prep_childenv (const char *const *deltaenv) +{ +extern char **environ; +char **childenv; +int childenv_nr =0 , childenv_alloc =0 ; +int i; + +for (i =0 ; environ[i]; i++) + childenv_nr++; +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_alloc++; +/* Add one for the NULL termination */ + childenv_alloc += childenv_nr +1 ;
run-command.c static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) · line 339
Quoted 10 lines
run-command.c +10
-0+ + childenv =xcalloc (childenv_alloc,sizeof (char *)); +memcpy (childenv, environ, childenv_nr *sizeof (char *)); + +/* merge in deltaenv */ +for (i =0 ; deltaenv && deltaenv[i]; i++) + childenv_nr =do_putenv (childenv, childenv_nr, deltaenv[i]); + +return childenv; +}
OK, the above seems to make sense; copy parent environment and then make changes from deltaenv on top of it...
Quoted 20 lines
run-command.c +2
-0#endif static inline void set_cloexec (int fd)@@ -395,12 +465,14 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;prepare_cmd (&argv, cmd); + childenv =prep_childenv (cmd->env); cmd->pid =fork (); failed_errno = errno;@@ -456,14 +528,6 @@ int start_command(struct child_process *cmd)if (cmd->dir &&chdir (cmd->dir))
run-command.c int start_command(struct child_process *cmd) · line 466
Quoted 10 lines
run-command.c +0
-8die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir); -if (cmd->env) { -for (; *cmd->env; cmd->env++) { -if (strchr (*cmd->env,'=' )) -putenv ((char *)*cmd->env); -else -unsetenv (*cmd->env); - } - }
... which was what the original code did inside the forked child.
So, everything above made sense to me except the use of tolower.
So it looks like Brandon is reusing some of Karsten's compat/mingw.c changes in
commit 343ff06da7d83f40892b10a3b653c7d0e6cb526c, ("Win32: keep the environment sorted")
But, since these changes to run-command are *nix only, the sorting the env makes no sense, and neither does case-insensitivity.
However, reading Karsten's commit; it only seems the use of case-insensitive qsort is correct. I'm not sure if mingw is case-insensitive for actual env modifications, but I know *nix env names are case-sensitive.
So, there _may_ be a bug in compat/mingw.c with the use of bsearchenv and compareenv. It does not seem bsearch is correct to use there, nor can a case-insensitive compareenv be used for searching, only sorting...
Re: [PATCH v4 05/10] run-command: prepare child environment before forking
Quoted 10 lines · Eric Wong
run-command.c static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) · line 277
Quoted 7 lines
run-command.c +7
-0+static int env_isequal (const char *e1,const char *e2) +{ +for (;;) { +char c1 = *e1++; +char c2 = *e2++; + c1 = (c1 =='=' ) ?'\0' :tolower (c1); + c2 = (c2 =='=' ) ?'\0' :tolower (c2);Dealing with C strings scares me so maybe I'm misreading;
but: why is this comparison case-insensitive?
Well i was pulling inspiration from the stuff in mingw.c...looks like i probably shouldn't have done so as you're correct, they should be case-sensitive. Jonathan pointed out that doing this env stuff in vanilla C may not be a good idea...and I kinda forgot about that cause it worked (it passed tests) Let me re-write this section of code to make it correct, and saner.
Signature · 1 line
Brandon Williams
PATCH v4 6/10 run-command: don't die in child when duping /dev/null
In reply to Brandon Williams--- run-command.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-)
run-command.c +13 -15
@@ -117,18 +117,6 @@ static inline void close_pair(int fd[2])close (fd[1 ]); } -#ifndef GIT_WINDOWS_NATIVE -static inline void dup_devnull (int to) -{ -int fd =open ("/dev/null" ,O_RDWR ); -if (fd <0 ) -die_errno (_ ("open /dev/null failed" )); -if (dup2 (fd, to) <0 ) -die_errno (_ ("dup2(%d,%d) failed" ), fd, to); -close (fd); -} -#endif -static char *locate_in_PATH (const char *file) {const char *p =getenv ("PATH" );
@@ -465,12 +453,20 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +int null_fd =-1 ;char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; +if (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) { + null_fd =open ("/dev/null" ,O_RDWR |O_CLOEXEC ); +if (null_fd <0 ) +die_errno (_ ("open /dev/null failed" )); +set_cloexec (null_fd); + } +prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env);
@@ -494,7 +490,7 @@ int start_command(struct child_process *cmd)atexit (notify_parent);if (cmd->no_stdin) -dup_devnull (0 ); +dup2 (null_fd,0 );else if (need_in) {dup2 (fdin[0 ],0 );close_pair (fdin);
@@ -504,7 +500,7 @@ int start_command(struct child_process *cmd) }if (cmd->no_stderr) -dup_devnull (2 ); +dup2 (null_fd,2 );else if (need_err) {dup2 (fderr[1 ],2 );close_pair (fderr);
@@ -514,7 +510,7 @@ int start_command(struct child_process *cmd) }if (cmd->no_stdout) -dup_devnull (1 ); +dup2 (null_fd,1 );else if (cmd->stdout_to_stderr)dup2 (2 ,1 );else if (need_out) {
@@ -574,6 +570,8 @@ int start_command(struct child_process *cmd) }close (notify_pipe[0 ]); +if (null_fd >=0 ) +close (null_fd);argv_array_clear (&argv);free (childenv); }
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v4 7/10 run-command: eliminate calls to error handling functions in child
In reply to Brandon WilliamsAll of our standard error handling paths have the potential to call malloc or take stdio locks; so we must avoid them inside the forked child.
Instead, the child only writes an 8 byte struct atomically to the parent through the notification pipe to propagate an error. All user-visible error reporting happens from the parent; even avoiding functions like atexit(3) and exit(3).
--- run-command.c | 121 ++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 89 insertions(+), 32 deletions(-)
run-command.c +89 -32
@@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ; -static void notify_parent (void ) +enum child_errcode { +CHILD_ERR_CHDIR , +CHILD_ERR_ENOENT , +CHILD_ERR_SILENT , +CHILD_ERR_ERRNO +}; + +struct child_err { +enum child_errcode err; +int syserr;/* errno */ +}; + +static void child_die (enum child_errcode err) { -/* -* execvp failed. If possible, we'd like to let start_command -* know, so failures like ENOENT can be handled right away; but -* otherwise, finish_command will still report the error. -*/ -xwrite (child_notifier,"" ,1 ); +struct child_err buf; + + buf.err = err; + buf.syserr = errno; + +/* write(2) on buf smaller than PIPE_BUF (min 512) is atomic: */ +xwrite (child_notifier, &buf,sizeof (buf)); +_exit (1 ); +} + +/* +* parent will make it look like the child spewed a fatal error and died +* this is needed to prevent changes to t0061. +*/ +static void fake_fatal (const char *err,va_list params) +{ +vreportf ("fatal: " , err, params); +} + +static void child_error_fn (const char *err,va_list params) +{ +const char msg[] ="error() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +} + +static void child_warn_fn (const char *err,va_list params) +{ +const char msg[] ="warn() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +} + +static void NORETURN child_die_fn(const char *err, va_list params) +{ +const char msg[] ="die() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +_exit (2 ); +} + +/* this runs in the parent process */ +static void child_err_spew (struct child_process *cmd,struct child_err *cerr) +{ +static void (*old_errfn)(const char *err,va_list params); + + old_errfn =get_error_routine (); +set_error_routine (fake_fatal); + errno = cerr->syserr; + +switch (cerr->err) { +case CHILD_ERR_CHDIR : +error_errno ("exec '%s': cd to '%s' failed" , + cmd->argv[0 ], cmd->dir); +break ; +case CHILD_ERR_ENOENT : +error_errno ("cannot run %s" , cmd->argv[0 ]); +break ; +case CHILD_ERR_SILENT : +break ; +case CHILD_ERR_ERRNO : +error_errno ("cannot exec '%s'" , cmd->argv[0 ]); +break ; + } +set_error_routine (old_errfn); }static void prepare_cmd (struct argv_array *out,const struct child_process *cmd)
@@ -362,13 +430,6 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal) code +=128 ; }else if (WIFEXITED (status)) { code =WEXITSTATUS (status); -/* -* Convert special exit code when execvp failed. -*/ -if (code ==127 ) { - code =-1 ; - failed_errno =ENOENT ; - } }else {error ("waitpid is confused (%s)" , argv0); }
@@ -456,6 +517,7 @@ int start_command(struct child_process *cmd)int null_fd =-1 ;char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ; +struct child_err cerr;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;
@@ -474,20 +536,16 @@ int start_command(struct child_process *cmd) failed_errno = errno;if (!cmd->pid) {/* -* Redirect the channel to write syscall error messages to -* before redirecting the process's stderr so that all die() -* in subsequent call paths use the parent's stderr. +* Ensure the default die/error/warn routines do not get +* called, they can take stdio locks and malloc. */ -if (cmd->no_stderr || need_err) { -int child_err =dup (2 ); -set_cloexec (child_err); -set_error_handle (fdopen (child_err,"w" )); - } +set_die_routine (child_die_fn); +set_error_routine (child_error_fn); +set_warn_routine (child_warn_fn);close (notify_pipe[0 ]);set_cloexec (notify_pipe[1 ]); child_notifier = notify_pipe[1 ]; -atexit (notify_parent);if (cmd->no_stdin)dup2 (null_fd,0 );
@@ -522,8 +580,7 @@ int start_command(struct child_process *cmd) }if (cmd->dir &&chdir (cmd->dir)) -die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], - cmd->dir); +child_die (CHILD_ERR_CHDIR );/* * Attempt to exec using the command and arguments starting at
@@ -538,12 +595,11 @@ int start_command(struct child_process *cmd) (char *const *) childenv);if (errno ==ENOENT ) { -if (!cmd->silent_exec_failure) -error ("cannot run %s: %s" , cmd->argv[0 ], -strerror (ENOENT )); -exit (127 ); +if (cmd->silent_exec_failure) +child_die (CHILD_ERR_SILENT ); +child_die (CHILD_ERR_ENOENT ); }else { -die_errno ("cannot exec '%s'" , cmd->argv[0 ]); +child_die (CHILD_ERR_ERRNO ); } }if (cmd->pid <0 )
@@ -554,17 +610,18 @@ int start_command(struct child_process *cmd)/* * Wait for child's exec. If the exec succeeds (or if fork() * failed), EOF is seen immediately by the parent. Otherwise, the -* child process sends a single byte. +* child process sends a child_err struct. * Note that use of this infrastructure is completely advisory, * therefore, we keep error checks minimal. */ close (notify_pipe[1 ]); -if (read (notify_pipe[0 ], ¬ify_pipe[1 ],1 ) ==1 ) { +if (xread (notify_pipe[0 ], &cerr,sizeof (cerr)) ==sizeof (cerr)) {/* * At this point we know that fork() succeeded, but exec() * failed. Errors have been reported to our stderr. */ wait_or_whine (cmd->pid, cmd->argv[0 ],0 ); +child_err_spew (cmd, &cerr); failed_errno = errno; cmd->pid =-1 ; }
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v4 8/10 run-command: handle dup2 and close errors in child
In reply to Brandon Williams--- run-command.c | 58 ++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 42 insertions(+), 16 deletions(-)
run-command.c +42 -16
@@ -213,6 +213,8 @@ static int child_notifier = -1;enum child_errcode {CHILD_ERR_CHDIR , +CHILD_ERR_DUP2 , +CHILD_ERR_CLOSE ,CHILD_ERR_ENOENT ,CHILD_ERR_SILENT ,CHILD_ERR_ERRNO
@@ -235,6 +237,24 @@ static void child_die(enum child_errcode err)_exit (1 ); } +static void child_dup2 (int fd,int to) +{ +if (dup2 (fd, to) <0 ) +child_die (CHILD_ERR_DUP2 ); +} + +static void child_close (int fd) +{ +if (close (fd)) +child_die (CHILD_ERR_CLOSE ); +} + +static void child_close_pair (int fd[2 ]) +{ +child_close (fd[0 ]); +child_close (fd[1 ]); +} +/* * parent will make it look like the child spewed a fatal error and died * this is needed to prevent changes to t0061.
@@ -277,6 +297,12 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)error_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir);break ; +case CHILD_ERR_DUP2 : +error_errno ("dup2() in child failed" ); +break ; +case CHILD_ERR_CLOSE : +error_errno ("close() in child failed" ); +break ;case CHILD_ERR_ENOENT :error_errno ("cannot run %s" , cmd->argv[0 ]);break ;
@@ -548,35 +574,35 @@ int start_command(struct child_process *cmd) child_notifier = notify_pipe[1 ];if (cmd->no_stdin) -dup2 (null_fd,0 ); +child_dup2 (null_fd,0 );else if (need_in) { -dup2 (fdin[0 ],0 ); -close_pair (fdin); +child_dup2 (fdin[0 ],0 ); +child_close_pair (fdin); }else if (cmd->in) { -dup2 (cmd->in,0 ); -close (cmd->in); +child_dup2 (cmd->in,0 ); +child_close (cmd->in); }if (cmd->no_stderr) -dup2 (null_fd,2 ); +child_dup2 (null_fd,2 );else if (need_err) { -dup2 (fderr[1 ],2 ); -close_pair (fderr); +child_dup2 (fderr[1 ],2 ); +child_close_pair (fderr); }else if (cmd->err >1 ) { -dup2 (cmd->err,2 ); -close (cmd->err); +child_dup2 (cmd->err,2 ); +child_close (cmd->err); }if (cmd->no_stdout) -dup2 (null_fd,1 ); +child_dup2 (null_fd,1 );else if (cmd->stdout_to_stderr) -dup2 (2 ,1 ); +child_dup2 (2 ,1 );else if (need_out) { -dup2 (fdout[1 ],1 ); -close_pair (fdout); +child_dup2 (fdout[1 ],1 ); +child_close_pair (fdout); }else if (cmd->out >1 ) { -dup2 (cmd->out,1 ); -close (cmd->out); +child_dup2 (cmd->out,1 ); +child_close (cmd->out); }if (cmd->dir &&chdir (cmd->dir))
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v4 9/10 run-command: add note about forking and threading
In reply to Brandon WilliamsAll non-Async-Signal-Safe functions (e.g. malloc and die) were removed between 'fork' and 'exec' in start_command in order to avoid potential deadlocking when forking while multiple threads are running. This deadlocking is possible when a thread (other than the one forking) has acquired a lock and didn't get around to releasing it before the fork. This leaves the lock in a locked state in the resulting process with no hope of it ever being released.
Add a note describing this potential pitfall before the call to 'fork()' so people working in this section of the code know to only use Async-Signal-Safe functions in the child process.
--- run-command.c | 9 +++++++++ 1 file changed, 9 insertions(+)
run-command.c +9 -0
@@ -558,6 +558,15 @@ int start_command(struct child_process *cmd)prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env); +/* +* NOTE: In order to prevent deadlocking when using threads special +* care should be taken with the function calls made in between the +* fork() and exec() calls. No calls should be made to functions which +* require acquiring a lock (e.g. malloc) as the lock could have been +* held by another thread at the time of forking, causing the lock to +* never be released in the child process. This means only +* Async-Signal-Safe functions are permitted in the child. +*/ cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) {
Signature · 1 line
2.12.2.762.g0e3151a226-goog
PATCH v4 10/10 run-command: block signals between fork and execve
In reply to Brandon WilliamsFrom: Eric Wong <[email protected]>
Signal handlers of the parent firing in the forked child may have unintended side effects. Rather than auditing every signal handler we have and will ever have, block signals while forking and restore default signal handlers in the child before execve.
Restoring default signal handlers is required because execve does not unblock signals, it only restores default signal handlers. So we must restore them with sigprocmask before execve, leaving a window when signal handlers we control can fire in the child. Continue ignoring ignored signals, but reset the rest to defaults.
Similarly, disable pthread cancellation to future-proof our code in case we start using cancellation; as cancellation is implemented with signals in glibc.
--- run-command.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+)
run-command.c +68 -0
@@ -215,6 +215,7 @@ enum child_errcode {CHILD_ERR_CHDIR ,CHILD_ERR_DUP2 ,CHILD_ERR_CLOSE , +CHILD_ERR_SIGPROCMASK ,CHILD_ERR_ENOENT ,CHILD_ERR_SILENT ,CHILD_ERR_ERRNO
@@ -303,6 +304,9 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)case CHILD_ERR_CLOSE :error_errno ("close() in child failed" );break ; +case CHILD_ERR_SIGPROCMASK : +error_errno ("sigprocmask failed restoring signals" ); +break ;case CHILD_ERR_ENOENT :error_errno ("cannot run %s" , cmd->argv[0 ]);break ;
@@ -421,6 +425,53 @@ static char **prep_childenv(const char *const *deltaenv) }#endif +struct atfork_state { +#ifndef NO_PTHREADS +int cs; +#endif +sigset_t old; +}; + +#ifndef NO_PTHREADS +static void bug_die (int err,const char *msg) +{ +if (err) { + errno = err; +die_errno ("BUG: %s" , msg); + } +} +#endif + +static void atfork_prepare (struct atfork_state *as) +{ +sigset_t all; + +if (sigfillset (&all)) +die_errno ("sigfillset" ); +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &all, &as->old)) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_sigmask (SIG_SETMASK , &all, &as->old), +"blocking all signals" ); +bug_die (pthread_setcancelstate (PTHREAD_CANCEL_DISABLE , &as->cs), +"disabling cancellation" ); +#endif +} + +static void atfork_parent (struct atfork_state *as) +{ +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &as->old,NULL )) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_setcancelstate (as->cs,NULL ), +"re-enabling cancellation" ); +bug_die (pthread_sigmask (SIG_SETMASK , &as->old,NULL ), +"restoring signal mask" ); +#endif +} +static inline void set_cloexec (int fd) {int flags =fcntl (fd,F_GETFD );
@@ -544,6 +595,7 @@ int start_command(struct child_process *cmd)char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;struct child_err cerr; +struct atfork_state as;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;
@@ -557,6 +609,7 @@ int start_command(struct child_process *cmd)prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env); +atfork_prepare (&as);/* * NOTE: In order to prevent deadlocking when using threads special
@@ -570,6 +623,7 @@ int start_command(struct child_process *cmd) cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) { +int sig;/* * Ensure the default die/error/warn routines do not get * called, they can take stdio locks and malloc.
@@ -618,6 +672,19 @@ int start_command(struct child_process *cmd)child_die (CHILD_ERR_CHDIR );/* +* restore default signal handlers here, in case +* we catch a signal right before execve below +*/ +for (sig =1 ; sig <NSIG ; sig++) { +/* ignored signals get reset to SIG_DFL on execve */ +if (signal (sig,SIG_DFL ) ==SIG_IGN ) +signal (sig,SIG_IGN ); + } + +if (sigprocmask (SIG_SETMASK , &as.old,NULL ) !=0 ) +child_die (CHILD_ERR_SIGPROCMASK ); + +/* * Attempt to exec using the command and arguments starting at * argv.argv[1]. argv.argv[0] contains SHELL_PATH which will * be used in the event exec failed with ENOEXEC at which point
@@ -637,6 +704,7 @@ int start_command(struct child_process *cmd)child_die (CHILD_ERR_ERRNO ); } } +atfork_parent (&as);if (cmd->pid <0 )error_errno ("cannot fork() for %s" , cmd->argv[0 ]);else if (cmd->clean_on_exit)
Signature · 1 line
2.12.2.762.g0e3151a226-goog
v5 cover forking and threading
In reply to Brandon Williamsv5 addresses the issues with creating the child's environment. Instead of doing vanilla C string manipulation, I used a struct string_list and struct strbuf. The code should be much easier to read and understand now. I also needed to add a function to remove a string_list_item from a string_list to the string_list API.
Brandon Williams (10):
t5550: use write_script to generate post-update hook
t0061: run_command executes scripts without a #! line
run-command: prepare command before forking
run-command: use the async-signal-safe execv instead of execvp
string-list: add string_list_remove function
run-command: prepare child environment before forking
run-command: don't die in child when duping /dev/null
run-command: eliminate calls to error handling functions in child
run-command: handle dup2 and close errors in child
run-command: add note about forking and threading
Eric Wong (1):
run-command: block signals between fork and execve
run-command.c | 404 +++++++++++++++++++++++++++++++++++---------- string-list.c | 18 ++ string-list.h | 5 + t/t0061-run-command.sh | 11 ++ t/t5550-http-fetch-dumb.sh | 5 +- 5 files changed, 358 insertions(+), 85 deletions(-)
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v5 1/11 t5550: use write_script to generate post-update hook
The post-update hooks created in t5550-http-fetch-dumb.sh is missing the "!#/bin/sh" line which can cause issues with portability. Instead create the hook using the 'write_script' function which includes the proper "#!" line.
--- t/t5550-http-fetch-dumb.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
t/t5550-http-fetch-dumb.sh +3 -2
@@ -20,8 +20,9 @@ test_expect_success 'create http-accessible bare repository with loose objects' (cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" && git config core.bare true && mkdir -p hooks && - echo "exec git update-server-info" >hooks/post-update && - chmod +x hooks/post-update && + write_script "hooks/post-update" <<-\EOF && + exec git update-server-info + EOF hooks/post-update ) && git remote add public "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v5 2/11 t0061: run_command executes scripts without a #! line
In reply to Brandon WilliamsAdd a test to 't0061-run-command.sh' to ensure that run_command can continue to execute scripts which don't include a '#!' line.
--- t/t0061-run-command.sh | 11 +++++++++++ 1 file changed, 11 insertions(+)
t/t0061-run-command.sh +11 -0
@@ -26,6 +26,17 @@ test_expect_success 'run_command can run a command' ' test_cmp empty err ' +test_expect_success 'run_command can run a script without a #! line' ' + cat >hello <<-\EOF && + cat hello-script + EOF + chmod +x hello && + test-run-command run-command ./hello >actual 2>err && + + test_cmp hello-script actual && + test_cmp empty err +' + test_expect_success POSIXPERM 'run_command reports EACCES' ' cat hello-script >hello.sh && chmod -x hello.sh &&
Signature · 1 line
2.12.2.816.g2cccc81164-goog
Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line
Am 19.04.2017 um 01:17 schrieb Brandon Williams:
Quoted 2 lines
Add a test to '
t0061-run-command.sh' to ensure thatrun_commandcan continue to execute scripts which don't include a '#!' line.
Why is this necessary? I am pretty certain that our emulation layer on Windows can only run scripts with a shbang line.
-- Hannes
Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line
Am 19.04.2017 um 07:43 schrieb Johannes Sixt:
Quoted 6 lines
Am 19.04.2017 um 01:17 schrieb Brandon Williams:
Quoted 2 lines
Add a test to '
t0061-run-command.sh' to ensure thatrun_commandcan continue to execute scripts which don't include a '#!' line.Why is this necessary? I am pretty certain that our emulation layer on Windows can only run scripts with a shbang line.
Nevermind. It is a compatibility feature: People may have written their hooks and scripts without #!, and these must continue to work where they worked before.
Please protect the new test with !MINGW.
Thanks,
-- Hannes
Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line
Quoted 7 lines · Johannes Sixt
Am 19.04.2017 um 07:43 schrieb Johannes Sixt:
Quoted 6 lines
Am 19.04.2017 um 01:17 schrieb Brandon Williams:
Quoted 2 lines
Add a test to '
t0061-run-command.sh' to ensure thatrun_commandcan continue to execute scripts which don't include a '#!' line.Why is this necessary? I am pretty certain that our emulation layer on Windows can only run scripts with a shbang line.
Out of curiosity how did you have t5550 passing on windows then? Since the first patch in this series fixes a that test which doesn't have a '#!' line.
Quoted 6 lines
Nevermind. It is a compatibility feature: People may have written their hooks and scripts without #!, and these must continue to work where they worked before.
Please protect the new test with !MINGW.
Will do.
Quoted 4 lines
Thanks,
-- Hannes
Signature · 1 line
Brandon Williams
Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line
Am 19.04.2017 um 17:56 schrieb Brandon Williams:
Quoted 7 lines · Johannes Sixt
Quoted 2 lines
Am 19.04.2017 um 07:43 schrieb Johannes Sixt:
Quoted 1 line
Windows can only run scripts with a shbang line.
Out of curiosity how did you have t5550 passing on windows then? Since the first patch in this series fixes a that test which doesn't have a '#!' line.
I guess, I don't run it at all in my setup.
-- Hannes
Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line
In reply to Brandon WilliamsHi Brandon,
Quoted 10 lines · Brandon Williams
Quoted 7 lines · Johannes Sixt
Am 19.04.2017 um 07:43 schrieb Johannes Sixt:
Quoted 6 lines
Am 19.04.2017 um 01:17 schrieb Brandon Williams:
Quoted 2 lines
Add a test to '
t0061-run-command.sh' to ensure thatrun_commandcan continue to execute scripts which don't include a '#!' line.Why is this necessary? I am pretty certain that our emulation layer on Windows can only run scripts with a shbang line.
Out of curiosity how did you have t5550 passing on windows then?
This is the reason:
1..0 # SKIP no web server found at '/usr/sbin/apache2'
As predicted by Hannes, your new test fails miserably on Windows:
https://travis-ci.org/git/git/jobs/223830474#L2656-L2674
Ciao,
Dscho
Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line
Quoted 10 lines · Johannes Schindelin
Hi Brandon,
Quoted 6 lines · Brandon Williams
Quoted 5 lines · Johannes Sixt
Am 19.04.2017 um 07:43 schrieb Johannes Sixt:
Quoted 4 lines
Am 19.04.2017 um 01:17 schrieb Brandon Williams:
Quoted 2 lines
Add a test to '
t0061-run-command.sh' to ensure thatrun_commandcan continue to execute scripts which don't include a '#!' line.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Why is this necessary? I am pretty certain that our emulation layer on Windows can only run scripts with a shbang line.
Out of curiosity how did you have t5550 passing on windows then?
This is the reason:
1..0 # SKIP no web server found at '
/usr/sbin/apache2'
Hmm, that's interesting. So do any of the http tests get run on windows then? I wonder if that lack of coverage could be an issue at some point in the future.
Quoted 1 line
As predicted by Hannes, your new test fails miserably on Windows:
Isn't 'miserably' just a bit harsh ;P haha
Signature · 1 line
Brandon Williams
Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line
Hi Brandon,
Quoted 15 lines · Brandon Williams
Quoted 14 lines · Johannes Schindelin
Quoted 10 lines · Brandon Williams
Quoted 7 lines · Johannes Sixt
Am 19.04.2017 um 07:43 schrieb Johannes Sixt:
Quoted 6 lines
Am 19.04.2017 um 01:17 schrieb Brandon Williams:
Quoted 2 lines
Add a test to '
t0061-run-command.sh' to ensure thatrun_commandcan continue to execute scripts which don't include a '#!' line.Why is this necessary? I am pretty certain that our emulation layer on Windows can only run scripts with a shbang line.
Out of curiosity how did you have t5550 passing on windows then?
Quoted 7 lines
Quoted 3 lines
This is the reason:
1..0 # SKIP no web server found at '
/usr/sbin/apache2'Hmm, that's interesting. So do any of the http tests get run on windows then? I wonder if that lack of coverage could be an issue at some point in the future.
Possibly. I'll put it at the bottom of my TODO list ;-)
Quoted 3 lines
Quoted 1 line
As predicted by Hannes, your new test fails miserably on Windows:
Isn't 'miserably' just a bit harsh ;P haha
Ah, I tried to be funny. So much for my future career as a comedian.
Ciao,
Dscho
Re: [PATCH v5 02/11] t0061: run_command executes scripts without a #! line
Quoted 27 lines · Johannes Schindelin
Hi Brandon,
Quoted 15 lines · Brandon Williams
Quoted 14 lines · Johannes Schindelin
Quoted 10 lines · Brandon Williams
Quoted 7 lines · Johannes Sixt
Am 19.04.2017 um 07:43 schrieb Johannes Sixt:
Quoted 6 lines
Am 19.04.2017 um 01:17 schrieb Brandon Williams:
Quoted 2 lines
Add a test to '
t0061-run-command.sh' to ensure thatrun_commandcan continue to execute scripts which don't include a '#!' line.Why is this necessary? I am pretty certain that our emulation layer on Windows can only run scripts with a shbang line.
Out of curiosity how did you have t5550 passing on windows then?
Quoted 7 lines
Quoted 3 lines
This is the reason:
1..0 # SKIP no web server found at '
/usr/sbin/apache2'Hmm, that's interesting. So do any of the http tests get run on windows then? I wonder if that lack of coverage could be an issue at some point in the future.
Quoted 7 lines
Possibly. I'll put it at the bottom of my TODO list ;-)
Quoted 3 lines
Quoted 1 line
As predicted by Hannes, your new test fails miserably on Windows:
Isn't 'miserably' just a bit harsh ;P haha
Ah, I tried to be funny. So much for my future career as a comedian.
Haha yeah I figured, no worries. I was just throwing it back at you :D
Quoted 3 lines
Ciao,
Dscho
Signature · 1 line
Brandon Williams
PATCH v5 3/11 run-command: prepare command before forking
In reply to Brandon WilliamsAccording to [1] we need to only call async-signal-safe operations between fork and exec. Using malloc to build the argv array isn't async-signal-safe.
In order to avoid allocation between 'fork()' and 'exec()' prepare the argv array used in the exec call prior to forking the process.
[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html
--- run-command.c | 46 ++++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-)
run-command.c +26 -20
@@ -221,18 +221,6 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv) }#ifndef GIT_WINDOWS_NATIVE -static int execv_shell_cmd (const char **argv) -{ -struct argv_array nargv =ARGV_ARRAY_INIT ; -prepare_shell_cmd (&nargv, argv); -trace_argv_printf (nargv.argv,"trace: exec:" ); -sane_execvp (nargv.argv[0 ], (char **)nargv.argv); -argv_array_clear (&nargv); -return -1 ; -} -#endif - -#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ;static void notify_parent (void )
@@ -244,6 +232,21 @@ static void notify_parent(void)*/ xwrite(child_notifier, "", 1); } + +static void prepare_cmd (struct argv_array *out,const struct child_process *cmd) +{ +if (!cmd->argv[0 ]) +die ("BUG: command is empty" ); + +if (cmd->git_cmd) { +argv_array_push (out,"git" ); +argv_array_pushv (out, cmd->argv); + }else if (cmd->use_shell) { +prepare_shell_cmd (out, cmd->argv); + }else { +argv_array_pushv (out, cmd->argv); + } +}#endif static inline void set_cloexec (int fd)
@@ -372,9 +375,13 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +struct argv_array argv =ARGV_ARRAY_INIT ; +if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; +prepare_cmd (&argv, cmd); + cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) {
@@ -437,12 +444,9 @@ int start_command(struct child_process *cmd)unsetenv (*cmd->env); } } -if (cmd->git_cmd) -execv_git_cmd (cmd->argv); -else if (cmd->use_shell) -execv_shell_cmd (cmd->argv); -else -sane_execvp (cmd->argv[0 ], (char *const *) cmd->argv); + +sane_execvp (argv.argv[0 ], (char *const *) argv.argv); +if (errno ==ENOENT ) {if (!cmd->silent_exec_failure) error("cannot run %s: %s", cmd->argv[0],
@@ -458,7 +462,7 @@ int start_command(struct child_process *cmd)mark_child_for_cleanup (cmd->pid, cmd);/* -* Wait for child's execvp. If the execvp succeeds (or if fork() +* Wait for child's exec. If the exec succeeds (or if fork() * failed), EOF is seen immediately by the parent. Otherwise, the * child process sends a single byte. * Note that use of this infrastructure is completely advisory,
@@ -467,7 +471,7 @@ int start_command(struct child_process *cmd)close (notify_pipe[1 ]);if (read (notify_pipe[0 ], ¬ify_pipe[1 ],1 ) ==1 ) {/* -* At this point we know that fork() succeeded, but execvp() +* At this point we know that fork() succeeded, but exec() * failed. Errors have been reported to our stderr. */ wait_or_whine (cmd->pid, cmd->argv[0 ],0 );
@@ -475,6 +479,8 @@ int start_command(struct child_process *cmd) cmd->pid = -1; }close (notify_pipe[0 ]); + +argv_array_clear (&argv); }#else {
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v5 4/11 run-command: use the async-signal-safe execv instead of execvp
In reply to Brandon WilliamsConvert the function used to exec from 'execvp()' to 'execv()' as the (p) variant of exec isn't async-signal-safe and has the potential to call malloc during the path resolution it performs. Instead we simply do the path resolution ourselves during the preparation stage prior to forking. There also don't exist any portable (p) variants which also take in an environment to use in the exec'd process. This allows easy migration to using 'execve()' in a future patch.
Also, as noted in [1], in the event of an ENOEXEC the (p) variants of exec will attempt to execute the command by interpreting it with the 'sh' utility. To maintain this functionality, if 'execv()' fails with ENOEXEC, start_command will atempt to execute the command by interpreting it with 'sh'.
[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html
--- run-command.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
run-command.c +29 -1
@@ -238,6 +238,12 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)if (!cmd->argv[0 ])die ("BUG: command is empty" ); +/* +* Add SHELL_PATH so in the event exec fails with ENOEXEC we can +* attempt to interpret the command with 'sh'. +*/ +argv_array_push (out,SHELL_PATH ); +if (cmd->git_cmd) {argv_array_push (out,"git" );argv_array_pushv (out, cmd->argv);
@@ -246,6 +252,20 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) } else {argv_array_pushv (out, cmd->argv); } + +/* +* If there are no '/' characters in the command then perform a path +* lookup and use the resolved path as the command to exec. If there +* are no '/' characters or if the command wasn't found in the path, +* have exec attempt to invoke the command directly. +*/ +if (!strchr (out->argv[1 ],'/' )) { +char *program =locate_in_PATH (out->argv[1 ]); +if (program) { +free ((char *)out->argv[1 ]); + out->argv[1 ] = program; + } + } }#endif
@@ -445,7 +465,15 @@ int start_command(struct child_process *cmd) } } -sane_execvp (argv.argv[0 ], (char *const *) argv.argv); +/* +* Attempt to exec using the command and arguments starting at +* argv.argv[1]. argv.argv[0] contains SHELL_PATH which will +* be used in the event exec failed with ENOEXEC at which point +* we will try to interpret the command using 'sh'. +*/ +execv (argv.argv[1 ], (char *const *) argv.argv +1 ); +if (errno ==ENOEXEC ) +execv (argv.argv[0 ], (char *const *) argv.argv);if (errno ==ENOENT ) {if (!cmd->silent_exec_failure)
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v5 5/11 string-list: add string_list_remove function
In reply to Brandon WilliamsTeach string-list to be able to remove a string from a sorted 'struct string_list'.
--- string-list.c | 18 ++++++++++++++++++ string-list.h | 5 +++++ 2 files changed, 23 insertions(+)
string-list.c +18 -0
@@ -67,6 +67,24 @@ struct string_list_item *string_list_insert(struct string_list *list, const charreturn list->items + index; } +void string_list_remove (struct string_list *list,const char *string, +int free_util) +{ +int exact_match; +int i =get_entry_index (list, string, &exact_match); + +if (exact_match) { +if (list->strdup_strings) +free (list->items[i].string); +if (free_util) +free (list->items[i].util); + + list->nr--; +memmove (list->items + i, list->items + i +1 , + (list->nr - i) *sizeof (struct string_list_item )); + } +} +int string_list_has_string (const struct string_list *list,const char *string) {int exact_match;
string-list.h +5 -0
@@ -63,6 +63,11 @@ int string_list_find_insert_index(const struct string_list *list, const char *ststruct string_list_item *string_list_insert (struct string_list *list,const char *string);/* +* Removes the given string from the sorted list. +*/ +void string_list_remove (struct string_list *list,const char *string,int free_util); + +/* * Checks if the given string is part of a sorted list. If it is part of the list, * return the coresponding string_list_item, NULL otherwise. */
Signature · 1 line
2.12.2.816.g2cccc81164-goog
Re: [PATCH v5 05/11] string-list: add string_list_remove function
Quoted 7 lines · Brandon Williams
string-list.c +6
-0+void string_list_remove (struct string_list *list,const char *string, +int free_util) +{ +int exact_match; +int i =get_entry_index (list, string, &exact_match); +
string-list.c struct string_list_item *string_list_insert(struct string_list *list, const char · line 85
Quoted 10 lines
string-list.c +10
-0+if (exact_match) { +if (list->strdup_strings) +free (list->items[i].string); +if (free_util) +free (list->items[i].util); + + list->nr--; +memmove (list->items + i, list->items + i +1 , + (list->nr - i) *sizeof (struct string_list_item )); + }
Looks correct. I shortly wondered if we'd have any value in returing exact_match, as that may save the caller some code, as I imagine the caller to be:
if (!string_list_has_string(&list, string))
die("BUG: ...");
string_list_remove(&list, string, 0);which could be simplified if we had the exact_match returned, i.e. the string_list_remove returns the implicit string_list_has_string.
Quoted 2 lines
/*
+ * Removes the given string from the sorted list.
What happens when the string is not found?
Quoted 2 lines
+ */
+voidstring_list_remove(struct string_list *list, const char *string, int free_util);
How much do we care about (eventual) consistency? ;) i.e. mark it extern ?
Thanks,
Stefan
Re: [PATCH v5 05/11] string-list: add string_list_remove function
Quoted 24 lines · Stefan Beller
Quoted 7 lines · Brandon Williams
string-list.c +6
-0+void string_list_remove (struct string_list *list,const char *string, +int free_util) +{ +int exact_match; +int i =get_entry_index (list, string, &exact_match); +string-list.c struct string_list_item *string_list_insert(struct string_list *list, const char · line 85
Quoted 10 lines
string-list.c +10
-0+if (exact_match) { +if (list->strdup_strings) +free (list->items[i].string); +if (free_util) +free (list->items[i].util); + + list->nr--; +memmove (list->items + i, list->items + i +1 , + (list->nr - i) *sizeof (struct string_list_item )); + }Looks correct. I shortly wondered if we'd have any value in returing
exact_match, as that may save the caller some code, as I imagine the caller to be:
Quoted 6 lines
if (!string_list_has_string(&list, string)) die("BUG: ..."); string_list_remove(&list, string, 0);which could be simplified if we had the
exact_matchreturned, i.e. thestring_list_removereturns the implicitstring_list_has_string.
I don't really see the value in this, as the only caller doesn't need it right now.
Quoted 5 lines
Quoted 3 lines
/*
+ * Removes the given string from the sorted list.What happens when the string is not found?
nothing.
Quoted 6 lines
Quoted 3 lines
+ */
+voidstring_list_remove(struct string_list *list, const char *string, int free_util);How much do we care about (eventual) consistency? ;) i.e. mark it extern ?
If I need to do another reroll I can mark it extern.
Signature · 1 line
Brandon Williams
Re: [PATCH v5 05/11] string-list: add string_list_remove function
Quoted 29 lines · Brandon Williams
Quoted 28 lines · Stefan Beller
Quoted 7 lines · Brandon Williams
string-list.c +6
-0+void string_list_remove (struct string_list *list,const char *string, +int free_util) +{ +int exact_match; +int i =get_entry_index (list, string, &exact_match); +string-list.c struct string_list_item *string_list_insert(struct string_list *list, const char · line 85
Quoted 10 lines
string-list.c +10
-0+if (exact_match) { +if (list->strdup_strings) +free (list->items[i].string); +if (free_util) +free (list->items[i].util); + + list->nr--; +memmove (list->items + i, list->items + i +1 , + (list->nr - i) *sizeof (struct string_list_item )); + }Looks correct. I shortly wondered if we'd have any value in returing
exact_match, as that may save the caller some code, as I imagine the caller to be:if (!string_list_has_string(&list, string)) die("BUG: ..."); string_list_remove(&list, string, 0);
Quoted 5 lines
Quoted 2 lines
which could be simplified if we had the
exact_matchreturned, i.e. thestring_list_removereturns the implicitstring_list_has_string.I don't really see the value in this, as the only caller doesn't need it right now.
yeah, I guess we can add such functionality later.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
/*
+ * Removes the given string from the sorted list.What happens when the string is not found?
nothing.
yeah that is what I figured from reading the code. The question could have been worded: Do we want to document what happens if the string is not found? (A reader in the future may wonder if it is even allowed to call this function without having consulted string_list_has_string).
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
+ */
+voidstring_list_remove(struct string_list *list, const char *string, int free_util);How much do we care about (eventual) consistency? ;) i.e. mark it extern ?
If I need to do another reroll I can mark it extern.
Thanks! This doesn't require a reroll on its own and all other patches look sane to me.
Thanks,
Stefan
PATCH v5 6/11 run-command: prepare child environment before forking
In reply to Brandon WilliamsIn order to avoid allocation between 'fork()' and 'exec()' prepare the environment to be used in the child process prior to forking.
Switch to using 'execve()' so that the construct child environment can used in the exec'd process.
--- run-command.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 56 insertions(+), 10 deletions(-)
run-command.c +56 -10
@@ -267,6 +267,55 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) } } } + +static char **prep_childenv (const char *const *deltaenv) +{ +extern char **environ; +char **childenv; +struct string_list env =STRING_LIST_INIT_DUP ; +struct strbuf key =STRBUF_INIT ; +const char *const *p; +int i; + +/* Construct a sorted string list consisting of the current environ */ +for (p = (const char *const *) environ; p && *p; p++) { +const char *equals =strchr (*p,'=' ); + +if (equals) { +strbuf_reset (&key); +strbuf_add (&key, *p, equals - *p); +string_list_append (&env, key.buf)->util = (void *) *p; + }else { +string_list_append (&env, *p)->util = (void *) *p; + } + } +string_list_sort (&env); + +/* Merge in 'deltaenv' with the current environ */ +for (p = deltaenv; p && *p; p++) { +const char *equals =strchr (*p,'=' ); + +if (equals) { +/* ('key=value'), insert or replace entry */ +strbuf_reset (&key); +strbuf_add (&key, *p, equals - *p); +string_list_insert (&env, key.buf)->util = (void *) *p; + }else { +/* otherwise ('key') remove existing entry */ +string_list_remove (&env, *p,0 ); + } + } + +/* Create an array of 'char *' to be used as the childenv */ + childenv =xmalloc ((env.nr +1 ) *sizeof (char *)); +for (i =0 ; i < env.nr; i++) + childenv[i] = env.items[i].util; + childenv[env.nr] =NULL ; + +string_list_clear (&env,0 ); +strbuf_release (&key); +return childenv; +}#endif static inline void set_cloexec (int fd)
@@ -395,12 +444,14 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;prepare_cmd (&argv, cmd); + childenv =prep_childenv (cmd->env); cmd->pid =fork (); failed_errno = errno;
@@ -456,14 +507,6 @@ int start_command(struct child_process *cmd)if (cmd->dir &&chdir (cmd->dir))die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir); -if (cmd->env) { -for (; *cmd->env; cmd->env++) { -if (strchr (*cmd->env,'=' )) -putenv ((char *)*cmd->env); -else -unsetenv (*cmd->env); - } - }/* * Attempt to exec using the command and arguments starting at
@@ -471,9 +514,11 @@ int start_command(struct child_process *cmd)* be used in the event exec failed with ENOEXEC at which point * we will try to interpret the command using 'sh'. */ -execv (argv.argv[1 ], (char *const *) argv.argv +1 ); +execve (argv.argv[1 ], (char *const *) argv.argv +1 , + (char *const *) childenv);if (errno ==ENOEXEC ) -execv (argv.argv[0 ], (char *const *) argv.argv); +execve (argv.argv[0 ], (char *const *) argv.argv, + (char *const *) childenv);if (errno ==ENOENT ) {if (!cmd->silent_exec_failure)
@@ -509,6 +554,7 @@ int start_command(struct child_process *cmd)close (notify_pipe[0 ]);argv_array_clear (&argv); + free(childenv); }#else {
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v5 7/11 run-command: don't die in child when duping /dev/null
In reply to Brandon Williams--- run-command.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-)
run-command.c +13 -15
@@ -117,18 +117,6 @@ static inline void close_pair(int fd[2])close (fd[1 ]); } -#ifndef GIT_WINDOWS_NATIVE -static inline void dup_devnull (int to) -{ -int fd =open ("/dev/null" ,O_RDWR ); -if (fd <0 ) -die_errno (_ ("open /dev/null failed" )); -if (dup2 (fd, to) <0 ) -die_errno (_ ("dup2(%d,%d) failed" ), fd, to); -close (fd); -} -#endif -static char *locate_in_PATH (const char *file) {const char *p =getenv ("PATH" );
@@ -444,12 +432,20 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +int null_fd =-1 ;char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; +if (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) { + null_fd =open ("/dev/null" ,O_RDWR |O_CLOEXEC ); +if (null_fd <0 ) +die_errno (_ ("open /dev/null failed" )); +set_cloexec (null_fd); + } +prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env);
@@ -473,7 +469,7 @@ int start_command(struct child_process *cmd)atexit (notify_parent);if (cmd->no_stdin) -dup_devnull (0 ); +dup2 (null_fd,0 );else if (need_in) {dup2 (fdin[0 ],0 );close_pair (fdin);
@@ -483,7 +479,7 @@ int start_command(struct child_process *cmd) }if (cmd->no_stderr) -dup_devnull (2 ); +dup2 (null_fd,2 );else if (need_err) {dup2 (fderr[1 ],2 );close_pair (fderr);
@@ -493,7 +489,7 @@ int start_command(struct child_process *cmd) }if (cmd->no_stdout) -dup_devnull (1 ); +dup2 (null_fd,1 );else if (cmd->stdout_to_stderr)dup2 (2 ,1 );else if (need_out) {
@@ -553,6 +549,8 @@ int start_command(struct child_process *cmd) }close (notify_pipe[0 ]); +if (null_fd >=0 ) +close (null_fd);argv_array_clear (&argv);free (childenv); }
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v5 8/11 run-command: eliminate calls to error handling functions in child
In reply to Brandon WilliamsAll of our standard error handling paths have the potential to call malloc or take stdio locks; so we must avoid them inside the forked child.
Instead, the child only writes an 8 byte struct atomically to the parent through the notification pipe to propagate an error. All user-visible error reporting happens from the parent; even avoiding functions like atexit(3) and exit(3).
--- run-command.c | 121 ++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 89 insertions(+), 32 deletions(-)
run-command.c +89 -32
@@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ; -static void notify_parent (void ) +enum child_errcode { +CHILD_ERR_CHDIR , +CHILD_ERR_ENOENT , +CHILD_ERR_SILENT , +CHILD_ERR_ERRNO +}; + +struct child_err { +enum child_errcode err; +int syserr;/* errno */ +}; + +static void child_die (enum child_errcode err) { -/* -* execvp failed. If possible, we'd like to let start_command -* know, so failures like ENOENT can be handled right away; but -* otherwise, finish_command will still report the error. -*/ -xwrite (child_notifier,"" ,1 ); +struct child_err buf; + + buf.err = err; + buf.syserr = errno; + +/* write(2) on buf smaller than PIPE_BUF (min 512) is atomic: */ +xwrite (child_notifier, &buf,sizeof (buf)); +_exit (1 ); +} + +/* +* parent will make it look like the child spewed a fatal error and died +* this is needed to prevent changes to t0061. +*/ +static void fake_fatal (const char *err,va_list params) +{ +vreportf ("fatal: " , err, params); +} + +static void child_error_fn (const char *err,va_list params) +{ +const char msg[] ="error() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +} + +static void child_warn_fn (const char *err,va_list params) +{ +const char msg[] ="warn() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +} + +static void NORETURN child_die_fn(const char *err, va_list params) +{ +const char msg[] ="die() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +_exit (2 ); +} + +/* this runs in the parent process */ +static void child_err_spew (struct child_process *cmd,struct child_err *cerr) +{ +static void (*old_errfn)(const char *err,va_list params); + + old_errfn =get_error_routine (); +set_error_routine (fake_fatal); + errno = cerr->syserr; + +switch (cerr->err) { +case CHILD_ERR_CHDIR : +error_errno ("exec '%s': cd to '%s' failed" , + cmd->argv[0 ], cmd->dir); +break ; +case CHILD_ERR_ENOENT : +error_errno ("cannot run %s" , cmd->argv[0 ]); +break ; +case CHILD_ERR_SILENT : +break ; +case CHILD_ERR_ERRNO : +error_errno ("cannot exec '%s'" , cmd->argv[0 ]); +break ; + } +set_error_routine (old_errfn); }static void prepare_cmd (struct argv_array *out,const struct child_process *cmd)
@@ -341,13 +409,6 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal) code +=128 ; }else if (WIFEXITED (status)) { code =WEXITSTATUS (status); -/* -* Convert special exit code when execvp failed. -*/ -if (code ==127 ) { - code =-1 ; - failed_errno =ENOENT ; - } }else {error ("waitpid is confused (%s)" , argv0); }
@@ -435,6 +496,7 @@ int start_command(struct child_process *cmd)int null_fd =-1 ;char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ; +struct child_err cerr;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;
@@ -453,20 +515,16 @@ int start_command(struct child_process *cmd) failed_errno = errno;if (!cmd->pid) {/* -* Redirect the channel to write syscall error messages to -* before redirecting the process's stderr so that all die() -* in subsequent call paths use the parent's stderr. +* Ensure the default die/error/warn routines do not get +* called, they can take stdio locks and malloc. */ -if (cmd->no_stderr || need_err) { -int child_err =dup (2 ); -set_cloexec (child_err); -set_error_handle (fdopen (child_err,"w" )); - } +set_die_routine (child_die_fn); +set_error_routine (child_error_fn); +set_warn_routine (child_warn_fn);close (notify_pipe[0 ]);set_cloexec (notify_pipe[1 ]); child_notifier = notify_pipe[1 ]; -atexit (notify_parent);if (cmd->no_stdin)dup2 (null_fd,0 );
@@ -501,8 +559,7 @@ int start_command(struct child_process *cmd) }if (cmd->dir &&chdir (cmd->dir)) -die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], - cmd->dir); +child_die (CHILD_ERR_CHDIR );/* * Attempt to exec using the command and arguments starting at
@@ -517,12 +574,11 @@ int start_command(struct child_process *cmd) (char *const *) childenv);if (errno ==ENOENT ) { -if (!cmd->silent_exec_failure) -error ("cannot run %s: %s" , cmd->argv[0 ], -strerror (ENOENT )); -exit (127 ); +if (cmd->silent_exec_failure) +child_die (CHILD_ERR_SILENT ); +child_die (CHILD_ERR_ENOENT ); }else { -die_errno ("cannot exec '%s'" , cmd->argv[0 ]); +child_die (CHILD_ERR_ERRNO ); } }if (cmd->pid <0 )
@@ -533,17 +589,18 @@ int start_command(struct child_process *cmd)/* * Wait for child's exec. If the exec succeeds (or if fork() * failed), EOF is seen immediately by the parent. Otherwise, the -* child process sends a single byte. +* child process sends a child_err struct. * Note that use of this infrastructure is completely advisory, * therefore, we keep error checks minimal. */ close (notify_pipe[1 ]); -if (read (notify_pipe[0 ], ¬ify_pipe[1 ],1 ) ==1 ) { +if (xread (notify_pipe[0 ], &cerr,sizeof (cerr)) ==sizeof (cerr)) {/* * At this point we know that fork() succeeded, but exec() * failed. Errors have been reported to our stderr. */ wait_or_whine (cmd->pid, cmd->argv[0 ],0 ); +child_err_spew (cmd, &cerr); failed_errno = errno; cmd->pid =-1 ; }
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v5 9/11 run-command: handle dup2 and close errors in child
In reply to Brandon Williams--- run-command.c | 58 ++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 42 insertions(+), 16 deletions(-)
run-command.c +42 -16
@@ -213,6 +213,8 @@ static int child_notifier = -1;enum child_errcode {CHILD_ERR_CHDIR , +CHILD_ERR_DUP2 , +CHILD_ERR_CLOSE ,CHILD_ERR_ENOENT ,CHILD_ERR_SILENT ,CHILD_ERR_ERRNO
@@ -235,6 +237,24 @@ static void child_die(enum child_errcode err)_exit (1 ); } +static void child_dup2 (int fd,int to) +{ +if (dup2 (fd, to) <0 ) +child_die (CHILD_ERR_DUP2 ); +} + +static void child_close (int fd) +{ +if (close (fd)) +child_die (CHILD_ERR_CLOSE ); +} + +static void child_close_pair (int fd[2 ]) +{ +child_close (fd[0 ]); +child_close (fd[1 ]); +} +/* * parent will make it look like the child spewed a fatal error and died * this is needed to prevent changes to t0061.
@@ -277,6 +297,12 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)error_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir);break ; +case CHILD_ERR_DUP2 : +error_errno ("dup2() in child failed" ); +break ; +case CHILD_ERR_CLOSE : +error_errno ("close() in child failed" ); +break ;case CHILD_ERR_ENOENT :error_errno ("cannot run %s" , cmd->argv[0 ]);break ;
@@ -527,35 +553,35 @@ int start_command(struct child_process *cmd) child_notifier = notify_pipe[1 ];if (cmd->no_stdin) -dup2 (null_fd,0 ); +child_dup2 (null_fd,0 );else if (need_in) { -dup2 (fdin[0 ],0 ); -close_pair (fdin); +child_dup2 (fdin[0 ],0 ); +child_close_pair (fdin); }else if (cmd->in) { -dup2 (cmd->in,0 ); -close (cmd->in); +child_dup2 (cmd->in,0 ); +child_close (cmd->in); }if (cmd->no_stderr) -dup2 (null_fd,2 ); +child_dup2 (null_fd,2 );else if (need_err) { -dup2 (fderr[1 ],2 ); -close_pair (fderr); +child_dup2 (fderr[1 ],2 ); +child_close_pair (fderr); }else if (cmd->err >1 ) { -dup2 (cmd->err,2 ); -close (cmd->err); +child_dup2 (cmd->err,2 ); +child_close (cmd->err); }if (cmd->no_stdout) -dup2 (null_fd,1 ); +child_dup2 (null_fd,1 );else if (cmd->stdout_to_stderr) -dup2 (2 ,1 ); +child_dup2 (2 ,1 );else if (need_out) { -dup2 (fdout[1 ],1 ); -close_pair (fdout); +child_dup2 (fdout[1 ],1 ); +child_close_pair (fdout); }else if (cmd->out >1 ) { -dup2 (cmd->out,1 ); -close (cmd->out); +child_dup2 (cmd->out,1 ); +child_close (cmd->out); }if (cmd->dir &&chdir (cmd->dir))
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v5 10/11 run-command: add note about forking and threading
In reply to Brandon WilliamsAll non-Async-Signal-Safe functions (e.g. malloc and die) were removed between 'fork' and 'exec' in start_command in order to avoid potential deadlocking when forking while multiple threads are running. This deadlocking is possible when a thread (other than the one forking) has acquired a lock and didn't get around to releasing it before the fork. This leaves the lock in a locked state in the resulting process with no hope of it ever being released.
Add a note describing this potential pitfall before the call to 'fork()' so people working in this section of the code know to only use Async-Signal-Safe functions in the child process.
--- run-command.c | 9 +++++++++ 1 file changed, 9 insertions(+)
run-command.c +9 -0
@@ -537,6 +537,15 @@ int start_command(struct child_process *cmd)prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env); +/* +* NOTE: In order to prevent deadlocking when using threads special +* care should be taken with the function calls made in between the +* fork() and exec() calls. No calls should be made to functions which +* require acquiring a lock (e.g. malloc) as the lock could have been +* held by another thread at the time of forking, causing the lock to +* never be released in the child process. This means only +* Async-Signal-Safe functions are permitted in the child. +*/ cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) {
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v5 11/11 run-command: block signals between fork and execve
In reply to Brandon WilliamsFrom: Eric Wong <[email protected]>
Signal handlers of the parent firing in the forked child may have unintended side effects. Rather than auditing every signal handler we have and will ever have, block signals while forking and restore default signal handlers in the child before execve.
Restoring default signal handlers is required because execve does not unblock signals, it only restores default signal handlers. So we must restore them with sigprocmask before execve, leaving a window when signal handlers we control can fire in the child. Continue ignoring ignored signals, but reset the rest to defaults.
Similarly, disable pthread cancellation to future-proof our code in case we start using cancellation; as cancellation is implemented with signals in glibc.
--- run-command.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+)
run-command.c +68 -0
@@ -215,6 +215,7 @@ enum child_errcode {CHILD_ERR_CHDIR ,CHILD_ERR_DUP2 ,CHILD_ERR_CLOSE , +CHILD_ERR_SIGPROCMASK ,CHILD_ERR_ENOENT ,CHILD_ERR_SILENT ,CHILD_ERR_ERRNO
@@ -303,6 +304,9 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)case CHILD_ERR_CLOSE :error_errno ("close() in child failed" );break ; +case CHILD_ERR_SIGPROCMASK : +error_errno ("sigprocmask failed restoring signals" ); +break ;case CHILD_ERR_ENOENT :error_errno ("cannot run %s" , cmd->argv[0 ]);break ;
@@ -400,6 +404,53 @@ static char **prep_childenv(const char *const *deltaenv) }#endif +struct atfork_state { +#ifndef NO_PTHREADS +int cs; +#endif +sigset_t old; +}; + +#ifndef NO_PTHREADS +static void bug_die (int err,const char *msg) +{ +if (err) { + errno = err; +die_errno ("BUG: %s" , msg); + } +} +#endif + +static void atfork_prepare (struct atfork_state *as) +{ +sigset_t all; + +if (sigfillset (&all)) +die_errno ("sigfillset" ); +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &all, &as->old)) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_sigmask (SIG_SETMASK , &all, &as->old), +"blocking all signals" ); +bug_die (pthread_setcancelstate (PTHREAD_CANCEL_DISABLE , &as->cs), +"disabling cancellation" ); +#endif +} + +static void atfork_parent (struct atfork_state *as) +{ +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &as->old,NULL )) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_setcancelstate (as->cs,NULL ), +"re-enabling cancellation" ); +bug_die (pthread_sigmask (SIG_SETMASK , &as->old,NULL ), +"restoring signal mask" ); +#endif +} +static inline void set_cloexec (int fd) {int flags =fcntl (fd,F_GETFD );
@@ -523,6 +574,7 @@ int start_command(struct child_process *cmd)char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;struct child_err cerr; +struct atfork_state as;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;
@@ -536,6 +588,7 @@ int start_command(struct child_process *cmd)prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env); +atfork_prepare (&as);/* * NOTE: In order to prevent deadlocking when using threads special
@@ -549,6 +602,7 @@ int start_command(struct child_process *cmd) cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) { +int sig;/* * Ensure the default die/error/warn routines do not get * called, they can take stdio locks and malloc.
@@ -597,6 +651,19 @@ int start_command(struct child_process *cmd)child_die (CHILD_ERR_CHDIR );/* +* restore default signal handlers here, in case +* we catch a signal right before execve below +*/ +for (sig =1 ; sig <NSIG ; sig++) { +/* ignored signals get reset to SIG_DFL on execve */ +if (signal (sig,SIG_DFL ) ==SIG_IGN ) +signal (sig,SIG_IGN ); + } + +if (sigprocmask (SIG_SETMASK , &as.old,NULL ) !=0 ) +child_die (CHILD_ERR_SIGPROCMASK ); + +/* * Attempt to exec using the command and arguments starting at * argv.argv[1]. argv.argv[0] contains SHELL_PATH which will * be used in the event exec failed with ENOEXEC at which point
@@ -616,6 +683,7 @@ int start_command(struct child_process *cmd)child_die (CHILD_ERR_ERRNO ); } } +atfork_parent (&as);if (cmd->pid <0 )error_errno ("cannot fork() for %s" , cmd->argv[0 ]);else if (cmd->clean_on_exit)
Signature · 1 line
2.12.2.816.g2cccc81164-goog
Re: [PATCH v5 11/11] run-command: block signals between fork and execve
Am 19.04.2017 um 01:18 schrieb Brandon Williams:
Review static char **prep_childenv(const char *const *deltaenv) · line 406
Quoted 4 lines
(quoted diff) +0
-0@@ -400,6 +404,53 @@ static char **prep_childenv(const char *const *deltaenv) } #endif
Does this #endif in this hunk context belong to an #ifndef GIT_WINDOWS_NATIVE? If so, I wonder why these new functions are outside these brackets? An oversight?
run-command.c static char **prep_childenv(const char *const *deltaenv) · line 412
Quoted 6 lines
run-command.c +6
-0+struct atfork_state { +#ifndef NO_PTHREADS +int cs; +#endif +sigset_t old; +};
...
-- Hannes
Re: [PATCH v5 11/11] run-command: block signals between fork and execve
Quoted 9 lines · Johannes Sixt
Am 19.04.2017 um 01:18 schrieb Brandon Williams:
Review static char **prep_childenv(const char *const *deltaenv) · line 406
Quoted 4 lines
(quoted diff) +0
-0@@ -400,6 +404,53 @@ static char **prep_childenv(const char *const *deltaenv) } #endifDoes this #endif in this hunk context belong to an #ifndef
GIT_WINDOWS_NATIVE? If so, I wonder why these new functions are outside these brackets? An oversight?
Seems like an oversight, sorry about that.
All the new atfork stuff I added should be protected by #ifndef GIT_WINDOWS_NATIVE.
Brandon / Johannes: can you fixup on your end?
I wonder if some of this OS-specific code would be more easily maintained if split out further to OS-specific files, even at the risk of some code duplication.
And/or perhaps label all #else and #endif statements with comments, and limit the scope of each ifdef block to be per-function for with tiny attention spans like me :x
Quoted 10 lines
run-command.c static char **prep_childenv(const char *const *deltaenv) · line 412
Quoted 6 lines
run-command.c +6
-0+struct atfork_state { +#ifndef NO_PTHREADS +int cs; +#endif +sigset_t old; +};...
-- Hannes
Re: [PATCH v5 11/11] run-command: block signals between fork and execve
Quoted 11 lines · Eric Wong
Quoted 9 lines · Johannes Sixt
Am 19.04.2017 um 01:18 schrieb Brandon Williams:
Review static char **prep_childenv(const char *const *deltaenv) · line 406
Quoted 4 lines
(quoted diff) +0
-0@@ -400,6 +404,53 @@ static char **prep_childenv(const char *const *deltaenv) } #endifDoes this #endif in this hunk context belong to an #ifndef
GIT_WINDOWS_NATIVE? If so, I wonder why these new functions are outside these brackets? An oversight?
Quoted 5 lines
Seems like an oversight, sorry about that.
All the new atfork stuff I added should be protected by #ifndefGIT_WINDOWS_NATIVE.Brandon / Johannes: can you fixup on your end?
Correct, this is an oversight I should have caught :)
No worries though, I'll fix it up in a reroll (since I'm going to be need to send out another version to fix up another patch in the series for Windows)
Quoted 8 lines
I wonder if some of this OS-specific code would be more easily maintained if split out further to OS-specific files, even at the risk of some code duplication.
And/or perhaps label all #else and #endif statements with comments, and limit the scope of each ifdef block to be per-function for with tiny attention spans like me :x
Yeah I'm not sure I know the best way to prevent this from happening, thankfully we have windows folk who keep us honest :D
Quoted 11 lines
Quoted 11 lines
run-command.c static char **prep_childenv(const char *const *deltaenv) · line 412
Quoted 7 lines
run-command.c +6
-0+struct atfork_state { +#ifndef NO_PTHREADS +int cs; +#endif +sigset_t old; +};...
-- Hannes
Signature · 1 line
Brandon Williams
v6 cover forking and threading
In reply to Brandon WilliamsChanges in v6:
* fix some windows compat issues
* better comment on the string_list_remove function (also marked extern)
Brandon Williams (10):
t5550: use write_script to generate post-update hook
t0061: run_command executes scripts without a #! line
run-command: prepare command before forking
run-command: use the async-signal-safe execv instead of execvp
string-list: add string_list_remove function
run-command: prepare child environment before forking
run-command: don't die in child when duping /dev/null
run-command: eliminate calls to error handling functions in child
run-command: handle dup2 and close errors in child
run-command: add note about forking and threading
Eric Wong (1):
run-command: block signals between fork and execve
run-command.c | 404 +++++++++++++++++++++++++++++++++++---------- string-list.c | 18 ++ string-list.h | 7 + t/t0061-run-command.sh | 11 ++ t/t5550-http-fetch-dumb.sh | 5 +- 5 files changed, 360 insertions(+), 85 deletions(-)
--- interdiff with 'origin/bw/forking-and-threading'
run-command.c +1 -1
@@ -402,7 +402,6 @@ static char **prep_childenv(const char *const *deltaenv)strbuf_release (&key);return childenv; } -#endif struct atfork_state {#ifndef NO_PTHREADS
@@ -450,6 +449,7 @@ static void atfork_parent(struct atfork_state *as) "restoring signal mask"); #endif } +#endif /* GIT_WINDOWS_NATIVE */ static inline void set_cloexec (int fd) {
string-list.h +3 -1
@@ -64,8 +64,10 @@ struct string_list_item *string_list_insert(struct string_list *list, const char/* * Removes the given string from the sorted list. +* If the string doesn't exist, the list is not altered. */ -void string_list_remove (struct string_list *list,const char *string,int free_util); +extern void string_list_remove (struct string_list *list,const char *string, +int free_util);/* * Checks if the given string is part of a sorted list. If it is part of the list,
t/t0061-run-command.sh +1 -1
@@ -26,7 +26,7 @@ test_expect_success 'run_command can run a command' ' test_cmp empty err ' -test_expect_success 'run_command can run a script without a #! line' ' +test_expect_success !MINGW 'run_command can run a script without a #! line' ' cat >hello <<-\EOF && cat hello-script EOF
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v6 1/11 t5550: use write_script to generate post-update hook
The post-update hooks created in t5550-http-fetch-dumb.sh is missing the "!#/bin/sh" line which can cause issues with portability. Instead create the hook using the 'write_script' function which includes the proper "#!" line.
--- t/t5550-http-fetch-dumb.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
t/t5550-http-fetch-dumb.sh +3 -2
@@ -20,8 +20,9 @@ test_expect_success 'create http-accessible bare repository with loose objects' (cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" && git config core.bare true && mkdir -p hooks && - echo "exec git update-server-info" >hooks/post-update && - chmod +x hooks/post-update && + write_script "hooks/post-update" <<-\EOF && + exec git update-server-info + EOF hooks/post-update ) && git remote add public "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v6 2/11 t0061: run_command executes scripts without a #! line
In reply to Brandon WilliamsAdd a test to 't0061-run-command.sh' to ensure that run_command can continue to execute scripts which don't include a '#!' line.
--- t/t0061-run-command.sh | 11 +++++++++++ 1 file changed, 11 insertions(+)
t/t0061-run-command.sh +11 -0
@@ -26,6 +26,17 @@ test_expect_success 'run_command can run a command' ' test_cmp empty err ' +test_expect_success !MINGW 'run_command can run a script without a #! line' ' + cat >hello <<-\EOF && + cat hello-script + EOF + chmod +x hello && + test-run-command run-command ./hello >actual 2>err && + + test_cmp hello-script actual && + test_cmp empty err +' + test_expect_success POSIXPERM 'run_command reports EACCES' ' cat hello-script >hello.sh && chmod -x hello.sh &&
Signature · 1 line
2.12.2.816.g2cccc81164-goog
Re: [PATCH v6 02/11] t0061: run_command executes scripts without a #! line
Hi Brandon,
Quoted 4 lines · Brandon Williams
Add a test to '
t0061-run-command.sh' to ensure thatrun_commandcan continue to execute scripts which don't include a '#!' line.Signed-off-by Brandon Williams <[email protected]>
Please add something like this to the commit message lest future readers wonder where that !MINGW comes from:
On Windows, shell scripts are not natively executable. Git has a workaround to execute them, looking for the shebang line. Shell scripts without a shebang line will simply not execute on Windows. Therefore, disable the new test on Windows.
Thanks,
Dscho
Re: [PATCH v6 02/11] t0061: run_command executes scripts without a #! line
Quoted 9 lines · Johannes Schindelin
Hi Brandon,
Quoted 4 lines · Brandon Williams
Add a test to '
t0061-run-command.sh' to ensure thatrun_commandcan continue to execute scripts which don't include a '#!' line.Signed-off-by Brandon Williams <[email protected]>
Quoted 7 lines
Please add something like this to the commit message lest future readers wonder where that !MINGW comes from:
On Windows, shell scripts are not natively executable. Git has a workaround to execute them, looking for the shebang line. Shell scripts without a shebang line will simply not execute on Windows. Therefore, disable the new test on Windows.
I'm fine with including this in the commit message. If another reroll needs to happen then I can make the update otherwise hopefully Junio can just squash that in.
Quoted 3 lines
Thanks,
Dscho
Signature · 1 line
Brandon Williams
PATCH v6 3/11 run-command: prepare command before forking
In reply to Brandon WilliamsAccording to [1] we need to only call async-signal-safe operations between fork and exec. Using malloc to build the argv array isn't async-signal-safe.
In order to avoid allocation between 'fork()' and 'exec()' prepare the argv array used in the exec call prior to forking the process.
[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html
--- run-command.c | 46 ++++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-)
run-command.c +26 -20
@@ -221,18 +221,6 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv) }#ifndef GIT_WINDOWS_NATIVE -static int execv_shell_cmd (const char **argv) -{ -struct argv_array nargv =ARGV_ARRAY_INIT ; -prepare_shell_cmd (&nargv, argv); -trace_argv_printf (nargv.argv,"trace: exec:" ); -sane_execvp (nargv.argv[0 ], (char **)nargv.argv); -argv_array_clear (&nargv); -return -1 ; -} -#endif - -#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ;static void notify_parent (void )
@@ -244,6 +232,21 @@ static void notify_parent(void)*/ xwrite(child_notifier, "", 1); } + +static void prepare_cmd (struct argv_array *out,const struct child_process *cmd) +{ +if (!cmd->argv[0 ]) +die ("BUG: command is empty" ); + +if (cmd->git_cmd) { +argv_array_push (out,"git" ); +argv_array_pushv (out, cmd->argv); + }else if (cmd->use_shell) { +prepare_shell_cmd (out, cmd->argv); + }else { +argv_array_pushv (out, cmd->argv); + } +}#endif static inline void set_cloexec (int fd)
@@ -372,9 +375,13 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +struct argv_array argv =ARGV_ARRAY_INIT ; +if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; +prepare_cmd (&argv, cmd); + cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) {
@@ -437,12 +444,9 @@ int start_command(struct child_process *cmd)unsetenv (*cmd->env); } } -if (cmd->git_cmd) -execv_git_cmd (cmd->argv); -else if (cmd->use_shell) -execv_shell_cmd (cmd->argv); -else -sane_execvp (cmd->argv[0 ], (char *const *) cmd->argv); + +sane_execvp (argv.argv[0 ], (char *const *) argv.argv); +if (errno ==ENOENT ) {if (!cmd->silent_exec_failure) error("cannot run %s: %s", cmd->argv[0],
@@ -458,7 +462,7 @@ int start_command(struct child_process *cmd)mark_child_for_cleanup (cmd->pid, cmd);/* -* Wait for child's execvp. If the execvp succeeds (or if fork() +* Wait for child's exec. If the exec succeeds (or if fork() * failed), EOF is seen immediately by the parent. Otherwise, the * child process sends a single byte. * Note that use of this infrastructure is completely advisory,
@@ -467,7 +471,7 @@ int start_command(struct child_process *cmd)close (notify_pipe[1 ]);if (read (notify_pipe[0 ], ¬ify_pipe[1 ],1 ) ==1 ) {/* -* At this point we know that fork() succeeded, but execvp() +* At this point we know that fork() succeeded, but exec() * failed. Errors have been reported to our stderr. */ wait_or_whine (cmd->pid, cmd->argv[0 ],0 );
@@ -475,6 +479,8 @@ int start_command(struct child_process *cmd) cmd->pid = -1; }close (notify_pipe[0 ]); + +argv_array_clear (&argv); }#else {
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v6 4/11 run-command: use the async-signal-safe execv instead of execvp
In reply to Brandon WilliamsConvert the function used to exec from 'execvp()' to 'execv()' as the (p) variant of exec isn't async-signal-safe and has the potential to call malloc during the path resolution it performs. Instead we simply do the path resolution ourselves during the preparation stage prior to forking. There also don't exist any portable (p) variants which also take in an environment to use in the exec'd process. This allows easy migration to using 'execve()' in a future patch.
Also, as noted in [1], in the event of an ENOEXEC the (p) variants of exec will attempt to execute the command by interpreting it with the 'sh' utility. To maintain this functionality, if 'execv()' fails with ENOEXEC, start_command will atempt to execute the command by interpreting it with 'sh'.
[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html
--- run-command.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
run-command.c +29 -1
@@ -238,6 +238,12 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)if (!cmd->argv[0 ])die ("BUG: command is empty" ); +/* +* Add SHELL_PATH so in the event exec fails with ENOEXEC we can +* attempt to interpret the command with 'sh'. +*/ +argv_array_push (out,SHELL_PATH ); +if (cmd->git_cmd) {argv_array_push (out,"git" );argv_array_pushv (out, cmd->argv);
@@ -246,6 +252,20 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) } else {argv_array_pushv (out, cmd->argv); } + +/* +* If there are no '/' characters in the command then perform a path +* lookup and use the resolved path as the command to exec. If there +* are no '/' characters or if the command wasn't found in the path, +* have exec attempt to invoke the command directly. +*/ +if (!strchr (out->argv[1 ],'/' )) { +char *program =locate_in_PATH (out->argv[1 ]); +if (program) { +free ((char *)out->argv[1 ]); + out->argv[1 ] = program; + } + } }#endif
@@ -445,7 +465,15 @@ int start_command(struct child_process *cmd) } } -sane_execvp (argv.argv[0 ], (char *const *) argv.argv); +/* +* Attempt to exec using the command and arguments starting at +* argv.argv[1]. argv.argv[0] contains SHELL_PATH which will +* be used in the event exec failed with ENOEXEC at which point +* we will try to interpret the command using 'sh'. +*/ +execv (argv.argv[1 ], (char *const *) argv.argv +1 ); +if (errno ==ENOEXEC ) +execv (argv.argv[0 ], (char *const *) argv.argv);if (errno ==ENOENT ) {if (!cmd->silent_exec_failure)
Signature · 1 line
2.12.2.816.g2cccc81164-goog
Re: [PATCH v6 04/11] run-command: use the async-signal-safe execv instead of execvp
run-command.c static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) · line 249
Quoted 13 lines · Brandon Williams
run-command.c +6
-0@@ -238,6 +238,12 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd)if (!cmd->argv[0 ])die ("BUG: command is empty" ); +/* +* Add SHELL_PATH so in the event exec fails with ENOEXEC we can +* attempt to interpret the command with 'sh'. +*/ +argv_array_push (out,SHELL_PATH ); +if (cmd->git_cmd) {argv_array_push (out,"git" );argv_array_pushv (out, cmd->argv);
So, given "cat-file", "-t", "HEAD" with cmd->git_cmd == TRUE, this will now prepare ("/bin/sh", "git", "cat-file", "-t", "HEAD", NULL) in the argv-array, and then ...
Quoted 8 lines
run-command.c +4
-0@@ -246,6 +252,20 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) }else {argv_array_pushv (out, cmd->argv); } + +/* +* If there are no '/' characters in the command then perform a path +* lookup and use the resolved path as the command to exec. If there
run-command.c static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) · line 268
Quoted 10 lines
run-command.c +10
-0+* are no '/' characters or if the command wasn't found in the path, +* have exec attempt to invoke the command directly. +*/ +if (!strchr (out->argv[1 ],'/' )) { +char *program =locate_in_PATH (out->argv[1 ]); +if (program) { +free ((char *)out->argv[1 ]); + out->argv[1 ] = program; + } + }
... turn the first element from "git" to "/usr/bin/git", i.e.
("/bin/sh", "/usr/bin/git", "cat-file", "-t", "HEAD", NULL)which ...
Quoted 4 lines
run-command.c +0
-0#endif @@ -445,7 +465,15 @@ int start_command(struct child_process *cmd) }
run-command.c int start_command(struct child_process *cmd) · line 474
Quoted 10 lines
run-command.c +7
-1} -sane_execvp (argv.argv[0 ], (char *const *) argv.argv); +/* +* Attempt to exec using the command and arguments starting at +* argv.argv[1]. argv.argv[0] contains SHELL_PATH which will +* be used in the event exec failed with ENOEXEC at which point +* we will try to interpret the command using 'sh'. +*/ +execv (argv.argv[1 ], (char *const *) argv.argv +1 );
... first is given without the leading "/bin/sh", as the end-user intended (sort of), but if it fails
run-command.c int start_command(struct child_process *cmd) · line 476
Quoted 2 lines
run-command.c +2
-0+if (errno ==ENOEXEC ) +execv (argv.argv[0 ], (char *const *) argv.argv);
"/bin/sh" tries to run "/usr/bin/git" that was not executable (well, the one in "usr/bin/" would have +x bit, but let's pretend that we are trying to run one from bin-wrappers/ and somehow forgot +x bit)?
I think all of that is sensible, but there is one "huh?" I can't figure out. Typically we do "sh -c git cat-file -t HEAD" but this lacks the "-c" (cf. the original prepare_shell_cmd()); why do we not need it in this case?
Thanks.
Re: [PATCH v6 04/11] run-command: use the async-signal-safe execv instead of execvp
Quoted 11 lines · Junio C Hamano
run-command.c int start_command(struct child_process *cmd) · line 476
Quoted 2 lines
run-command.c +2
-0+if (errno ==ENOEXEC ) +execv (argv.argv[0 ], (char *const *) argv.argv);"
/bin/sh" tries to run "/usr/bin/git" that was not executable (well, the one in "usr/bin/" would have +x bit, but let's pretend that we are trying to run one from bin-wrappers/ and somehow forgot +x bit)?I think all of that is sensible, but there is one "huh?" I can't figure out. Typically we do "sh -c git cat-file -t HEAD" but this lacks the "-c" (cf. the original
prepare_shell_cmd()); why do we not need it in this case?
I think this is the same case we were discussing over in the "rebase" thread. This isn't about running the user's command as a shell command. Note that this kicks in even when cmd->shell_cmd isn't set.
This is about finding "/usr/bin/foo", realizing it cannot be exec'd because it lacks a shebang line, and then pretending that it did have "#!/bin/sh". IOW, maintaining compatibility with execvp().
So the command itself isn't a shell command, but it may execute a shell script. If that makes sense.
-Peff
Re: [PATCH v6 04/11] run-command: use the async-signal-safe execv instead of execvp
Quoted 18 lines · Jeff King
Quoted 11 lines · Junio C Hamano
run-command.c int start_command(struct child_process *cmd) · line 476
Quoted 2 lines
run-command.c +2
-0+if (errno ==ENOEXEC ) +execv (argv.argv[0 ], (char *const *) argv.argv);"
/bin/sh" tries to run "/usr/bin/git" that was not executable (well, the one in "usr/bin/" would have +x bit, but let's pretend that we are trying to run one from bin-wrappers/ and somehow forgot +x bit)?I think all of that is sensible, but there is one "huh?" I can't figure out. Typically we do "sh -c git cat-file -t HEAD" but this lacks the "-c" (cf. the original
prepare_shell_cmd()); why do we not need it in this case?I think this is the same case we were discussing over in the "rebase" thread. This isn't about running the user's command as a shell command. Note that this kicks in even when
cmd->shell_cmdisn't set.
Quoted 6 lines
This is about finding "
/usr/bin/foo", realizing it cannot be exec'd because it lacks a shebang line, and then pretending that it did have "#!/bin/sh". IOW, maintaining compatibility withexecvp().So the command itself isn't a shell command, but it may execute a shell script. If that makes sense.
And note that this isn't about the "+x" bit. That would result in EACCES. Getting ENOEXEC means that the contents of the binary are not something execv thought it could run.
-Peff
Re: [PATCH v6 04/11] run-command: use the async-signal-safe execv instead of execvp
In reply to Jeff KingQuoted 6 lines · Jeff King
This is about finding "
/usr/bin/foo", realizing it cannot be exec'd because it lacks a shebang line, and then pretending that it did have "#!/bin/sh". IOW, maintaining compatibility withexecvp().So the command itself isn't a shell command, but it may execute a shell script. If that makes sense.
Ah, OK. Thanks.
Re: [PATCH v6 04/11] run-command: use the async-signal-safe execv instead of execvp
In reply to Jeff KingQuoted 14 lines · Jeff King
Quoted 11 lines · Junio C Hamano
run-command.c int start_command(struct child_process *cmd) · line 476
Quoted 2 lines
run-command.c +2
-0+if (errno ==ENOEXEC ) +execv (argv.argv[0 ], (char *const *) argv.argv);"
/bin/sh" tries to run "/usr/bin/git" that was not executable (well, the one in "usr/bin/" would have +x bit, but let's pretend that we are trying to run one from bin-wrappers/ and somehow forgot +x bit)?I think all of that is sensible, but there is one "huh?" I can't figure out. Typically we do "sh -c git cat-file -t HEAD" but this lacks the "-c" (cf. the original
prepare_shell_cmd()); why do we not need it in this case?
Quoted 7 lines
I think this is the same case we were discussing over in the "rebase" thread. This isn't about running the user's command as a shell command. Note that this kicks in even when
cmd->shell_cmdisn't set.This is about finding "
/usr/bin/foo", realizing it cannot be exec'd because it lacks a shebang line, and then pretending that it did have "#!/bin/sh". IOW, maintaining compatibility withexecvp().
Exactly this is all about ensuring we do the same thing the execvp does, because there isn't a portable variant which allows for passing in an environment.
Quoted 5 lines
So the command itself isn't a shell command, but it may execute a shell script. If that makes sense.
-Peff
Signature · 1 line
Brandon Williams
PATCH v6 5/11 string-list: add string_list_remove function
In reply to Brandon WilliamsTeach string-list to be able to remove a string from a sorted 'struct string_list'.
--- string-list.c | 18 ++++++++++++++++++ string-list.h | 7 +++++++ 2 files changed, 25 insertions(+)
string-list.c +18 -0
@@ -67,6 +67,24 @@ struct string_list_item *string_list_insert(struct string_list *list, const charreturn list->items + index; } +void string_list_remove (struct string_list *list,const char *string, +int free_util) +{ +int exact_match; +int i =get_entry_index (list, string, &exact_match); + +if (exact_match) { +if (list->strdup_strings) +free (list->items[i].string); +if (free_util) +free (list->items[i].util); + + list->nr--; +memmove (list->items + i, list->items + i +1 , + (list->nr - i) *sizeof (struct string_list_item )); + } +} +int string_list_has_string (const struct string_list *list,const char *string) {int exact_match;
string-list.h +7 -0
@@ -63,6 +63,13 @@ int string_list_find_insert_index(const struct string_list *list, const char *ststruct string_list_item *string_list_insert (struct string_list *list,const char *string);/* +* Removes the given string from the sorted list. +* If the string doesn't exist, the list is not altered. +*/ +extern void string_list_remove (struct string_list *list,const char *string, +int free_util); + +/* * Checks if the given string is part of a sorted list. If it is part of the list, * return the coresponding string_list_item, NULL otherwise. */
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v6 6/11 run-command: prepare child environment before forking
In reply to Brandon WilliamsIn order to avoid allocation between 'fork()' and 'exec()' prepare the environment to be used in the child process prior to forking.
Switch to using 'execve()' so that the construct child environment can used in the exec'd process.
--- run-command.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 56 insertions(+), 10 deletions(-)
run-command.c +56 -10
@@ -267,6 +267,55 @@ static void prepare_cmd(struct argv_array *out, const struct child_process *cmd) } } } + +static char **prep_childenv (const char *const *deltaenv) +{ +extern char **environ; +char **childenv; +struct string_list env =STRING_LIST_INIT_DUP ; +struct strbuf key =STRBUF_INIT ; +const char *const *p; +int i; + +/* Construct a sorted string list consisting of the current environ */ +for (p = (const char *const *) environ; p && *p; p++) { +const char *equals =strchr (*p,'=' ); + +if (equals) { +strbuf_reset (&key); +strbuf_add (&key, *p, equals - *p); +string_list_append (&env, key.buf)->util = (void *) *p; + }else { +string_list_append (&env, *p)->util = (void *) *p; + } + } +string_list_sort (&env); + +/* Merge in 'deltaenv' with the current environ */ +for (p = deltaenv; p && *p; p++) { +const char *equals =strchr (*p,'=' ); + +if (equals) { +/* ('key=value'), insert or replace entry */ +strbuf_reset (&key); +strbuf_add (&key, *p, equals - *p); +string_list_insert (&env, key.buf)->util = (void *) *p; + }else { +/* otherwise ('key') remove existing entry */ +string_list_remove (&env, *p,0 ); + } + } + +/* Create an array of 'char *' to be used as the childenv */ + childenv =xmalloc ((env.nr +1 ) *sizeof (char *)); +for (i =0 ; i < env.nr; i++) + childenv[i] = env.items[i].util; + childenv[env.nr] =NULL ; + +string_list_clear (&env,0 ); +strbuf_release (&key); +return childenv; +}#endif static inline void set_cloexec (int fd)
@@ -395,12 +444,14 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;prepare_cmd (&argv, cmd); + childenv =prep_childenv (cmd->env); cmd->pid =fork (); failed_errno = errno;
@@ -456,14 +507,6 @@ int start_command(struct child_process *cmd)if (cmd->dir &&chdir (cmd->dir))die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir); -if (cmd->env) { -for (; *cmd->env; cmd->env++) { -if (strchr (*cmd->env,'=' )) -putenv ((char *)*cmd->env); -else -unsetenv (*cmd->env); - } - }/* * Attempt to exec using the command and arguments starting at
@@ -471,9 +514,11 @@ int start_command(struct child_process *cmd)* be used in the event exec failed with ENOEXEC at which point * we will try to interpret the command using 'sh'. */ -execv (argv.argv[1 ], (char *const *) argv.argv +1 ); +execve (argv.argv[1 ], (char *const *) argv.argv +1 , + (char *const *) childenv);if (errno ==ENOEXEC ) -execv (argv.argv[0 ], (char *const *) argv.argv); +execve (argv.argv[0 ], (char *const *) argv.argv, + (char *const *) childenv);if (errno ==ENOENT ) {if (!cmd->silent_exec_failure)
@@ -509,6 +554,7 @@ int start_command(struct child_process *cmd)close (notify_pipe[0 ]);argv_array_clear (&argv); + free(childenv); }#else {
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v6 7/11 run-command: don't die in child when duping /dev/null
In reply to Brandon Williams--- run-command.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-)
run-command.c +13 -15
@@ -117,18 +117,6 @@ static inline void close_pair(int fd[2])close (fd[1 ]); } -#ifndef GIT_WINDOWS_NATIVE -static inline void dup_devnull (int to) -{ -int fd =open ("/dev/null" ,O_RDWR ); -if (fd <0 ) -die_errno (_ ("open /dev/null failed" )); -if (dup2 (fd, to) <0 ) -die_errno (_ ("dup2(%d,%d) failed" ), fd, to); -close (fd); -} -#endif -static char *locate_in_PATH (const char *file) {const char *p =getenv ("PATH" );
@@ -444,12 +432,20 @@ int start_command(struct child_process *cmd)#ifndef GIT_WINDOWS_NATIVE {int notify_pipe[2 ]; +int null_fd =-1 ;char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ; +if (cmd->no_stdin || cmd->no_stdout || cmd->no_stderr) { + null_fd =open ("/dev/null" ,O_RDWR |O_CLOEXEC ); +if (null_fd <0 ) +die_errno (_ ("open /dev/null failed" )); +set_cloexec (null_fd); + } +prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env);
@@ -473,7 +469,7 @@ int start_command(struct child_process *cmd)atexit (notify_parent);if (cmd->no_stdin) -dup_devnull (0 ); +dup2 (null_fd,0 );else if (need_in) {dup2 (fdin[0 ],0 );close_pair (fdin);
@@ -483,7 +479,7 @@ int start_command(struct child_process *cmd) }if (cmd->no_stderr) -dup_devnull (2 ); +dup2 (null_fd,2 );else if (need_err) {dup2 (fderr[1 ],2 );close_pair (fderr);
@@ -493,7 +489,7 @@ int start_command(struct child_process *cmd) }if (cmd->no_stdout) -dup_devnull (1 ); +dup2 (null_fd,1 );else if (cmd->stdout_to_stderr)dup2 (2 ,1 );else if (need_out) {
@@ -553,6 +549,8 @@ int start_command(struct child_process *cmd) }close (notify_pipe[0 ]); +if (null_fd >=0 ) +close (null_fd);argv_array_clear (&argv);free (childenv); }
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v6 8/11 run-command: eliminate calls to error handling functions in child
In reply to Brandon WilliamsAll of our standard error handling paths have the potential to call malloc or take stdio locks; so we must avoid them inside the forked child.
Instead, the child only writes an 8 byte struct atomically to the parent through the notification pipe to propagate an error. All user-visible error reporting happens from the parent; even avoiding functions like atexit(3) and exit(3).
--- run-command.c | 121 ++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 89 insertions(+), 32 deletions(-)
run-command.c +89 -32
@@ -211,14 +211,82 @@ static const char **prepare_shell_cmd(struct argv_array *out, const char **argv)#ifndef GIT_WINDOWS_NATIVE static int child_notifier =-1 ; -static void notify_parent (void ) +enum child_errcode { +CHILD_ERR_CHDIR , +CHILD_ERR_ENOENT , +CHILD_ERR_SILENT , +CHILD_ERR_ERRNO +}; + +struct child_err { +enum child_errcode err; +int syserr;/* errno */ +}; + +static void child_die (enum child_errcode err) { -/* -* execvp failed. If possible, we'd like to let start_command -* know, so failures like ENOENT can be handled right away; but -* otherwise, finish_command will still report the error. -*/ -xwrite (child_notifier,"" ,1 ); +struct child_err buf; + + buf.err = err; + buf.syserr = errno; + +/* write(2) on buf smaller than PIPE_BUF (min 512) is atomic: */ +xwrite (child_notifier, &buf,sizeof (buf)); +_exit (1 ); +} + +/* +* parent will make it look like the child spewed a fatal error and died +* this is needed to prevent changes to t0061. +*/ +static void fake_fatal (const char *err,va_list params) +{ +vreportf ("fatal: " , err, params); +} + +static void child_error_fn (const char *err,va_list params) +{ +const char msg[] ="error() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +} + +static void child_warn_fn (const char *err,va_list params) +{ +const char msg[] ="warn() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +} + +static void NORETURN child_die_fn(const char *err, va_list params) +{ +const char msg[] ="die() should not be called in child\n" ; +xwrite (2 , msg,sizeof (msg) -1 ); +_exit (2 ); +} + +/* this runs in the parent process */ +static void child_err_spew (struct child_process *cmd,struct child_err *cerr) +{ +static void (*old_errfn)(const char *err,va_list params); + + old_errfn =get_error_routine (); +set_error_routine (fake_fatal); + errno = cerr->syserr; + +switch (cerr->err) { +case CHILD_ERR_CHDIR : +error_errno ("exec '%s': cd to '%s' failed" , + cmd->argv[0 ], cmd->dir); +break ; +case CHILD_ERR_ENOENT : +error_errno ("cannot run %s" , cmd->argv[0 ]); +break ; +case CHILD_ERR_SILENT : +break ; +case CHILD_ERR_ERRNO : +error_errno ("cannot exec '%s'" , cmd->argv[0 ]); +break ; + } +set_error_routine (old_errfn); }static void prepare_cmd (struct argv_array *out,const struct child_process *cmd)
@@ -341,13 +409,6 @@ static int wait_or_whine(pid_t pid, const char *argv0, int in_signal) code +=128 ; }else if (WIFEXITED (status)) { code =WEXITSTATUS (status); -/* -* Convert special exit code when execvp failed. -*/ -if (code ==127 ) { - code =-1 ; - failed_errno =ENOENT ; - } }else {error ("waitpid is confused (%s)" , argv0); }
@@ -435,6 +496,7 @@ int start_command(struct child_process *cmd)int null_fd =-1 ;char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ; +struct child_err cerr;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;
@@ -453,20 +515,16 @@ int start_command(struct child_process *cmd) failed_errno = errno;if (!cmd->pid) {/* -* Redirect the channel to write syscall error messages to -* before redirecting the process's stderr so that all die() -* in subsequent call paths use the parent's stderr. +* Ensure the default die/error/warn routines do not get +* called, they can take stdio locks and malloc. */ -if (cmd->no_stderr || need_err) { -int child_err =dup (2 ); -set_cloexec (child_err); -set_error_handle (fdopen (child_err,"w" )); - } +set_die_routine (child_die_fn); +set_error_routine (child_error_fn); +set_warn_routine (child_warn_fn);close (notify_pipe[0 ]);set_cloexec (notify_pipe[1 ]); child_notifier = notify_pipe[1 ]; -atexit (notify_parent);if (cmd->no_stdin)dup2 (null_fd,0 );
@@ -501,8 +559,7 @@ int start_command(struct child_process *cmd) }if (cmd->dir &&chdir (cmd->dir)) -die_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], - cmd->dir); +child_die (CHILD_ERR_CHDIR );/* * Attempt to exec using the command and arguments starting at
@@ -517,12 +574,11 @@ int start_command(struct child_process *cmd) (char *const *) childenv);if (errno ==ENOENT ) { -if (!cmd->silent_exec_failure) -error ("cannot run %s: %s" , cmd->argv[0 ], -strerror (ENOENT )); -exit (127 ); +if (cmd->silent_exec_failure) +child_die (CHILD_ERR_SILENT ); +child_die (CHILD_ERR_ENOENT ); }else { -die_errno ("cannot exec '%s'" , cmd->argv[0 ]); +child_die (CHILD_ERR_ERRNO ); } }if (cmd->pid <0 )
@@ -533,17 +589,18 @@ int start_command(struct child_process *cmd)/* * Wait for child's exec. If the exec succeeds (or if fork() * failed), EOF is seen immediately by the parent. Otherwise, the -* child process sends a single byte. +* child process sends a child_err struct. * Note that use of this infrastructure is completely advisory, * therefore, we keep error checks minimal. */ close (notify_pipe[1 ]); -if (read (notify_pipe[0 ], ¬ify_pipe[1 ],1 ) ==1 ) { +if (xread (notify_pipe[0 ], &cerr,sizeof (cerr)) ==sizeof (cerr)) {/* * At this point we know that fork() succeeded, but exec() * failed. Errors have been reported to our stderr. */ wait_or_whine (cmd->pid, cmd->argv[0 ],0 ); +child_err_spew (cmd, &cerr); failed_errno = errno; cmd->pid =-1 ; }
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v6 9/11 run-command: handle dup2 and close errors in child
In reply to Brandon Williams--- run-command.c | 58 ++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 42 insertions(+), 16 deletions(-)
run-command.c +42 -16
@@ -213,6 +213,8 @@ static int child_notifier = -1;enum child_errcode {CHILD_ERR_CHDIR , +CHILD_ERR_DUP2 , +CHILD_ERR_CLOSE ,CHILD_ERR_ENOENT ,CHILD_ERR_SILENT ,CHILD_ERR_ERRNO
@@ -235,6 +237,24 @@ static void child_die(enum child_errcode err)_exit (1 ); } +static void child_dup2 (int fd,int to) +{ +if (dup2 (fd, to) <0 ) +child_die (CHILD_ERR_DUP2 ); +} + +static void child_close (int fd) +{ +if (close (fd)) +child_die (CHILD_ERR_CLOSE ); +} + +static void child_close_pair (int fd[2 ]) +{ +child_close (fd[0 ]); +child_close (fd[1 ]); +} +/* * parent will make it look like the child spewed a fatal error and died * this is needed to prevent changes to t0061.
@@ -277,6 +297,12 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)error_errno ("exec '%s': cd to '%s' failed" , cmd->argv[0 ], cmd->dir);break ; +case CHILD_ERR_DUP2 : +error_errno ("dup2() in child failed" ); +break ; +case CHILD_ERR_CLOSE : +error_errno ("close() in child failed" ); +break ;case CHILD_ERR_ENOENT :error_errno ("cannot run %s" , cmd->argv[0 ]);break ;
@@ -527,35 +553,35 @@ int start_command(struct child_process *cmd) child_notifier = notify_pipe[1 ];if (cmd->no_stdin) -dup2 (null_fd,0 ); +child_dup2 (null_fd,0 );else if (need_in) { -dup2 (fdin[0 ],0 ); -close_pair (fdin); +child_dup2 (fdin[0 ],0 ); +child_close_pair (fdin); }else if (cmd->in) { -dup2 (cmd->in,0 ); -close (cmd->in); +child_dup2 (cmd->in,0 ); +child_close (cmd->in); }if (cmd->no_stderr) -dup2 (null_fd,2 ); +child_dup2 (null_fd,2 );else if (need_err) { -dup2 (fderr[1 ],2 ); -close_pair (fderr); +child_dup2 (fderr[1 ],2 ); +child_close_pair (fderr); }else if (cmd->err >1 ) { -dup2 (cmd->err,2 ); -close (cmd->err); +child_dup2 (cmd->err,2 ); +child_close (cmd->err); }if (cmd->no_stdout) -dup2 (null_fd,1 ); +child_dup2 (null_fd,1 );else if (cmd->stdout_to_stderr) -dup2 (2 ,1 ); +child_dup2 (2 ,1 );else if (need_out) { -dup2 (fdout[1 ],1 ); -close_pair (fdout); +child_dup2 (fdout[1 ],1 ); +child_close_pair (fdout); }else if (cmd->out >1 ) { -dup2 (cmd->out,1 ); -close (cmd->out); +child_dup2 (cmd->out,1 ); +child_close (cmd->out); }if (cmd->dir &&chdir (cmd->dir))
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v6 10/11 run-command: add note about forking and threading
In reply to Brandon WilliamsAll non-Async-Signal-Safe functions (e.g. malloc and die) were removed between 'fork' and 'exec' in start_command in order to avoid potential deadlocking when forking while multiple threads are running. This deadlocking is possible when a thread (other than the one forking) has acquired a lock and didn't get around to releasing it before the fork. This leaves the lock in a locked state in the resulting process with no hope of it ever being released.
Add a note describing this potential pitfall before the call to 'fork()' so people working in this section of the code know to only use Async-Signal-Safe functions in the child process.
--- run-command.c | 9 +++++++++ 1 file changed, 9 insertions(+)
run-command.c +9 -0
@@ -537,6 +537,15 @@ int start_command(struct child_process *cmd)prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env); +/* +* NOTE: In order to prevent deadlocking when using threads special +* care should be taken with the function calls made in between the +* fork() and exec() calls. No calls should be made to functions which +* require acquiring a lock (e.g. malloc) as the lock could have been +* held by another thread at the time of forking, causing the lock to +* never be released in the child process. This means only +* Async-Signal-Safe functions are permitted in the child. +*/ cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) {
Signature · 1 line
2.12.2.816.g2cccc81164-goog
PATCH v6 11/11 run-command: block signals between fork and execve
In reply to Brandon WilliamsFrom: Eric Wong <[email protected]>
Signal handlers of the parent firing in the forked child may have unintended side effects. Rather than auditing every signal handler we have and will ever have, block signals while forking and restore default signal handlers in the child before execve.
Restoring default signal handlers is required because execve does not unblock signals, it only restores default signal handlers. So we must restore them with sigprocmask before execve, leaving a window when signal handlers we control can fire in the child. Continue ignoring ignored signals, but reset the rest to defaults.
Similarly, disable pthread cancellation to future-proof our code in case we start using cancellation; as cancellation is implemented with signals in glibc.
--- run-command.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+)
run-command.c +68 -0
@@ -215,6 +215,7 @@ enum child_errcode {CHILD_ERR_CHDIR ,CHILD_ERR_DUP2 ,CHILD_ERR_CLOSE , +CHILD_ERR_SIGPROCMASK ,CHILD_ERR_ENOENT ,CHILD_ERR_SILENT ,CHILD_ERR_ERRNO
@@ -303,6 +304,9 @@ static void child_err_spew(struct child_process *cmd, struct child_err *cerr)case CHILD_ERR_CLOSE :error_errno ("close() in child failed" );break ; +case CHILD_ERR_SIGPROCMASK : +error_errno ("sigprocmask failed restoring signals" ); +break ;case CHILD_ERR_ENOENT :error_errno ("cannot run %s" , cmd->argv[0 ]);break ;
@@ -398,7 +402,54 @@ static char **prep_childenv(const char *const *deltaenv)strbuf_release (&key);return childenv; } + +struct atfork_state { +#ifndef NO_PTHREADS +int cs;#endif +sigset_t old; +}; + +#ifndef NO_PTHREADS +static void bug_die (int err,const char *msg) +{ +if (err) { + errno = err; +die_errno ("BUG: %s" , msg); + } +} +#endif + +static void atfork_prepare (struct atfork_state *as) +{ +sigset_t all; + +if (sigfillset (&all)) +die_errno ("sigfillset" ); +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &all, &as->old)) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_sigmask (SIG_SETMASK , &all, &as->old), +"blocking all signals" ); +bug_die (pthread_setcancelstate (PTHREAD_CANCEL_DISABLE , &as->cs), +"disabling cancellation" ); +#endif +} + +static void atfork_parent (struct atfork_state *as) +{ +#ifdef NO_PTHREADS +if (sigprocmask (SIG_SETMASK , &as->old,NULL )) +die_errno ("sigprocmask" ); +#else +bug_die (pthread_setcancelstate (as->cs,NULL ), +"re-enabling cancellation" ); +bug_die (pthread_sigmask (SIG_SETMASK , &as->old,NULL ), +"restoring signal mask" ); +#endif +} +#endif /* GIT_WINDOWS_NATIVE */ static inline void set_cloexec (int fd) {
@@ -523,6 +574,7 @@ int start_command(struct child_process *cmd)char **childenv;struct argv_array argv =ARGV_ARRAY_INIT ;struct child_err cerr; +struct atfork_state as;if (pipe (notify_pipe)) notify_pipe[0 ] = notify_pipe[1 ] =-1 ;
@@ -536,6 +588,7 @@ int start_command(struct child_process *cmd)prepare_cmd (&argv, cmd); childenv =prep_childenv (cmd->env); +atfork_prepare (&as);/* * NOTE: In order to prevent deadlocking when using threads special
@@ -549,6 +602,7 @@ int start_command(struct child_process *cmd) cmd->pid =fork (); failed_errno = errno;if (!cmd->pid) { +int sig;/* * Ensure the default die/error/warn routines do not get * called, they can take stdio locks and malloc.
@@ -597,6 +651,19 @@ int start_command(struct child_process *cmd)child_die (CHILD_ERR_CHDIR );/* +* restore default signal handlers here, in case +* we catch a signal right before execve below +*/ +for (sig =1 ; sig <NSIG ; sig++) { +/* ignored signals get reset to SIG_DFL on execve */ +if (signal (sig,SIG_DFL ) ==SIG_IGN ) +signal (sig,SIG_IGN ); + } + +if (sigprocmask (SIG_SETMASK , &as.old,NULL ) !=0 ) +child_die (CHILD_ERR_SIGPROCMASK ); + +/* * Attempt to exec using the command and arguments starting at * argv.argv[1]. argv.argv[0] contains SHELL_PATH which will * be used in the event exec failed with ENOEXEC at which point
@@ -616,6 +683,7 @@ int start_command(struct child_process *cmd)child_die (CHILD_ERR_ERRNO ); } } +atfork_parent (&as);if (cmd->pid <0 )error_errno ("cannot fork() for %s" , cmd->argv[0 ]);else if (cmd->clean_on_exit)
Signature · 1 line
2.12.2.816.g2cccc81164-goog
Quoted 16 lines · Brandon Williams
Changes in v6:
* fix some windows compat issues
* better comment on thestring_list_removefunction (also marked extern)Brandon Williams (10):
t5550: usewrite_scriptto generate post-update hook
t0061:run_commandexecutes scripts without a #! line
run-command: prepare command before forking
run-command: use the async-signal-safe execv instead of execvp
string-list: addstring_list_removefunction
run-command: prepare child environment before forking
run-command: don't die in child when duping/dev/null
run-command: eliminate calls to error handling functions in child
run-command: handle dup2 and close errors in child
run-command: add note about forking and threading
Quoted 2 lines
Eric Wong (1):
run-command: block signals between fork and execve
Just as an FYI there's a bug with this code where it'll try to execute a directory. I'm adding a test and fixing it. Since this topic is in next I'll base the patch on top of this series.
Signature · 1 line
Brandon Williams
[PATCH v6 12/11] run-command: don't try to execute directories
In some situations run-command will incorrectly try (and fail) to execute a directory instead of an executable. For example:
Lets suppose a user has PATH=~/bin (where 'bin' is a directory) and they happen to have another directory inside 'bin' named 'git-remote-blah'. Then git tries to execute the directory:
$ git ls-remote blah://blah fatal: cannot exec 'git-remote-blah': Permission denied
This is due to only checking 'access()' when locating an executable in PATH, which doesn't distinguish between files and directories. Instead use 'stat()' and check that the path is to a regular file. Now run-command won't try to execute the directory 'git-remote-blah':
$ git ls-remote blah://blah fatal: Unable to find remote helper for 'blah'
--- run-command.c | 3 ++- t/t0061-run-command.sh | 7 +++++++ 2 files changed, 9 insertions(+), 1 deletion(-)
run-command.c +2 -1
@@ -127,6 +127,7 @@ static char *locate_in_PATH(const char *file)while (1 ) {const char *end =strchrnul (p,':' ); +struct stat st;strbuf_reset (&buf);
@@ -137,7 +138,7 @@ static char *locate_in_PATH(const char *file) }strbuf_addstr (&buf, file); -if (!access (buf.buf,F_OK )) +if (!stat (buf.buf, &st) &&S_ISREG (st.st_mode))return strbuf_detach (&buf,NULL );if (!*end)
t/t0061-run-command.sh +7 -0
@@ -37,6 +37,13 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' test_cmp empty err ' +test_expect_success 'run_command should not try to execute a directory' ' + test_when_finished "rm -rf bin/blah" && + mkdir -p bin/blah && + PATH=bin:$PATH test_must_fail test-run-command run-command blah 2>err && + test_i18ngrep "No such file or directory" err +' + test_expect_success POSIXPERM 'run_command reports EACCES' ' cat hello-script >hello.sh && chmod -x hello.sh &&
Signature · 1 line
2.13.0.rc0.306.g87b477812d-goog
Re: [PATCH v6 12/11] run-command: don't try to execute directories
Quoted 15 lines · Brandon Williams
In some situations run-command will incorrectly try (and fail) to execute a directory instead of an executable. For example:
Lets suppose a user has PATH=~/bin (where 'bin' is a directory) and they happen to have another directory inside 'bin' named 'git-remote-blah'. Then git tries to execute the directory:
$ git ls-remote blah://blah fatal: cannot exec 'git-remote-blah': Permission deniedThis is due to only checking '
access()' when locating an executable in PATH, which doesn't distinguish between files and directories. Instead use 'stat()' and check that the path is to a regular file. Now run-command won't try to execute the directory 'git-remote-blah':
Quoted 4 lines
$ git ls-remote blah://blah fatal: Unable to find remote helper for 'blah'Signed-off-by Brandon Williams <[email protected]>
For the interested, the context in which this was reported was trying to execute a directory named 'ssh'. Thanks for a quick fix.
Technically this bug has existed since
commit 38f865c27d1f2560afb48efd2b7b105c1278c4b5 Author: Jeff King <[email protected]> Date: Fri Mar 30 03:52:18 2012 -0400 run-command: treat inaccessible directories as ENOENT
Until we switched from using execvp to execve, the symptom was very
subtle: it only affected the error message when a program could not be found, instead of affecting functionality more substantially.
[...]
Quoted 11 lines
run-command.c +1
-0@@ -127,6 +127,7 @@ static char *locate_in_PATH(const char *file)while (1 ) {const char *end =strchrnul (p,':' ); +struct stat st;strbuf_reset (&buf);
run-command.c static char *locate_in_PATH(const char *file) · line 142
Quoted 6 lines
run-command.c +1
-1@@ -137,7 +138,7 @@ static char *locate_in_PATH(const char *file) }strbuf_addstr (&buf, file); -if (!access (buf.buf,F_OK )) +if (!stat (buf.buf, &st) &&S_ISREG (st.st_mode))return strbuf_detach (&buf,NULL );
Should this share code with help.c's is_executable()?
I suppose not, since that would have trouble finding scripts without the executable bit set.
I was momentarily nervous about what happens if this gets run on Windows. This is just looking for a file's existence, not executability, so it should be fine.
t/t0061-run-command.sh test_expect_success !MINGW 'run_command can run a script without a #! line' ' · line 43
Quoted 10 lines
t/t0061-run-command.sh +4
-0@@ -37,6 +37,13 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' test_cmp empty err ' +test_expect_success 'run_command should not try to execute a directory' ' + test_when_finished "rm -rf bin/blah" && + mkdir -p bin/blah && + PATH=bin:$PATH test_must_fail test-run-command run-command blah 2>err &&
Two nits:
- this environment variable setting leaks past the test_must_fail invocation in some shells. When running external comments, they update the environment after forking, but when running shell functions, they update the environment first and never set it back.
A search with "git grep -e '=.* test_must_fail'" finds no other instances of this pattern, so apparently we've done a good job of being careful about that. *surprised* t/check-non-portable-shell.pl doesn't check for this. Perhaps it should.
Standard workarounds:
(
PATH=... && export PATH && test_must_fail ... )
or
test_must_fail env PATH=... ...
- using a relative path (other than '.') in $PATH feels unusual. We can mimic a typical user setup more closely by using "$PWD/bin".
t/t0061-run-command.sh test_expect_success !MINGW 'run_command can run a script without a #! line' ' · line 44
Quoted 1 line
t/t0061-run-command.sh +1
-0+ test_i18ngrep "No such file or directory" err
This string comes from libc. Is there some other way to test for what this patch does?
E.g. how about something like the following?
Thanks,
Jonathan
t/t0061-run-command.sh +10 -4
@@ -38,10 +38,16 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' ' test_expect_success 'run_command should not try to execute a directory' ' - test_when_finished "rm -rf bin/blah" && - mkdir -p bin/blah && - PATH=bin:$PATH test_must_fail test-run-command run-command blah 2>err && - test_i18ngrep "No such file or directory" err + test_when_finished "rm -rf bin1 bin2" && + mkdir -p bin1/blah && + mkdir bin2 && + cat hello-script >bin2/blah && + chmod +x bin2/blah && + PATH=$PWD/bin1:$PWD/bin2:$PATH \ + test-run-command run-command blah >actual 2>err && + + test_cmp hello-script actual && + test_cmp empty err ' test_expect_success POSIXPERM 'run_command reports EACCES' '
Re: [PATCH v6 12/11] run-command: don't try to execute directories
Quoted 3 lines · Jonathan Nieder
Until we switched from using execvp to execve, the symptom was very
subtle: it only affected the error message when a program could not be found, instead of affecting functionality more substantially.
Hmph, what if you had bin/ssh/ directory and bin2/ssh executable and had bin:bin2 listed in this order in your $PATH? Without this change you'll get an error and that's the end of it. With this change, you'd be able to execute bin2/ssh executable, no? So I am not sure if I agree with the "this is just an error message subtlety".
What does execvp() do when bin/ssh/ directory, bin2/ssh non-executable regular file, and bin3/ssh executable file exist and you have bin:bin2:bin3 on your $PATH? That is what locate_in_PATH() should emulate, I would think.
Quoted 11 lines
Review
Quoted 2 lines
(quoted diff) +1
-0+ if (!stat(buf.buf, &st) && S_ISREG(st.st_mode)) return strbuf_detach(&buf, NULL);Should this share code with help.c's
is_executable()?I suppose not, since that would have trouble finding scripts without the executable bit set.
I was momentarily nervous about what happens if this gets run on Windows. This is just looking for a file's existence, not executability, so it should be fine.
When we are looking for "ssh" with locate_in_PATH(), shouldn't we look for "ssh.exe" on Windows, though?
Re: [PATCH v6 12/11] run-command: don't try to execute directories
Quoted 11 lines · Junio C Hamano
Quoted 3 lines · Jonathan Nieder
Until we switched from using execvp to execve, the symptom was very
subtle: it only affected the error message when a program could not be found, instead of affecting functionality more substantially.Hmph, what if you had bin/ssh/ directory and bin2/ssh executable and had bin:bin2 listed in this order in your $PATH? Without this change you'll get an error and that's the end of it. With this change, you'd be able to execute bin2/ssh executable, no? So I am not sure if I agree with the "this is just an error message subtlety".
I think you misunderstood what I meant. execvp() does not have this bug. In current master, run_command() (within function sane_execvp()) double-checks execvp()'s work when it sees EACCES to decide whether to convert it into a more user-friendly ENOENT. Because of this bug, if you have a bin/ssh/ directory and no bin2/ssh executable, instead of reporting this condition as a user-friendly ENOENT, it would leave it as EACCES.
Quoted 4 lines
What does
execvp()do when bin/ssh/ directory, bin2/ssh non-executable regular file, and bin3/ssh executable file exist and you have bin:bin2:bin3 on your $PATH? That is whatlocate_in_PATH()should emulate, I would think.
Good catch.
$ mkdir -p $HOME/bin1/greet
$ mkdir $HOME/bin2
$ printf '%s\n' 'echo bin2' >$HOME/bin2/greet
$ mkdir $HOME/bin3
$ printf '%s\n' '#!/bin/sh' 'echo bin3' >$HOME/bin3/greet
$ chmod +x $HOME/bin3/greet
$ PATH=$HOME/bin1:$HOME/bin2:$HOME/bin3:$PATH perl -e 'exec("greet")'
bin3It needs to skip over non-executable files.
I think this means we'd want to reuse something like is_executable from help.c.
[...]
Quoted 7 lines
Quoted 7 lines
Review
Quoted 2 lines
(quoted diff) +1
-0+ if (!stat(buf.buf, &st) && S_ISREG(st.st_mode)) return strbuf_detach(&buf, NULL);Should this share code with help.c's
is_executable()?I suppose not, since that would have trouble finding scripts without the executable bit set.
I confused myself about the script special-case: they are supposed to have the executable bit set, too --- the special-casing is just about lacking #!/bin/sh at the top (and hence not being directly executable with execve).
Quoted 6 lines
Quoted 3 lines
I was momentarily nervous about what happens if this gets run on Windows. This is just looking for a file's existence, not executability, so it should be fine.
When we are looking for "ssh" with
locate_in_PATH(), shouldn't we look for "ssh.exe" on Windows, though?
Fortunately this is in a #if !defined(GIT_WINDOWS_NATIVE) block. It's probably worth adding a comment so people know not to rely on it matching Windows path search behavior.
Thanks for looking it over,
Jonathan
Re: [PATCH v6 12/11] run-command: don't try to execute directories
In reply to Jonathan NiederQuoted 21 lines · Jonathan Nieder
Quoted 9 lines
This is due to only checking '
access()' when locating an executable in PATH, which doesn't distinguish between files and directories. Instead use 'stat()' and check that the path is to a regular file. Now run-command won't try to execute the directory 'git-remote-blah':$ git ls-remote blah://blah fatal: Unable to find remote helper for 'blah'Signed-off-by Brandon Williams <[email protected]>For the interested, the context in which this was reported was trying to execute a directory named 'ssh'. Thanks for a quick fix.
Technically this bug has existed since
commit 38f865c27d1f2560afb48efd2b7b105c1278c4b5 Author: Jeff King <[email protected]> Date: Fri Mar 30 03:52:18 2012 -0400 run-command: treat inaccessible directories as ENOENT
Quoted 3 lines
Until we switched from using execvp to execve, the symptom was very
subtle: it only affected the error message when a program could not be found, instead of affecting functionality more substantially.
Yeah, I'm pretty sure I didn't think at all about access() matching directories when doing that commit. Using stat() does seem like the right solution.
-Peff
Re: [PATCH v6 12/11] run-command: don't try to execute directories
In reply to Brandon WilliamsQuoted 7 lines · Brandon Williams
This is due to only checking '
access()' when locating an executable in PATH, which doesn't distinguish between files and directories. Instead use 'stat()' and check that the path is to a regular file. Now run-command won't try to execute the directory 'git-remote-blah':$ git ls-remote blah://blah fatal: Unable to find remote helper for 'blah'
The above is not a very interesting example.
More important is that $PATH may have a directory with git-remote-blah directory (your setup above) and then another directory with the git-remote-blah executable that the user wanted to use. Without this change, we won't get to the real one, and that makes this change truly valuable.
The added test demostrates the "uninteresting" behaviour. Even though it is correct and technically sufficient, it would make it more relevant to do something like this:
mkdir -p bin/blah bin2 && write_script bin2/blah <<-\EOF && echo We found blah in bin2 EOF PATH=bin:$PATH test_must_fail ... what you have ... PATH=bin:bin2:$PATH test-run-command run-command blah >actual && bin2/blah >expect && test_cmp expect actual
as the point of locate_in_PATH() is to successfully find one, without getting confused by an earlier unusable one.
Thanks.
Quoted 45 lines
Signed-off-by Brandon Williams <[email protected]>--- run-command.c | 3 ++- t/t0061-run-command.sh | 7 +++++++ 2 files changed, 9 insertions(+), 1 deletion(-)run-command.c +2
-1@@ -127,6 +127,7 @@ static char *locate_in_PATH(const char *file)while (1 ) {const char *end =strchrnul (p,':' ); +struct stat st;strbuf_reset (&buf);@@ -137,7 +138,7 @@ static char *locate_in_PATH(const char *file) }strbuf_addstr (&buf, file); -if (!access (buf.buf,F_OK )) +if (!stat (buf.buf, &st) &&S_ISREG (st.st_mode))return strbuf_detach (&buf,NULL );if (!*end)t/t0061-run-command.sh +7
-0@@ -37,6 +37,13 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' test_cmp empty err ' +test_expect_success 'run_command should not try to execute a directory' ' + test_when_finished "rm -rf bin/blah" && + mkdir -p bin/blah && + PATH=bin:$PATH test_must_fail test-run-command run-command blah 2>err && + test_i18ngrep "No such file or directory" err +' + test_expect_success POSIXPERM 'run_command reports EACCES' ' cat hello-script >hello.sh && chmod -x hello.sh &&
Re: [PATCH v6 12/11] run-command: don't try to execute directories
In reply to Brandon Williams(cc-ing the reporter)
Quoted 10 lines · Brandon Williams
In some situations run-command will incorrectly try (and fail) to execute a directory instead of an executable. For example:
Lets suppose a user has PATH=~/bin (where 'bin' is a directory) and they happen to have another directory inside 'bin' named 'git-remote-blah'. Then git tries to execute the directory:
$ git ls-remote blah://blah fatal: cannot exec 'git-remote-blah': Permission denied
Quoted 7 lines
This is due to only checking '
access()' when locating an executable in PATH, which doesn't distinguish between files and directories. Instead use 'stat()' and check that the path is to a regular file. Now run-command won't try to execute the directory 'git-remote-blah':$ git ls-remote blah://blah fatal: Unable to find remote helper for 'blah'
This was observed by having a directory called "ssh" in $PATH before the real ssh and trying to use ssh protoccol. Thanks for catching it.
Quoted 45 lines
Signed-off-by Brandon Williams <[email protected]>--- run-command.c | 3 ++- t/t0061-run-command.sh | 7 +++++++ 2 files changed, 9 insertions(+), 1 deletion(-)run-command.c +2
-1@@ -127,6 +127,7 @@ static char *locate_in_PATH(const char *file)while (1 ) {const char *end =strchrnul (p,':' ); +struct stat st;strbuf_reset (&buf);@@ -137,7 +138,7 @@ static char *locate_in_PATH(const char *file) }strbuf_addstr (&buf, file); -if (!access (buf.buf,F_OK )) +if (!stat (buf.buf, &st) &&S_ISREG (st.st_mode))return strbuf_detach (&buf,NULL );if (!*end)t/t0061-run-command.sh +7
-0@@ -37,6 +37,13 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' test_cmp empty err ' +test_expect_success 'run_command should not try to execute a directory' ' + test_when_finished "rm -rf bin/blah" && + mkdir -p bin/blah && + PATH=bin:$PATH test_must_fail test-run-command run-command blah 2>err && + test_i18ngrep "No such file or directory" err +' + test_expect_success POSIXPERM 'run_command reports EACCES' ' cat hello-script >hello.sh && chmod -x hello.sh &&
PATCH v7 1/2 exec_cmd: expose is_executable function
In reply to Brandon WilliamsMove the logic for 'is_executable()' from help.c to exec_cmd.c and expose it so that callers from outside help.c can access the function. This is to enable run-command to be able to query if a file is executable in a future patch.
--- exec_cmd.c | 42 ++++++++++++++++++++++++++++++++++++++++++ exec_cmd.h | 1 + help.c | 42 ------------------------------------------ 3 files changed, 43 insertions(+), 42 deletions(-)
exec_cmd.c +42 -0
@@ -149,3 +149,45 @@ int execl_git_cmd(const char *cmd,...) argv[argc] =NULL ;return execv_git_cmd (argv); } + +int is_executable (const char *name) +{ +struct stat st; + +if (stat (name, &st) ||/* stat, not lstat */ + !S_ISREG (st.st_mode)) +return 0 ; + +#if defined(GIT_WINDOWS_NATIVE ) +/* +* On Windows there is no executable bit. The file extension +* indicates whether it can be run as an executable, and Git +* has special-handling to detect scripts and launch them +* through the indicated script interpreter. We test for the +* file extension first because virus scanners may make +* it quite expensive to open many files. +*/ +if (ends_with (name,".exe" )) +return S_IXUSR ; + +{ +/* +* Now that we know it does not have an executable extension, +* peek into the file instead. +*/ +char buf[3 ] = {0 }; +int n; +int fd =open (name,O_RDONLY ); + st.st_mode &= ~S_IXUSR ; +if (fd >=0 ) { + n =read (fd, buf,2 ); +if (n ==2 ) +/* look for a she-bang */ +if (!strcmp (buf,"#!" )) + st.st_mode |=S_IXUSR ; +close (fd); + } +} +#endif +return st.st_mode &S_IXUSR ; +}
exec_cmd.h +1 -0
@@ -12,5 +12,6 @@ extern int execv_git_cmd(const char **argv); /* NULL terminated */LAST_ARG_MUST_BE_NULL extern int execl_git_cmd(const char *cmd, ...);extern char *system_path (const char *path); +extern int is_executable (const char *name);#endif /* GIT_EXEC_CMD_H */
help.c +0 -42
48 lines not on this page · raw message
Signature · 1 line
2.13.0.rc0.306.g87b477812d-goog
PATCH v7 2/2 run-command: don't try to execute directories
In some situations run-command will incorrectly try (and fail) to execute a directory instead of an executable file. This was observed by having a directory called "ssh" in $PATH before the real ssh and trying to use ssh protoccol, reslting in the following:
$ git ls-remote ssh://url fatal: cannot exec 'ssh': Permission denied
It ends up being worse and run-command will even try to execute a non-executable file if it preceeds the executable version of a file on the PATH. For example, if PATH=~/bin1:~/bin2:~/bin3 and there exists a directory 'git-hello' in 'bin1', a non-executable file 'git-hello' in bin2 and an executable file 'git-hello' (which prints "Hello World!") in bin3 the following will occur:
$ git hello fatal: cannot exec 'git-hello': Permission denied
This is due to only checking 'access()' when locating an executable in PATH, which doesn't distinguish between files and directories. Instead use 'is_executable()' which check that the path is to a regular, executable file. Now run-command won't try to execute the directory or non-executable file 'git-hello':
$ git hello Hello World!
--- run-command.c | 2 +- t/t0061-run-command.sh | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-)
run-command.c +1 -1
@@ -137,7 +137,7 @@ static char *locate_in_PATH(const char *file) }strbuf_addstr (&buf, file); -if (!access (buf.buf,F_OK )) +if (is_executable (buf.buf))return strbuf_detach (&buf,NULL );if (!*end)
t/t0061-run-command.sh +23 -0
@@ -37,6 +37,29 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' test_cmp empty err ' +test_expect_success 'run_command should not try to execute a directory' ' + test_when_finished "rm -rf bin1 bin2 bin3" && + mkdir -p bin1/greet bin2 bin3 && + write_script bin2/greet <<-\EOF && + cat bin2/greet + EOF + chmod -x bin2/greet && + write_script bin3/greet <<-\EOF && + cat bin3/greet + EOF + + # Test that run-command does not try to execute the "greet" directory in + # "bin1", or the non-executable file "greet" in "bin2", but rather + # correcty executes the "greet" script located in bin3. + ( + PATH=$PWD/bin1:$PWD/bin2:$PWD/bin3:$PATH && + export PATH && + test-run-command run-command greet >actual 2>err + ) && + test_cmp bin3/greet actual && + test_cmp empty err +' + test_expect_success POSIXPERM 'run_command reports EACCES' ' cat hello-script >hello.sh && chmod -x hello.sh &&
Signature · 1 line
2.13.0.rc0.306.g87b477812d-goog
Re: [PATCH v7 2/2] run-command: don't try to execute directories
Quoted 1 line · Brandon Williams
Subject: run-command: don't try to execute directories
nit: this is also about non-executable files, now. That would mean something like
run-command: don't try to execute directories or non-executable files
or
run-command: restrict PATH search to files we can execute
[...]
Quoted 12 lines
Reported-by Brian Hatfield <[email protected]>Signed-off-by Brandon Williams <[email protected]>--- run-command.c | 2 +- t/t0061-run-command.sh | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-)run-command.c +0
-0
run-command.c static char *locate_in_PATH(const char *file) · line 141
Quoted 6 lines
run-command.c +1
-1@@ -137,7 +137,7 @@ static char *locate_in_PATH(const char *file) }strbuf_addstr (&buf, file); -if (!access (buf.buf,F_OK )) +if (is_executable (buf.buf))return strbuf_detach (&buf,NULL );
It's probably worth a docstring for this function to explain that this is not a complete emulation of execvp on Windows, since it doesn't look for .com and .exe files.
Quoted 7 lines
run-command.c +0
-0if (!*end)t/t0061-run-command.sh +0
-0
t/t0061-run-command.sh test_expect_success !MINGW 'run_command can run a script without a #! line' ' · line 46
Quoted 10 lines
t/t0061-run-command.sh +7
-0@@ -37,6 +37,29 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' test_cmp empty err ' +test_expect_success 'run_command should not try to execute a directory' ' + test_when_finished "rm -rf bin1 bin2 bin3" && + mkdir -p bin1/greet bin2 bin3 && + write_script bin2/greet <<-\EOF && + cat bin2/greet + EOF + chmod -x bin2/greet &&
This probably implies that the test needs a POSIXPERM dependency. Should it be a separate test_expect_success case so that the other part can still run on Windows?
The rest looks good. Thanks for your patient work.
With whatever subset of the changes described makes sense,
Thanks.
run-command.c +15 -0
@@ -117,6 +117,21 @@ static inline void close_pair(int fd[2])close (fd[1 ]); } +/* +* Search $PATH for a command. This emulates the path search that +* execvp would perform, without actually executing the command so it +* can be used before fork() to prepare to run a command using +* execve() or after execvp() to diagnose why it failed. +* +* The caller should ensure that file contains no directory +* separators. +* +* Returns NULL if the command could not be found. +* +* This should not be used on Windows, where the $PATH search rules +* are more complicated (e.g., a search for "foo" should find +* "foo.exe"). +*/ static char *locate_in_PATH (const char *file) {const char *p =getenv ("PATH" );
t/t0061-run-command.sh +21 -14
@@ -37,26 +37,33 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' test_cmp empty err ' -test_expect_success 'run_command should not try to execute a directory' ' +test_expect_success 'run_command does not try to execute a directory' ' test_when_finished "rm -rf bin1 bin2" && - mkdir -p bin1/greet bin2 bin3 && + mkdir -p bin1/greet bin2 && write_script bin2/greet <<-\EOF && cat bin2/greet EOF - chmod -x bin2/greet && - write_script bin3/greet <<-\EOF && - cat bin3/greet + + PATH=$PWD/bin1:$PWD/bin2:$PATH \ + test-run-command run-command greet >actual 2>err && + test_cmp bin2/greet actual && + test_cmp empty err +' + +test_expect_success POSIXPERM 'run_command passes over non-executable file' ' + test_when_finished "rm -rf bin1 bin2" && + mkdir -p bin1 bin2 && + write_script bin1/greet <<-\EOF && + cat bin1/greet + EOF + chmod -x bin1/greet && + write_script bin2/greet <<-\EOF && + cat bin2/greet EOF - # Test that run-command does not try to execute the "greet" directory in - # "bin1", or the non-executable file "greet" in "bin2", but rather - # correcty executes the "greet" script located in bin3. - ( - PATH=$PWD/bin1:$PWD/bin2:$PWD/bin3:$PATH && - export PATH && - test-run-command run-command greet >actual 2>err - ) && - test_cmp bin3/greet actual && + PATH=$PWD/bin1:$PWD/bin2:$PATH \ + test-run-command run-command greet >actual 2>err && + test_cmp bin2/greet actual && test_cmp empty err '
Re: [PATCH v7 2/2] run-command: don't try to execute directories
Quoted 135 lines · Jonathan Nieder
Quoted 1 line · Brandon Williams
Subject: run-command: don't try to execute directories
nit: this is also about non-executable files, now. That would mean something like
run-command: don't try to execute directories or non-executable files
or
run-command: restrict PATH search to files we can execute
[...]
Quoted 12 lines
Reported-by Brian Hatfield <[email protected]>Signed-off-by Brandon Williams <[email protected]>--- run-command.c | 2 +- t/t0061-run-command.sh | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-)run-command.c +0
-0run-command.c static char *locate_in_PATH(const char *file) · line 141
Quoted 6 lines
run-command.c +1
-1@@ -137,7 +137,7 @@ static char *locate_in_PATH(const char *file) }strbuf_addstr (&buf, file); -if (!access (buf.buf,F_OK )) +if (is_executable (buf.buf))return strbuf_detach (&buf,NULL );It's probably worth a docstring for this function to explain that this is not a complete emulation of execvp on Windows, since it doesn't look for .com and .exe files.
Quoted 7 lines
run-command.c +0
-0if (!*end)t/t0061-run-command.sh +0
-0t/t0061-run-command.sh test_expect_success !MINGW 'run_command can run a script without a #! line' ' · line 46
Quoted 10 lines
t/t0061-run-command.sh +7
-0@@ -37,6 +37,29 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' test_cmp empty err ' +test_expect_success 'run_command should not try to execute a directory' ' + test_when_finished "rm -rf bin1 bin2 bin3" && + mkdir -p bin1/greet bin2 bin3 && + write_script bin2/greet <<-\EOF && + cat bin2/greet + EOF + chmod -x bin2/greet &&This probably implies that the test needs a POSIXPERM dependency. Should it be a separate
test_expect_successcase so that the other part can still run on Windows?The rest looks good. Thanks for your patient work.
With whatever subset of the changes described makes sense,
Reviewed-by Jonathan Nieder <[email protected]>Thanks.
run-command.c +15
-0@@ -117,6 +117,21 @@ static inline void close_pair(int fd[2])close (fd[1 ]); } +/* +* Search $PATH for a command. This emulates the path search that +* execvp would perform, without actually executing the command so it +* can be used before fork() to prepare to run a command using +* execve() or after execvp() to diagnose why it failed. +* +* The caller should ensure that file contains no directory +* separators. +* +* Returns NULL if the command could not be found. +* +* This should not be used on Windows, where the $PATH search rules +* are more complicated (e.g., a search for "foo" should find +* "foo.exe"). +*/ static char *locate_in_PATH (const char *file) {const char *p =getenv ("PATH" );t/t0061-run-command.sh +18
-10@@ -37,26 +37,33 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' test_cmp empty err ' -test_expect_success 'run_command should not try to execute a directory' ' +test_expect_success 'run_command does not try to execute a directory' ' test_when_finished "rm -rf bin1 bin2" && - mkdir -p bin1/greet bin2 bin3 && + mkdir -p bin1/greet bin2 && write_script bin2/greet <<-\EOF && cat bin2/greet EOF - chmod -x bin2/greet && - write_script bin3/greet <<-\EOF && - cat bin3/greet + + PATH=$PWD/bin1:$PWD/bin2:$PATH \ + test-run-command run-command greet >actual 2>err && + test_cmp bin2/greet actual && + test_cmp empty err +' + +test_expect_success POSIXPERM 'run_command passes over non-executable file' ' + test_when_finished "rm -rf bin1 bin2" && + mkdir -p bin1 bin2 && + write_script bin1/greet <<-\EOF && + cat bin1/greet + EOF + chmod -x bin1/greet && + write_script bin2/greet <<-\EOF && + cat bin2/greet EOF - # Test that run-command does not try to execute the "greet" directory in - # "bin1", or the non-executable file "greet" in "bin2", but rather - # correcty executes the "greet" script located in bin3. - ( - PATH=$PWD/bin1:$PWD/bin2:$PWD/bin3:$PATH &&
t/t0061-run-command.sh test_expect_success !MINGW 'run_command can run a script without a #! line' ' · line 69
Quoted 10 lines
t/t0061-run-command.sh +3
-4- export PATH && - test-run-command run-command greet >actual 2>err - ) && - test_cmp bin3/greet actual && + PATH=$PWD/bin1:$PWD/bin2:$PATH \ + test-run-command run-command greet >actual 2>err && + test_cmp bin2/greet actual && test_cmp empty err '
Yeah the POSIXPERM is going to be necessary. I'll roll in some of these changes when I do a reroll.
Signature · 1 line
Brandon Williams
Re: [PATCH v7 1/2] exec_cmd: expose is_executable function
In reply to Brandon WilliamsHi,
Quoted 11 lines · Brandon Williams
Move the logic for '
is_executable()' fromhelp.ctoexec_cmd.cand expose it so that callers from outsidehelp.ccan access the function. This is to enable run-command to be able to query if a file is executable in a future patch.Signed-off-by Brandon Williams <[email protected]>--- exec_cmd.c | 42 ++++++++++++++++++++++++++++++++++++++++++ exec_cmd.h | 1 + help.c | 42 ------------------------------------------ 3 files changed, 43 insertions(+), 42 deletions(-)
Makes sense. Seems like as fine place for it. (exec_cmd.c is mostly an implementation detail of run_command, to hold the logic for executing a git command. This is another run_command helper, and it doesn't feel illogical to find it there. Another alternative place to put t would be in run-command.c.)
exec_cmd.c int execl_git_cmd(const char *cmd,...) · line 153
Quoted 10 lines
exec_cmd.c +2
-0@@ -149,3 +149,45 @@ int execl_git_cmd(const char *cmd,...) argv[argc] =NULL ;return execv_git_cmd (argv); } + +int is_executable (const char *name)
nit: it's a good practice to find a logical place for a new function in the existing file, instead of defaulting to the end. That way, the file is easier to read sequentially, and if two different patches want to add a function to the same file then they are less likely to conflict.
This could go before prepare_git_cmd, for example.
With or without such a tweak,
exec_cmd.c +42 -42
@@ -104,6 +104,48 @@ void setup_path(void)strbuf_release (&new_path); } +int is_executable (const char *name) +{ +struct stat st; + +if (stat (name, &st) ||/* stat, not lstat */ + !S_ISREG (st.st_mode)) +return 0 ; + +#if defined(GIT_WINDOWS_NATIVE ) +/* +* On Windows there is no executable bit. The file extension +* indicates whether it can be run as an executable, and Git +* has special-handling to detect scripts and launch them +* through the indicated script interpreter. We test for the +* file extension first because virus scanners may make +* it quite expensive to open many files. +*/ +if (ends_with (name,".exe" )) +return S_IXUSR ; + +{ +/* +* Now that we know it does not have an executable extension, +* peek into the file instead. +*/ +char buf[3 ] = {0 }; +int n; +int fd =open (name,O_RDONLY ); + st.st_mode &= ~S_IXUSR ; +if (fd >=0 ) { + n =read (fd, buf,2 ); +if (n ==2 ) +/* look for a she-bang */ +if (!strcmp (buf,"#!" )) + st.st_mode |=S_IXUSR ; +close (fd); + } +} +#endif +return st.st_mode &S_IXUSR ; +} +const char **prepare_git_cmd (struct argv_array *out,const char **argv) {argv_array_push (out,"git" );
@@ -149,45 +191,3 @@ int execl_git_cmd(const char *cmd,...) argv[argc] =NULL ;return execv_git_cmd (argv); } - -int is_executable (const char *name) -{ -struct stat st; - -if (stat (name, &st) ||/* stat, not lstat */ - !S_ISREG (st.st_mode)) -return 0 ; - -#if defined(GIT_WINDOWS_NATIVE ) -/* -* On Windows there is no executable bit. The file extension -* indicates whether it can be run as an executable, and Git -* has special-handling to detect scripts and launch them -* through the indicated script interpreter. We test for the -* file extension first because virus scanners may make -* it quite expensive to open many files. -*/ -if (ends_with (name,".exe" )) -return S_IXUSR ; - -{ -/* -* Now that we know it does not have an executable extension, -* peek into the file instead. -*/ -char buf[3 ] = {0 }; -int n; -int fd =open (name,O_RDONLY ); - st.st_mode &= ~S_IXUSR ; -if (fd >=0 ) { - n =read (fd, buf,2 ); -if (n ==2 ) -/* look for a she-bang */ -if (!strcmp (buf,"#!" )) - st.st_mode |=S_IXUSR ; -close (fd); - } -} -#endif -return st.st_mode &S_IXUSR ; -}
exec_cmd.h +1 -1
@@ -7,11 +7,11 @@ extern void git_set_argv_exec_path(const char *exec_path);extern void git_extract_argv0_path (const char *path);extern const char *git_exec_path (void );extern void setup_path (void ); +extern int is_executable (const char *name);extern const char **prepare_git_cmd (struct argv_array *out,const char **argv);extern int execv_git_cmd (const char **argv);/* NULL terminated */ LAST_ARG_MUST_BE_NULL extern int execl_git_cmd(const char *cmd, ...);extern char *system_path (const char *path); -extern int is_executable (const char *name);#endif /* GIT_EXEC_CMD_H */
Re: [PATCH v7 1/2] exec_cmd: expose is_executable function
In reply to Brandon WilliamsAm 25.04.2017 um 19:54 schrieb Brandon Williams:
Quoted 2 lines
Move the logic for '
is_executable()' fromhelp.ctoexec_cmd.cand expose it so that callers from outsidehelp.ccan access the function.
The function is quite low-level. IMO, run-command.[ch] would be a better home for it. Additionally, that would reduce the number of files that contain #ifdef GIT_WINDOWS_NATIVE.
-- Hannes
Re: [PATCH v7 1/2] exec_cmd: expose is_executable function
Quoted 7 lines · Johannes Sixt
Am 25.04.2017 um 19:54 schrieb Brandon Williams:
Quoted 2 lines
Move the logic for '
is_executable()' fromhelp.ctoexec_cmd.cand expose it so that callers from outsidehelp.ccan access the function.The function is quite low-level. IMO, run-command.[ch] would be a better home for it. Additionally, that would reduce the number of files that contain #ifdef
GIT_WINDOWS_NATIVE.
Fair enough, Jonathan suggested the same so I'll move it to run-command in a re-roll.
Signature · 1 line
Brandon Williams
PATCH v8 1/2 run-command: expose is_executable function
In reply to Brandon WilliamsMove the logic for 'is_executable()' from help.c to run_command.c and expose it so that callers from outside help.c can access the function. This is to enable run-command to be able to query if a file is executable in a future patch.
--- help.c | 43 +------------------------------------------ run-command.c | 42 ++++++++++++++++++++++++++++++++++++++++++ run-command.h | 1 + 3 files changed, 44 insertions(+), 42 deletions(-)
help.c +1 -42
@@ -1,6 +1,7 @@#include "cache.h" #include "builtin.h" #include "exec_cmd.h" +#include "run-command.h" #include "levenshtein.h" #include "help.h" #include "common-cmds.h"
@@ -96,48 +97,6 @@ static void pretty_print_cmdnames(struct cmdnames *cmds, unsigned int colopts)string_list_clear (&list,0 ); } -static int is_executable (const char *name) -{ -struct stat st; - -if (stat (name, &st) ||/* stat, not lstat */ - !S_ISREG (st.st_mode)) -return 0 ; - -#if defined(GIT_WINDOWS_NATIVE ) -/* -* On Windows there is no executable bit. The file extension -* indicates whether it can be run as an executable, and Git -* has special-handling to detect scripts and launch them -* through the indicated script interpreter. We test for the -* file extension first because virus scanners may make -* it quite expensive to open many files. -*/ -if (ends_with (name,".exe" )) -return S_IXUSR ; - -{ -/* -* Now that we know it does not have an executable extension, -* peek into the file instead. -*/ -char buf[3 ] = {0 }; -int n; -int fd =open (name,O_RDONLY ); - st.st_mode &= ~S_IXUSR ; -if (fd >=0 ) { - n =read (fd, buf,2 ); -if (n ==2 ) -/* look for a she-bang */ -if (!strcmp (buf,"#!" )) - st.st_mode |=S_IXUSR ; -close (fd); - } -} -#endif -return st.st_mode &S_IXUSR ; -} -static void list_commands_in_dir (struct cmdnames *cmds,const char *path,const char *prefix)
run-command.c +42 -0
48 lines not on this page · raw message
run-command.h +1 -0
@@ -51,6 +51,7 @@ struct child_process {#define CHILD_PROCESS_INIT { NULL, ARGV_ARRAY_INIT, ARGV_ARRAY_INIT }void child_process_init (struct child_process *);void child_process_clear (struct child_process *); +extern int is_executable (const char *name);int start_command (struct child_process *);int finish_command (struct child_process *);
Signature · 1 line
2.13.0.rc0.306.g87b477812d-goog
PATCH v8 2/2 run-command: restrict PATH search to executable files
In some situations run-command will incorrectly try (and fail) to execute a directory instead of an executable file. This was observed by having a directory called "ssh" in $PATH before the real ssh and trying to use ssh protoccol, reslting in the following:
$ git ls-remote ssh://url fatal: cannot exec 'ssh': Permission denied
It ends up being worse and run-command will even try to execute a non-executable file if it preceeds the executable version of a file on the PATH. For example, if PATH=~/bin1:~/bin2:~/bin3 and there exists a directory 'git-hello' in 'bin1', a non-executable file 'git-hello' in bin2 and an executable file 'git-hello' (which prints "Hello World!") in bin3 the following will occur:
$ git hello fatal: cannot exec 'git-hello': Permission denied
This is due to only checking 'access()' when locating an executable in PATH, which doesn't distinguish between files and directories. Instead use 'is_executable()' which check that the path is to a regular, executable file. Now run-command won't try to execute the directory or non-executable file 'git-hello':
$ git hello Hello World!
--- run-command.c | 19 ++++++++++++++++++- t/t0061-run-command.sh | 30 ++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 1 deletion(-)
run-command.c +18 -1
@@ -159,6 +159,23 @@ int is_executable(const char *name)return st.st_mode &S_IXUSR ; } +/* +* Search $PATH for a command. This emulates the path search that +* execvp would perform, without actually executing the command so it +* can be used before fork() to prepare to run a command using +* execve() or after execvp() to diagnose why it failed. +* +* The caller should ensure that file contains no directory +* separators. +* +* Returns the path to the command, as found in $PATH or NULL if the +* command could not be found. The caller inherits ownership of the memory +* used to store the resultant path. +* +* This should not be used on Windows, where the $PATH search rules +* are more complicated (e.g., a search for "foo" should find +* "foo.exe"). +*/ static char *locate_in_PATH (const char *file) {const char *p =getenv ("PATH" );
@@ -179,7 +196,7 @@ static char *locate_in_PATH(const char *file) }strbuf_addstr (&buf, file); -if (!access (buf.buf,F_OK )) +if (is_executable (buf.buf))return strbuf_detach (&buf,NULL );if (!*end)
t/t0061-run-command.sh +30 -0
@@ -37,6 +37,36 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' test_cmp empty err ' +test_expect_success 'run_command does not try to execute a directory' ' + test_when_finished "rm -rf bin1 bin2" && + mkdir -p bin1/greet bin2 && + write_script bin2/greet <<-\EOF && + cat bin2/greet + EOF + + PATH=$PWD/bin1:$PWD/bin2:$PATH \ + test-run-command run-command greet >actual 2>err && + test_cmp bin2/greet actual && + test_cmp empty err +' + +test_expect_success POSIXPERM 'run_command passes over non-executable file' ' + test_when_finished "rm -rf bin1 bin2" && + mkdir -p bin1 bin2 && + write_script bin1/greet <<-\EOF && + cat bin1/greet + EOF + chmod -x bin1/greet && + write_script bin2/greet <<-\EOF && + cat bin2/greet + EOF + + PATH=$PWD/bin1:$PWD/bin2:$PATH \ + test-run-command run-command greet >actual 2>err && + test_cmp bin2/greet actual && + test_cmp empty err +' + test_expect_success POSIXPERM 'run_command reports EACCES' ' cat hello-script >hello.sh && chmod -x hello.sh &&
Signature · 1 line
2.13.0.rc0.306.g87b477812d-goog
Re: [PATCH v8 2/2] run-command: restrict PATH search to executable files
Quoted 27 lines · Brandon Williams
In some situations run-command will incorrectly try (and fail) to execute a directory instead of an executable file. This was observed by having a directory called "ssh" in $PATH before the real ssh and trying to use ssh protoccol, reslting in the following:
$ git ls-remote ssh://url fatal: cannot exec 'ssh': Permission deniedIt ends up being worse and run-command will even try to execute a non-executable file if it preceeds the executable version of a file on the PATH. For example, if PATH=~/bin1:~/bin2:~/bin3 and there exists a directory 'git-hello' in 'bin1', a non-executable file 'git-hello' in bin2 and an executable file 'git-hello' (which prints "Hello World!") in bin3 the following will occur:
$ git hello fatal: cannot exec 'git-hello': Permission deniedThis is due to only checking '
access()' when locating an executable in PATH, which doesn't distinguish between files and directories. Instead use 'is_executable()' which check that the path is to a regular, executable file. Now run-command won't try to execute the directory or non-executable file 'git-hello':$ git hello Hello World!
Quoted 6 lines
Reported-by Brian Hatfield <[email protected]>Signed-off-by Brandon Williams <[email protected]>--- run-command.c | 19 ++++++++++++++++++- t/t0061-run-command.sh | 30 ++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 1 deletion(-)
Thanks. Patch left unsnipped for reference.
Quoted 78 lines
run-command.c +18
-1@@ -159,6 +159,23 @@ int is_executable(const char *name)return st.st_mode &S_IXUSR ; } +/* +* Search $PATH for a command. This emulates the path search that +* execvp would perform, without actually executing the command so it +* can be used before fork() to prepare to run a command using +* execve() or after execvp() to diagnose why it failed. +* +* The caller should ensure that file contains no directory +* separators. +* +* Returns the path to the command, as found in $PATH or NULL if the +* command could not be found. The caller inherits ownership of the memory +* used to store the resultant path. +* +* This should not be used on Windows, where the $PATH search rules +* are more complicated (e.g., a search for "foo" should find +* "foo.exe"). +*/ static char *locate_in_PATH (const char *file) {const char *p =getenv ("PATH" );@@ -179,7 +196,7 @@ static char *locate_in_PATH(const char *file) }strbuf_addstr (&buf, file); -if (!access (buf.buf,F_OK )) +if (is_executable (buf.buf))return strbuf_detach (&buf,NULL );if (!*end)t/t0061-run-command.sh +30
-0@@ -37,6 +37,36 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' test_cmp empty err ' +test_expect_success 'run_command does not try to execute a directory' ' + test_when_finished "rm -rf bin1 bin2" && + mkdir -p bin1/greet bin2 && + write_script bin2/greet <<-\EOF && + cat bin2/greet + EOF + + PATH=$PWD/bin1:$PWD/bin2:$PATH \ + test-run-command run-command greet >actual 2>err && + test_cmp bin2/greet actual && + test_cmp empty err +' + +test_expect_success POSIXPERM 'run_command passes over non-executable file' ' + test_when_finished "rm -rf bin1 bin2" && + mkdir -p bin1 bin2 && + write_script bin1/greet <<-\EOF && + cat bin1/greet + EOF + chmod -x bin1/greet && + write_script bin2/greet <<-\EOF && + cat bin2/greet + EOF + + PATH=$PWD/bin1:$PWD/bin2:$PATH \ + test-run-command run-command greet >actual 2>err && + test_cmp bin2/greet actual && + test_cmp empty err +' + test_expect_success POSIXPERM 'run_command reports EACCES' ' cat hello-script >hello.sh && chmod -x hello.sh &&
Re: [PATCH v8 2/2] run-command: restrict PATH search to executable files
In reply to Brandon WilliamsQuoted 18 lines · Brandon Williams
In some situations run-command will incorrectly try (and fail) to execute a directory instead of an executable file. This was observed by having a directory called "ssh" in $PATH before the real ssh and trying to use ssh protoccol, reslting in the following:
$ git ls-remote ssh://url fatal: cannot exec 'ssh': Permission deniedIt ends up being worse and run-command will even try to execute a non-executable file if it preceeds the executable version of a file on the PATH. For example, if PATH=~/bin1:~/bin2:~/bin3 and there exists a directory 'git-hello' in 'bin1', a non-executable file 'git-hello' in bin2 and an executable file 'git-hello' (which prints "Hello World!") in bin3 the following will occur:
$ git hello fatal: cannot exec 'git-hello': Permission denied
Quoted 8 lines
This is due to only checking '
access()' when locating an executable in PATH, which doesn't distinguish between files and directories. Instead use 'is_executable()' which check that the path is to a regular, executable file. Now run-command won't try to execute the directory or non-executable file 'git-hello':$ git hello Hello World!
Could you add a line after this example, that says something like "which matches what execvp() would have done with a request to execute git-hello with such a $PATH."
That is because it can be argued that bin1/git-hello should be found and get complaint "not an executable file", or that bin1/git-hello should be skipped but bin2/git-hello should be found and get complaint "not an executable file", both to help the user diagnose and fix the broken $PATH (or director contents). It is the easiest to justify why we chose this other definition to skip both git-hello in bin1 and bin2 if that is an established existing practice---we can say "sure, what you propose also may make sense, but we match what execvp(3) does".
The patch text looks good.
Thanks.
PATCH v9 2/2 run-command: restrict PATH search to executable files
In some situations run-command will incorrectly try (and fail) to execute a directory instead of an executable file. This was observed by having a directory called "ssh" in $PATH before the real ssh and trying to use ssh protoccol, reslting in the following:
$ git ls-remote ssh://url fatal: cannot exec 'ssh': Permission denied
It ends up being worse and run-command will even try to execute a non-executable file if it preceeds the executable version of a file on the PATH. For example, if PATH=~/bin1:~/bin2:~/bin3 and there exists a directory 'git-hello' in 'bin1', a non-executable file 'git-hello' in bin2 and an executable file 'git-hello' (which prints "Hello World!") in bin3 the following will occur:
$ git hello fatal: cannot exec 'git-hello': Permission denied
This is due to only checking 'access()' when locating an executable in PATH, which doesn't distinguish between files and directories. Instead use 'is_executable()' which check that the path is to a regular, executable file. Now run-command won't try to execute the directory or non-executable file 'git-hello':
$ git hello Hello World!
This matches what 'execvp()' would have done with a request to execute 'git-hello' with such a $PATH.
---
This [2/2] patch has the exact same diff as v8, the only difference is to the commit message per Junio's request to add an explanation for why this particular behavior is desirable (because it matches what execvp() does).
I didn't resend out [1/2] of this fixup because it is identical to the v8 version.
run-command.c | 19 ++++++++++++++++++- t/t0061-run-command.sh | 30 ++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 1 deletion(-)
run-command.c +18 -1
@@ -159,6 +159,23 @@ int is_executable(const char *name)return st.st_mode &S_IXUSR ; } +/* +* Search $PATH for a command. This emulates the path search that +* execvp would perform, without actually executing the command so it +* can be used before fork() to prepare to run a command using +* execve() or after execvp() to diagnose why it failed. +* +* The caller should ensure that file contains no directory +* separators. +* +* Returns the path to the command, as found in $PATH or NULL if the +* command could not be found. The caller inherits ownership of the memory +* used to store the resultant path. +* +* This should not be used on Windows, where the $PATH search rules +* are more complicated (e.g., a search for "foo" should find +* "foo.exe"). +*/ static char *locate_in_PATH (const char *file) {const char *p =getenv ("PATH" );
@@ -179,7 +196,7 @@ static char *locate_in_PATH(const char *file) }strbuf_addstr (&buf, file); -if (!access (buf.buf,F_OK )) +if (is_executable (buf.buf))return strbuf_detach (&buf,NULL );if (!*end)
t/t0061-run-command.sh +30 -0
@@ -37,6 +37,36 @@ test_expect_success !MINGW 'run_command can run a script without a #! line' ' test_cmp empty err ' +test_expect_success 'run_command does not try to execute a directory' ' + test_when_finished "rm -rf bin1 bin2" && + mkdir -p bin1/greet bin2 && + write_script bin2/greet <<-\EOF && + cat bin2/greet + EOF + + PATH=$PWD/bin1:$PWD/bin2:$PATH \ + test-run-command run-command greet >actual 2>err && + test_cmp bin2/greet actual && + test_cmp empty err +' + +test_expect_success POSIXPERM 'run_command passes over non-executable file' ' + test_when_finished "rm -rf bin1 bin2" && + mkdir -p bin1 bin2 && + write_script bin1/greet <<-\EOF && + cat bin1/greet + EOF + chmod -x bin1/greet && + write_script bin2/greet <<-\EOF && + cat bin2/greet + EOF + + PATH=$PWD/bin1:$PWD/bin2:$PATH \ + test-run-command run-command greet >actual 2>err && + test_cmp bin2/greet actual && + test_cmp empty err +' + test_expect_success POSIXPERM 'run_command reports EACCES' ' cat hello-script >hello.sh && chmod -x hello.sh &&
Signature · 1 line
2.13.0.rc0.306.g87b477812d-goog
Re: [PATCH v9 2/2] run-command: restrict PATH search to executable files
Quoted 6 lines · Brandon Williams
This [2/2] patch has the exact same diff as v8, the only difference is to the commit message per Junio's request to add an explanation for why this particular behavior is desirable (because it matches what
execvp()does).I didn't resend out [1/2] of this fixup because it is identical to the v8 version.
Thanks. It is already explained well in the in-code comment, but it is good to help "git log" readers with this information.
Will queue.
Re: [PATCH v8 1/2] run-command: expose is_executable function
In reply to Brandon WilliamsQuoted 11 lines · Brandon Williams
Move the logic for '
is_executable()' fromhelp.ctorun_command.cand expose it so that callers from outsidehelp.ccan access the function. This is to enable run-command to be able to query if a file is executable in a future patch.Signed-off-by Brandon Williams <[email protected]>--- help.c | 43 +------------------------------------------ run-command.c | 42 ++++++++++++++++++++++++++++++++++++++++++ run-command.h | 1 + 3 files changed, 44 insertions(+), 42 deletions(-)