Safe-guard the Windows code a bit more against getenv() problems
cover Safe-guard the Windows code a bit more against getenv() problems
We saw a couple of getenv() cleanups, where we now immediately duplicate the return value of getenv() in order to avoid problems. However, I am really uncomfortable with the current value (30) of GETENV_MAX_RETAIN in compat/mingw.c: it strikes me as low, given that the average number of getenv() call per Git process in Git's test suite is already 40.
I'd really like to increase it, even if it won't help problems where getenv() is called in a loop whose number of iteration depends on user input (like the problem fixed in 6776a84dae (diff: ensure correct lifetime of external_diff_cmd, 2019-01-11)). It will still fend off plenty of other cases, I believe.
And yes, it's in my TODOs to look back over those getenv() issues after v2.21.0 (see https://github.com/git-for-windows/git/pull/2019 for my progress).
Johannes Schindelin (1):
mingw: safe-guard a bit more against getenv() problems
compat/mingw.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
base-commit: 2d08f3deb9feb73dc8d21d75bfd367839fc1322c
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-127%2Fdscho%2Fmingw-retain-more-getenv-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-127/dscho/mingw-retain-more-getenv-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/127
Signature · 1 line
gitgitgadget
PATCH 1/1 mingw: safe-guard a bit more against getenv() problems
From: Johannes Schindelin <[email protected]>
Running up to v2.21.0, we fixed two bugs that were made prominent by the Windows-specific change to retain copies of only the 30 latest getenv() calls' returned strings, invalidating any copies of previous getenv() calls' return values.
While this really shines a light onto bugs of the form where we hold onto getenv()'s return values without copying them, it is also a real problem for users.
And even if Jeff King's patches merged via 773e408881 (Merge branch 'jk/save-getenv-result', 2019-01-29) provide further work on that front, we are far from done. Just one example: on Windows, we unset environment variables when spawning new processes, which potentially invalidates strings that were previously obtained via getenv(), and therefore we have to duplicate environment values that are somehow involved in spawning new processes (e.g. GIT_MAN_VIEWER in show_man_page()).
We do not have a chance to investigate, let address, all of those issues in time for v2.21.0, so let's at least help Windows users by increasing the number of getenv() calls' return values that are kept valid. The number 64 was determined by looking at the average number of getenv() calls per process in the entire test suite run on Windows (which is around 40) and then adding a bit for good measure. And it is a power of two (which would have hit yesterday's theme perfectly).
--- compat/mingw.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
compat/mingw.c +1 -1
@@ -1632,7 +1632,7 @@ int mingw_kill(pid_t pid, int sig)*/ char *mingw_getenv (const char *name) { -#define GETENV_MAX_RETAIN 30 +#define GETENV_MAX_RETAIN 64static char *values[GETENV_MAX_RETAIN ];static int value_counter;int len_key, len_value;
Signature · 1 line
gitgitgadget
Re: [PATCH 1/1] mingw: safe-guard a bit more against getenv() problems
Quoted 38 lines · Johannes Schindelin via GitGitGadget
From: Johannes Schindelin <[email protected]>
Running up to v2.21.0, we fixed two bugs that were made prominent by the Windows-specific change to retain copies of only the 30 latest
getenv()calls' returned strings, invalidating any copies of previousgetenv()calls' return values.While this really shines a light onto bugs of the form where we hold onto getenv()'s return values without copying them, it is also a real problem for users.
And even if Jeff King's patches merged via
773e408881(Merge branch 'jk/save-getenv-result', 2019-01-29) provide further work on that front, we are far from done. Just one example: on Windows, we unset environment variables when spawning new processes, which potentially invalidates strings that were previously obtained viagetenv(), and therefore we have to duplicate environment values that are somehow involved in spawning new processes (e.g.GIT_MAN_VIEWERinshow_man_page()).We do not have a chance to investigate, let address, all of those issues in time for v2.21.0, so let's at least help Windows users by increasing the number of
getenv()calls' return values that are kept valid. The number 64 was determined by looking at the average number ofgetenv()calls per process in the entire test suite run on Windows (which is around 40) and then adding a bit for good measure. And it is a power of two (which would have hit yesterday's theme perfectly).Signed-off-by Johannes Schindelin <[email protected]>--- compat/mingw.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)compat/mingw.c +0
-0
compat/mingw.c int mingw_kill(pid_t pid, int sig) · line 1638
Quoted 8 lines
compat/mingw.c +1
-1@@ -1632,7 +1632,7 @@ int mingw_kill(pid_t pid, int sig)*/ char *mingw_getenv (const char *name) { -#define GETENV_MAX_RETAIN 30 +#define GETENV_MAX_RETAIN 64static char *values[GETENV_MAX_RETAIN ];static int value_counter;int len_key, len_value;
Why not use a mem_pool for this? We have that code isolated and re-usable now. Have mingw_getenv() copy the string into the pool always return the pointer from within the pool. The pool automatically handles allocating new blocks as necessary. And (if we care) we can bulk free the pool before existing.
Jeff
Re: [PATCH 1/1] mingw: safe-guard a bit more against getenv() problems
Quoted 12 lines · Jeff Hostetler
Quoted 12 lines
compat/mingw.c +1
-1@@ -1632,7 +1632,7 @@ int mingw_kill(pid_t pid, int sig)*/ char *mingw_getenv (const char *name) { -#define GETENV_MAX_RETAIN 30 +#define GETENV_MAX_RETAIN 64static char *values[GETENV_MAX_RETAIN ];static int value_counter;
Quoted 8 lines
compat/mingw.c int mingw_kill(pid_t pid, int sig) · line 1638
Quoted 2 lines
compat/mingw.c +0
-0int len_key, len_value;Why not use a
mem_poolfor this? We have that code isolated and re-usable now. Havemingw_getenv()copy the string into the pool always return the pointer from within the pool. The pool automatically handles allocating new blocks as necessary. And (if we care) we can bulk free the pool before existing.
Probably a good idea in the longer term. The patch as posted would do for the upcoming release, though.
Thanks, both.
Re: [PATCH 1/1] mingw: safe-guard a bit more against getenv() problems
In reply to Jeff HostetlerHi Jeff,
Quoted 15 lines · Jeff Hostetler
Quoted 14 lines · Johannes Schindelin via GitGitGadget
From: Johannes Schindelin <[email protected]>
compat/mingw.c +1
-1@@ -1632,7 +1632,7 @@ int mingw_kill(pid_t pid, int sig)*/ char *mingw_getenv (const char *name) { -#define GETENV_MAX_RETAIN 30 +#define GETENV_MAX_RETAIN 64static char *values[GETENV_MAX_RETAIN ];static int value_counter;
Quoted 8 lines
compat/mingw.c int mingw_kill(pid_t pid, int sig) · line 1638
Quoted 2 lines
compat/mingw.c +0
-0int len_key, len_value;Why not use a
mem_poolfor this? We have that code isolated and re-usable now. Havemingw_getenv()copy the string into the pool always return the pointer from within the pool. The pool automatically handles allocating new blocks as necessary. And (if we care) we can bulk free the pool before existing.
The problem with the mem_pool would be that I do *not* want to accumulate tons and tons of stale copies (remember that difftool bug? Apparently it calls getenv() in a loop that scales with the number of modified files; I would hate to have copies for all of those).
And all you could do to remedy this would be to blow away all of them it the mem_pool grew too much, *including* the latest getenv() (which we painfully keep valid, e.g. in the case of git_committer_info() which calls getenv() *three* times *inside* the parameter list of the fmt_ident() function call.
Of course, we could use a hashmap combined with a priority queue that would cap at a given capacity, and once that is reached, culls the oldest entries until the capacity is maintained again.
Taking a step back, thought, we are talking only about the Windows-specific code, which is code we happen to have *some* control over. On other platforms, we have a lot less control.
Meaning: the true fix would be some sort of static analysis that ensures that getenv()'s returned values are duplicated unless it is super easy to reason that the use is safe.
Ciao,
Dscho
Re: [PATCH 1/1] mingw: safe-guard a bit more against getenv() problems
In reply to Johannes Schindelin via GitGitGadgetQuoted 11 lines · Johannes Schindelin via GitGitGadget
From: Johannes Schindelin <[email protected]>
Running up to v2.21.0, we fixed two bugs that were made prominent by the Windows-specific change to retain copies of only the 30 latest
getenv()calls' returned strings, invalidating any copies of previousgetenv()calls' return values.While this really shines a light onto bugs of the form where we hold onto getenv()'s return values without copying them, it is also a real problem for users.
Quoted 7 lines
And even if Jeff King's patches merged via
773e408881(Merge branch 'jk/save-getenv-result', 2019-01-29) provide further work on that front, we are far from done. Just one example: on Windows, we unset environment variables when spawning new processes, which potentially invalidates strings that were previously obtained viagetenv(), and therefore we have to duplicate environment values that are somehow involved in spawning new processes (e.g.GIT_MAN_VIEWERinshow_man_page()).
Belated review, as this is already in master, but: yes, I absolutely support this patch, even on top of my patches. Those were just the cases I found by poking around for a few minutes, and I'm sure there are many more.
-Peff
Re: [PATCH 1/1] mingw: safe-guard a bit more against getenv() problems
In reply to Johannes SchindelinQuoted 11 lines · Johannes Schindelin
Quoted 5 lines
Why not use a
mem_poolfor this? We have that code isolated and re-usable now. Havemingw_getenv()copy the string into the pool always return the pointer from within the pool. The pool automatically handles allocating new blocks as necessary. And (if we care) we can bulk free the pool before existing.The problem with the
mem_poolwould be that I do *not* want to accumulate tons and tons of stale copies (remember thatdifftoolbug? Apparently it callsgetenv()in a loop that scales with the number of modified files; I would hate to have copies for all of those).
Quoted 5 lines
And all you could do to remedy this would be to blow away all of them it the
mem_poolgrew too much, *including* the latestgetenv()(which we painfully keep valid, e.g. in the case ofgit_committer_info()which callsgetenv()*three* times *inside* the parameter list of thefmt_ident()function call.
One suggestion I made elsewhere[1] is to use strintern() to de-dup the responses, and then keep them around forever. That bounds the memory usage by the number of unique environment variables requested (which is probably finite and limited), rather than the number of getenv() calls (which as we have seen can be done in an arbitrary-sized loop).
I say "probably" because it's possible that there is code which mutates the environment, reads it back, etc, in a loop. But that would be pretty out-of-character for Git, I think.
-Peff