git-credential-store: skip empty lines and comments from store
PATCH 1/1 git-credential-store: skip empty lines and comments from store
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported as failing[1] to parse as valid credentials.
instead of passing every line to the matcher as read, trim them from spaces and skip the ones that will be otherwise empty or start with "#" (assumed to be comments)
[1] https://stackoverflow.com/a/61420852/5005936
--- credential-store.c | 3 +++ t/t0302-credential-store.sh | 17 +++++++++++++++++ 2 files changed, 20 insertions(+)
credential-store.c +3 -0
@@ -24,6 +24,9 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { +strbuf_trim (&line); +if (line.len ==0 || *line.buf =='#' ) +continue ;credential_from_url (&entry, line.buf);if (entry.username && entry.password &&credential_match (c, &entry)) {
t/t0302-credential-store.sh +17 -0
@@ -120,4 +120,21 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi test_must_be_empty "$HOME/.config/git/credentials" ' +test_expect_success 'get: allow for empty lines or comments in store file' ' + echo "#this is a comment" >"$HOME/.git-credentials" && + echo "" >>"$HOME/.git-credentials" && + echo "https://user:[email protected]" >>"$HOME/.git-credentials" && + echo " " >>"$HOME/.git-credentials" && + check fill store <<-\EOF + protocol=https + host=example.com + -- + protocol=https + host=example.com + username=user + password=pass + -- + EOF +' + test_done
Signature · 1 line
2.26.2.569.g1d74ac4d14
[Cc:+Dirk -- so he knows his bug report[1] wasn't ignored]
Quoted 13 lines · Carlo Marcelo Arenas Belón
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported as failing[1] to parse as valid credentials.
instead of passing every line to the matcher as read, trim them from spaces and skip the ones that will be otherwise empty or start with "#" (assumed to be comments)
Reported-by Dirk <[email protected]>Signed-off-by Carlo Marcelo Arenas Belón <[email protected]>---
t/t0302-credential-store.sh test_expect_success 'erase: erase matching credentials from both xdg and home fi · line 124
Quoted 5 lines
t/t0302-credential-store.sh +5
-0@@ -120,4 +120,21 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi +test_expect_success 'get: allow for empty lines or comments in store file' ' + echo "#this is a comment" >"$HOME/.git-credentials" && + echo "" >>"$HOME/.git-credentials" && + echo "https://user:[email protected]" >>"$HOME/.git-credentials" && + echo " " >>"$HOME/.git-credentials" &&
Is there a reason you don't use a here-doc for the above (which would be less noisy)? For instance:
q_to_tab >"$HOME/.git-credentials" <<-\EOF &&
#this is a comment
Quoted 13 lines · Eric Sunshine
[Cc:+Dirk -- so he knows his bug report[1] wasn't ignored]
t/t0302-credential-store.sh test_expect_success 'erase: erase matching credentials from both xdg and home fi · line 124
Quoted 7 lines · Carlo Marcelo Arenas Belón
t/t0302-credential-store.sh +5
-0@@ -120,4 +120,21 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi +test_expect_success 'get: allow for empty lines or comments in store file' ' + echo "#this is a comment" >"$HOME/.git-credentials" && + echo "" >>"$HOME/.git-credentials" && + echo "https://user:[email protected]" >>"$HOME/.git-credentials" && + echo " " >>"$HOME/.git-credentials" &&Is there a reason you don't use a here-doc for the above (which would be less noisy)? For instance:
because I didn't knew it existed ;), thanks for your help and will include for next version.
Carlo
PATCH v2 1/1 git-credential-store: skip empty lines and comments from store
In reply to Carlo Marcelo Arenas Belónwith the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported failing[1] to parse as valid credentials.
instead of passing every line to the matcher for processing, trim them from spaces and skip the ones that will be otherwise empty or that start with "#" (assumed to be comments)
[1] https://stackoverflow.com/a/61420852/5005936
---
v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentation
Documentation/git-credential-store.txt | 7 +++++++ credential-store.c | 3 +++ t/t0302-credential-store.sh | 19 +++++++++++++++++++ 3 files changed, 29 insertions(+)
Documentation/git-credential-store.txt +7 -0
@@ -101,6 +101,13 @@ username (if we already have one) match, then the password is returned to Git. See the discussion of configuration in linkgit:gitcredentials[7] for more information. +Note that the file used is not a configuration file and should be ideally +managed only through git, as any manually introduced typos will compromise +the validation of credentials. + +The parser will ignore any lines starting with the '#' character during +the processing of credentials, though. + GIT --- Part of the linkgit:git[1] suite
credential-store.c +3 -0
@@ -24,6 +24,9 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { +strbuf_trim (&line); +if (line.len ==0 || *line.buf =='#' ) +continue ;credential_from_url (&entry, line.buf);if (entry.username && entry.password &&credential_match (c, &entry)) {
t/t0302-credential-store.sh +19 -0
@@ -120,4 +120,23 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi test_must_be_empty "$HOME/.config/git/credentials" ' +test_expect_success 'get: allow for empty lines or comments in store file' ' + q_to_cr >"$HOME/.git-credentials" <<-\EOF && + #this is a comment and the next line contains leading spaces + Q + https://user:[email protected] + Q + EOF + check fill store <<-\EOF + protocol=https + host=example.com + -- + protocol=https + host=example.com + username=user + password=pass + -- + EOF +' + test_done
Signature · 1 line
2.26.2.569.g1d74ac4d14
Quoted 3 lines · Carlo Marcelo Arenas Belón
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported failing[1] to parse as valid credentials.
Those were never supposed to work. I'm mildly surprised that they did. It looks like the username/password check here is what prevented us from matching an empty line against a very broad pattern:
while (strbuf_getline_lf(&line, fh) != EOF) {
credential_from_url(&entry, line.buf);
if (entry.username && entry.password &&
credential_match(c, &entry)) {
found_credential = 1;And there was no such thing as a comment character. E.g., a "commented-out" url like this:
$ echo "#https://user:[email protected]/" >creds
would still be matched:
$ echo host=example.com | git credential-store --file creds get host=example.com username=user password=pass
I guess I'm not really opposed to adding this as a feature, but I think the justification should be "because it is somehow useful" and not because it's a bugfix.
Quoted 5 lines
t/t0302-credential-store.sh +0
-0
t/t0302-credential-store.sh test_expect_success 'erase: erase matching credentials from both xdg and home fi · line 129
Quoted 10 lines
t/t0302-credential-store.sh +7
-0@@ -120,4 +120,23 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi test_must_be_empty "$HOME/.config/git/credentials" ' +test_expect_success 'get: allow for empty lines or comments in store file' ' + q_to_cr >"$HOME/.git-credentials" <<-\EOF && + #this is a comment and the next line contains leading spaces + Q + https://user:[email protected] + Q + EOF
q_to_cr is a little weird here, as we wouldn't expect there to be CRs in the file. They do get removed by strbuf_trim(), even in non-comment lines. But perhaps "sed s/Q//" would accomplish the same thing (making the whitespace more visible) without making anyone wonder whether the CR is an important part of the test?
-Peff
Quoted 7 lines · Jeff King
Quoted 3 lines · Carlo Marcelo Arenas Belón
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported failing[1] to parse as valid credentials.
Those were never supposed to work. I'm mildly surprised that they did.
agree on that, which is why I have to admit I couldn't find the right wording for the previous paragraph, and also why I didn't pinpoint a specific commit as the one that introduced a bug.
it was instead just meant to describe a "current state of affairs" and I was hoping the documentation entry will help guide future users to be more careful with this file.
Quoted 3 lines
I guess I'm not really opposed to adding this as a feature, but I think the justification should be "because it is somehow useful" and not because it's a bugfix.
I think that is a matter of semantics, because for some users it seems like a regression on the last bugfix and therefore might be used (albeit IMHO incorrectly) as an excuse not to upgrade, which will be even worst.
so yes, I am implementint this "feature" to prevent them to have an excuse but it also means we would most likely need to fastrack this into maint.
Quoted 8 lines
Quoted 8 lines
t/t0302-credential-store.sh +0
-0@@ -120,4 +120,23 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi test_must_be_empty "$HOME/.config/git/credentials" '
Quoted 13 lines
t/t0302-credential-store.sh test_expect_success 'erase: erase matching credentials from both xdg and home fi · line 129
Quoted 7 lines
t/t0302-credential-store.sh +7
-0+test_expect_success 'get: allow for empty lines or comments in store file' ' + q_to_cr >"$HOME/.git-credentials" <<-\EOF && + #this is a comment and the next line contains leading spaces + Q + https://user:[email protected] + Q + EOF
q_to_cris a little weird here, as we wouldn't expect there to be CRs in the file. They do get removed bystrbuf_trim(), even in non-comment lines. But perhaps "sed s/Q//" would accomplish the same thing (making the whitespace more visible) without making anyone wonder whether the CR is an important part of the test?
I know, but commiting a line with 1 tab and 4 empty spaces instead of using Q seemed even worst and q_to_cr almost fits the bill and might even make this test "windows compatible" for once ;)
will rewrite then using test_write_lines and I hope it is still as readable without having to resort to the originl ugly chain of echo
Carlo
Quoted 16 lines · Carlo Marcelo Arenas Belón
Quoted 9 lines · Jeff King
t/t0302-credential-store.sh test_expect_success 'erase: erase matching credentials from both xdg and home fi · line 129
Quoted 7 lines · Carlo Marcelo Arenas Belón
t/t0302-credential-store.sh +7
-0+test_expect_success 'get: allow for empty lines or comments in store file' ' + q_to_cr >"$HOME/.git-credentials" <<-\EOF && + #this is a comment and the next line contains leading spaces + Q + https://user:[email protected] + Q + EOFQuoted 5 lines
q_to_cris a little weird here, as we wouldn't expect there to be CRs in the file. They do get removed bystrbuf_trim(), even in non-comment lines. But perhaps "sed s/Q//" would accomplish the same thing (making the whitespace more visible) without making anyone wonder whether the CR is an important part of the test?
Quoted 3 lines
I know, but commiting a line with 1 tab and 4 empty spaces instead of using Q seemed even worst and
q_to_cralmost fits the bill and might even make this test "windows compatible" for once ;)
Hmm? In the proposal[1], q_to_tab() was used, not q_to_cr(), and q_to_tab() seemed to fit quite well as a replacement for the original 'echo' chain[2] which emitted one whitespace-only line. Substituting that whitespace-only line with "Q" (which gets translated into a TAB) seems to fit much better than q_to_cr().
Quoted 2 lines
will rewrite then using
test_write_linesand I hope it is still as readable without having to resort to the originl ugly chain of echo
test_write_lines() doesn't do a very good job of representing -- at a glance -- what the "expected" output should be. The q_to_tab() block made it much more clear.
(Having said that, it's probably not worth another re-roll just to change that.)
[1]: https://lore.kernel.org/git/CAPig+cR8HKcbNxxw65ERz0iHvnO5aC6RXvF9NjvFTySXpcHCSQ@mail.gmail.com/
[2]: https://lore.kernel.org/git/[email protected]/
Quoted 1 line · Jeff King
Those were never supposed to work. I'm mildly surprised that they did.
I know that feeling X-<. It probably was a mistake to pretend that the file were editable with an editor, but it is too late to fix that.
The next complainer may say that "; comment" no longer works. We probably should draw a line somewhere, but I am not sure where.
Quoted 7 lines · Junio C Hamano
Quoted 1 line · Jeff King
Those were never supposed to work. I'm mildly surprised that they did.
I know that feeling X-<. It probably was a mistake to pretend that the file were editable with an editor, but it is too late to fix that.
I guess it is somewhat my fault for documenting the format at all git-credential-store.txt. I probably should have said "this is a black box and is subject to change".
Quoted 2 lines
The next complainer may say that "; comment" no longer works. We probably should draw a line somewhere, but I am not sure where.
If we get a bug report for "credential-store does not respect core.commentChar" I think my head would explode. :)
-Peff
PATCH v3 1/1 git-credential-store: skip empty lines and comments from store
In reply to Carlo Marcelo Arenas Belónwith the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
using the store file in this manner wasn't intended by the original code and it had latent issues which the new code dutifully prevented but since the strings used wouldn't had been valid credentials anyway we could instead detect them and skip the matching logic and therefore formalize this "feature".
trim all lines as they are being read from the store file and skip the ones that will be otherwise empty or that start with "#" (therefore assuming them to be comments)
[1] https://stackoverflow.com/a/61420852/5005936
---
v3:
* avoid using q_to_cr as suggested by Peff
* a more verbose commit message and slightly more complete documentation v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentation
Documentation/git-credential-store.txt | 7 +++++++ credential-store.c | 3 +++ t/t0302-credential-store.sh | 15 +++++++++++++++ 3 files changed, 25 insertions(+)
Documentation/git-credential-store.txt +7 -0
@@ -101,6 +101,13 @@ username (if we already have one) match, then the password is returned to Git. See the discussion of configuration in linkgit:gitcredentials[7] for more information. +Note that the file used is not a configuration file and should be ideally +managed only through git, as any manually introduced typos will compromise +the validation of credentials. + +The parser will ignore any lines starting with the '#' character during +the processing of credentials for read, though. + GIT --- Part of the linkgit:git[1] suite
credential-store.c +3 -0
@@ -24,6 +24,9 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { +strbuf_trim (&line); +if (line.len ==0 || *line.buf =='#' ) +continue ;credential_from_url (&entry, line.buf);if (entry.username && entry.password &&credential_match (c, &entry)) {
t/t0302-credential-store.sh +15 -0
@@ -120,4 +120,19 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi test_must_be_empty "$HOME/.config/git/credentials" ' +test_expect_success 'get: allow for empty lines or comments in store file' ' + test_write_lines "#comment" " " "" \ + https://user:[email protected] >"$HOME/.git-credentials" && + check fill store <<-\EOF + protocol=https + host=example.com + -- + protocol=https + host=example.com + username=user + password=pass + -- + EOF +' + test_done
Signature · 1 line
2.26.2.569.g1d74ac4d14
Hi Carlo,
Quoted 11 lines · Carlo Marcelo Arenas Belón
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
using the store file in this manner wasn't intended by the original code and it had latent issues which the new code dutifully prevented but since the strings used wouldn't had been valid credentials anyway we could instead detect them and skip the matching logic and therefore formalize this "feature".
trim all lines as they are being read from the store file and skip the
Does trimming affect any credentials that may have spaces either end? I've not looked at the format.
Quoted 26 lines
ones that will be otherwise empty or that start with "#" (therefore assuming them to be comments)
[1] https://stackoverflow.com/a/61420852/5005936
Reported-by Dirk <[email protected]>Helped-by Eric Sunshine <[email protected]>Signed-off-by Carlo Marcelo Arenas Belón <[email protected]>---
v3:
* avoid usingq_to_cras suggested by Peff
* a more verbose commit message and slightly more complete documentation v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentationDocumentation/git-credential-store.txt | 7 +++++++ credential-store.c | 3 +++ t/t0302-credential-store.sh | 15 +++++++++++++++ 3 files changed, 25 insertions(+)Documentation/git-credential-store.txt +0
-0
Documentation/git-credential-store.txt username (if we already have one) match, then the password is returned · line 109
Quoted 9 lines
Documentation/git-credential-store.txt +6
-0@@ -101,6 +101,13 @@ username (if we already have one) match, then the password is returned to Git. See the discussion of configuration in linkgit:gitcredentials[7] for more information. +Note that the file used is not a configuration file and should be ideally +managed only through git, as any manually introduced typos will compromise +the validation of credentials. + +The parser will ignore any lines starting with the '#' character during +the processing of credentials for read, though.
Should the trimming of leading/trailing spaces be mentioned?
Also the git-credential page mentions that the credential must end with a blank line ("don't forget.."). Should that be mentioned here, or have I misunderstood?
Quoted 32 lines
Documentation/git-credential-store.txt +1
-0+ GIT --- Part of the linkgit:git[1] suitecredential-store.c +3
-0@@ -24,6 +24,9 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { +strbuf_trim (&line); +if (line.len ==0 || *line.buf =='#' ) +continue ;credential_from_url (&entry, line.buf);if (entry.username && entry.password &&credential_match (c, &entry)) {t/t0302-credential-store.sh +6
-0@@ -120,4 +120,19 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi test_must_be_empty "$HOME/.config/git/credentials" ' +test_expect_success 'get: allow for empty lines or comments in store file' ' + test_write_lines "#comment" " " "" \ + https://user:[email protected] >"$HOME/.git-credentials" && + check fill store <<-\EOF + protocol=https + host=example.com
t/t0302-credential-store.sh test_expect_success 'erase: erase matching credentials from both xdg and home fi · line 138
Quoted 10 lines
t/t0302-credential-store.sh +9
-0+ -- + protocol=https + host=example.com + username=user + password=pass + -- + EOF +' + test_done
Philip
Quoted 11 lines · Philip Oakley
Quoted 10 lines · Carlo Marcelo Arenas Belón
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
using the store file in this manner wasn't intended by the original code and it had latent issues which the new code dutifully prevented but since the strings used wouldn't had been valid credentials anyway we could instead detect them and skip the matching logic and therefore formalize this "feature".
Quoted 3 lines
Quoted 1 line
trim all lines as they are being read from the store file and skip the
Does trimming affect any credentials that may have spaces either end?
all credentials are url encoded so the only spaces that are affected are the ones that were added by careless editing of that file.
as Eric pointed out, any tabs will be silently "cleaned" as well.
Quoted 1 line
Should the trimming of leading/trailing spaces be mentioned?
I wanted to keep that as an undocumented "feature" as it is just meant to help people to avoid a fatal error during the transition but didn't want to encourage people thinking it is a supported part or even worse encourage people to start editing their files to add tabs and spaces.
Junio's suggested documentation fix[1] makes that clearer that I could
Quoted 3 lines
Also the git-credential page mentions that the credential must end with a blank line ("don't forget.."). Should that be mentioned here, or have I misunderstood?
that is for the protocol part of it (which is used between git and the credential helper), the file format for this specific helper doesn't require that.
Carlo
Hi Carlo,
Thanks for the clarification.
Philip
Quoted 37 lines · Carlo Marcelo Arenas Belón
Quoted 5 lines · Philip Oakley
Quoted 4 lines · Carlo Marcelo Arenas Belón
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
Quoted 8 lines
Quoted 7 lines
using the store file in this manner wasn't intended by the original code and it had latent issues which the new code dutifully prevented but since the strings used wouldn't had been valid credentials anyway we could instead detect them and skip the matching logic and therefore formalize this "feature".
trim all lines as they are being read from the store file and skip the
Does trimming affect any credentials that may have spaces either end?
all credentials are url encoded so the only spaces that are affected are the ones that were added by careless editing of that file.
as Eric pointed out, any tabs will be silently "cleaned" as well.
Quoted 1 line
Should the trimming of leading/trailing spaces be mentioned?
I wanted to keep that as an undocumented "feature" as it is just meant to help people to avoid a fatal error during the transition but didn't want to encourage people thinking it is a supported part or even worse encourage people to start editing their files to add tabs and spaces.
Junio's suggested documentation fix[1] makes that clearer that I could
Quoted 3 lines
Also the git-credential page mentions that the credential must end with a blank line ("don't forget.."). Should that be mentioned here, or have I misunderstood?
that is for the protocol part of it (which is used between git and the credential helper), the file format for this specific helper doesn't require that.
Carlo
Thank you very much. That's correct and ideal in my eyes.
Regarding the comments, this is a new feature, of course. I think it's a worthless discussion about the question if the correct handling of empty lines are a bugfix or a new feature. In fact empty lines were handled correcty (ignored). So this might be considered a feature. But it wasn't documented, so it's a bugfix...
Anyway. Thank you all.
Dirk
Am 27.04.20 um 14:59 schrieb Carlo Marcelo Arenas Belón:
Quoted 88 lines
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
using the store file in this manner wasn't intended by the original code and it had latent issues which the new code dutifully prevented but since the strings used wouldn't had been valid credentials anyway we could instead detect them and skip the matching logic and therefore formalize this "feature".
trim all lines as they are being read from the store file and skip the ones that will be otherwise empty or that start with "#" (therefore assuming them to be comments)
[1] https://stackoverflow.com/a/61420852/5005936
Reported-by Dirk <[email protected]>Helped-by Eric Sunshine <[email protected]>Signed-off-by Carlo Marcelo Arenas Belón <[email protected]>---
v3:
* avoid usingq_to_cras suggested by Peff
* a more verbose commit message and slightly more complete documentation v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentationDocumentation/git-credential-store.txt | 7 +++++++ credential-store.c | 3 +++ t/t0302-credential-store.sh | 15 +++++++++++++++ 3 files changed, 25 insertions(+)Documentation/git-credential-store.txt +7
-0@@ -101,6 +101,13 @@ username (if we already have one) match, then the password is returned to Git. See the discussion of configuration in linkgit:gitcredentials[7] for more information. +Note that the file used is not a configuration file and should be ideally +managed only through git, as any manually introduced typos will compromise +the validation of credentials. + +The parser will ignore any lines starting with the '#' character during +the processing of credentials for read, though. + GIT --- Part of the linkgit:git[1] suitecredential-store.c +3
-0@@ -24,6 +24,9 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { +strbuf_trim (&line); +if (line.len ==0 || *line.buf =='#' ) +continue ;credential_from_url (&entry, line.buf);if (entry.username && entry.password &&credential_match (c, &entry)) {t/t0302-credential-store.sh +15
-0@@ -120,4 +120,19 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi test_must_be_empty "$HOME/.config/git/credentials" ' +test_expect_success 'get: allow for empty lines or comments in store file' ' + test_write_lines "#comment" " " "" \ + https://user:[email protected] >"$HOME/.git-credentials" && + check fill store <<-\EOF + protocol=https + host=example.com + -- + protocol=https + host=example.com + username=user + password=pass + -- + EOF +' + test_done
Quoted 1 line · Carlo Marcelo Arenas Belón
with the added checks for invalid URLs in credentials, any locally
s/with/With/
Quoted 2 lines
modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
These files are not supposed to be viewed or edited without the help of the credential helpers. Do these blank lines and comments even survive when a new credential is approved, or do we just overwrite and lose them?
I'd rather not to do either, if we did not have to, but if it were necessary for us to do something, I am OK to ignore empty lines. But I'd prefer not to mix the new "# comment" feature in, if we did not have to.
Also, triming the lines that are not empty is unwarranted. IIUC, what the "store" action writes encodes whitespaces, so as soon as you see whitespace on either end, (or anywhere on the line for that matter), it is a hand-edited cruft in the file. If you ignore comments, you probably should ignore those lines, too.
Quoted 7 lines · Junio C Hamano
Quoted 2 lines
modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
These files are not supposed to be viewed or edited without the help of the credential helpers. Do these blank lines and comments even survive when a new credential is approved, or do we just overwrite and lose them?
That's a good question. In the older code we do save them, because credential-store passes through lines which don't match the credential we're operating on.
But in Carlo's patch, the immediate "continue" means we wouldn't ever call "other_cb", which is what does that pass-through.
Quoted 10 lines
I'd rather not to do either, if we did not have to, but if it were necessary for us to do something, I am OK to ignore empty lines. But I'd prefer not to mix the new "# comment" feature in, if we did not have to.
Also, triming the lines that are not empty is unwarranted. IIUC, what the "store" action writes encodes whitespaces, so as soon as you see whitespace on either end, (or anywhere on the line for that matter), it is a hand-edited cruft in the file. If you ignore comments, you probably should ignore those lines, too.
Yeah, all of that seems quite sensible.
-Peff
Quoted 10 lines · Jeff King
Quoted 7 lines · Junio C Hamano
Quoted 2 lines
modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
These files are not supposed to be viewed or edited without the help of the credential helpers. Do these blank lines and comments even survive when a new credential is approved, or do we just overwrite and lose them?
Quoted 6 lines
That's a good question. In the older code we do save them, because credential-store passes through lines which don't match the credential we're operating on.
But in Carlo's patch, the immediate "continue" means we wouldn't ever call "
other_cb", which is what does that pass-through.
So, does that mean the patch that started this thread will still help users who wrote custom comments and blank lines in their credential store by letting git-credential-store start up, but leaves a ticking bomb for them to lose these precious comments and blanks once they add a new site, change password, etc., at which point the user realizes that comments and blanks are not supported after all?
Quoted 5 lines
Quoted 5 lines
I'd rather not to do either, if we did not have to, but if it were necessary for us to do something, I am OK to ignore empty lines. But I'd prefer not to mix the new "# comment" feature in, if we did not have to.
Quoted 7 lines
Quoted 5 lines
Also, triming the lines that are not empty is unwarranted. IIUC, what the "store" action writes encodes whitespaces, so as soon as you see whitespace on either end, (or anywhere on the line for that matter), it is a hand-edited cruft in the file. If you ignore comments, you probably should ignore those lines, too.
Yeah, all of that seems quite sensible.
I think the first patch we need is a (belated) documentation patch, that adds to the existing "STORAGE FORMAT". We already say "Each credential is stored on its own line as a URL", but we do not say anything about allowing other cruft in the file. We probably should. Adding a "comment" feature, if anybody feels like it, is OK and we can loosen the paragraph when that happens.
-- >8 --
Subject: credential-store: document the file format a bit more
Reading a malformed credential URL line and silently ignoring it does not mean that we promise to torelate and/or keep empty lines and "# commented" lines forever.
Some people seem to take anything that is not explicitly forbidden as allowed, but the world does not work that way.
--- Documentation/git-credential-store.txt | 4 ++++ 1 file changed, 4 insertions(+)
Documentation/git-credential-store.txt +4 -0
@@ -94,6 +94,10 @@ stored on its own line as a URL like: https://user:[email protected] ------------------------------ +No other kinds of lines (e.g. empty lines or comment lines) are +allowed in the file, even though some may be silently ignored. Do +not view or edit the file with editors. + When Git needs authentication for a particular URL context, credential-store will consider that context a pattern to match against each entry in the credentials file. If the protocol, hostname, and
Quoted 12 lines · Junio C Hamano
Quoted 11 lines
Quoted 4 lines
These files are not supposed to be viewed or edited without the help of the credential helpers. Do these blank lines and comments even survive when a new credential is approved, or do we just overwrite and lose them?
That's a good question. In the older code we do save them, because credential-store passes through lines which don't match the credential we're operating on.
But in Carlo's patch, the immediate "continue" means we wouldn't ever call "
other_cb", which is what does that pass-through.
Quoted 6 lines
So, does that mean the patch that started this thread will still help users who wrote custom comments and blank lines in their credential store by letting git-credential-store start up, but leaves a ticking bomb for them to lose these precious comments and blanks once they add a new site, change password, etc., at which point the user realizes that comments and blanks are not supported after all?
Yes, exactly. If I start with:
cat >creds <<\EOF
# this is a comment https://user:[email protected]/ EOF
then:
echo url=https://other:[email protected] | git credential-store --file=creds store
with v2.26.0 results in:
https://other:[email protected]
# this is a comment https://user:[email protected]
but applying the patch on top:
https://other:[email protected]
https://user:[email protected]/
That raises another issue about comments, too: we make no promises about where we write new entries. They could be inserted between a comment line and one it is meant to annotate (that line could even be moved if we reject and then re-approve a credential).
Quoted 6 lines
I think the first patch we need is a (belated) documentation patch, that adds to the existing "STORAGE FORMAT". We already say "Each credential is stored on its own line as a URL", but we do not say anything about allowing other cruft in the file. We probably should. Adding a "comment" feature, if anybody feels like it, is OK and we can loosen the paragraph when that happens.
The more I look into it, the more negative I am on adding such a comment feature.
Quoted 2 lines
-- >8 --
Subject: credential-store: document the file format a bit more
Yeah, this certainly clarifies things. I suppose one could ask why we'd bother documenting the format at all, then, though I do think it could be helpful for debugging.
Quoted 3 lines
Reading a malformed credential URL line and silently ignoring it does not mean that we promise to torelate and/or keep empty lines and "# commented" lines forever.
s/torelate/tolerate/
-Peff
Quoted 20 lines · Jeff King
Quoted 12 lines · Junio C Hamano
Quoted 11 lines
Quoted 4 lines
These files are not supposed to be viewed or edited without the help of the credential helpers. Do these blank lines and comments even survive when a new credential is approved, or do we just overwrite and lose them?
That's a good question. In the older code we do save them, because credential-store passes through lines which don't match the credential we're operating on.
But in Carlo's patch, the immediate "continue" means we wouldn't ever call "
other_cb", which is what does that pass-through.Quoted 6 lines
So, does that mean the patch that started this thread will still help users who wrote custom comments and blank lines in their credential store by letting git-credential-store start up, but leaves a ticking bomb for them to lose these precious comments and blanks once they add a new site, change password, etc., at which point the user realizes that comments and blanks are not supported after all?
Quoted 2 lines
The more I look into it, the more negative I am on adding such a comment feature.
FWIW I am not interested on implementing a comment feature (regardless of what the subject says, and indeed was going to use quotes around it in my original patch), but it was already too long ;)
my objective was to allow people that were not upgrading because of this "regression" a path forward, but after all this discussion I am convinced too that my approach is misguided.
if users are already manually editing this file, they should be able to fix the corrupted lines themselves but the error shown needs to be improved. and the failure mode corrected as well so they are not blocked.
this also means that my "sneaky" way to fix formatting issues is gone and the passthrough is called so nothing will be lost, but as you also said the location is not warranted to be kept so this comment "feature" is as useless as intended.
would something like the following (wouldn't pass the last test yet as it will need special handling to be able to handle the full path of the broken file as shown in the warning sent to stderr) on top of Junio's make more sense?
it still "silently" fixes the formatting issues where a leading tab (test added) or spaces were added to easy the transition, but probably should also error out as Junio suggested originally.
had also updated the commit message and what is left of the documentation.
Carlo
-- >8 --
Subject: [RFC PATCH v4] git-credential-store: skip empty lines and comments
from store
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
using the store file in this manner wasn't intended by the original code and it had latent issues which the new code dutifully prevented but since the strings used wouldn't had been valid credentials anyway we could instead detect them and skip the matching logic and therefore prevent a fatal failure.
trim all lines as they are being read from the store file and skip the ones that will be otherwise empty or that start with "#" (therefore assuming them to be comments) but warn users so they will be make aware of the issues introduced by their manual editing.
[1] https://stackoverflow.com/a/61420852/5005936
--- Documentation/git-credential-store.txt | 4 ++++ credential-store.c | 7 ++++++ t/t0302-credential-store.sh | 31 ++++++++++++++++++++++++++ 3 files changed, 42 insertions(+)
Documentation/git-credential-store.txt +4 -0
@@ -101,6 +101,10 @@ username (if we already have one) match, then the password is returned to Git. See the discussion of configuration in linkgit:gitcredentials[7] for more information. +The parser will ignore any lines that are empty or starting with '#' character +during the processing of credentials for read, but warn the user so they can +be corrected. + GIT --- Part of the linkgit:git[1] suite
credential-store.c +7 -0
@@ -24,6 +24,13 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { +strbuf_trim (&line); +if (line.len ==0 || *line.buf =='#' ) { +warning ("ignoring corrupted credential entry \"%s\" in %s" , line.buf, fn); +if (other_cb) +other_cb (&line); +continue ; + }credential_from_url (&entry, line.buf);if (entry.username && entry.password &&credential_match (c, &entry)) {
t/t0302-credential-store.sh +31 -0
@@ -120,4 +120,35 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi test_must_be_empty "$HOME/.config/git/credentials" ' +test_expect_success 'get: attempt to workaround formatting isssues (ex: tab)' ' + q_to_tab >"$HOME/.git-credentials" <<-\EOF && + Qhttps://user:[email protected] + EOF + check fill store <<-\EOF + protocol=https + host=example.com + -- + protocol=https + host=example.com + username=user + password=pass + -- + EOF +' + +test_expect_success 'get: skip empty lines or "comments" in store file' ' + test_write_lines "#comment" " " "" \ + https://user:[email protected] >"$HOME/.git-credentials" && + check fill store <<-\EOF + protocol=https + host=example.com + -- + protocol=https + host=example.com + username=user + password=pass + -- + EOF +' + test_done
Signature · 1 line
2.26.2.569.g1d74ac4d14
Quoted 19 lines · Junio C Hamano
Quoted 10 lines · Jeff King
Quoted 7 lines · Junio C Hamano
Quoted 2 lines
modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
These files are not supposed to be viewed or edited without the help of the credential helpers. Do these blank lines and comments even survive when a new credential is approved, or do we just overwrite and lose them?
Quoted 6 lines
That's a good question. In the older code we do save them, because credential-store passes through lines which don't match the credential we're operating on.
But in Carlo's patch, the immediate "continue" means we wouldn't ever call "
other_cb", which is what does that pass-through.
Quoted 6 lines
So, does that mean the patch that started this thread will still help users who wrote custom comments and blank lines in their credential store by letting git-credential-store start up, but leaves a ticking bomb for them to lose these precious comments and blanks once they add a new site, change password, etc., at which point the user realizes that comments and blanks are not supported after all?
yes, and it also helps users that might have added spaces or tabs around their credentials while editing them to still be able to use those instead of just failing to match them.
IMHO the only "regression" I was fixing was the fact that the current code will get to throw a fatal error with an unhelpful message and prevent access to valid credentials as shown by :
$ /git credential fill protocol=http host=example.com
warning: url has no scheme:
fatal: credential url cannot be parsed:
Username for 'http://example.com':
Quoted 40 lines
Quoted 5 lines
Quoted 5 lines
I'd rather not to do either, if we did not have to, but if it were necessary for us to do something, I am OK to ignore empty lines. But I'd prefer not to mix the new "# comment" feature in, if we did not have to.
Quoted 7 lines
Quoted 5 lines
Also, triming the lines that are not empty is unwarranted. IIUC, what the "store" action writes encodes whitespaces, so as soon as you see whitespace on either end, (or anywhere on the line for that matter), it is a hand-edited cruft in the file. If you ignore comments, you probably should ignore those lines, too.
Yeah, all of that seems quite sensible.
I think the first patch we need is a (belated) documentation patch, that adds to the existing "STORAGE FORMAT". We already say "Each credential is stored on its own line as a URL", but we do not say anything about allowing other cruft in the file. We probably should. Adding a "comment" feature, if anybody feels like it, is OK and we can loosen the paragraph when that happens.
-- >8 --
Subject: credential-store: document the file format a bit moreReading a malformed credential URL line and silently ignoring it does not mean that we promise to torelate and/or keep empty lines and "# commented" lines forever.
Some people seem to take anything that is not explicitly forbidden as allowed, but the world does not work that way.
Signed-off-by Junio C Hamano <[email protected]>--- Documentation/git-credential-store.txt | 4 ++++ 1 file changed, 4 insertions(+)Documentation/git-credential-store.txt +0
-0
Documentation/git-credential-store.txt stored on its own line as a URL like: · line 99
Quoted 6 lines
Documentation/git-credential-store.txt +3
-0@@ -94,6 +94,10 @@ stored on its own line as a URL like: https://user:[email protected] ------------------------------ +No other kinds of lines (e.g. empty lines or comment lines) are +allowed in the file, even though some may be silently ignored. Do +not view or edit the file with editors.
view should be ok; mentioning that any typos or extraneous characters will compromise the validation of credentials like I mentioned in my proposed documentation update probably worth doing here instead, too.
Carlo
Quoted 20 lines · Junio C Hamano
-- >8 --
Subject: credential-store: document the file format a bit moreReading a malformed credential URL line and silently ignoring it does not mean that we promise to torelate and/or keep empty lines and "# commented" lines forever.
Some people seem to take anything that is not explicitly forbidden as allowed, but the world does not work that way.
Signed-off-by Junio C Hamano <[email protected]>--- Documentation/git-credential-store.txt | 4 ++++ 1 file changed, 4 insertions(+)Documentation/git-credential-store.txt +0
-0
Documentation/git-credential-store.txt stored on its own line as a URL like: · line 101
Quoted 8 lines
Documentation/git-credential-store.txt +4
-0@@ -94,6 +94,10 @@ stored on its own line as a URL like: https://user:[email protected] ------------------------------ +No other kinds of lines (e.g. empty lines or comment lines) are +allowed in the file, even though some may be silently ignored. Do +not view or edit the file with editors. + When Git needs authentication for a particular URL context,
I like this.
I do suspect this is easy to run into accidentally. In $DAYJOB (in the context of [1]) there was a service that accidentally wrote a \n\n at the end of a line that was used by git-credential-store. Once the cause was tracked down, it was straightforward to fix, but I don't like the idea that others in a similar position may end up tempted to just not upgrade Git.
Independently, there is the thread we are replying to.
Independently, in Debian's bug tracking system, Stefan (cc-ed) reports[2]:
| the vulnerability in CVE-2020-11008 is related to the handling | of credential helpers in git. In Buster this has been fixed in | 1:2.20.1-2+deb10u3. This broke my existing configuration where | repositories have credential.helper=store set. This is | documented in /usr/share/man/man1/git-credential-store.1.gz | and other files from git, git-doc etc. | I am unsure how to proceed... is this helper now unsupported?
(Stefan, do you have more details? Did you manually populate your credential store? What error message do you get?)
I wonder if in addition to the above documentation change we may want something guaranteed to catch all cases where people would have experienced a regression, like
credential-store.c +2 -2
@@ -24,8 +24,8 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && +if (!credential_from_url_gently (&entry, line.buf,1 ) && + entry.username && entry.password &&credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {
And then we can tighten the handling of unrecognized lines to first warn and then error out, as a controlled change that doesn't lead people to regret updating git.
Thoughts?
Thanks,
Jonathan
[1] https://cs.opensource.google/copybara/copybara/+/master:java/com/google/copybara/git/GitOptions.java;drc=bc79a0b1ffe18f79dea0b75ba3a24b641a50a9fc;l=46
[2] https://bugs.debian.org/958929
Quoted 20 lines · Jonathan Nieder
I wonder if in addition to the above documentation change we may want something guaranteed to catch all cases where people would have experienced a regression, like
credential-store.c +2
-2@@ -24,8 +24,8 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && +if (!credential_from_url_gently (&entry, line.buf,1 ) && + entry.username && entry.password &&credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {
Quoted 3 lines
And then we can tighten the handling of unrecognized lines to first warn and then error out, as a controlled change that doesn't lead people to regret updating git.
I like that solution, as it mostly brings us back to the original behavior, as weird or unexpected as it was.
The only I think would be different is:
://user:[email protected]
which I think the old code would have treated as matching any protocol (and the new code will reject as a bogus URL).
I wondered if you'd need to free the partially-parsed struct fields on failure, but the answer is no; we'd clear it the next time through the loop, or at the very end of the function.
-Peff
Quoted 22 lines · Jeff King
Quoted 20 lines · Jonathan Nieder
I wonder if in addition to the above documentation change we may want something guaranteed to catch all cases where people would have experienced a regression, like
credential-store.c +2
-2@@ -24,8 +24,8 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && +if (!credential_from_url_gently (&entry, line.buf,1 ) && + entry.username && entry.password &&credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {
Quoted 6 lines
Quoted 3 lines
And then we can tighten the handling of unrecognized lines to first warn and then error out, as a controlled change that doesn't lead people to regret updating git.
I like that solution, as it mostly brings us back to the original behavior, as weird or unexpected as it was.
I like this version better as well, and we could even reuse my test case.
it wouldn't cover cases where there were leading spaces/tabs around the credential which I have to admit I liked just because it is more robust to bad input, and there is no sane way now to tell the user that there is invalid data anyway, but I am ok eitherway.
Carlo
Quoted 16 lines · Carlo Marcelo Arenas Belón
Quoted 9 lines
Quoted 9 lines
credential-store.c +2
-2while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && +if (!credential_from_url_gently (&entry, line.buf,1 ) && + entry.username && entry.password &&credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {Quoted 6 lines
Quoted 3 lines
And then we can tighten the handling of unrecognized lines to first warn and then error out, as a controlled change that doesn't lead people to regret updating git.
I like that solution, as it mostly brings us back to the original behavior, as weird or unexpected as it was.
Quoted 7 lines
I like this version better as well, and we could even reuse my test case.
it wouldn't cover cases where there were leading spaces/tabs around the credential which I have to admit I liked just because it is more robust to bad input, and there is no sane way now to tell the user that there is invalid data anyway, but I am ok eitherway.
I think if we're discouraging people from hand-editing the file, then that feature would be going in the wrong direction anyway.
Did you or Jonathan want to wrap it up with the test and commit message?
-Peff
Quoted 9 lines · Jeff King
Quoted 5 lines · Carlo Marcelo Arenas Belón
it wouldn't cover cases where there were leading spaces/tabs around the credential which I have to admit I liked just because it is more robust to bad input, and there is no sane way now to tell the user that there is invalid data anyway, but I am ok eitherway.
I think if we're discouraging people from hand-editing the file, then that feature would be going in the wrong direction anyway.
I think it will actually encourage them to edit the file (and complain of a regression), since they can see the credential is correct, but somehow git fails to work with it unlike their previous version.
remember there is no warning that could explain why that happens and with this pach there is no error either.
Quoted 1 line
Did you or Jonathan want to wrap it up with the test and commit message?
I was going to say Jonathan will have to do it, since he was the only one that didn't provide a patch (and therefore a sign off) but guess I can save everyone some time and practice my git-send-email skills (which obviously failed me)
Jonathan,
make sure your patch[1] is complete and correct and to resend it with a sign off, feel free to add me as Tested-by if you feel like, but I added myself as Helped-by anyway ;)
Junio,
didn't fix the typo that Peff suggested fixed[2], because I forgot and I wasn't sure if that also means I should SOB it, but if you could also consider my suggestion[3] then at you get my SOB warranted ;)
Carlo
PS. guess I should had send it as an RFC instead
[1] https://lore.kernel.org/git/[email protected]/
[2] https://lore.kernel.org/git/[email protected]/
[3] https://lore.kernel.org/git/20200427234909.GC61348@Carlos-MBP/
Quoted 11 lines · On Mon, 27 Apr 2020 22:25:10 -0700 Jonathan Nieder <[email protected]> wrote:
Independently, in Debian's bug tracking system, Stefan (cc-ed) reports[2]:
| the vulnerability in CVE-2020-11008 is related to the handling | of credential helpers in git. In Buster this has been fixed in | 1:2.20.1-2+deb10u3. This broke my existing configuration where | repositories have credential.helper=store set. This is | documented in /usr/share/man/man1/git-credential-store.1.gz | and other files from git, git-doc etc. | I am unsure how to proceed... is this helper now unsupported?
Quoted 2 lines
(Stefan, do you have more details? Did you manually populate your credential store? What error message do you get?)
I can't remember for sure - it's literally years ago - but I am pretty sure I did not *populate* it manually initially. However, I might have edited it to separate entries (https vs. smtp) and I might have added additional entries by c&p & editing.
I have replaced the one instance of \n\n used for separation with a normal single line break between entries and how it works fine. Before that I got the following output (that's obviously not very helpful if you don't know the whole story :)
Quoted 5 lines
git pull
warning: url has no scheme:
fatal: credential url cannot be parsed:
Already up to date.
Current branch master is up to date.
Signature · 1 line
Kind regards/Mit freundlichen Grüßen, Stefan Tauner
Quoted 9 lines · Jonathan Nieder
I wonder if in addition to the above documentation change we may want something guaranteed to catch all cases where people would have experienced a regression, like
credential-store.c +0
-0
credential-store.c static int parse_credential_file(const char *fn, · line 31
Quoted 10 lines
credential-store.c +2
-2@@ -24,8 +24,8 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && +if (!credential_from_url_gently (&entry, line.buf,1 ) && + entry.username && entry.password &&credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {
Hmph, so the idea is, instead of ignoring the potential error detected by credential_from_url() and using credential when it is available, we loudly attempt to parse and give warning on malformed entries when we discard a line?
I think that is an excellent idea.
It would be nicer if we can somehow add where we found the offending line, e.g.
/home/me/.gitcredential:396: url has no scheme: ://u:p@host/path
Do we feel it a bit disturbing that u:p is shown in the error message, without redacting, by the way?
Quoted 13 lines · Junio C Hamano
Quoted 11 lines · Jonathan Nieder
I wonder if in addition to the above documentation change we may want something guaranteed to catch all cases where people would have experienced a regression, like
credential-store.c +0
-0@@ -24,8 +24,8 @@ static int parse_credential_file(const char *fn, }
Quoted 13 lines
credential-store.c static int parse_credential_file(const char *fn, · line 31
Quoted 8 lines
credential-store.c +2
-2while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && +if (!credential_from_url_gently (&entry, line.buf,1 ) && + entry.username && entry.password &&credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {Hmph, so the idea is, instead of ignoring the potential error detected by
credential_from_url()and using credential when it is available, we loudly attempt to parse and give warning on malformed entries when we discard a line?
the idea was to silently ignore the line (notice quiet = 1), which is the "original behaviour" as Peff pointed out in his reply in [email protected][1]
Quoted 9 lines
I think that is an excellent idea.
It would be nicer if we can somehow add where we found the offending line, e.g.
/home/me/.gitcredential:396: url has no scheme: ://u:p@host/pathDo we feel it a bit disturbing that u:p is shown in the error message, without redacting, by the way?
we could do so with the following on top as a 5/4.
most of the test changes would go away in a reroll, and are similar to what would have been needed for my original suggested patch[2] with the exception that in this series we won't support silently fixing credentials which is something you mentioned a preference on.
Carlo
[1] https://lore.kernel.org/git/[email protected]/
[2] https://lore.kernel.org/git/20200428013726.GD61348@Carlos-MBP/
-- >8 --
Subject: [RFC PATCH] credential-store: SQUASH!!
add warnings
--- credential-store.c | 55 +++++++++++++++++++++++++++++++------ t/t0302-credential-store.sh | 42 ++++++++++++++++------------ 2 files changed, 70 insertions(+), 27 deletions(-)
credential-store.c +46 -9
@@ -6,6 +6,32 @@static struct lock_file credential_lock; +static char *redact_credential (const struct strbuf *line) +{ +struct strbuf redacted_line =STRBUF_INIT ; +char *at =strchr (line->buf,'@' ); +char *colon; +int redacted =0 ; + +if (at) { +strbuf_addf (&redacted_line,"%.*s" , + (int )(at - line->buf), line->buf); + colon =strrchr (redacted_line.buf,':' ); +if (colon && *(colon +1 ) !='/' && colon > redacted_line.buf) { + redacted_line.len = colon - redacted_line.buf +1 ; +strbuf_addstr (&redacted_line,"<redacted>" ); +strbuf_addstr (&redacted_line, at); + redacted =1 ; + } +else +strbuf_reset (&redacted_line); + } +if (!redacted) +strbuf_addbuf (&redacted_line, line); + +return redacted_line.buf; +} + static int parse_credential_file(const char *fn, struct credential *c, void (*match_cb)(struct credential *),
@@ -15,6 +41,7 @@ static int parse_credential_file(const char *fn,struct strbuf line =STRBUF_INIT ;struct credential entry =CREDENTIAL_INIT ;int found_credential =0 ; +int lineno =0 ; fh =fopen (fn,"r" );if (!fh) {
@@ -24,17 +51,27 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -if (!credential_from_url_gently (&entry, line.buf,1 ) && - entry.username && entry.password && -credential_match (c, &entry)) { - found_credential =1 ; -if (match_cb) { -match_cb (&entry); -break ; + lineno++; +if (!credential_from_url_gently (&entry, line.buf,1 )) { +if (entry.username && entry.password && +credential_match (c, &entry)) { + found_credential =1 ; +if (match_cb) { +match_cb (&entry); +break ; + } } +else if (other_cb) +other_cb (&line); + } +else { +if (other_cb) +other_cb (&line); +char *redacted =redact_credential (&line); +warning ("%s:%d ignoring invalid credential: %s" , + fn, lineno, redacted); +free (redacted); } -else if (other_cb) -other_cb (&line); }credential_clear (&entry);
t/t0302-credential-store.sh +24 -18
60 lines not on this page · raw message
Signature · 1 line
2.26.2.569.g1d74ac4d14
Quoted 11 lines · Carlo Marcelo Arenas Belón
Quoted 10 lines
Review
Quoted 5 lines
(quoted diff) +2
-0+ if (!credential_from_url_gently(&entry, line.buf, 1) && + entry.username && entry.password && credential_match(c, &entry)) { found_credential = 1; if (match_cb) {Hmph, so the idea is, instead of ignoring the potential error detected by
credential_from_url()and using credential when it is available, we loudly attempt to parse and give warning on malformed entries when we discard a line?
Quoted 1 line
the idea was to silently ignore the line (notice quiet = 1), which
Ah, sorry, I apparently did not read the patch carefully enough.
Thanks for a correction.
v4 cover credential-store: prevent fatal errors
In reply to Carlo Marcelo Arenas Belónwith the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] as failing to parse as valid credentials.
document better the format for the credential file and make sure any failures are still handled gently by the new code as a first step to allow people to upgrade without having first to fix their corrupted credential files.
[1] https://stackoverflow.com/a/61420852/5005936
Carlo Marcelo Arenas Belón (2):
git-credential-store: skip empty lines and comments from store
credential-store: make sure there is no regression with missing scheme
Jonathan Nieder (1):
git-credential-store: fix (WIP)
Junio C Hamano (1):
credential-store: document the file format a bit more
Documentation/git-credential-store.txt | 4 ++++ credential-store.c | 4 ++-- t/t0302-credential-store.sh | 32 ++++++++++++++++++++++++++ 3 files changed, 38 insertions(+), 2 deletions(-)
Signature · 1 line
2.26.2.569.g1d74ac4d14
PATCH v4 1/4 credential-store: document the file format a bit more
From: Junio C Hamano <[email protected]>
Reading a malformed credential URL line and silently ignoring it does not mean that we promise to torelate and/or keep empty lines and "# commented" lines forever.
Some people seem to take anything that is not explicitly forbidden as allowed, but the world does not work that way.
--- Documentation/git-credential-store.txt | 4 ++++ 1 file changed, 4 insertions(+)
Documentation/git-credential-store.txt +4 -0
@@ -94,6 +94,10 @@ stored on its own line as a URL like: https://user:[email protected] ------------------------------ +No other kinds of lines (e.g. empty lines or comment lines) are +allowed in the file, even though some may be silently ignored. Do +not view or edit the file with editors. + When Git needs authentication for a particular URL context, credential-store will consider that context a pattern to match against each entry in the credentials file. If the protocol, hostname, and
Signature · 1 line
2.26.2.569.g1d74ac4d14
PATCH v4 2/4 git-credential-store: skip empty lines and comments from store
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
add corresponding failing cases
[1] https://stackoverflow.com/a/61420852/5005936
--- t/t0302-credential-store.sh | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
t/t0302-credential-store.sh +15 -0
@@ -120,4 +120,19 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi test_must_be_empty "$HOME/.config/git/credentials" ' +test_expect_failure 'get: store file can contain empty/bogus lines' ' + test_write_lines "#comment" " " "" \ + https://user:[email protected] >"$HOME/.git-credentials" && + check fill store <<-\EOF + protocol=https + host=example.com + -- + protocol=https + host=example.com + username=user + password=pass + -- + EOF +' + test_done
Signature · 1 line
2.26.2.569.g1d74ac4d14
Quoted 1 line · Carlo Marcelo Arenas Belón
git-credential-store: skip empty lines and comments from store
I don't see anything in this patch which makes it skip anything at all; it only introduces a new test.
Quoted 9 lines
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
add corresponding failing cases
[1] https://stackoverflow.com/a/61420852/5005936
Signed-off-by Carlo Marcelo Arenas Belón <[email protected]>
Quoted 6 lines · Eric Sunshine
Quoted 1 line · Carlo Marcelo Arenas Belón
git-credential-store: skip empty lines and comments from store
I don't see anything in this patch which makes it skip anything at all; it only introduces a new test.
my bad, forgot to fix the subject when rerolling; does something like:
credential-store: show "regression" when bogus credentials in file make sense for the reroll?
Carlo
PATCH v4 3/4 git-credential-store: fix (WIP)
In reply to Carlo Marcelo Arenas BelónFrom: Jonathan Nieder <[email protected]>
--- credential-store.c | 4 ++-- t/t0302-credential-store.sh | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
credential-store.c +2 -2
@@ -24,8 +24,8 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && +if (!credential_from_url_gently (&entry, line.buf,1 ) && + entry.username && entry.password &&credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {
t/t0302-credential-store.sh +1 -1
@@ -120,7 +120,7 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi test_must_be_empty "$HOME/.config/git/credentials" ' -test_expect_failure 'get: store file can contain empty/bogus lines' ' +test_expect_success 'get: store file can contain empty/bogus lines' ' test_write_lines "#comment" " " "" \ https://user:[email protected] >"$HOME/.git-credentials" && check fill store <<-\EOF
Signature · 1 line
2.26.2.569.g1d74ac4d14
Re: [PATCH v4 3/4] git-credential-store: fix (WIP)
Quoted 2 lines · Carlo Marcelo Arenas Belón
From: Jonathan Nieder <[email protected]>
Subject: git-credential-store: fix (WIP)
Um, what? Did you forget to squash this into the previous patch?
Quoted 1 line
No sign-off? (Jonathan's)
Re: [PATCH v4 3/4] git-credential-store: fix (WIP)
Quoted 6 lines · Eric Sunshine
Quoted 2 lines · Carlo Marcelo Arenas Belón
From: Jonathan Nieder <[email protected]>
Subject: git-credential-store: fix (WIP)Um, what? Did you forget to squash this into the previous patch?
no; was documenting his proposal from [email protected][1] with the hope (unsuccesfully) to save everyone time and easy testing by having a series ready to apply.
Quoted 3 lines
Quoted 1 line
No sign-off? (Jonathan's)
really my fault, but was expected as documented in:
https://lore.kernel.org/git/20200428112518.GA15981@Carlos-MBP/
sorry for the confussion and thanks for your review
Carlo
PATCH v4 4/4 credential-store: make sure there is no regression with missing scheme
In reply to Carlo Marcelo Arenas Belónc44088ecc4 (credential: treat URL without scheme as invalid, 2020-04-18) made scheme (AKA protocol) mandatory so make sure that still fails.
--- t/t0302-credential-store.sh | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
t/t0302-credential-store.sh +17 -0
@@ -135,4 +135,21 @@ test_expect_success 'get: store file can contain empty/bogus lines' ' EOF ' +test_expect_success 'get: ignore credentials without scheme' ' + echo "://user:[email protected]" >"$HOME/.git-credentials" && + check fill store <<-\EOF + protocol=https + host=example.com + -- + protocol=https + host=example.com + username=askpass-username + password=askpass-password + -- + askpass: Username for '\''https://example.com'\'': + askpass: Password for '\''https://[email protected]'\'': + -- + EOF +' + test_done
Signature · 1 line
2.26.2.569.g1d74ac4d14
Re: [PATCH v4 1/4] credential-store: document the file format a bit more
In reply to Carlo Marcelo Arenas BelónQuoted 11 lines · Carlo Marcelo Arenas Belón
Reading a malformed credential URL line and silently ignoring it does not mean that we promise to torelate and/or keep empty lines and "# commented" lines forever.
Some people seem to take anything that is not explicitly forbidden as allowed, but the world does not work that way.
Signed-off-by Junio C Hamano <[email protected]>---
Documentation/git-credential-store.txt stored on its own line as a URL like: · line 96
Quoted 3 lines
Documentation/git-credential-store.txt +3
-0@@ -94,6 +94,10 @@ stored on its own line as a URL like: +No other kinds of lines (e.g. empty lines or comment lines) are +allowed in the file, even though some may be silently ignored. Do +not view or edit the file with editors.
I suggest dropping the "even though some may be silently ignored" bit since it's both mysterious (providing no concrete information) and unnecessarily confusing since it flat out contradicts the earlier part of the sentence. The fact that the implementation has accidentally been loose in its parsing doesn't warrant introducing such ambiguity into the (newly-added) documentation.
Re: [PATCH v4 1/4] credential-store: document the file format a bit more
Quoted 9 lines · Eric Sunshine
Quoted 7 lines · Carlo Marcelo Arenas Belón
Reading a malformed credential URL line and silently ignoring it does not mean that we promise to torelate and/or keep empty lines and "# commented" lines forever.
Some people seem to take anything that is not explicitly forbidden as allowed, but the world does not work that way.
Quoted 14 lines
Documentation/git-credential-store.txt stored on its own line as a URL like: · line 96
Quoted 7 lines
Signed-off-by Junio C Hamano <[email protected]>---
Documentation/git-credential-store.txt +3
-0@@ -94,6 +94,10 @@ stored on its own line as a URL like: +No other kinds of lines (e.g. empty lines or comment lines) are +allowed in the file, even though some may be silently ignored. Do +not view or edit the file with editors.I suggest dropping the "even though some may be silently ignored" bit since it's both mysterious (providing no concrete information) and unnecessarily confusing since it flat out contradicts the earlier part of the sentence. The fact that the implementation has accidentally been loose in its parsing doesn't warrant introducing such ambiguity into the (newly-added) documentation.
I do not think it is an ambiguity. The phrase is there just to remind the readers that they are not allowed to take a loose implementation in the past as an excuse to throw in crufts and expect the result to continue working.
Re: [PATCH v4 1/4] credential-store: document the file format a bit more
In reply to Carlo Marcelo Arenas BelónQuoted 4 lines · Carlo Marcelo Arenas Belón
From: Junio C Hamano <[email protected]>
Reading a malformed credential URL line and silently ignoring it does not mean ...
I'd reuse the one I've already queued as 272281ef (credential-store:
document the file format a bit more, 2020-04-27) for this step.
Thanks.
-- >8 --
Subject: [PATCH] credential-store: document the file format a bit more
Reading a malformed credential URL line and silently ignoring it does not mean that we support empty lines and/or "# commented" lines forever. We should document it to avoid confusion.
--- Documentation/git-credential-store.txt | 4 ++++ 1 file changed, 4 insertions(+)
Documentation/git-credential-store.txt +4 -0
@@ -94,6 +94,10 @@ stored on its own line as a URL like: https://user:[email protected] ------------------------------ +No other kinds of lines (e.g. empty lines or comment lines) are +allowed in the file, even though some may be silently ignored. Do +not view or edit the file with editors. + When Git needs authentication for a particular URL context, credential-store will consider that context a pattern to match against each entry in the credentials file. If the protocol, hostname, and
Signature · 1 line
2.26.2-266-ge870325ee8
PATCH v5 1/1 credential-store: warn instead of fatal for bogus lines from store
In reply to Carlo Marcelo Arenas Belónwith the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
instead of doing a hard check for credentials, do a soft one and warn the user so any invalid entries could be corrected.
[1] https://stackoverflow.com/a/61420852/5005936
---
v5:
* q_to_tab this round, with a single echo to make sure empty line is covered, as that seems to be a popular issue
* rebase on top of jc/credential-store-file-format-doc
* implement a redacter for credentials to use on errora to avoid leaking passwordss
v4:
* use credential_from_url_gently instead as shown by Jonathan
* add documentation to clarify "comments" is not a supported feature v3:
* avoid using q_to_cr as suggested by Peff
* a more verbose commit message and slightly more complete documentation v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentation
Documentation/git-credential-store.txt | 12 +++++- credential-store.c | 51 ++++++++++++++++++++++---- t/t0302-credential-store.sh | 43 ++++++++++++++++++++++ 3 files changed, 96 insertions(+), 10 deletions(-)
Documentation/git-credential-store.txt +10 -2
@@ -95,8 +95,16 @@ https://user:[email protected] ------------------------------ No other kinds of lines (e.g. empty lines or comment lines) are -allowed in the file, even though some may be silently ignored. Do -not view or edit the file with editors. +allowed in the file, even though historically the parser was very +lenient and some might had been silently ignored. + +Do not edit the file with editors as it could compromise the validity +of your credentials by sometimes subtle formatting issues (like spaces) + +In cases where those formatting issues are detected during parsing a +warning indicating the line found will be printed to stderr so it can +be corrected at your earliest convenience, but the remaining valid +credentials will be used to try to find a match as described below. When Git needs authentication for a particular URL context, credential-store will consider that context a pattern to match against
credential-store.c +43 -8
71 lines not on this page · raw message
t/t0302-credential-store.sh +43 -0
47 lines not on this page · raw message
base-commit: 272281efcc18fcedd248597b8859f343cae1c5a0
Signature · 1 line
2.26.2.569.g1d74ac4d14
Re: [PATCH v5] credential-store: warn instead of fatal for bogus lines from store
credential-store.c line 19
Quoted 11 lines · Carlo Marcelo Arenas Belón
credential-store.c +11
-0+static char *redact_credential (const struct strbuf *line) +{ +struct strbuf redacted_line =STRBUF_INIT ; +char *at =strchr (line->buf,'@' ); +char *colon; +int redacted =0 ; + +if (at) { +strbuf_addf (&redacted_line,"%.*s" , + (int )(at - line->buf), line->buf); + colon =strrchr (redacted_line.buf,':' );
Just showing my ignorance, but ...
- Is the above strrchr() that forbids a colon in the password intended, or should it be strchr() that only forbids a colon in the username instead?
- Would it hurt to redact both username and password as sensitive? If not, it would certainly make it simpler to unconditionally:
int i;
for (i = 0; i < redacted_line.len; i++) {
if (redacted_line.buf[i] != ':')
redacted_line.buf[i] = 'x';
}Re: [PATCH v5] credential-store: warn instead of fatal for bogus lines from store
Quoted 15 lines · Junio C Hamano
credential-store.c line 19
Quoted 11 lines · Carlo Marcelo Arenas Belón
credential-store.c +11
-0+static char *redact_credential (const struct strbuf *line) +{ +struct strbuf redacted_line =STRBUF_INIT ; +char *at =strchr (line->buf,'@' ); +char *colon; +int redacted =0 ; + +if (at) { +strbuf_addf (&redacted_line,"%.*s" , + (int )(at - line->buf), line->buf); + colon =strrchr (redacted_line.buf,':' );Just showing my ignorance, but ...
Quoted 3 lines
- Is the above
strrchr()that forbids a colon in the password intended, or should it bestrchr()that only forbids a colon in the username instead?
neither; that ":" is the separator we put in there between username and password (unless it was edited out), as any original ":" in both is already urlencoded.
strrchr() was used in purpose to back only as much as it is needed from the string to hopefully find the beginning of the password.
indeed, most of the uglyness of the code comes from the fact it tries really hard to only redact the password so the rest of the credential is still useful for context and do so without changing the original line.
Quoted 8 lines
- Would it hurt to redact both username and password as sensitive? If not, it would certainly make it simpler to unconditionally:
int i; for (i = 0; i < redacted_line.len; i++) { if (redacted_line.buf[i] != ':') redacted_line.buf[i] = 'x'; }
that would leak the length of the password which might also be considered sensitive
IMHO if we are redacting both might as well not print anything and let the user find the offending credential by line number instead ;) after all I got a feeling most of those entries are empty lines or "comments".
for example with the current code merged in pu we get :
$ cat secretfile
://u:[email protected]
http://user:some%[email protected]
$ git credential-store --file secretfile get protocol=http
warning: secretfile:1 ignoring invalid credential:
warning: secretfile:2 ignoring invalid credential: ://u:<redacted>@example.com username=user
password=some:pass
Carlo
Re: [PATCH v5] credential-store: warn instead of fatal for bogus lines from store
Quoted 3 lines · Carlo Marcelo Arenas Belón
IMHO if we are redacting both might as well not print anything and let the user find the offending credential by line number instead ;) after all I got a feeling most of those entries are empty lines or "comments".
Very sensible.
v6 cover credential-store: prevent fatal errors
In reply to Carlo Marcelo Arenas Belónwith the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] as failing to parse as valid credentials.
document better the format for the credential file and make sure any failures are still handled gently by the new code to avoid regressions but make sure that any invalid credentials are flagged as such instead of being silently ignored.
ideally would like to add a test against the use of cert:// to make sure that is still well supported but don't know of any use case or if it is being used with store (AFAIK, it could and at least in theory it should still work fine with the added validations), hence sending this as an RFC.
also, there is no real reason to have 2 patches but I am keeping them separated because the first one should be AFAIK otherwise almost ready.
but considering that we are validating since v4 against a missing protocol and printing a warning since v5, it seems we should then fully avoid all those silently ignored issues as well (as shown in patch 2) which I hope we could squash and release together (if agreed)
[1] https://stackoverflow.com/a/61420852/5005936
Carlo Marcelo Arenas Belón (2):
credential-store: warn instead of fatal for bogus lines from store
credential-store: warn for any incomplete credentials instead of using
---
v6:
* get rid of redacter and only use line number for context
* make validation more strict to also catch incomplete credentials v5:
* q_to_tab this round, with a single echo to make sure empty line is covered, as that seems to be a popular issue
* rebase on top of jc/credential-store-file-format-doc
* implement a redacter for credentials to use on errors to avoid leaking passwordss
v4:
* use credential_from_url_gently instead as shown by Jonathan
* add documentation to clarify "comments" is not a supported feature v3:
* avoid using q_to_cr as suggested by Peff
* a more verbose commit message and slightly more complete documentation v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentation
Documentation/git-credential-store.txt | 12 +++- credential-store.c | 23 +++++--- t/t0302-credential-store.sh | 80 ++++++++++++++++++++++++++ 3 files changed, 105 insertions(+), 10 deletions(-)
base-commit: 272281efcc18fcedd248597b8859f343cae1c5a0
Signature · 1 line
2.26.2.569.g1d74ac4d14
RFC PATCH v6 1/2 credential-store: warn instead of fatal for bogus lines from store
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
instead of doing a hard check for credentials, do a soft one and warn the user so any invalid entries could be corrected.
[1] https://stackoverflow.com/a/61420852/5005936
--- Documentation/git-credential-store.txt | 12 ++++++-- credential-store.c | 22 +++++++++----- t/t0302-credential-store.sh | 42 ++++++++++++++++++++++++++ 3 files changed, 66 insertions(+), 10 deletions(-)
Documentation/git-credential-store.txt +10 -2
@@ -95,8 +95,16 @@ https://user:[email protected] ------------------------------ No other kinds of lines (e.g. empty lines or comment lines) are -allowed in the file, even though some may be silently ignored. Do -not view or edit the file with editors. +allowed in the file, even though historically the parser was very +lenient and some might had been silently ignored. + +Do not edit the file with editors as it could compromise the validity +of your credentials by sometimes subtle formatting issues (like spaces) + +In cases where those formatting issues are detected during parsing a +warning indicating the line found will be printed to stderr so it can +be corrected at your earliest convenience, but the remaining valid +credentials will be used to try to find a match as described below. When Git needs authentication for a particular URL context, credential-store will consider that context a pattern to match against
credential-store.c +14 -8
@@ -15,6 +15,7 @@ static int parse_credential_file(const char *fn,struct strbuf line =STRBUF_INIT ;struct credential entry =CREDENTIAL_INIT ;int found_credential =0 ; +int lineno =0 ; fh =fopen (fn,"r" );if (!fh) {
@@ -24,16 +25,21 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && -credential_match (c, &entry)) { - found_credential =1 ; -if (match_cb) { -match_cb (&entry); -break ; + lineno++; +if (!credential_from_url_gently (&entry, line.buf,1 )) { +if (entry.username && entry.password && +credential_match (c, &entry)) { + found_credential =1 ; +if (match_cb) { +match_cb (&entry); +break ; + } } } -else if (other_cb) +else +warning (_ ("ignoring invalid credential in %s:%d" ), + fn, lineno); +if (!found_credential && other_cb)other_cb (&line); }
t/t0302-credential-store.sh +42 -0
46 lines not on this page · raw message
Signature · 1 line
2.26.2.569.g1d74ac4d14
Re: [RFC PATCH v6 1/2] credential-store: warn instead of fatal for bogus lines from store
Quoted 6 lines · Carlo Marcelo Arenas Belón
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
instead of doing a hard check for credentials, do a soft one and warn the user so any invalid entries could be corrected.
Yeah, this version looks be best so far. No need to worry about redacting anything; we are dealing with folks who have edited the file, and they shouldn't have any trouble going to the line with a problem given an exact line number.
Quoted 5 lines
Documentation/git-credential-store.txt +0
-0
Documentation/git-credential-store.txt https://user:[email protected] · line 102
Quoted 10 lines
Documentation/git-credential-store.txt +5
-2@@ -95,8 +95,16 @@ https://user:[email protected] ------------------------------ No other kinds of lines (e.g. empty lines or comment lines) are -allowed in the file, even though some may be silently ignored. Do -not view or edit the file with editors. +allowed in the file, even though historically the parser was very +lenient and some might had been silently ignored. + +Do not edit the file with editors as it could compromise the validity +of your credentials by sometimes subtle formatting issues (like spaces)
I do not think dropping "view or" is warranted. There is no need to invite the risk of opening with the intention to only view and then end up saving a modification. In other words, do not encourage use of an *editor* in any way.
Documentation/git-credential-store.txt https://user:[email protected] · line 107
Quoted 4 lines
Documentation/git-credential-store.txt +4
-0+In cases where those formatting issues are detected during parsing a +warning indicating the line found will be printed to stderr so it can +be corrected at your earliest convenience, but the remaining valid +credentials will be used to try to find a match as described below.
I am often just as guilty, but the above four lines in a single sentence is hard to read, especially without minimum number of comma. With a comma after "during parsing", and another after "to stderr", might make it more readable, but at that point, it would become far more readable if we split them into two sentences.
Perhaps
An unparseable line is ignored, and a warning message points out the line number the problematic line appears in (you may want to delete them with an editor).
Quoted 8 lines
Documentation/git-credential-store.txt +7
-0while (strbuf_getline_lf(&line, fh) != EOF) { + lineno++; + if (!credential_from_url_gently(&entry, line.buf, 1)) { + if (entry.username && entry.password && + credential_match(c, &entry)) { + found_credential = 1; + if (match_cb) { + match_cb(&entry);
Documentation/git-credential-store.txt https://user:[email protected]
Quoted 10 lines
Documentation/git-credential-store.txt +6
-0+ break; + } } } + else + warning(_("ignoring invalid credential in %s:%d"), + fn, lineno); + if (!found_credential && other_cb) other_cb(&line); }
The above is harder to follow than necessary.
while (... != EOF) {
lineno++;
if (credential is not well formed) {
warning(_("ignoring..."));
} else if (the entry matches) {
found_credential = 1;
if (match_cb) {
match_cb(&entry);
break; /* stop at the first match */
}
continue;
}
if (other_cb)
other_cb(&line);
}may make the intention a bit clearer, with the loud "continue" inside.
(1) we give warning for malformed one; and
(2) we do not let other_cb touch a matching entry.
Re: [RFC PATCH v6 1/2] credential-store: warn instead of fatal for bogus lines from store
Quoted 21 lines · Junio C Hamano
Quoted 8 lines
credential-store.c +7
-0while (strbuf_getline_lf (&line, fh) !=EOF ) { + lineno++; +if (!credential_from_url_gently (&entry, line.buf,1 )) { +if (entry.username && entry.password && +credential_match (c, &entry)) { + found_credential =1 ; +if (match_cb) { +match_cb (&entry);credential-store.c static int parse_credential_file(const char *fn, · line 44
Quoted 10 lines
credential-store.c +6
-0+break ; + } } } +else +warning (_ ("ignoring invalid credential in %s:%d" ), + fn, lineno); +if (!found_credential && other_cb)other_cb (&line); }The above is harder to follow than necessary.
Quoted 3 lines
while (... != EOF) {
lineno++;
if (credential is not well formed) {After reading your [2/2], I think this part deserves a bit of explanation. By "is not well formed", what I mean is not just credential_from_url_gently() returns non-zero, but also user/pass are specified. And your step [2/2] may make the condition for a line to be "well formed" even stricter by requiring at least one of host or path, and that would also belong to this test.
Quoted 2 lines
warning(_("ignoring..."));
} else if (the entry matches) {And the "entry matches" test here would only be "what does credential_match() say?"
Quoted 16 lines
found_credential = 1; if (match_cb) { match_cb(&entry); break; /* stop at the first match */ } continue; } if (other_cb) other_cb(&line); }may make the intention a bit clearer, with the loud "continue" inside.
(1) we give warning for malformed one; and
(2) we do not letother_cbtouch a matching entry.
RFC PATCH v6 2/2 credential-store: warn for any incomplete credentials instead of using
In reply to Carlo Marcelo Arenas Belónoriginally any credential found was tried for matching as far as it had a username and password, but that resulted in fatal errors as the rules were harden.
now that we have a way to report malformed credentials, use it to notify the user when username/password was missing, instead of just silently skipping.
do the same for credentials that are missing host (or had one that is empty) or that are missing a path (for supporting cert://) as well.
--- credential-store.c | 7 ++++--- t/t0302-credential-store.sh | 38 +++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 3 deletions(-)
credential-store.c +4 -3
@@ -26,9 +26,10 @@ static int parse_credential_file(const char *fn,while (strbuf_getline_lf (&line, fh) !=EOF ) { lineno++; -if (!credential_from_url_gently (&entry, line.buf,1 )) { -if (entry.username && entry.password && -credential_match (c, &entry)) { +if (!credential_from_url_gently (&entry, line.buf,1 ) && + ((entry.host && *entry.host) || entry.path) && + entry.username && entry.password) { +if (credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {match_cb (&entry);
t/t0302-credential-store.sh +38 -0
@@ -139,6 +139,44 @@ test_expect_success 'get: credentials without scheme are invalid' ' test_i18ngrep "ignoring invalid credential" stderr ' +test_expect_success 'get: credentials without valid host/path are invalid' ' + echo "https://user:pass@" >"$HOME/.git-credentials" && + cat >expect-stdout <<-\STDOUT && + protocol=https + host=example.com + username=askpass-username + password=askpass-password + STDOUT + test_config credential.helper store && + git credential fill <<-\EOF >stdout 2>stderr && + protocol=https + host=example.com + EOF + test_cmp expect-stdout stdout && + grep "askpass: Username for '\''https://example.com'\'':" stderr && + grep "askpass: Password for '\''https://[email protected]'\'':" stderr && + test_i18ngrep "ignoring invalid credential" stderr +' + +test_expect_success 'get: credentials without username/password are invalid' ' + echo "https://[email protected]" >"$HOME/.git-credentials" && + cat >expect-stdout <<-\STDOUT && + protocol=https + host=example.com + username=askpass-username + password=askpass-password + STDOUT + test_config credential.helper store && + git credential fill <<-\EOF >stdout 2>stderr && + protocol=https + host=example.com + EOF + test_cmp expect-stdout stdout && + grep "askpass: Username for '\''https://example.com'\'':" stderr && + grep "askpass: Password for '\''https://[email protected]'\'':" stderr && + test_i18ngrep "ignoring invalid credential" stderr +' + test_expect_success 'get: store file can contain empty/bogus lines' ' echo "" > "$HOME/.git-credentials" && q_to_tab <<-\CONFIG >>"$HOME/.git-credentials" &&
Signature · 1 line
2.26.2.569.g1d74ac4d14
Re: [RFC PATCH v6 2/2] credential-store: warn for any incomplete credentials instead of using
Quoted 3 lines · Carlo Marcelo Arenas Belón
originally any credential found was tried for matching as far as it had a username and password, but that resulted in fatal errors as the rules were harden.
harden -> hardened
Quoted 3 lines
now that we have a way to report malformed credentials, use it to notify the user when username/password was missing, instead of just silently skipping.
Sorry, but isn't that what happend already in the previous step? What are you ordering the codebase (after applying the previous stpe) do further? It already is "using it to notify the user when username and/or password is missing".
Quoted 2 lines
do the same for credentials that are missing host (or had one that is empty) or that are missing a path (for supporting cert://) as well.
While the intention may be a good one ...
Quoted 11 lines
Signed-off-by Carlo Marcelo Arenas Belón <[email protected]>--- credential-store.c | 7 ++++--- t/t0302-credential-store.sh | 38 +++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 3 deletions(-)credential-store.c +0
-0
credential-store.c static int parse_credential_file(const char *fn, · line 32
Quoted 10 lines
credential-store.c +4
-3@@ -26,9 +26,10 @@ static int parse_credential_file(const char *fn,while (strbuf_getline_lf (&line, fh) !=EOF ) { lineno++; -if (!credential_from_url_gently (&entry, line.buf,1 )) { -if (entry.username && entry.password && -credential_match (c, &entry)) { +if (!credential_from_url_gently (&entry, line.buf,1 ) && + ((entry.host && *entry.host) || entry.path) && + entry.username && entry.password) { +if (credential_match (c, &entry)) {
... this makes the code even harder to follow than it already was after the previous step. In the previous step, at least it was clear that we'd catch *all* the well-formed/parseable input will come into the first if () {...} block, but with the extra logic, that is no longer true. Even a line that is well formed may be bypassed from matching and will be fed to "else" side.
Re: [RFC PATCH v6 2/2] credential-store: warn for any incomplete credentials instead of usingy
Quoted 8 lines · Junio C Hamano
Quoted 3 lines · Carlo Marcelo Arenas Belón
originally any credential found was tried for matching as far as it had a username and password, but that resulted in fatal errors as the rules were harden.
harden -> hardened
Quoted 8 lines
Quoted 3 lines
now that we have a way to report malformed credentials, use it to notify the user when username/password was missing, instead of just silently skipping.
Sorry, but isn't that what happend already in the previous step? What are you ordering the codebase (after applying the previous stpe) do further? It already is "using it to notify the user when username and/or password is missing".
not sure I follow, but the current code (as well as the one after patch 1) just silently ignores any credential that was missing username and password since the _gently version of the call will only really fail for a missing protocol.
so patch1 will notify ONLY(*) when the credential is so malformed that we can't figure out which protocol to use (like empty lines and comments)
(*) it will also fail if we have a '\n' in one of the components but that is hopefully not relevant here.
Quoted 6 lines
Quoted 6 lines
credential-store.c +0
-0@@ -26,9 +26,10 @@ static int parse_credential_file(const char *fn,
Quoted 16 lines
credential-store.c static int parse_credential_file(const char *fn, · line 32
Quoted 9 lines
credential-store.c +4
-3while (strbuf_getline_lf (&line, fh) !=EOF ) { lineno++; -if (!credential_from_url_gently (&entry, line.buf,1 )) { -if (entry.username && entry.password && -credential_match (c, &entry)) { +if (!credential_from_url_gently (&entry, line.buf,1 ) && + ((entry.host && *entry.host) || entry.path) && + entry.username && entry.password) { +if (credential_match (c, &entry)) {... this makes the code even harder to follow than it already was after the previous step. In the previous step, at least it was clear that we'd catch *all* the well-formed/parseable input will come into the first if () {...} block, but with the extra logic, that is no longer true. Even a line that is well formed may be bypassed from matching and will be fed to "else" side.
I think it will be clearer with the reformatting you suggested and that I agree is needed, but wanted to make sure first that we agreed on the direction.
FWIW the test cases show the format of the lines affected and while they are parseable enough that were processed they were obviously not valid, specially considering the updated rules.
maybe we should make the _gently version less gently instead?, so that callers will be able to know they have an incomplete credential for matching after calling it without having to do their own check?
if the later, I am affraid this will be a far bigger change, which is why I would rather duplicate the rule checking for at least the first iteration.
Carlo
Re: [RFC PATCH v6 2/2] credential-store: warn for any incomplete credentials instead of usingy
Quoted 9 lines · Carlo Marcelo Arenas Belón
Quoted 4 lines
Sorry, but isn't that what happend already in the previous step? What are you ordering the codebase (after applying the previous stpe) do further? It already is "using it to notify the user when username and/or password is missing".
not sure I follow, but the current code (as well as the one after patch 1) just silently ignores any credential that was missing username and password since the
_gentlyversion of the call will only really fail for a missing protocol.
Ah, I misread the intention of [1/1]. While I was rewriting the if/else structure inside the loop, somehow I thought that we want to warn when from_url_gently() fails or user/pass are missing, and in this step, we are adding to the "warning-worthy" set an entry that does not have either host or path.
Sorry about the confusion.
PATCH v6 1/1 credential-store: warn instead of fatal for bogus lines from store
In reply to Carlo Marcelo Arenas Belónwith the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
instead of doing a hard check for credentials, do a soft one and warn the user so any invalid entries could be corrected.
make sure that the credential to use is complete before calling credential_match by confirming it has all fields set as expected by the updated rules.
[1] https://stackoverflow.com/a/61420852/5005936
---
v6:
* get rid of redacter and only use line number for context in warning
* make validation more strict to also catch incomplete credentials
* reorder check as suggested by Junio
v5:
* q_to_tab this round, with a single echo to make sure empty line is covered, as that seems to be a popular issue
* rebase on top of jc/credential-store-file-format-doc
* implement a redacter for credentials to use on errors to avoid leaking passwords
v4:
* use credential_from_url_gently instead as shown by Jonathan
* add documentation to clarify "comments" is not a supported feature v3:
* avoid using q_to_cr as suggested by Peff
* a more verbose commit message and slightly more complete documentation v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentation
Documentation/git-credential-store.txt | 11 +++- credential-store.c | 26 +++++++-- t/t0302-credential-store.sh | 80 ++++++++++++++++++++++++++ 3 files changed, 111 insertions(+), 6 deletions(-)
Documentation/git-credential-store.txt +9 -2
@@ -95,8 +95,15 @@ https://user:[email protected] ------------------------------ No other kinds of lines (e.g. empty lines or comment lines) are -allowed in the file, even though some may be silently ignored. Do -not view or edit the file with editors. +allowed in the file, even though historically the parser was very +lenient and some might had been silently ignored. + +Do not view or edit the file with editors as it could compromise the +validity of your credentials by sometimes subtle formatting issues, +like spaces, line wrapping or text encoding. + +An unparseable or otherwise invalid line is ignored, and a warning +message points out the problematic line number and file it appears in. When Git needs authentication for a particular URL context, credential-store will consider that context a pattern to match against
credential-store.c +22 -4
@@ -6,6 +6,15 @@static struct lock_file credential_lock; +/* +* entry->protocol comes validated from credential_from_url_gently +*/ +static int valid_credential (struct credential *entry) +{ +return (entry->username && entry->password && + ((entry->host && *entry->host) || entry->path)); +} + static int parse_credential_file(const char *fn, struct credential *c, void (*match_cb)(struct credential *),
@@ -15,6 +24,7 @@ static int parse_credential_file(const char *fn,struct strbuf line =STRBUF_INIT ;struct credential entry =CREDENTIAL_INIT ;int found_credential =0 ; +int lineno =0 ; fh =fopen (fn,"r" );if (!fh) {
@@ -24,16 +34,24 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && -credential_match (c, &entry)) { + lineno++; + +if (credential_from_url_gently (&entry, line.buf,1 ) || + !valid_credential (&entry)) { +warning (_ ("%s:%d: ignoring invalid credential" ), + fn, lineno); + } +else if (credential_match (c, &entry)) + { found_credential =1 ;if (match_cb) {match_cb (&entry);break ; } +continue ; } -else if (other_cb) + +if (other_cb)other_cb (&line); }
t/t0302-credential-store.sh +80 -0
84 lines not on this page · raw message
base-commit: 272281efcc18fcedd248597b8859f343cae1c5a0
Signature · 1 line
2.26.2.569.g1d74ac4d14
Re: [PATCH v6] credential-store: warn instead of fatal for bogus lines from store
Quoted 3 lines · Carlo Marcelo Arenas Belón
with the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
with -> With
Quoted 2 lines
instead of doing a hard check for credentials, do a soft one and warn the user so any invalid entries could be corrected.
instead -> instead
Quoted 3 lines
make sure that the credential to use is complete before calling
credential_matchby confirming it has all fields set as expected by the updated rules.
make -> Make
Quoted 6 lines
Documentation/git-credential-store.txt +0
-0@@ -95,8 +95,15 @@ https://user:[email protected] ------------------------------
Documentation/git-credential-store.txt https://user:[email protected] · line 103
Quoted 10 lines
Documentation/git-credential-store.txt +6
-2No other kinds of lines (e.g. empty lines or comment lines) are -allowed in the file, even though some may be silently ignored. Do -not view or edit the file with editors. +allowed in the file, even though historically the parser was very +lenient and some might had been silently ignored. + +Do not view or edit the file with editors as it could compromise the +validity of your credentials by sometimes subtle formatting issues, +like spaces, line wrapping or text encoding.
I do not think dropping "view or" is justifiable. There is no need to invite the risk of opening with the intention to only view and then end up saving a modification. In other words, do not encourage use of an *editor* in any way.
Documentation/git-credential-store.txt https://user:[email protected] · line 106
Quoted 3 lines
Documentation/git-credential-store.txt +3
-0+ +An unparseable or otherwise invalid line is ignored, and a warning +message points out the problematic line number and file it appears in.
OK. You didn't want to tell them they can remove the problematic line as a whole with their editor?
credential-store.c line 11
Quoted 11 lines
credential-store.c +3
-0@@ -6,6 +6,15 @@static struct lock_file credential_lock; +/* +* entry->protocol comes validated from credential_from_url_gently +*/
Sure, but wouldn't it be simpler to do without such a comment and check it also here, just as a belt-and-suspender safety?
Quoted 17 lines
credential-store.c +6
-0+static int valid_credential (struct credential *entry) +{ +return (entry->username && entry->password && + ((entry->host && *entry->host) || entry->path)); +}@@ -15,6 +24,7 @@ static int parse_credential_file(const char *fn,struct strbuf line =STRBUF_INIT ;struct credential entry =CREDENTIAL_INIT ;int found_credential =0 ; +int lineno =0 ; fh =fopen (fn,"r" );if (!fh) {@@ -24,16 +34,24 @@ static int parse_credential_file(const char *fn, }
credential-store.c static int parse_credential_file(const char *fn, · line 45
Quoted 10 lines
credential-store.c +9
-0while (strbuf_getline_lf (&line, fh) !=EOF ) { + lineno++; + +if (credential_from_url_gently (&entry, line.buf,1 ) || + !valid_credential (&entry)) { +warning (_ ("%s:%d: ignoring invalid credential" ), + fn, lineno); + } +else if (credential_match (c, &entry)) + {
Style: "} else if (...) {" on a single line.
credential-store.c static int parse_credential_file(const char *fn, · line 56
Quoted 11 lines
credential-store.c +3
-0found_credential =1 ;if (match_cb) {match_cb (&entry);break ; } +continue ; } + +if (other_cb)other_cb (&line); }
This got vastly easier to read, thanks to the additional helper. Looking really good.
Quoted 14 lines
t/t0302-credential-store.sh +6
-0@@ -120,4 +120,84 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi test_must_be_empty "$HOME/.config/git/credentials" ' +test_expect_success 'get: credentials without scheme are invalid' ' + echo "://user:[email protected]" >"$HOME/.git-credentials" && + cat >expect-stdout <<-\STDOUT && + protocol=https + host=example.com + username=askpass-username
t/t0302-credential-store.sh test_expect_success 'erase: erase matching credentials from both xdg and home fi · line 138
Quoted 10 lines
t/t0302-credential-store.sh +10
-0+ password=askpass-password + STDOUT + test_config credential.helper store && + git credential fill <<-\EOF >stdout 2>stderr && + protocol=https + host=example.com + EOF + test_cmp expect-stdout stdout && + grep "askpass: Username for '\''https://example.com'\'':" stderr && + grep "askpass: Password for '\''https://[email protected]'\'':" stderr &&
These messages are passed from credential.c::credential_ask_one() to git_prompt() without any i18n, so use of a bare "grep" is good.
t/t0302-credential-store.sh test_expect_success 'erase: erase matching credentials from both xdg and home fi · line 139
Quoted 1 line
t/t0302-credential-store.sh +1
-0+ test_i18ngrep "ignoring invalid credential" stderr
And this is new message inside _(), so i18ngrep is good.
Nice to see such an attention to the details.
t/t0302-credential-store.sh test_expect_success 'erase: erase matching credentials from both xdg and home fi · line 200
Quoted 8 lines
t/t0302-credential-store.sh +8
-0+ test_config credential.helper store && + git credential fill <<-\EOF >stdout 2>stderr && + protocol=https + host=example.com + EOF + test_cmp expect-stdout stdout && + test_i18ngrep "ignoring invalid credential" stderr && + test_line_count = 3 stderr
The "ignoring invalid credential" message could be translated into two or more lines, but I think that is worrying too much about theoretical possibility, so checking line count would probably be sufficient.
Thanks.
Re: [PATCH v6] credential-store: warn instead of fatal for bogus lines from store
Quoted 5 lines · Junio C Hamano
Documentation/git-credential-store.txt https://user:[email protected] · line 103
Quoted 3 lines
Documentation/git-credential-store.txt +3
-0+Do not view or edit the file with editors as it could compromise the +validity of your credentials by sometimes subtle formatting issues, +like spaces, line wrapping or text encoding.I do not think dropping "view or" is justifiable....
Sorry, this comment is now stale.
Quoted 5 lines
Documentation/git-credential-store.txt https://user:[email protected] · line 106
Quoted 2 lines
Documentation/git-credential-store.txt +2
-0+An unparseable or otherwise invalid line is ignored, and a warning +message points out the problematic line number and file it appears in.OK. You didn't want to tell them they can remove the problematic line as a whole with their editor?
This one I do not care too deeply either way. Probably it is obvious to the readers that they can remove the lines (otherwise there is no good reason to give warnings in the first place).
Re: [PATCH v6] credential-store: warn instead of fatal for bogus lines from store
In reply to Junio C HamanoQuoted 4 lines · Junio C Hamano
Quoted 2 lines
instead of doing a hard check for credentials, do a soft one and warn the user so any invalid entries could be corrected.
instead -> instead
;)
Quoted 10 lines
I do not think dropping "view or" is justifiable. There is no need to invite the risk of opening with the intention to only view and then end up saving a modification. In other words, do not encourage use of an *editor* in any way.
Documentation/git-credential-store.txt https://user:[email protected] · line 106
Quoted 2 lines
Documentation/git-credential-store.txt +2
-0+An unparseable or otherwise invalid line is ignored, and a warning +message points out the problematic line number and file it appears in.OK. You didn't want to tell them they can remove the problematic line as a whole with their editor?
someone said "do not encourage use of an *editor* in any way" just a few lines before ;)
Quoted 5 lines
Quoted 5 lines
t/t0302-credential-store.sh +5
-0+ test_config credential.helper store && + git credential fill <<-\EOF >stdout 2>stderr && + protocol=https + host=example.com + EOF
Quoted 8 lines
t/t0302-credential-store.sh test_expect_success 'erase: erase matching credentials from both xdg and home fi · line 200
Quoted 3 lines
t/t0302-credential-store.sh +3
-0+ test_cmp expect-stdout stdout && + test_i18ngrep "ignoring invalid credential" stderr && + test_line_count = 3 stderrThe "ignoring invalid credential" message could be translated into two or more lines, but I think that is worrying too much about theoretical possibility, so checking line count would probably be sufficient.
yes, and specially considering that if I even ended up adding a -c option to test_i18ngrep as I intended originally we will still the same issue.
what is the preferred way to do something like this, or is it better to add a check for each line formatting issue this is handling?
Carlo
PATCH v7 1/1 credential-store: warn instead of fatal for bogus lines from store
In reply to Carlo Marcelo Arenas BelónWith the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
Instead of doing a hard check for credentials, do a soft one and warn the user so any invalid entries could be corrected.
Make sure that the credential to use is complete before calling credential_match by confirming it has all fields set as expected by the updated rules.
[1] https://stackoverflow.com/a/61420852/5005936
---
v7:
* check for protocol in helper as suggested by Junio
v6:
* get rid of redacter and only use line number for context in warning
* make validation more strict to also catch incomplete credentials
* reorder check as suggested by Junio
v5:
* q_to_tab this round, with a single echo to make sure empty line is covered, as that seems to be a popular issue
* rebase on top of jc/credential-store-file-format-doc
* implement a redacter for credentials to use on errors to avoid leaking passwords
v4:
* use credential_from_url_gently instead as shown by Jonathan
* add documentation to clarify "comments" is not a supported feature v3:
* avoid using q_to_cr as suggested by Peff
* a more verbose commit message and slightly more complete documentation v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentation
Documentation/git-credential-store.txt | 11 +++- credential-store.c | 22 +++++-- t/t0302-credential-store.sh | 80 ++++++++++++++++++++++++++ 3 files changed, 107 insertions(+), 6 deletions(-)
Documentation/git-credential-store.txt +9 -2
@@ -95,8 +95,15 @@ https://user:[email protected] ------------------------------ No other kinds of lines (e.g. empty lines or comment lines) are -allowed in the file, even though some may be silently ignored. Do -not view or edit the file with editors. +allowed in the file, even though historically the parser was very +lenient and some might had been silently ignored. + +Do not view or edit the file with editors as it could compromise the +validity of your credentials by sometimes subtle formatting issues, +like spaces, line wrapping or text encoding. + +An unparseable or otherwise invalid line is ignored, and a warning +message points out the problematic line number and file it appears in. When Git needs authentication for a particular URL context, credential-store will consider that context a pattern to match against
credential-store.c +18 -4
@@ -6,6 +6,13 @@static struct lock_file credential_lock; +static int valid_credential (struct credential *entry) +{ +return (entry->username && entry->password && + entry->protocol && *entry->protocol && + ((entry->host && *entry->host) || entry->path)); +} + static int parse_credential_file(const char *fn, struct credential *c, void (*match_cb)(struct credential *),
@@ -15,6 +22,7 @@ static int parse_credential_file(const char *fn,struct strbuf line =STRBUF_INIT ;struct credential entry =CREDENTIAL_INIT ;int found_credential =0 ; +int lineno =0 ; fh =fopen (fn,"r" );if (!fh) {
@@ -24,16 +32,22 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && -credential_match (c, &entry)) { + lineno++; + +if (credential_from_url_gently (&entry, line.buf,1 ) || + !valid_credential (&entry)) { +warning (_ ("%s:%d: ignoring invalid credential" ), + fn, lineno); + }else if (credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {match_cb (&entry);break ; } +continue ; } -else if (other_cb) + +if (other_cb)other_cb (&line); }
t/t0302-credential-store.sh +80 -0
84 lines not on this page · raw message
base-commit: 272281efcc18fcedd248597b8859f343cae1c5a0
Signature · 1 line
2.26.2.569.g1d74ac4d14
PATCH v8 1/1 credential-store: warn instead of fatal for bogus lines from store
With the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
Instead of doing a hard check for credentials, do a soft one and therefore avoid the reported fatal error.
Warn the user indicating the filename and line number so any invalid entries could be corrected but continue parsing until a match is found or all valid credentials are processed.
Make sure that the credential that we will use to match is complete by confirming it has all fields set as expected by the updated rules.
[1] https://stackoverflow.com/a/61420852/5005936
---
v8:
* only warn during get operations as otherwise line number might be incorrect
v7:
* check for protocol in helper as suggested by Junio
v6:
* get rid of redacter and only use line number for context in warning
* make validation more strict to also catch incomplete credentials
* reorder check as suggested by Junio
v5:
* q_to_tab this round, with a single echo to make sure empty line is covered, as that seems to be a popular issue
* rebase on top of jc/credential-store-file-format-doc
* implement a redacter for credentials to use on errors to avoid leaking passwords
v4:
* use credential_from_url_gently instead as shown by Jonathan
* add documentation to clarify "comments" is not a supported feature v3:
* avoid using q_to_cr as suggested by Peff
* a more verbose commit message and slightly more complete documentation v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentation
Documentation/git-credential-store.txt | 11 +++- credential-store.c | 31 +++++++-- t/t0302-credential-store.sh | 89 ++++++++++++++++++++++++++ 3 files changed, 123 insertions(+), 8 deletions(-)
Documentation/git-credential-store.txt +9 -2
@@ -95,8 +95,15 @@ https://user:[email protected] ------------------------------ No other kinds of lines (e.g. empty lines or comment lines) are -allowed in the file, even though some may be silently ignored. Do -not view or edit the file with editors. +allowed in the file, even though historically the parser was very +lenient and some might had been silently ignored. + +Do not view or edit the file with editors as it could compromise the +validity of your credentials by sometimes subtle formatting issues, +like spaces, line wrapping or text encoding. + +An unparseable or otherwise invalid line is ignored, and a warning +message points out the problematic file and line number it appears in. When Git needs authentication for a particular URL context, credential-store will consider that context a pattern to match against
credential-store.c +25 -6
@@ -4,10 +4,20 @@#include "string-list.h" #include "parse-options.h" +#define PARSE_VERBOSE 0x01 +static struct lock_file credential_lock; +static int valid_credential (struct credential *entry) +{ +return (entry->username && entry->password && + entry->protocol && *entry->protocol && + ((entry->host && *entry->host) || entry->path)); +} +static int parse_credential_file (const char *fn,struct credential *c, +int flags,void (*match_cb)(struct credential *),void (*other_cb)(struct strbuf *)) {
@@ -15,6 +25,7 @@ static int parse_credential_file(const char *fn,struct strbuf line =STRBUF_INIT ;struct credential entry =CREDENTIAL_INIT ;int found_credential =0 ; +int lineno =0 ; fh =fopen (fn,"r" );if (!fh) {
@@ -24,16 +35,23 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && -credential_match (c, &entry)) { + lineno++; + +if (credential_from_url_gently (&entry, line.buf,1 ) || + !valid_credential (&entry)) { +if (flags &PARSE_VERBOSE ) +warning (_ ("%s:%d: ignoring invalid credential" ), + fn, lineno); + }else if (credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {match_cb (&entry);break ; } +continue ; } -else if (other_cb) + +if (other_cb)other_cb (&line); }
@@ -62,7 +80,7 @@ static void rewrite_credential_file(const char *fn, struct credential *c,die_errno ("unable to get credential storage lock" );if (extra)print_line (extra); -parse_credential_file (fn, c,NULL , print_line); +parse_credential_file (fn, c,0 ,NULL , print_line);if (commit_lock_file (&credential_lock) <0 )die_errno ("unable to write credential store" ); }
@@ -139,7 +157,8 @@ static void lookup_credential(const struct string_list *fns, struct credential *struct string_list_item *fn;for_each_string_list_item (fn, fns) -if (parse_credential_file (fn->string, c, print_entry,NULL )) +if (parse_credential_file (fn->string, c,PARSE_VERBOSE , + print_entry,NULL ))return ;/* Found credential */ }
t/t0302-credential-store.sh +89 -0
93 lines not on this page · raw message
base-commit: 272281efcc18fcedd248597b8859f343cae1c5a0
Signature · 1 line
2.26.2.2.g9c647e0d3b.dirty
PATCH v9 1/1 credential-store: warn instead of fatal for bogus lines from store
In reply to Carlo Marcelo Arenas BelónWith the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
Instead of doing a hard check for credentials, do a soft one and therefore avoid the reported fatal error.
Warn the user indicating the filename and line number so any invalid entries could be corrected but continue parsing until a match is found or all valid credentials are processed.
Make sure that the credential that we will use to match is complete by confirming it has all fields set as expected by the updated rules.
[1] https://stackoverflow.com/a/61420852/5005936
---
v9:
* use strbuf_getline() instead to better handle files with CRLF v8:
* only warn during get operations as otherwise line number might be incorrect
v7:
* check for protocol in helper as suggested by Junio
v6:
* get rid of redacter and only use line number for context in warning
* make validation more strict to also catch incomplete credentials
* reorder check as suggested by Junio
v5:
* q_to_tab this round, with a single echo to make sure empty line is covered, as that seems to be a popular issue
* rebase on top of jc/credential-store-file-format-doc
* implement a redacter for credentials to use on errors to avoid leaking passwords
v4:
* use credential_from_url_gently instead as shown by Jonathan
* add documentation to clarify "comments" is not a supported feature v3:
* avoid using q_to_cr as suggested by Peff
* a more verbose commit message and slightly more complete documentation v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentation
Documentation/git-credential-store.txt | 11 +++- credential-store.c | 33 ++++++++-- t/t0302-credential-store.sh | 90 +++++++++++++++++++++++++- 3 files changed, 124 insertions(+), 10 deletions(-)
Documentation/git-credential-store.txt +9 -2
@@ -95,8 +95,15 @@ https://user:[email protected] ------------------------------ No other kinds of lines (e.g. empty lines or comment lines) are -allowed in the file, even though some may be silently ignored. Do -not view or edit the file with editors. +allowed in the file, even though historically the parser was very +lenient and some might had been silently ignored. + +Do not view or edit the file with editors as it could compromise the +validity of your credentials by sometimes subtle formatting issues, +like spaces, line wrapping or text encoding. + +An unparseable or otherwise invalid line is ignored, and a warning +message points out the problematic file and line number it appears in. When Git needs authentication for a particular URL context, credential-store will consider that context a pattern to match against
credential-store.c +26 -7
@@ -4,10 +4,20 @@#include "string-list.h" #include "parse-options.h" +#define PARSE_VERBOSE 0x01 +static struct lock_file credential_lock; +static int valid_credential (struct credential *entry) +{ +return (entry->username && entry->password && + entry->protocol && *entry->protocol && + ((entry->host && *entry->host) || entry->path)); +} +static int parse_credential_file (const char *fn,struct credential *c, +int flags,void (*match_cb)(struct credential *),void (*other_cb)(struct strbuf *)) {
@@ -15,6 +25,7 @@ static int parse_credential_file(const char *fn,struct strbuf line =STRBUF_INIT ;struct credential entry =CREDENTIAL_INIT ;int found_credential =0 ; +int lineno =0 ; fh =fopen (fn,"r" );if (!fh) {
@@ -23,17 +34,24 @@ static int parse_credential_file(const char *fn,return found_credential; } -while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && -credential_match (c, &entry)) { +while (strbuf_getline (&line, fh) !=EOF ) { + lineno++; + +if (credential_from_url_gently (&entry, line.buf,1 ) || + !valid_credential (&entry)) { +if (flags &PARSE_VERBOSE ) +warning (_ ("%s:%d: ignoring invalid credential" ), + fn, lineno); + }else if (credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {match_cb (&entry);break ; } +continue ; } -else if (other_cb) + +if (other_cb)other_cb (&line); }
@@ -62,7 +80,7 @@ static void rewrite_credential_file(const char *fn, struct credential *c,die_errno ("unable to get credential storage lock" );if (extra)print_line (extra); -parse_credential_file (fn, c,NULL , print_line); +parse_credential_file (fn, c,0 ,NULL , print_line);if (commit_lock_file (&credential_lock) <0 )die_errno ("unable to write credential store" ); }
@@ -139,7 +157,8 @@ static void lookup_credential(const struct string_list *fns, struct credential *struct string_list_item *fn;for_each_string_list_item (fn, fns) -if (parse_credential_file (fn->string, c, print_entry,NULL )) +if (parse_credential_file (fn->string, c,PARSE_VERBOSE , + print_entry,NULL ))return ;/* Found credential */ }
t/t0302-credential-store.sh +89 -1
100 lines not on this page · raw message
base-commit: 272281efcc18fcedd248597b8859f343cae1c5a0
Signature · 1 line
2.26.2.2.gf219a08795
Re: [PATCH v9] credential-store: warn instead of fatal for bogus lines from store
Quoted 14 lines · Carlo Marcelo Arenas Belón
With the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
Instead of doing a hard check for credentials, do a soft one and therefore avoid the reported fatal error.
Warn the user indicating the filename and line number so any invalid entries could be corrected but continue parsing until a match is found or all valid credentials are processed.
Make sure that the credential that we will use to match is complete by confirming it has all fields set as expected by the updated rules.
Quoted 7 lines
[1] https://stackoverflow.com/a/61420852/5005936
Reported-by Dirk <[email protected]>Helped-by Eric Sunshine <[email protected]>Based-on-patch-by Jonathan Nieder <[email protected]>Signed-off-by Carlo Marcelo Arenas Belón <[email protected]>---
Looks good.
Quoted 8 lines
credential-store.c +0
-0@@ -4,10 +4,20 @@#include "string-list.h" #include "parse-options.h"
credential-store.c line 16
Quoted 10 lines
credential-store.c +8
-0+#define PARSE_VERBOSE 0x01 +static struct lock_file credential_lock; +static int valid_credential (struct credential *entry) +{ +return (entry->username && entry->password && + entry->protocol && *entry->protocol && + ((entry->host && *entry->host) || entry->path)); +}
OK.
Quoted 15 lines
credential-store.c +2
-0static int parse_credential_file (const char *fn,struct credential *c, +int flags,void (*match_cb)(struct credential *),void (*other_cb)(struct strbuf *)) {@@ -15,6 +25,7 @@ static int parse_credential_file(const char *fn,struct strbuf line =STRBUF_INIT ;struct credential entry =CREDENTIAL_INIT ;int found_credential =0 ; +int lineno =0 ; fh =fopen (fn,"r" );if (!fh) {
credential-store.c static int parse_credential_file(const char *fn, · line 37
Quoted 8 lines
credential-store.c +1
-4@@ -23,17 +34,24 @@ static int parse_credential_file(const char *fn,return found_credential; } -while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && -credential_match (c, &entry)) { +while (strbuf_getline (&line, fh) !=EOF ) {
Hmph, I probably have missed some discussion that happened in the last few days, but from the use of _lf() in the original, I sense that it is very much deliberate that the file format is designed to be LF delimited records, *not* a text file in which each line is terminated with some end-of-line marker that is platform dependent. IOW, an explicit use of _lf() shouts, at least to me, that we want a sequence of LF terminated records even on Windows and Macs.
So, I am not sure why this change is desirable.
credential-store.c static int parse_credential_file(const char *fn, · line 41
Quoted 4 lines
credential-store.c +4
-0+ lineno++; + +if (credential_from_url_gently (&entry, line.buf,1 ) || + !valid_credential (&entry)) {
OK, so we read a line, fed it to the parser, and if we had trouble parsing or the line did not have enough credential material, we discard and warn (when told to).
credential-store.c static int parse_credential_file(const char *fn, · line 51
Quoted 10 lines
credential-store.c +5
-0+if (flags &PARSE_VERBOSE ) +warning (_ ("%s:%d: ignoring invalid credential" ), + fn, lineno); + }else if (credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {match_cb (&entry);break ; } +continue ;
And if the credential material on a good line matches, we remember we saw a match. If there is match callback, we call it and leave the loop to return from the function. Otherwise we go to the next line.
credential-store.c static int parse_credential_file(const char *fn, · line 55
Quoted 4 lines
credential-store.c +2
-0} + +if (other_cb)other_cb (&line);
A malformed/underspecified line and an unmatched line is fed to the other callback.
credential-store.c static void rewrite_credential_file(const char *fn, struct credential *c, · line 83
Quoted 8 lines
credential-store.c +1
-1}@@ -62,7 +80,7 @@ static void rewrite_credential_file(const char *fn, struct credential *c,die_errno ("unable to get credential storage lock" );if (extra)print_line (extra); -parse_credential_file (fn, c,NULL , print_line); +parse_credential_file (fn, c,0 ,NULL , print_line);
This helper function is called from two places.
- In store_credential_file, we write the credential we were asked to store to a strbuf, and then call this function. The function first writes that new credential and then calls the parse helper without PARSE_VERBOSE. We do not give any match callback, and other callback is to just print the line read from the credential file. So, the final output would be the new credential, followed by the current contents of the credential file except for the records that match the one we are storing. We do this without warning about malformed entries at all.
- In remove_credential, we go over multiple files and copy the lines in each of them, except the ones that match the credential we are given. We also do this without warning about malformed entries at all.
credential-store.c static void lookup_credential(const struct string_list *fns, struct credential * · line 163
Quoted 12 lines
credential-store.c +2
-1if (commit_lock_file (&credential_lock) <0 )die_errno ("unable to write credential store" ); }@@ -139,7 +157,8 @@ static void lookup_credential(const struct string_list *fns, struct credential *struct string_list_item *fn;for_each_string_list_item (fn, fns) -if (parse_credential_file (fn->string, c, print_entry,NULL )) +if (parse_credential_file (fn->string, c,PARSE_VERBOSE , + print_entry,NULL ))return ;/* Found credential */ }
This is triggered by the "get" operation. We read until we hit the entry we are looking for, at that point the match-callback will print out "username=... / password=..." lines and we ignore the remainder of the file. While we look for the first matching credential, we do warn about malformed or underspecified lines that we are ignoring, but because we leave the loop once a matching line is found, bad lines that come after it won't be even looked at to be warned.
Validating and warning about bad entries is *not* the main purpose of the "credential-store" program, so I fully agree with the design of the "get" part. I am not so sure about the other two operations (i.e. "store" and "erase") that do scan all the entries and has chance to warn about bad ones, though (note: I am not saying that we should parse verbosely---it is just that I do not know why you chose not to and I am not convinced that it is a good idea not to warn).
The tests looked good.
Thanks.
Re: [PATCH v9] credential-store: warn instead of fatal for bogus lines from store
Junio C Hamano <[email protected]> writes:
[part of the commit log message]
Quoted 3 lines
Quoted 3 lines
Warn the user indicating the filename and line number so any invalid entries could be corrected but continue parsing until a match is found or all valid credentials are processed.
We should say "Do so only during the 'get' operation; give up giving any warning for 'erase' or 'store' operation, as keeping track of the line number of the input file, while having to issue a warning that has the line number of the output file, is too hard for us" or something like that here.
I suspect that it might not be "too hard", but we'd need to rename other_cb to a more specific name first and limit the possibility of repurposing parse_credential_file().
For example, other_cb can become "copy_cb", we declare that the function works in two (and no other) modes, one is to look-up (which is read-only so we report the input line number in our warning messges) and the other is to copy-out-with-filtering (which will leave a file that is different from the input, so we report the output line number).
To support the copy-out-wiht-filtering mode, we pass the starting line number into the function (i.e. 'store' may store a new line, so the first line copied out to the file may be the second line in the output), and count the output lines there:
if (other_cb) {
lines++;
other_cb(&line);
}and of course we won't increment lines++ when we saw a match in the copy-out-with-filtering mode. In look-up mode (i.e. copy_cb == NULL), we count the input lines just like your code.
But I suspect that it may not be worth doing. But if we decide not to do so, we should document why we chose (not) to in the commit log message.
Quoted 7 lines
Validating and warning about bad entries is *not* the main purpose of the "credential-store" program, so I fully agree with the design of the "get" part. I am not so sure about the other two operations (i.e. "store" and "erase") that do scan all the entries and has chance to warn about bad ones, though (note: I am not saying that we should parse verbosely---it is just that I do not know why you chose not to and I am not convinced that it is a good idea not to warn).
I re-read the incremental log for v8 and found your explanation.
Thanks.
Re: [PATCH v9] credential-store: warn instead of fatal for bogus lines from store
In reply to Junio C HamanoQuoted 5 lines · Junio C Hamano
Quoted 4 lines · Carlo Marcelo Arenas Belón
credential-store.c +0
-0@@ -23,17 +34,24 @@ static int parse_credential_file(const char *fn,return found_credential; }
Quoted 13 lines
credential-store.c static int parse_credential_file(const char *fn, · line 37
Quoted 5 lines
credential-store.c +1
-4-while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && -credential_match (c, &entry)) { +while (strbuf_getline (&line, fh) !=EOF ) {Hmph, I probably have missed some discussion that happened in the last few days, but from the use of
_lf()in the original, I sense that it is very much deliberate that the file format is designed to be LF delimited records, *not* a text file in which each line is terminated with some end-of-line marker that is platform dependent. IOW, an explicit use of_lf()shouts, at least to me, that we want a sequence of LF terminated records even on Windows and Macs.
apologize for the confusion, the only discussion was the one I had with myself late at night when I realized that specific corruption was not being detected, and might be related to issues that seemed to be common[1]
the problem is that practically speaking, if users in Windows and Macs edited the file they "might" had saved lines with the wrong line ending (*) (part of the reason I added a warning about "encoding" to the documentation), and those are difficult to "see" as invalid.
using the non _lf() version ensures any trailing '\r' character would get removed from the credential and allow us to use them, instead of silently failing.
originally I added a explicit check for it, which I assume you would prefer but ..
Quoted 1 line
So, I am not sure why this change is desirable.
to give users immediate relief from what seems to be a commonly seen issue.
IMHO after we get this released out and they see the updated documentation explaining the risks, and some learn how to fix their credential files ( and hopefully use an editor that lets them see the problem); we could then add also a detection for this edge case and go back to _lf()
I also had to admit I might had overreacted, as I was testing before v8 with a very corrupted file and was seeing warnings twice on each operation and somehow even got myself into the original fatal, which I had to admit I can't replicate now after some needed rest.
Carlo
[1] https://github.com/desktop/desktop/issues/9597
(*) textedit, which comes with macOS doesn't even write an EOF record when
creating files, for example, and that behaviour seems to be common for most other native editors but a credential line WITHOUT line ending works
Re: [PATCH v9] credential-store: warn instead of fatal for bogus lines from store
Quoted 8 lines · Carlo Marcelo Arenas Belón
the problem is that practically speaking, if users in Windows and Macs edited the file they "might" had saved lines with the wrong line ending (*) (part of the reason I added a warning about "encoding" to the documentation), and those are difficult to "see" as invalid.
using the non
_lf()version ensures any trailing '\r' character would get removed from the credential and allow us to use them, instead of silently failing.
You are forgetting why we are fixing credential-store, aren't you?
It is primarily to help those who damaged their files by editing, and introducing cruft that cannot be parsed, from a stricter parsing introduced recently in 2.17.4 and above. Without the fix, they cannot operate with the store they already have.
Now, think about those users who saved their file, after adding CR at the end of each line, but didn't do any other edit (like adding a blank line or "# comment"). It may have happened 3 years ago. What did they see from the system back then? It may have happened 3 minutes ago. What would they see with the stricter parser now?
With or without the recent parser change, they would have seen that these lines no longer match the URLs they wanted to match, but the credential store does not die on them for malformed lines, no?
In other words, the stricter parsing did nothing to these users.
In fact, those users who added CR at the end of each line 3 years ago may have even depended on the disappearance of these entries for all these years. Maybe lines that record their ancient passwords for sites are still buried in the later parts of the file, with CR, but doing no harm because these lines do not match anything. These users may have changed their password since then and wrote new records with "credential store", and these new records are stored without CR at the end of the line, so they match the URLs.
By using the non _lf() variant, you are suddenly resurrecting these old records that the users thought are already gone and have been causing no harm to them. Do we know that resurrecting these old records is a good thing to do? I don't. For example, once the user decides to "sort" the file (after all, we are talking about users who edit the file, so we cannot assume they won't do so), they would end up with duplicate records that record two passwords to the same site and they cannot tell which one is current, as you even lost the CR at the end of line that would have told you which ones are broken.
In short, you wouldn't know what ramification it has by suddenly using non _lf() variant. And it has nothing to do with the fix we are trying to make.
When fixing something, it is tempting to see if you can cover a bit more area with the same change. But you should make it a habit to stick to the absolute minimum first for safety. When contemplating to step out of the absolute minimum, you need to think twice to see if that extra "fix" really fixes, or if it potentially harms.
And I am not convinced that turning CRLF into LF in this case is a good change. In any case, it certainly does not belong to the same commit as the one that fixes the fallout from stricter parser introduced in 2.17.4 and above.
Thanks.
Re: [PATCH v9] credential-store: warn instead of fatal for bogus lines from store
Quoted 8 lines · Junio C Hamano
Quoted 5 lines · Carlo Marcelo Arenas Belón
the problem is that practically speaking, if users in Windows and Macs edited the file they "might" had saved lines with the wrong line ending (*) (part of the reason I added a warning about "encoding" to the documentation), and those are difficult to "see" as invalid.
Quoted 5 lines
Quoted 3 lines
using the non
_lf()version ensures any trailing '\r' character would get removed from the credential and allow us to use them, instead of silently failing.You are forgetting why we are fixing credential-store, aren't you?
indeed, and thanks for the clarification.
you are correct this was a bad idea and is better to warn them instead (even if only
during get operations and for the lines that were processed) than my "solution".
FWIW though, my change wasn't going to change the file on disk but only allow the line to be processed.
gave up already on sneakily "fixing" corruption issues after Peff called me on it
and yes, another version you will never see had a PARSER_TRIM flag added ;)
Carlo
PS. should we really do the warn even in store/erase operations as a followup or is the warning not useful as is, and probably then all operations should be quiet (as Jonathan suggested originally?) and we could do warn (and maybe fix) in a different change (maybe adding a fsck command of sorts)?
Quoted 5 lines
When fixing something, it is tempting to see if you can cover a bit more area with the same change. But you should make it a habit to stick to the absolute minimum first for safety. When contemplating to step out of the absolute minimum, you need to think twice to see if that extra "fix" really fixes, or if it potentially harms.
Re: [PATCH v9] credential-store: warn instead of fatal for bogus lines from store
Quoted 5 lines · Carlo Arenas
PS. should we really do the warn even in store/erase operations as a followup or is the warning not useful as is, and probably then all operations should be quiet (as Jonathan suggested originally?) and we could do warn (and maybe fix) in a different change (maybe adding a fsck command of sorts)?
Yeah, I think I like the "no warning, just ignore" much better. The implementation I suspect would become a lot simpler, right?
Thanks for thinking one extra step.
Re: [PATCH v9] credential-store: warn instead of fatal for bogus lines from store
Quoted 10 lines · Junio C Hamano
Quoted 5 lines · Carlo Arenas
PS. should we really do the warn even in store/erase operations as a followup or is the warning not useful as is, and probably then all operations should be quiet (as Jonathan suggested originally?) and we could do warn (and maybe fix) in a different change (maybe adding a fsck command of sorts)?
Yeah, I think I like the "no warning, just ignore" much better. The implementation I suspect would become a lot simpler, right?
sure, it will be basically the version that Jonathan suggested[1] on top of yours and that I included in v4 but that was missing a SOB.
Jonathan,
would you want to submit it formally?, IMHO better if with base (from pu):272281efcc (credential-store: document the file format a bit more, 2020-04-27)
it might be still worth including some of the test cases, so I could do it instead too; but let me know what would be your preference otherwise, as they will need to be wrapped, probably better than on what became my confusing attempt to save everyone's time with the v4 series[2], and that was made obsolete by Junio's creation of jc/credential-store-file-format-doc.
Carlo
[1] https://lore.kernel.org/git/[email protected]/
[2] https://lore.kernel.org/git/[email protected]/
Re: [PATCH v9] credential-store: warn instead of fatal for bogus lines from store
Quoted 2 lines · Carlo Marcelo Arenas Belón
attempt to save everyone's time with the v4 series[2], and that was made obsolete by Junio's creation of jc/credential-store-file-format-doc.
I actually discarded that "file format only" patch after you included in your series. One fewer patch I have to keep track of is always better ;-)
Thanks.
RFC PATCH v10 1/1 credential-store: warn/ignore for bogus lines from store file
In reply to Carlo Marcelo Arenas BelónWith the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
Instead of doing a hard check for credentials, do a soft one and therefore avoid the reported fatal error.
Warn the user indicating the filename and line number during get, for all entries that were processed, but continue parsing until a match is found or all valid credentials are processed.
Warning during store and erase operations has been suppressed for now by adding a PARSER_VERBOSE flag that is only enabled during get operations.
Make sure that the credential that we will use to match is complete by confirming it has all fields set as expected by the updated rules.
[1] https://stackoverflow.com/a/61420852/5005936
---
v10:
* insted of trying to use broken credentials in DOS format, make sure they are flagged as broken
v9:
* use strbuf_getline() instead to better handle files with CRLF v8:
* only warn during get operations as otherwise line number might be incorrect
v7:
* check for protocol in helper as suggested by Junio
v6:
* get rid of redacter and only use line number for context in warning
* make validation more strict to also catch incomplete credentials
* reorder check as suggested by Junio
v5:
* q_to_tab this round, with a single echo to make sure empty line is covered, as that seems to be a popular issue
* rebase on top of jc/credential-store-file-format-doc
* implement a redacter for credentials to use on errors to avoid leaking passwords
v4:
* use credential_from_url_gently instead as shown by Jonathan
* add documentation to clarify "comments" is not a supported feature v3:
* avoid using q_to_cr as suggested by Peff
* a more verbose commit message and slightly more complete documentation v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentation
Documentation/git-credential-store.txt | 18 +++- credential-store.c | 32 ++++++-- t/t0302-credential-store.sh | 109 ++++++++++++++++++++++++- 3 files changed, 150 insertions(+), 9 deletions(-)
Documentation/git-credential-store.txt +16 -2
@@ -95,8 +95,22 @@ https://user:[email protected] ------------------------------ No other kinds of lines (e.g. empty lines or comment lines) are -allowed in the file, even though some may be silently ignored. Do -not view or edit the file with editors. +allowed in the file, even though historically the parser was very +lenient and some might had been silently ignored. + +Do not view or edit the file with editors as it could compromise the +validity of your credentials by sometimes subtle formatting issues, +like spaces, line wrapping or text encoding. + +An unparseable or otherwise invalid line is ignored, and a warning +message points out the problematic file and line number it appears in. + +Note that because of deficiencies of the current implementation, these +warnings will be only generated for the entries that were processed +while reading the credential files during a get operation, and so they +are not comprehensive, and will require you to use an editor (or +better a sed/awk/perl one liner) to edit the credential file and remove +them. When Git needs authentication for a particular URL context, credential-store will consider that context a pattern to match against
credential-store.c +26 -6
@@ -4,10 +4,20 @@#include "string-list.h" #include "parse-options.h" +#define PARSE_VERBOSE 0x01 +static struct lock_file credential_lock; +static int valid_credential (struct credential *entry) +{ +return (entry->username && entry->password && + entry->protocol && *entry->protocol && + ((entry->host && *entry->host) || entry->path)); +} +static int parse_credential_file (const char *fn,struct credential *c, +int flags,void (*match_cb)(struct credential *),void (*other_cb)(struct strbuf *)) {
@@ -15,6 +25,7 @@ static int parse_credential_file(const char *fn,struct strbuf line =STRBUF_INIT ;struct credential entry =CREDENTIAL_INIT ;int found_credential =0 ; +int lineno =0 ; fh =fopen (fn,"r" );if (!fh) {
@@ -24,16 +35,24 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && -credential_match (c, &entry)) { + lineno++; + +if (strchr (line.buf,'\r' ) || +credential_from_url_gently (&entry, line.buf,1 ) || + !valid_credential (&entry)) { +if (flags &PARSE_VERBOSE ) +warning (_ ("%s:%d: ignoring invalid credential" ), + fn, lineno); + }else if (credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {match_cb (&entry);break ; } +continue ; } -else if (other_cb) + +if (other_cb)other_cb (&line); }
@@ -62,7 +81,7 @@ static void rewrite_credential_file(const char *fn, struct credential *c,die_errno ("unable to get credential storage lock" );if (extra)print_line (extra); -parse_credential_file (fn, c,NULL , print_line); +parse_credential_file (fn, c,0 ,NULL , print_line);if (commit_lock_file (&credential_lock) <0 )die_errno ("unable to write credential store" ); }
@@ -139,7 +158,8 @@ static void lookup_credential(const struct string_list *fns, struct credential *struct string_list_item *fn;for_each_string_list_item (fn, fns) -if (parse_credential_file (fn->string, c, print_entry,NULL )) +if (parse_credential_file (fn->string, c,PARSE_VERBOSE , + print_entry,NULL ))return ;/* Found credential */ }
t/t0302-credential-store.sh +108 -1
119 lines not on this page · raw message
base-commit: 272281efcc18fcedd248597b8859f343cae1c5a0
Signature · 1 line
2.26.2.2.gf219a08795
[RFC PATCH v10 2/1] credential-store: warn also for store and erase
done in a hacky way and only as a POC, but at least works, if we are still thinking that warning should be at least comprehensive for the first iteration.
didn't change the callback as you suggested but I think this doesn't look that bad either.
squash for final v10 or ..?
--- Documentation/git-credential-store.txt | 7 ------- credential-store.c | 19 +++++++++---------- t/t0302-credential-store.sh | 20 +++++++++++++++++--- 3 files changed, 26 insertions(+), 20 deletions(-)
Documentation/git-credential-store.txt +0 -7
@@ -105,13 +105,6 @@ like spaces, line wrapping or text encoding. An unparseable or otherwise invalid line is ignored, and a warning message points out the problematic file and line number it appears in. -Note that because of deficiencies of the current implementation, these -warnings will be only generated for the entries that were processed -while reading the credential files during a get operation, and so they -are not comprehensive, and will require you to use an editor (or -better a sed/awk/perl one liner) to edit the credential file and remove -them. - When Git needs authentication for a particular URL context, credential-store will consider that context a pattern to match against each entry in the credentials file. If the protocol, hostname, and
credential-store.c +9 -10
@@ -4,8 +4,6 @@#include "string-list.h" #include "parse-options.h" -#define PARSE_VERBOSE 0x01 -static struct lock_file credential_lock;static int valid_credential (struct credential *entry)
@@ -17,7 +15,7 @@ static int valid_credential(struct credential *entry)static int parse_credential_file (const char *fn,struct credential *c, -int flags, +int initial_line,void (*match_cb)(struct credential *),void (*other_cb)(struct strbuf *)) {
@@ -25,7 +23,7 @@ static int parse_credential_file(const char *fn,struct strbuf line =STRBUF_INIT ;struct credential entry =CREDENTIAL_INIT ;int found_credential =0 ; -int lineno =0 ; +int lineno = initial_line; fh =fopen (fn,"r" );if (!fh) {
@@ -40,8 +38,7 @@ static int parse_credential_file(const char *fn,if (strchr (line.buf,'\r' ) ||credential_from_url_gently (&entry, line.buf,1 ) || !valid_credential (&entry)) { -if (flags &PARSE_VERBOSE ) -warning (_ ("%s:%d: ignoring invalid credential" ), +warning (_ ("%s:%d: ignoring invalid credential" ), fn, lineno); }else if (credential_match (c, &entry)) { found_credential =1 ;
@@ -77,11 +74,14 @@ static void print_line(struct strbuf *buf)static void rewrite_credential_file (const char *fn,struct credential *c,struct strbuf *extra) { +int initial_line =0 ;if (hold_lock_file_for_update (&credential_lock, fn,0 ) <0 )die_errno ("unable to get credential storage lock" ); -if (extra) +if (extra) {print_line (extra); -parse_credential_file (fn, c,0 ,NULL , print_line); + initial_line++; + } +parse_credential_file (fn, c, initial_line,NULL , print_line);if (commit_lock_file (&credential_lock) <0 )die_errno ("unable to write credential store" ); }
@@ -158,8 +158,7 @@ static void lookup_credential(const struct string_list *fns, struct credential *struct string_list_item *fn;for_each_string_list_item (fn, fns) -if (parse_credential_file (fn->string, c,PARSE_VERBOSE , - print_entry,NULL )) +if (parse_credential_file (fn->string, c,0 , print_entry,NULL ))return ;/* Found credential */ }
t/t0302-credential-store.sh +17 -3
@@ -218,13 +218,27 @@ test_expect_success 'get: store file can contain empty/bogus lines' ' test_line_count = 3 stderr ' -test_expect_success 'store: store succeeds silently in corrupted file' ' +test_expect_success 'store: succeeds in corrupted file but correctly warn' ' echo "#comment" >"$HOME/.git-credentials" && - check approve store <<-\EOF && + test_config credential.helper store && + git credential approve <<-\EOF && url=https://user:[email protected] EOF grep "https://user:[email protected]" "$HOME/.git-credentials" && - test_must_be_empty stderr + test_i18ngrep "git-credentials:2: ignoring invalid credential" stderr +' + +test_expect_success 'erase: succeeds in corrupted file but correctly warn' ' + cat <<-\CONFIG >"$HOME/.git-credentials" && + #comment + https://user:[email protected] + CONFIG + test_config credential.helper store && + git credential reject <<-\EOF && + url=https://example.com + EOF + ! grep "https://user:[email protected]" "$HOME/.git-credentials" && + test_i18ngrep "git-credentials:1: ignoring invalid credential" stderr ' test_done
Signature · 1 line
2.26.2.3.g90bd2f7136
Re: [RFC PATCH v10 2/1] credential-store: warn also for store and erase
Quoted 3 lines · Carlo Marcelo Arenas Belón
done in a hacky way and only as a POC, but at least works, if we are still thinking that warning should be at least comprehensive for the first iteration.
...
Quoted 10 lines
credential-store.c +1
-1@@ -25,7 +23,7 @@ static int parse_credential_file(const char *fn,struct strbuf line =STRBUF_INIT ;struct credential entry =CREDENTIAL_INIT ;int found_credential =0 ; -int lineno =0 ; +int lineno = initial_line; fh =fopen (fn,"r" );if (!fh) {
credential-store.c static int parse_credential_file(const char *fn, · line 44
Quoted 9 lines
credential-store.c +1
-2@@ -40,8 +38,7 @@ static int parse_credential_file(const char *fn,if (strchr (line.buf,'\r' ) ||credential_from_url_gently (&entry, line.buf,1 ) || !valid_credential (&entry)) { -if (flags &PARSE_VERBOSE ) -warning (_ ("%s:%d: ignoring invalid credential" ), +warning (_ ("%s:%d: ignoring invalid credential" ), fn, lineno); }else if (credential_match (c, &entry)) { found_credential =1 ;
I am not sure how this would correctly count when "store" and "erase" is used. During "store", the initial_line may let us correctly count the new line, and as long as we keep copying, our output line number would stay to be correct, as we increment line by one every time we read in, but as soon as we see the older version of the credential record this "store" is updating, i.e. when the credential_match() reports a match, we will discard the line we read by hitting the "continue" and not calling other_cb(). We increment line, but we don't output a line when that happens, so any warning after that point will be out of sync, no? And "erase" is the same deal. It behaves the same way as "store" except that "erase" does not emit a replacement record at the beginning. As soon as the matching record from the input is dropped, our line number will be out of sync.
In any case, I think it probably is a good idea to start with a version that ignores without warning, and I had an impression that you were on the same page from reading your response to v9 review, so...
PATCH v10 1/1 credential-store: ignore bogus lines from store file
In reply to Carlo Marcelo Arenas BelónWith the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
Instead of doing a hard check for credentials, do a soft one and therefore avoid the reported fatal error.
As a special case, flag files with CRLF endings as invalid early to prevent current problems in credential_from_url_gently() with handling of '\r' in the host.
While at it add tests for all known corruptions that are currently ignored to keep track of them and avoid the risk of regressions.
[1] https://stackoverflow.com/a/61420852/5005936
---
v10:
* go back to v4 but with better testing and commit message
* make sure broken CR characters are ignored early
v9:
* use strbuf_getline() instead to better handle files with CRLF v8:
* only warn during get operations as otherwise line number might be incorrect
v7:
* check for protocol in helper as suggested by Junio
v6:
* get rid of redacter and only use line number for context in warning
* make validation more strict to also catch incomplete credentials
* reorder check as suggested by Junio
v5:
* q_to_tab this round, with a single echo to make sure empty line is covered, as that seems to be a popular issue
* rebase on top of jc/credential-store-file-format-doc
* implement a redacter for credentials to use on errors to avoid leaking passwords
v4:
* use credential_from_url_gently instead as shown by Jonathan
* add documentation to clarify "comments" is not a supported feature v3:
* avoid using q_to_cr as suggested by Peff
* a more verbose commit message and slightly more complete documentation v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentation
credential-store.c | 5 ++-- t/t0302-credential-store.sh | 60 ++++++++++++++++++++++++++++++++++++- 2 files changed, 62 insertions(+), 3 deletions(-)
credential-store.c +3 -2
@@ -24,8 +24,9 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && +if (strchr (line.buf,'\r' ) ==NULL && + !credential_from_url_gently (&entry, line.buf,1 ) && + entry.username && entry.password &&credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {
t/t0302-credential-store.sh +59 -1
70 lines not on this page · raw message
base-commit: 49458fb91d61461938881559ce23abbb1a2f8c35
Signature · 1 line
2.26.2.686.gfaf46a9ccd
Re: [PATCH v10] credential-store: ignore bogus lines from store file
Quoted 10 lines · Carlo Marcelo Arenas Belón
With the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
Instead of doing a hard check for credentials, do a soft one and therefore avoid the reported fatal error.
As a special case, flag files with CRLF endings as invalid early to prevent current problems in
credential_from_url_gently()with handling of '\r' in the host.
I do not think it hurts to silently ignore a line that ends with CR, but only because I do not think credential_from_url_gently() would not match such a line when asked to match something without complaining.
In other words, isn't the new "!strchr() &&" in the condition a no-op?
Quoted 6 lines
credential-store.c +0
-0@@ -24,8 +24,9 @@ static int parse_credential_file(const char *fn, }
credential-store.c static int parse_credential_file(const char *fn, · line 32
Quoted 10 lines
credential-store.c +3
-2while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && +if (strchr (line.buf,'\r' ) ==NULL && + !credential_from_url_gently (&entry, line.buf,1 ) && + entry.username && entry.password &&credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {
In any case, among the ones we discussed, this probably has the least chance of unintended regression, I would think (with or without the added "!strchr() &&" check), so let's queue it and quickly merge it down thru 'next' to 'master'.
Quoted 28 lines
t/t0302-credential-store.sh +12
-1@@ -107,7 +107,6 @@ test_expect_success 'store: if both xdg and home files exist, only store in home test_must_be_empty "$HOME/.config/git/credentials" ' - test_expect_success 'erase: erase matching credentials from both xdg and home files' ' echo "https://home-user:[email protected]" >"$HOME/.git-credentials" && mkdir -p "$HOME/.config/git" &&@@ -120,4 +119,63 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi test_must_be_empty "$HOME/.config/git/credentials" ' +invalid_credential_test() { + test_expect_success 'get: ignore credentials without $1 as invalid' ' + echo "$2" >"$HOME/.git-credentials" && + check fill store <<-\EOF + protocol=https + host=example.com + -- + protocol=https + host=example.com + username=askpass-username + password=askpass-password + --
t/t0302-credential-store.sh test_expect_success 'erase: erase matching credentials from both xdg and home fi · line 143
Quoted 10 lines
t/t0302-credential-store.sh +10
-0+ askpass: Username for '\''https://example.com'\'': + askpass: Password for '\''https://[email protected]'\'': + -- + EOF + ' +} + +invalid_credential_test "scheme" ://user:[email protected] +invalid_credential_test "valid host/path" https://user:pass@ +invalid_credential_test "username/password" https://[email protected]
These are quite clear to see. Nicely done.
Quoted 6 lines
t/t0302-credential-store.sh +6
-0+test_expect_success 'get: credentials with DOS line endings are invalid' ' + printf "https://user:[email protected]\r\n" >"$HOME/.git-credentials" && + check fill store <<-\EOF + protocol=https + host=example.com + --
t/t0302-credential-store.sh test_expect_success 'erase: erase matching credentials from both xdg and home fi · line 160
Quoted 10 lines
t/t0302-credential-store.sh +10
-0+ protocol=https + host=example.com + username=askpass-username + password=askpass-password + -- + askpass: Username for '\''https://example.com'\'': + askpass: Password for '\''https://[email protected]'\'': + -- + EOF +'
What I am curious is if anything breaks around this test if you lost the extra "!strchr() &&" check. I suspect that this test will pass.
Quoted 22 lines
t/t0302-credential-store.sh +19
-0+test_expect_success 'get: store file can contain empty/bogus lines' ' + echo "" >"$HOME/.git-credentials" && + q_to_tab <<-\CREDENTIAL >>"$HOME/.git-credentials" && + #comment + Q + https://user:[email protected] + CREDENTIAL + check fill store <<-\EOF + protocol=https + host=example.com + -- + protocol=https + host=example.com + username=user + password=pass + -- + EOF +' + test_donebase-commit:
49458fb91d61461938881559ce23abbb1a2f8c35
Re: [PATCH v10] credential-store: ignore bogus lines from store file
Quoted 9 lines · Junio C Hamano
Quoted 7 lines · Carlo Marcelo Arenas Belón
With the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
Instead of doing a hard check for credentials, do a soft one and therefore avoid the reported fatal error.
Quoted 8 lines
Quoted 3 lines
As a special case, flag files with CRLF endings as invalid early to prevent current problems in
credential_from_url_gently()with handling of '\r' in the host.I do not think it hurts to silently ignore a line that ends with CR, but only because I do not think
credential_from_url_gently()would not match such a line when asked to match something without complaining.
for a credential like the one in the testcase (meaning no url), it will append \r to the hostname, which would cause havoc if that credential is printed (meaning you will end up without a host line) and be back in the die() in credential_apply()
Quoted 2 lines
In other words, isn't the new "!strchr() &&" in the condition a no-op?
you are correct that it will be unlikely (but not imposible) to get an embedded CR from the other side to match, which is what I want to address in the next patchset.
IMHO adding the proposed early check gives us space to fix the other issues at our own leasure and it is meant to be gone eventually.
Quoted 7 lines
Quoted 7 lines
credential-store.c +0
-0@@ -24,8 +24,9 @@ static int parse_credential_file(const char *fn, }
Quoted 14 lines
credential-store.c static int parse_credential_file(const char *fn, · line 32
Quoted 9 lines
credential-store.c +3
-2while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && +if (strchr (line.buf,'\r' ) ==NULL && + !credential_from_url_gently (&entry, line.buf,1 ) && + entry.username && entry.password &&credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {In any case, among the ones we discussed, this probably has the least chance of unintended regression, I would think (with or without the added "!strchr() &&" check), so let's queue it and quickly merge it down thru 'next' to 'master'.
considering the only line that I wrote was the strchr and the other one was written by Jonathan and reviewed by Peff I definitly agree.
don't forget this is also a good candidate for maint (most likely all the way to maint-2.17)
Carlo
Re: [PATCH v10] credential-store: ignore bogus lines from store file
In reply to Junio C HamanoQuoted 3 lines · Junio C Hamano
What I am curious is if anything breaks around this test if you lost the extra "!strchr() &&" check. I suspect that this test will pass.
correct
and with the strchr I am introducing a regression (compared against 2.26.0) in cases where the '\r' gets appended to the path and therefore gets ignored for matching (unless useHttpPath is true, which is not the default)
will add 2 more test cases to cover those, and guess we are going to 11 :(
Carlo
PS. thanks for all your patience and reviewing
Re: [PATCH v10] credential-store: ignore bogus lines from store file
Quoted 9 lines · Carlo Marcelo Arenas Belón
and with the strchr I am introducing a regression (compared against 2.26.0) in cases where the '\r' gets appended to the path and therefore gets ignored for matching (unless useHttpPath is true, which is not the default)
will add 2 more test cases to cover those, and guess we are going to 11 :(
Carlo
PS. thanks for all your patience and reviewing
Among those 11, I also am responsible for a few while we ere pursuing the approach to warn X-<.
Thanks for _your_ patience.
Re: [PATCH v10] credential-store: ignore bogus lines from store file
In reply to Junio C HamanoQuoted 8 lines · Junio C Hamano
Quoted 3 lines
As a special case, flag files with CRLF endings as invalid early to prevent current problems in
credential_from_url_gently()with handling of '\r' in the host.I do not think it hurts to silently ignore a line that ends with CR, but only because I do not think
credential_from_url_gently()would not match such a line when asked to match something without complaining.
I wondered if we might hit a case where the CR ends up in the path, like:
$ printf 'https://user:[email protected]/\r\n' >creds $ echo url=https://example.com/ | git credential-store --file=creds get username=user password=pass
because the path is parsed as missing in the incoming pattern (and thus we match any path, even "\r").
But credential-store would never write such a path in the first place. Even with the trailing slash on an incoming URL, it will write:
without a slash at all (and thus any inserted CR would be part of the hostname). So somebody would have to have inserted it themselves, or have turned useHTTPPath on (in which case we _would_ complain on the matching side, too, because we'd try matching the path with a CR in it).
I think it's reasonable to assume that any CR would have been a problem even in older versions.
-Peff
Re: [PATCH v10] credential-store: ignore bogus lines from store file
In reply to Carlo Marcelo Arenas BelónDscho,
this was tested[1] in windows and was able to reproduce the bogus issues with credential store in macOS using t0302.50
[1] https://github.com/carenas/git/actions/runs/93858709
Junio,
will followup with a more comprehensive fix for the problems with '\r' which will obiously need also changes in credential.c, but that IMHO might be more of a prerequisite for the next iteration (the one that warns)
would be very helpful if you squash the following, otherwise
Carlo
-- >8 --
Subject: [PATCH] SQUASH!!!
--- t/t0302-credential-store.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
t/t0302-credential-store.sh +1 -1
@@ -120,7 +120,7 @@ test_expect_success 'erase: erase matching credentials from both xdg and home fi ' invalid_credential_test() { - test_expect_success 'get: ignore credentials without $1 as invalid' ' + test_expect_success "get: ignore credentials without $1 as invalid" ' echo "$2" >"$HOME/.git-credentials" && check fill store <<-\EOF protocol=https
Signature · 1 line
2.26.2.686.gfaf46a9ccd
PATCH v11 1/1 credential-store: ignore bogus lines from store file
In reply to Carlo Marcelo Arenas BelónWith the added checks for invalid URLs in credentials, any locally modified store files which might have empty lines or even comments were reported[1] failing to parse as valid credentials.
Instead of doing a hard check for credentials, do a soft one and therefore avoid the reported fatal error.
While at it add tests for all known corruptions that are currently ignored to keep track of them and avoid the risk of regressions.
[1] https://stackoverflow.com/a/61420852/5005936
---
v11:
* remove bogus early check for CR characters
v10:
* go back to v4 but with better testing and commit message
* make sure broken CR characters are ignored early
v9:
* use strbuf_getline() instead to better handle files with CRLF v8:
* only warn during get operations as otherwise line number might be incorrect
v7:
* check for protocol in helper as suggested by Junio
v6:
* get rid of redacter and only use line number for context in warning
* make validation more strict to also catch incomplete credentials
* reorder check as suggested by Junio
v5:
* q_to_tab this round, with a single echo to make sure empty line is covered, as that seems to be a popular issue
* rebase on top of jc/credential-store-file-format-doc
* implement a redacter for credentials to use on errors to avoid leaking passwords
v4:
* use credential_from_url_gently instead as shown by Jonathan
* add documentation to clarify "comments" is not a supported feature v3:
* avoid using q_to_cr as suggested by Peff
* a more verbose commit message and slightly more complete documentation v2:
* use a here-doc for clarity as suggested by Eric
* improve commit message and include documentation
credential-store.c | 4 +- t/t0302-credential-store.sh | 91 ++++++++++++++++++++++++++++++++++++- 2 files changed, 92 insertions(+), 3 deletions(-)
credential-store.c +2 -2
@@ -24,8 +24,8 @@ static int parse_credential_file(const char *fn, }while (strbuf_getline_lf (&line, fh) !=EOF ) { -credential_from_url (&entry, line.buf); -if (entry.username && entry.password && +if (!credential_from_url_gently (&entry, line.buf,1 ) && + entry.username && entry.password &&credential_match (c, &entry)) { found_credential =1 ;if (match_cb) {
t/t0302-credential-store.sh +90 -1
101 lines not on this page · raw message
base-commit: 49458fb91d61461938881559ce23abbb1a2f8c35
Signature · 1 line
2.26.2.686.gfaf46a9ccd