tag: avoid NULL pointer arithmetic
PATCH 1/1 tag: avoid NULL pointer arithmetic
lookup_blob() etc. can return NULL if the referenced object isn't of the expected type. In theory it's wrong to reference the object member in that case. In practice it's OK because it's located at offset 0 for all types, so the pointer arithmetic (NULL + 0) is optimized out by the compiler. The issue is reported by Clang's AddressSanitizer, though.
Avoid the ASan error by casting the results of the lookup functions to struct object pointers. That works fine with NULL pointers as well. We already rely on the object member being first in all object types in other places in the code.
--- tag.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
tag.c +4 -4
@@ -142,13 +142,13 @@ int parse_tag_buffer(struct tag *item, const void *data, unsigned long size) bufptr = nl +1 ;if (!strcmp (type, blob_type)) { - item->tagged = &lookup_blob (&oid)->object; + item->tagged = (struct object *)lookup_blob (&oid); }else if (!strcmp (type, tree_type)) { - item->tagged = &lookup_tree (&oid)->object; + item->tagged = (struct object *)lookup_tree (&oid); }else if (!strcmp (type, commit_type)) { - item->tagged = &lookup_commit (&oid)->object; + item->tagged = (struct object *)lookup_commit (&oid); }else if (!strcmp (type, tag_type)) { - item->tagged = &lookup_tag (&oid)->object; + item->tagged = (struct object *)lookup_tag (&oid); }else {error ("Unknown type %s" , type); item->tagged =NULL ;
Signature · 1 line
2.14.2
Quoted 6 lines · René Scharfe
lookup_blob()etc. can return NULL if the referenced object isn't of the expected type. In theory it's wrong to reference the object member in that case. In practice it's OK because it's located at offset 0 for all types, so the pointer arithmetic (NULL + 0) is optimized out by the compiler. The issue is reported by Clang's AddressSanitizer, though.
Quoted 7 lines
Avoid the ASan error by casting the results of the lookup functions to struct object pointers. That works fine with NULL pointers as well. We already rely on the object member being first in all object types in other places in the code.
Signed-off-by Rene Scharfe <[email protected]>---
Yikes, that is tricky.
Taking the address of .object field appears to be a lot cleaner and more kosher than casting, but from the compiler's point of view it's quite the opposite. Sigh.
Thanks. The patch itself looks like the right solution to me.
Quoted 25 lines
tag.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)tag.c +4
-4@@ -142,13 +142,13 @@ int parse_tag_buffer(struct tag *item, const void *data, unsigned long size) bufptr = nl +1 ;if (!strcmp (type, blob_type)) { - item->tagged = &lookup_blob (&oid)->object; + item->tagged = (struct object *)lookup_blob (&oid); }else if (!strcmp (type, tree_type)) { - item->tagged = &lookup_tree (&oid)->object; + item->tagged = (struct object *)lookup_tree (&oid); }else if (!strcmp (type, commit_type)) { - item->tagged = &lookup_commit (&oid)->object; + item->tagged = (struct object *)lookup_commit (&oid); }else if (!strcmp (type, tag_type)) { - item->tagged = &lookup_tag (&oid)->object; + item->tagged = (struct object *)lookup_tag (&oid); }else {error ("Unknown type %s" , type); item->tagged =NULL ;
Quoted 10 lines · René Scharfe
lookup_blob()etc. can return NULL if the referenced object isn't of the expected type. In theory it's wrong to reference the object member in that case. In practice it's OK because it's located at offset 0 for all types, so the pointer arithmetic (NULL + 0) is optimized out by the compiler. The issue is reported by Clang's AddressSanitizer, though.Avoid the ASan error by casting the results of the lookup functions to struct object pointers. That works fine with NULL pointers as well. We already rely on the object member being first in all object types in other places in the code.
Out of curiosity, did you have to do anything to coax this out of ASan (e.g., a specific version)? I've been running it pretty regularly and didn't see this one (I did switch from clang to gcc a month or two ago, but this code is pretty old, I think).
-Peff
Am 02.10.2017 um 07:08 schrieb Jeff King:
Quoted 13 lines · René Scharfe
Quoted 11 lines
lookup_blob()etc. can return NULL if the referenced object isn't of the expected type. In theory it's wrong to reference the object member in that case. In practice it's OK because it's located at offset 0 for all types, so the pointer arithmetic (NULL + 0) is optimized out by the compiler. The issue is reported by Clang's AddressSanitizer, though.Avoid the ASan error by casting the results of the lookup functions to struct object pointers. That works fine with NULL pointers as well. We already rely on the object member being first in all object types in other places in the code.
Quoted 4 lines
Out of curiosity, did you have to do anything to coax this out of ASan (e.g., a specific version)? I've been running it pretty regularly and didn't see this one (I did switch from clang to gcc a month or two ago, but this code is pretty old, I think).
I did "make -j4 SANITIZE=undefined,address BLK_SHA1=1 test" with clang version 4.0.1-1 (tags/RELEASE_401/final), and t1450-fsck.sh failed.
René
Quoted 5 lines · René Scharfe
Quoted 5 lines
Quoted 4 lines
Avoid the ASan error by casting the results of the lookup functions to struct object pointers. That works fine with NULL pointers as well. We already rely on the object member being first in all object types in other places in the code.
Quoted 7 lines
Quoted 4 lines
Out of curiosity, did you have to do anything to coax this out of ASan (e.g., a specific version)? I've been running it pretty regularly and didn't see this one (I did switch from clang to gcc a month or two ago, but this code is pretty old, I think).
I did "make -j4 SANITIZE=undefined,address BLK_SHA1=1 test" with clang version 4.0.1-1 (tags/RELEASE_401/final), and
t1450-fsck.shfailed.
Interesting. I can trigger it with the same setup, but not if:
1. I use gcc instead of clang.
2. If I only use one of UBSan and ASan. It's the combination that triggers it.
-Peff
Quoted 10 lines
lookup_blob()etc. can return NULL if the referenced object isn't of the expected type. In theory it's wrong to reference the object member in that case. In practice it's OK because it's located at offset 0 for all types, so the pointer arithmetic (NULL + 0) is optimized out by the compiler. The issue is reported by Clang's AddressSanitizer, though.Avoid the ASan error by casting the results of the lookup functions to struct object pointers. That works fine with NULL pointers as well. We already rely on the object member being first in all object types in other places in the code.
This sounds like the main goal of the patch is to avoid an ASan error, but I think it's more important to avoid (and to be more explicit about avoiding) the undefined behavior. I.e. along the lines of s/In theory it's wrong/It's undefined behavior/ and
s/ASan error/undefined behavior/
Furthermore, fsck.c:fsck_walk_tree() does the same "immediately reference the object member in lookup_blob()'s and lookup_tree()'s return value" thing. I think those should receive the same treatment as well.
Quoted 30 lines
Signed-off-by Rene Scharfe <[email protected]>--- tag.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)tag.c +4
-4@@ -142,13 +142,13 @@ int parse_tag_buffer(struct tag *item, const void *data, unsigned long size) bufptr = nl +1 ;if (!strcmp (type, blob_type)) { - item->tagged = &lookup_blob (&oid)->object; + item->tagged = (struct object *)lookup_blob (&oid); }else if (!strcmp (type, tree_type)) { - item->tagged = &lookup_tree (&oid)->object; + item->tagged = (struct object *)lookup_tree (&oid); }else if (!strcmp (type, commit_type)) { - item->tagged = &lookup_commit (&oid)->object; + item->tagged = (struct object *)lookup_commit (&oid); }else if (!strcmp (type, tag_type)) { - item->tagged = &lookup_tag (&oid)->object; + item->tagged = (struct object *)lookup_tag (&oid); }else {error ("Unknown type %s" , type); item->tagged =NULL ;Signature · 1 line
2.14.2
Am 03.10.2017 um 12:22 schrieb SZEDER Gábor:
Quoted 11 lines
Quoted 10 lines
lookup_blob()etc. can return NULL if the referenced object isn't of the expected type. In theory it's wrong to reference the object member in that case. In practice it's OK because it's located at offset 0 for all types, so the pointer arithmetic (NULL + 0) is optimized out by the compiler. The issue is reported by Clang's AddressSanitizer, though.Avoid the ASan error by casting the results of the lookup functions to struct object pointers. That works fine with NULL pointers as well. We already rely on the object member being first in all object types in other places in the code.
Quoted 5 lines
This sounds like the main goal of the patch is to avoid an ASan error, but I think it's more important to avoid (and to be more explicit about avoiding) the undefined behavior. I.e. along the lines of s/In theory it's wrong/It's undefined behavior/ and
s/ASan error/undefined behavior/
You are probably right, but I struggle to pin down the difference. Another
try: Adding zero to a NULL pointer is undefined, but is either optimized out or has no effect. That's why the code doesn't crash without ASan and UBSan. Relying on undefined behavior is wrong nevertheless, so let's stop doing it.
Quoted 4 lines
Furthermore, fsck.c:fsck_walk_tree() does the same "immediately reference the object member in lookup_blob()'s and lookup_tree()'s return value" thing. I think those should receive the same treatment as well.
Hmm, are put_object_name() and all the walk() implementations ready for a NULL object handed to them? Or would we rather need to error out right there? Other suspicious places in this regard are (at least):builtin/merge-tree.c, cache-tree.c, http-push.c, list-objects.c, merge-recursive.c, and shallow.c. :-/
René
PATCH 1/1 fsck: check results of lookup_blob() and lookup_tree() for NULL
Am 03.10.2017 um 14:51 schrieb René Scharfe:
Quoted 9 lines
Am 03.10.2017 um 12:22 schrieb SZEDER Gábor:
Quoted 4 lines
Furthermore, fsck.c:fsck_walk_tree() does the same "immediately reference the object member in lookup_blob()'s and lookup_tree()'s return value" thing. I think those should receive the same treatment as well.
Hmm, are
put_object_name()and all thewalk()implementations ready for a NULL object handed to them? Or would we rather need to error out right there?
How about this?
-- >8 --lookup_blob() and lookup_tree() can return NULL if they find an object of an unexpected type. Error out of fsck_walk_tree() in that case, like we do when encountering a bad file mode. An error message is already shown by object_as_type(), which gets called by the lookup functions.
--- fsck.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
fsck.c +8 -2
@@ -358,14 +358,20 @@ static int fsck_walk_tree(struct tree *tree, void *data, struct fsck_options *opcontinue ;if (S_ISDIR (entry.mode)) { - obj = &lookup_tree (entry.oid)->object; +struct tree *tree =lookup_tree (entry.oid); +if (!tree) +return -1 ; + obj = &tree->object;if (name)put_object_name (options, obj,"%s%s/" , name, entry.path); result = options->walk (obj,OBJ_TREE , data, options); }else if (S_ISREG (entry.mode) ||S_ISLNK (entry.mode)) { - obj = &lookup_blob (entry.oid)->object; +struct blob *blob =lookup_blob (entry.oid); +if (!blob) +return -1 ; + obj = &blob->object;if (name)put_object_name (options, obj,"%s%s" , name, entry.path);
Signature · 1 line
2.14.2
Re: [PATCH] fsck: check results of lookup_blob() and lookup_tree() for NULL
Quoted 10 lines · René Scharfe
Am 03.10.2017 um 14:51 schrieb René Scharfe:
Quoted 9 lines
Am 03.10.2017 um 12:22 schrieb SZEDER Gábor:
Quoted 4 lines
Furthermore, fsck.c:fsck_walk_tree() does the same "immediately reference the object member in lookup_blob()'s and lookup_tree()'s return value" thing. I think those should receive the same treatment as well.
Hmm, are
put_object_name()and all thewalk()implementations ready for a NULL object handed to them? Or would we rather need to error out right there?
Quoted 7 lines
How about this?
-- >8 --
lookup_blob()andlookup_tree()can return NULL if they find an object of an unexpected type. Error out offsck_walk_tree()in that case, like we do when encountering a bad file mode. An error message is already shown byobject_as_type(), which gets called by the lookup functions.
The result from options->walk() is checked, and among the callbacks that are assigned to the .walk field:
- mark_object() in builtin/fsck.c gives its own error message to diagnose broken link and returns 1;
- mark_used() in builtin/fsck.c silently returns 1;
- mark_link() in builtin/index-pack.c does the same; and
- check_object() in builtin/unpack-objects.c does the same,
when they see a NULL object.
This patch may avoid the "unexpected behaviour" coming from expecting that &((struct tree *)NULL)->object == NULL the current code does, but it also changes the behaviour. The loop used to diagnose a fishy entry in the tree we are walking, and kept checking the remaining entries in the tree. You now immediately return, not seeing if the later entries in the tree are good and losing objects that are referenced by these entries as dangling.
I am not sure if this is a good change. I suspect that the "bad mode" handling should be made less severe instead.
Quoted 33 lines
Signed-off-by Rene Scharfe <[email protected]>--- fsck.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)fsck.c +8
-2@@ -358,14 +358,20 @@ static int fsck_walk_tree(struct tree *tree, void *data, struct fsck_options *opcontinue ;if (S_ISDIR (entry.mode)) { - obj = &lookup_tree (entry.oid)->object; +struct tree *tree =lookup_tree (entry.oid); +if (!tree) +return -1 ; + obj = &tree->object;if (name)put_object_name (options, obj,"%s%s/" , name, entry.path); result = options->walk (obj,OBJ_TREE , data, options); }else if (S_ISREG (entry.mode) ||S_ISLNK (entry.mode)) { - obj = &lookup_blob (entry.oid)->object; +struct blob *blob =lookup_blob (entry.oid); +if (!blob) +return -1 ; + obj = &blob->object;if (name)put_object_name (options, obj,"%s%s" , name, entry.path);
Re: [PATCH] fsck: check results of lookup_blob() and lookup_tree() for NULL
Am 04.10.2017 um 06:00 schrieb Junio C Hamano:
Quoted 28 lines
Quoted 4 lines · René Scharfe
lookup_blob()andlookup_tree()can return NULL if they find an object of an unexpected type. Error out offsck_walk_tree()in that case, like we do when encountering a bad file mode. An error message is already shown byobject_as_type(), which gets called by the lookup functions.The result from
options->walk()is checked, and among the callbacks that are assigned to the .walk field:-
mark_object()inbuiltin/fsck.cgives its own error message to diagnose broken linkand returns 1;-
mark_used()inbuiltin/fsck.csilently returns 1;-
mark_link()inbuiltin/index-pack.cdoes the same; and-
check_object()inbuiltin/unpack-objects.cdoes the same,when they see a NULL object.
This patch may avoid the "unexpected behaviour" coming from expecting that &((struct tree *)NULL)->object == NULL the current code does, but it also changes the behaviour. The loop used to diagnose a fishy entry in the tree we are walking, and kept checking the remaining entries in the tree. You now immediately return, not seeing if the later entries in the tree are good and losing objects that are referenced by these entries as dangling.
Quoted 2 lines
I am not sure if this is a good change. I suspect that the "bad mode" handling should be made less severe instead.
Makes sense. Replacement patch coming up. I'll pass on the mode handling change, though (at least for now).
René
PATCH v2 1/1 fsck: handle NULL return of lookup_blob() and lookup_tree()
In reply to René Scharfelookup_blob() and lookup_tree() can return NULL if they find an object of an unexpected type. Accessing the object member is undefined in that case. Cast the result to a struct object pointer instead; we can do that because object is the first member of all object types. This trick is already used in other places in the code.
An error message is already shown by object_as_type(), which is called by the lookup functions. The walk callback functions are expected to handle NULL object pointers passed to them, but put_object_name() needs a valid object, so avoid calling it without one.
---
Changes from v1:
- Don't abort on encountering an object with a mismatched type.
- Added a test for showing the problem with SANITIZE=address,undefined.
fsck.c | 8 ++++---- t/t1450-fsck.sh | 22 ++++++++++++++++++++++ 2 files changed, 26 insertions(+), 4 deletions(-)
fsck.c +4 -4
@@ -358,15 +358,15 @@ static int fsck_walk_tree(struct tree *tree, void *data, struct fsck_options *opcontinue ;if (S_ISDIR (entry.mode)) { - obj = &lookup_tree (entry.oid)->object; -if (name) + obj = (struct object *)lookup_tree (entry.oid); +if (name && obj)put_object_name (options, obj,"%s%s/" , name, entry.path); result = options->walk (obj,OBJ_TREE , data, options); }else if (S_ISREG (entry.mode) ||S_ISLNK (entry.mode)) { - obj = &lookup_blob (entry.oid)->object; -if (name) + obj = (struct object *)lookup_blob (entry.oid); +if (name && obj)put_object_name (options, obj,"%s%s" , name, entry.path); result = options->walk (obj,OBJ_BLOB , data, options);
t/t1450-fsck.sh +22 -0
@@ -222,6 +222,28 @@ test_expect_success 'unparseable tree object' ' test_i18ngrep ! "fatal: empty filename in tree entry" out ' +hex2oct() { + perl -ne 'printf "\\%03o", hex for /../g' +} + +test_expect_success 'tree entry with type mismatch' ' + test_when_finished "remove_object \$blob" && + test_when_finished "remove_object \$tree" && + test_when_finished "remove_object \$commit" && + test_when_finished "git update-ref -d refs/heads/type_mismatch" && + blob=$(echo blob | git hash-object -w --stdin) && + blob_bin=$(echo $blob | hex2oct) && + tree=$( + printf "40000 dir\0${blob_bin}100644 file\0${blob_bin}" | + git hash-object -t tree --stdin -w --literally + ) && + commit=$(git commit-tree $tree) && + git update-ref refs/heads/type_mismatch $commit && + test_must_fail git fsck >out 2>&1 && + test_i18ngrep "is a blob, not a tree" out && + test_i18ngrep ! "dangling blob" out +' + test_expect_success 'tag pointing to nonexistent' ' cat >invalid-tag <<-\EOF && object ffffffffffffffffffffffffffffffffffffffff
Signature · 1 line
2.14.2
Re: [PATCH v2] fsck: handle NULL return of lookup_blob() and lookup_tree()
Quoted 4 lines · René Scharfe
An error message is already shown by
object_as_type(), which is called by the lookup functions. The walk callback functions are expected to handle NULL object pointers passed to them, butput_object_name()needs a valid object, so avoid calling it without one.
Thanks for getting the details right ;-)
t/t1450-fsck.sh test_expect_success 'unparseable tree object' ' · line 237
Quoted 3 lines
t/t1450-fsck.sh +3
-0+ blob_bin=$(echo $blob | hex2oct) && + tree=$( + printf "40000 dir\0${blob_bin}100644 file\0${blob_bin}" |
Wow, that's ... cute.
t/t1450-fsck.sh test_expect_success 'unparseable tree object' ' · line 238
Quoted 1 line
t/t1450-fsck.sh +1
-0+ git hash-object -t tree --stdin -w --literally
Makes me curious why --literally is here. Even if we let check_tree() called from index_mem() by taking the normal path, it wouldn't complain the type mismatch, I suspect. I guess doing it this way is a future-proof against check_tree() getting tightened in the future, in which case I think it makes sense.
And for the same reason, hashing "--literally" like this patch does is a better solution than using "git mktree", which would have allowed us to avoid the hex2oct and instead feed the tree in a bit more human-readable way.
Thanks, will queue.
Quoted 11 lines
t/t1450-fsck.sh +8
-0+ ) && + commit=$(git commit-tree $tree) && + git update-ref refs/heads/type_mismatch $commit && + test_must_fail git fsck >out 2>&1 && + test_i18ngrep "is a blob, not a tree" out && + test_i18ngrep ! "dangling blob" out +' + test_expect_success 'tag pointing to nonexistent' ' cat >invalid-tag <<-\EOF && object ffffffffffffffffffffffffffffffffffffffff
Re: [PATCH v2] fsck: handle NULL return of lookup_blob() and lookup_tree()
Am 06.10.2017 um 04:23 schrieb Junio C Hamano:
Quoted 15 lines
t/t1450-fsck.sh test_expect_success 'unparseable tree object' ' · line 237
Quoted 3 lines · René Scharfe
t/t1450-fsck.sh +3
-0+ blob_bin=$(echo $blob | hex2oct) && + tree=$( + printf "40000 dir\0${blob_bin}100644 file\0${blob_bin}" |Wow, that's ... cute.
t/t1450-fsck.sh test_expect_success 'unparseable tree object' ' · line 238
Quoted 1 line
t/t1450-fsck.sh +1
-0+ git hash-object -t tree --stdin -w --literallyMakes me curious why --literally is here. Even if we let
check_tree()called fromindex_mem()by taking the normal path, it wouldn't complain the type mismatch, I suspect. I guess doing it this way is a future-proof againstcheck_tree()getting tightened in the future, in which case I think it makes sense.
Quoted 4 lines
And for the same reason, hashing "--literally" like this patch does is a better solution than using "git mktree", which would have allowed us to avoid the hex2oct and instead feed the tree in a bit more human-readable way.
git mktree errors out already, complaining about the object type mismatch. But I added "--literally" only accidentally, when I copied the invocation from a few lines up. The test works fine without that flag currently. The flag captures the intent, however, of knowingly building a flawed tree object.
René