Skip to contentSkip to conversation

Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default

PATCH In review Ævar Arnfjörð Bjarmason 9 participants 17 messages May 15, 2017 – Jun 1, 2017 2 messages not in the archive Kbuild

lore
Share & tools
lore
#0 lore raw

Since 2.13.0 just running "git status" on a newly init'd repo on Solaris SPARC[1] segfaults. Michael (CC'd) reported this issue on #git and I helped him debug it.

Just compiling with BLK_SHA1=YesPlease solves the issue.

There are at least two different issues with DC_SHA1 here:

* We don't properly detect that this platform is big endian. The
check at the top of sha1dc/sha1.c needs to test for _BIG_ENDIAN. This comes from sys/isa_defs.h which (I'm told by #solaris) is included on Solaris by default, at least by stdio.h.

Hacking the endian detection makes t0013-sha1dc.sh pass.

* Even with that & the test passing just a plain "git init x && cd x
&& touch A && git add A && git commit" will segfault.

This is some bug in the sha1dc code, presumably some big endian issue that's not resolved by the change above. Backtrace for that (censored actual author info):

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1 (LWP 1)]
0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5,
W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291
291 SHA1COMPRESS_FULL_ROUND1_STEP_LOAD(a, b, c, d, e, m,
W, 0, temp);
(gdb) bt
#0 0x002f8c84 in sha1_compression_states (ihv=0xffbf8268,
m=0x398ad5, W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291
#1 0x00300b60 in sha1_process (ctx=0xffbf8260, block=0x398ad5) at
sha1dc/sha1.c:1616
#2 0x00301188 in SHA1DCUpdate (ctx=0xffbf8260,

     buf=0x398ad5 "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\nauthor Au
Thor <[email protected]> 123456789 +0000\ncommitter Au Thor
<[email protected]> 123456789
                  +0000\n\nBlah Blah"..., len=220)
     at sha1dc/sha1.c:1731

#3 0x0030168c in git_SHA1DCUpdate (ctx=0xffbf8260, vdata=0x398aa0,
len=273) at sha1dc/sha1.c:1808
#4 0x002a6f7c in write_sha1_file_prepare (buf=0x398aa0, len=273,
type=0x959c8 "commit", sha1=0xffbfd630 "",

     hdr=0xffbf8c28 "commit 273", hdrlen=0xffbf8c24) at sha1_file.c:3207

#5 0x002a71ac in hash_sha1_file (buf=0x398aa0, len=273, type=0x959c8
"commit", sha1=0xffbfd630 "") at sha1_file.c:3266
#6 0x002a25f8 in check_sha1_signature (sha1=0xffbfdbb8
"\375\067\356\337\002", map=0x398aa0, size=273, type=0x959c8 "commit")

     at sha1_file.c:1644

#7 0x0022816c in parse_object (sha1=0xffbfdbb8
"\375\067\356\337\002") at object.c:269
#8 0x0027c258 in get_reference (revs=0xffbfdc88, name=0xa87f0
"HEAD", sha1=0xffbfdbb8 "\375\067\356\337\002", flags=0)

     at revision.c:196

#9 0x00284714 in setup_revisions (argc=0, argv=0x0, revs=0xffbfdc88,
opt=0xffbfdc74) at revision.c:2295
#10 0x002ee4d8 in wt_status_collect_changes_index (s=0x348ea8
<s.24114>) at wt-status.c:585
#11 0x002eeae8 in wt_status_collect (s=0x348ea8 <s.24114>) at wt-status.c:701 #12 0x000db4fc in cmd_status (argc=0, argv=0xffbfe6fc, prefix=0x0) at
builtin/commit.c:1393
#13 0x000acc24 in run_builtin (p=0x340ccc <commands+1200>, argc=1,
argv=0xffbfe6fc) at git.c:371
#14 0x000acf98 in handle_builtin (argc=1, argv=0xffbfe6fc) at git.c:572 #15 0x000ad1a4 in run_argv (argcp=0xffbfe66c, argv=0xffbfe670) at git.c:624 #16 0x000ad3fc in cmd_main (argc=1, argv=0xffbfe6fc) at git.c:701 #17 0x001747a0 in main (argc=5, argv=0xffbfe6ec) at common-main.c:43

1. Relevant part of uname -a: "SunOS <hostname> 5.11 11.3 sun4v sparc sun4v".

#1 lore raw

Hi Aevar,

Thank you for notifying us of this issue.
Big endianness is a tricky issue, also since I don't have access or accurate knowledge about all big endian systems. Our github repo does check correct functioning, including an endianness mistake, with 'make test'. But I guess this is not included for SHA1DC in Git.

Anyway, we can easily add the _BIG_ENDIAN macrotest to the git repo and will do so soon.

I don't think the segfault is caused by buffer overflow, inproper access, or the endianness issue. But I did notice an unexpected issue: the message block pointer m=0x398ad5 is odd.
Can you confirm whether loading an uint32_t from an odd address triggers a hardware interrupt on your platform? This is not problem for x86, but maybe for your platform it is?
If it is then we should always copy buffer contents to the sha1context to avoid this issue.

Best regards,
Marc Stevens

-----Oorspronkelijk bericht-----
Van: Ævar Arnfjörð Bjarmason [mailto:[email protected]]
Verzonden: maandag 15 mei 2017 14:49
Aan: Git Mailing List <[email protected]>
CC: [email protected]; Jeff King <[email protected]>; Marc Stevens <[email protected]>
Onderwerp: Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default

Since 2.13.0 just running "git status" on a newly init'd repo on Solaris SPARC[1] segfaults. Michael (CC'd) reported this issue on #git and I helped him debug it.

Just compiling with BLK_SHA1=YesPlease solves the issue.

There are at least two different issues with DC_SHA1 here:

* We don't properly detect that this platform is big endian. The check at the top of sha1dc/sha1.c needs to test for _BIG_ENDIAN. This comes from sys/isa_defs.h which (I'm told by #solaris) is included on Solaris by default, at least by stdio.h.

Hacking the endian detection makes t0013-sha1dc.sh pass.

* Even with that & the test passing just a plain "git init x && cd x && touch A && git add A && git commit" will segfault.

This is some bug in the sha1dc code, presumably some big endian issue that's not resolved by the change above. Backtrace for that (censored actual author info):

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1 (LWP 1)]
0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5, W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291
291 SHA1COMPRESS_FULL_ROUND1_STEP_LOAD(a, b, c, d, e, m,
W, 0, temp);
(gdb) bt
#0 0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5, W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291 #1 0x00300b60 in sha1_process (ctx=0xffbf8260, block=0x398ad5) at
sha1dc/sha1.c:1616
#2 0x00301188 in SHA1DCUpdate (ctx=0xffbf8260,

     buf=0x398ad5 "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\nauthor Au Thor <[email protected]> 123456789 +0000\ncommitter Au Thor <[email protected]> 123456789
                  +0000\n\nBlah Blah"..., len=220)
     at sha1dc/sha1.c:1731

#3 0x0030168c in git_SHA1DCUpdate (ctx=0xffbf8260, vdata=0x398aa0,
len=273) at sha1dc/sha1.c:1808
#4 0x002a6f7c in write_sha1_file_prepare (buf=0x398aa0, len=273,
type=0x959c8 "commit", sha1=0xffbfd630 "",

     hdr=0xffbf8c28 "commit 273", hdrlen=0xffbf8c24) at sha1_file.c:3207
 #5  0x002a71ac in hash_sha1_file (buf=0x398aa0, len=273, type=0x959c8 "commit", sha1=0xffbfd630 "") at sha1_file.c:3266
 #6  0x002a25f8 in check_sha1_signature (sha1=0xffbfdbb8 "\375\067\356\337\002", map=0x398aa0, size=273, type=0x959c8 "commit")
     at sha1_file.c:1644
 #7  0x0022816c in parse_object (sha1=0xffbfdbb8
"\375\067\356\337\002") at object.c:269
 #8  0x0027c258 in get_reference (revs=0xffbfdc88, name=0xa87f0 "HEAD", sha1=0xffbfdbb8 "\375\067\356\337\002", flags=0)
     at revision.c:196

#9 0x00284714 in setup_revisions (argc=0, argv=0x0, revs=0xffbfdc88,
opt=0xffbfdc74) at revision.c:2295
#10 0x002ee4d8 in wt_status_collect_changes_index (s=0x348ea8
<s.24114>) at wt-status.c:585
#11 0x002eeae8 in wt_status_collect (s=0x348ea8 <s.24114>) at wt-status.c:701 #12 0x000db4fc in cmd_status (argc=0, argv=0xffbfe6fc, prefix=0x0) at
builtin/commit.c:1393
#13 0x000acc24 in run_builtin (p=0x340ccc <commands+1200>, argc=1,
argv=0xffbfe6fc) at git.c:371
#14 0x000acf98 in handle_builtin (argc=1, argv=0xffbfe6fc) at git.c:572 #15 0x000ad1a4 in run_argv (argcp=0xffbfe66c, argv=0xffbfe670) at git.c:624 #16 0x000ad3fc in cmd_main (argc=1, argv=0xffbfe6fc) at git.c:701 #17 0x001747a0 in main (argc=5, argv=0xffbfe6ec) at common-main.c:43

1. Relevant part of uname -a: "SunOS <hostname> 5.11 11.3 sun4v sparc sun4v".

#2 lore raw
Quoted 7 lines · Marc Stevens

Hi Aevar,

Thank you for notifying us of this issue.
Big endianness is a tricky issue, also since I don't have access or accurate knowledge about all big endian systems. Our github repo does check correct functioning, including an endianness mistake, with 'make test'. But I guess this is not included for SHA1DC in Git.

Quoted 7 lines

Anyway, we can easily add the _BIG_ENDIAN macrotest to the git repo and will do so soon.

I don't think the segfault is caused by buffer overflow, inproper access, or the endianness issue. But I did notice an unexpected issue: the message block pointer m=0x398ad5 is odd.
Can you confirm whether loading an uint32_t from an odd address triggers a hardware interrupt on your platform? This is not problem for x86, but maybe for your platform it is?
If it is then we should always copy buffer contents to the sha1context to avoid this issue.

I don't have access to the box in question, Michael was testing this code for me. But unaligned access is probably the cause, although according to some info I found online that should give a SIGBUS not a SIGSEGV, but that may have changed:

https://bugs.python.org/issue12181
https://github.com/magnumripper/JohnTheRipper/issues/2187

Quoted 65 lines

Best regards,
Marc Stevens

-----Oorspronkelijk bericht-----
Van: Ævar Arnfjörð Bjarmason [mailto:[email protected]]
Verzonden: maandag 15 mei 2017 14:49
Aan: Git Mailing List <[email protected]>
CC: [email protected]; Jeff King <[email protected]>; Marc Stevens <[email protected]>
Onderwerp: Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default

Since 2.13.0 just running "git status" on a newly init'd repo on Solaris SPARC[1] segfaults. Michael (CC'd) reported this issue on #git and I helped him debug it.

Just compiling with BLK_SHA1=YesPlease solves the issue.

There are at least two different issues with DC_SHA1 here:

* We don't properly detect that this platform is big endian. The check at the top of sha1dc/sha1.c needs to test for _BIG_ENDIAN. This comes from sys/isa_defs.h which (I'm told by #solaris) is included on Solaris by default, at least by stdio.h.

Hacking the endian detection makes t0013-sha1dc.sh pass.

* Even with that & the test passing just a plain "git init x && cd x && touch A && git add A && git commit" will segfault.

This is some bug in the sha1dc code, presumably some big endian issue that's not resolved by the change above. Backtrace for that (censored actual author info):

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1 (LWP 1)]
0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5, W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291
291 SHA1COMPRESS_FULL_ROUND1_STEP_LOAD(a, b, c, d, e, m,
W, 0, temp);
(gdb) bt
#0 0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5, W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291 #1 0x00300b60 in sha1_process (ctx=0xffbf8260, block=0x398ad5) at
sha1dc/sha1.c:1616
#2 0x00301188 in SHA1DCUpdate (ctx=0xffbf8260,

     buf=0x398ad5 "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\nauthor Au Thor <[email protected]> 123456789 +0000\ncommitter Au Thor <[email protected]> 123456789
                  +0000\n\nBlah Blah"..., len=220)
     at sha1dc/sha1.c:1731

#3 0x0030168c in git_SHA1DCUpdate (ctx=0xffbf8260, vdata=0x398aa0,
len=273) at sha1dc/sha1.c:1808
#4 0x002a6f7c in write_sha1_file_prepare (buf=0x398aa0, len=273,
type=0x959c8 "commit", sha1=0xffbfd630 "",

     hdr=0xffbf8c28 "commit 273", hdrlen=0xffbf8c24) at sha1_file.c:3207
 #5  0x002a71ac in hash_sha1_file (buf=0x398aa0, len=273, type=0x959c8 "commit", sha1=0xffbfd630 "") at sha1_file.c:3266
 #6  0x002a25f8 in check_sha1_signature (sha1=0xffbfdbb8 "\375\067\356\337\002", map=0x398aa0, size=273, type=0x959c8 "commit")
     at sha1_file.c:1644
 #7  0x0022816c in parse_object (sha1=0xffbfdbb8
"\375\067\356\337\002") at object.c:269
 #8  0x0027c258 in get_reference (revs=0xffbfdc88, name=0xa87f0 "HEAD", sha1=0xffbfdbb8 "\375\067\356\337\002", flags=0)
     at revision.c:196

#9 0x00284714 in setup_revisions (argc=0, argv=0x0, revs=0xffbfdc88,
opt=0xffbfdc74) at revision.c:2295
#10 0x002ee4d8 in wt_status_collect_changes_index (s=0x348ea8
<s.24114>) at wt-status.c:585
#11 0x002eeae8 in wt_status_collect (s=0x348ea8 <s.24114>) at wt-status.c:701 #12 0x000db4fc in cmd_status (argc=0, argv=0xffbfe6fc, prefix=0x0) at
builtin/commit.c:1393
#13 0x000acc24 in run_builtin (p=0x340ccc <commands+1200>, argc=1,
argv=0xffbfe6fc) at git.c:371
#14 0x000acf98 in handle_builtin (argc=1, argv=0xffbfe6fc) at git.c:572 #15 0x000ad1a4 in run_argv (argcp=0xffbfe66c, argv=0xffbfe670) at git.c:624 #16 0x000ad3fc in cmd_main (argc=1, argv=0xffbfe6fc) at git.c:701 #17 0x001747a0 in main (argc=5, argv=0xffbfe6ec) at common-main.c:43

1. Relevant part of uname -a: "SunOS <hostname> 5.11 11.3 sun4v sparc sun4v".

#3 lore raw
Quoted 16 lines · Ævar Arnfjörð Bjarmason
Quoted 7 lines · Marc Stevens

Hi Aevar,

Thank you for notifying us of this issue.
Big endianness is a tricky issue, also since I don't have access or accurate knowledge about all big endian systems. Our github repo does check correct functioning, including an endianness mistake, with 'make test'. But I guess this is not included for SHA1DC in Git.

Quoted 7 lines

Anyway, we can easily add the _BIG_ENDIAN macrotest to the git repo and will do so soon.

I don't think the segfault is caused by buffer overflow, inproper access, or the endianness issue. But I did notice an unexpected issue: the message block pointer m=0x398ad5 is odd.
Can you confirm whether loading an uint32_t from an odd address triggers a hardware interrupt on your platform? This is not problem for x86, but maybe for your platform it is?
If it is then we should always copy buffer contents to the sha1context to avoid this issue.

Quoted 4 lines

I don't have access to the box in question, Michael was testing this code for me. But unaligned access is probably the cause, although according to some info I found online that should give a SIGBUS not a SIGSEGV, but that may have changed:

Yeah, I would have expected SIGBUS there. If we have alignment issues, though, I'd expect that ARM systems will experience problems.

Block-sha1 uses a macro which allows unaligned loads on platforms that support it, and otherwise does the endian conversion on the fly as we load the bytes into a local variable (which presumably happens all in-register). That may be faster than doing a mass copy of the buffer.

-Peff

#4 lore raw
Quoted 23 lines · Jeff King
Quoted 16 lines · Ævar Arnfjörð Bjarmason
Quoted 7 lines · Marc Stevens

Hi Aevar,

Thank you for notifying us of this issue.
Big endianness is a tricky issue, also since I don't have access or accurate knowledge about all big endian systems. Our github repo does check correct functioning, including an endianness mistake, with 'make test'. But I guess this is not included for SHA1DC in Git.

Quoted 7 lines

Anyway, we can easily add the _BIG_ENDIAN macrotest to the git repo and will do so soon.

I don't think the segfault is caused by buffer overflow, inproper access, or the endianness issue. But I did notice an unexpected issue: the message block pointer m=0x398ad5 is odd.
Can you confirm whether loading an uint32_t from an odd address triggers a hardware interrupt on your platform? This is not problem for x86, but maybe for your platform it is?
If it is then we should always copy buffer contents to the sha1context to avoid this issue.

Quoted 4 lines

I don't have access to the box in question, Michael was testing this code for me. But unaligned access is probably the cause, although according to some info I found online that should give a SIGBUS not a SIGSEGV, but that may have changed:

Quoted 7 lines

Yeah, I would have expected SIGBUS there. If we have alignment issues, though, I'd expect that ARM systems will experience problems.

Block-sha1 uses a macro which allows unaligned loads on platforms that support it, and otherwise does the endian conversion on the fly as we load the bytes into a local variable (which presumably happens all in-register). That may be faster than doing a mass copy of the buffer.

I agree. It is fairly normal to use that kind of macro and not do a memcpy with hash functions.

In fact many hash functions ONLY use that kind of macro, as decent compilers will automagically convert the macro into an unaligned load on platforms that support fast unaligned loads.

The only reason to expose the direct unaligned load is to make sure that the hashing code is fast on such platforms even when compiled under -g.

Yves

Signature · 1 line
perl -Mre=debug -e "/just|another|perl|hacker/"

A message that is not in the archive (sent off-list or not yet received): CAKKM46vwM9pxyMxTc4jA0z_8vGKdDGCGg9ziKkFAsqr5ULYJxA@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw

Hi Marc,

works like a charm!

Michael

2017-05-15 16:33 GMT+02:00 Marc Stevens <[email protected]>:

Quoted 157 lines

Hi Michael,

See the latest commit to the SHA1DC repo:

https://github.com/cr-marcstevens/sha1collisiondetection/commit/33a694a9ee1b79c24be45f9eab5ac0e1aeeaf271

Just apply this same patch to SHA1DC in Git.

Best regards,

Marc Stevens

Van: Michael Kebe [mailto:[email protected]]
Verzonden: maandag 15 mei 2017 16:22
Aan: Ævar Arnfjörð Bjarmason <[email protected]>
CC: Jeff King <[email protected]>; Git Mailing List <[email protected]>; Marc Stevens <[email protected]>
Onderwerp: Re: Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default

Just give me patch or repo, I will test it tomorrow.

Michael

Am 15.05.2017 4:14 nachm. schrieb "Ævar Arnfjörð Bjarmason" <[email protected]>:

Quoted 12 lines · Marc Stevens

Hi Aevar,

Thank you for notifying us of this issue.
Big endianness is a tricky issue, also since I don't have access or accurate knowledge about all big endian systems.
Our github repo does check correct functioning, including an endianness mistake, with 'make test'.
But I guess this is not included for SHA1DC in Git.

Anyway, we can easily add the _BIG_ENDIAN macrotest to the git repo and will do so soon.

Quoted 9 lines

I don't think the segfault is caused by buffer overflow, inproper access, or the endianness issue.
But I did notice an unexpected issue: the message block pointer m=0x398ad5 is odd.
Can you confirm whether loading an uint32_t from an odd address triggers a hardware interrupt on your platform?
This is not problem for x86, but maybe for your platform it is? If it is then we should always copy buffer contents to the sha1context to avoid this issue.

I don't have access to the box in question, Michael was testing this code for me. But unaligned access is probably the cause, although according to some info I found online that should give a SIGBUS not a SIGSEGV, but that may have changed:

https://bugs.python.org/issue12181
https://github.com/magnumripper/JohnTheRipper/issues/2187

Quoted 85 lines

Best regards,
Marc Stevens

-----Oorspronkelijk bericht-----
Van: Ævar Arnfjörð Bjarmason [mailto:[email protected]]
Verzonden: maandag 15 mei 2017 14:49
Aan: Git Mailing List <[email protected]>
CC: [email protected]; Jeff King <[email protected]>; Marc Stevens <[email protected]>
Onderwerp: Git 2.13.0 segfaults on Solaris SPARC due to DC_SHA1=YesPlease being on by default

Since 2.13.0 just running "git status" on a newly init'd repo on Solaris SPARC[1] segfaults. Michael (CC'd) reported this issue on #git and I helped him debug it.

Just compiling with BLK_SHA1=YesPlease solves the issue.

There are at least two different issues with DC_SHA1 here:

* We don't properly detect that this platform is big endian. The check at
the top of sha1dc/sha1.c needs to test for _BIG_ENDIAN. This comes from sys/isa_defs.h which (I'm told by #solaris) is included on Solaris by default, at least by stdio.h.

Hacking the endian detection makes t0013-sha1dc.sh pass.

* Even with that & the test passing just a plain "git init x && cd x &&
touch A && git add A && git commit" will segfault.

This is some bug in the sha1dc code, presumably some big endian issue that's not resolved by the change above. Backtrace for that (censored actual author info):

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1 (LWP 1)]
0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5,
W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291
291 SHA1COMPRESS_FULL_ROUND1_STEP_LOAD(a, b, c, d, e, m,
W, 0, temp);
(gdb) bt
#0 0x002f8c84 in sha1_compression_states (ihv=0xffbf8268, m=0x398ad5,
W=0xffbf82fc, states=0xffbf857c) at sha1dc/sha1.c:291
#1 0x00300b60 in sha1_process (ctx=0xffbf8260, block=0x398ad5) at
sha1dc/sha1.c:1616
#2 0x00301188 in SHA1DCUpdate (ctx=0xffbf8260,

     buf=0x398ad5 "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef\nauthor Au
Thor <[email protected]> 123456789 +0000\ncommitter Au Thor
<[email protected]> 123456789
                  +0000\n\nBlah Blah"..., len=220)
     at sha1dc/sha1.c:1731

#3 0x0030168c in git_SHA1DCUpdate (ctx=0xffbf8260, vdata=0x398aa0,
len=273) at sha1dc/sha1.c:1808
#4 0x002a6f7c in write_sha1_file_prepare (buf=0x398aa0, len=273,
type=0x959c8 "commit", sha1=0xffbfd630 "",

     hdr=0xffbf8c28 "commit 273", hdrlen=0xffbf8c24) at sha1_file.c:3207

#5 0x002a71ac in hash_sha1_file (buf=0x398aa0, len=273, type=0x959c8
"commit", sha1=0xffbfd630 "") at sha1_file.c:3266
#6 0x002a25f8 in check_sha1_signature (sha1=0xffbfdbb8
"\375\067\356\337\002", map=0x398aa0, size=273, type=0x959c8 "commit")

     at sha1_file.c:1644

#7 0x0022816c in parse_object (sha1=0xffbfdbb8
"\375\067\356\337\002") at object.c:269
#8 0x0027c258 in get_reference (revs=0xffbfdc88, name=0xa87f0 "HEAD",
sha1=0xffbfdbb8 "\375\067\356\337\002", flags=0)

     at revision.c:196

#9 0x00284714 in setup_revisions (argc=0, argv=0x0, revs=0xffbfdc88,
opt=0xffbfdc74) at revision.c:2295
#10 0x002ee4d8 in wt_status_collect_changes_index (s=0x348ea8
<s.24114>) at wt-status.c:585
#11 0x002eeae8 in wt_status_collect (s=0x348ea8 <s.24114>) at
wt-status.c:701
#12 0x000db4fc in cmd_status (argc=0, argv=0xffbfe6fc, prefix=0x0) at
builtin/commit.c:1393
#13 0x000acc24 in run_builtin (p=0x340ccc <commands+1200>, argc=1,
argv=0xffbfe6fc) at git.c:371
#14 0x000acf98 in handle_builtin (argc=1, argv=0xffbfe6fc) at git.c:572 #15 0x000ad1a4 in run_argv (argcp=0xffbfe66c, argv=0xffbfe670) at
git.c:624
#16 0x000ad3fc in cmd_main (argc=1, argv=0xffbfe6fc) at git.c:701 #17 0x001747a0 in main (argc=5, argv=0xffbfe6ec) at common-main.c:43

1. Relevant part of uname -a: "SunOS <hostname> 5.11 11.3 sun4v sparc sun4v".

#8 lore raw

PATCH 1/1 sha1dc: fix issues with a big endian platform

From: Marc Stevens <[email protected]>

Some big-endian platforms define _BIG_ENDIAN, which the test at the beginning of file has missed. Also, when the input is not aligned, some platforms trigger SIGBUS.

This change corresponds to 33a694a9 ("Fix issues with a big endian platform", 2017-05-15) in the history of the upstream repository
https://github.com/cr-marcstevens/sha1collisiondetection

---

* So here is my attempt to clarify the log message (I left the title as-is, but this change deals both with endianness and alignment requirement).

Please look it over, and then sign-off your patch ;-)

Thanks.

P.S. I wonder how often "buf" is not aligned---could we somehow optimize out memcpy when it is not necessary, or is it not worth it?

 sha1dc/sha1.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
sha1dc/sha1.c +3 -2
index 35e9dd5bf4..ae25318c47 100644
@@ -20,7 +20,7 @@
  */
 #if (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || \
     (defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __BIG_ENDIAN__)) || \
-    defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) ||  defined(__AARCH64EB__) || \
+    defined(_BIG_ENDIAN) || defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) ||  defined(__AARCH64EB__) || \
     defined(_MIPSEB) || defined(__MIPSEB) || defined(__MIPSEB__)
 
 #define SHA1DC_BIGENDIAN	1
@@ -1728,7 +1728,8 @@ void SHA1DCUpdate(SHA1_CTX* ctx, const char* buf, size_t len)
 	while (len >= 64)
 	{
 		ctx->total += 64;
-		sha1_process(ctx, (uint32_t*)(buf));
+		memcpy(ctx->buffer, buf, 64);
+		sha1_process(ctx, (uint32_t*)(ctx->buffer));
 		buf += 64;
 		len -= 64;
 	}
Signature · 1 line
2.13.0-416-g4c6b804423
#9 lore raw

Re: [PATCH] sha1dc: fix issues with a big endian platform

Quoted 9 lines · Junio C Hamano

From: Marc Stevens <[email protected]>

Some big-endian platforms define _BIG_ENDIAN, which the test at the beginning of file has missed. Also, when the input is not aligned, some platforms trigger SIGBUS.

This change corresponds to 33a694a9 ("Fix issues with a big endian platform", 2017-05-15) in the history of the upstream repository
https://github.com/cr-marcstevens/sha1collisiondetection

Then why not just update sha1dc from upstream instead of cherry-picking one patch from them? My update-sha1dc change does this:
https://github.com/git/git/compare/master...avar:update-sha1dc

This follows the instructions Jeff detailed in 28dc98e343. I solved a few merge conflicts, I think I got them right, but review of that especially welcome.

Maybe you just meant this patch for maint, but if that's the case it's not clear from your commit message or E-Mail.

Quoted 43 lines

---

* So here is my attempt to clarify the log message (I left the title as-is, but this change deals both with endianness and alignment requirement).

Please look it over, and then sign-off your patch ;-)

Thanks.

P.S. I wonder how often "buf" is not aligned---could we somehow optimize out memcpy when it is not necessary, or is it not worth it?

 sha1dc/sha1.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
sha1dc/sha1.c +3 -3
index 35e9dd5bf4..ae25318c47 100644
@@ -20,7 +20,7 @@
  */
 #if (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || \
     (defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __BIG_ENDIAN__)) || \
-    defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) ||  defined(__AARCH64EB__) || \
+    defined(_BIG_ENDIAN) || defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) ||  defined(__AARCH64EB__) || \
     defined(_MIPSEB) || defined(__MIPSEB) || defined(__MIPSEB__)
 
 #define SHA1DC_BIGENDIAN       1
@@ -1728,7 +1728,8 @@ void SHA1DCUpdate(SHA1_CTX* ctx, const char* buf, size_t len)
        while (len >= 64)
        {
                ctx->total += 64;
-               sha1_process(ctx, (uint32_t*)(buf));
+               memcpy(ctx->buffer, buf, 64);
+               sha1_process(ctx, (uint32_t*)(ctx->buffer));
                buf += 64;
                len -= 64;
        }
--

2.13.0-416-g4c6b804423

#10 lore raw

Re: [PATCH] sha1dc: fix issues with a big endian platform

Quoted 7 lines · Ævar Arnfjörð Bjarmason
Quoted 6 lines · Junio C Hamano

From: Marc Stevens <[email protected]>

Some big-endian platforms define _BIG_ENDIAN, which the test at the beginning of file has missed. Also, when the input is not aligned, some platforms trigger SIGBUS.

Quoted 6 lines
Quoted 3 lines

This change corresponds to 33a694a9 ("Fix issues with a big endian platform", 2017-05-15) in the history of the upstream repository
https://github.com/cr-marcstevens/sha1collisiondetection

Then why not just update sha1dc from upstream instead of cherry-picking one patch from them?

See the original message upthread. I did the cherry-pick simply because that was what Marc instructed the patch recipient to do ;-).

#11 lore raw

cover Use sha1collisiondetection as a submodule

Quoted 13 lines · Junio C Hamano
Quoted 11 lines · Ævar Arnfjörð Bjarmason
Quoted 9 lines · Junio C Hamano

From: Marc Stevens <[email protected]>

Some big-endian platforms define _BIG_ENDIAN, which the test at the beginning of file has missed. Also, when the input is not aligned, some platforms trigger SIGBUS.

This change corresponds to 33a694a9 ("Fix issues with a big endian platform", 2017-05-15) in the history of the upstream repository
https://github.com/cr-marcstevens/sha1collisiondetection

Quoted 5 lines
Quoted 2 lines

Then why not just update sha1dc from upstream instead of cherry-picking one patch from them?

See the original message upthread. I did the cherry-pick simply because that was what Marc instructed the patch recipient to do ;-).

Since that patch is now in Marc's upstream code we can just update to that.

While we're at it let's see if Marc will take a patch so that we can use his code as-is with some Makefile trickery of our own, instead of having to solve merge conflicts each time we update from him. I'll submit a pull request for that shortly.

And since if and when that pull request gets accepted we're at the point of being able to use the upstream code as-is & don't need to locally modify it, we can just use a submodule to track it.

Ævar Arnfjörð Bjarmason (3):
sha1dc: update from my fork of upstream
sha1dc: use sha1collisiondetection as a submodule
sha1dc: remove our old copy of the sha1dc code

 .gitmodules            |    4 +
 Makefile               |   13 +-
 hash.h                 |    2 +-
 sha1collisiondetection |    1 +
 sha1dc/LICENSE.txt     |   30 -
 sha1dc/sha1.c          | 1809 ------------------------------------------------
 sha1dc/sha1.h          |  122 ----
 sha1dc/ubc_check.c     |  363 ----------
 sha1dc/ubc_check.h     |   44 --
 sha1dc_git.c           |   19 +
 sha1dc_git.h           |   14 +
 11 files changed, 49 insertions(+), 2372 deletions(-)
 create mode 100644 .gitmodules
 create mode 160000 sha1collisiondetection
 delete mode 100644 sha1dc/LICENSE.txt
 delete mode 100644 sha1dc/sha1.c
 delete mode 100644 sha1dc/sha1.h
 delete mode 100644 sha1dc/ubc_check.c
 delete mode 100644 sha1dc/ubc_check.h
 create mode 100644 sha1dc_git.c
 create mode 100644 sha1dc_git.h
Signature · 1 line
2.13.0.303.g4ebf302169
#12 lore raw

PATCH 1/3 sha1dc: update from my fork of upstream

Upgrade the sha1dc code from my fork of the project, which has one patch I'm hoping will be integrated upstream after this series demonstrates its usefulness:

https://github.com/cr-marcstevens/sha1collisiondetection/compare/master...avar:easier-inclusion-in-other-programs

That minor change allows us to in the future just use the upstream code as-is, without any local modifications.

Now all the local modifications we've done in this directory (see the git history of sha1dc/) are done either with defines, or via the newly added sha1dc_git.[ch] files.

---
 Makefile           |  9 +++++-
 sha1dc/sha1.c      | 79 ++++++++++++++++++++++++++---------------------
 sha1dc/sha1.h      | 90 +++++++++++++++++++++++-------------------------------
 sha1dc/ubc_check.c |  9 ++++--
 sha1dc/ubc_check.h | 10 ++++--
 sha1dc_git.c       | 19 ++++++++++++
 sha1dc_git.h       | 14 +++++++++
 7 files changed, 139 insertions(+), 91 deletions(-)
 create mode 100644 sha1dc_git.c
 create mode 100644 sha1dc_git.h
Makefile +8 -1
index e35542e631..342466d83a 100644
@@ -1414,7 +1414,14 @@ else
 	DC_SHA1 := YesPlease
 	LIB_OBJS += sha1dc/sha1.o
 	LIB_OBJS += sha1dc/ubc_check.o
-	BASIC_CFLAGS += -DSHA1_DC
+	BASIC_CFLAGS += \
+		-DSHA1_DC \
+		-DSHA1DC_NO_STANDARD_INCLUDES \
+		-DSHA1DC_INIT_SAFE_HASH_DEFAULT=0 \
+		-DSHA1DC_CUSTOM_INCLUDE_SHA1_C="\"cache.h\"" \
+		-DSHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_C="\"../sha1dc_git.c\"" \
+		-DSHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_H="\"../sha1dc_git.h\"" \
+		-DSHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C="\"git-compat-util.h\""
 endif
 endif
 endif
sha1dc/sha1.c +45 -34

137 lines not on this page · raw message

sha1dc/sha1.h +39 -51

136 lines not on this page · raw message

sha1dc/ubc_check.c +7 -2
index 089dd4743d..0614926648 100644
@@ -24,8 +24,13 @@
 // ubc_check has been verified against ubc_check_verify using the 'ubc_check_test' program in the tools section
 */
 
-#include "git-compat-util.h"
-#include "sha1dc/ubc_check.h"
+#ifndef SHA1DC_NO_STANDARD_INCLUDES
+#include <stdint.h>
+#endif
+#ifdef SHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C
+#include SHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C
+#endif
+#include "ubc_check.h"
 
 static const uint32_t DV_I_43_0_bit 	= (uint32_t)(1) << 0;
 static const uint32_t DV_I_44_0_bit 	= (uint32_t)(1) << 1;
sha1dc/ubc_check.h +7 -3
index b64c306d77..f6bb62547a 100644
@@ -20,13 +20,17 @@
 // thus one needs to do the recompression check for each DV that has its bit set
 */
 
-#ifndef UBC_CHECK_H
-#define UBC_CHECK_H
+#ifndef SHA1DC_UBC_CHECK_H
+#define SHA1DC_UBC_CHECK_H
 
 #if defined(__cplusplus)
 extern "C" {
 #endif
 
+#ifndef SHA1DC_NO_STANDARD_INCLUDES
+#include <stdint.h>
+#endif
+
 #define DVMASKSIZE 1
 typedef struct { int dvType; int dvK; int dvB; int testt; int maski; int maskb; uint32_t dm[80]; } dv_info_t;
 extern dv_info_t sha1_dvs[];
@@ -41,4 +45,4 @@ void ubc_check(const uint32_t W[80], uint32_t dvmask[DVMASKSIZE]);
 }
 #endif
 
-#endif /* UBC_CHECK_H */
+#endif
sha1dc_git.c new file +19 -0
new file mode 100644
index 0000000000..e59c1d8103
@@ -0,0 +1,19 @@
+void git_SHA1DCFinal(unsigned char hash[20], SHA1_CTX *ctx)
+{
+	if (!SHA1DCFinal(hash, ctx))
+		return;
+	die("SHA-1 appears to be part of a collision attack: %s",
+	    sha1_to_hex(hash));
+}
+
+void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *vdata, unsigned long len)
+{
+	const char *data = vdata;
+	/* We expect an unsigned long, but sha1dc only takes an int */
+	while (len > INT_MAX) {
+		SHA1DCUpdate(ctx, data, INT_MAX);
+		data += INT_MAX;
+		len -= INT_MAX;
+	}
+	SHA1DCUpdate(ctx, data, len);
+}
sha1dc_git.h new file +14 -0
new file mode 100644
index 0000000000..801b5a1fcb
@@ -0,0 +1,14 @@
+/*
+ * Same as SHA1DCFinal, but convert collision attack case into a verbose die().
+ */
+void git_SHA1DCFinal(unsigned char [20], SHA1_CTX *);
+
+/*
+ * Same as SHA1DCUpdate, but adjust types to match git's usual interface.
+ */
+void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *data, unsigned long len);
+
+#define platform_SHA_CTX SHA1_CTX
+#define platform_SHA1_Init SHA1DCInit
+#define platform_SHA1_Update git_SHA1DCUpdate
+#define platform_SHA1_Final git_SHA1DCFinal
Signature · 1 line
2.13.0.303.g4ebf302169
#13 lore raw

PATCH 2/3 sha1dc: use sha1collisiondetection as a submodule

In reply to Ævar Arnfjörð Bjarmason

Now that we don't need local modifications to the code anymore with my avar/sha1collisiondetection:easier-inclusion-in-other-programs branch we can use the upstream code-as is.

If and when this change gets accepted by upstream we should of course point to the https://github.com/cr-marcstevens/sha1collisiondetection
master branch instead.

---
 .gitmodules            | 4 ++++
 Makefile               | 8 ++++----
 hash.h                 | 2 +-
 sha1collisiondetection | 1 +
 4 files changed, 10 insertions(+), 5 deletions(-)
 create mode 100644 .gitmodules
 create mode 160000 sha1collisiondetection
.gitmodules new file +4 -0
new file mode 100644
index 0000000000..b60238a087
@@ -0,0 +1,4 @@
+[submodule "sha1collisiondetection"]
+	path = sha1collisiondetection
+	url = https://github.com/avar/sha1collisiondetection.git
+	branch = easier-inclusion-in-other-programs
Makefile +4 -4
index 342466d83a..63223fd02f 100644
@@ -1412,15 +1412,15 @@ ifdef APPLE_COMMON_CRYPTO
 	BASIC_CFLAGS += -DSHA1_APPLE
 else
 	DC_SHA1 := YesPlease
-	LIB_OBJS += sha1dc/sha1.o
-	LIB_OBJS += sha1dc/ubc_check.o
+	LIB_OBJS += sha1dc/lib/sha1.o
+	LIB_OBJS += sha1dc/lib/ubc_check.o
 	BASIC_CFLAGS += \
 		-DSHA1_DC \
 		-DSHA1DC_NO_STANDARD_INCLUDES \
 		-DSHA1DC_INIT_SAFE_HASH_DEFAULT=0 \
 		-DSHA1DC_CUSTOM_INCLUDE_SHA1_C="\"cache.h\"" \
-		-DSHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_C="\"../sha1dc_git.c\"" \
-		-DSHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_H="\"../sha1dc_git.h\"" \
+		-DSHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_C="\"../../sha1dc_git.c\"" \
+		-DSHA1DC_CUSTOM_TRAILING_INCLUDE_SHA1_H="\"../../sha1dc_git.h\"" \
 		-DSHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C="\"git-compat-util.h\""
 endif
 endif
hash.h +1 -1
index a11fc9233f..e585237136 100644
@@ -8,7 +8,7 @@
 #elif defined(SHA1_OPENSSL)
 #include <openssl/sha.h>
 #elif defined(SHA1_DC)
-#include "sha1dc/sha1.h"
+#include "sha1dc/lib/sha1.h"
 #else /* SHA1_BLK */
 #include "block-sha1/sha1.h"
 #endif
sha1collisiondetection new file +1 -0
new file mode 160000
index 0000000000..5423dd8bcf
@@ -0,0 +1 @@
+Subproject commit 5423dd8bcf3cdbd029dfa08d5d5f6dc044f7ac6d
Signature · 1 line
2.13.0.303.g4ebf302169
#14 lore raw

Re: [PATCH/RFC 0/3] Use sha1collisiondetection as a submodule

In reply to Ævar Arnfjörð Bjarmason
Quoted 23 lines · Ævar Arnfjörð Bjarmason
Quoted 13 lines · Junio C Hamano
Quoted 11 lines · Ævar Arnfjörð Bjarmason
Quoted 9 lines · Junio C Hamano

From: Marc Stevens <[email protected]>

Some big-endian platforms define _BIG_ENDIAN, which the test at the beginning of file has missed. Also, when the input is not aligned, some platforms trigger SIGBUS.

This change corresponds to 33a694a9 ("Fix issues with a big endian platform", 2017-05-15) in the history of the upstream repository
https://github.com/cr-marcstevens/sha1collisiondetection

Quoted 5 lines
Quoted 2 lines

Then why not just update sha1dc from upstream instead of cherry-picking one patch from them?

See the original message upthread. I did the cherry-pick simply because that was what Marc instructed the patch recipient to do ;-).

Since that patch is now in Marc's upstream code we can just update to that.

Quoted 8 lines

While we're at it let's see if Marc will take a patch so that we can use his code as-is with some Makefile trickery of our own, instead of having to solve merge conflicts each time we update from him. I'll submit a pull request for that shortly.

And since if and when that pull request gets accepted we're at the point of being able to use the upstream code as-is & don't need to locally modify it, we can just use a submodule to track it.

As someone who works on submodules: YAY! This sounds wonderful in the sense that more git developers experience the sharp edges (if any) of submodules and would fix them.

With the reset work on submodules (checkout, reset, ls-files, grep) and more in the works (better push/pull) we may be close to prime time.

However we do distribute tarballs (well, Junio does), which is not yet supported to include submodules.

I did follow the SHA1DC discussion just from the sideline, how crucial is that library for us?

Also now that we discuss having submodules:
Would we just point the submodule URL to github\
and call it a day?

We could make a friendly fork of it and put it next to all the repositories of https://git-blame.blogspot.com/p/git-public-repositories.html
and then use relative URLs in the .gitmodules file.

On a tangent, in an off-list discussion we discussed having the git-annex tests as an optional submodule instead of an "external" test, but in conclusion we considered that idea not ideal to implement (because our tests are like contracts, we should actually write our own tests and not rely on downstream to write tests for us)

looking forward for a discussion here,
Stefan

#15 lore raw

Re: [PATCH/RFC 0/3] Use sha1collisiondetection as a submodule

Quoted 30 lines · Stefan Beller
Quoted 28 lines · Ævar Arnfjörð Bjarmason
Quoted 13 lines · Junio C Hamano
Quoted 11 lines · Ævar Arnfjörð Bjarmason
Quoted 9 lines · Junio C Hamano

From: Marc Stevens <[email protected]>

Some big-endian platforms define _BIG_ENDIAN, which the test at the beginning of file has missed. Also, when the input is not aligned, some platforms trigger SIGBUS.

This change corresponds to 33a694a9 ("Fix issues with a big endian platform", 2017-05-15) in the history of the upstream repository
https://github.com/cr-marcstevens/sha1collisiondetection

Quoted 5 lines
Quoted 2 lines

Then why not just update sha1dc from upstream instead of cherry-picking one patch from them?

See the original message upthread. I did the cherry-pick simply because that was what Marc instructed the patch recipient to do ;-).

Since that patch is now in Marc's upstream code we can just update to that.

While we're at it let's see if Marc will take a patch so that we can use his code as-is with some Makefile trickery of our own, instead of having to solve merge conflicts each time we update from him. I'll submit a pull request for that shortly.

Quoted 7 lines
Quoted 3 lines

And since if and when that pull request gets accepted we're at the point of being able to use the upstream code as-is & don't need to locally modify it, we can just use a submodule to track it.

As someone who works on submodules: YAY! This sounds wonderful in the sense that more git developers experience the sharp edges (if any) of submodules and would fix them.

Yeah I agree git.git should dogfood submodules, and this seemed like a perfect opportunity for thaht.

As noted later in your mail everything may not be ready, so when I submit a non-RFC version of this (after upstream pulls my changes, hopefully), the 2nd and 3rd patches can just be dropped.

Quoted 6 lines

With the reset work on submodules (checkout, reset, ls-files, grep) and more in the works (better push/pull) we may be close to prime time.

However we do distribute tarballs (well, Junio does), which is not yet supported to include submodules.

Ouch. So there's no non-manual way with git-archive to make a tarball release of a git project using submodules?

Quoted 2 lines

I did follow the SHA1DC discussion just from the sideline, how crucial is that library for us?

Since 2.13.0 it's git's default sha1 implementation.

Quoted 7 lines

Also now that we discuss having submodules:
Would we just point the submodule URL to github\
and call it a day?

We could make a friendly fork of it and put it next to all the repositories of https://git-blame.blogspot.com/p/git-public-repositories.html
and then use relative URLs in the .gitmodules file.

Wouldn't we lose the ability to just "pull" the module to see if upstream changed, i.e. now we'd need to manually munge the URL first.

If so having a fetchurl be a relative URL similar to git-push's pushurl would solve it wouldn't it? I.e. a project like git could say "here's my mirrors" different from "here's my upstream".

#16 lore raw

Re: [PATCH/RFC 0/3] Use sha1collisiondetection as a submodule

Quoted 50 lines · Ævar Arnfjörð Bjarmason
Quoted 30 lines · Stefan Beller
Quoted 28 lines · Ævar Arnfjörð Bjarmason
Quoted 13 lines · Junio C Hamano
Quoted 11 lines · Ævar Arnfjörð Bjarmason
Quoted 9 lines · Junio C Hamano

From: Marc Stevens <[email protected]>

Some big-endian platforms define _BIG_ENDIAN, which the test at the beginning of file has missed. Also, when the input is not aligned, some platforms trigger SIGBUS.

This change corresponds to 33a694a9 ("Fix issues with a big endian platform", 2017-05-15) in the history of the upstream repository
https://github.com/cr-marcstevens/sha1collisiondetection

Quoted 5 lines
Quoted 2 lines

Then why not just update sha1dc from upstream instead of cherry-picking one patch from them?

See the original message upthread. I did the cherry-pick simply because that was what Marc instructed the patch recipient to do ;-).

Since that patch is now in Marc's upstream code we can just update to that.

While we're at it let's see if Marc will take a patch so that we can use his code as-is with some Makefile trickery of our own, instead of having to solve merge conflicts each time we update from him. I'll submit a pull request for that shortly.

Quoted 7 lines
Quoted 3 lines

And since if and when that pull request gets accepted we're at the point of being able to use the upstream code as-is & don't need to locally modify it, we can just use a submodule to track it.

As someone who works on submodules: YAY! This sounds wonderful in the sense that more git developers experience the sharp edges (if any) of submodules and would fix them.

Yeah I agree git.git should dogfood submodules, and this seemed like a perfect opportunity for thaht.

As noted later in your mail everything may not be ready, so when I submit a non-RFC version of this (after upstream pulls my changes, hopefully), the 2nd and 3rd patches can just be dropped.

Quoted 4 lines

With the reset work on submodules (checkout, reset, ls-files, grep) and more in the works (better push/pull) we may be close to prime time.

Quoted 5 lines
Quoted 2 lines

However we do distribute tarballs (well, Junio does), which is not yet supported to include submodules.

Ouch. So there's no non-manual way with git-archive to make a tarball release of a git project using submodules?

Yes. Ouch.

Quoted 6 lines
Quoted 3 lines

We could make a friendly fork of it and put it next to all the repositories of https://git-blame.blogspot.com/p/git-public-repositories.html
and then use relative URLs in the .gitmodules file.

Wouldn't we lose the ability to just "pull" the module to see if upstream changed, i.e. now we'd need to manually munge the URL first.

I assumed more people would fetch git and trust the community rather than people checking if SHA1DC is up-to-date, so I thought it is easier for the minority to rewrite a URL.

Quoted 3 lines

If so having a fetchurl be a relative URL similar to git-push's pushurl would solve it wouldn't it? I.e. a project like git could say "here's my mirrors" different from "here's my upstream".

I would think so.

#17 lore raw

Re: [PATCH/RFC 0/3] Use sha1collisiondetection as a submodule

In reply to Stefan Beller
Quoted 30 lines · Stefan Beller
Quoted 28 lines · Ævar Arnfjörð Bjarmason
Quoted 13 lines · Junio C Hamano
Quoted 11 lines · Ævar Arnfjörð Bjarmason
Quoted 9 lines · Junio C Hamano

From: Marc Stevens <[email protected]>

Some big-endian platforms define _BIG_ENDIAN, which the test at the beginning of file has missed. Also, when the input is not aligned, some platforms trigger SIGBUS.

This change corresponds to 33a694a9 ("Fix issues with a big endian platform", 2017-05-15) in the history of the upstream repository
https://github.com/cr-marcstevens/sha1collisiondetection

Quoted 5 lines
Quoted 2 lines

Then why not just update sha1dc from upstream instead of cherry-picking one patch from them?

See the original message upthread. I did the cherry-pick simply because that was what Marc instructed the patch recipient to do ;-).

Since that patch is now in Marc's upstream code we can just update to that.

While we're at it let's see if Marc will take a patch so that we can use his code as-is with some Makefile trickery of our own, instead of having to solve merge conflicts each time we update from him. I'll submit a pull request for that shortly.

Quoted 7 lines
Quoted 3 lines

And since if and when that pull request gets accepted we're at the point of being able to use the upstream code as-is & don't need to locally modify it, we can just use a submodule to track it.

As someone who works on submodules: YAY! This sounds wonderful in the sense that more git developers experience the sharp edges (if any) of submodules and would fix them.

Haha, I wish I had your enthusiasm for this. That is a good argument for us to include them in git.git, but I still feel apprehensive. There is still so much to do in submodule land! We'll make them great though!

Quoted 28 lines

With the reset work on submodules (checkout, reset, ls-files, grep) and more in the works (better push/pull) we may be close to prime time.

However we do distribute tarballs (well, Junio does), which is not yet supported to include submodules.

I did follow the SHA1DC discussion just from the sideline, how crucial is that library for us?

Also now that we discuss having submodules:
Would we just point the submodule URL to github\
and call it a day?

We could make a friendly fork of it and put it next to all the repositories of https://git-blame.blogspot.com/p/git-public-repositories.html
and then use relative URLs in the .gitmodules file.

On a tangent, in an off-list discussion we discussed having the git-annex tests as an optional submodule instead of an "external" test, but in conclusion we considered that idea not ideal to implement (because our tests are like contracts, we should actually write our own tests and not rely on downstream to write tests for us)

looking forward for a discussion here,
Stefan

Signature · 1 line
Brandon Williams
#18 lore raw

Re: [PATCH] sha1dc: fix issues with a big endian platform

In reply to Junio C Hamano

Hi Junio,

Quoted 14 lines · Junio C Hamano
sha1dc/sha1.c +1 -1
index 35e9dd5bf4..ae25318c47 100644
@@ -20,7 +20,7 @@
  */
 #if (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || \
     (defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __BIG_ENDIAN__)) || \
-    defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) ||  defined(__AARCH64EB__) || \
+    defined(_BIG_ENDIAN) || defined(__BIG_ENDIAN__) || defined(__ARMEB__) || defined(__THUMBEB__) ||  defined(__AARCH64EB__) || \
     defined(_MIPSEB) || defined(__MIPSEB) || defined(__MIPSEB__)
 
 #define SHA1DC_BIGENDIAN	1

sha1dc/sha1.c void SHA1DCUpdate(SHA1_CTX* ctx, const char* buf, size_t len) · line 1733

Quoted 7 lines
sha1dc/sha1.c +2 -1
@@ -1728,7 +1728,8 @@ void SHA1DCUpdate(SHA1_CTX* ctx, const char* buf, size_t len)
 	while (len >= 64)
 	{
 		ctx->total += 64;
-		sha1_process(ctx, (uint32_t*)(buf));
+		memcpy(ctx->buffer, buf, 64);
+		sha1_process(ctx, (uint32_t*)(ctx->buffer));
 		buf += 64;

There is actually a discussion going on about this. See

https://github.com/cr-marcstevens/sha1collisiondetection/commit/33a694a9ee1b79c24be45f9eab5ac0e1aeeaf271#commitcomment-22158243

for details.

The fixup commit under discussion is here:

https://github.com/cr-marcstevens/sha1collisiondetection/commit/40f07a0c12d525b24ac1235ee8a81bbf33957ca5

Ciao,
dscho