archive: flush deflate stream until Z_STREAM_END
PATCH 1/1 archive: flush deflate stream until Z_STREAM_END
In archive-zip.c:write_zip_entry() when using a stream as input for deflating a file, the call to git_deflate() with Z_FINISH always expects Z_STREAM_END to be returned. Per zlib documentation[1]:
If the parameter flush is set to Z_FINISH, pending input is processed, pending output is flushed and deflate returns with Z_STREAM_END if there was enough output space. If deflate returns with Z_OK or Z_BUF_ERROR, this function must be called again with Z_FINISH and more output space (updated avail_out) but no more input data, until it returns with Z_STREAM_END or an error. After deflate has returned Z_STREAM_END, the only possible operations on the stream are deflateReset or deflateEnd.
In scenarios where the output buffer is not large enough to write all the compressed data, it is perfectly valid for the underlying deflate() to return Z_OK. Thus, expecting a single pass of deflate() here to always return Z_STREAM_END is a bug. Update the code to flush the deflate stream until Z_STREAM_END is returned.
[1]: https://zlib.net/manual.html
---
Greetings,
At GitLab, we received a report of a user getting the following error when generating a zip archive of their repository via git-archive(1):
fatal: deflate error (0)
I've so far only been able to reproduce this issue in the chromium.git repository with a specific file:
git clone --depth=1 https://github.com/chromium/chromium.git cd chromium git -c core.bigFileThreshold=1 archive -o foo.zip --format=zip HEAD -- \ ui/events/ozone/evdev/touch_filter/palm_model/onedevice_train_palm_detection_filter_inference.cc
In the above example, core.bigFileThreshold is set to a low value to cause more files to use a stream as input while being deflated. This is the codepath that produces the specific error.
I've tested the patch against this specific file, and it fixes the issue, but I'm uncertain how to reproduce and test this issue more generically. I'm open to suggestions if anyone has some ideas :)
Thanks,
-Justin
--- archive-zip.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
archive-zip.c +14 -6
@@ -492,14 +492,22 @@ static int write_zip_entry(struct archiver_args *args, zstream.next_in = buf; zstream.avail_in =0 ; - result =git_deflate (&zstream,Z_FINISH ); -if (result !=Z_STREAM_END ) -die ("deflate error (%d)" , result); + +do { + result =git_deflate (&zstream,Z_FINISH ); +if (result !=Z_OK && result !=Z_STREAM_END ) +die ("deflate error (%d)" , result); + + out_len = zstream.next_out - compressed; +if (out_len >0 ) { +write_or_die (1 , compressed, out_len); + compressed_size += out_len; + zstream.next_out = compressed; + zstream.avail_out =sizeof (compressed); + } + }while (result !=Z_STREAM_END );git_deflate_end (&zstream); - out_len = zstream.next_out - compressed; -write_or_die (1 , compressed, out_len); - compressed_size += out_len; zip_offset += compressed_size;write_zip_data_desc (size, compressed_size, crc);
Signature · 1 line
2.50.1.214.ga30f80fde9
Quoted 14 lines · Justin Tobler
In
archive-zip.c:write_zip_entry()when using a stream as input for deflating a file, the call togit_deflate()withZ_FINISHalways expectsZ_STREAM_ENDto be returned. Per zlib documentation[1]:If the parameter flush is set to
Z_FINISH, pending input is processed, pending output is flushed and deflate returns withZ_STREAM_ENDif there was enough output space. If deflate returns withZ_OKorZ_BUF_ERROR, this function must be called again withZ_FINISHand more output space (updatedavail_out) but no more input data, until it returns withZ_STREAM_ENDor an error. After deflate has returnedZ_STREAM_END, the only possible operations on the stream are deflateReset or deflateEnd.
Quoted 7 lines
In scenarios where the output buffer is not large enough to write all the compressed data, it is perfectly valid for the underlying
deflate()to returnZ_OK. Thus, expecting a single pass ofdeflate()here to always returnZ_STREAM_ENDis a bug. Update the code to flush the deflate stream untilZ_STREAM_ENDis returned.
Good find. I guess back then I thought making the output buffer twice as big as the input buffer was sufficient, as deflateBound() guarantees compression is possible with a much lower ratio. But this doesn't take the internal state of a stream into account. Oof!
Quoted 19 lines
Helped-by Toon Claes <[email protected]>Signed-off-by Justin Tobler <[email protected]>---
Greetings,
At GitLab, we received a report of a user getting the following error when generating a zip archive of their repository via git-archive(1):
fatal: deflate error (0)
I've so far only been able to reproduce this issue in the chromium.git repository with a specific file:
git clone --depth=1 https://github.com/chromium/chromium.git cd chromium git -c core.bigFileThreshold=1 archive -o foo.zip --format=zip HEAD -- \ ui/events/ozone/evdev/touch_filter/palm_model/onedevice_train_palm_detection_filter_inference.cc
Quoted 7 lines
In the above example,
core.bigFileThresholdis set to a low value to cause more files to use a stream as input while being deflated. This is the codepath that produces the specific error.I've tested the patch against this specific file, and it fixes the issue, but I'm uncertain how to reproduce and test this issue more generically. I'm open to suggestions if anyone has some ideas :)
Not sure how to fill up zlib's pending buffer most efficiently. Reducing the size of the output buffer would make the bug easier to trigger, though.
Quoted 27 lines
Thanks,
(quoted diff) +0
-2-Justin --- archive-zip.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)archive-zip.c +9
-3@@ -492,14 +492,22 @@ static int write_zip_entry(struct archiver_args *args, zstream.next_in = buf; zstream.avail_in =0 ; - result =git_deflate (&zstream,Z_FINISH ); -if (result !=Z_STREAM_END ) -die ("deflate error (%d)" , result); + + do { + result = git_deflate(&zstream, Z_FINISH); + if (result != Z_OK && result != Z_STREAM_END) + die("deflate error (%d)", result); + + out_len = zstream.next_out - compressed; + if (out_len > 0) { + write_or_die(1, compressed, out_len);
archive-zip.c static int write_zip_entry(struct archiver_args *args, · line 502
Quoted 10 lines
archive-zip.c +5
-3+ compressed_size += out_len; + zstream.next_out = compressed; + zstream.avail_out =sizeof (compressed); + } + }while (result !=Z_STREAM_END );git_deflate_end (&zstream); - out_len = zstream.next_out - compressed; -write_or_die (1 , compressed, out_len); - compressed_size += out_len;
Looks good. Could probably rolled into the first loop, but that just would make this fix more complicated.
Quoted 3 lines
archive-zip.c +0
-0zip_offset += compressed_size;write_zip_data_desc (size, compressed_size, crc);
Quoted 8 lines · René Scharfe
Quoted 3 lines · Justin Tobler
I've tested the patch against this specific file, and it fixes the issue, but I'm uncertain how to reproduce and test this issue more generically. I'm open to suggestions if anyone has some ideas :)
Not sure how to fill up zlib's pending buffer most efficiently. Reducing the size of the output buffer would make the bug easier to trigger, though.
Ya, I was able to trigger this issue more frequently by making the output buffer smaller than the input buffer. I was really hoping though to find a way to reproduce this without code changes so we could add a test. Not sure if that is really feasible though in this case.
Quoted 26 lines
Quoted 26 lines
archive-zip.c +14
-3@@ -492,14 +492,22 @@ static int write_zip_entry(struct archiver_args *args, zstream.next_in = buf; zstream.avail_in =0 ; - result =git_deflate (&zstream,Z_FINISH ); -if (result !=Z_STREAM_END ) -die ("deflate error (%d)" , result); + +do { + result =git_deflate (&zstream,Z_FINISH ); +if (result !=Z_OK && result !=Z_STREAM_END ) +die ("deflate error (%d)" , result); + + out_len = zstream.next_out - compressed; +if (out_len >0 ) { +write_or_die (1 , compressed, out_len); + compressed_size += out_len; + zstream.next_out = compressed; + zstream.avail_out =sizeof (compressed); + } + }while (result !=Z_STREAM_END );
Quoted 7 lines
archive-zip.c static int write_zip_entry(struct archiver_args *args, · line 502
Quoted 4 lines
archive-zip.c +0
-3git_deflate_end (&zstream); - out_len = zstream.next_out - compressed; -write_or_die (1 , compressed, out_len); - compressed_size += out_len;Looks good. Could probably rolled into the first loop, but that just would make this fix more complicated.
I was also considering rolling this into the first loop, but ultimately went with the minimal patch to fix the issue. I don't mind rerolling if we prefer it the other way though. :)
Thanks for the review!
-Justin
Quoted 9 lines · Justin Tobler
Quoted 8 lines · René Scharfe
Quoted 3 lines · Justin Tobler
I've tested the patch against this specific file, and it fixes the issue, but I'm uncertain how to reproduce and test this issue more generically. I'm open to suggestions if anyone has some ideas :)
Not sure how to fill up zlib's pending buffer most efficiently. Reducing the size of the output buffer would make the bug easier to trigger, though.
Yes, in my local testing I also have been reducing the size of the output buffer.
Quoted 4 lines
Ya, I was able to trigger this issue more frequently by making the output buffer smaller than the input buffer. I was really hoping though to find a way to reproduce this without code changes so we could add a test. Not sure if that is really feasible though in this case.
I think it's really nice you were able find a repository that reproduces this issue. Having an existing example in the "wild" is better proof to verify this fix work.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
I've so far only been able to reproduce this issue in the chromium.git repository with a specific file:
git clone --depth=1 https://github.com/chromium/chromium.git cd chromium git -c core.bigFileThreshold=1 archive -o foo.zip --format=zip HEAD -- \ ui/events/ozone/evdev/touch_filter/palm_model/onedevice_train_palm_detection_filter_inference.cc
Weird, for me this file didn't trigger the error, but the following file did:
chrome/test/data/third_party/kraken/tests/kraken-1.1/imaging-darkroom-data.js
But also with this file, I can confirm this fix works.
Quoted 34 lines
Quoted 26 lines
Quoted 26 lines
archive-zip.c +14
-3@@ -492,14 +492,22 @@ static int write_zip_entry(struct archiver_args *args, zstream.next_in = buf; zstream.avail_in =0 ; - result =git_deflate (&zstream,Z_FINISH ); -if (result !=Z_STREAM_END ) -die ("deflate error (%d)" , result); + +do { + result =git_deflate (&zstream,Z_FINISH ); +if (result !=Z_OK && result !=Z_STREAM_END ) +die ("deflate error (%d)" , result); + + out_len = zstream.next_out - compressed; +if (out_len >0 ) { +write_or_die (1 , compressed, out_len); + compressed_size += out_len; + zstream.next_out = compressed; + zstream.avail_out =sizeof (compressed); + } + }while (result !=Z_STREAM_END );Quoted 7 lines
archive-zip.c static int write_zip_entry(struct archiver_args *args, · line 502
Quoted 4 lines
archive-zip.c +0
-3git_deflate_end (&zstream); - out_len = zstream.next_out - compressed; -write_or_die (1 , compressed, out_len); - compressed_size += out_len;Looks good. Could probably rolled into the first loop, but that just would make this fix more complicated.
Quoted 3 lines
I was also considering rolling this into the first loop, but ultimately went with the minimal patch to fix the issue. I don't mind rerolling if we prefer it the other way though. :)
When I was researching, I have been working on a version that rolls the changes into the first loop, but that diff is a lot more substantial. The thing is, you cannot seem to avoid introducing another (nested) loop.
My refactoring is based on the zlib example page[1] page and example source code[2], and also there two nested loops are used. Ideally, we'd modify our code so it follows the example code more closely, but I'm worried this might introduce new breakage, so it's probably not worth it.
On the other hand, I was able to trigger failure when I made the output buffer equally large as the input buffer. In this case git-archive completes successfully, but when inflated, the content is mangled. It seems my version is protected against that.
I've submitted my version too[3]. I'm on the fence which solution is better, so I'd love to get input from the both of you. So I'm happy to hear any thoughts.
[1]: https://zlib.net/zlib_how.html
[2]: https://zlib.net/zpipe.c
[3]: https://lore.kernel.org/git/[email protected]/
Signature · 2 lines
Cheers, Toon