Cryptoapi deflate fix.
cover Cryptoapi deflate fix.
The deflate module doesn't properly complete the handling of PAGE_SIZE chunks of data. This patch corrects that so that it can be used with Suspend2.
PATCH 1/2 Add cryptoapi LZF support.
Add a cryptoapi LZF module. Performance in suspending to disk can be greatly improved by a fast compression algorithm. LZF provides this. It was donated by Marc Alexander Lehmann under a dual BSD/GPL license.
crypto/Kconfig | 7 + crypto/Makefile | 1 crypto/lzf.c | 335 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 343 insertions(+), 0 deletions(-)
crypto/Kconfig +7 -0
@@ -316,6 +316,13 @@ config CRYPTO_DEFLATE You will most probably want this if using IPSec. +config CRYPTO_LZF + tristate "LZF compression algorithm" + depends on CRYPTO + help + This is the LZF algorithm. It is especially useful for Suspend2, + because it achieves good compression quickly. + config CRYPTO_MICHAEL_MIC tristate "Michael MIC keyed digest algorithm" depends on CRYPTO
crypto/Makefile +1 -0
@@ -30,5 +30,6 @@ obj-$(CONFIG_CRYPTO_ANUBIS) += anubis.o obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o obj-$(CONFIG_CRYPTO_CRC32C) += crc32c.o +obj-$(CONFIG_CRYPTO_LZF) += lzf.o obj-$(CONFIG_CRYPTO_TEST) += tcrypt.o
crypto/lzf.c new file +335 -0
335 lines not on this page · raw message
--
Nigel Cunningham nigel at suspend2 dot net
PATCH 2/2 Make cryptoapi deflate module handle full pages.
In reply to Nigel CunninghamThe cryptoapi deflate module currently returns an error when it successfully handles a full page of data at the end of a compressed stream. This patch addresses that behaviour.
crypto/deflate.c | 11 +++++++++-- 1 files changed, 9 insertions(+), 2 deletions(-)
crypto/deflate.c +9 -2
@@ -142,8 +142,15 @@ static int deflate_compress(void *ctx, c ret =zlib_deflate (stream,Z_FINISH );if (ret !=Z_STREAM_END ) { - ret = -EINVAL ; - goto out; +if (!(ret ==Z_OK && !stream->avail_in && !stream->avail_out)) { + ret = -EINVAL ; + goto out; + }else { +u8 zerostuff =0 ; + stream->next_out = &zerostuff; + stream->avail_out =1 ; + ret =zlib_deflate (stream,Z_FINISH ); + } } ret =0 ; *dlen = stream->total_out;
--
Nigel Cunningham nigel at suspend2 dot net
Quoted 4 lines · Nigel Cunningham
The deflate module doesn't properly complete the handling of
PAGE_SIZEchunks of data. This patch corrects that so that it can be used with Suspend2.
Well, it also adds the LZF support to the cryptoapi. These are two different things.
If you think the deflate modules needs to be fixed, you should post the patch for it separately, I think.
Rafael
Hi.
Quoted 5 lines · Rafael J. Wysocki
Quoted 3 lines · Nigel Cunningham
The deflate module doesn't properly complete the handling of
PAGE_SIZEchunks of data. This patch corrects that so that it can be used with Suspend2.
Quoted 7 lines
Well, it also adds the LZF support to the cryptoapi. These are two different things.
If you think the deflate modules needs to be fixed, you should post the patch for it separately, I think.
Rafael
Sorry for the bad choice of heading. I have posted this before and emailed it direct to Herbert, but he (rightly) doesn't see the need while suspend2 isn't merged.
Regards,
Nigel
Signature · 2 lines
See http://www.suspend2.net for Howtos, FAQs, mailing lists, wiki and bugzilla info.
Herbert Xu
maintainerQuoted 4 lines · Nigel Cunningham
Sorry for the bad choice of heading. I have posted this before and emailed it direct to Herbert, but he (rightly) doesn't see the need while suspend2 isn't merged.
Hmm, I don't recall ever telling you that.
I searched through my mail archive here is what I said to you last year on these two patches. As far as I can see I don't have a reply from you on either subject. So if you could reply to my questions below then I can consider your patches again.
1. Deflate fix:
Quoted 7 lines
Quoted 3 lines
Hi Nigel, I need a bit more information about this patch. Do you have a specific input stream that requires a fix like this?
Yes, Suspend2 if the user selects deflate as their compressor. The output data will be
PAGE_SIZEchunks, but deflate sometimes thinks it has an extra byte to give us back.
Are you saying that you're feeding it PAGE_SIZE chunks and it's giving you back something bigger than a page? That is expected since the nature of compression in general is that not everything is compressible. Data streams which are not compressible will end up bigger than what you start with.
What would be a bug is if you feed it something that's deflateBound^-1(PAGE_SIZE) bytes long and it spits back something that's longer than a page.
Quoted 3 lines
I agree that it's ugly and don't recall using it when I had gzip support in an earlier version of Suspend2. Are you thinking there might be a better way? If so, I can dig out the old (non crypto api) code.
Well if you could give me a snippet of code that actually uses this stuff to compress pages then I might have a better idea of what it's trying to do.
2. LZF:
Review
Quoted 6 lines
(quoted diff) +6
-0+static int lzf_compress_init(void *context) +{ + struct lzf_ctx *ctx = (struct lzf_ctx *)context; + + /* Get LZF ready to go */ + ctx->hbuf = vmalloc_32((1 << hlog) * sizeof(char *));
Any reason why vmalloc can't be used?
Review
Quoted 5 lines
(quoted diff) +4
-0+ /* Allocate local buffer */ + ctx->local_buffer = (char *)get_zeroed_page(GFP_ATOMIC); + /* Allocate page buffer */ + ctx->page_buffer = (char *)get_zeroed_page(GFP_ATOMIC);
Where are these two buffers used anyway?
Review
Quoted 3 lines
(quoted diff) +3
-0+ if (!ctx->page_buffer) { + free_page((unsigned long)ctx->local_buffer); + lzf_compress_exit(ctx);
This is a double-free of local_buffer.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtHi.
Quoted 6 lines · Herbert Xu
Quoted 3 lines · Nigel Cunningham
Sorry for the bad choice of heading. I have posted this before and emailed it direct to Herbert, but he (rightly) doesn't see the need while suspend2 isn't merged.
Hmm, I don't recall ever telling you that.
Ok. Sorry for my wonky memory then :)
Quoted 20 lines
I searched through my mail archive here is what I said to you last year on these two patches. As far as I can see I don't have a reply from you on either subject. So if you could reply to my questions below then I can consider your patches again.
1. Deflate fix:
Quoted 7 lines
Quoted 3 lines
Hi Nigel, I need a bit more information about this patch. Do you have a specific input stream that requires a fix like this?
Yes, Suspend2 if the user selects deflate as their compressor. The output data will be
PAGE_SIZEchunks, but deflate sometimes thinks it has an extra byte to give us back.Are you saying that you're feeding it
PAGE_SIZEchunks and it's giving you back something bigger than a page? That is expected since the nature of compression in general is that not everything is compressible. Data streams which are not compressible will end up bigger than what you start with.
Quoted 3 lines
What would be a bug is if you feed it something that's deflateBound^-1(PAGE_SIZE) bytes long and it spits back something that's longer than a page.
Yes, I'm always feeding it PAGE_SIZE chunks and compressing each page separately. It's a long time since I looked at or thought about this, so I'll spend some more time getting it fresh in my head if you like.
Quoted 9 lines
Quoted 3 lines
I agree that it's ugly and don't recall using it when I had gzip support in an earlier version of Suspend2. Are you thinking there might be a better way? If so, I can dig out the old (non crypto api) code.
Well if you could give me a snippet of code that actually uses this stuff to compress pages then I might have a better idea of what it's trying to do.
2. LZF:
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+static int lzf_compress_init(void *context) +{ + struct lzf_ctx *ctx = (struct lzf_ctx *)context; + + /* Get LZF ready to go */ + ctx->hbuf = vmalloc_32((1 << hlog) * sizeof(char *));Any reason why vmalloc can't be used?
I just left Marc's original code as it was, so I'm not completely sure, but I guess it's because we want lowmem.
Quoted 7 lines
Review
Quoted 5 lines
(quoted diff) +4
-0+ /* Allocate local buffer */ + ctx->local_buffer = (char *)get_zeroed_page(GFP_ATOMIC); + /* Allocate page buffer */ + ctx->page_buffer = (char *)get_zeroed_page(GFP_ATOMIC);Where are these two buffers used anyway?
Page_buffer is the destination buffer passed to the compressor. Local buffer is used to buffer the output for writing (and vv).
Quoted 5 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ if (!ctx->page_buffer) { + free_page((unsigned long)ctx->local_buffer); + lzf_compress_exit(ctx);This is a double-free of
local_buffer.
Is that from our previous correspondence? I can't find anything like it now.
Regards,
Nigel
Quoted 1 line
Cheers,
Signature · 2 lines
See http://www.suspend2.net for Howtos, FAQs, mailing lists, wiki and bugzilla info.
Herbert Xu
maintainerQuoted 2 lines · Nigel Cunningham
Ok. Sorry for my wonky memory then :)
No problems.
Quoted 3 lines
Yes, I'm always feeding it
PAGE_SIZEchunks and compressing each page separately. It's a long time since I looked at or thought about this, so I'll spend some more time getting it fresh in my head if you like.
If you could give me a minimal test case to work with that'd be great.
Quoted 2 lines
I just left Marc's original code as it was, so I'm not completely sure, but I guess it's because we want lowmem.
The question is why do you need lowmem? It doesn't appear to be obvious by looking at the code.
Quoted 3 lines
Quoted 1 line
This is a double-free of
local_buffer.Is that from our previous correspondence? I can't find anything like it now.
Yes, that was from a year ago before you fixed the code :) Feel free to resubmit this once you've addressed the vmalloc question.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt