Skip to contentSkip to conversation

Cryptoapi deflate fix.

PATCH In review Nigel Cunningham 3 participants 8 messages Jun 26, 2006 – Jun 27, 2006 Crypto API

lore
Share & tools
lore
#0 lore raw

cover Cryptoapi deflate fix.

The deflate module doesn't properly complete the handling of PAGE_SIZE chunks of data. This patch corrects that so that it can be used with Suspend2.

#1 lore raw

PATCH 1/2 Add cryptoapi LZF support.

Add a cryptoapi LZF module. Performance in suspending to disk can be greatly improved by a fast compression algorithm. LZF provides this. It was donated by Marc Alexander Lehmann under a dual BSD/GPL license.

 crypto/Kconfig  |    7 +
 crypto/Makefile |    1 
 crypto/lzf.c    |  335 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 343 insertions(+), 0 deletions(-)
crypto/Kconfig +7 -0
index c442f2e..36e0b71 100644
@@ -316,6 +316,13 @@ config CRYPTO_DEFLATE
 	  
 	  You will most probably want this if using IPSec.
 
+config CRYPTO_LZF
+	tristate "LZF compression algorithm"
+	depends on CRYPTO
+	help
+	  This is the LZF algorithm. It is especially useful for Suspend2,
+	  because it achieves good compression quickly.
+
 config CRYPTO_MICHAEL_MIC
 	tristate "Michael MIC keyed digest algorithm"
 	depends on CRYPTO
crypto/Makefile +1 -0
index d287b9e..9845a6d 100644
@@ -30,5 +30,6 @@ obj-$(CONFIG_CRYPTO_ANUBIS) += anubis.o
 obj-$(CONFIG_CRYPTO_DEFLATE) += deflate.o
 obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o
 obj-$(CONFIG_CRYPTO_CRC32C) += crc32c.o
+obj-$(CONFIG_CRYPTO_LZF) += lzf.o
 
 obj-$(CONFIG_CRYPTO_TEST) += tcrypt.o
crypto/lzf.c new file +335 -0

335 lines not on this page · raw message

--
Nigel Cunningham nigel at suspend2 dot net

#2 lore raw

PATCH 2/2 Make cryptoapi deflate module handle full pages.

In reply to Nigel Cunningham

The cryptoapi deflate module currently returns an error when it successfully handles a full page of data at the end of a compressed stream. This patch addresses that behaviour.

 crypto/deflate.c |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)
crypto/deflate.c +9 -2
index f209368..2a38593 100644
@@ -142,8 +142,15 @@ static int deflate_compress(void *ctx, c
 
 	ret = zlib_deflate(stream, Z_FINISH);
 	if (ret != Z_STREAM_END) {
-		ret = -EINVAL;
-		goto out;
+	    	if (!(ret == Z_OK && !stream->avail_in && !stream->avail_out)) {
+			ret = -EINVAL;
+			goto out;
+		} else {
+			u8 zerostuff = 0;
+			stream->next_out = &zerostuff;
+			stream->avail_out = 1; 
+			ret = zlib_deflate(stream, Z_FINISH);
+		}
 	}
 	ret = 0;
 	*dlen = stream->total_out;

--
Nigel Cunningham nigel at suspend2 dot net

#3 lore raw
In reply to Nigel Cunningham
Quoted 4 lines · Nigel Cunningham

The deflate module doesn't properly complete the handling of PAGE_SIZE chunks of data. This patch corrects that so that it can be used with Suspend2.

Well, it also adds the LZF support to the cryptoapi. These are two different things.

If you think the deflate modules needs to be fixed, you should post the patch for it separately, I think.

Rafael

#4 lore raw

Hi.

Quoted 5 lines · Rafael J. Wysocki
Quoted 3 lines · Nigel Cunningham

The deflate module doesn't properly complete the handling of PAGE_SIZE chunks of data. This patch corrects that so that it can be used with Suspend2.

Quoted 7 lines

Well, it also adds the LZF support to the cryptoapi. These are two different things.

If you think the deflate modules needs to be fixed, you should post the patch for it separately, I think.

Rafael

Sorry for the bad choice of heading. I have posted this before and emailed it direct to Herbert, but he (rightly) doesn't see the need while suspend2 isn't merged.

Regards,

Nigel

Signature · 2 lines
See http://www.suspend2.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.
maintainer
#5 lore raw
Quoted 4 lines · Nigel Cunningham

Sorry for the bad choice of heading. I have posted this before and emailed it direct to Herbert, but he (rightly) doesn't see the need while suspend2 isn't merged.

Hmm, I don't recall ever telling you that.

I searched through my mail archive here is what I said to you last year on these two patches. As far as I can see I don't have a reply from you on either subject. So if you could reply to my questions below then I can consider your patches again.

1. Deflate fix:

Quoted 7 lines
Quoted 3 lines

Hi Nigel, I need a bit more information about this patch. Do you have a specific input stream that requires a fix like this?

Yes, Suspend2 if the user selects deflate as their compressor. The output data will be PAGE_SIZE chunks, but deflate sometimes thinks it has an extra byte to give us back.

Are you saying that you're feeding it PAGE_SIZE chunks and it's giving you back something bigger than a page? That is expected since the nature of compression in general is that not everything is compressible. Data streams which are not compressible will end up bigger than what you start with.

What would be a bug is if you feed it something that's deflateBound^-1(PAGE_SIZE) bytes long and it spits back something that's longer than a page.

Quoted 3 lines

I agree that it's ugly and don't recall using it when I had gzip support in an earlier version of Suspend2. Are you thinking there might be a better way? If so, I can dig out the old (non crypto api) code.

Well if you could give me a snippet of code that actually uses this stuff to compress pages then I might have a better idea of what it's trying to do.

2. LZF:

Review

Quoted 6 lines
(quoted diff) +6 -0
+static int lzf_compress_init(void *context)
+{
+     struct lzf_ctx *ctx = (struct lzf_ctx *)context;
+
+     /* Get LZF ready to go */
+     ctx->hbuf = vmalloc_32((1 << hlog) * sizeof(char *));

Any reason why vmalloc can't be used?

Review

Quoted 5 lines
(quoted diff) +4 -0
+     /* Allocate local buffer */
+     ctx->local_buffer = (char *)get_zeroed_page(GFP_ATOMIC);
 
+     /* Allocate page buffer */
+     ctx->page_buffer = (char *)get_zeroed_page(GFP_ATOMIC);

Where are these two buffers used anyway?

Review

Quoted 3 lines
(quoted diff) +3 -0
+     if (!ctx->page_buffer) {
+             free_page((unsigned long)ctx->local_buffer);
+             lzf_compress_exit(ctx);

This is a double-free of local_buffer.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#6 lore raw

Hi.

Quoted 6 lines · Herbert Xu
Quoted 3 lines · Nigel Cunningham

Sorry for the bad choice of heading. I have posted this before and emailed it direct to Herbert, but he (rightly) doesn't see the need while suspend2 isn't merged.

Hmm, I don't recall ever telling you that.

Ok. Sorry for my wonky memory then :)

Quoted 20 lines

I searched through my mail archive here is what I said to you last year on these two patches. As far as I can see I don't have a reply from you on either subject. So if you could reply to my questions below then I can consider your patches again.

1. Deflate fix:

Quoted 7 lines
Quoted 3 lines

Hi Nigel, I need a bit more information about this patch. Do you have a specific input stream that requires a fix like this?

Yes, Suspend2 if the user selects deflate as their compressor. The output data will be PAGE_SIZE chunks, but deflate sometimes thinks it has an extra byte to give us back.

Are you saying that you're feeding it PAGE_SIZE chunks and it's giving you back something bigger than a page? That is expected since the nature of compression in general is that not everything is compressible. Data streams which are not compressible will end up bigger than what you start with.

Quoted 3 lines

What would be a bug is if you feed it something that's deflateBound^-1(PAGE_SIZE) bytes long and it spits back something that's longer than a page.

Yes, I'm always feeding it PAGE_SIZE chunks and compressing each page separately. It's a long time since I looked at or thought about this, so I'll spend some more time getting it fresh in my head if you like.

Quoted 9 lines
Quoted 3 lines

I agree that it's ugly and don't recall using it when I had gzip support in an earlier version of Suspend2. Are you thinking there might be a better way? If so, I can dig out the old (non crypto api) code.

Well if you could give me a snippet of code that actually uses this stuff to compress pages then I might have a better idea of what it's trying to do.

2. LZF:

Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+static int lzf_compress_init(void *context)
+{
+     struct lzf_ctx *ctx = (struct lzf_ctx *)context;
+
+     /* Get LZF ready to go */
+     ctx->hbuf = vmalloc_32((1 << hlog) * sizeof(char *));

Any reason why vmalloc can't be used?

I just left Marc's original code as it was, so I'm not completely sure, but I guess it's because we want lowmem.

Quoted 7 lines

Review

Quoted 5 lines
(quoted diff) +4 -0
+     /* Allocate local buffer */
+     ctx->local_buffer = (char *)get_zeroed_page(GFP_ATOMIC);
 
+     /* Allocate page buffer */
+     ctx->page_buffer = (char *)get_zeroed_page(GFP_ATOMIC);

Where are these two buffers used anyway?

Page_buffer is the destination buffer passed to the compressor. Local buffer is used to buffer the output for writing (and vv).

Quoted 5 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+     if (!ctx->page_buffer) {
+             free_page((unsigned long)ctx->local_buffer);
+             lzf_compress_exit(ctx);

This is a double-free of local_buffer.

Is that from our previous correspondence? I can't find anything like it now.

Regards,

Nigel

Quoted 1 line

Cheers,

Signature · 2 lines
See http://www.suspend2.net for Howtos, FAQs, mailing
lists, wiki and bugzilla info.
maintainer
#7 lore raw
Quoted 2 lines · Nigel Cunningham

Ok. Sorry for my wonky memory then :)

No problems.

Quoted 3 lines

Yes, I'm always feeding it PAGE_SIZE chunks and compressing each page separately. It's a long time since I looked at or thought about this, so I'll spend some more time getting it fresh in my head if you like.

If you could give me a minimal test case to work with that'd be great.

Quoted 2 lines

I just left Marc's original code as it was, so I'm not completely sure, but I guess it's because we want lowmem.

The question is why do you need lowmem? It doesn't appear to be obvious by looking at the code.

Quoted 3 lines
Quoted 1 line

This is a double-free of local_buffer.

Is that from our previous correspondence? I can't find anything like it now.

Yes, that was from a year ago before you fixed the code :) Feel free to resubmit this once you've addressed the vmalloc question.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt