LZO irreversible output?
Subsystems (3)
Hi all.
(Not sent to LKML yesterday; no reply from linux-crypto yet, so resending).
A while back now, I stopped supplying the LZF compression algorithm with TuxOnIce and made LZO the default algorithm. Around the same time, we started getting occasional errors when reading images; decompression failures.
I've finally managed to find the time to properly look at this, and have managed to find a data page that LZO compresses, but seems to be unable to decompress back to the original contents. I'm wondering whether this is because I'm doing something wrong, or because there really is some data the LZO (or the kernel implementation) can't do reversible compression on.
I've turned the test case and the way TuxOnIce invokes the compression and decompression code into a standalone kernel module (attached). On my
64 bit machine, insmoding the module results in the error path being
triggered (skipping the dump of the original page):
[52205.333463] Output 716 bytes. Result 0. [52205.333468] Compressed to: 06 f8 8f 09 00 00 ea ff ff 00 a1 00 40 7a 00 ff [52205.333473] Compressed to: ff 27 3c 00 78 01 2b 0c 00 01 30 90 09 00 7c 05 [52205.333478] Compressed to: fc 00 d0 03 3f dd 00 68 dc 06 fc 00 20 07 dd 00 [52205.333483] Compressed to: a0 dc 06 fc 00 20 07 dd 00 d8 dc 06 fc 00 20 07 [52205.333488] Compressed to: de 00 10 91 bc 1b fc 00 20 07 dd 00 48 dc 06 fc [52205.333493] Compressed to: 00 20 07 dd 00 80 dc 06 fc 00 20 07 dd 00 b8 dc [52205.333498] Compressed to: 06 fc 00 20 07 dd 00 f0 dc 06 fc 00 20 07 de 00 [52205.333503] Compressed to: 28 92 bc 22 fc 00 20 07 dd 00 60 dc 06 fc 00 20 [52205.333508] Compressed to: 07 dd 00 98 dc 06 fc 00 20 07 dd 00 d0 dc 06 fc [52205.333512] Compressed to: 00 20 07 de 00 08 93 bc 1b fc 00 20 07 dd 00 40 [52205.333517] Compressed to: dc 06 fc 00 20 07 dd 00 78 dc 06 fc 00 20 07 dd [52205.333522] Compressed to: 00 b0 dc 06 fc 00 20 07 dd 00 e8 dc 06 fc 00 20 [52205.333527] Compressed to: 07 de 00 20 94 bc 22 fc 00 20 07 dd 00 58 dc 06 [52205.333532] Compressed to: fc 00 20 07 dd 00 90 dc 06 fc 00 20 07 dd 00 c8 [52205.333537] Compressed to: dc 06 fc 00 20 07 de 00 00 95 bc 1b 27 1c 00 a0 [52205.333542] Compressed to: 07 3f 3d 13 38 dc 06 fc 00 b9 06 00 3f dd 00 70 [52205.333546] Compressed to: dc 06 fc 00 20 07 dd 00 a8 dc 06 fc 00 20 07 dd [52205.333551] Compressed to: 00 e0 dc 06 fc 00 20 07 de 00 18 96 bc 22 fc 00 [52205.333556] Compressed to: 20 07 dd 00 50 dc 06 fc 00 20 07 dd 00 88 dc 06 [52205.333561] Compressed to: fc 00 20 07 dd 00 c0 dc 06 fc 00 20 07 dd 00 f8 [52205.333566] Compressed to: dc 06 fc 00 20 07 de 00 30 97 bc 22 fc 00 20 07 [52205.333571] Compressed to: dd 00 68 dc 06 fc 00 20 07 dd 00 a0 dc 06 fc 00 [52205.333576] Compressed to: 20 07 dd 00 d8 dc 06 fc 00 20 07 de 00 10 98 bc [52205.333581] Compressed to: 1b fc 00 20 07 dd 00 48 dc 06 fc 00 20 07 dd 00 [52205.333586] Compressed to: 80 dc 06 fc 00 20 07 dd 00 b8 dc 06 fc 00 20 07 [52205.333591] Compressed to: dd 00 f0 dc 06 fc 00 20 07 de 00 28 99 bc 22 fc [52205.333596] Compressed to: 00 20 07 dd 00 60 dc 06 fc 00 20 07 dd 00 98 dc [52205.333601] Compressed to: 06 fc 00 20 07 dd 00 d0 dc 06 fc 00 20 07 de 00 [52205.333606] Compressed to: 08 9a bc 1b fc 00 20 07 dd 00 40 dc 06 fc 00 20 [52205.333611] Compressed to: 07 dd 00 78 dc 06 fc 00 20 07 dd 00 b0 dc 06 fc [52205.333616] Compressed to: 00 20 07 dd 00 e8 dc 06 fc 00 20 07 de 00 20 9b [52205.333621] Compressed to: bc 22 fc 00 20 07 dd 00 58 dc 06 fc 00 20 07 dd [52205.333625] Compressed to: 00 90 dc 06 fc 00 20 07 dd 00 c8 dc 06 fc 00 20 [52205.333630] Compressed to: 07 de 00 00 9c bc 1b 27 1c 00 a0 07 3f 1d 1b 38 [52205.333635] Compressed to: dc 06 fc 00 b8 06 20 02 fc 1b dd 06 70 dc 00 20 [52205.333640] Compressed to: 07 dd 00 a8 dc 05 fc 00 20 07 dd 00 e0 dc 06 fc [52205.333645] Compressed to: 00 20 07 de 00 18 9d bc 22 fc 00 20 07 dd 00 50 [52205.333650] Compressed to: dc 06 fc 00 20 07 dd 00 88 dc 06 fc 00 20 07 dd [52205.333655] Compressed to: 00 c0 dc 06 fc 00 20 07 dd 00 f8 dc 06 fc 00 20 [52205.333660] Compressed to: 07 de 00 30 9e bc 22 fc 00 20 07 dd 00 68 dc 06 [52205.333665] Compressed to: fc 00 20 07 dd 00 a0 dc 06 fc 00 20 07 dd 00 d8 [52205.333670] Compressed to: dc 06 fc 00 20 07 de 00 10 9f bc 1b fc 00 20 07 [52205.333675] Compressed to: dd 00 48 dc 06 fc 00 20 07 dd 00 80 dc 06 fc 00 [52205.333679] Compressed to: 20 07 dd 00 b8 dc 06 fc 00 20 07 dd 00 f0 dc 06 [52205.333684] Compressed to: 05 f0 9f 09 00 00 ea ff ff 11 00 00 [52205.333689] [52205.333691] Restored to 0 bytes, result code -22.
Would someone be willing and able to tell me what (if anything) I'm doing wrong, or whether there is something wrong with the algo or its implementation?
Thanks!
Nigel
Kconfig | 6 + Makefile | 2 lzo_test.c | 340 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 348 insertions(+)
kernel/power/Kconfig +6 -0
@@ -461,3 +461,9 @@ config PM_RUNTIME and the bus type drivers of the buses the devices are on are responsible for the actual handling of the autosuspend requests and wake-up events. + +config TOI_LZO_TEST + tristate "Test LZO" + depends on CRYPTO + ---help--- + Test LZO compression.
kernel/power/lzo_test.c +340 -0
340 lines not on this page · raw message
kernel/power/Makefile +2 -0
@@ -41,3 +41,5 @@ obj-$(CONFIG_HIBERNATION) += swsusp.o hi obj-$(CONFIG_HIBERNATION_NVS) += hibernate_nvs.o obj-$(CONFIG_MAGIC_SYSRQ) += poweroff.o + +obj-$(CONFIG_TOI_LZO_TEST) += lzo_test.o
Quoted 9 lines · Nigel Cunningham
Hi all.
(Not sent to LKML yesterday; no reply from linux-crypto yet, so resending).
A while back now, I stopped supplying the LZF compression algorithm with TuxOnIce and made LZO the default algorithm. Around the same time, we started getting occasional errors when reading images; decompression failures.
Quoted 6 lines
I've finally managed to find the time to properly look at this, and have managed to find a data page that LZO compresses, but seems to be unable to decompress back to the original contents. I'm wondering whether this is because I'm doing something wrong, or because there really is some data the LZO (or the kernel implementation) can't do reversible compression on.
Well, FWIW, we have never had any problems with the userland LZO in s2disk, so if anything is wrong with LZO here, I guess it's the kernel code.
Rafael
Hi Rafael.
Quoted 17 lines · Rafael J. Wysocki
Quoted 9 lines · Nigel Cunningham
Hi all.
(Not sent to LKML yesterday; no reply from linux-crypto yet, so resending).
A while back now, I stopped supplying the LZF compression algorithm with TuxOnIce and made LZO the default algorithm. Around the same time, we started getting occasional errors when reading images; decompression failures.
Quoted 6 lines
I've finally managed to find the time to properly look at this, and have managed to find a data page that LZO compresses, but seems to be unable to decompress back to the original contents. I'm wondering whether this is because I'm doing something wrong, or because there really is some data the LZO (or the kernel implementation) can't do reversible compression on.
Quoted 2 lines
Well, FWIW, we have never had any problems with the userland LZO in s2disk, so if anything is wrong with LZO here, I guess it's the kernel code.
Okay. Guess I have to start shipping LZF again and make it the default again then.
Regards,
Nigel
Quoted 19 lines · Nigel Cunningham
Hi Rafael.
Quoted 16 lines · Rafael J. Wysocki
Quoted 15 lines · Nigel Cunningham
Hi all.
(Not sent to LKML yesterday; no reply from linux-crypto yet, so resending).
A while back now, I stopped supplying the LZF compression algorithm with TuxOnIce and made LZO the default algorithm. Around the same time, we started getting occasional errors when reading images; decompression failures.
I've finally managed to find the time to properly look at this, and have managed to find a data page that LZO compresses, but seems to be unable to decompress back to the original contents. I'm wondering whether this is because I'm doing something wrong, or because there really is some data the LZO (or the kernel implementation) can't do reversible compression on.
Quoted 6 lines
Quoted 2 lines
Well, FWIW, we have never had any problems with the userland LZO in s2disk, so if anything is wrong with LZO here, I guess it's the kernel code.
Okay. Guess I have to start shipping LZF again and make it the default again then.
I would hope someone will look at the real problem, though, that LZO isn't working properly. I have to assume that either the kernel decompress is broken or that the page you have given is invalid, and the error lies in the compression.
It doesn't look as if you are doing something wrong, it looks broken.
Signature · 3 lines
Bill Davidsen <[email protected]> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot
Hi Bill.
Quoted 6 lines · Bill Davidsen
I would hope someone will look at the real problem, though, that LZO isn't working properly. I have to assume that either the kernel decompress is broken or that the page you have given is invalid, and the error lies in the compression.
It doesn't look as if you are doing something wrong, it looks broken.
I did get hold of Richard Purdie and Nitin Gupta, who were the guys in the know. We discovered that LZO is expecting decomp_size to be initialised to the amount of available space when the decompression code is called, so there was a bug in my testing code. Nitin was talking about sending a patch to the documentation to make this requirement clearer.
That said, the actual code that TuxOnIce uses does already initialise the variable to PAGE_SIZE, so it seems that I might just have to run with the checking code enabled (with this fix) for a while, until the issue is found.
Regards,
Nigel
On 8 Feb 2010, Nigel Cunningham verbalised:
Quoted 4 lines
That said, the actual code that TuxOnIce uses does already initialise the variable to
PAGE_SIZE, so it seems that I might just have to run with the checking code enabled (with this fix) for a while, until the issue is found.
If you push the fix somewhere I'll run with it as well and see if I can make this problem happen again. (So far I've only seen it with both compression and max_workers > 1, but I haven't done a systematic check for this so it could be an illusion.)
Hi.
Quoted 10 lines · Nix
On 8 Feb 2010, Nigel Cunningham verbalised:
Quoted 4 lines
That said, the actual code that TuxOnIce uses does already initialise the variable to
PAGE_SIZE, so it seems that I might just have to run with the checking code enabled (with this fix) for a while, until the issue is found.If you push the fix somewhere I'll run with it as well and see if I can make this problem happen again. (So far I've only seen it with both compression and
max_workers> 1, but I haven't done a systematic check for this so it could be an illusion.)
I will do - I've just been very slow to because I'm in the middle of switching website hosting at the moment (as well as the week-to-week parts of my real job!). Too much to do and not enough time :)
Nigel