Swap Compression

Some time before rmoser started HIS discussion on compressed swap, I started a discussion on the same topic. Out of that discussion came mention of an existing project which did EXACTLY what I was proposing. That made me happy. For some reason rmoser wants to start from scratch. He can do that if he wants. I had only hoped that my earlier mention of it would spark interest in taking the existing implementation and adding it to the mainline kernel, after some incompatbilities were dealt with. Unfortunately, I don't have the time to engage directly in that particular aspect the kernel development.

*********** REPLY SEPARATOR ***********

I want to start from scratch because I am trying a different angle, and I don't understand fully current projects on it. I am not trying to discredit anyone. In fact, I would hope that the existing implimentation (different as it is) would manage to get itself in as well. Remember, modularity is all about options. Options aren't just "Do you want this or that?", but also "How do you want this or that done?" Plus, I am not compressing the page cache, assuming I understand what this means. AFAIK (from others' help), the page cache is a cached copy of pages that have gone to the swap recently, or been pulled off recently. I am not touching that.

Other things (not brought up by me) include making a central compression library for the kernel (modular, i.e. modprobe zlib and such) (Jorn's idea). That is something that I think would be essential for swap compression; why should one module horde all the compression algo's?

--Bluefox

There is nothing wrong from having a separate, parallel project running - it's a good idea, and generally helps to spark interest in both projects.

John.

*********** REPLY SEPARATOR ***********

THANK YOU!

You've saved me from having to defend meself.

--Bluefox Icy

I don't think a parallel project is a bad idea either. I was just suggesting adding the minilzo algorithm from the linuxcompressed project as one of the compression algorithms available.

Con

Actually, I'd like a central compression library with a large assortment of algorithms. That way the really common code is shared between both (or more) projects is shared.

Also, yet another unused compression algorithm hurts about as bad, as yet another unused device driver. It just grows the kernel .tar.bz2.

Jörn

Time? What's that? Time is only worth what you do with it.
-- Theo de Raadt

*********** REPLY SEPARATOR ***********

That's just great, heh. It will in the future cut kernel code size and matinence work.

Not to get side tracked, but there is an answer for this.....

Well then let's restructure the kernel's build system soon. It's 15 seconds since I read this, and already I'm working out in my head the high-level workings of a modular build system (what it would look like).

For an example, future kernel releases could be in directories, which contain the kernel .tar.bz2, then separate sets of drivers and such. Simply unpacking these drivers into your kernel source tree could place files that make would read, i.e. the kernel build system would adapt. For example:

linux/options/foo.lod <-- Linux Option Definition file.

This file would reference other LO's that it requires, and give configuration options. These options would have references to LO's that they require. They would also reference which version ranges of themselves they are compatible with. Thus:

make menuconfig

<M>Foo
----<M>Foo with Bar support

<SELECT> <EXIT> <HELP> <MISSING>

Hit missing on an option, say Foo:

Dependencies for CONFIG_FOO
< >Foo
----< >Bar
----< >Baz [FAILED DEPENDS!]

Hit baz:

Dependencies for CONFIG_FOO_BAZ:

Foo with Baz support needs Baz compatible with Baz v1.32

So you get the Baz option tarball, or the tarball containing it:

linux/options/baz.lod

make menuconfig:

<M>Foo
----<M>Foo with Bar support
----<M>Foo with Baz support

Yes I know this is a hassle, but it's not like Microsoft drivers where you spend 12 hours all over the net searching for stuff; the main kernel source distribution tree would contain the kernel, and the kernel.org mirrors would all have the Linux Options. There'd be a tarball for the "default must-have drivers" separate from the kernel core.

It's just a suggestion for the future.

--Bluefox Icy

Had a thought. Why wait for a compression algorithm? Jorn, if you are going to work on putting the code into the kernel and making the stuff to allow the swap code to use it, why not start coding it before the compression code is finished? i.e. get the stuff down for the swap filtering (filtering i.e. passing through a compression or encryption routine) and swap-on-ram stuff, and later take the compression algo code and place the module interface on it and make a kernel module.

At this point, I'd say to allow specified order filters, to allow for swap cyphering and compression. Security, you know; swap files are a security hazard. Just power-off, boot a root disk, pull up the swap partition, rip out the blocks, and look for what looks to be the root password.

--Bluefox Icy

While we're having thoughts, this thread keeps me thinking it would make sense to have a block device driver that would be assigned unused memory.

I don't mean memory on video cards etc. I'm thinking of the 10% of RAM unused when 1GB systems are booted with MEM=900M because they run faster with HIGHMEM turned off.

The primary use for this "device" would be high priority swap. Even with whatever overhead it takes to access it should be orders of magnitude faster than any spinning media.

________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		[email protected]

		Remember Cernan and Schmitt