Skip to contentSkip to conversation

Packing data in kernel memory

DISCUSSION John Richard Moser 3 participants 5 messages Jun 13, 2006

lore
Share & tools
lore
#0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Is there a way to pack and store arbitrary data in the kernel, or do I need to roll my own?

As a simple example, let's say I wanted to store 8 elements. Assume PAGE_SIZE is 4. If I store these individually, their size and use is:

ELEM      SIZE    PAGES    USE
[A.]      4        1        4
[B...]    6        2        8
[C]       3        1        4
[D....]   7        2        8
[E]       3        1        4
[F.]      4        1        4

[G......] 9 3 12
[H.....] 8 2 8

TOTAL 44 13 52

2 excess pages, 8 excess units of memory wasted.

Now let's say I used groups of 4 pages:

E                 S   P   U
[A.][H.....]      12   4  16
[B...][C][D....]  16   4  16
[E][F.][G......]  16   4  16
                  44  12  48

1 excess pages, 4 units wasted memory.
- --
All content of all messages exchanged herein are left in the Public Domain, unless otherwise explicitly stated.

Creative brains are a valuable, limited resource. They shouldn't be wasted on re-inventing the wheel when there are so many fascinating new problems waiting out there.
-- Eric Steven Raymond

We will enslave their women, eat their children and rape their cattle!
-- Bosc, Evil alien overlord from the fifth dimension
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIVAwUBRI8IkQs1xW0HCTEFAQJcCxAAnucDd+TBSS7hjHmYMdG64WRHwcidvGkS prFXPSWKXZh0tTvV5+DGDwUvGeLew1VvEY5IJl7i6ak7V0nehAKuEIdMiL9x1RSI OeZN2cA/eGl1Bv1r3QJdhutfCPF10yFMEUkkLwa2+z/bbPlN4zdoUaR0yAFRigwL zuJbOTlrEQcY8lAIlPbtdm+c1mQ+FNfpe7dGEYWwOZZYeE8wglCQbFhzmda9nmP5 fJrod94dJnsGsD03MOs17+0+GaLEWz09KKCjMIXRi9fG4q1N9Kgi/GwH+kUR/Cnq kT44JgcNIuHYMFVss4qUA+wBit3HuT7zL4U9BhxIMpg7BFaYPVCDD+Lpi4J50WXk sYJs9GpOHwSAmeK1RQ68FU9iO7HI2g0LPKrAWxXj+nclaMhmBqQAF2Eue7LxnSbm Cy2Ne3nWIPnIFA4y76L8VslsYgFhaBdrzydIHKTH/mqDHWvNMDmcCy6sSnueGXOR qBopgebvKRpnCtwQvpQ+RgwvGx7/XyftP+W+Lag5BCC1yezP6vk7HDj1lvKrrKVf kyGNcNx0t3yWnSqeJC1M8nOdpSfHsuDjoG03HdPcmdurQdaGtUmA5k37GpjvATBf 1JAst4FXf9LWJ8mwQakbp7r4vUrEyfMfub9J/j2u1zxq37bJCyAs6jxb5D7YmSno zzV/7vqPdU0=
=kiK1
-----END PGP SIGNATURE-----

#1 lore raw
Quoted 3 lines

Subject: Packing data in kernel memory

Can't you just use mlock(), if you want to keep it in RAM?

Or do you need it in kernel memory, because you need it in the lowmem area? Or for interaction with other kernel code?

Quoted 3 lines

Is there a way to pack and store arbitrary data in the kernel, or do I need to roll my own?

Write a device driver, kmalloc some buffer, and copy data via a write function from userspace to that buffer. Should be trivial.

Quoted 1 line

1 excess pages, 4 units wasted memory.

Of course, kmalloc only works up to some boundary AFIACS.

Jan Engelhardt

#2 lore raw
Quoted 11 lines · On Tue, 13 Jun 2006 22:18:55 +0200 (MEST) Jan Engelhardt wrote:
Quoted 3 lines

Subject: Packing data in kernel memory

Can't you just use mlock(), if you want to keep it in RAM?

Or do you need it in kernel memory, because you need it in the lowmem area? Or for interaction with other kernel code?

Quoted 2 lines

Is there a way to pack and store arbitrary data in the kernel, or do I need to roll my own?

Sounds a bit like a slab cache to me.

Quoted 6 lines

Write a device driver, kmalloc some buffer, and copy data via a write function from userspace to that buffer. Should be trivial.

Quoted 1 line

1 excess pages, 4 units wasted memory.

Of course, kmalloc only works up to some boundary AFIACS.

128 KB on some arches. More on a few IIRC.

---
~Randy

#3 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 6 lines · Randy.Dunlap
Quoted 4 lines · On Tue, 13 Jun 2006 22:18:55 +0200 (MEST) Jan Engelhardt wrote:
Quoted 2 lines

Subject: Packing data in kernel memory

Can't you just use mlock(), if you want to keep it in RAM?

Quoted 8 lines
Quoted 5 lines

Or do you need it in kernel memory, because you need it in the lowmem area? Or for interaction with other kernel code?

Quoted 2 lines

Is there a way to pack and store arbitrary data in the kernel, or do I need to roll my own?

Sounds a bit like a slab cache to me.

OK cool, can I make that non-swappable? I'm going to be trying to do this between where kernel swaps a page out and swapped page actually is written to disk. The result will be a "Swap Zone," in-memory storage of pages that the rest of the kernel thinks have been swapped to disk. (Code here will use the swap interface, so the rest of the kernel thinks it's just swapping; I'll handle whether to pull it out of memory or off disk behind that)

The need for packing pages comes because eventually (using above infrastructure) I'll be taking sets of 32KiB of data and compressing it; I don't want to pad up to 4095 excess unused bytes if that stuff compresses to 28KiB+1 :) (more likely 16KiB+1 +/-8KiB)

This is all, of course, assuming I ever figure out how the heck to get in the middle of the swapping process. I'm looking at mm/page_io.c swap_writepage() and friends and scratching my head. I have no idea wtf...

Quoted 11 lines
Quoted 5 lines

Write a device driver, kmalloc some buffer, and copy data via a write function from userspace to that buffer. Should be trivial.

Quoted 1 line

1 excess pages, 4 units wasted memory.

Of course, kmalloc only works up to some boundary AFIACS.

128 KB on some arches. More on a few IIRC.

---
~Randy

- --
All content of all messages exchanged herein are left in the Public Domain, unless otherwise explicitly stated.

Creative brains are a valuable, limited resource. They shouldn't be wasted on re-inventing the wheel when there are so many fascinating new problems waiting out there.
-- Eric Steven Raymond

We will enslave their women, eat their children and rape their cattle!
-- Bosc, Evil alien overlord from the fifth dimension
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQIVAwUBRI8mAAs1xW0HCTEFAQJa4w/+Jrhnnp+DyOmuuQPL0A2QbydRlhvyeK6g mixAd41AJPN8CmMqFZzWTPFbhN65BiM3oaKv+5YX8kvzJiKfhi8BabLlkapUgljx qlFr2yOSTIBEkPiPaUTjjYSLFfVBqca1kAAcjO7qJGjcrCJK0AkVp11XKvF8xiLI Hg8kEV1GzQKtMo65s+HQQR8XDTDRPuyTpGgWbVSwHyZnJY1pwFd2gVNpW63y52QM pJw7WyLBa4NNDLLNRvX8/DbSjvaN3fYy223GItS67QaSOv5G9MNXQnhmUQV8dV0J k4xiOhPBRoV1tDpIbdFTWajPp5facVZfLklsNv1uPyDUxdsrMDa8ETNsd6Kn3A5V 8Zp6EQWScyqoDa8u7aL2IZ0BCm69aJnaAXLm3miNheW1vUPmKYOZJn3+lmEX1vMh JuXZzUYjgFgIss3djrpC2GoWqlMYgQ92ZBxecBoMQowPGkLwtcbz4J3qfwBalr1+ q9Ho85mH7eFUyod7ftWS8r6SQ1WtxCGTl9aPnwqlroq2RG1a/3bhJ2NIHyoLc+zt y4IpzZ7B1JzTpBVKKkksOMv7B3XmxCwNzr4Qc1ilx+cLlqwsChiiAzv0IFXz6cBT LVeTTHGyQOk7Yd5jO+Wi/s+9XsXEPFQGLDIimxqjqhXitcErAWOodg7tidYgMgFW snfgQan4PZI=
=B4VW
-----END PGP SIGNATURE-----

#4 lore raw
Quoted 10 lines · On Tue, 13 Jun 2006 16:54:26 -0400 John Richard Moser wrote:
Quoted 9 lines · Randy.Dunlap
Quoted 7 lines · On Tue, 13 Jun 2006 22:18:55 +0200 (MEST) Jan Engelhardt wrote:
Quoted 2 lines

Subject: Packing data in kernel memory

Can't you just use mlock(), if you want to keep it in RAM?

Or do you need it in kernel memory, because you need it in the lowmem area? Or for interaction with other kernel code?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

Is there a way to pack and store arbitrary data in the kernel, or do I need to roll my own?

Sounds a bit like a slab cache to me.

OK cool, can I make that non-swappable?

kernel (allocated) memory is non-swappable.

Quoted 8 lines

I'm going to be trying to do
this between where kernel swaps a page out and swapped page actually is written to disk. The result will be a "Swap Zone," in-memory storage of pages that the rest of the kernel thinks have been swapped to disk. (Code here will use the swap interface, so the rest of the kernel thinks it's just swapping; I'll handle whether to pull it out of memory or off disk behind that)

Quoted 17 lines

The need for packing pages comes because eventually (using above infrastructure) I'll be taking sets of 32KiB of data and compressing it; I don't want to pad up to 4095 excess unused bytes if that stuff compresses to 28KiB+1 :) (more likely 16KiB+1 +/-8KiB)

This is all, of course, assuming I ever figure out how the heck to get in the middle of the swapping process. I'm looking at mm/page_io.c swap_writepage() and friends and scratching my head. I have no idea wtf...

Quoted 7 lines
Quoted 5 lines

Write a device driver, kmalloc some buffer, and copy data via a write function from userspace to that buffer. Should be trivial.

Quoted 1 line

1 excess pages, 4 units wasted memory.

Of course, kmalloc only works up to some boundary AFIACS.

128 KB on some arches. More on a few IIRC.

---
~Randy