Unbounded growth of slab caches and how to shrink them

I've observed a rather strange unbounded growth of the kmalloc-192 slab cache:

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
711124869 411527215 3% 0.19K 16934908 42 135479264K kmalloc-192

Essentially the kmalloc is around 130 GB , yet only 3 percent of this are being used. In this case I'd like to essentially shrink the overall size of the cache. How is it possible to achieve that? I tried echoing '1' to /sys/kernel/slab/kmalloc-192/shrink but nothing changed.

This is on 3.12 which is rather old kernel, but still I believe it is entirely possible for someone to find a way to flood a machine with network requests which would cause a lot of objects to be allocate, resulting in a particular slab cache growing, then later when the request flood stops the cache would be almost empty, yet the memory won't be usable for anything other than satisfying memory allocation from this cache.

Quoted 6 lines · Nikolay Borisov

I've observed a rather strange unbounded growth of the kmalloc-192 slab cache:

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
711124869 411527215 3% 0.19K 16934908 42 135479264K kmalloc-192

Quoted 4 lines

Essentially the kmalloc is around 130 GB , yet only 3 percent of this are being used. In this case I'd like to essentially shrink the overall size of the cache. How is it possible to achieve that? I tried echoing '1' to /sys/kernel/slab/kmalloc-192/shrink but nothing changed.

Ok this probably means that most slabs have just a few or one objects? Some workloads can result in situations like that. Can you enable debugging and get a list of functions where these objects are allocated?

Quoted 7 lines

This is on 3.12 which is rather old kernel, but still I believe it is entirely possible for someone to find a way to flood a machine with network requests which would cause a lot of objects to be allocate, resulting in a particular slab cache growing, then later when the request flood stops the cache would be almost empty, yet the memory won't be usable for anything other than satisfying memory allocation from this cache.

True. Long known problem and all my attempts to facilitate a solution here did not go anywhere. The essential solution would require objects being movable or removable from the sparsely allocated page frames. And this goes way beyond my subsystem.

If you can figure out which subsystem allocates or frees these objects (through the call traces) then we may find a knob in the subsystem to clear those out once in a while.

Right, so what debugging concretely do you have in mind. So far what I did was reboot the machine with SLUB merging disabled, since there are quite a lot of slabs being merged into that particular one:

:t-0000192 <- cred_jar pid_3 inet_peer_cache request_sock_TCPv6
kmalloc-192 file_lock_cache bio-0 ip_dst_cache key_jar

I'm quite sure it's likely it's one of the either networking or bio-0 slab cache, since the others seems generally not very used.