Deprecating and removing SLOB
Hi Vlastimil,
I was actually using SLOB until recently for a device flasher program (kernel + initramfs + dtb, booted over USB) for Ingenic SoCs. I picked SLOB just because it said "embedded systems" in menuconfig and some of my boards have as little as 32 MiB RAM.
It worked fine on some boards, but on others it had about a 25% chance of booting, and 75% chance of hanging at boot. I tried printk-debugging it, and was coming to the conclusion that it's memory corruption of some sort.
Then I switched to SLUB and all the problems are gone. Same with SLAB.
So while I can't say for sure that SLOB is broken (it might be triggering a bug somewhere else), I am highly suspicious that it is.
So yeah... axe it.
Cheers,
-Paul
Quoted 10 lines · Paul Cercueil
It worked fine on some boards, but on others it had about a 25% chance of booting, and 75% chance of hanging at boot. I tried printk-debugging it, and was coming to the conclusion that it's memory corruption of some sort.
Then I switched to SLUB and all the problems are gone. Same with SLAB.
So while I can't say for sure that SLOB is broken (it might be triggering a bug somewhere else), I am highly suspicious that it is.
I have this distinct memory of having seen other reports like this, but my google-fu is not strong enough to back that up.
There definitely has been recurring noise about SLOB issues. There's a reason people have wanted to remove it for years and years.
Linus
Hi,
Quoted 9 lines · Linus Torvalds
Quoted 8 lines · Paul Cercueil
It worked fine on some boards, but on others it had about a 25% chance of booting, and 75% chance of hanging at boot. I tried printk-debugging it, and was coming to the conclusion that it's memory corruption of some sort.
Then I switched to SLUB and all the problems are gone. Same with SLAB.
Quoted 8 lines
Quoted 2 lines
So while I can't say for sure that SLOB is broken (it might be triggering a bug somewhere else), I am highly suspicious that it is.
I have this distinct memory of having seen other reports like this, but my google-fu is not strong enough to back that up.
There definitely has been recurring noise about SLOB issues. There's a reason people have wanted to remove it for years and years.
Some of the reported SLOB issues have been actually real driver bugs, that go unnoticed when SLUB/SLAB are used (unless perhaps debug stuff is enabled). I'm not saying kernel should keep SLUB, but it's good at failing early when there is a bug. See e.g. commit 120ee599b5bf ("staging:
octeon-usb: prevent memory corruption")
A.
Quoted 16 lines · Aaro Koskinen
Quoted 15 lines · Linus Torvalds
Quoted 10 lines · Paul Cercueil
It worked fine on some boards, but on others it had about a 25% chance of booting, and 75% chance of hanging at boot. I tried printk-debugging it, and was coming to the conclusion that it's memory corruption of some sort.
Then I switched to SLUB and all the problems are gone. Same with SLAB.
So while I can't say for sure that SLOB is broken (it might be triggering a bug somewhere else), I am highly suspicious that it is.
I have this distinct memory of having seen other reports like this, but my google-fu is not strong enough to back that up.
Quoted 6 lines
Quoted 2 lines
There definitely has been recurring noise about SLOB issues. There's a reason people have wanted to remove it for years and years.
Some of the reported SLOB issues have been actually real driver bugs, that go unnoticed when SLUB/SLAB are used (unless perhaps debug stuff is enabled). I'm not saying kernel should keep SLUB, but it's good at
^^^^
SLOB, sorry for a typo.
Quoted 4 lines
failing early when there is a bug. See e.g. commit
120ee599b5bf("staging:
octeon-usb: prevent memory corruption")A.
Quoted 6 lines · Aaro Koskinen
Some of the reported SLOB issues have been actually real driver bugs, that go unnoticed when SLUB/SLAB are used (unless perhaps debug stuff is enabled). I'm not saying kernel should keep SLOB, but it's good at failing early when there is a bug. See e.g. commit
120ee599b5bf("staging:
octeon-usb: prevent memory corruption")
Out of curiosity, are these bugs that would have been found using KASAN or some of the other kernel sanitizers and/or other debugging tools we have at our disposal?
- Ted
Quoted 11 lines · Theodore Ts'o
Quoted 6 lines · Aaro Koskinen
Some of the reported SLOB issues have been actually real driver bugs, that go unnoticed when SLUB/SLAB are used (unless perhaps debug stuff is enabled). I'm not saying kernel should keep SLOB, but it's good at failing early when there is a bug. See e.g. commit
120ee599b5bf("staging:
octeon-usb: prevent memory corruption")Out of curiosity, are these bugs that would have been found using KASAN or some of the other kernel sanitizers and/or other debugging tools we have at our disposal?
Hopefully slub_debug redzoning would be able to trigger the bug described in commit 120ee599b5bf above, which is:
Quoted 6 lines
octeon-hcd will crash the kernel when SLOB is used. This usually happens after the 18-byte control transfer when a device descriptor is read. The DMA engine is always transfering full 32-bit words and if the transfer is shorter, some random garbage appears after the buffer. The problem is not visible with SLUB since it rounds up the allocations to word boundary, and the extra bytes will go undetected.
Ah, actually it wouldn't *now* as SLUB would make the allocation fall into kmalloc-32 cache and only add redzone beyond 32 bytes. But with upcoming changes by Feng Tang, this should work.
slub_debug would also have a chance of catching buffer overflows by kernel code itself, not DMA, and tell you about it more sooner and gracefully than crashing. KASAN also, even with a higher chance and precision, if it's available for your arch and your device constraints can tolerate its larger overhead.
Quoted 3 lines
- Ted
Quoted 23 lines · Vlastimil Babka
Quoted 11 lines · Theodore Ts'o
Quoted 6 lines · Aaro Koskinen
Some of the reported SLOB issues have been actually real driver bugs, that go unnoticed when SLUB/SLAB are used (unless perhaps debug stuff is enabled). I'm not saying kernel should keep SLOB, but it's good at failing early when there is a bug. See e.g. commit
120ee599b5bf("staging:
octeon-usb: prevent memory corruption")Out of curiosity, are these bugs that would have been found using KASAN or some of the other kernel sanitizers and/or other debugging tools we have at our disposal?
Hopefully
slub_debugredzoning would be able to trigger the bug described in commit120ee599b5bfabove, which is:Quoted 6 lines
octeon-hcd will crash the kernel when SLOB is used. This usually happens after the 18-byte control transfer when a device descriptor is read. The DMA engine is always transfering full 32-bit words and if the transfer is shorter, some random garbage appears after the buffer. The problem is not visible with SLUB since it rounds up the allocations to word boundary, and the extra bytes will go undetected.
Quoted 3 lines
Ah, actually it wouldn't *now* as SLUB would make the allocation fall into kmalloc-32 cache and only add redzone beyond 32 bytes. But with upcoming changes by Feng Tang, this should work.
I wrote a simple case trying simulating this:
static noinline void dma_align_test(void)
{
char *buf;
buf = kmalloc(18, GFP_KERNEL);
buf[18] = 0;
buf[19] = 0;
kfree(buf);
}And with slub_debug on and the slub_redzone patchset[1], it did catch the out-of-bound access, as in the dmesg:
"
=============================================================================
BUG kmalloc-32 (Not tainted): kmalloc Redzone overwritten
-----------------------------------------------------------------------------
0xffff888005ebb032-0xffff888005ebb033 @offset=50. First byte 0x0 instead of 0xcc Allocated in dma_align_test+0x1b/0x29 age=6554 cpu=1 pid=1
__kmem_cache_alloc_node+0x2a7/0x320 kmalloc_trace+0x27/0xa0 dma_align_test+0x1b/0x29 late_slub_debug+0xa/0x11 do_one_initcall+0x87/0x2a0 ...
Slab 0xffffea000017aec0 objects=21 used=19 fp=0xffff888005ebbf20 flags=0xfffffc0000200(slab|node=0|zone=1|lastcpupid=0x1fffff) Object 0xffff888005ebb020 @offset=32 fp=0x0000000000000000
Redzone ffff888005ebb000: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................ Redzone ffff888005ebb010: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc ................ Object ffff888005ebb020: 50 92 28 00 81 88 ff ff 01 00 00 00 11 00 b6 07 P.(............. Object ffff888005ebb030: 6b a5 00 00 cc cc cc cc cc cc cc cc cc cc cc cc k............... Redzone ffff888005ebb040: cc cc cc cc cc cc cc cc ........ Padding ffff888005ebb0a4: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ Padding ffff888005ebb0b4: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZ
"
[1]. https://lore.kernel.org/lkml/[email protected]/
Thanks,
Feng
Quoted 11 lines
slub_debugwould also have a chance of catching buffer overflows by kernel code itself, not DMA, and tell you about it more sooner and gracefully than crashing. KASAN also, even with a higher chance and precision, if it's available for your arch and your device constraints can tolerate its larger overhead.Quoted 3 lines
- Ted
Quoted 10 lines · Vlastimil Babka
Quoted 6 lines
octeon-hcd will crash the kernel when SLOB is used. This usually happens after the 18-byte control transfer when a device descriptor is read. The DMA engine is always transfering full 32-bit words and if the transfer is shorter, some random garbage appears after the buffer. The problem is not visible with SLUB since it rounds up the allocations to word boundary, and the extra bytes will go undetected.
Ah, actually it wouldn't *now* as SLUB would make the allocation fall into kmalloc-32 cache and only add redzone beyond 32 bytes. But with upcoming changes by Feng Tang, this should work.
This is kind of "if a bug stings a tree in a forest, does it hurt" problem. If all allocations of 18 bytes are rounded up to 20 or more bytes, then it doesn't matter that the device has this bug. Sure, it may end up hurting in the future if we decide to create 18-byte slab caches, but it's not actually going to affect anything today (and we seem to be moving towards less precision in order to get more performance)
From: Matthew Wilcox
Quoted 14 lines
Sent: 10 November 2022 16:20
Quoted 10 lines · Vlastimil Babka
Quoted 6 lines
octeon-hcd will crash the kernel when SLOB is used. This usually happens after the 18-byte control transfer when a device descriptor is read. The DMA engine is always transfering full 32-bit words and if the transfer is shorter, some random garbage appears after the buffer. The problem is not visible with SLUB since it rounds up the allocations to word boundary, and the extra bytes will go undetected.
Ah, actually it wouldn't *now* as SLUB would make the allocation fall into kmalloc-32 cache and only add redzone beyond 32 bytes. But with upcoming changes by Feng Tang, this should work.
Quoted 7 lines
This is kind of "if a bug stings a tree in a forest, does it hurt" problem. If all allocations of 18 bytes are rounded up to 20 or more bytes, then it doesn't matter that the device has this bug. Sure, it may end up hurting in the future if we decide to create 18-byte slab caches, but it's not actually going to affect anything today (and we seem to be moving towards less precision in order to get more performance)
Yes, even on dma-coherent systems allocated blocks have to be moderately aligned - so the space after an 18 byte block can't be used. I also doubt there is any benefit (and many bugs) from allowing 2 bytes alignment on m68k.
So the 'overwrite to a whole number of words' maybe reasonably expected to not cause any real bugs.
x86 (even 32bit) probably requires 16 byte alignment (for some corner cases) - ok for a power-of-2 allocator that doesn't add a header. (Although 1, 2, 4 and 8 byte allocates are valid.)
To reduce memory wastage what you really don't want is an allocator that adds a header/trailer and then rounds up to a power of 2. Coders write in binary and do kmalloc(256) not kmalloc(200) and rounding 256 up to 512 is rather wasteful.
(Search for the kmalloc(PAGE_SIZE+1) :-)
I also think that one of the allocators only cuts pages into power-of-2 sizes.
It is probably sensible to return cache-aligned (probably 64 byte) buffers for requests larger than a cache line.
But a 4k page can be split into 21 192-byte buffers.
As well as using less memory for allocates between 129 and 192 bytes it may reduce pressure on the d-cache by evening out cache line usage.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)