SLUB: Lockless freelists for objects V7
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest17 patches 22 repliesIn review
V2->V3 from the cover letter
- Provide statistics
- Fallback logic to page lock if cmpxchg16b is not available.
- Better counter support
- More cleanups and clarifications
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/17 | slub: Push irq disable into allocate_slab()mm/ | +13 | 0 replies | |
| 2/17 | slub: Do not use frozen page flag but a bit in the page countersinclude/ include/ mm/ | +9 | 0 replies | |
| 3/17 | slub: Move page->frozen handling near where the page->freelist handling occursmm/ | +6 | 0 replies | |
| 4/17 | x86: Add support for cmpxchg_doublearch/ arch/ arch/ +1 more | +98 | 13 replies | |
| 5/17 | mm: Rearrange struct pageinclude/ | +60 | 3 replies | |
| 6/17 | slub: Add cmpxchg_double_slab()include/ mm/ | +61 | 7 replies | |
| 7/17 | slub: explicit list_lock takingmm/ | +49 | 0 replies | |
| 8/17 | slub: Pass kmem_cache struct to lock and freeze slabmm/ | +8 | 0 replies | |
| 9/17 | slub: Rework allocator fastpathsmm/ | +280 | 0 replies | |
| 10/17 | slub: Invert locking and avoid slab lockmm/ | +53 | 0 replies | |
| 11/17 | slub: Disable interrupts in free_debug processingmm/ | +10 | 0 replies | |
| 12/17 | slub: Avoid disabling interrupts in free slowpathmm/ | +5 | 0 replies | |
| 13/17 | slub: Get rid of the another_slab labelmm/ | +5 | 0 replies | |
| 14/17 | slub: Add statistics for the case that the current slab does not match the nodeinclude/ mm/ | +4 | 0 replies | |
| 15/17 | slub: fast release on full slabinclude/ mm/ | +20 | 0 replies | |
| 16/17 | slub: Not necessary to check for empty slab on load_freelistmm/ | +2 | 0 replies | |
| 17/17 | slub: slabinfo update for cmpxchg handlingtools/ | +39 | 0 replies |
Reviews trailers per patch
3 of 17 patches have a Reviewed-by or Acked-by.
| Patch | David Rientjes | H. Peter Anvin |
|---|---|---|
| 1/17 | ||
| 2/17 | ||
| 3/17 | A-b | |
| 4/17 | R-b R-b | |
| 5/17 | ||
| 6/17 | ||
| 7/17 | ||
| 8/17 | A-b | |
| 9/17 | ||
| 10/17 | ||
| 11/17 | ||
| 12/17 | ||
| 13/17 | ||
| 14/17 | ||
| 15/17 | ||
| 16/17 | ||
| 17/17 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [slubllv7 00/17] SLUB: Lockless freelists for objects V7
V6->V7 - Work out issues with the x86 arch specific patch.
- Add review tags.
V5->V6 - Diffed against current Linus for -next integration.
- Rework descriptions
- Patches could use some review.
V4->V5 - More cleanup. Remove gotos from __slab_alloc and __slab_free
- Some structural changes to alloc and free to clean up the code
- Statistics modifications folded in other patches.
- Fixes to patches already in Pekka's slabnext.
- Include missing upstream fixes
V3->V4 - Diffed against Pekka's slab/next tree.
- Numerous cleanups in particular as a result of the removal of the #ifdef CMPXCHG_LOCAL stuff.
- Smaller cleanups whereever I saw something.
V2->V3
- Provide statistics
- Fallback logic to page lock if cmpxchg16b is not available.
- Better counter support
- More cleanups and clarifications
Well here is another result of my obsession with SLAB allocators. There must be some way to get an allocator done that is faster without queueing and I hope that we are now there (maybe only almost...). Any help with cleaning up the rough edges would be appreciated.
This patchset implement wider lockless operations in slub affecting most of the slowpaths. In particular the patch decreases the overhead in the performance critical section of __slab_free.
One test that I ran was "hackbench 200 process 200" on 2.6.39-rc3 under KVM
Run SLAB SLUB SLUB LL
1st 35.2 35.9 31.9
2nd 34.6 30.8 27.9
3rd 33.8 29.9 28.8
Note that the SLUB version in 2.6.29-rc1 already has an optimized allocation and free path using this_cpu_cmpxchg_double(). SLUB LL takes it to new heights by also using cmpxchg_double() in the slowpaths (especially in the kfree() case where we frequently cannot use the fastpath because there is no queue).
The patch uses a cmpxchg_double (also introduced here) to do an atomic change on the state of a slab page that includes the following pieces of information:
1. Freelist pointer
2. Number of objects inuse
3. Frozen state of a slab
Disabling of interrupts (which is a significant latency in the allocator paths) is avoided in the __slab_free case.
There are some concerns with this patch. The use of cmpxchg_double on fields of the page struct requires alignment of the fields to double word boundaries. That can only be accomplished by adding some padding to struct page which blows it up to 64 byte (on x86_64). Comments in the source describe these things in more detail.
The cmpxchg_double() operation introduced here could also be used to update other doublewords in the page struct in a lockless fashion. One can envision page state changes that involved flags and mappings or maybe do list operations locklessly (but with the current scheme we would need to update two other words elsewhere at the same time too, so another scheme would be needed).