Lockless memory allocator and list
Subsystems (3)
cover Lockless memory allocator and list
This patchset adds a lockless memory allocator and a lock-less list.
v6:
- Revise ARCH_HAVE_NMI_SAFE_CMPXCHG definition for some architectures according to architecture maitainers' comments.
v5:
- Add ARCH_HAVE_NMI_SAFE_CMPXCHG
- Add spin_trylock_irqsave based fallback in lockless memory allocator if ARCH_HAVE_NMI_SAFE_CMPXCHG=n
- Make lockless list depends on ARCH_HAVE_NMI_SAFE_CMPXCHG
v4:
- Split from APEI patchset
- Update patch description and comments according to ML comments
v3:
- Rework lockless memory allocator and list according to ML comments
[PATCH -v6 1/3] Add Kconfig option ARCH_HAVE_NMI_SAFE_CMPXCHG [PATCH -v6 2/3] lib, Make gen_pool memory allocator lockless [PATCH -v6 3/3] lib, Add lock-less NULL terminated single list
PATCH 2/3 lib, Make gen_pool memory allocator lockless
This version of the gen_pool memory allocator supports lockless operation.
This makes it safe to use in NMI handlers and other special unblockable contexts that could otherwise deadlock on locks. This is implemented by using atomic operations and retries on any conflicts. The disadvantage is that there may be livelocks in extreme cases. For better scalability, one gen_pool allocator can be used for each CPU.
The lockless operation only works if there is enough memory available. If new memory is added to the pool a lock has to be still taken. So any user relying on locklessness has to ensure that sufficient memory is preallocated.
The basic atomic operation of this allocator is cmpxchg on long. On architectures that don't have NMI-safe cmpxchg implementation, a spin_trylock_irqsave based fallback is used for gen_pool_alloc, so it can be used in NMI handler safely. But gen_pool_free can not be used in NMI handler in these architectures, because memory free can not fail.
--- include/linux/bitmap.h | 1 include/linux/genalloc.h | 48 ++++++- lib/bitmap.c | 2 lib/genalloc.c | 309 ++++++++++++++++++++++++++++++++++++++++------- 4 files changed, 312 insertions(+), 48 deletions(-)
include/linux/bitmap.h +1 -0
@@ -142,6 +142,7 @@ extern void bitmap_release_region(unsignextern int bitmap_allocate_region (unsigned long *bitmap,int pos,int order);extern void bitmap_copy_le (void *dst,const unsigned long *src,int nbits); +#define BITMAP_FIRST_WORD_MASK (start) (~0UL << ((start) % BITS_PER_LONG))#define BITMAP_LAST_WORD_MASK (nbits) \ ( \ ((nbits) % BITS_PER_LONG) ? \
include/linux/genalloc.h +43 -5
73 lines not on this page · raw message
lib/bitmap.c +0 -2
@@ -271,8 +271,6 @@ int __bitmap_weight(const unsigned long }EXPORT_SYMBOL (__bitmap_weight); -#define BITMAP_FIRST_WORD_MASK (start) (~0UL << ((start) % BITS_PER_LONG)) -void bitmap_set (unsigned long *map,int start,int nr) {unsigned long *p = map +BIT_WORD (start);
lib/genalloc.c +268 -41
384 lines not on this page · raw message
Re: [PATCH -v6 2/3] lib, Make gen_pool memory allocator lockless
Quoted 14 lines · Huang Ying
This version of the
gen_poolmemory allocator supports lockless operation.This makes it safe to use in NMI handlers and other special unblockable contexts that could otherwise deadlock on locks. This is implemented by using atomic operations and retries on any conflicts. The disadvantage is that there may be livelocks in extreme cases. For better scalability, one
gen_poolallocator can be used for each CPU.The lockless operation only works if there is enough memory available. If new memory is added to the pool a lock has to be still taken. So any user relying on locklessness has to ensure that sufficient memory is preallocated.
Quoted 6 lines
The basic atomic operation of this allocator is cmpxchg on long. On architectures that don't have NMI-safe cmpxchg implementation, a
spin_trylock_irqsavebased fallback is used forgen_pool_alloc, so it can be used in NMI handler safely. Butgen_pool_freecan not be used in NMI handler in these architectures, because memory free can not fail.
I still don't see a reason to merge this.
It makes the genalloc thing slower for every other user (more LOCK'ed ops) and there is no new user presented in this series.
Re: [PATCH -v6 2/3] lib, Make gen_pool memory allocator lockless
Hi, Peter,
Quoted 22 lines · Peter Zijlstra
Quoted 14 lines · Huang Ying
This version of the
gen_poolmemory allocator supports lockless operation.This makes it safe to use in NMI handlers and other special unblockable contexts that could otherwise deadlock on locks. This is implemented by using atomic operations and retries on any conflicts. The disadvantage is that there may be livelocks in extreme cases. For better scalability, one
gen_poolallocator can be used for each CPU.The lockless operation only works if there is enough memory available. If new memory is added to the pool a lock has to be still taken. So any user relying on locklessness has to ensure that sufficient memory is preallocated.
Quoted 6 lines
The basic atomic operation of this allocator is cmpxchg on long. On architectures that don't have NMI-safe cmpxchg implementation, a
spin_trylock_irqsavebased fallback is used forgen_pool_alloc, so it can be used in NMI handler safely. Butgen_pool_freecan not be used in NMI handler in these architectures, because memory free can not fail.
Quoted 4 lines
I still don't see a reason to merge this.
It makes the genalloc thing slower for every other user (more LOCK'ed ops) and there is no new user presented in this series.
As far as I know, all genalloc users are not performance sensitive. After all, it is mainly used to manage some device memory before. As for users, I think this is a "chicken and egg" problem. And I have plan to use it in APEI code.
Best Regards,
Huang Ying
PATCH 3/3 lib, Add lock-less NULL terminated single list
In reply to Huang YingCmpxchg is used to implement adding new entry to the list, deleting all entries from the list, deleting first entry of the list and some other operations.
Because this is a single list, so the tail can not be accessed in O(1).
This can be used in NMI handler.
If there are multiple producers and multiple consumers, llist_add can be used in producers and llist_del_all can be used in consumers. They can work simultaneously without lock. But llist_del_first can not be used here. Because llist_del_first depends on list->first->next does not changed if list->first is not changed during its operation, but llist_del_first, llist_add, llist_add sequence in another consumer may violate that.
If there are multiple producers and one consumer, llist_add can be used in producers and llist_del_all or llist_del_first can be used in the consumer.
The list entries deleted via llist_del_all can be traversed with traversing function such as llist_for_each etc. But the list entries can not be traversed safely before deleted from the list without proper synchronization with the list consumers.
--- include/linux/llist.h | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++ lib/Kconfig | 4 ++ lib/Makefile | 2 + lib/llist.c | 80 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 178 insertions(+) create mode 100644 include/linux/llist.h create mode 100644 lib/llist.c
include/linux/llist.h new file +92 -0
@@ -0,0 +1,92 @@ +#ifndef LLIST_H +#define LLIST_H +/* +* Lock-less NULL terminated single linked list +* +* If there are multiple producers and multiple consumers, llist_add +* can be used in producers and llist_del_all can be used in +* consumers. They can work simultaneously without lock. But +* llist_del_first can not be used here. Because llist_del_first +* depends on list->first->next does not changed if list->first is not +* changed during its operation, but llist_del_first, llist_add, +* llist_add sequence in another consumer may violate that. +* +* If there are multiple producers and one consumer, llist_add can be +* used in producers and llist_del_all or llist_del_first can be used +* in the consumer. +* +* The list entries deleted via llist_del_all can be traversed with +* traversing function such as llist_for_each etc. But the list +* entries can not be traversed safely before deleted from the list +* without proper synchronization with the list consumers. +*/ + +struct llist_head { +struct llist_node *first; +}; + +struct llist_node { +struct llist_node *next; +}; + +#define LLIST_HEAD_INIT (name) { NULL } +#define LLIST_HEAD (name) struct llist_head name = LLIST_HEAD_INIT(name) + +/** +* init_llist_head - initialize lock-less list head +* @head: the head for your lock-less list +*/ +static inline void init_llist_head (struct llist_head *list) +{ + list->first =NULL ; +} + +/** +* llist_entry - get the struct of this entry +* @ptr: the &struct llist_node pointer. +* @type: the type of the struct this is embedded in. +* @member: the name of the llist_node within the struct. +*/ +#define llist_entry (ptr, type, member) \ + container_of(ptr, type, member) + +/** +* llist_for_each - iterate over some entries of a lock-less list +* @pos: the &struct llist_node to use as a loop cursor +* @node: the first entry of deleted list entries +* +* In general, some entries of the lock-less list can be traversed +* safely only after being deleted from list, so start with an entry +* instead of list head. +*/ +#define llist_for_each (pos, node) \ + for (pos = (node); pos; pos = pos->next) + +/** +* llist_for_each_entry - iterate over some entries of lock-less list of given type +* @pos: the type * to use as a loop cursor. +* @node: the fist entry of deleted list entries. +* @member: the name of the llist_node with the struct. +* +* In general, some entries of the lock-less list can be traversed +* safely only after being removed from list, so start with an entry +* instead of list head. +*/ +#define llist_for_each_entry (pos, node, member) \ + for (pos = llist_entry((node), typeof(*pos), member); \ + &pos->member != NULL; \ + pos = llist_entry(pos->member.next, typeof(*pos), member)) + +/** +* llist_empty - tests whether a lock-less list is empty +* @head: the list to test +*/ +static inline int llist_empty (const struct llist_head *head) +{ +return head->first ==NULL ; +} + +void llist_add (struct llist_node *new,struct llist_head *head); +struct llist_node *llist_del_first (struct llist_head *head); +struct llist_node *llist_del_all (struct llist_head *head); +#endif /* LLIST_H */
lib/Kconfig +4 -0
@@ -210,4 +210,8 @@ config GENERIC_ATOMIC64 config LRU_CACHE tristate +config LLIST + bool + depends on ARCH_HAVE_NMI_SAFE_CMPXCHG + endmenu
lib/Makefile +2 -0
@@ -106,6 +106,8 @@ obj-$(CONFIG_GENERIC_ATOMIC64) += atomic obj-$(CONFIG_ATOMIC64_SELFTEST) += atomic64_test.o +obj-$(CONFIG_LLIST) += llist.o + hostprogs-y := gen_crc32table clean-files := crc32table.h
lib/llist.c new file +80 -0
80 lines not on this page · raw message