nvm page allocator for bcache
v10 cover nvm page allocator for bcache
From: Jianpeng Ma <[email protected]>
This series implements nvm pages allocator for bcache. This idea is from one discussion about nvdimm use case in kernel together with Coly. Coly sent the following email about this idea to give some introduction on what we will do before:
https://lore.kernel.org/linux-bcache/[email protected]/
Here this series focus on the first step in above email, that is to say, this patch set implements a generic framework in bcache to allocate/release NV-memory pages, and provide allocated pages for each requestor after reboot. In order to do this, one simple buddy system is implemented to manage NV-memory pages.
Coly Li (1):
bcache: add initial data structures for nvm pages
Jianpeng Ma (5):
bcache: initialize the nvm pages allocator
bcache: initialization of the buddy
cache: bch_nvm_alloc_pages() of the buddy
bcache: bch_nvm_free_pages() of the buddy
bcache: get allocated pages from specific owner
drivers/md/bcache/Kconfig | 9 + drivers/md/bcache/Makefile | 1 + drivers/md/bcache/nvm-pages.c | 769 ++++++++++++++++++++++++++++++++ drivers/md/bcache/nvm-pages.h | 92 ++++ drivers/md/bcache/super.c | 3 + include/uapi/linux/bcache-nvm.h | 206 +++++++++ 6 files changed, 1080 insertions(+) create mode 100644 drivers/md/bcache/nvm-pages.c create mode 100644 drivers/md/bcache/nvm-pages.h create mode 100644 include/uapi/linux/bcache-nvm.h
Signature · 1 line
2.25.1
PATCH v10 1/6 bcache: add initial data structures for nvm pages
From: Coly Li <[email protected]>
This patch initializes the prototype data structures for nvm pages allocator,
- struct bch_nvm_pages_sb
This is the super block allocated on each nvdimm namespace. A nvdimm set may have multiple namespaces, bch_nvm_pages_sb->set_uuid is used to mark which nvdimm set this name space belongs to. Normally we will use the bcache's cache set UUID to initialize this uuid, to connect this nvdimm set to a specified bcache cache set.
- struct bch_owner_list_head
This is a table for all heads of all owner lists. A owner list records which page(s) allocated to which owner. After reboot from power failure, the ownwer may find all its requested and allocated pages from the owner list by a handler which is converted by a UUID.
- struct bch_nvm_pages_owner_head
This is a head of an owner list. Each owner only has one owner list, and a nvm page only belongs to an specific owner. uuid[] will be set to owner's uuid, for bcache it is the bcache's cache set uuid. label is not mandatory, it is a human-readable string for debug purpose. The pointer *recs references to separated nvm page which hold the table of struct bch_nvm_pgalloc_rec.
- struct bch_nvm_pgalloc_recs
This struct occupies a whole page, owner_uuid should match the uuid in struct bch_nvm_pages_owner_head. recs[] is the real table contains all allocated records.
- struct bch_nvm_pgalloc_rec
Each structure records a range of allocated nvm pages.
- Bits 0 - 51: is pages offset of the allocated pages.
- Bits 52 - 57: allocaed size in page_size * order-of-2
- Bits 58 - 63: reserved.
Since each of the allocated nvm pages are power of 2, using 6 bits to represent allocated size can have (1<<(1<<64) - 1) * PAGE_SIZE maximum value. It can be a 76 bits width range size in byte for 4KB page size, which is large enough currently.
--- include/uapi/linux/bcache-nvm.h | 200 ++++++++++++++++++++++++++++++++ 1 file changed, 200 insertions(+) create mode 100644 include/uapi/linux/bcache-nvm.h
include/uapi/linux/bcache-nvm.h new file +200 -0
@@ -0,0 +1,200 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ + +#ifndef _UAPI_BCACHE_NVM_H +#define _UAPI_BCACHE_NVM_H + +#ifdef CONFIG_64BIT +/* +* Bcache on NVDIMM data structures +*/ + +/* +* - struct bch_nvm_pages_sb +* This is the super block allocated on each nvdimm namespace. A nvdimm +* set may have multiple namespaces, bch_nvm_pages_sb->set_uuid is used to mark +* which nvdimm set this name space belongs to. Normally we will use the +* bcache's cache set UUID to initialize this uuid, to connect this nvdimm +* set to a specified bcache cache set. +* +* - struct bch_owner_list_head +* This is a table for all heads of all owner lists. A owner list records +* which page(s) allocated to which owner. After reboot from power failure, +* the ownwer may find all its requested and allocated pages from the owner +* list by a handler which is converted by a UUID. +* +* - struct bch_nvm_pages_owner_head +* This is a head of an owner list. Each owner only has one owner list, +* and a nvm page only belongs to an specific owner. uuid[] will be set to +* owner's uuid, for bcache it is the bcache's cache set uuid. label is not +* mandatory, it is a human-readable string for debug purpose. The pointer +* recs references to separated nvm page which hold the table of struct +* bch_pgalloc_rec. +* +*- struct bch_nvm_pgalloc_recs +* This structure occupies a whole page, owner_uuid should match the uuid +* in struct bch_nvm_pages_owner_head. recs[] is the real table contains all +* allocated records. +* +* - struct bch_pgalloc_rec +* Each structure records a range of allocated nvm pages. pgoff is offset +* in unit of page size of this allocated nvm page range. The adjoint page +* ranges of same owner can be merged into a larger one, therefore pages_nr +* is NOT always power of 2. +* +* +* Memory layout on nvdimm namespace 0 +* +* 0 +---------------------------------+ +* | | +* 4KB +---------------------------------+ +* | bch_nvm_pages_sb | +* 8KB +---------------------------------+ <--- bch_nvm_pages_sb.bch_owner_list_head +* | bch_owner_list_head | +* | | +* 16KB +---------------------------------+ <--- bch_owner_list_head.heads[0].recs[0] +* | bch_nvm_pgalloc_recs | +* | (nvm pages internal usage) | +* 24KB +---------------------------------+ +* | | +* | | +* 16MB +---------------------------------+ +* | allocable nvm pages | +* | for buddy allocator | +* end +---------------------------------+ +* +* +* +* Memory layout on nvdimm namespace N +* (doesn't have owner list) +* +* 0 +---------------------------------+ +* | | +* 4KB +---------------------------------+ +* | bch_nvm_pages_sb | +* 8KB +---------------------------------+ +* | | +* | | +* | | +* | | +* | | +* | | +* 16MB +---------------------------------+ +* | allocable nvm pages | +* | for buddy allocator | +* end +---------------------------------+ +* +*/ + +#include <linux/types.h> + +/* In sectors */ +#define BCH_NVM_PAGES_SB_OFFSET 4096 +#define BCH_NVM_PAGES_OFFSET (16 << 20) + +#define BCH_NVM_PAGES_LABEL_SIZE 32 +#define BCH_NVM_PAGES_NAMESPACES_MAX 8 + +#define BCH_NVM_PAGES_OWNER_LIST_HEAD_OFFSET (8<<10) +#define BCH_NVM_PAGES_SYS_RECS_HEAD_OFFSET (16<<10) + +#define BCH_NVM_PAGES_SB_VERSION 0 +#define BCH_NVM_PAGES_SB_VERSION_MAX 0 + +static const unsigned char bch_nvm_pages_magic[] = { +0x17 ,0xbd ,0x53 ,0x7f ,0x1b ,0x23 ,0xd6 ,0x83 , +0x46 ,0xa4 ,0xf8 ,0x28 ,0x17 ,0xda ,0xec ,0xa9 }; +static const unsigned char bch_nvm_pages_pgalloc_magic[] = { +0x39 ,0x25 ,0x3f ,0xf7 ,0x27 ,0x17 ,0xd0 ,0xb9 , +0x10 ,0xe6 ,0xd2 ,0xda ,0x38 ,0x68 ,0x26 ,0xae }; + +/* takes 64bit width */ +struct bch_pgalloc_rec { +__u64 pgoff:52 ; +__u64 order:6 ; +__u64 reserved:6 ; +}; + +struct bch_nvm_pgalloc_recs { +union { +struct { +struct bch_nvm_pages_owner_head *owner; +struct bch_nvm_pgalloc_recs *next; +unsigned char magic[16 ]; +unsigned char owner_uuid[16 ]; +unsigned int size; +unsigned int used; +unsigned long _pad[4 ]; +struct bch_pgalloc_rec recs[]; + }; +unsigned char pad[8192 ]; +}; +}; + +#define BCH_MAX_RECS \ + ((sizeof(struct bch_nvm_pgalloc_recs) - \ + offsetof(struct bch_nvm_pgalloc_recs, recs)) / \ + sizeof(struct bch_pgalloc_rec)) + +struct bch_nvm_pages_owner_head { +unsigned char uuid[16 ]; +unsigned char label[BCH_NVM_PAGES_LABEL_SIZE ]; +/* Per-namespace own lists */ +struct bch_nvm_pgalloc_recs *recs[BCH_NVM_PAGES_NAMESPACES_MAX ]; +}; + +/* heads[0] is always for nvm_pages internal usage */ +struct bch_owner_list_head { +union { +struct { +unsigned int size; +unsigned int used; +unsigned long _pad[4 ]; +struct bch_nvm_pages_owner_head heads[]; + }; +unsigned char pad[8192 ]; +}; +}; +#define BCH_MAX_OWNER_LIST \ + ((sizeof(struct bch_owner_list_head) - \ + offsetof(struct bch_owner_list_head, heads)) / \ + sizeof(struct bch_nvm_pages_owner_head)) + +/* The on-media bit order is local CPU order */ +struct bch_nvm_pages_sb { +unsigned long csum; +unsigned long ns_start; +unsigned long sb_offset; +unsigned long version; +unsigned char magic[16 ]; +unsigned char uuid[16 ]; +unsigned int page_size; +unsigned int total_namespaces_nr; +unsigned int this_namespace_nr; +union { +unsigned char set_uuid[16 ]; +unsigned long set_magic; + }; + +unsigned long flags; +unsigned long seq; + +unsigned long feature_compat; +unsigned long feature_incompat; +unsigned long feature_ro_compat; + +/* For allocable nvm pages from buddy systems */ +unsigned long pages_offset; +unsigned long pages_total; + +unsigned long pad[8 ]; + +/* Only on the first name space */ +struct bch_owner_list_head *owner_list_head; + +/* Just for csum_set() */ +unsigned int keys; +unsigned long d[0 ]; +}; +#endif /* CONFIG_64BIT */ + +#endif /* _UAPI_BCACHE_NVM_H */
Signature · 1 line
2.25.1
PATCH v10 2/6 bcache: initialize the nvm pages allocator
In reply to Qiaowei RenFrom: Jianpeng Ma <[email protected]>
This patch define the prototype data structures in memory and initializes the nvm pages allocator.
The nvm address space which is managed by this allocator can consist of many nvm namespaces, and some namespaces can compose into one nvm set, like cache set. For this initial implementation, only one set can be supported.
The users of this nvm pages allocator need to call register_namespace() to register the nvdimm device (like /dev/pmemX) into this allocator as the instance of struct nvm_namespace.
--- drivers/md/bcache/Kconfig | 9 ++ drivers/md/bcache/Makefile | 1 + drivers/md/bcache/nvm-pages.c | 291 ++++++++++++++++++++++++++++++++++ drivers/md/bcache/nvm-pages.h | 74 +++++++++ drivers/md/bcache/super.c | 3 + 5 files changed, 378 insertions(+) create mode 100644 drivers/md/bcache/nvm-pages.c create mode 100644 drivers/md/bcache/nvm-pages.h
drivers/md/bcache/Kconfig +9 -0
@@ -35,3 +35,12 @@ config BCACHE_ASYNC_REGISTRATION device path into this file will returns immediately and the real registration work is handled in kernel work queue in asynchronous way. + +config BCACHE_NVM_PAGES + bool "NVDIMM support for bcache (EXPERIMENTAL)" + depends on BCACHE + depends on LIBNVDIMM + depends on DAX + help + Allocate/release NV-memory pages for bcache and provide allocated pages + for each requestor after system reboot.
drivers/md/bcache/Makefile +1 -0
@@ -5,3 +5,4 @@ obj-$(CONFIG_BCACHE) += bcache.o bcache-y := alloc.o bset.o btree.o closure.o debug.o extents.o\ io.o journal.o movinggc.o request.o stats.o super.o sysfs.o trace.o\ util.o writeback.o features.o +bcache-$(CONFIG_BCACHE_NVM_PAGES) += nvm-pages.o
drivers/md/bcache/nvm-pages.c new file +291 -0
291 lines not on this page · raw message
drivers/md/bcache/nvm-pages.h new file +74 -0
74 lines not on this page · raw message
drivers/md/bcache/super.c +3 -0
@@ -14,6 +14,7 @@#include "request.h" #include "writeback.h" #include "features.h" +#include "nvm-pages.h" #include <linux/blkdev.h> #include <linux/debugfs.h>
@@ -2816,6 +2817,7 @@ static void bcache_exit(void) {bch_debug_exit ();bch_request_exit (); +bch_nvm_exit ();if (bcache_kobj)kobject_put (bcache_kobj);if (bcache_wq)
@@ -2914,6 +2916,7 @@ static int __init bcache_init(void)bch_debug_init ();closure_debug_init (); +bch_nvm_init (); bcache_is_reboot = false;
Signature · 1 line
2.25.1
PATCH v10 3/6 bcache: initialization of the buddy
In reply to Qiaowei RenFrom: Jianpeng Ma <[email protected]>
This nvm pages allocator will implement the simple buddy to manage the nvm address space. This patch initializes this buddy for new namespace.
the unit of alloc/free of the buddy is page. DAX device has their struct page(in dram or PMEM).
struct { /* ZONE_DEVICE pages */
/** @pgmap: Points to the hosting device page map. */ struct dev_pagemap *pgmap;
void *zone_device_data;
/*
* ZONE_DEVICE private pages are counted as being
* mapped so the next 3 words hold the mapping, index,
* and private fields from the source anonymous or
* page cache page while the page is migrated to device
* private memory.
* ZONE_DEVICE MEMORY_DEVICE_FS_DAX pages also
* use the mapping, index, and private fields when
* pmem backed DAX files are mapped.
*/
};
ZONE_DEVICE pages only use pgmap. Other 4 words[16/32 bytes] don't use. So the second/third word will be used as 'struct list_head ' which list in buddy. The fourth word(that is normal struct page::index) store pgoff which the page-offset in the dax device. And the fifth word (that is normal struct page::private) store order of buddy. page_type will be used to store buddy flags.
--- drivers/md/bcache/nvm-pages.c | 156 +++++++++++++++++++++++++++++++- drivers/md/bcache/nvm-pages.h | 6 ++ include/uapi/linux/bcache-nvm.h | 10 +- 3 files changed, 165 insertions(+), 7 deletions(-)
drivers/md/bcache/nvm-pages.c +152 -4
@@ -32,6 +32,10 @@ static void release_nvm_namespaces(struct bch_nvm_set *nvm_set)for (i =0 ; i < nvm_set->total_namespaces_nr; i++) { ns = nvm_set->nss[i];if (ns) { +kvfree (ns->pages_bitmap); +if (ns->pgalloc_recs_bitmap) +bitmap_free (ns->pgalloc_recs_bitmap); +blkdev_put (ns->bdev,FMODE_READ |FMODE_WRITE |FMODE_EXEC );kfree (ns); }
@@ -46,17 +50,130 @@ static void release_nvm_set(struct bch_nvm_set *nvm_set)kfree (nvm_set); } +static struct page *nvm_vaddr_to_page (struct bch_nvm_namespace *ns,void *addr) +{ +return virt_to_page (addr); +} + +static void *nvm_pgoff_to_vaddr (struct bch_nvm_namespace *ns,pgoff_t pgoff) +{ +return ns->kaddr + (pgoff <<PAGE_SHIFT ); +} + +static inline void remove_owner_space (struct bch_nvm_namespace *ns, +pgoff_t pgoff,u64 nr) +{ +while (nr >0 ) { +unsigned int num = nr >UINT_MAX ?UINT_MAX : nr; + +bitmap_set (ns->pages_bitmap, pgoff, num); + nr -= num; + pgoff += num; + } +} + +#define BCH_PGOFF_TO_KVADDR (pgoff) ((void *)((unsigned long)pgoff << PAGE_SHIFT)) +static int init_owner_info (struct bch_nvm_namespace *ns) {struct bch_owner_list_head *owner_list_head = ns->sb->owner_list_head; +struct bch_nvm_pgalloc_recs *sys_recs; +int i, j, k, rc =0 ;mutex_lock (&only_set->lock); only_set->owner_list_head = owner_list_head; only_set->owner_list_size = owner_list_head->size; only_set->owner_list_used = owner_list_head->used; + +/* remove used space */ +remove_owner_space (ns,0 ,div_u64 (ns->pages_offset, ns->page_size)); + + sys_recs = ns->kaddr +BCH_NVM_PAGES_SYS_RECS_HEAD_OFFSET ; +/* suppose no hole in array */ +for (i =0 ; i < owner_list_head->used; i++) { +struct bch_nvm_pages_owner_head *head = &owner_list_head->heads[i]; + +for (j =0 ; j <BCH_NVM_PAGES_NAMESPACES_MAX ; j++) { +struct bch_nvm_pgalloc_recs *pgalloc_recs = head->recs[j]; +unsigned long offset = (unsigned long )ns->kaddr >>PAGE_SHIFT ; +struct page *page; + +while (pgalloc_recs) { +u32 pgalloc_recs_pos = (unsigned int )(pgalloc_recs - sys_recs); + +if (memcmp (pgalloc_recs->magic, bch_nvm_pages_pgalloc_magic,16 )) { +pr_info ("invalid bch_nvm_pages_pgalloc_magic\n" ); + rc = -EINVAL ; + goto unlock; + } +if (memcmp (pgalloc_recs->owner_uuid, head->uuid,16 )) { +pr_info ("invalid owner_uuid in bch_nvm_pgalloc_recs\n" ); + rc = -EINVAL ; + goto unlock; + } +if (pgalloc_recs->owner != head) { +pr_info ("invalid owner in bch_nvm_pgalloc_recs\n" ); + rc = -EINVAL ; + goto unlock; + } + +/* recs array can has hole */ +for (k =0 ; k < pgalloc_recs->size; k++) { +struct bch_pgalloc_rec *rec = &pgalloc_recs->recs[k]; + +if (rec->pgoff) { +BUG_ON (rec->pgoff <= offset); + +/* init struct page: index/private */ + page =nvm_vaddr_to_page (ns, +BCH_PGOFF_TO_KVADDR (rec->pgoff)); + +set_page_private (page, rec->order); + page->index = rec->pgoff - offset; + +remove_owner_space (ns, + rec->pgoff - offset, +1L << rec->order); + } + } +bitmap_set (ns->pgalloc_recs_bitmap, pgalloc_recs_pos,1 ); + pgalloc_recs = pgalloc_recs->next; + } + } + } +unlock:mutex_unlock (&only_set->lock); -return 0 ; +return rc; +} + +static void init_nvm_free_space (struct bch_nvm_namespace *ns) +{ +unsigned int start, end, pages; +int i; +struct page *page; +pgoff_t pgoff_start; + +bitmap_for_each_clear_region (ns->pages_bitmap, start, end,0 , ns->pages_total) { + pgoff_start = start; + pages = end - start; + +while (pages) { +for (i =BCH_MAX_ORDER -1 ; i >=0 ; i--) { +if ((pgoff_start % (1L << i) ==0 ) && (pages >= (1L << i))) +break ; + } + + page =nvm_vaddr_to_page (ns,nvm_pgoff_to_vaddr (ns, pgoff_start)); + page->index = pgoff_start; +set_page_private (page, i); +__SetPageBuddy (page); +list_add ((struct list_head *)&page->zone_device_data, &ns->free_area[i]); + + pgoff_start +=1L << i; + pages -=1L << i; + } + } }static int attach_nvm_set (struct bch_nvm_namespace *ns)
@@ -163,7 +280,7 @@ static int read_nvdimm_meta_super(struct block_device *bdev,struct bch_nvm_namespace *bch_register_namespace (const char *dev_path) {struct bch_nvm_namespace *ns; -int err; +int i, err;pgoff_t pgoff;char buf[BDEVNAME_SIZE ];struct block_device *bdev;
@@ -247,18 +364,49 @@ struct bch_nvm_namespace *bch_register_namespace(const char *dev_path) ns->nvm_set = only_set;mutex_init (&ns->lock); +/* +* parameters of bitmap_set/clear are unsigned int. +* Given currently size of nvm is far from exceeding this limit, +* so only add a WARN_ON message. +*/ +WARN_ON (BITS_TO_LONGS (ns->pages_total) >UINT_MAX ); + ns->pages_bitmap =kvcalloc (BITS_TO_LONGS (ns->pages_total), +sizeof (unsigned long ),GFP_KERNEL ); +if (!ns->pages_bitmap) { + err = -ENOMEM ; + goto clear_ns_nr; + } + +if (ns->sb->this_namespace_nr ==0 ) { + ns->pgalloc_recs_bitmap =bitmap_zalloc (BCH_MAX_PGALLOC_RECS ,GFP_KERNEL ); +if (ns->pgalloc_recs_bitmap ==NULL ) { + err = -ENOMEM ; + goto free_pages_bitmap; + } + } + +for (i =0 ; i <BCH_MAX_ORDER ; i++) +INIT_LIST_HEAD (&ns->free_area[i]); +if (ns->sb->this_namespace_nr ==0 ) {pr_info ("only first namespace contain owner info\n" ); err =init_owner_info (ns);if (err <0 ) {pr_info ("init_owner_info met error %d\n" , err); - only_set->nss[ns->sb->this_namespace_nr] =NULL ; - goto free_ns; + goto free_recs_bitmap; } +/* init buddy allocator */ +init_nvm_free_space (ns); }kfree (path);return ns; +free_recs_bitmap: +bitmap_free (ns->pgalloc_recs_bitmap); +free_pages_bitmap: +kvfree (ns->pages_bitmap); +clear_ns_nr: + only_set->nss[ns->sb->this_namespace_nr] =NULL ; free_ns:kfree (ns); bdput:
drivers/md/bcache/nvm-pages.h +6 -0
@@ -16,6 +16,7 @@* to which owner. After reboot from power failure, they will be initialized * based on nvm pages superblock in NVDIMM device. */ +#define BCH_MAX_ORDER 20struct bch_nvm_namespace {struct bch_nvm_pages_sb *sb;void *kaddr;
@@ -27,6 +28,11 @@ struct bch_nvm_namespace {u64 pages_total;pfn_t start_pfn; +unsigned long *pages_bitmap; +struct list_head free_area[BCH_MAX_ORDER ]; + +unsigned long *pgalloc_recs_bitmap; +struct dax_device *dax_dev;struct block_device *bdev;struct bch_nvm_set *nvm_set;
include/uapi/linux/bcache-nvm.h +7 -3
@@ -130,11 +130,15 @@ union { }; }; -#define BCH_MAX_RECS \ - ((sizeof(struct bch_nvm_pgalloc_recs) - \ - offsetof(struct bch_nvm_pgalloc_recs, recs)) / \ +#define BCH_MAX_RECS \ + ((sizeof(struct bch_nvm_pgalloc_recs) - \ + offsetof(struct bch_nvm_pgalloc_recs, recs)) / \ sizeof(struct bch_pgalloc_rec)) +#define BCH_MAX_PGALLOC_RECS \ + ((BCH_NVM_PAGES_OFFSET - BCH_NVM_PAGES_SYS_RECS_HEAD_OFFSET) / \ + sizeof(struct bch_nvm_pgalloc_recs)) +struct bch_nvm_pages_owner_head {unsigned char uuid[16 ];unsigned char label[BCH_NVM_PAGES_LABEL_SIZE ];
Signature · 1 line
2.25.1
PATCH v10 4/6 cache: bch_nvm_alloc_pages() of the buddy
In reply to Qiaowei RenFrom: Jianpeng Ma <[email protected]>
This patch implements the bch_nvm_alloc_pages() of the buddy. In terms of function, this func is like current-page-buddy-alloc. But the differences are:
a: it need owner_uuid as parameter which record owner info. And it make those info persistence.
b: it don't need flags like GFP_*. All allocs are the equal.
c: it don't trigger other ops etc swap/recycle.
--- drivers/md/bcache/nvm-pages.c | 174 ++++++++++++++++++++++++++++++++ drivers/md/bcache/nvm-pages.h | 6 ++ include/uapi/linux/bcache-nvm.h | 6 +- 3 files changed, 184 insertions(+), 2 deletions(-)
drivers/md/bcache/nvm-pages.c +174 -0
@@ -72,6 +72,180 @@ static inline void remove_owner_space(struct bch_nvm_namespace *ns, } } +/* If not found, it will create if create == true */ +static struct bch_nvm_pages_owner_head *find_owner_head (const char *owner_uuid,bool create) +{ +struct bch_owner_list_head *owner_list_head = only_set->owner_list_head; +struct bch_nvm_pages_owner_head *owner_head =NULL ; +int i; + +if (owner_list_head ==NULL ) + goto out; + +for (i =0 ; i < only_set->owner_list_used; i++) { +if (!memcmp (owner_uuid, owner_list_head->heads[i].uuid,16 )) { + owner_head = &(owner_list_head->heads[i]); +break ; + } + } + +if (!owner_head && create) { +u32 used = only_set->owner_list_used; + +if (only_set->owner_list_size > used) { +memcpy_flushcache (owner_list_head->heads[used].uuid, owner_uuid,16 ); + only_set->owner_list_used++; + + owner_list_head->used++; + owner_head = &(owner_list_head->heads[used]); + }else +pr_info ("no free bch_nvm_pages_owner_head\n" ); + } + +out: +return owner_head; +} + +static struct bch_nvm_pgalloc_recs *find_empty_pgalloc_recs (void ) +{ +unsigned int start; +struct bch_nvm_namespace *ns = only_set->nss[0 ]; +struct bch_nvm_pgalloc_recs *recs; + + start =bitmap_find_next_zero_area (ns->pgalloc_recs_bitmap,BCH_MAX_PGALLOC_RECS ,0 ,1 ,0 ); +if (start >BCH_MAX_PGALLOC_RECS ) { +pr_info ("no free struct bch_nvm_pgalloc_recs\n" ); +return NULL ; + } + +bitmap_set (ns->pgalloc_recs_bitmap, start,1 ); + recs = (struct bch_nvm_pgalloc_recs *)(ns->kaddr +BCH_NVM_PAGES_SYS_RECS_HEAD_OFFSET ) + + start; +return recs; +} + +static struct bch_nvm_pgalloc_recs *find_nvm_pgalloc_recs (struct bch_nvm_namespace *ns, +struct bch_nvm_pages_owner_head *owner_head,bool create) +{ +int ns_nr = ns->sb->this_namespace_nr; +struct bch_nvm_pgalloc_recs *prev_recs =NULL , *recs = owner_head->recs[ns_nr]; + +/* If create=false, we return recs[nr] */ +if (!create) +return recs; + +/* +* If create=true, it mean we need a empty struct bch_pgalloc_rec +* So we should find non-empty struct bch_nvm_pgalloc_recs or alloc +* new struct bch_nvm_pgalloc_recs. And return this bch_nvm_pgalloc_recs +*/ +while (recs && (recs->used == recs->size)) { + prev_recs = recs; + recs = recs->next; + } + +/* Found empty struct bch_nvm_pgalloc_recs */ +if (recs) +return recs; +/* Need alloc new struct bch_nvm_galloc_recs */ + recs =find_empty_pgalloc_recs (); +if (recs) { + recs->next =NULL ; + recs->owner = owner_head; +memcpy_flushcache (recs->magic, bch_nvm_pages_pgalloc_magic,16 ); +memcpy_flushcache (recs->owner_uuid, owner_head->uuid,16 ); + recs->size =BCH_MAX_RECS ; + recs->used =0 ; + +if (prev_recs) + prev_recs->next = recs; +else + owner_head->recs[ns_nr] = recs; + } + +return recs; +} + +static void add_pgalloc_rec (struct bch_nvm_pgalloc_recs *recs,void *kaddr,int order) +{ +int i; + +for (i =0 ; i < recs->size; i++) { +if (recs->recs[i].pgoff ==0 ) { + recs->recs[i].pgoff = (unsigned long )kaddr >>PAGE_SHIFT ; + recs->recs[i].order = order; + recs->used++; +break ; + } + } +BUG_ON (i == recs->size); +} + +void *bch_nvm_alloc_pages (int order,const char *owner_uuid) +{ +void *kaddr =NULL ; +struct bch_nvm_pgalloc_recs *pgalloc_recs; +struct bch_nvm_pages_owner_head *owner_head; +int i, j; + +mutex_lock (&only_set->lock); + owner_head =find_owner_head (owner_uuid, true); + +if (!owner_head) { +pr_err ("can't find bch_nvm_pgalloc_recs by(uuid=%s)\n" , owner_uuid); + goto unlock; + } + +for (j =0 ; j < only_set->total_namespaces_nr; j++) { +struct bch_nvm_namespace *ns = only_set->nss[j]; + +if (!ns || (ns->free < (1L << order))) +continue ; + +for (i = order; i <BCH_MAX_ORDER ; i++) { +struct list_head *list; +struct page *page, *buddy_page; + +if (list_empty (&ns->free_area[i])) +continue ; + + list = ns->free_area[i].next; + page = container_of((void *)list, struct page, zone_device_data); + +list_del (list); + +while (i != order) { + buddy_page =nvm_vaddr_to_page (ns, +nvm_pgoff_to_vaddr (ns, page->index + (1L << (i -1 )))); +set_page_private (buddy_page, i -1 ); + buddy_page->index = page->index + (1L << (i -1 )); +__SetPageBuddy (buddy_page); +list_add ((struct list_head *)&buddy_page->zone_device_data, + &ns->free_area[i -1 ]); + i--; + } + +set_page_private (page, order); +__ClearPageBuddy (page); + ns->free -=1L << order; + kaddr =nvm_pgoff_to_vaddr (ns, page->index); +break ; + } + +if (i <BCH_MAX_ORDER ) { + pgalloc_recs =find_nvm_pgalloc_recs (ns, owner_head, true); +/* ToDo: handle pgalloc_recs==NULL */ +add_pgalloc_rec (pgalloc_recs, kaddr, order); +break ; + } + } + +unlock: +mutex_unlock (&only_set->lock); +return kaddr; +} +EXPORT_SYMBOL_GPL (bch_nvm_alloc_pages); +#define BCH_PGOFF_TO_KVADDR (pgoff) ((void *)((unsigned long)pgoff << PAGE_SHIFT))static int init_owner_info (struct bch_nvm_namespace *ns)
drivers/md/bcache/nvm-pages.h +6 -0
@@ -62,6 +62,7 @@ extern struct bch_nvm_set *only_set;struct bch_nvm_namespace *bch_register_namespace (const char *dev_path);int bch_nvm_init (void );void bch_nvm_exit (void ); +void *bch_nvm_alloc_pages (int order,const char *owner_uuid);#else
@@ -74,6 +75,11 @@ static inline int bch_nvm_init(void) return 0; }static inline void bch_nvm_exit (void ) { } +static inline void *bch_nvm_alloc_pages (int order,const char *owner_uuid) +{ +return NULL ; +} +#endif /* CONFIG_BCACHE_NVM_PAGES */
include/uapi/linux/bcache-nvm.h +4 -2
@@ -135,9 +135,11 @@ union { offsetof(struct bch_nvm_pgalloc_recs, recs)) / \ sizeof(struct bch_pgalloc_rec)) +/* Currently 64 struct bch_nvm_pgalloc_recs is enough */ #define BCH_MAX_PGALLOC_RECS \ - ((BCH_NVM_PAGES_OFFSET - BCH_NVM_PAGES_SYS_RECS_HEAD_OFFSET) / \ - sizeof(struct bch_nvm_pgalloc_recs)) + (min_t(unsigned int, 64, \ + (BCH_NVM_PAGES_OFFSET - BCH_NVM_PAGES_SYS_RECS_HEAD_OFFSET) / \ + sizeof(struct bch_nvm_pgalloc_recs)))struct bch_nvm_pages_owner_head {unsigned char uuid[16 ];
Signature · 1 line
2.25.1
PATCH v10 5/6 bcache: bch_nvm_free_pages() of the buddy
In reply to Qiaowei RenFrom: Jianpeng Ma <[email protected]>
This patch implements the bch_nvm_free_pages() of the buddy.
The difference between this and page-buddy-free:
it need owner_uuid to free owner allocated pages.And must persistent after free.
--- drivers/md/bcache/nvm-pages.c | 164 ++++++++++++++++++++++++++++++++-- drivers/md/bcache/nvm-pages.h | 3 +- 2 files changed, 159 insertions(+), 8 deletions(-)
drivers/md/bcache/nvm-pages.c +157 -7
@@ -50,7 +50,7 @@ static void release_nvm_set(struct bch_nvm_set *nvm_set)kfree (nvm_set); } -static struct page *nvm_vaddr_to_page (struct bch_nvm_namespace *ns,void *addr) +static struct page *nvm_vaddr_to_page (void *addr) {return virt_to_page (addr); }
@@ -181,6 +181,155 @@ static void add_pgalloc_rec(struct bch_nvm_pgalloc_recs *recs, void *kaddr, intBUG_ON (i == recs->size); } +static inline void *nvm_end_addr (struct bch_nvm_namespace *ns) +{ +return ns->kaddr + (ns->pages_total <<PAGE_SHIFT ); +} + +static inline bool in_nvm_range (struct bch_nvm_namespace *ns, +void *start_addr,void *end_addr) +{ +return (start_addr >= ns->kaddr) && (end_addr <nvm_end_addr (ns)); +} + +static struct bch_nvm_namespace *find_nvm_by_addr (void *addr,int order) +{ +int i; +struct bch_nvm_namespace *ns; + +for (i =0 ; i < only_set->total_namespaces_nr; i++) { + ns = only_set->nss[i]; +if (ns &&in_nvm_range (ns, addr, addr + (1L << order))) +return ns; + } +return NULL ; +} + +static int remove_pgalloc_rec (struct bch_nvm_pgalloc_recs *pgalloc_recs,int ns_nr, +void *kaddr,int order) +{ +struct bch_nvm_pages_owner_head *owner_head = pgalloc_recs->owner; +struct bch_nvm_pgalloc_recs *prev_recs, *sys_recs; +u64 pgoff = (unsigned long )kaddr >>PAGE_SHIFT ; +struct bch_nvm_namespace *ns = only_set->nss[0 ]; +int i; + + prev_recs = pgalloc_recs; + sys_recs = ns->kaddr +BCH_NVM_PAGES_SYS_RECS_HEAD_OFFSET ; +while (pgalloc_recs) { +for (i =0 ; i < pgalloc_recs->size; i++) { +struct bch_pgalloc_rec *rec = &(pgalloc_recs->recs[i]); + +if (rec->pgoff == pgoff) { +WARN_ON (rec->order != order); + rec->pgoff =0 ; + rec->order =0 ; + pgalloc_recs->used--; + +if (pgalloc_recs->used ==0 ) { +int recs_pos = pgalloc_recs - sys_recs; + +if (pgalloc_recs == prev_recs) + owner_head->recs[ns_nr] = pgalloc_recs->next; +else + prev_recs->next = pgalloc_recs->next; + + pgalloc_recs->next =NULL ; + pgalloc_recs->owner =NULL ; + +bitmap_clear (ns->pgalloc_recs_bitmap, recs_pos,1 ); + } + goto exit; + } + } + prev_recs = pgalloc_recs; + pgalloc_recs = pgalloc_recs->next; + } +exit: +return pgalloc_recs ?0 : -ENOENT ; +} + +static void __free_space (struct bch_nvm_namespace *ns,void *addr,int order) +{ +unsigned long add_pages = (1L << order); +pgoff_t pgoff; +struct page *page; + + page =nvm_vaddr_to_page (addr); +WARN_ON ((!page) || (page->private != order)); + pgoff = page->index; + +while (order <BCH_MAX_ORDER -1 ) { +struct page *buddy_page; + +pgoff_t buddy_pgoff = pgoff ^ (1L << order); +pgoff_t parent_pgoff = pgoff & ~(1L << order); + +if ((parent_pgoff + (1L << (order +1 )) > ns->pages_total)) +break ; + + buddy_page =nvm_vaddr_to_page (nvm_pgoff_to_vaddr (ns, buddy_pgoff)); +WARN_ON (!buddy_page); + +if (PageBuddy (buddy_page) && (buddy_page->private == order)) { +list_del ((struct list_head *)&buddy_page->zone_device_data); +__ClearPageBuddy (buddy_page); + pgoff = parent_pgoff; + order++; +continue ; + } +break ; + } + + page =nvm_vaddr_to_page (nvm_pgoff_to_vaddr (ns, pgoff)); +WARN_ON (!page); +list_add ((struct list_head *)&page->zone_device_data, &ns->free_area[order]); + page->index = pgoff; +set_page_private (page, order); +__SetPageBuddy (page); + ns->free += add_pages; +} + +void bch_nvm_free_pages (void *addr,int order,const char *owner_uuid) +{ +struct bch_nvm_namespace *ns; +struct bch_nvm_pages_owner_head *owner_head; +struct bch_nvm_pgalloc_recs *pgalloc_recs; +int r; + +mutex_lock (&only_set->lock); + + ns =find_nvm_by_addr (addr, order); +if (!ns) { +pr_err ("can't find nvm_dev by kaddr %px\n" , addr); + goto unlock; + } + + owner_head =find_owner_head (owner_uuid, false); +if (!owner_head) { +pr_err ("can't found bch_nvm_pages_owner_head by(uuid=%s)\n" , owner_uuid); + goto unlock; + } + + pgalloc_recs =find_nvm_pgalloc_recs (ns, owner_head, false); +if (!pgalloc_recs) { +pr_err ("can't find bch_nvm_pgalloc_recs by(uuid=%s)\n" , owner_uuid); + goto unlock; + } + + r =remove_pgalloc_rec (pgalloc_recs, ns->sb->this_namespace_nr, addr, order); +if (r <0 ) { +pr_err ("can't find bch_pgalloc_rec\n" ); + goto unlock; + } + +__free_space (ns, addr, order); + +unlock: +mutex_unlock (&only_set->lock); +} +EXPORT_SYMBOL_GPL (bch_nvm_free_pages); +void *bch_nvm_alloc_pages (int order,const char *owner_uuid) {void *kaddr =NULL ;
@@ -215,7 +364,7 @@ void *bch_nvm_alloc_pages(int order, const char *owner_uuid)list_del (list);while (i != order) { - buddy_page =nvm_vaddr_to_page (ns, + buddy_page =nvm_vaddr_to_page (nvm_pgoff_to_vaddr (ns, page->index + (1L << (i -1 ))));set_page_private (buddy_page, i -1 ); buddy_page->index = page->index + (1L << (i -1 ));
@@ -299,7 +448,7 @@ static int init_owner_info(struct bch_nvm_namespace *ns)BUG_ON (rec->pgoff <= offset);/* init struct page: index/private */ - page =nvm_vaddr_to_page (ns, + page =nvm_vaddr_to_page (BCH_PGOFF_TO_KVADDR (rec->pgoff));set_page_private (page, rec->order);
@@ -338,11 +487,12 @@ static void init_nvm_free_space(struct bch_nvm_namespace *ns)break ; } - page =nvm_vaddr_to_page (ns,nvm_pgoff_to_vaddr (ns, pgoff_start)); + page =nvm_vaddr_to_page (nvm_pgoff_to_vaddr (ns, pgoff_start)); page->index = pgoff_start;set_page_private (page, i); -__SetPageBuddy (page); -list_add ((struct list_head *)&page->zone_device_data, &ns->free_area[i]); + +/* in order to update ns->free */ +__free_space (ns,nvm_pgoff_to_vaddr (ns, pgoff_start), i); pgoff_start +=1L << i; pages -=1L << i;
@@ -533,7 +683,7 @@ struct bch_nvm_namespace *bch_register_namespace(const char *dev_path) ns->page_size = ns->sb->page_size; ns->pages_offset = ns->sb->pages_offset; ns->pages_total = ns->sb->pages_total; - ns->free =0 ; + ns->free =0 ;/* increase by __free_space() */ ns->bdev = bdev; ns->nvm_set = only_set;mutex_init (&ns->lock);
drivers/md/bcache/nvm-pages.h +2 -1
@@ -63,6 +63,7 @@ struct bch_nvm_namespace *bch_register_namespace(const char *dev_path);int bch_nvm_init (void );void bch_nvm_exit (void );void *bch_nvm_alloc_pages (int order,const char *owner_uuid); +void bch_nvm_free_pages (void *addr,int order,const char *owner_uuid);#else
@@ -79,7 +80,7 @@ static inline void *bch_nvm_alloc_pages(int order, const char *owner_uuid) {return NULL ; } - +static inline void bch_nvm_free_pages (void *addr,int order,const char *owner_uuid) { }#endif /* CONFIG_BCACHE_NVM_PAGES */
Signature · 1 line
2.25.1
PATCH v10 6/6 bcache: get allocated pages from specific owner
In reply to Qiaowei RenFrom: Jianpeng Ma <[email protected]>
This patch implements bch_get_allocated_pages() of the buddy to be used to get allocated pages from specific owner.
--- drivers/md/bcache/nvm-pages.c | 6 ++++++ drivers/md/bcache/nvm-pages.h | 5 +++++ 2 files changed, 11 insertions(+)
drivers/md/bcache/nvm-pages.c +6 -0
@@ -395,6 +395,12 @@ void *bch_nvm_alloc_pages(int order, const char *owner_uuid) }EXPORT_SYMBOL_GPL (bch_nvm_alloc_pages); +struct bch_nvm_pages_owner_head *bch_get_allocated_pages (const char *owner_uuid) +{ +return find_owner_head (owner_uuid, false); +} +EXPORT_SYMBOL_GPL (bch_get_allocated_pages); +#define BCH_PGOFF_TO_KVADDR (pgoff) ((void *)((unsigned long)pgoff << PAGE_SHIFT))static int init_owner_info (struct bch_nvm_namespace *ns)
drivers/md/bcache/nvm-pages.h +5 -0
@@ -64,6 +64,7 @@ int bch_nvm_init(void);void bch_nvm_exit (void );void *bch_nvm_alloc_pages (int order,const char *owner_uuid);void bch_nvm_free_pages (void *addr,int order,const char *owner_uuid); +struct bch_nvm_pages_owner_head *bch_get_allocated_pages (const char *owner_uuid);#else
@@ -81,6 +82,10 @@ static inline void *bch_nvm_alloc_pages(int order, const char *owner_uuid) return NULL; }static inline void bch_nvm_free_pages (void *addr,int order,const char *owner_uuid) { } +static inline struct bch_nvm_pages_owner_head *bch_get_allocated_pages (const char *owner_uuid) +{ +return NULL ; +}#endif /* CONFIG_BCACHE_NVM_PAGES */
Signature · 1 line
2.25.1