libnvdimm: reset seeds for next namespace creation
cover libnvdimm: reset seeds for next namespace creation
From: Ocean He <[email protected]>
When pmem namespaces created are smaller than section size twice, the second creation would fail and meanwhile there is a kernel call trace which comes from commit 15d36fecd0bdc7510b70 ("mm: disallow mappings that conflict for devm_memremap_pages()").
------------[ cut here ]------------ nd_pmem pfn1.1: Conflicting mapping in same section WARNING: CPU: 84 PID: 51974 at kernel/memremap.c:194 devm_memremap_pages+0x4a0/0x4e0 CPU: 84 PID: 51974 Comm: ndctl Kdump: loaded Tainted: G W E 4.19.0-rc2-23-default+ #27 RIP: 0010:devm_memremap_pages+0x4a0/0x4e0 Call Trace: pmem_attach_disk+0x3ab/0x581 [nd_pmem] nvdimm_bus_probe+0x69/0x150 [libnvdimm] really_probe+0x262/0x3d0 driver_probe_device+0x60/0x120 bind_store+0x102/0x190 kernfs_fop_write+0x105/0x180 __vfs_write+0x36/0x1a0 ? common_file_perm+0x47/0x130 ? security_file_permission+0x2c/0xb0 vfs_write+0xad/0x1a0 ksys_write+0x52/0xc0 do_syscall_64+0x5b/0x180 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Here is an example (section size is 128MB) based on kernel 4.19-rc2.
# ndctl create-namespace -r region1 -s 100m -t pmem -m fsdax
{
"dev":"namespace1.0",
"mode":"fsdax",
"map":"dev",
"size":"96.00 MiB (100.66 MB)",
"uuid":"ef9a0556-a610-40b5-8c71-43991765a2cc",
"raw_uuid":"177b22e2-b7e8-482f-a063-2b8de876d979",
"sector_size":512,
"blockdev":"pmem1",
"numa_node":1
}
# ndctl create-namespace -r region1 -s 100m -t pmem -m fsdax
libndctl: ndctl_pfn_enable: pfn1.1: failed to enable
Error: namespace1.1: failed to enable
failed to create namespace: No such device or addressWhen above second creation failure occurs, the expectation is to destroy namespace1.0 to create a new namespace which size is aligned with section size. However, both namespace seed and pfn seed have been consumed, the new namespace creation still fails.
# ndctl destroy-namespace namespace1.0 -f destroyed 1 namespace # ndctl create-namespace -r region1 -s 128m -t pmem -m fsdax failed to create namespace: Device or resource busy
To ensure pfn_seed/dax_seed and namespace_seed are always ready for next namespace creation, this patch set enables seed detach and reset. Back to the example, the new namespace creation never fails if this patch set applied.
# ndctl destroy-namespace namespace1.0 -f
destroyed 1 namespace
# ndctl create-namespace -r region1 -s 128m -t pmem -m fsdax
{
"dev":"namespace1.0",
"mode":"fsdax",
"map":"dev",
"size":"124.00 MiB (130.02 MB)",
"uuid":"0d0e7506-d108-4a88-824a-edef26fd0399",
"raw_uuid":"efeb9647-12f5-44cd-8a52-2f3a0d14589a",
"sector_size":512,
"blockdev":"pmem1",
"numa_node":1
}
# ndctl create-namespace -r region1 -s 128m -t pmem -m fsdax
{
"dev":"namespace1.1",
"mode":"fsdax",
"map":"dev",
"size":130023424,
"uuid":"689828dc-8779-434d-8e93-0406d4e1e536",
"raw_uuid":"d86e1025-c224-48b6-b2a7-6ccef152d5fd",
"sector_size":512,
"blockdev":"pmem1.1",
"numa_node":1
}The mode devdax (-m devdax) has the same issue, this patch set could cover it.
Ocean He (3):
libnvdimm, claim: remove static attribute of nd_detach_and_reset libnvdimm, namespace_devs: add function nd_region_reset_ns_seed for
namespace seed reset
libnvdimm, region_devs: reset related seeds when fail to create
namespace
drivers/nvdimm/claim.c | 2 +- drivers/nvdimm/namespace_devs.c | 32 ++++++++++++++++++++++++++++++++ drivers/nvdimm/nd-core.h | 2 ++ drivers/nvdimm/region_devs.c | 34 ++++++++++++++++++++++++++++++++++ 4 files changed, 69 insertions(+), 1 deletion(-)
Signature · 6 lines
1.8.3.1 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 1/3 libnvdimm, claim: remove static attribute of nd_detach_and_reset
From: Ocean He <[email protected]>
The function nd_detach_and_reset needs to be called externally, so remove the static attribute and declare it in nd-core.h.
--- drivers/nvdimm/claim.c | 2 +- drivers/nvdimm/nd-core.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
drivers/nvdimm/claim.c +1 -1
@@ -117,7 +117,7 @@ struct nd_pfn *to_nd_pfn_safe(struct device *dev)return NULL ; } -static void nd_detach_and_reset (struct device *dev, +void nd_detach_and_reset (struct device *dev,struct nd_namespace_common **_ndns) {/* detach the namespace and destroy / reset the device */
drivers/nvdimm/nd-core.h +1 -0
@@ -123,6 +123,7 @@ struct resource *nsblk_add_resource(struct nd_region *nd_region,resource_size_t __nvdimm_namespace_capacity (struct nd_namespace_common *ndns);void nd_detach_ndns (struct device *dev,struct nd_namespace_common **_ndns);void __nd_detach_ndns (struct device *dev,struct nd_namespace_common **_ndns); +void nd_detach_and_reset (struct device *dev,struct nd_namespace_common **_ndns);bool nd_attach_ndns (struct device *dev,struct nd_namespace_common *attach,struct nd_namespace_common **_ndns); bool __nd_attach_ndns(struct device *dev, struct nd_namespace_common *attach,
Signature · 6 lines
1.8.3.1 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 2/3 libnvdimm, namespace_devs: add function nd_region_reset_ns_seed for namespace seed reset
In reply to Ocean HeFrom: Ocean He <[email protected]>
During runtime, if a namespace seed is used for new namespace creation but fail, then it needs to be reset for next namespace creation.
Add function nd_region_reset_ns_seed for namespace seed reset and declare it in nd-core.h.
--- drivers/nvdimm/namespace_devs.c | 32 ++++++++++++++++++++++++++++++++ drivers/nvdimm/nd-core.h | 1 + 2 files changed, 33 insertions(+)
drivers/nvdimm/namespace_devs.c +32 -0
@@ -2129,6 +2129,38 @@ void nd_region_create_ns_seed(struct nd_region *nd_region)nd_device_register (nd_region->ns_seed); } +void nd_region_reset_ns_seed (struct nd_region *nd_region) +{ +struct device *dev = nd_region->ns_seed; +unsigned long long val =0 ; +ssize_t rc; +u8 **uuid =NULL ; + + rc =__holder_class_store (dev,"" ); +dev_dbg (dev,"%s(%zd)\n" , rc <0 ?"fail to reset claim_class " :"" , rc); + + rc =__size_store (dev, val); +if (rc >=0 ) + rc =nd_namespace_label_update (nd_region, dev); +dev_dbg (dev,"%s(%zd)\n" , rc <0 ?"fail to reset size " :"" , rc); + +if (is_namespace_pmem (dev)) { +struct nd_namespace_pmem *nspm =to_nd_namespace_pmem (dev); + + uuid = &nspm->uuid; + }else if (is_namespace_blk (dev)) { +struct nd_namespace_blk *nsblk =to_nd_namespace_blk (dev); + + uuid = &nsblk->uuid; + } + +if (rc ==0 && val ==0 && uuid) { +/* setting size zero == 'delete namespace' */ +kfree (*uuid); + *uuid =NULL ; + } +} +void nd_region_create_dax_seed (struct nd_region *nd_region) {WARN_ON (!is_nvdimm_bus_locked (&nd_region->dev));
drivers/nvdimm/nd-core.h +1 -0
@@ -81,6 +81,7 @@ static inline bool is_memory(struct device *dev)void nd_region_probe_success (struct nvdimm_bus *nvdimm_bus,struct device *dev);struct nd_region ;void nd_region_create_ns_seed (struct nd_region *nd_region); +void nd_region_reset_ns_seed (struct nd_region *nd_region);void nd_region_create_btt_seed (struct nd_region *nd_region);void nd_region_create_pfn_seed (struct nd_region *nd_region);void nd_region_create_dax_seed (struct nd_region *nd_region);
Signature · 6 lines
1.8.3.1 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
PATCH 3/3 libnvdimm, region_devs: reset related seeds when fail to create namespace
In reply to Ocean HeFrom: Ocean He <[email protected]>
During runtime, namespace creation may fail if blocked by commit 15d36fecd0bdc7510b70 ("mm: disallow mappings that conflict for devm_memremap_pages()"). To ensure pfn_seed/dax_seed and namespace_seed are ready for next namespace creation, here to do detach and reset.
--- drivers/nvdimm/region_devs.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+)
drivers/nvdimm/region_devs.c +34 -0
@@ -703,7 +703,27 @@ void nd_mapping_free_labels(struct nd_mapping *nd_mapping)kfree (label_ent); } } +/* +* To ensure pfn_seed/dax_seed and namespace_seed are ready for +* next namespace creation, here to do detach and reset. +*/ +void nd_region_detach_and_reset (struct device *dev, +struct nd_region *nd_region) +{ +/* Only nd_pmem has been verified, fix me for other dev type. */ +if (!is_nd_pmem (&nd_region->dev)) +return ; +if (is_nd_pfn (dev) ||is_nd_dax (dev)) { +struct nd_pfn *nd_pfn =to_nd_pfn_safe (dev); +struct nd_namespace_common *ndns =to_ndns (nd_region->ns_seed); + +if (nd_pfn->ndns == ndns && ndns->claim == dev) { +nd_detach_and_reset (dev, &nd_pfn->ndns); +nd_region_reset_ns_seed (nd_region); + } + } +}/* * Upon successful probe/remove, take/release a reference on the * associated interleave set (if present), and plant new btt + namespace
@@ -774,6 +794,20 @@ static void nd_region_notify_driver_action(struct nvdimm_bus *nvdimm_bus,nd_region_create_ns_seed (nd_region);nvdimm_bus_unlock (dev); } +if (is_nd_pfn (dev) && !probe) { + nd_region =to_nd_region (dev->parent); +nvdimm_bus_lock (dev); +if (nd_region->pfn_seed == dev) +nd_region_detach_and_reset (dev, nd_region); +nvdimm_bus_unlock (dev); + } +if (is_nd_dax (dev) && !probe) { + nd_region =to_nd_region (dev->parent); +nvdimm_bus_lock (dev); +if (nd_region->dax_seed == dev) +nd_region_detach_and_reset (dev, nd_region); +nvdimm_bus_unlock (dev); + } }void nd_region_probe_success (struct nvdimm_bus *nvdimm_bus,struct device *dev)
Signature · 6 lines
1.8.3.1 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
kbuild test robot
build bot[RFC PATCH] libnvdimm, region_devs: nd_region_detach_and_reset() can be static
Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
--- region_devs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvdimm/region_devs.c +1 -1
@@ -707,7 +707,7 @@ void nd_mapping_free_labels(struct nd_mapping *nd_mapping)* To ensure pfn_seed/dax_seed and namespace_seed are ready for * next namespace creation, here to do detach and reset. */ -void nd_region_detach_and_reset (struct device *dev, +static void nd_region_detach_and_reset (struct device *dev,struct nd_region *nd_region) {/* Only nd_pmem has been verified, fix me for other dev type. */
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
kbuild test robot
build botRe: [PATCH 3/3] libnvdimm, region_devs: reset related seeds when fail to create namespace
In reply to Ocean HeThank you for the patch! Perhaps something to improve:
Hi Ocean,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on linux-nvdimm/libnvdimm-for-next]
[also build test WARNING on v4.19-rc3 next-20180910]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Ocean-He/libnvdimm-reset-seeds-for-next-namespace-creation/20180911-005505 base: https://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm.git libnvdimm-for-next reproduce: # apt-get install sparse make ARCH=x86_64 allmodconfig make C=1 CF=-D__CHECK_ENDIAN__
sparse warnings: (new ones prefixed by >>)
drivers/nvdimm/region_devs.c:84:29: sparse: expression using sizeof(void) drivers/nvdimm/region_devs.c:84:29: sparse: expression using sizeof(void) drivers/nvdimm/region_devs.c:405:37: sparse: expression using sizeof(void) drivers/nvdimm/region_devs.c:405:37: sparse: expression using sizeof(void)
>> drivers/nvdimm/region_devs.c:710:6: sparse: symbol 'nd_region_detach_and_reset' was not declared. Should it be static?
drivers/nvdimm/region_devs.c:1127:31: sparse: expression using sizeof(void) drivers/nvdimm/region_devs.c:985:9: sparse: context imbalance in 'nd_region_acquire_lane' - wrong count at exit drivers/nvdimm/region_devs.c:998:36: sparse: context imbalance in 'nd_region_release_lane' - unexpected unlock
Please review and possibly fold the followup patch.
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 84 lines · Ocean He
From: Ocean He <[email protected]>
When pmem namespaces created are smaller than section size twice, the second creation would fail and meanwhile there is a kernel call trace which comes from commit
15d36fecd0bdc7510b70("mm: disallow mappings that conflict fordevm_memremap_pages()").------------[ cut here ]------------ nd_pmem pfn1.1: Conflicting mapping in same section WARNING: CPU: 84 PID: 51974 at kernel/memremap.c:194 devm_memremap_pages+0x4a0/0x4e0 CPU: 84 PID: 51974 Comm: ndctl Kdump: loaded Tainted: G W E 4.19.0-rc2-23-default+ #27 RIP: 0010:devm_memremap_pages+0x4a0/0x4e0 Call Trace: pmem_attach_disk+0x3ab/0x581 [nd_pmem] nvdimm_bus_probe+0x69/0x150 [libnvdimm] really_probe+0x262/0x3d0 driver_probe_device+0x60/0x120 bind_store+0x102/0x190 kernfs_fop_write+0x105/0x180 __vfs_write+0x36/0x1a0 ? common_file_perm+0x47/0x130 ? security_file_permission+0x2c/0xb0 vfs_write+0xad/0x1a0 ksys_write+0x52/0xc0 do_syscall_64+0x5b/0x180 entry_SYSCALL_64_after_hwframe+0x44/0xa9Here is an example (section size is 128MB) based on kernel 4.19-rc2.
# ndctl create-namespace -r region1 -s 100m -t pmem -m fsdax { "dev":"namespace1.0", "mode":"fsdax", "map":"dev", "size":"96.00 MiB (100.66 MB)", "uuid":"ef9a0556-a610-40b5-8c71-43991765a2cc", "raw_uuid":"177b22e2-b7e8-482f-a063-2b8de876d979", "sector_size":512, "blockdev":"pmem1", "numa_node":1 } # ndctl create-namespace -r region1 -s 100m -t pmem -m fsdax libndctl: ndctl_pfn_enable: pfn1.1: failed to enable Error: namespace1.1: failed to enable failed to create namespace: No such device or addressWhen above second creation failure occurs, the expectation is to destroy namespace1.0 to create a new namespace which size is aligned with section size. However, both namespace seed and pfn seed have been consumed, the new namespace creation still fails.
# ndctl destroy-namespace namespace1.0 -f destroyed 1 namespace # ndctl create-namespace -r region1 -s 128m -t pmem -m fsdax failed to create namespace: Device or resource busyTo ensure pfn_seed/dax_seed and
namespace_seedare always ready for next namespace creation, this patch set enables seed detach and reset. Back to the example, the new namespace creation never fails if this patch set applied.# ndctl destroy-namespace namespace1.0 -f destroyed 1 namespace # ndctl create-namespace -r region1 -s 128m -t pmem -m fsdax { "dev":"namespace1.0", "mode":"fsdax", "map":"dev", "size":"124.00 MiB (130.02 MB)", "uuid":"0d0e7506-d108-4a88-824a-edef26fd0399", "raw_uuid":"efeb9647-12f5-44cd-8a52-2f3a0d14589a", "sector_size":512, "blockdev":"pmem1", "numa_node":1 } # ndctl create-namespace -r region1 -s 128m -t pmem -m fsdax { "dev":"namespace1.1", "mode":"fsdax", "map":"dev", "size":130023424, "uuid":"689828dc-8779-434d-8e93-0406d4e1e536", "raw_uuid":"d86e1025-c224-48b6-b2a7-6ccef152d5fd", "sector_size":512, "blockdev":"pmem1.1", "numa_node":1 }
Quoted 2 lines
The mode devdax (-m devdax) has the same issue, this patch set could cover it.
This is good analysis, but I believe this is better fixed / handled in ndctl directly. This is just one of a few reasons that namespace creation can fail, and it should be ndctl's job to recover from failed creation. The kernel only provides the mechanism the policy of what to do with errors and interrupted namespace creation is up to userspace.
Also, in the future, the plan is to allow namespaces smaller than a section size which will fix this particular failing condition properly.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 103 lines
-----Original Message-----
From: Dan Williams <[email protected]>
Sent: Tuesday, September 11, 2018 8:51 AM
To: Ocean He <[email protected]>
Cc: [email protected]; Vishal L Verma <[email protected]>; Dave Jiang <[email protected]>; linux-nvdimm <[email protected]>; Linux Kernel Mailing List <[email protected]>; Ocean HY1 He <[email protected]>
Subject: [External] Re: [PATCH 0/3] libnvdimm: reset seeds for next namespace creationQuoted 5 lines · Ocean He
From: Ocean He <[email protected]>
When pmem namespaces created are smaller than section size twice, the second creation would fail and meanwhile there is a kernel call trace which comes from commit
15d36fecd0bdc7510b70("mm: disallow mappingsthat
Quoted 4 lines
conflict for
devm_memremap_pages()").------------[ cut here ]------------ nd_pmem pfn1.1: Conflicting mapping in same section WARNING: CPU: 84 PID: 51974 at kernel/memremap.c:194
devm_memremap_pages+0x4a0/0x4e0Quoted 1 line
CPU: 84 PID: 51974 Comm: ndctl Kdump: loaded Tainted: G W E 4.19.0-rc2-23-default+ #27
Quoted 34 lines
RIP: 0010:devm_memremap_pages+0x4a0/0x4e0 Call Trace: pmem_attach_disk+0x3ab/0x581 [nd_pmem] nvdimm_bus_probe+0x69/0x150 [libnvdimm] really_probe+0x262/0x3d0 driver_probe_device+0x60/0x120 bind_store+0x102/0x190 kernfs_fop_write+0x105/0x180 __vfs_write+0x36/0x1a0 ? common_file_perm+0x47/0x130 ? security_file_permission+0x2c/0xb0 vfs_write+0xad/0x1a0 ksys_write+0x52/0xc0 do_syscall_64+0x5b/0x180 entry_SYSCALL_64_after_hwframe+0x44/0xa9Here is an example (section size is 128MB) based on kernel 4.19-rc2.
# ndctl create-namespace -r region1 -s 100m -t pmem -m fsdax { "dev":"namespace1.0", "mode":"fsdax", "map":"dev", "size":"96.00 MiB (100.66 MB)", "uuid":"ef9a0556-a610-40b5-8c71-43991765a2cc", "raw_uuid":"177b22e2-b7e8-482f-a063-2b8de876d979", "sector_size":512, "blockdev":"pmem1", "numa_node":1 } # ndctl create-namespace -r region1 -s 100m -t pmem -m fsdax libndctl: ndctl_pfn_enable: pfn1.1: failed to enable Error: namespace1.1: failed to enable failed to create namespace: No such device or addressQuoted 3 lines
When above second creation failure occurs, the expectation is to destroy namespace1.0 to create a new namespace which size is aligned with section size. However, both namespace seed and pfn seed have been consumed,
the
Quoted 7 lines
new namespace creation still fails.
# ndctl destroy-namespace namespace1.0 -f destroyed 1 namespace # ndctl create-namespace -r region1 -s 128m -t pmem -m fsdax failed to create namespace: Device or resource busyTo ensure pfn_seed/dax_seed and
namespace_seedare always ready fornext
Quoted 30 lines
namespace creation, this patch set enables seed detach and reset. Back to the example, the new namespace creation never fails if this patch set applied.
# ndctl destroy-namespace namespace1.0 -f destroyed 1 namespace # ndctl create-namespace -r region1 -s 128m -t pmem -m fsdax { "dev":"namespace1.0", "mode":"fsdax", "map":"dev", "size":"124.00 MiB (130.02 MB)", "uuid":"0d0e7506-d108-4a88-824a-edef26fd0399", "raw_uuid":"efeb9647-12f5-44cd-8a52-2f3a0d14589a", "sector_size":512, "blockdev":"pmem1", "numa_node":1 } # ndctl create-namespace -r region1 -s 128m -t pmem -m fsdax { "dev":"namespace1.1", "mode":"fsdax", "map":"dev", "size":130023424, "uuid":"689828dc-8779-434d-8e93-0406d4e1e536", "raw_uuid":"d86e1025-c224-48b6-b2a7-6ccef152d5fd", "sector_size":512, "blockdev":"pmem1.1", "numa_node":1 }
Quoted 9 lines
Quoted 2 lines
The mode devdax (-m devdax) has the same issue, this patch set could cover it.
This is good analysis, but I believe this is better fixed / handled in ndctl directly. This is just one of a few reasons that namespace creation can fail, and it should be ndctl's job to recover from failed creation. The kernel only provides the mechanism the policy of what to do with errors and interrupted namespace creation is up to userspace.
Well, thanks for your review. I just send out the patch of ndctl for this issue, please help to review again. Many thanks!
https://lists.01.org/pipermail/linux-nvdimm/2018-September/017778.html
Ocean.
Quoted 3 lines
Also, in the future, the plan is to allow namespaces smaller than a section size which will fix this particular failing condition properly.
I am interesting that what minimal size is allowed for namespace creation. I need this to guide the NVDIMM enablement on Lenovo ThinkSystem Servers.
I see function nvdimm_namespace_common_probe return error if size is less than ND_MIN_NAMESPACE_SIZE(equals to PAGE_SIZE).
size = nvdimm_namespace_capacity(ndns);
if (size < ND_MIN_NAMESPACE_SIZE) {
dev_dbg(&ndns->dev, "%pa, too small must be at least %#x\n",
&size, ND_MIN_NAMESPACE_SIZE);
return ERR_PTR(-ENODEV);
}I also see function nd_namespace_store return error if size is less than SZ_16M.
if (__nvdimm_namespace_capacity(ndns) < SZ_16M) {
dev_dbg(dev, "%s too small to host\n", name);
len = -ENXIO;
goto out_attach;
}Ocean.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm