Skip to contentSkip to conversation

PMEM driver for v4.1

GIT PULL Ingo Molnar 8 participants 30 messages Apr 13, 2015 – May 27, 2015

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Linus,

Please pull the latest x86-pmem-for-linus git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86-pmem-for-linus

# HEAD: 4c1eaa2344fb26bb5e936fb4d8ee307343ea0089 drivers/block/pmem: Fix 32-bit build warning in pmem_alloc()

This is the initial support for the pmem block device driver:
persistent non-volatile memory space mapped into the system's physical memory space as large physical memory regions.

The driver is based on Intel code, written by Ross Zwisler, with fixes by Boaz Harrosh, integrated with x86 e820 memory resource management and tidied up by Christoph Hellwig.

Note that there were two other separate pmem driver submissions to
lkml: but apparently all parties (Ross Zwisler, Boaz Harrosh) are reasonably happy with this initial version.

This version enables minimal support that enables persistent memory devices out in the wild to work as block devices, identified through a magic (non-standard) e820 flag and auto-discovered if CONFIG_X86_PMEM_LEGACY=y, or added explicitly through manipulating the memory maps via the "memmap=..." boot option with the new, special '!' modifier character.

Limitations: this is a regular block device, and since the pmem areas are not struct page backed, they are invisible to the rest of the system (other than the block IO device), so direct IO to/from pmem areas, direct mmap() or XIP is not possible yet. The page cache will also shadow and double buffer pmem contents, etc.

Initial support is for x86.

Thanks,

Ingo

------------------>
Christoph Hellwig (1):
x86/mm: Add support for the non-standard protected e820 type

Ingo Molnar (1):
drivers/block/pmem: Fix 32-bit build warning in pmem_alloc()

Ross Zwisler (1):
drivers/block/pmem: Add a driver for persistent memory

 Documentation/kernel-parameters.txt |   6 +
 MAINTAINERS                         |   6 +
 arch/x86/Kconfig                    |  10 ++
 arch/x86/include/uapi/asm/e820.h    |  10 ++
 arch/x86/kernel/Makefile            |   1 +
 arch/x86/kernel/e820.c              |  26 +++-
 arch/x86/kernel/pmem.c              |  53 ++++++++
 drivers/block/Kconfig               |  11 ++
 drivers/block/Makefile              |   1 +
 drivers/block/pmem.c                | 262 ++++++++++++++++++++++++++++++++++++
 10 files changed, 380 insertions(+), 6 deletions(-)
 create mode 100644 arch/x86/kernel/pmem.c
 create mode 100644 drivers/block/pmem.c
Documentation/kernel-parameters.txt +6 -0
index bfcb1a62a7b4..c87122dd790f 100644
@@ -1965,6 +1965,12 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			         or
 			         memmap=0x10000$0x18690000
 
+	memmap=nn[KMG]!ss[KMG]
+			[KNL,X86] Mark specific memory as protected.
+			Region of memory to be used, from ss to ss+nn.
+			The memory region may be marked as e820 type 12 (0xc)
+			and is NVDIMM or ADR memory.
+
 	memory_corruption_check=0/1 [X86]
 			Some BIOSes seem to corrupt the first 64k of
 			memory when doing things like suspend/resume.
MAINTAINERS +6 -0
index 1de6afa8ee51..4517613dc638 100644
@@ -8071,6 +8071,12 @@ S:	Maintained
 F:	Documentation/blockdev/ramdisk.txt
 F:	drivers/block/brd.c
 
+PERSISTENT MEMORY DRIVER
+M:	Ross Zwisler <[email protected]>
+L:	[email protected]
+S:	Supported
+F:	drivers/block/pmem.c
+
 RANDOM NUMBER DRIVER
 M:	"Theodore Ts'o" <[email protected]>
 S:	Maintained
arch/x86/Kconfig +10 -0
index b7d31ca55187..9e3bcd6f4a48 100644
@@ -1430,6 +1430,16 @@ config ILLEGAL_POINTER_VALUE
 
 source "mm/Kconfig"
 
+config X86_PMEM_LEGACY
+	bool "Support non-standard NVDIMMs and ADR protected memory"
+	help
+	  Treat memory marked using the non-standard e820 type of 12 as used
+	  by the Intel Sandy Bridge-EP reference BIOS as protected memory.
+	  The kernel will offer these regions to the 'pmem' driver so
+	  they can be used for persistent storage.
+
+	  Say Y if unsure.
+
 config HIGHPTE
 	bool "Allocate 3rd-level pagetables from highmem"
 	depends on HIGHMEM
arch/x86/include/uapi/asm/e820.h +10 -0
index d993e33f5236..960a8a9dc4ab 100644
@@ -33,6 +33,16 @@
 #define E820_NVS	4
 #define E820_UNUSABLE	5
 
+/*
+ * This is a non-standardized way to represent ADR or NVDIMM regions that
+ * persist over a reboot.  The kernel will ignore their special capabilities
+ * unless the CONFIG_X86_PMEM_LEGACY=y option is set.
+ *
+ * ( Note that older platforms also used 6 for the same type of memory,
+ *   but newer versions switched to 12 as 6 was assigned differently.  Some
+ *   time they will learn... )
+ */
+#define E820_PRAM	12
 
 /*
  * reserved RAM used by kernel itself
arch/x86/kernel/Makefile +1 -0
index cdb1b70ddad0..971f18cd9ca0 100644
@@ -94,6 +94,7 @@ obj-$(CONFIG_KVM_GUEST)		+= kvm.o kvmclock.o
 obj-$(CONFIG_PARAVIRT)		+= paravirt.o paravirt_patch_$(BITS).o
 obj-$(CONFIG_PARAVIRT_SPINLOCKS)+= paravirt-spinlocks.o
 obj-$(CONFIG_PARAVIRT_CLOCK)	+= pvclock.o
+obj-$(CONFIG_X86_PMEM_LEGACY)	+= pmem.o
 
 obj-$(CONFIG_PCSPKR_PLATFORM)	+= pcspeaker.o
 
arch/x86/kernel/e820.c +20 -6
index 46201deee923..11cc7d54ec3f 100644
@@ -149,6 +149,9 @@ static void __init e820_print_type(u32 type)
 	case E820_UNUSABLE:
 		printk(KERN_CONT "unusable");
 		break;
+	case E820_PRAM:
+		printk(KERN_CONT "persistent (type %u)", type);
+		break;
 	default:
 		printk(KERN_CONT "type %u", type);
 		break;
@@ -343,7 +346,7 @@ int __init sanitize_e820_map(struct e820entry *biosmap, int max_nr_map,
 		 * continue building up new bios map based on this
 		 * information
 		 */
-		if (current_type != last_type)	{
+		if (current_type != last_type || current_type == E820_PRAM) {
 			if (last_type != 0)	 {
 				new_bios[new_bios_entry].size =
 					change_point[chgidx]->addr - last_addr;
@@ -688,6 +691,7 @@ void __init e820_mark_nosave_regions(unsigned long limit_pfn)
 			register_nosave_region(pfn, PFN_UP(ei->addr));
 
 		pfn = PFN_DOWN(ei->addr + ei->size);
+
 		if (ei->type != E820_RAM && ei->type != E820_RESERVED_KERN)
 			register_nosave_region(PFN_UP(ei->addr), pfn);
 
@@ -748,7 +752,7 @@ u64 __init early_reserve_e820(u64 size, u64 align)
 /*
  * Find the highest page frame number we have available
  */
-static unsigned long __init e820_end_pfn(unsigned long limit_pfn, unsigned type)
+static unsigned long __init e820_end_pfn(unsigned long limit_pfn)
 {
 	int i;
 	unsigned long last_pfn = 0;
@@ -759,7 +763,11 @@ static unsigned long __init e820_end_pfn(unsigned long limit_pfn, unsigned type)
 		unsigned long start_pfn;
 		unsigned long end_pfn;
 
-		if (ei->type != type)
+		/*
+		 * Persistent memory is accounted as ram for purposes of
+		 * establishing max_pfn and mem_map.
+		 */
+		if (ei->type != E820_RAM && ei->type != E820_PRAM)
 			continue;
 
 		start_pfn = ei->addr >> PAGE_SHIFT;
@@ -784,12 +792,12 @@ static unsigned long __init e820_end_pfn(unsigned long limit_pfn, unsigned type)
 }
 unsigned long __init e820_end_of_ram_pfn(void)
 {
-	return e820_end_pfn(MAX_ARCH_PFN, E820_RAM);
+	return e820_end_pfn(MAX_ARCH_PFN);
 }
 
 unsigned long __init e820_end_of_low_ram_pfn(void)
 {
-	return e820_end_pfn(1UL<<(32 - PAGE_SHIFT), E820_RAM);
+	return e820_end_pfn(1UL << (32-PAGE_SHIFT));
 }
 
 static void early_panic(char *msg)
@@ -866,6 +874,9 @@ static int __init parse_memmap_one(char *p)
 	} else if (*p == '$') {
 		start_at = memparse(p+1, &p);
 		e820_add_region(start_at, mem_size, E820_RESERVED);
+	} else if (*p == '!') {
+		start_at = memparse(p+1, &p);
+		e820_add_region(start_at, mem_size, E820_PRAM);
 	} else
 		e820_remove_range(mem_size, ULLONG_MAX - mem_size, E820_RAM, 1);
 
@@ -907,6 +918,7 @@ static inline const char *e820_type_to_string(int e820_type)
 	case E820_ACPI:	return "ACPI Tables";
 	case E820_NVS:	return "ACPI Non-volatile Storage";
 	case E820_UNUSABLE:	return "Unusable memory";
+	case E820_PRAM: return "Persistent RAM";
 	default:	return "reserved";
 	}
 }
@@ -940,7 +952,9 @@ void __init e820_reserve_resources(void)
 		 * pci device BAR resource and insert them later in
 		 * pcibios_resource_survey()
 		 */
-		if (e820.map[i].type != E820_RESERVED || res->start < (1ULL<<20)) {
+		if (((e820.map[i].type != E820_RESERVED) &&
+		     (e820.map[i].type != E820_PRAM)) ||
+		     res->start < (1ULL<<20)) {
 			res->flags |= IORESOURCE_BUSY;
 			insert_resource(&iomem_resource, res);
 		}
arch/x86/kernel/pmem.c new file +53 -0

53 lines not on this page · raw message

drivers/block/Kconfig +11 -0
index 1b8094d4d7af..eb1fed5bd516 100644
@@ -404,6 +404,17 @@ config BLK_DEV_RAM_DAX
 	  and will prevent RAM block device backing store memory from being
 	  allocated from highmem (only a problem for highmem systems).
 
+config BLK_DEV_PMEM
+	tristate "Persistent memory block device support"
+	help
+	  Saying Y here will allow you to use a contiguous range of reserved
+	  memory as one or more persistent block devices.
+
+	  To compile this driver as a module, choose M here: the module will be
+	  called 'pmem'.
+
+	  If unsure, say N.
+
 config CDROM_PKTCDVD
 	tristate "Packet writing on CD/DVD media"
 	depends on !UML
drivers/block/Makefile +1 -0
index 02b688d1438d..9cc6c18a1c7e 100644
@@ -14,6 +14,7 @@ obj-$(CONFIG_PS3_VRAM)		+= ps3vram.o
 obj-$(CONFIG_ATARI_FLOPPY)	+= ataflop.o
 obj-$(CONFIG_AMIGA_Z2RAM)	+= z2ram.o
 obj-$(CONFIG_BLK_DEV_RAM)	+= brd.o
+obj-$(CONFIG_BLK_DEV_PMEM)	+= pmem.o
 obj-$(CONFIG_BLK_DEV_LOOP)	+= loop.o
 obj-$(CONFIG_BLK_CPQ_DA)	+= cpqarray.o
 obj-$(CONFIG_BLK_CPQ_CISS_DA)  += cciss.o
drivers/block/pmem.c new file +262 -0

262 lines not on this page · raw message

#1 lore raw
Quoted 5 lines · Ingo Molnar

Limitations: this is a regular block device, and since the pmem areas are not struct page backed, they are invisible to the rest of the system (other than the block IO device), so direct IO to/from pmem areas, direct mmap() or XIP is not possible yet. The page cache will also shadow and double buffer pmem contents, etc.

Unless you use the DAX support in ext2/4 and soon XFS, in which case we avoid that double buffering when doing read/write and mmap

#2 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 5 lines · Ingo Molnar

Limitations: this is a regular block device, and since the pmem areas are not struct page backed, they are invisible to the rest of the system (other than the block IO device), so direct IO to/from pmem areas, direct mmap() or XIP is not possible yet. The page cache will also shadow and double buffer pmem contents, etc.

Unless you use the DAX support in ext2/4 and soon XFS, in which case we avoid that double buffering when doing read/write and mmap

Indeed, I missed that DAX support just went upstream in v4.0 - nice!

DAX may have some other limitations though that comes from not having struct page * backing and using VM_MIXEDMAP, the following APIs might not work on DAX files:

- splice
- zero copy O_DIRECT into DAX areas.
- futexes

- ( AFAICS hugetlbs won't work on DAX mmap()s yet - although with the current nocache mapping that's probable the least of the performance issues for now. )

Btw., what's the future design plan here? Enable struct page backing, or provide special codepaths for all DAX uses like the special pte based approach for mmap()s?

Thanks,

Ingo

#3 lore raw
Quoted 19 lines · Ingo Molnar
Quoted 9 lines · Christoph Hellwig
Quoted 5 lines · Ingo Molnar

Limitations: this is a regular block device, and since the pmem areas are not struct page backed, they are invisible to the rest of the system (other than the block IO device), so direct IO to/from pmem areas, direct mmap() or XIP is not possible yet. The page cache will also shadow and double buffer pmem contents, etc.

Unless you use the DAX support in ext2/4 and soon XFS, in which case we avoid that double buffering when doing read/write and mmap

Indeed, I missed that DAX support just went upstream in v4.0 - nice!

DAX may have some other limitations though that comes from not having struct page * backing and using VM_MIXEDMAP, the following APIs might not work on DAX files:

Quoted 7 lines

- splice
- zero copy O_DIRECT into DAX areas.
- futexes

- ( AFAICS hugetlbs won't work on DAX mmap()s yet - although with the current nocache mapping that's probable the least of the performance issues for now. )

mlock() and MAP_POPULATE don't work with DAX files as well.

Quoted 12 lines

Btw., what's the future design plan here? Enable struct page backing, or provide special codepaths for all DAX uses like the special pte based approach for mmap()s?

Thanks,

Ingo
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#6 lore raw
In reply to Ingo Molnar
Quoted 13 lines · Ingo Molnar
Quoted 9 lines · Christoph Hellwig
Quoted 5 lines · Ingo Molnar

Limitations: this is a regular block device, and since the pmem areas are not struct page backed, they are invisible to the rest of the system (other than the block IO device), so direct IO to/from pmem areas, direct mmap() or XIP is not possible yet. The page cache will also shadow and double buffer pmem contents, etc.

Unless you use the DAX support in ext2/4 and soon XFS, in which case we avoid that double buffering when doing read/write and mmap

Quoted 7 lines

Indeed, I missed that DAX support just went upstream in v4.0 - nice!

DAX may have some other limitations though that comes from not having struct page * backing and using VM_MIXEDMAP, the following APIs might not work on DAX files:

- splice

splice works fine. Also I sent a cleanup in this area to Andrew it will be in for 4.1

Quoted 1 line

- zero copy O_DIRECT into DAX areas.

DAX is always O_DIRECT.

What does not work is mmap of DAX file and use that pointer in an O_DIRECT operation of another device. (unless it is a DAX device)

Also mmap of DAX file and RDMA or direct-networking. Will need a copy.

All this is fixable by applying my page-struct patch for pmem

Quoted 10 lines

- futexes

- ( AFAICS hugetlbs won't work on DAX mmap()s yet - although with the current nocache mapping that's probable the least of the performance issues for now. )

Btw., what's the future design plan here? Enable struct page backing, or provide special codepaths for all DAX uses like the special pte based approach for mmap()s?

I'm hopping for struct page, 4k pages at first and 2M pages later on, which needs more work in IO stacks, where I need this most.

Quoted 3 lines

Thanks,
Ingo

Thanks
Boaz

#7 lore raw
Quoted 16 lines · Boaz Harrosh
Quoted 15 lines · Ingo Molnar
Quoted 9 lines · Christoph Hellwig
Quoted 5 lines · Ingo Molnar

Limitations: this is a regular block device, and since the pmem areas are not struct page backed, they are invisible to the rest of the system (other than the block IO device), so direct IO to/from pmem areas, direct mmap() or XIP is not possible yet. The page cache will also shadow and double buffer pmem contents, etc.

Unless you use the DAX support in ext2/4 and soon XFS, in which case we avoid that double buffering when doing read/write and mmap

Indeed, I missed that DAX support just went upstream in v4.0 - nice!

Quoted 8 lines
Quoted 5 lines

DAX may have some other limitations though that comes from not having struct page * backing and using VM_MIXEDMAP, the following APIs might not work on DAX files:

- splice

splice works fine. Also I sent a cleanup in this area to Andrew it will be in for 4.1

How does splice work with DAX files? AFAICS vmsplice() won't work, as it uses get_user_pages(), which needs struct page backing. Also, how will f_op->sendpage() work? That too needs page backing.

Quoted 6 lines
Quoted 1 line

- zero copy O_DIRECT into DAX areas.

DAX is always O_DIRECT.

What does not work is mmap of DAX file and use that pointer in an O_DIRECT operation of another device. (unless it is a DAX device)

That is what I meant: O_DIRECT into (or from) DAX mmap()-ed areas, from a different device.

I'm not promoting the use of these APIs, some of them are quirky, just wanted to list the known limitations. The pmem driver is already useful as-is.

Thanks,

Ingo

#8 lore raw
Quoted 1 line · Ingo Molnar

<>

Quoted 5 lines

How does splice work with DAX files? AFAICS vmsplice() won't work, as it uses get_user_pages(), which needs struct page backing. Also, how will f_op->sendpage() work? That too needs page backing.

I'm not sure about f_op->sendpage(). I do not see it in ext4 so I assumed it works through f_op->splice_read/write.

The way f_op->splice_XX works is through the read/write_iter system and this one is fully supported with DAX. It will do an extra copy.

[You can see this patch in linux-next:

	8b22559 dax: unify ext2/4_{dax,}_file_operations

that explains a bit about this issue
]

Quoted 10 lines
Quoted 6 lines
Quoted 1 line

- zero copy O_DIRECT into DAX areas.

DAX is always O_DIRECT.

What does not work is mmap of DAX file and use that pointer in an O_DIRECT operation of another device. (unless it is a DAX device)

That is what I meant: O_DIRECT into (or from) DAX mmap()-ed areas, from a different device.

Yes. For me the one missing the most is RDMA to/from pmem copy-less.

Is why I'm pushing for my page-struct patches. But so far people rejected it. I hope to resend it soon, once the dust settles and see if people might change their mind.

Quoted 7 lines

I'm not promoting the use of these APIs, some of them are quirky, just wanted to list the known limitations. The pmem driver is already useful as-is.

Thanks,
Ingo

Thanks
Boaz

#10 lore raw
In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

Btw., what's the future design plan here? Enable struct page backing, or provide special codepaths for all DAX uses like the special pte based approach for mmap()s?

There are a couple approaches proposed, but we don't have consensus which way to go yet (to put it mildly).

- the old Intel patches just allocate pages for E820_PMEM regions. I think this is a good way forward for the "old-school" small pmem regions which usually are battery/capacitor + flash backed DRAM anyway. This could easily be resurrected for the current code, but it couldn't be used for PCI backed pmem regions, and would work although waste a lot of resources for the gigantic pmem devices some Intel people talk about (400TB+ IIRC).

- Intel has proposed changes that allow block I/O on regions that aren't page backed, by supporting PFN-based scatterlists which would have to be supported all over the I/O path. Reception of that code has been rather mediocre in general, although I wouldn't rule it out.

- Boaz has shown code that creates pages dynamically for pmem regions. Unlike the old Intel e820 code that would also work for PCI backed pmem regions. Boaz says he has such a card, but until someone actually publishes specs and/or the trivial pci_driver for them I'm inclined to just ignore that option.

- There have been proposals for temporary struct page mappings, or variable sized pages, but as far as I can tell no code to actually implement these schemes.

#9 lore raw
In reply to Ingo Molnar
Quoted 1 line · Ingo Molnar

How does splice work with DAX files?

By falling back to default_file_splice_read/default_file_splice_write which doesn't use the iter ops, but instead requires a copy in the splice code. But given that the actual underlying reads and writes bypass the pagecache it's not any less effiecient than the normal pagecache based splice.

Quoted 2 lines

AFAICS vmsplice() won't work, as
it uses get_user_pages(), which needs struct page backing.

Exactly.

Quoted 2 lines

Also, how
will f_op->sendpage() work? That too needs page backing.

default_file_splice_read allocates it's own kernel pages, which are then passed to ->sendpage.

#5 lore raw
In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig
Quoted 1 line · Yigal Korman

mlock()

DAX files always are in-memory so this just sounds like an oversight. method.

Yes mlock on DAX can just return true, but mlock implies MAP_POPULATE.

Which means "I would like to page-fault the all mmap range at mmap time so at access time I'm guarantied not to sleep". This is usually done for latency sensitive applications.

But current code fails on MAP_POPULATE for DAX because it is only implemented for pages, and therefor mlock fails as well.

One thing I do not understand. does mlock also protects against truncate?

Thanks
Boaz

#11 lore raw
In reply to Christoph Hellwig
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Ingo Molnar

Btw., what's the future design plan here? Enable struct page backing, or provide special codepaths for all DAX uses like the special pte based approach for mmap()s?

There are a couple approaches proposed, but we don't have consensus which way to go yet (to put it mildly).

Quoted 7 lines

- the old Intel patches just allocate pages for E820_PMEM regions. I think this is a good way forward for the "old-school" small pmem regions which usually are battery/capacitor + flash backed DRAM anyway. This could easily be resurrected for the current code, but it couldn't be used for PCI backed pmem regions, and would work although waste a lot of resources for the gigantic pmem devices some Intel people talk about (400TB+ IIRC).

So here's how I calculate the various scenarios:

There are two main usecases visible currently for pmem devices: 'pmem as storage' and 'pmem as memory', and they have rather distinct characteristics.

1) pmem devices as 'storage':

So the size of 'struct page' is 64 bytes currently.

So even if a pmem device is just 1 TB (small storage), for example to replace storage on a system, we'd have to allocate 64 bytes per 4096 bytes of storage, which would use up 16 GB of main memory just for the page struct arrays...

So in this case 'struct page' allocation is not acceptable even for relatively small pmem device sizes.

For this usecase I think the current pmem driver is perfectly acceptable and in fact ideal:

- it double buffers, giving higher performance and also protecting storage from wear (that is likely flash based)

- the double buffering makes most struct page based APIs work just fine.

- it offers the DAX APIs for those weird special cases that really want to do their own cache and memory management (databases and other crazy usecases)

2) pmem devices as 'memory':

Battery backed and similar solutions of nv-dram, these are probably a lot smaller (for cost reasons) and are also a lot more RAM-alike, so the 'struct page' allocation in main RAM makes sense and possibly people would want to avoid the double buffering as well.

Furthermore, in this case we could also do another trick:

Quoted 11 lines

- Intel has proposed changes that allow block I/O on regions that aren't page backed, by supporting PFN-based scatterlists which would have to be supported all over the I/O path. Reception of that code has been rather mediocre in general, although I wouldn't rule it out.

- Boaz has shown code that creates pages dynamically for pmem regions. Unlike the old Intel e820 code that would also work for PCI backed pmem regions. Boaz says he has such a card, but until someone actually publishes specs and/or the trivial pci_driver for them I'm inclined to just ignore that option.

Quoted 3 lines

- There have been proposals for temporary struct page mappings, or variable sized pages, but as far as I can tell no code to actually implement these schemes.

None of this gives me warm fuzzy feelings...

... has anyone explored the possibility of putting 'struct page' into the pmem device itself, essentially using it as metadata?

Since it's directly mapped it should just work for most things if it's at least write-through cached (UC would be a horror), and it would also solve all the size problems. With write-through caching it should also be pretty OK performance-wise. The 64 bytes size is ideal as well.

( This would create a bit of a dependency with the kernel version, and
would complicate things of how to acquire a fresh page array after bootup, but that could be solved relatively easily IMHO. )

This would eliminate all the negative effects dynamic allocation of page structs or sg-lists brings.

Anyway:

Since both the 'pmem as storage' and 'pmem as memory' usecases will likely be utilized in practice, IMHO we should hedge our bets by supporting both equally well: we should start with the simpler one (the current driver) and then support both in the end, with as much end user flexibility as possible.

Thanks,

Ingo

#12 lore raw
Quoted 70 lines · Ingo Molnar
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines · Ingo Molnar

Btw., what's the future design plan here? Enable struct page backing, or provide special codepaths for all DAX uses like the special pte based approach for mmap()s?

There are a couple approaches proposed, but we don't have consensus which way to go yet (to put it mildly).

Quoted 7 lines

- the old Intel patches just allocate pages for E820_PMEM regions. I think this is a good way forward for the "old-school" small pmem regions which usually are battery/capacitor + flash backed DRAM anyway. This could easily be resurrected for the current code, but it couldn't be used for PCI backed pmem regions, and would work although waste a lot of resources for the gigantic pmem devices some Intel people talk about (400TB+ IIRC).

So here's how I calculate the various scenarios:

There are two main usecases visible currently for pmem devices: 'pmem as storage' and 'pmem as memory', and they have rather distinct characteristics.

1) pmem devices as 'storage':

So the size of 'struct page' is 64 bytes currently.

So even if a pmem device is just 1 TB (small storage), for example to replace storage on a system, we'd have to allocate 64 bytes per 4096 bytes of storage, which would use up 16 GB of main memory just for the page struct arrays...

So in this case 'struct page' allocation is not acceptable even for relatively small pmem device sizes.

For this usecase I think the current pmem driver is perfectly acceptable and in fact ideal:

- it double buffers, giving higher performance and also protecting storage from wear (that is likely flash based)

- the double buffering makes most struct page based APIs work just fine.

- it offers the DAX APIs for those weird special cases that really want to do their own cache and memory management (databases and other crazy usecases)

2) pmem devices as 'memory':

Battery backed and similar solutions of nv-dram, these are probably a lot smaller (for cost reasons) and are also a lot more RAM-alike, so the 'struct page' allocation in main RAM makes sense and possibly people would want to avoid the double buffering as well.

Furthermore, in this case we could also do another trick:

Quoted 11 lines

- Intel has proposed changes that allow block I/O on regions that aren't page backed, by supporting PFN-based scatterlists which would have to be supported all over the I/O path. Reception of that code has been rather mediocre in general, although I wouldn't rule it out.

- Boaz has shown code that creates pages dynamically for pmem regions. Unlike the old Intel e820 code that would also work for PCI backed pmem regions. Boaz says he has such a card, but until someone actually publishes specs and/or the trivial pci_driver for them I'm inclined to just ignore that option.

Quoted 9 lines
Quoted 3 lines

- There have been proposals for temporary struct page mappings, or variable sized pages, but as far as I can tell no code to actually implement these schemes.

None of this gives me warm fuzzy feelings...

... has anyone explored the possibility of putting 'struct page' into the pmem device itself, essentially using it as metadata?

Is what I've been saying from the first time it was asked

Yes this works today.
With my patchset it can be done.

The way it works is through memory_hotplug. You can hotplug in a pmem range as additional memory. This adds to the total amount of memory in the system.

you than load the pmem device with page-struct support that will allocate from the regular memory pools, but which are now bigger.

What I thought is that one "partition" on the device can be used as RAM, and its contents is re-initialized on boot. (I'm saying "partition" because it is more a pre-defined range not
a real block-layer partition)

That said, it should be very easy to build on my patchset and instruct add_section to use a pre-allocated area for the page-structs. (I will add an RFC patch that will do this)

Quoted 13 lines

Since it's directly mapped it should just work for most things if it's at least write-through cached (UC would be a horror), and it would also solve all the size problems. With write-through caching it should also be pretty OK performance-wise. The 64 bytes size is ideal as well.

( This would create a bit of a dependency with the kernel version, and
would complicate things of how to acquire a fresh page array after bootup, but that could be solved relatively easily IMHO. )

This would eliminate all the negative effects dynamic allocation of page structs or sg-lists brings.

Quoted 8 lines

Anyway:

Since both the 'pmem as storage' and 'pmem as memory' usecases will likely be utilized in practice, IMHO we should hedge our bets by supporting both equally well: we should start with the simpler one (the current driver) and then support both in the end, with as much end user flexibility as possible.

I have a pending patch to let user specify the mapping used per pmem device, (And default set at Kconfig). Including a per device use of pages. pages just being one of the mapping modes.

So all this can just be under the same driver and each admin can configure for his own needs.

I will send a new patchset once the merge window is over. And we can then see how it all looks.

Quoted 3 lines

Thanks,
Ingo

Thanks Ingo very much
Boaz

#13 lore raw
In reply to Ingo Molnar
Quoted 10 lines

-----Original Message-----
From: Linux-nvdimm [mailto:[email protected]] On Behalf Of Ingo Molnar
Sent: Tuesday, April 14, 2015 7:42 AM
To: Christoph Hellwig
Subject: Re: [Linux-nvdimm] [GIT PULL] PMEM driver for v4.1

...

Quoted 4 lines

Since it's directly mapped it should just work for most things if it's at least write-through cached (UC would be a horror), and it would also solve all the size problems. With write-through caching it should also be pretty OK performance-wise. The 64 bytes size is ideal as

Are the WT support patches going to make it into 4.1?

#17 lore raw
In reply to Ingo Molnar
Quoted 20 lines · Ingo Molnar

2) pmem devices as 'memory':

Battery backed and similar solutions of nv-dram, these are probably a lot smaller (for cost reasons) and are also a lot more RAM-alike, so the 'struct page' allocation in main RAM makes sense and possibly people would want to avoid the double buffering as well.

Furthermore, in this case we could also do another trick:

Quoted 11 lines

- Intel has proposed changes that allow block I/O on regions that aren't page backed, by supporting PFN-based scatterlists which would have to be supported all over the I/O path. Reception of that code has been rather mediocre in general, although I wouldn't rule it out.

- Boaz has shown code that creates pages dynamically for pmem regions. Unlike the old Intel e820 code that would also work for PCI backed pmem regions. Boaz says he has such a card, but until someone actually publishes specs and/or the trivial pci_driver for them I'm inclined to just ignore that option.

Quoted 8 lines
Quoted 3 lines

- There have been proposals for temporary struct page mappings, or variable sized pages, but as far as I can tell no code to actually implement these schemes.

None of this gives me warm fuzzy feelings...

... has anyone explored the possibility of putting 'struct page' into the pmem device itself, essentially using it as metadata?

Yes, the impetus for proposing the pfn conversion of the block layer was the consideration that persistent memory may have less write endurance than DRAM. The kernel preserving write endurance exclusively for user data and the elimination of struct page overhead motivated the patchset [1].

[1]: https://lwn.net/Articles/636968/

#14 lore raw
In reply to Elliott, Robert (Server Storage)
Quoted 10 lines · Robert (Server Storage)
Quoted 10 lines

-----Original Message-----
From: Linux-nvdimm [mailto:[email protected]] On Behalf Of Ingo Molnar
Sent: Tuesday, April 14, 2015 7:42 AM
To: Christoph Hellwig
Subject: Re: [Linux-nvdimm] [GIT PULL] PMEM driver for v4.1

Quoted 7 lines

...

Quoted 4 lines

Since it's directly mapped it should just work for most things if it's at least write-through cached (UC would be a horror), and it would also solve all the size problems. With write-through caching it should also be pretty OK performance-wise. The 64 bytes size is ideal as

Are the WT support patches going to make it into 4.1?

Which patches are these? Maybe I missed them, but I don't see anything in the archives.

#15 lore raw
Quoted 11 lines

-----Original Message-----
From: Dan Williams [mailto:[email protected]]
Sent: Tuesday, April 14, 2015 11:34 AM
To: Elliott, Robert (Server Storage)

Quoted 6 lines

...

Quoted 4 lines

Since it's directly mapped it should just work for most things if it's at least write-through cached (UC would be a horror), and it would also solve all the size problems. With write-through caching it should also be pretty OK performance-wise. The 64 bytes size is ideal as

Quoted 4 lines
Quoted 1 line

Are the WT support patches going to make it into 4.1?

Which patches are these? Maybe I missed them, but I don't see anything in the archives.

These have been baking in linux-mm and linux-next:
* [PATCH v3 0/6] Kernel huge I/O mapping support
https://lkml.org/lkml/2015/3/3/589
* [PATCH v4 0/7] mtrr, mm, x86: Enhance MTRR checks for huge I/O mapping
https://lkml.org/lkml/2015/3/24/1056

I don't think this made it into a subsystem tree yet:
* [PATCH v8 0/7] Support Write-Through mapping on x86
https://lkml.org/lkml/2015/2/24/773

I guess we could target 4.2 for both the WT series and pmem patches that support the new ioremap_wt() function.

---
Robert Elliott, HP Server Storage

#16 lore raw
In reply to Elliott, Robert (Server Storage)
Quoted 8 lines · Elliott, Robert (Server Storage)
Quoted 8 lines

-----Original Message-----
From: Linux-nvdimm [mailto:[email protected]] On Behalf Of Ingo Molnar
Sent: Tuesday, April 14, 2015 7:42 AM
To: Christoph Hellwig
Subject: Re: [Linux-nvdimm] [GIT PULL] PMEM driver for v4.1

Quoted 7 lines

...

Quoted 4 lines

Since it's directly mapped it should just work for most things if it's at least write-through cached (UC would be a horror), and it would also solve all the size problems. With write-through caching it should also be pretty OK performance-wise. The 64 bytes size is ideal as

Are the WT support patches going to make it into 4.1?

Not that I know of, they are still under review.

Thanks,

Ingo

#18 lore raw
In reply to Dan Williams
Quoted 5 lines · Dan Williams
Quoted 4 lines

None of this gives me warm fuzzy feelings...

... has anyone explored the possibility of putting 'struct page' into the pmem device itself, essentially using it as metadata?

Quoted 7 lines

Yes, the impetus for proposing the pfn conversion of the block layer was the consideration that persistent memory may have less write endurance than DRAM. The kernel preserving write endurance exclusively for user data and the elimination of struct page overhead motivated the patchset [1].

[1]: https://lwn.net/Articles/636968/

(Is there a Git URL where I could take a look at these patches?)

But, I think the usage of pfn's in the block layer is relatively independent of the question whether a pmem region should be permanently struct page backed or not.

I think the main confusion comes from the fact that 'pfn' can have two roles with sufficiently advanced MMIO interfaces: describing main RAM page (struct page), but also describing essentially sectors on a large, MMIO-accessible storage device, directly visible to the CPU but otherwise not RAM.

So for that reason I think pmem devices should be both struct page backed and not struct page backed, depending on their physical characteristics:

------------

1)

If a pmem device is in any way expected to be write-unreliable (i.e. it's not DRAM but flash) then it's going to be potentially large and we simply cannot use struct page backing for it, full stop.

Users very likely want a filesystem on it, with double buffering that both reduces wear and makes better use of main RAM and CPU caches.

In this case the pmem device is a simple storage device that has a refreshlingly clean hardware ABI that exposes all of its contents in a large, directly mapped MMIO region in essence.

We don't back mass storage with struct page, we never did with any of the other storage devices either.

I'd expect this to be the 90% dominant 'pmem usecase' in the future.

In this case any 'direct mapping' system calls, DIO or non-double-buffering mmaps() and DAX on the other hand will stay a 'weird' secondary usecases for user-space operating systems like databases that want to take caching out of the hands of the kernel.

The majority of users will use it as storage, with a filesystem on it and regular RAM caching it for everyone's gain. All the struct page based APIs and system calls will work just fine, and the rare usecases will be served by DAX.

------------

2)

But if a pmem device is RAM, with no write unreliability, then we obviously want it to have struct page backing, and we probably want to think about it more in terms of hot-pluggable memory, than a storage device.

This scenario will be less common than the mass-storage scenario.

Note that this is similar to how GPU memory is categorized: it's essentially RAM-alike, which naturally results in struct page backing.

------------

Note that scenarios 1) and 2) are not under our control, they are essentially a physical property, with some user policy influencing it as well. So we have to support both and we have no 'opinion' about which one is right, as it's simply physical reality as-is.

In that sense I think this driver does the right thing as a first
step: it exposes pmem regions in the more conservative fashion, as a block storage device, assuming write unreliability.

Patches that would turn the pmem driver into unconditionally struct page backed would be misguided for this usecase. Allocating and freeing struct page arrays on the fly would be similarly misguided.

But patches that allow pmem regions that declare themselves true RAM to be inserted as hotplug memory would be the right approach IMHO - while still preserving the pmem block device and the non-struct-page backed approach for other pmem devices.

Note how in this picture the question of how IO scatter-gather lists are constructed is an implementational detail that does not impact the main design: they are essentially DMA abstractions for storage devices, implemented efficiently via memcpy() in the pmem case, and both pfn lists and struct page lists are pretty equivalent approaches for most usages.

The only exception are the 'weird' usecases like DAX, DIO and RDMA:
these have to be pfn driven, due to the lack of struct page descriptors for storage devices in general. In that case the 'pfn' isn't really memory, but a sector_t equivalent, for this new type of storage DMA that is implemented via a memcpy().

In that sense the special DAX page fault handler looks like a natural approach as well: the pfn's in the page table aren't really describing memory pages, but 'sectors' on an IO device - with special rules, limited APIs and ongoing complications to be expected.

At least that's how I see it.

Thanks,

Ingo

#19 lore raw
Quoted 14 lines · Ingo Molnar
Quoted 11 lines · Dan Williams
Quoted 4 lines

None of this gives me warm fuzzy feelings...

... has anyone explored the possibility of putting 'struct page' into the pmem device itself, essentially using it as metadata?

Yes, the impetus for proposing the pfn conversion of the block layer was the consideration that persistent memory may have less write endurance than DRAM. The kernel preserving write endurance exclusively for user data and the elimination of struct page overhead motivated the patchset [1].

Quoted 3 lines
Quoted 1 line

[1]: https://lwn.net/Articles/636968/

(Is there a Git URL where I could take a look at these patches?)

git://git.kernel.org/pub/scm/linux/kernel/git/djbw/nvdimm
evacuate-struct-page-v1

Note that the bulk of the change is automated via Coccinelle.

For v2, I'm looking at enabling a kmap primitive to operate on a pfn_t, kmap_pfn().

#20 lore raw
In reply to Ingo Molnar

So we'be been busy discussing future steps in this thread for a few days, but it seems like Linus doesn't like the pull request. Linus, can you say what you don't like here?

#21 lore raw
Quoted 3 lines · Christoph Hellwig

So we'be been busy discussing future steps in this thread for a few days, but it seems like Linus doesn't like the pull request. Linus, can you say what you don't like here?

I basically don't do pulls that generate a lot of discussion before the discussion is over.

Of course, most of the time that discussion is because the pull request itself is contentious. This time it seems to be more about secondary issues related to the area rather than the pull request itself being contentious. I assume nobody is actually objecting to the pull itself, and will be going ahead with it.

Linus

#22 lore raw
In reply to Ingo Molnar
Quoted 9 lines · Ingo Molnar

Please pull the latest x86-pmem-for-linus git tree from:

   git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86-pmem-for-linus

   # HEAD: 4c1eaa2344fb26bb5e936fb4d8ee307343ea0089 drivers/block/pmem: Fix 32-bit build warning in pmem_alloc()

This is the initial support for the pmem block device driver:
persistent non-volatile memory space mapped into the system's physical memory space as large physical memory regions.

Ingo, this sucks. You collapsed all of the separate patches into a single "add new driver" patch, which makes it impossible to bisect which of the recent changes broke xfstests. Please don't do this again.

Quoted 9 lines

------------------>
Christoph Hellwig (1):
x86/mm: Add support for the non-standard protected e820 type

Ingo Molnar (1):
drivers/block/pmem: Fix 32-bit build warning in pmem_alloc()

Ross Zwisler (1):
drivers/block/pmem: Add a driver for persistent memory

#23 lore raw
Quoted 7 lines · Matthew Wilcox
Quoted 6 lines · Ingo Molnar

Please pull the latest x86-pmem-for-linus git tree from:

   git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86-pmem-for-linus

   # HEAD: 4c1eaa2344fb26bb5e936fb4d8ee307343ea0089 drivers/block/pmem: Fix 32-bit build warning in pmem_alloc()
Quoted 8 lines
Quoted 3 lines

This is the initial support for the pmem block device driver:
persistent non-volatile memory space mapped into the system's physical memory space as large physical memory regions.

Ingo, this sucks. You collapsed all of the separate patches into a single "add new driver" patch, which makes it impossible to bisect which of the recent changes broke xfstests. Please don't do this again.

I didn't do any collapsing.

Thanks,

Ingo

#24 lore raw
In reply to Matthew Wilcox

On 05/25/2015 09:16 PM, Matthew Wilcox wrote:
<>

Quoted 5 lines

Ingo, this sucks. You collapsed all of the separate patches into a single "add new driver" patch, which makes it impossible to bisect which of the recent changes broke xfstests. Please don't do this again.

Matthew hi

Below is a splitout of the patches I sent to Christoph. Christoph in his turn has added more changes.

I would please like to help. What is the breakage you see with DAX.

I'm routinely testing with DAX so it is a surprise, Though I'm testing with my version with pages and __copy_from_user_nocache, and so on.
Or I might have missed it. What test are you failing?

Find patches at:
[web: http://git.open-osd.org/gitweb.cgi?p=pmem.git;a=shortlog;h=refs/tags/PMEM-SPLITOUT-4.0-rc5]
[git tag PMEM-SPLITOUT-4.0-rc5 on git://git.open-osd.org / pmem.git]

The interesting list for you:
bc465aa Linux 4.0-rc5
e489c02 x86: add support for the non-standard protected e820 type 8b06a64 SQUASHME: Don't let e820_PMEM section merge emulated segments.

There were a few changes to what got upstream on top of these 2

a069890 pmem: Initial version of persistent memory driver

This one already contains changes made by Christoph

d4f6df5 SQUASHME: pmem: Remove getgeo
2f3f1dd SQUASHME: pmem: Streamline pmem driver
c921d23 SQUSHME: pmem: Micro cleaning
bb373a7 SQUASHME: pmem: Remove SECTOR_SHIFT
ad0070e SQUASHME: pmem: Remove "heavily based on brd.c" + Copyright

This is not the final pmem version that got merged, but these are the changes I made which got incorporated.

Thanks
Boaz

#25 lore raw
Quoted 7 lines · Boaz Harrosh

I would please like to help. What is the breakage you see with DAX.

I'm routinely testing with DAX so it is a surprise, Though I'm testing with my version with pages and __copy_from_user_nocache, and so on.
Or I might have missed it. What test are you failing?

generic/019 fails in several fun ways.

The first way, which I fixed yesterday, is that the test was using the wrong way to find the 'make-it-fail' switch for the block device. That's now in xfstests. The messages from xfstests were unnecessarily worrying; they were complaining about an inconsistent filesystem, which might be expected as the test had failed to abort cleanly and left a couple of tasks actively writing to the filesystem.

(I hadn't seen the problem before because I was using two devices pmem0 and pmem1; with the new pmem driver, I got one device and partitioned it instead. The problem only occurs when using partitions, not when using entire devices).

The second way is that we hit two BUG/WARN messages. The first (which we hit simultaneously on three CPUs in this run!) is:

WARNING: CPU: 7 PID: 2922 at fs/buffer.c:1143 mark_buffer_dirty+0x19e/0x270()

The stack trace probably isn't useful, and anyway it's horribly corrupted due to triggering the stack trace simultaneously on three CPUs.

The second one we hit was this one:

 ------------[ cut here ]------------
 WARNING: CPU: 0 PID: 2930 at fs/block_dev.c:56 __blkdev_put+0xc5/0x210()
 Modules linked in: ext4 crc16 jbd2 pmem binfmt_misc nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc snd_hda_codec_hdmi iTCO_wdt iTCO_vendor_support evdev x86_pkg_temp_thermal coretemp kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse serio_raw pcspkr i2c_i801 snd_hda_codec_realtek snd_hda_codec_generic lpc_ich mfd_core mei_me mei i915 snd_hda_intel i2c_algo_bit snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_hda_core loop video drm_kms_helper fuse snd_timer snd drm soundcore button processor parport_pc ppdev lp parport sg sd_mod ehci_pci ehci_hcd ahci libahci crc32c_intel libata fan scsi_mod xhci_pci nvme xhci_hcd e1000e ptp pps_core usbcore usb_common thermal thermal_sys
 CPU: 0 PID: 2930 Comm: umount Tainted: G        W       4.1.0-rc4+ #10
 Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Q87M-D2H, BIOS F6 08/03/2013
  ffffffff81a04063 ffff8800a58e3d98 ffffffff81653644 0000000000000000
  0000000000000000 ffff8800a58e3dd8 ffffffff81081fea 0000000000000000
  ffff880236580880 ffff880236580ae8 ffff880236580a60 ffff880236580898
 Call Trace:
  [<ffffffff81653644>] dump_stack+0x4c/0x65
  [<ffffffff81081fea>] warn_slowpath_common+0x8a/0xc0
  [<ffffffff810820da>] warn_slowpath_null+0x1a/0x20
  [<ffffffff81260475>] __blkdev_put+0xc5/0x210
  [<ffffffff81260f72>] blkdev_put+0x52/0x180
  [<ffffffff8121e631>] kill_block_super+0x41/0x80
  [<ffffffff8121ea94>] deactivate_locked_super+0x44/0x80
  [<ffffffff8121ef0c>] deactivate_super+0x6c/0x80
  [<ffffffff81242133>] cleanup_mnt+0x43/0xa0
  [<ffffffff812421e2>] __cleanup_mnt+0x12/0x20
  [<ffffffff810a7104>] task_work_run+0xc4/0xf0
  [<ffffffff8101bdd9>] do_notify_resume+0x59/0x80
  [<ffffffff8165cd66>] int_signal+0x12/0x17
 ---[ end trace 73da47765ccceacf ]---

I suspect these are generic ext4 problems that will occur without DAX. DAX just makes them more likely to occur since only metadata I/O now goes through the 'likely to fail' path.

Are you skipping generic/019 or just not seeing these failures?

author Tested-by Ross Zwisler Acked-by Dan Williams
#29 lore raw
In reply to Matthew Wilcox
Quoted 7 lines · Matthew Wilcox
Quoted 6 lines · Ingo Molnar

Please pull the latest x86-pmem-for-linus git tree from:

   git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86-pmem-for-linus

   # HEAD: 4c1eaa2344fb26bb5e936fb4d8ee307343ea0089 drivers/block/pmem: Fix 32-bit build warning in pmem_alloc()
Quoted 7 lines
Quoted 3 lines

This is the initial support for the pmem block device driver:
persistent non-volatile memory space mapped into the system's physical memory space as large physical memory regions.

Ingo, this sucks. You collapsed all of the separate patches into a single "add new driver" patch, which makes it impossible to bisect which of the recent changes broke xfstests. Please don't do this again.

You ignored my previous reply, so let me ask this again more forcefully: what the hell are you talking about??

As you can see it from the fine commit 9e853f2313e5:

...

I didn't do any 'collapsing' of patches, I applied the patches as sent by hch:

9e853f2313e5 ("drivers/block/pmem: Add a driver for persistent memory") ec776ef6bbe1 ("x86/mm: Add support for the non-standard protected e820 type")

Thanks,

Ingo

#26 lore raw
In reply to Matthew Wilcox
Quoted 9 lines · Matthew Wilcox
Quoted 7 lines · Boaz Harrosh

I would please like to help. What is the breakage you see with DAX.

I'm routinely testing with DAX so it is a surprise, Though I'm testing with my version with pages and __copy_from_user_nocache, and so on.
Or I might have missed it. What test are you failing?

Quoted 9 lines

generic/019 fails in several fun ways.

The first way, which I fixed yesterday, is that the test was using the wrong way to find the 'make-it-fail' switch for the block device. That's now in xfstests. The messages from xfstests were unnecessarily worrying; they were complaining about an inconsistent filesystem, which might be expected as the test had failed to abort cleanly and left a couple of tasks actively writing to the filesystem.

OK Apparently I never ran generic/019 see below

Quoted 5 lines

(I hadn't seen the problem before because I was using two devices pmem0 and pmem1; with the new pmem driver, I got one device and partitioned it instead. The problem only occurs when using partitions, not when using entire devices).

My version of pmem+pages has back this option of slicing up pmem to arbitrary sized pieces, because each piece can load with one of EPMEM_UNCACHED, EPMEM_WRITE_TROUGH, EPMEM_WRITE_COMBINED, EPMEM_CACHED, EPMEM_PAGES memory mapping mode.
(I have almost all the proper code for EPMEM_CACHED / EPMEM_PAGES
support in surrounding code including m/fsync and mmap, memcpy_nt and so on, only left out is sync/freeze with regard to mmap)

But Yes I will tell the guys to add a partitions testing as well it is important that it will work.

So would you suspect that we should have the same problem with the original driver?

Quoted 4 lines

The second way is that we hit two BUG/WARN messages. The first (which we hit simultaneously on three CPUs in this run!) is:

WARNING: CPU: 7 PID: 2922 at fs/buffer.c:1143 mark_buffer_dirty+0x19e/0x270()

Hum, so that must be from some directory handling code, no? Regular DAX IO does not do write_begin. But why would it make any difference if its a partition or not, that's weird. Since this is a real page not a pmem buffer at all.

Quoted 29 lines

The stack trace probably isn't useful, and anyway it's horribly corrupted due to triggering the stack trace simultaneously on three CPUs.

The second one we hit was this one:

 ------------[ cut here ]------------
 WARNING: CPU: 0 PID: 2930 at fs/block_dev.c:56 __blkdev_put+0xc5/0x210()
 Modules linked in: ext4 crc16 jbd2 pmem binfmt_misc nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc snd_hda_codec_hdmi iTCO_wdt iTCO_vendor_support evdev x86_pkg_temp_thermal coretemp kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd psmouse serio_raw pcspkr i2c_i801 snd_hda_codec_realtek snd_hda_codec_generic lpc_ich mfd_core mei_me mei i915 snd_hda_intel i2c_algo_bit snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_hda_core loop video drm_kms_helper fuse snd_timer snd drm soundcore button processor parport_pc ppdev lp parport sg sd_mod ehci_pci ehci_hcd ahci libahci crc32c_intel libata fan scsi_mod xhci_pci nvme xhci_hcd e1000e ptp pps_core usbcore usb_common thermal thermal_sys
 CPU: 0 PID: 2930 Comm: umount Tainted: G        W       4.1.0-rc4+ #10
 Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Q87M-D2H, BIOS F6 08/03/2013
  ffffffff81a04063 ffff8800a58e3d98 ffffffff81653644 0000000000000000
  0000000000000000 ffff8800a58e3dd8 ffffffff81081fea 0000000000000000
  ffff880236580880 ffff880236580ae8 ffff880236580a60 ffff880236580898
 Call Trace:
  [<ffffffff81653644>] dump_stack+0x4c/0x65
  [<ffffffff81081fea>] warn_slowpath_common+0x8a/0xc0
  [<ffffffff810820da>] warn_slowpath_null+0x1a/0x20
  [<ffffffff81260475>] __blkdev_put+0xc5/0x210
  [<ffffffff81260f72>] blkdev_put+0x52/0x180
  [<ffffffff8121e631>] kill_block_super+0x41/0x80
  [<ffffffff8121ea94>] deactivate_locked_super+0x44/0x80
  [<ffffffff8121ef0c>] deactivate_super+0x6c/0x80
  [<ffffffff81242133>] cleanup_mnt+0x43/0xa0
  [<ffffffff812421e2>] __cleanup_mnt+0x12/0x20
  [<ffffffff810a7104>] task_work_run+0xc4/0xf0
  [<ffffffff8101bdd9>] do_notify_resume+0x59/0x80
  [<ffffffff8165cd66>] int_signal+0x12/0x17
 ---[ end trace 73da47765ccceacf ]---
Quoted 4 lines

I suspect these are generic ext4 problems that will occur without DAX. DAX just makes them more likely to occur since only metadata I/O now goes through the 'likely to fail' path.

But why would WARN_ON_ONCE(write_inode_now(inode, true)) fail? None of the pmem.c IO paths ever return any error. It must be failing in block core even before reaching pmem. It looks like we are stuck with dirty mappings after the devices are already being torn down, or something ...

Quoted 2 lines

Are you skipping generic/019 or just not seeing these failures?

Hu funny I just looked and I see with ./check auto I get generic/018 1s ... [not run] defragmentation not supported for fstype "m1fs" generic/020 0s ... 0s

019 is not even printing a skip. But if I run it directly I get:
generic/019 [not run] /sys/kernel/debug/fail_make_request not found. \
Seems that CONFIG_FAIL_MAKE_REQUEST kernel config option not enabled

So my bad, I will try to properly configure and recreate this failure here as well.

Thanks
Boaz

#27 lore raw
Quoted 10 lines · Boaz Harrosh

Hu funny I just looked and I see with ./check auto I get generic/018 1s ... [not run] defragmentation not supported for fstype "m1fs" generic/020 0s ... 0s

019 is not even printing a skip. But if I run it directly I get:
generic/019 [not run] /sys/kernel/debug/fail_make_request not found. \
Seems that CONFIG_FAIL_MAKE_REQUEST kernel config option not enabled

So my bad, I will try to properly configure and recreate this failure here as well.

It fails I/O above the driver. Any failure in generic/019 is very unlіkely to be driver related.

#28 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Boaz Harrosh

Hu funny I just looked and I see with ./check auto I get generic/018 1s ... [not run] defragmentation not supported for fstype "m1fs" generic/020 0s ... 0s

019 is not even printing a skip. But if I run it directly I get:
generic/019 [not run] /sys/kernel/debug/fail_make_request not found. \
Seems that CONFIG_FAIL_MAKE_REQUEST kernel config option not enabled

Quoted 6 lines
Quoted 2 lines

So my bad, I will try to properly configure and recreate this failure here as well.

It fails I/O above the driver. Any failure in generic/019 is very unlіkely to be driver related.

Hm, so then that would be expected right?
__blkdev_put fails to WARN_ON_ONCE(write_inode_now(inode, true)) because the test tells the Kernel to fail IO requests.

__blkdev_put then complains because it has no way to report the error back to caller.

I would then say that 019 test should expect that this might happen. Or it should somehow make sure to remove the requests block before the umount.

Thanks
Boaz