Skip to contentSkip to conversation

dax: fix NULL pointer in __dax_pmd_fault()

PATCH In review Ross Zwisler 5 participants 11 messages Sep 22, 2015 – Sep 24, 2015

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 dax: fix NULL pointer in __dax_pmd_fault()

The following commit:

commit 46c043ede471 ("mm: take i_mmap_lock in unmap_mapping_range() for

	DAX")

moved some code in __dax_pmd_fault() that was responsible for zeroing newly allocated PMD pages. The new location didn't properly set up 'kaddr', though, so when run this code resulted in a NULL pointer BUG.

Fix this by getting the correct 'kaddr' via bdev_direct_access().

---
 fs/dax.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
fs/dax.c +12 -1
index 7ae6df7..bcfb14b 100644
@@ -569,8 +569,20 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
 	if (!buffer_size_valid(&bh) || bh.b_size < PMD_SIZE)
 		goto fallback;
 
+	sector = bh.b_blocknr << (blkbits - 9);
+
 	if (buffer_unwritten(&bh) || buffer_new(&bh)) {
 		int i;
+
+		length = bdev_direct_access(bh.b_bdev, sector, &kaddr, &pfn,
+						bh.b_size);
+		if (length < 0) {
+			result = VM_FAULT_SIGBUS;
+			goto out;
+		}
+		if ((length < PMD_SIZE) || (pfn & PG_PMD_COLOUR))
+			goto fallback;
+
 		for (i = 0; i < PTRS_PER_PMD; i++)
 			clear_pmem(kaddr + i * PAGE_SIZE, PAGE_SIZE);
 		wmb_pmem();
@@ -623,7 +635,6 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
 		result = VM_FAULT_NOPAGE;
 		spin_unlock(ptl);
 	} else {
-		sector = bh.b_blocknr << (blkbits - 9);
 		length = bdev_direct_access(bh.b_bdev, sector, &kaddr, &pfn,
 						bh.b_size);
 		if (length < 0) {
Signature · 1 line
2.1.0
Reviewed-by on v2 1/1
#1 lore raw

[ adding Andrew ]

Quoted 5 lines · Ross Zwisler

The following commit:

commit 46c043ede471 ("mm: take i_mmap_lock in unmap_mapping_range() for

        DAX")
Quoted 8 lines

moved some code in __dax_pmd_fault() that was responsible for zeroing newly allocated PMD pages. The new location didn't properly set up 'kaddr', though, so when run this code resulted in a NULL pointer BUG.

Fix this by getting the correct 'kaddr' via bdev_direct_access().

Taking into account the comment below,

Quoted 16 lines
---
 fs/dax.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
fs/dax.c +2 -0
index 7ae6df7..bcfb14b 100644
@@ -569,8 +569,20 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
        if (!buffer_size_valid(&bh) || bh.b_size < PMD_SIZE)
                goto fallback;
 
+       sector = bh.b_blocknr << (blkbits - 9);
+
        if (buffer_unwritten(&bh) || buffer_new(&bh)) {
                int i;

fs/dax.c int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, · line 585

Quoted 10 lines
fs/dax.c +10 -0
+
+               length = bdev_direct_access(bh.b_bdev, sector, &kaddr, &pfn,
+                                               bh.b_size);
+               if (length < 0) {
+                       result = VM_FAULT_SIGBUS;
+                       goto out;
+               }
+               if ((length < PMD_SIZE) || (pfn & PG_PMD_COLOUR))
+                       goto fallback;
+

Hmm, we don't need the PG_PMD_COLOUR check since we aren't using the pfn in this path, right?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw
Quoted 43 lines · Dan Williams

[ adding Andrew ]

Quoted 5 lines · Ross Zwisler

The following commit:

commit 46c043ede471 ("mm: take i_mmap_lock in unmap_mapping_range() for

        DAX")
Quoted 8 lines

moved some code in __dax_pmd_fault() that was responsible for zeroing newly allocated PMD pages. The new location didn't properly set up 'kaddr', though, so when run this code resulted in a NULL pointer BUG.

Fix this by getting the correct 'kaddr' via bdev_direct_access().

Taking into account the comment below,

Quoted 21 lines
---
 fs/dax.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
fs/dax.c +7 -0
index 7ae6df7..bcfb14b 100644
@@ -569,8 +569,20 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
        if (!buffer_size_valid(&bh) || bh.b_size < PMD_SIZE)
                goto fallback;
 
+       sector = bh.b_blocknr << (blkbits - 9);
+
        if (buffer_unwritten(&bh) || buffer_new(&bh)) {
                int i;
+
+               length = bdev_direct_access(bh.b_bdev, sector, &kaddr, &pfn,
+                                               bh.b_size);
+               if (length < 0) {
+                       result = VM_FAULT_SIGBUS;
Quoted 8 lines

fs/dax.c int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, · line 585

Quoted 5 lines
fs/dax.c +5 -0
+                       goto out;
+               }
+               if ((length < PMD_SIZE) || (pfn & PG_PMD_COLOUR))
+                       goto fallback;
+

Hmm, we don't need the PG_PMD_COLOUR check since we aren't using the pfn in this path, right?

I think we care, because we'll end up bailing anyway at the later PG_PMD_COLOUR check before we actually insert the pfn via vmf_insert_pfn_pmd(). If we don't check the alignment we'll do 2 MiB worth of zeroing to the media, then later fall back to PTE faults.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw
Quoted 53 lines · Ross Zwisler
Quoted 43 lines · Dan Williams

[ adding Andrew ]

Quoted 5 lines · Ross Zwisler

The following commit:

commit 46c043ede471 ("mm: take i_mmap_lock in unmap_mapping_range() for

        DAX")
Quoted 8 lines

moved some code in __dax_pmd_fault() that was responsible for zeroing newly allocated PMD pages. The new location didn't properly set up 'kaddr', though, so when run this code resulted in a NULL pointer BUG.

Fix this by getting the correct 'kaddr' via bdev_direct_access().

Taking into account the comment below,

Quoted 21 lines
---
 fs/dax.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
fs/dax.c +7 -0
index 7ae6df7..bcfb14b 100644
@@ -569,8 +569,20 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
        if (!buffer_size_valid(&bh) || bh.b_size < PMD_SIZE)
                goto fallback;
 
+       sector = bh.b_blocknr << (blkbits - 9);
+
        if (buffer_unwritten(&bh) || buffer_new(&bh)) {
                int i;
+
+               length = bdev_direct_access(bh.b_bdev, sector, &kaddr, &pfn,
+                                               bh.b_size);
+               if (length < 0) {
+                       result = VM_FAULT_SIGBUS;
Quoted 8 lines

fs/dax.c int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, · line 585

Quoted 5 lines
fs/dax.c +5 -0
+                       goto out;
+               }
+               if ((length < PMD_SIZE) || (pfn & PG_PMD_COLOUR))
+                       goto fallback;
+

Hmm, we don't need the PG_PMD_COLOUR check since we aren't using the pfn in this path, right?

Quoted 4 lines

I think we care, because we'll end up bailing anyway at the later PG_PMD_COLOUR check before we actually insert the pfn via vmf_insert_pfn_pmd(). If we don't check the alignment we'll do 2 MiB worth of zeroing to the media, then later fall back to PTE faults.

Ok, good point.

maintainer
#4 lore raw
In reply to Ross Zwisler
Quoted 10 lines · On Tue, 22 Sep 2015 13:36:22 -0600 Ross Zwisler <[email protected]>…

The following commit:

commit 46c043ede471 ("mm: take i_mmap_lock in unmap_mapping_range() for

	DAX")

moved some code in __dax_pmd_fault() that was responsible for zeroing newly allocated PMD pages. The new location didn't properly set up 'kaddr', though, so when run this code resulted in a NULL pointer BUG.

Fix this by getting the correct 'kaddr' via bdev_direct_access().

Why the heck didn't gcc warn?

I had a fiddle:

fs/dax.c renamed from fs/dax.c~a +4 -1
@@ -529,15 +529,18 @@ int __dax_pmd_fault(struct vm_area_struc
 	unsigned long pmd_addr = address & PMD_MASK;
 	bool write = flags & FAULT_FLAG_WRITE;
 	long length;
-	void __pmem *kaddr;
+	void *kaddr;
 	pgoff_t size, pgoff;
 	sector_t block, sector;
 	unsigned long pfn;
 	int result = 0;
 
+//	printk("%p\n", kaddr);
+
 	/* Fall back to PTEs if we're going to COW */
 	if (write && !(vma->vm_flags & VM_SHARED))
 		return VM_FAULT_FALLBACK;
+	printk("%p\n", kaddr);
 	/* If the PMD would extend outside the VMA */
 	if (pmd_addr < vma->vm_start)
 		return VM_FAULT_FALLBACK;

gcc warns about the first printk, but not about the second. So that "if (...) return ..." seems to have defeated gcc uninitialized-var detection. wtf?

Quoted 13 lines
fs/dax.c +5 -0
@@ -569,8 +569,20 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
 	if (!buffer_size_valid(&bh) || bh.b_size < PMD_SIZE)
 		goto fallback;
 
+	sector = bh.b_blocknr << (blkbits - 9);
+
 	if (buffer_unwritten(&bh) || buffer_new(&bh)) {
 		int i;
+
+		length = bdev_direct_access(bh.b_bdev, sector, &kaddr, &pfn,
+						bh.b_size);

fs/dax.c int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, · line 588

Quoted 10 lines
fs/dax.c +7 -0
+		if (length < 0) {
+			result = VM_FAULT_SIGBUS;
+			goto out;
+		}
+		if ((length < PMD_SIZE) || (pfn & PG_PMD_COLOUR))
+			goto fallback;
+
 		for (i = 0; i < PTRS_PER_PMD; i++)
 			clear_pmem(kaddr + i * PAGE_SIZE, PAGE_SIZE);
 		wmb_pmem();

hm, that's a lot of copy-n-paste. Do we really need to run bdev_direct_access() twice? Will kaddr' and pfn' change?

#5 lore raw
Quoted 62 lines · Andrew Morton
Quoted 10 lines · On Tue, 22 Sep 2015 13:36:22 -0600 Ross Zwisler <[email protected]>…

The following commit:

commit 46c043ede471 ("mm: take i_mmap_lock in unmap_mapping_range() for

      DAX")

moved some code in __dax_pmd_fault() that was responsible for zeroing newly allocated PMD pages. The new location didn't properly set up 'kaddr', though, so when run this code resulted in a NULL pointer BUG.

Fix this by getting the correct 'kaddr' via bdev_direct_access().

Why the heck didn't gcc warn?

I had a fiddle:

fs/dax.c renamed from fs/dax.c~a +4 -1
@@ -529,15 +529,18 @@ int __dax_pmd_fault(struct vm_area_struc
        unsigned long pmd_addr = address & PMD_MASK;
        bool write = flags & FAULT_FLAG_WRITE;
        long length;
-       void __pmem *kaddr;
+       void *kaddr;
        pgoff_t size, pgoff;
        sector_t block, sector;
        unsigned long pfn;
        int result = 0;
 
+//     printk("%p\n", kaddr);
+
        /* Fall back to PTEs if we're going to COW */
        if (write && !(vma->vm_flags & VM_SHARED))
                return VM_FAULT_FALLBACK;
+       printk("%p\n", kaddr);
        /* If the PMD would extend outside the VMA */
        if (pmd_addr < vma->vm_start)
                return VM_FAULT_FALLBACK;

gcc warns about the first printk, but not about the second. So that "if (...) return ..." seems to have defeated gcc uninitialized-var detection. wtf?

Quoted 18 lines
fs/dax.c +10 -0
@@ -569,8 +569,20 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
      if (!buffer_size_valid(&bh) || bh.b_size < PMD_SIZE)
              goto fallback;
 
+     sector = bh.b_blocknr << (blkbits - 9);
+
      if (buffer_unwritten(&bh) || buffer_new(&bh)) {
              int i;
+
+             length = bdev_direct_access(bh.b_bdev, sector, &kaddr, &pfn,
+                                             bh.b_size);
+             if (length < 0) {
+                     result = VM_FAULT_SIGBUS;
+                     goto out;
+             }
+             if ((length < PMD_SIZE) || (pfn & PG_PMD_COLOUR))
Quoted 9 lines

fs/dax.c int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, · line 588

Quoted 5 lines
fs/dax.c +2 -0
+                     goto fallback;
+
              for (i = 0; i < PTRS_PER_PMD; i++)
                      clear_pmem(kaddr + i * PAGE_SIZE, PAGE_SIZE);
              wmb_pmem();

hm, that's a lot of copy-n-paste. Do we really need to run bdev_direct_access() twice? Will kaddr' and pfn' change?

They shouldn't change, but I'm working on a fix for handling the race of unbinding the pmem device while that kaddr is in use (unbind invalidates kaddr). The proposal is a dax_map_bh()/dax_unmap_bh() interface to temporarily pin the mapping around each usage.

#6 lore raw
Quoted 74 lines · Dan Williams
Quoted 62 lines · Andrew Morton
Quoted 10 lines · On Tue, 22 Sep 2015 13:36:22 -0600 Ross Zwisler <[email protected]>…

The following commit:

commit 46c043ede471 ("mm: take i_mmap_lock in unmap_mapping_range() for

      DAX")

moved some code in __dax_pmd_fault() that was responsible for zeroing newly allocated PMD pages. The new location didn't properly set up 'kaddr', though, so when run this code resulted in a NULL pointer BUG.

Fix this by getting the correct 'kaddr' via bdev_direct_access().

Why the heck didn't gcc warn?

I had a fiddle:

fs/dax.c renamed from fs/dax.c~a +4 -1
@@ -529,15 +529,18 @@ int __dax_pmd_fault(struct vm_area_struc
        unsigned long pmd_addr = address & PMD_MASK;
        bool write = flags & FAULT_FLAG_WRITE;
        long length;
-       void __pmem *kaddr;
+       void *kaddr;
        pgoff_t size, pgoff;
        sector_t block, sector;
        unsigned long pfn;
        int result = 0;
 
+//     printk("%p\n", kaddr);
+
        /* Fall back to PTEs if we're going to COW */
        if (write && !(vma->vm_flags & VM_SHARED))
                return VM_FAULT_FALLBACK;
+       printk("%p\n", kaddr);
        /* If the PMD would extend outside the VMA */
        if (pmd_addr < vma->vm_start)
                return VM_FAULT_FALLBACK;

gcc warns about the first printk, but not about the second. So that "if (...) return ..." seems to have defeated gcc uninitialized-var detection. wtf?

Quoted 18 lines
fs/dax.c +10 -0
@@ -569,8 +569,20 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
      if (!buffer_size_valid(&bh) || bh.b_size < PMD_SIZE)
              goto fallback;
 
+     sector = bh.b_blocknr << (blkbits - 9);
+
      if (buffer_unwritten(&bh) || buffer_new(&bh)) {
              int i;
+
+             length = bdev_direct_access(bh.b_bdev, sector, &kaddr, &pfn,
+                                             bh.b_size);
+             if (length < 0) {
+                     result = VM_FAULT_SIGBUS;
+                     goto out;
+             }
+             if ((length < PMD_SIZE) || (pfn & PG_PMD_COLOUR))
Quoted 9 lines

fs/dax.c int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, · line 588

Quoted 5 lines
fs/dax.c +2 -0
+                     goto fallback;
+
              for (i = 0; i < PTRS_PER_PMD; i++)
                      clear_pmem(kaddr + i * PAGE_SIZE, PAGE_SIZE);
              wmb_pmem();

hm, that's a lot of copy-n-paste. Do we really need to run bdev_direct_access() twice? Will kaddr' and pfn' change?

Quoted 3 lines

They shouldn't change, but I'm working on a fix for handling the race of unbinding the pmem device while that kaddr is in use (unbind invalidates kaddr).

Exactly what does "unbinding the pmem device" mean, and why can (parts of) the pmem device "go away" when there are active references to it?

Quoted 2 lines

The proposal is a dax_map_bh()/dax_unmap_bh() interface to temporarily pin the mapping around each usage.

Which mapping? The bufferhead maps file offset to filesystem block addresses, so I'm not sure what problem you are actually refering to here...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#7 lore raw
Quoted 75 lines · Dave Chinner
Quoted 74 lines · Dan Williams
Quoted 71 lines · Andrew Morton
Quoted 10 lines · On Tue, 22 Sep 2015 13:36:22 -0600 Ross Zwisler <[email protected]>…

The following commit:

commit 46c043ede471 ("mm: take i_mmap_lock in unmap_mapping_range() for

      DAX")

moved some code in __dax_pmd_fault() that was responsible for zeroing newly allocated PMD pages. The new location didn't properly set up 'kaddr', though, so when run this code resulted in a NULL pointer BUG.

Fix this by getting the correct 'kaddr' via bdev_direct_access().

Why the heck didn't gcc warn?

I had a fiddle:

fs/dax.c renamed from fs/dax.c~a +4 -1
@@ -529,15 +529,18 @@ int __dax_pmd_fault(struct vm_area_struc
        unsigned long pmd_addr = address & PMD_MASK;
        bool write = flags & FAULT_FLAG_WRITE;
        long length;
-       void __pmem *kaddr;
+       void *kaddr;
        pgoff_t size, pgoff;
        sector_t block, sector;
        unsigned long pfn;
        int result = 0;
 
+//     printk("%p\n", kaddr);
+
        /* Fall back to PTEs if we're going to COW */
        if (write && !(vma->vm_flags & VM_SHARED))
                return VM_FAULT_FALLBACK;
+       printk("%p\n", kaddr);
        /* If the PMD would extend outside the VMA */
        if (pmd_addr < vma->vm_start)
                return VM_FAULT_FALLBACK;

gcc warns about the first printk, but not about the second. So that "if (...) return ..." seems to have defeated gcc uninitialized-var detection. wtf?

Quoted 13 lines
fs/dax.c +5 -0
@@ -569,8 +569,20 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
      if (!buffer_size_valid(&bh) || bh.b_size < PMD_SIZE)
              goto fallback;
 
+     sector = bh.b_blocknr << (blkbits - 9);
+
      if (buffer_unwritten(&bh) || buffer_new(&bh)) {
              int i;
+
+             length = bdev_direct_access(bh.b_bdev, sector, &kaddr, &pfn,
+                                             bh.b_size);

fs/dax.c int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, · line 588

Quoted 10 lines
fs/dax.c +7 -0
+             if (length < 0) {
+                     result = VM_FAULT_SIGBUS;
+                     goto out;
+             }
+             if ((length < PMD_SIZE) || (pfn & PG_PMD_COLOUR))
+                     goto fallback;
+
              for (i = 0; i < PTRS_PER_PMD; i++)
                      clear_pmem(kaddr + i * PAGE_SIZE, PAGE_SIZE);
              wmb_pmem();

hm, that's a lot of copy-n-paste. Do we really need to run bdev_direct_access() twice? Will kaddr' and pfn' change?

Quoted 5 lines
Quoted 3 lines

They shouldn't change, but I'm working on a fix for handling the race of unbinding the pmem device while that kaddr is in use (unbind invalidates kaddr).

Exactly what does "unbinding the pmem device" mean,

echo namespace0.0 > /sys/bus/nd/drivers/nd_pmem/unbind

Quoted 3 lines

and why can
(parts of) the pmem device "go away" when there are active references to it?

Normally we have outstanding i/o requests to hold off blk_cleanup_queue(), but in the dax case we don't have any mechanism (yet) to flag the queue as busy. I have some patches to add a percpu_refcount for this purpose.

Quoted 7 lines
Quoted 3 lines

The proposal is a dax_map_bh()/dax_unmap_bh() interface to temporarily pin the mapping around each usage.

Which mapping? The bufferhead maps file offset to filesystem block addresses, so I'm not sure what problem you are actually refering to here...

The kaddr is coming from the devm_memremap() in the pmem driver that gets unmapped after the device is released by the driver.

#8 lore raw
Quoted 76 lines · Dan Williams
Quoted 75 lines · Dave Chinner
Quoted 74 lines · Dan Williams
Quoted 71 lines · Andrew Morton
Quoted 10 lines · On Tue, 22 Sep 2015 13:36:22 -0600 Ross Zwisler <[email protected]>…

The following commit:

commit 46c043ede471 ("mm: take i_mmap_lock in unmap_mapping_range() for

      DAX")

moved some code in __dax_pmd_fault() that was responsible for zeroing newly allocated PMD pages. The new location didn't properly set up 'kaddr', though, so when run this code resulted in a NULL pointer BUG.

Fix this by getting the correct 'kaddr' via bdev_direct_access().

Why the heck didn't gcc warn?

I had a fiddle:

fs/dax.c renamed from fs/dax.c~a +4 -1
@@ -529,15 +529,18 @@ int __dax_pmd_fault(struct vm_area_struc
        unsigned long pmd_addr = address & PMD_MASK;
        bool write = flags & FAULT_FLAG_WRITE;
        long length;
-       void __pmem *kaddr;
+       void *kaddr;
        pgoff_t size, pgoff;
        sector_t block, sector;
        unsigned long pfn;
        int result = 0;
 
+//     printk("%p\n", kaddr);
+
        /* Fall back to PTEs if we're going to COW */
        if (write && !(vma->vm_flags & VM_SHARED))
                return VM_FAULT_FALLBACK;
+       printk("%p\n", kaddr);
        /* If the PMD would extend outside the VMA */
        if (pmd_addr < vma->vm_start)
                return VM_FAULT_FALLBACK;

gcc warns about the first printk, but not about the second. So that "if (...) return ..." seems to have defeated gcc uninitialized-var detection. wtf?

Quoted 13 lines
fs/dax.c +5 -0
@@ -569,8 +569,20 @@ int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address,
      if (!buffer_size_valid(&bh) || bh.b_size < PMD_SIZE)
              goto fallback;
 
+     sector = bh.b_blocknr << (blkbits - 9);
+
      if (buffer_unwritten(&bh) || buffer_new(&bh)) {
              int i;
+
+             length = bdev_direct_access(bh.b_bdev, sector, &kaddr, &pfn,
+                                             bh.b_size);

fs/dax.c int __dax_pmd_fault(struct vm_area_struct *vma, unsigned long address, · line 588

Quoted 10 lines
fs/dax.c +7 -0
+             if (length < 0) {
+                     result = VM_FAULT_SIGBUS;
+                     goto out;
+             }
+             if ((length < PMD_SIZE) || (pfn & PG_PMD_COLOUR))
+                     goto fallback;
+
              for (i = 0; i < PTRS_PER_PMD; i++)
                      clear_pmem(kaddr + i * PAGE_SIZE, PAGE_SIZE);
              wmb_pmem();

hm, that's a lot of copy-n-paste. Do we really need to run bdev_direct_access() twice? Will kaddr' and pfn' change?

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

They shouldn't change, but I'm working on a fix for handling the race of unbinding the pmem device while that kaddr is in use (unbind invalidates kaddr).

Exactly what does "unbinding the pmem device" mean,

echo namespace0.0 > /sys/bus/nd/drivers/nd_pmem/unbind

That tells me "how", not "what"..... :/

Quoted 8 lines
Quoted 3 lines

and why can
(parts of) the pmem device "go away" when there are active references to it?

Normally we have outstanding i/o requests to hold off blk_cleanup_queue(), but in the dax case we don't have any mechanism (yet) to flag the queue as busy. I have some patches to add a percpu_refcount for this purpose.

So this comes back to fact we allow a block device to be torn down and freed while a filesystem has active references to it?

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

The proposal is a dax_map_bh()/dax_unmap_bh() interface to temporarily pin the mapping around each usage.

Which mapping? The bufferhead maps file offset to filesystem block addresses, so I'm not sure what problem you are actually refering to here...

The kaddr is coming from the devm_memremap() in the pmem driver that gets unmapped after the device is released by the driver.

Perhaps the better solution is to not tear down the block device until all active references have gone away? i.e. unbind puts the device into a persistent error state and forces all active mappings to refault. Hence all future accesses error out and then when the user unmounts the unhappy filesystem the last reference to the blockdev goes away and the mappings can be torn down safely...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#9 lore raw
Quoted 1 line · Dave Chinner

On Tue, Sep 22, 2015 at 08:00:29PM -0700, Dan Williams wrote:

<>

Quoted 10 lines
Quoted 2 lines

The kaddr is coming from the devm_memremap() in the pmem driver that gets unmapped after the device is released by the driver.

Perhaps the better solution is to not tear down the block device until all active references have gone away? i.e. unbind puts the device into a persistent error state and forces all active mappings to refault. Hence all future accesses error out and then when the user unmounts the unhappy filesystem the last reference to the blockdev goes away and the mappings can be torn down safely...

Me too

Quoted 4 lines

Cheers,

Dave.

#10 lore raw
In reply to Dave Chinner
Quoted 10 lines · Dave Chinner
Quoted 2 lines · Dan Williams

The kaddr is coming from the devm_memremap() in the pmem driver that gets unmapped after the device is released by the driver.

Perhaps the better solution is to not tear down the block device until all active references have gone away? i.e. unbind puts the device into a persistent error state and forces all active mappings to refault. Hence all future accesses error out and then when the user unmounts the unhappy filesystem the last reference to the blockdev goes away and the mappings can be torn down safely...

In fact this is how it already works in the block layer, it's just that the pmem driver was not participating in that mechanism. The filesystem prevents the gendisk and hosting driver module from going away via the heavyweight get_disk(). The gendisk keeps the request_queue from being de-allocated, but the queue can go "dead" to new requests at any time. Single-queue based drivers take the queue_lock and check blk_queue_dying() before allowing new requests. Multi-queue drivers take a lighter-weight approach and try to get a new "live" reference from a percpu_refcount.

When the backing device is unplugged or otherwise unbound from its driver it calls blk_cleanup_queue() in its shutdown path. That marks the queue dead and flushes any outstanding requests. From that point forward all requests end in error until the final put_disk().

This is what I came up with for pmem:
https://lists.01.org/pipermail/linux-nvdimm/2015-September/002206.html