Skip to contentSkip to conversation

FIle copy to FAT FS on NVDIMM hits BUG_ON at fs/buffer.c:3305!

DISCUSSION Kani, Toshimitsu 7 participants 13 messages Jul 25, 2017 – Sep 20, 2017

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hi,

Copying files to vfat FS on an NVDIMM device hits BUG_ON(!PageLocked(page)) in try_to_free_buffers(). It happens on 4.13-rc1, and happens on older kernels as well.

A simple reproducer is shown below. It is 100% reproducible on my setup (8GB of regular memory and 16GB of NVDIMM). It usually hits in the 3rd or 4th file copy and does not repeat with the while-loop. Interestingly, it hits only when an NVDIMM device is set as raw or memory mode. It does not hit with sector mode.

==
DEV=pmem0
set -x
mkfs.vfat /dev/$DEV
mount /dev/$DEV /mnt/$DEV
dd if=/dev/zero of=/mnt/$DEV/1Gfile bs=1M count=1024 while true; do
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-1
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-2
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-3
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-4
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-5
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-6
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-7
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-8
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-9
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-10
done
==

 kernel BUG at fs/buffer.c:3305!
 invalid opcode: 0000 [#1] SMP
:

 Workqueue: writeback wb_workfn (flush-259:0)
 task: ffff8d02595b8000 task.stack: ffffa22242400000
 RIP: 0010:try_to_free_buffers+0xd2/0xe0
 RSP: 0018:ffffa22242403830 EFLAGS: 00010246
 RAX: 00afffc000001028 RBX: 0000000000000008 RCX: ffff8d012dcf19c0
 RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffc468e3b52b80
 RBP: ffffa22242403858 R08: 0000000000000000 R09: 000000000002067c
 R10: ffff8d027ffe6000 R11: 0000000000000000 R12: 0000000000000000
 R13: ffff8d022fccdbe0 R14: ffffc468e3b52b80 R15: ffffa22242403ad0

 FS:  0000000000000000(0000) GS:ffff8d027fd40000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 00007f9d2bb80b70 CR3: 000000084fe09000 CR4: 00000000007406e0
 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
 PKRU: 55555554

 Call Trace:
  clean_buffers+0x5d/0x70
  __mpage_writepage+0x567/0x760
  ? page_mkclean+0x6a/0xb0
  write_cache_pages+0x205/0x580
  ? clean_buffers+0x70/0x70
  ? fat_add_cluster+0x80/0x80 [fat]
  mpage_writepages+0x7c/0x100
  ? fat_add_cluster+0x80/0x80 [fat]
  ? __set_page_dirty+0x9b/0xc0
  ? fprop_fraction_percpu+0x2f/0x80
  fat_writepages+0x15/0x20 [fat]
  ? fat_writepages+0x15/0x20 [fat]
  do_writepages+0x25/0x80
  __writeback_single_inode+0x45/0x350
  writeback_sb_inodes+0x25e/0x610
  __writeback_inodes_wb+0x92/0xc0
  wb_writeback+0x29b/0x340
  wb_workfn+0x195/0x3d0
  ? wb_workfn+0x195/0x3d0
  process_one_work+0x193/0x3d0
  worker_thread+0x4e/0x3d0
  kthread+0x114/0x150
  ? process_one_work+0x3d0/0x3d0
  ? kthread_park+0x60/0x60
  ? kthread_park+0x60/0x60
  ret_from_fork+0x25/0x30
  :
 RIP: try_to_free_buffers+0xd2/0xe0 RSP: ffffa22242403830

Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#1 lore raw
Quoted 11 lines · Toshimitsu

Hi,

Copying files to vfat FS on an NVDIMM device hits BUG_ON(!PageLocked(page)) in try_to_free_buffers(). It happens on 4.13-rc1, and happens on older kernels as well.

A simple reproducer is shown below. It is 100% reproducible on my setup (8GB of regular memory and 16GB of NVDIMM). It usually hits in the 3rd or 4th file copy and does not repeat with the while-loop. Interestingly, it hits only when an NVDIMM device is set as raw or memory mode. It does not hit with sector mode.

Interesting, thanks for the report.. I was able to reproduce this in my KVM setup with a virtualized NVDIMM using v4.12. I also tried reproducing it with BRD, and wasn't able.

I'll take a look.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#2 lore raw
Quoted 13 lines · Ross Zwisler
Quoted 11 lines · Toshimitsu

Hi,

Copying files to vfat FS on an NVDIMM device hits BUG_ON(!PageLocked(page)) in try_to_free_buffers().  It happens on 4.13-rc1, and happens on older kernels as well.

A simple reproducer is shown below.  It is 100% reproducible on my setup (8GB of regular memory and 16GB of NVDIMM).  It usually hits in the 3rd or 4th file copy and does not repeat with the while- loop. Interestingly, it hits only when an NVDIMM device is set as raw or memory mode.  It does not hit with sector mode.

Quoted 5 lines

Interesting, thanks for the report..  I was able to reproduce this in my KVM setup with a virtualized NVDIMM using v4.12.  I also tried reproducing it with BRD, and wasn't able.

I'll take a look.

Great!! Thanks Ross!
-Toshi

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#3 lore raw
In reply to Ross Zwisler
Quoted 5 lines · Ross Zwisler

Interesting, thanks for the report.. I was able to reproduce this in my KVM setup with a virtualized NVDIMM using v4.12. I also tried reproducing it with BRD, and wasn't able.

I'll take a look.

Hi Ross and Toshi,

can you please keep me on Cc for the thread and possible patches? I'd be willing to play test monkey as well.

Byte,
Johannes

Signature · 10 lines
Johannes Thumshirn                                          Storage
[email protected]                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
#4 lore raw
Quoted 13 lines · Toshimitsu

kernel BUG at fs/buffer.c:3305!
invalid opcode: 0000 [#1] SMP
:

 Workqueue: writeback wb_workfn (flush-259:0)
 task: ffff8d02595b8000 task.stack: ffffa22242400000
 RIP: 0010:try_to_free_buffers+0xd2/0xe0
 RSP: 0018:ffffa22242403830 EFLAGS: 00010246
 RAX: 00afffc000001028 RBX: 0000000000000008 RCX: ffff8d012dcf19c0
 RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffc468e3b52b80
 RBP: ffffa22242403858 R08: 0000000000000000 R09: 000000000002067c
 R10: ffff8d027ffe6000 R11: 0000000000000000 R12: 0000000000000000
 R13: ffff8d022fccdbe0 R14: ffffc468e3b52b80 R15: ffffa22242403ad0
 FS:  0000000000000000(0000) GS:ffff8d027fd40000(0000)

The locking of this path seems to be broken. The guy familiar to bdev_write_page() path will made real fix though, The following patch should be explaining enough what is wrong.

In short, clean_buffers() must be called before unlocking lock_page().

---

 fs/block_dev.c |    2 --
 fs/mpage.c     |    1 +
 mm/page_io.c   |    1 +
 3 files changed, 2 insertions(+), 2 deletions(-)
fs/mpage.c renamed from fs/mpage.c~bdev_write_page-fix +1 -0
diff -puN fs/mpage.c~bdev_write_page-fix fs/mpage.c
@@ -605,6 +605,7 @@ alloc_new:
 			if (!bdev_write_page(bdev, blocks[0] << (blkbits - 9),
 								page, wbc)) {
 				clean_buffers(page, first_unmapped);
+				unlock_page(page);
 				goto out;
 			}
 		}
mm/page_io.c renamed from mm/page_io.c~bdev_write_page-fix +1 -0
diff -puN mm/page_io.c~bdev_write_page-fix mm/page_io.c
@@ -308,6 +308,7 @@ int __swap_writepage(struct page *page,
 
 	ret = bdev_write_page(sis->bdev, swap_page_sector(page), page, wbc);
 	if (!ret) {
+		unlock_page(page);
 		count_vm_event(PSWPOUT);
 		return 0;
 	}
fs/block_dev.c renamed from fs/block_dev.c~bdev_write_page-fix +0 -2
diff -puN fs/block_dev.c~bdev_write_page-fix fs/block_dev.c
@@ -714,8 +714,6 @@ int bdev_write_page(struct block_device
 	result = ops->rw_page(bdev, sector + get_start_sect(bdev), page, true);
 	if (result)
 		end_page_writeback(page);
-	else
-		unlock_page(page);
 	blk_queue_exit(bdev->bd_queue);
 	return result;
 }

_

Signature · 5 lines
OGAWA Hirofumi <[email protected]>
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
#5 lore raw
Quoted 16 lines · OGAWA Hirofumi
Quoted 13 lines · Toshimitsu

kernel BUG at fs/buffer.c:3305!
invalid opcode: 0000 [#1] SMP
:

 Workqueue: writeback wb_workfn (flush-259:0)
 task: ffff8d02595b8000 task.stack: ffffa22242400000
 RIP: 0010:try_to_free_buffers+0xd2/0xe0
 RSP: 0018:ffffa22242403830 EFLAGS: 00010246
 RAX: 00afffc000001028 RBX: 0000000000000008 RCX: ffff8d012dcf19c0
 RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffc468e3b52b80
 RBP: ffffa22242403858 R08: 0000000000000000 R09: 000000000002067c
 R10: ffff8d027ffe6000 R11: 0000000000000000 R12: 0000000000000000
 R13: ffff8d022fccdbe0 R14: ffffc468e3b52b80 R15: ffffa22242403ad0
 FS:  0000000000000000(0000) GS:ffff8d027fd40000(0000)
Quoted 3 lines

The locking of this path seems to be broken. The guy familiar to bdev_write_page() path will made real fix though, The following patch should be explaining enough what is wrong.

Is there someone in particular who is familiar with bdev_write_page() that is working on this fix, or does someone need to pick this up?
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#6 lore raw
Quoted 17 lines · Ross Zwisler
Quoted 16 lines · OGAWA Hirofumi
Quoted 13 lines · Toshimitsu

kernel BUG at fs/buffer.c:3305!
invalid opcode: 0000 [#1] SMP
:

 Workqueue: writeback wb_workfn (flush-259:0)
 task: ffff8d02595b8000 task.stack: ffffa22242400000
 RIP: 0010:try_to_free_buffers+0xd2/0xe0
 RSP: 0018:ffffa22242403830 EFLAGS: 00010246
 RAX: 00afffc000001028 RBX: 0000000000000008 RCX: ffff8d012dcf19c0
 RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffc468e3b52b80
 RBP: ffffa22242403858 R08: 0000000000000000 R09: 000000000002067c
 R10: ffff8d027ffe6000 R11: 0000000000000000 R12: 0000000000000000
 R13: ffff8d022fccdbe0 R14: ffffc468e3b52b80 R15: ffffa22242403ad0
 FS:  0000000000000000(0000) GS:ffff8d027fd40000(0000)
Quoted 6 lines
Quoted 3 lines

The locking of this path seems to be broken. The guy familiar to bdev_write_page() path will made real fix though, The following patch should be explaining enough what is wrong.

Is there someone in particular who is familiar with bdev_write_page() that is working on this fix, or does someone need to pick this up?

Another question, does ->rw_page() really buy us that much with the pmem driver? If applications want to enjoy the lowest latency access they can just use DAX. There's now only 4 drivers that use rw_page since nvme dropped its usage and I'd be inclined to just rip it out.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#7 lore raw
Quoted 4 lines · Dan Williams

Another question, does ->rw_page() really buy us that much with the pmem driver? If applications want to enjoy the lowest latency access they can just use DAX. There's now only 4 drivers that use rw_page since nvme dropped its usage and I'd be inclined to just rip it out.

nvme never supported rw_page (there was a page for it, but it fortunately never got merged).

rw_page are massive pain the ass and the method should go away. For make_request drivers that actually operate synchronous (e.g. the ramdisk) it's not much of a benefit, and even for normally asynchronous drivers like nvme the block layer polling interface is much more suitable.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#8 lore raw
In reply to OGAWA Hirofumi
Quoted 5 lines · OGAWA Hirofumi

The locking of this path seems to be broken. The guy familiar to bdev_write_page() path will made real fix though, The following patch should be explaining enough what is wrong.

In short, clean_buffers() must be called before unlocking lock_page().

Thanks for that. This should fix the problem while not leaking the unlock_page call outside bdev_write_page.

--- 8< ---

fs/block_dev.c +4 -2
index 9941dc8342df..3fbe75bdd257 100644
@@ -716,10 +716,12 @@ int bdev_write_page(struct block_device *bdev, sector_t sector,
 
 	set_page_writeback(page);
 	result = ops->rw_page(bdev, sector + get_start_sect(bdev), page, true);
-	if (result)
+	if (result) {
 		end_page_writeback(page);
-	else
+	} else {
+		clean_page_buffers(page);
 		unlock_page(page);
+	}
 	blk_queue_exit(bdev->bd_queue);
 	return result;
 }
fs/mpage.c +11 -3
index 2e4c41ccb5c9..d97b003f1607 100644
@@ -468,6 +468,16 @@ static void clean_buffers(struct page *page, unsigned first_unmapped)
 		try_to_free_buffers(page);
 }
 
+/*
+ * For situations where we want to clean all buffers attached to a page.
+ * We don't need to calculate how many buffers are attached to the page,
+ * we just need to specify a number larger than the maximum number of buffers.
+ */
+void clean_page_buffers(struct page *page)
+{
+	clean_buffers(page, PAGE_SIZE);
+}
+
 static int __mpage_writepage(struct page *page, struct writeback_control *wbc,
 		      void *data)
 {
@@ -605,10 +615,8 @@ static int __mpage_writepage(struct page *page, struct writeback_control *wbc,
 	if (bio == NULL) {
 		if (first_unmapped == blocks_per_page) {
 			if (!bdev_write_page(bdev, blocks[0] << (blkbits - 9),
-								page, wbc)) {
-				clean_buffers(page, first_unmapped);
+								page, wbc))
 				goto out;
-			}
 		}
 		bio = mpage_alloc(bdev, blocks[0] << (blkbits - 9),
 				BIO_MAX_PAGES, GFP_NOFS|__GFP_HIGH);
include/linux/buffer_head.h +1 -0
index c8dae555eccf..446b24cac67d 100644
@@ -232,6 +232,7 @@ int generic_write_end(struct file *, struct address_space *,
 				loff_t, unsigned, unsigned,
 				struct page *, void *);
 void page_zero_new_buffers(struct page *page, unsigned from, unsigned to);
+void clean_page_buffers(struct page *page);
 int cont_write_begin(struct file *, struct address_space *, loff_t,
 			unsigned, unsigned, struct page **, void **,
 			get_block_t *, loff_t *);

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

author Tested-by
#9 lore raw
Quoted 8 lines · Matthew Wilcox
Quoted 6 lines · OGAWA Hirofumi

The locking of this path seems to be broken. The guy familiar to bdev_write_page() path will made real fix though, The following patch should be explaining enough what is wrong.

In short, clean_buffers() must be called before unlocking lock_page().

Quoted 6 lines

Thanks for that.  This should fix the problem while not leaking the unlock_page call outside bdev_write_page.

--- 8< ---

Thanks Willy and Hirofumi for the quick fix! I've tested the change, and it works fine.

Out of curiosity, I am still wondering about the following:
- Why is this issue exposed with FAT FS, but not with other FSs?
- Why did I not see this issue with BTT?

Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#10 lore raw
Quoted 16 lines · Toshimitsu
Quoted 8 lines · Matthew Wilcox
Quoted 6 lines · OGAWA Hirofumi

The locking of this path seems to be broken. The guy familiar to bdev_write_page() path will made real fix though, The following patch should be explaining enough what is wrong.

In short, clean_buffers() must be called before unlocking lock_page().

Quoted 6 lines

Thanks for that.  This should fix the problem while not leaking the unlock_page call outside bdev_write_page.

--- 8< ---

Quoted 8 lines

Thanks Willy and Hirofumi for the quick fix! I've tested the change, and it works fine.

Out of curiosity, I am still wondering about the following:
- Why is this issue exposed with FAT FS, but not with other FSs?
- Why did I not see this issue with BTT?

I also didn't see it with FAT + BRD, so I think the answer to both of the above is just that the timings were different.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#11 lore raw
Quoted 20 lines · Ross Zwisler
Quoted 19 lines · Toshimitsu
Quoted 8 lines · Matthew Wilcox
Quoted 6 lines · OGAWA Hirofumi

The locking of this path seems to be broken. The guy familiar to bdev_write_page() path will made real fix though, The following patch should be explaining enough what is wrong.

In short, clean_buffers() must be called before unlocking lock_page().

Quoted 6 lines

Thanks for that.  This should fix the problem while not leaking the unlock_page call outside bdev_write_page.

--- 8< ---

Thanks Willy and Hirofumi for the quick fix!  I've tested the change, and it works fine.

Quoted 8 lines
Quoted 5 lines

Out of curiosity, I am still wondering about the following:
 - Why is this issue exposed with FAT FS, but not with other FSs?
 - Why did I not see this issue with BTT?

I also didn't see it with FAT + BRD, so I think the answer to both of the above is just that the timings were different.

Yeah, that may well be the case. Just a tough nature for testing.

Thanks!
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#12 lore raw
In reply to Kani, Toshimitsu
Quoted 12 lines · Toshimitsu
Quoted 11 lines · Matthew Wilcox
Quoted 6 lines · OGAWA Hirofumi

The locking of this path seems to be broken. The guy familiar to bdev_write_page() path will made real fix though, The following patch should be explaining enough what is wrong.

In short, clean_buffers() must be called before unlocking lock_page().

Thanks for that.  This should fix the problem while not leaking the unlock_page call outside bdev_write_page.

Quoted 8 lines
Quoted 3 lines

--- 8< ---

Thanks Willy and Hirofumi for the quick fix! I've tested the change, and it works fine.

Since removing rw_page would have to wait till 4.15, can we go with this fix for 4.14? I confirmed that Matthew's patch [1] applies cleanly to 4.14-rc1. This patch can also be applied to stable kernels.

Matthew, can you resend this patch with description?

[1] https://www.spinics.net/lists/linux-fsdevel/msg113835.html

Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm