FIle copy to FAT FS on NVDIMM hits BUG_ON at fs/buffer.c:3305!
Subsystems (4)
Hi,
Copying files to vfat FS on an NVDIMM device hits BUG_ON(!PageLocked(page)) in try_to_free_buffers(). It happens on 4.13-rc1, and happens on older kernels as well.
A simple reproducer is shown below. It is 100% reproducible on my setup (8GB of regular memory and 16GB of NVDIMM). It usually hits in the 3rd or 4th file copy and does not repeat with the while-loop. Interestingly, it hits only when an NVDIMM device is set as raw or memory mode. It does not hit with sector mode.
==
DEV=pmem0
set -x
mkfs.vfat /dev/$DEV
mount /dev/$DEV /mnt/$DEV
dd if=/dev/zero of=/mnt/$DEV/1Gfile bs=1M count=1024 while true; do
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-1
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-2
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-3
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-4
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-5
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-6
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-7
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-8
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-9
cp /mnt/$DEV/1Gfile /mnt/$DEV/file-10
done
==
kernel BUG at fs/buffer.c:3305!
invalid opcode: 0000 [#1] SMP
:
Workqueue: writeback wb_workfn (flush-259:0) task: ffff8d02595b8000 task.stack: ffffa22242400000 RIP: 0010:try_to_free_buffers+0xd2/0xe0 RSP: 0018:ffffa22242403830 EFLAGS: 00010246 RAX: 00afffc000001028 RBX: 0000000000000008 RCX: ffff8d012dcf19c0 RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffc468e3b52b80 RBP: ffffa22242403858 R08: 0000000000000000 R09: 000000000002067c R10: ffff8d027ffe6000 R11: 0000000000000000 R12: 0000000000000000 R13: ffff8d022fccdbe0 R14: ffffc468e3b52b80 R15: ffffa22242403ad0
FS: 0000000000000000(0000) GS:ffff8d027fd40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9d2bb80b70 CR3: 000000084fe09000 CR4: 00000000007406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace: clean_buffers+0x5d/0x70 __mpage_writepage+0x567/0x760 ? page_mkclean+0x6a/0xb0 write_cache_pages+0x205/0x580 ? clean_buffers+0x70/0x70 ? fat_add_cluster+0x80/0x80 [fat] mpage_writepages+0x7c/0x100 ? fat_add_cluster+0x80/0x80 [fat] ? __set_page_dirty+0x9b/0xc0 ? fprop_fraction_percpu+0x2f/0x80 fat_writepages+0x15/0x20 [fat] ? fat_writepages+0x15/0x20 [fat] do_writepages+0x25/0x80 __writeback_single_inode+0x45/0x350 writeback_sb_inodes+0x25e/0x610 __writeback_inodes_wb+0x92/0xc0 wb_writeback+0x29b/0x340 wb_workfn+0x195/0x3d0 ? wb_workfn+0x195/0x3d0 process_one_work+0x193/0x3d0 worker_thread+0x4e/0x3d0 kthread+0x114/0x150 ? process_one_work+0x3d0/0x3d0 ? kthread_park+0x60/0x60 ? kthread_park+0x60/0x60 ret_from_fork+0x25/0x30 : RIP: try_to_free_buffers+0xd2/0xe0 RSP: ffffa22242403830
Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 11 lines · Toshimitsu
Hi,
Copying files to vfat FS on an NVDIMM device hits BUG_ON(!PageLocked(page)) in
try_to_free_buffers(). It happens on 4.13-rc1, and happens on older kernels as well.A simple reproducer is shown below. It is 100% reproducible on my setup (8GB of regular memory and 16GB of NVDIMM). It usually hits in the 3rd or 4th file copy and does not repeat with the while-loop. Interestingly, it hits only when an NVDIMM device is set as raw or memory mode. It does not hit with sector mode.
Interesting, thanks for the report.. I was able to reproduce this in my KVM setup with a virtualized NVDIMM using v4.12. I also tried reproducing it with BRD, and wasn't able.
I'll take a look.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 13 lines · Ross Zwisler
Quoted 11 lines · Toshimitsu
Hi,
Copying files to vfat FS on an NVDIMM device hits BUG_ON(!PageLocked(page)) in try_to_free_buffers(). It happens on 4.13-rc1, and happens on older kernels as well.
A simple reproducer is shown below. It is 100% reproducible on my setup (8GB of regular memory and 16GB of NVDIMM). It usually hits in the 3rd or 4th file copy and does not repeat with the while- loop. Interestingly, it hits only when an NVDIMM device is set as raw or memory mode. It does not hit with sector mode.
Quoted 5 lines
Interesting, thanks for the report.. I was able to reproduce this in my KVM setup with a virtualized NVDIMM using v4.12. I also tried reproducing it with BRD, and wasn't able.
I'll take a look.
Great!! Thanks Ross!
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 5 lines · Ross Zwisler
Interesting, thanks for the report.. I was able to reproduce this in my KVM setup with a virtualized NVDIMM using v4.12. I also tried reproducing it with BRD, and wasn't able.
I'll take a look.
Hi Ross and Toshi,
can you please keep me on Cc for the thread and possible patches? I'd be willing to play test monkey as well.
Byte,
Johannes
Signature · 10 lines
Johannes Thumshirn Storage [email protected] +49 911 74053 689 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850 _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 13 lines · Toshimitsu
kernel BUG at
fs/buffer.c:3305!
invalid opcode: 0000 [#1] SMP
:Workqueue: writeback wb_workfn (flush-259:0) task: ffff8d02595b8000 task.stack: ffffa22242400000 RIP: 0010:try_to_free_buffers+0xd2/0xe0 RSP: 0018:ffffa22242403830 EFLAGS: 00010246 RAX: 00afffc000001028 RBX: 0000000000000008 RCX: ffff8d012dcf19c0 RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffc468e3b52b80 RBP: ffffa22242403858 R08: 0000000000000000 R09: 000000000002067c R10: ffff8d027ffe6000 R11: 0000000000000000 R12: 0000000000000000 R13: ffff8d022fccdbe0 R14: ffffc468e3b52b80 R15: ffffa22242403ad0 FS: 0000000000000000(0000) GS:ffff8d027fd40000(0000)
The locking of this path seems to be broken. The guy familiar to bdev_write_page() path will made real fix though, The following patch should be explaining enough what is wrong.
In short, clean_buffers() must be called before unlocking lock_page().
---
fs/block_dev.c | 2 -- fs/mpage.c | 1 + mm/page_io.c | 1 + 3 files changed, 2 insertions(+), 2 deletions(-)
fs/mpage.c renamed from fs/mpage.c~bdev_write_page-fix +1 -0
@@ -605,6 +605,7 @@ alloc_new:if (!bdev_write_page (bdev, blocks[0 ] << (blkbits -9 ), page, wbc)) {clean_buffers (page, first_unmapped); +unlock_page (page); goto out; } }
mm/page_io.c renamed from mm/page_io.c~bdev_write_page-fix +1 -0
@@ -308,6 +308,7 @@ int __swap_writepage(struct page *page, ret =bdev_write_page (sis->bdev,swap_page_sector (page), page, wbc);if (!ret) { +unlock_page (page);count_vm_event (PSWPOUT );return 0 ; }
fs/block_dev.c renamed from fs/block_dev.c~bdev_write_page-fix +0 -2
@@ -714,8 +714,6 @@ int bdev_write_page(struct block_device result = ops->rw_page (bdev, sector +get_start_sect (bdev), page, true);if (result)end_page_writeback (page); -else -unlock_page (page);blk_queue_exit (bdev->bd_queue);return result; }
_
Signature · 5 lines
OGAWA Hirofumi <[email protected]> _______________________________________________ Linux-nvdimm mailing list [email protected] https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 16 lines · OGAWA Hirofumi
Quoted 13 lines · Toshimitsu
kernel BUG at
fs/buffer.c:3305!
invalid opcode: 0000 [#1] SMP
:Workqueue: writeback wb_workfn (flush-259:0) task: ffff8d02595b8000 task.stack: ffffa22242400000 RIP: 0010:try_to_free_buffers+0xd2/0xe0 RSP: 0018:ffffa22242403830 EFLAGS: 00010246 RAX: 00afffc000001028 RBX: 0000000000000008 RCX: ffff8d012dcf19c0 RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffc468e3b52b80 RBP: ffffa22242403858 R08: 0000000000000000 R09: 000000000002067c R10: ffff8d027ffe6000 R11: 0000000000000000 R12: 0000000000000000 R13: ffff8d022fccdbe0 R14: ffffc468e3b52b80 R15: ffffa22242403ad0 FS: 0000000000000000(0000) GS:ffff8d027fd40000(0000)
Quoted 3 lines
The locking of this path seems to be broken. The guy familiar to
bdev_write_page()path will made real fix though, The following patch should be explaining enough what is wrong.
Is there someone in particular who is familiar with bdev_write_page() that is working on this fix, or does someone need to pick this up?
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 17 lines · Ross Zwisler
Quoted 16 lines · OGAWA Hirofumi
Quoted 13 lines · Toshimitsu
kernel BUG at
fs/buffer.c:3305!
invalid opcode: 0000 [#1] SMP
:Workqueue: writeback wb_workfn (flush-259:0) task: ffff8d02595b8000 task.stack: ffffa22242400000 RIP: 0010:try_to_free_buffers+0xd2/0xe0 RSP: 0018:ffffa22242403830 EFLAGS: 00010246 RAX: 00afffc000001028 RBX: 0000000000000008 RCX: ffff8d012dcf19c0 RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffc468e3b52b80 RBP: ffffa22242403858 R08: 0000000000000000 R09: 000000000002067c R10: ffff8d027ffe6000 R11: 0000000000000000 R12: 0000000000000000 R13: ffff8d022fccdbe0 R14: ffffc468e3b52b80 R15: ffffa22242403ad0 FS: 0000000000000000(0000) GS:ffff8d027fd40000(0000)
Quoted 6 lines
Quoted 3 lines
The locking of this path seems to be broken. The guy familiar to
bdev_write_page()path will made real fix though, The following patch should be explaining enough what is wrong.Is there someone in particular who is familiar with
bdev_write_page()that is working on this fix, or does someone need to pick this up?
Another question, does ->rw_page() really buy us that much with the pmem driver? If applications want to enjoy the lowest latency access they can just use DAX. There's now only 4 drivers that use rw_page since nvme dropped its usage and I'd be inclined to just rip it out.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 4 lines · Dan Williams
Another question, does ->rw_page() really buy us that much with the pmem driver? If applications want to enjoy the lowest latency access they can just use DAX. There's now only 4 drivers that use
rw_pagesince nvme dropped its usage and I'd be inclined to just rip it out.
nvme never supported rw_page (there was a page for it, but it fortunately never got merged).
rw_page are massive pain the ass and the method should go away. For make_request drivers that actually operate synchronous (e.g. the ramdisk) it's not much of a benefit, and even for normally asynchronous drivers like nvme the block layer polling interface is much more suitable.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 5 lines · OGAWA Hirofumi
The locking of this path seems to be broken. The guy familiar to
bdev_write_page()path will made real fix though, The following patch should be explaining enough what is wrong.In short,
clean_buffers()must be called before unlockinglock_page().
Thanks for that. This should fix the problem while not leaking the unlock_page call outside bdev_write_page.
--- 8< ---
fs/block_dev.c +4 -2
@@ -716,10 +716,12 @@ int bdev_write_page(struct block_device *bdev, sector_t sector,set_page_writeback (page); result = ops->rw_page (bdev, sector +get_start_sect (bdev), page, true); -if (result) +if (result) {end_page_writeback (page); -else + }else { +clean_page_buffers (page);unlock_page (page); + }blk_queue_exit (bdev->bd_queue);return result; }
fs/mpage.c +11 -3
@@ -468,6 +468,16 @@ static void clean_buffers(struct page *page, unsigned first_unmapped)try_to_free_buffers (page); } +/* +* For situations where we want to clean all buffers attached to a page. +* We don't need to calculate how many buffers are attached to the page, +* we just need to specify a number larger than the maximum number of buffers. +*/ +void clean_page_buffers (struct page *page) +{ +clean_buffers (page,PAGE_SIZE ); +} +static int __mpage_writepage (struct page *page,struct writeback_control *wbc,void *data) {
@@ -605,10 +615,8 @@ static int __mpage_writepage(struct page *page, struct writeback_control *wbc,if (bio ==NULL ) {if (first_unmapped == blocks_per_page) {if (!bdev_write_page (bdev, blocks[0 ] << (blkbits -9 ), - page, wbc)) { -clean_buffers (page, first_unmapped); + page, wbc)) goto out; - } } bio =mpage_alloc (bdev, blocks[0 ] << (blkbits -9 ),BIO_MAX_PAGES ,GFP_NOFS |__GFP_HIGH);
include/linux/buffer_head.h +1 -0
@@ -232,6 +232,7 @@ int generic_write_end(struct file *, struct address_space *, loff_t, unsigned, unsigned, struct page *, void *);void page_zero_new_buffers (struct page *page,unsigned from,unsigned to); +void clean_page_buffers (struct page *page);int cont_write_begin (struct file *,struct address_space *,loff_t ,unsigned ,unsigned ,struct page **,void **,get_block_t *,loff_t *);
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Kani, Toshimitsu
Tested-byQuoted 8 lines · Matthew Wilcox
Quoted 6 lines · OGAWA Hirofumi
The locking of this path seems to be broken. The guy familiar to
bdev_write_page()path will made real fix though, The following patch should be explaining enough what is wrong.In short,
clean_buffers()must be called before unlockinglock_page().
Quoted 6 lines
Thanks for that. This should fix the problem while not leaking the
unlock_pagecall outsidebdev_write_page.--- 8< ---
Signed-off-by Matthew Wilcox <[email protected]>
Thanks Willy and Hirofumi for the quick fix! I've tested the change, and it works fine.
Out of curiosity, I am still wondering about the following:
- Why is this issue exposed with FAT FS, but not with other FSs?
- Why did I not see this issue with BTT?
Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 16 lines · Toshimitsu
Quoted 8 lines · Matthew Wilcox
Quoted 6 lines · OGAWA Hirofumi
The locking of this path seems to be broken. The guy familiar to
bdev_write_page()path will made real fix though, The following patch should be explaining enough what is wrong.In short,
clean_buffers()must be called before unlockinglock_page().Quoted 6 lines
Thanks for that. This should fix the problem while not leaking the
unlock_pagecall outsidebdev_write_page.--- 8< ---
Signed-off-by Matthew Wilcox <[email protected]>
Quoted 8 lines
Thanks Willy and Hirofumi for the quick fix! I've tested the change, and it works fine.
Tested-by Toshi Kani <[email protected]>Out of curiosity, I am still wondering about the following:
- Why is this issue exposed with FAT FS, but not with other FSs?
- Why did I not see this issue with BTT?
I also didn't see it with FAT + BRD, so I think the answer to both of the above is just that the timings were different.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 20 lines · Ross Zwisler
Quoted 19 lines · Toshimitsu
Quoted 8 lines · Matthew Wilcox
Quoted 6 lines · OGAWA Hirofumi
The locking of this path seems to be broken. The guy familiar to
bdev_write_page()path will made real fix though, The following patch should be explaining enough what is wrong.In short,
clean_buffers()must be called before unlockinglock_page().Quoted 6 lines
Thanks for that. This should fix the problem while not leaking the
unlock_pagecall outsidebdev_write_page.--- 8< ---
Signed-off-by Matthew Wilcox <[email protected]>Thanks Willy and Hirofumi for the quick fix! I've tested the change, and it works fine.
Quoted 8 lines
Quoted 5 lines
Tested-by Toshi Kani <[email protected]>Out of curiosity, I am still wondering about the following:
- Why is this issue exposed with FAT FS, but not with other FSs?
- Why did I not see this issue with BTT?I also didn't see it with FAT + BRD, so I think the answer to both of the above is just that the timings were different.
Yeah, that may well be the case. Just a tough nature for testing.
Thanks!
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Quoted 12 lines · Toshimitsu
Quoted 11 lines · Matthew Wilcox
Quoted 6 lines · OGAWA Hirofumi
The locking of this path seems to be broken. The guy familiar to
bdev_write_page()path will made real fix though, The following patch should be explaining enough what is wrong.In short,
clean_buffers()must be called before unlockinglock_page().Thanks for that. This should fix the problem while not leaking the
unlock_pagecall outsidebdev_write_page.
Quoted 8 lines
Quoted 3 lines
--- 8< ---
Signed-off-by Matthew Wilcox <[email protected]>Thanks Willy and Hirofumi for the quick fix! I've tested the change, and it works fine.
Tested-by Toshi Kani <[email protected]>
Since removing rw_page would have to wait till 4.15, can we go with this fix for 4.14? I confirmed that Matthew's patch [1] applies cleanly to 4.14-rc1. This patch can also be applied to stable kernels.
Matthew, can you resend this patch with description?
[1] https://www.spinics.net/lists/linux-fsdevel/msg113835.html
Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm