kernel BUG at fs/direct-io.c:916!
Today I wanted to run xfs_fsr on my laptop. I got:
------------[ cut here ]------------ kernel BUG at fs/direct-io.c:916! invalid opcode: 0000 [#1] PREEMPT last sysfs file: /devices/pci0000:00/0000:00:13.0/usb1/idProduct Modules linked in: tda9887 tuner rtc rtc_dev rtc_ds1672 rtc_m48t86 rtc_pcf8563 rtc_rs5c372 rtc_sysfs rtc_x1205 rtc_core rtc_lib thermal fan button processor ac battery af_packet ide_scsi scsi_mod saa7134_dvb mt352 saa7134 compat_ioctl32 v4l2_common v4l1_compat ir_kbd_i2c ir_common videodev video_buf_dvb dvb_core video_buf nxt200x dvb_pll tda1004x i2c_core usbhid usbmouse tsdev pcmcia firmware_class yenta_socket psmouse evdev rt2500 rsrc_nonstatic pcmcia_core 8139too ehci_hcd ohci_hcd usbcore snd_atiixp_modem snd_atiixp snd_ac97_codec ide_cd ati_agp agpgart snd_ac97_bus snd_pcm snd_timer snd soundcore snd_page_alloc cdrom unix CPU: 0 EIP: 0060:[<c0182645>] Not tainted VLI EFLAGS: 00210246 (2.6.16-mm1 #1)
EIP is at __blockdev_direct_IO+0xe75/0xed5
eax: 00000007 ebx: dd029254 ecx: 00000009 edx: 00000000
esi: 00000000 edi: 00000000 ebp: dd029200 esp: d52e1c98
ds: 007b es: 007b ss: 0068
Process xfs_fsr (pid: 22319, threadinfo=d52e1000 task=c9278580)
Stack: <0>00000001 006e78ea 174a9067 c014a4bd 00002c00 dd029254 00011200 ddf394c0
c7bc425c d52e1ef8 00000001 01bd4040 00000009 00000009 c14f601c 00003000
00000000 00000000 00000000 c14f2fa0 00000009 00200286 00000008 00000001
Call Trace:
<c014a4bd> __handle_mm_fault+0x77d/0x850 <c01f760a> xfs_vm_direct_IO+0xda/0x100
<c01f7a40> xfs_get_blocks_direct+0x0/0x50 <c01f7170> xfs_end_io_direct+0x0/0x90
<c0175729> touch_atime+0x59/0xc0 <c013cd27> generic_file_direct_IO+0x87/0x130
<c013ce40> generic_file_direct_write+0x70/0x1b0 <c0175659> file_update_time+0x39/0xb0
<c01ff906> xfs_write+0x4a6/0xd40 <c013c520> file_read_actor+0x0/0xe0
<c01fb887> xfs_file_aio_write+0x87/0xb0 <c015b594> do_sync_write+0xc4/0x120
<c012e5b0> autoremove_wake_function+0x0/0x50 <c0205826> sys_shmdt+0x6/0x1a0
<c01fba15> xfs_file_ioctl+0x35/0x70 <c015baa3> vfs_write+0xa3/0x160
<c015b4d0> do_sync_write+0x0/0x120 <c015c451> sys_write+0x41/0x70
<c0102e93> sysenter_past_esp+0x54/0x75
Code: ff ff 8b 84 24 80 00 00 00 bb f1 ff ff ff e8 73 21 fc ff 8b 75 2c 8b 55 34 e9 65 f8 ff ff e8 63 44 17 00 8d 76 00 e9 e4 f9 ff ff <0f> 0b 94 03 55 01 31 c0 8d 76 00 e9 dd fb ff ff e8 46 44 17 00 # uname -a Linux knarzkiste 2.6.16-mm1 #1 PREEMPT Fri Mar 24 19:01:24 CET 2006 i686 GNU/Linux
Signature · 12 lines
_________________________________________________ Charité - Universitätsmedizin Berlin _________________________________________________ Ralf Hildebrandt i.A. Geschäftsbereich Informationsmanagement Campus Benjamin Franklin Hindenburgdamm 30 | Berlin Tel. +49 30 450 570155 | Fax +49 30 450 570962 [email protected] http://www.charite.de
* Ralf Hildebrandt <[email protected]>:
Quoted 1 line
Today I wanted to run
xfs_fsron my laptop. I got:
It's a 2.6.16-mm1 issue, since 2.6.16-git11 does not exhibit that failure.
Quoted 2 lines
# uname -a Linux knarzkiste 2.6.16-mm1 #1 PREEMPT Fri Mar 24 19:01:24 CET 2006 i686 GNU/Linux
Signature · 4 lines
Ralf Hildebrandt (i.A. des IT-Zentrums) [email protected] Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-962 IT-Zentrum Standort CBF send no mail to [email protected]
Quoted 7 lines · Ralf Hildebrandt
* Ralf Hildebrandt <[email protected]>:
Quoted 1 line
Today I wanted to run
xfs_fsron my laptop. I got:It's a 2.6.16-mm1 issue, since 2.6.16-git11 does not exhibit that failure.
Quoted 1 line
Linux knarzkiste 2.6.16-mm1 #1 PREEMPT Fri Mar 24 19:01:24 CET 2006 i686 GNU/Linux
Hmm, there were XFS patches in -mm last week, but they also got merged to mainline last week, not clear whether your git kernel had those changes or not. I think there's probably some direct I/O (generic) changes in -mm too based on list traffic from the last couple of weeks (I'm an -mm lamer, sorry, couldn't easily tell you exactly what patches those might be) - could you retry with todays git snapshot and see if mainline is affected? Else we'll need to find and analyse any -mm fs/direct-io.c patches.
cheers.
Signature · 1 line
Nathan
A message that is not in the archive (sent off-list or not yet received): [email protected]
* Nathan Scott <[email protected]>:
Quoted 8 lines
Hmm, there were XFS patches in -mm last week, but they also got merged to mainline last week, not clear whether your git kernel had those changes or not. I think there's probably some direct I/O (generic) changes in -mm too based on list traffic from the last couple of weeks (I'm an -mm lamer, sorry, couldn't easily tell you exactly what patches those might be) - could you retry with todays git snapshot and see if mainline is affected? Else we'll need to find and analyse any -mm
fs/direct-io.cpatches.
2.6.16-git12 also fails utterly:
Quoted 29 lines
Mar 27 00:51:55 knarzkiste kernel: ------------[ cut here ]------------ Mar 27 00:51:55 knarzkiste kernel: kernel BUG at fs/direct-io.c:916! Mar 27 00:51:55 knarzkiste kernel: invalid opcode: 0000 [#1] Mar 27 00:51:55 knarzkiste kernel: PREEMPT Mar 27 00:51:56 knarzkiste kernel: Modules linked in: thermal fan button ac battery af_packet ide_scsi sata_sil libata scsi_mod eeprom powernow_k8 freq_table processor saa7134_dvb mt352 video_buf_dvb dvb_core nxt200x dvb_pll tda1004x usbhid usbmouse tda9887 tuner saa7134 video_buf compat_ioctl32 v4l2_common v4l1_compat ir_kbd_i2c ir_common videodev tsdev pcmcia firmware_class 8250_pci ehci_hcd ohci_hcd evdev 8139too 8250 serial_core psmouse yenta_socket rsrc_nonstatic pcmcia_core ide_cd usbcore ati_agp agpgart snd_atiixp_modem snd_atiixp snd_ac97_codec snd_ac97_bus snd_pcm snd_timer snd soundcore snd_page_alloc cdrom unix Mar 27 00:51:56 knarzkiste kernel: CPU: 0 Mar 27 00:51:56 knarzkiste kernel: EIP: 0060:[__blockdev_direct_IO+3701/3797] Not tainted VLI Mar 27 00:51:56 knarzkiste kernel: EFLAGS: 00210246 (2.6.16-git12 #1) Mar 27 00:51:56 knarzkiste kernel: EIP is at __blockdev_direct_IO+0xe75/0xed5 Mar 27 00:51:56 knarzkiste kernel: eax: 00000008 ebx: dcc7d454 ecx: 00000009 edx: 00000000 Mar 27 00:51:56 knarzkiste kernel: esi: 00000000 edi: 00000000 ebp: dcc7d400 esp: dd5fec98 Mar 27 00:51:56 knarzkiste kernel: ds: 007b es: 007b ss: 0068 Mar 27 00:51:56 knarzkiste kernel: Process xfs_fsr (pid: 4828, threadinfo=dd5fe000 task=dd11f560) Mar 27 00:51:56 knarzkiste kernel: Stack: <0>00000001 022faf40 0f978067 c0148ebd 007d9200 dcc7d454 d1d00bd0 c01422d6 Mar 27 00:51:56 knarzkiste kernel: d1d0099c dd5feef8 00000001 016bf3c0 00000009 00000009 dd61d900 007da000 Mar 27 00:51:56 knarzkiste kernel: 00000000 00000000 00000000 00000000 00000009 00200286 00000008 00000000 Mar 27 00:51:56 knarzkiste kernel: Call Trace: Mar 27 00:51:56 knarzkiste kernel: <c0148ebd> __handle_mm_fault+0x77d/0x850 <c01422d6> __do_page_cache_readahead+0xc6/0x290 Mar 27 00:51:56 knarzkiste kernel: <c01f3baa> xfs_vm_direct_IO+0xda/0x100 <c01f3fe0> xfs_get_blocks_direct+0x0/0x50 Mar 27 00:51:56 knarzkiste kernel: <c01f3710> xfs_end_io_direct+0x0/0x90 <c0173019> touch_atime+0x59/0xc0 Mar 27 00:51:56 knarzkiste kernel: <c013bc87> generic_file_direct_IO+0x87/0x130 <c013bda0> generic_file_direct_write+0x70/0x1b0 Mar 27 00:51:56 knarzkiste kernel: <c0172f49> file_update_time+0x39/0xb0 <c01fbda6> xfs_write+0x4a6/0xd40 Mar 27 00:51:56 knarzkiste kernel: <c013b3b0> file_read_actor+0x0/0xe0 <c01f7d27> xfs_file_aio_write+0x87/0xb0 Mar 27 00:51:56 knarzkiste kernel: <c01596f4> do_sync_write+0xc4/0x120 <c01591d4> generic_file_llseek+0x34/0xe0 Mar 27 00:51:56 knarzkiste kernel: <c012de90> autoremove_wake_function+0x0/0x50 <c0205826> crypt+0x136/0x210 Mar 27 00:51:56 knarzkiste kernel: <c01f7eb5> xfs_file_ioctl+0x35/0x70 <c0159c03> vfs_write+0xa3/0x160 Mar 27 00:51:56 knarzkiste kernel: <c0159630> do_sync_write+0x0/0x120 <c015a5b1> sys_write+0x41/0x70 Mar 27 00:51:56 knarzkiste kernel: <c0102e93> sysenter_past_esp+0x54/0x75 Mar 27 00:51:56 knarzkiste kernel: Code: ff ff 8b 84 24 80 00 00 00 bb f1 ff ff ff e8 23 39 fc ff 8b 75 2c 8b 55 34 e9 65 f8 ff ff e8 83 32 17 00 8d 76 00 e9 e4 f9 ff ff <0f> 0b 94 03 1a a5 30 c0 8d 76 00 e9 dd fb ff ff e8 66 32 17 00
Signature · 4 lines
Ralf Hildebrandt (i.A. des IT-Zentrums) [email protected] Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-962 IT-Zentrum Standort CBF send no mail to [email protected]
Quoted 11 lines · Ralf Hildebrandt
* Nathan Scott <[email protected]>:
Quoted 8 lines
Hmm, there were XFS patches in -mm last week, but they also got merged to mainline last week, not clear whether your git kernel had those changes or not. I think there's probably some direct I/O (generic) changes in -mm too based on list traffic from the last couple of weeks (I'm an -mm lamer, sorry, couldn't easily tell you exactly what patches those might be) - could you retry with todays git snapshot and see if mainline is affected? Else we'll need to find and analyse any -mm
fs/direct-io.cpatches.
Quoted 1 line
2.6.16-git12 also fails utterly:
Could you find the inode number where its failing (fsr -v -d) and send me the xfs_bmap output for that file - use find -inum to get from an inum to a path for bmap.
thanks.
Signature · 1 line
Nathan
A message that is not in the archive (sent off-list or not yet received): 20060327060436.GC2481@frodo
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 15 lines · Ralf Hildebrandt
* Nathan Scott <[email protected]>:
Quoted 14 lines · Ralf Hildebrandt
Quoted 12 lines
* Nathan Scott <[email protected]>:
Quoted 8 lines
Hmm, there were XFS patches in -mm last week, but they also got merged to mainline last week, not clear whether your git kernel had those changes or not. I think there's probably some direct I/O (generic) changes in -mm too based on list traffic from the last couple of weeks (I'm an -mm lamer, sorry, couldn't easily tell you exactly what patches those might be) - could you retry with todays git snapshot and see if mainline is affected? Else we'll need to find and analyse any -mm
fs/direct-io.cpatches.2.6.16-git12 also fails utterly:
Quoted 8 lines
Quoted 5 lines
Could you also try reverting this patch:
and let me know if the problem still happens?
Reverting this particular patch does ELIMINATE the problem. Excellent!
OK, I think I see whats gone wrong here now. Ralf, could you try the patch below and check that it fixes your test case?
Badari, it looks like a regression from the "remove ->get_blocks() support" patch - can you look over the fix below and confirm/deny please?
I'm definately seeing block mapping requests that are smaller than the filesystem block size coming into XFS from direct-io.c - and it looks like that eventually blows up in do_direct_IO if dio_remainder becomes set and we could only map one block (if dio->blocks_available was 1 after get_more_blocks). We'll reduce that to zero right at the end of the branch that calls get_more_blocks in do_direct_IO... and mayhem ensues further on.
I have a couple of other .17 changes pending, if you could ACK this I'll get it merged in for ya.
cheers.
Signature · 41 lines
Nathan
Index: xfs-linux-2.6/fs/direct-io.c
===================================================================
--- xfs-linux-2.6.orig/fs/direct-io.c
+++ xfs-linux-2.6/fs/direct-io.c
@@ -524,8 +524,6 @@ static int get_more_blocks(struct dio *d
*/
ret = dio->page_errors;
if (ret == 0) {
- map_bh->b_state = 0;
- map_bh->b_size = 0;
BUG_ON(dio->block_in_file >= dio->final_block_in_request);
fs_startblk = dio->block_in_file >> dio->blkfactor;
dio_count = dio->final_block_in_request - dio->block_in_file;
@@ -534,6 +532,9 @@ static int get_more_blocks(struct dio *d
if (dio_count & blkmask)
fs_count++;
+ map_bh->b_state = 0;
+ map_bh->b_size = fs_count << dio->inode->i_blkbits;
+
create = dio->rw == WRITE;
if (dio->lock_type == DIO_LOCKING) {
if (dio->block_in_file < (i_size_read(dio->inode) >>
@@ -542,13 +543,13 @@ static int get_more_blocks(struct dio *d
} else if (dio->lock_type == DIO_NO_LOCKING) {
create = 0;
}
+
/*
* For writes inside i_size we forbid block creations: only
* overwrites are permitted. We fall back to buffered writes
* at a higher level for inside-i_size block-instantiating
* writes.
*/
- map_bh->b_size = fs_count << dio->blkbits;
ret = (*dio->get_block)(dio->inode, fs_startblk,
map_bh, create);
}* Nathan Scott <[email protected]>:
Quoted 2 lines
OK, I think I see whats gone wrong here now. Ralf, could you try the patch below and check that it fixes your test case?
The patch is against what? -git12? 2.6.16?
Signature · 12 lines
_________________________________________________ Charité - Universitätsmedizin Berlin _________________________________________________ Ralf Hildebrandt i.A. Geschäftsbereich Informationsmanagement Campus Benjamin Franklin Hindenburgdamm 30 | Berlin Tel. +49 30 450 570155 | Fax +49 30 450 570962 [email protected] http://www.charite.de
Quoted 6 lines · Ralf Hildebrandt
* Nathan Scott <[email protected]>:
Quoted 2 lines
OK, I think I see whats gone wrong here now. Ralf, could you try the patch below and check that it fixes your test case?
The patch is against what? -git12? 2.6.16?
Should apply cleanly to the current git tree (did yesterday, anyway).
cheers.
Signature · 1 line
Nathan
* Nathan Scott <[email protected]>:
Quoted 9 lines · Ralf Hildebrandt
Quoted 6 lines
* Nathan Scott <[email protected]>:
Quoted 2 lines
OK, I think I see whats gone wrong here now. Ralf, could you try the patch below and check that it fixes your test case?
The patch is against what? -git12? 2.6.16?
Should apply cleanly to the current git tree (did yesterday, anyway).
Alas, it fixes the problem (in -mm2, that is). Thanks!
Signature · 12 lines
_________________________________________________ Charité - Universitätsmedizin Berlin _________________________________________________ Ralf Hildebrandt i.A. Geschäftsbereich Informationsmanagement Campus Benjamin Franklin Hindenburgdamm 30 | Berlin Tel. +49 30 450 570155 | Fax +49 30 450 570962 [email protected] http://www.charite.de
Badari Pulavarty
Acked-byQuoted 40 lines · Nathan Scott
Quoted 15 lines · Ralf Hildebrandt
* Nathan Scott <[email protected]>:
Quoted 14 lines · Ralf Hildebrandt
Quoted 12 lines
* Nathan Scott <[email protected]>:
Quoted 8 lines
Hmm, there were XFS patches in -mm last week, but they also got merged to mainline last week, not clear whether your git kernel had those changes or not. I think there's probably some direct I/O (generic) changes in -mm too based on list traffic from the last couple of weeks (I'm an -mm lamer, sorry, couldn't easily tell you exactly what patches those might be) - could you retry with todays git snapshot and see if mainline is affected? Else we'll need to find and analyse any -mm
fs/direct-io.cpatches.2.6.16-git12 also fails utterly:
Quoted 8 lines
Quoted 5 lines
Could you also try reverting this patch:
and let me know if the problem still happens?
Reverting this particular patch does ELIMINATE the problem. Excellent!
OK, I think I see whats gone wrong here now. Ralf, could you try the patch below and check that it fixes your test case?
Badari, it looks like a regression from the "remove ->get_blocks() support" patch - can you look over the fix below and confirm/deny please?
I'm definately seeing block mapping requests that are smaller than the filesystem block size coming into XFS from
direct-io.c- and it looks like that eventually blows up indo_direct_IOifdio_remainderbecomes set and we could only map one block (ifdio->blocks_availablewas 1 afterget_more_blocks). We'll reduce that to zero right at the end of the branch that callsget_more_blocksindo_direct_IO... and mayhem ensues further on.
Quoted 5 lines
I have a couple of other .17 changes pending, if you could ACK this I'll get it merged in for ya.
cheers.
Nathan,
Thanks for working this out. You may want to add a description to the patch. Like:
"inode->i_blkbits should be used instead of dio->blkbits, as it may not indicate the filesystem block size all the time".
Thanks,
Badari
Quoted 5 lines · Badari Pulavarty
Thanks for working this out. You may want to add a description to the patch. Like:
"
inode->i_blkbitsshould be used instead ofdio->blkbits, as it may not indicate the filesystem block size all the time".
Will do, thanks. Oh, another thing - what is the situation where a NULL bdev would be passed into __blockdev_direct_IO? All the filesystems seem to pass i_sb->s_bdev, so I guess it must be blkdev_direct_IO - can I_BDEV(inode) ever be NULL on a block device inode (doesn't sound right)? If it cannot, I suppose we should remove those NULL bdev checks too...
cheers.
Signature · 41 lines
Nathan
Index: xfs-linux-2.6/fs/direct-io.c
===================================================================
--- xfs-linux-2.6.orig/fs/direct-io.c
+++ xfs-linux-2.6/fs/direct-io.c
@@ -1186,8 +1186,8 @@ __blockdev_direct_IO(int rw, struct kioc
size_t size;
unsigned long addr;
unsigned blkbits = inode->i_blkbits;
- unsigned bdev_blkbits = 0;
unsigned blocksize_mask = (1 << blkbits) - 1;
+ unsigned bdev_blkbits = blksize_bits(bdev_hardsect_size(bdev));
ssize_t retval = -EINVAL;
loff_t end = offset;
struct dio *dio;
@@ -1197,12 +1197,8 @@ __blockdev_direct_IO(int rw, struct kioc
if (rw & WRITE)
current->flags |= PF_SYNCWRITE;
- if (bdev)
- bdev_blkbits = blksize_bits(bdev_hardsect_size(bdev));
-
if (offset & blocksize_mask) {
- if (bdev)
- blkbits = bdev_blkbits;
+ blkbits = bdev_blkbits;
blocksize_mask = (1 << blkbits) - 1;
if (offset & blocksize_mask)
goto out;
@@ -1214,8 +1210,7 @@ __blockdev_direct_IO(int rw, struct kioc
size = iov[seg].iov_len;
end += size;
if ((addr & blocksize_mask) || (size & blocksize_mask)) {
- if (bdev)
- blkbits = bdev_blkbits;
+ blkbits = bdev_blkbits;
blocksize_mask = (1 << blkbits) - 1;
if ((addr & blocksize_mask) || (size & blocksize_mask))
goto out;Quoted 9 lines · Nathan Scott
Quoted 6 lines · Badari Pulavarty
Thanks for working this out. You may want to add a description to the patch. Like:
"
inode->i_blkbitsshould be used instead ofdio->blkbits, as it may not indicate the filesystem block size all the time".
Quoted 9 lines
Will do, thanks. Oh, another thing - what is the situation where a NULL bdev would be passed into
__blockdev_direct_IO? All the filesystems seem to passi_sb->s_bdev, so I guess it must beblkdev_direct_IO- canI_BDEV(inode)ever be NULL on a block device inode (doesn't sound right)? If it cannot, I suppose we should remove those NULL bdev checks too...cheers.
I can't think of a case, where we would end up getting b_dev = NULL in direct IO code.
Thanks,
Badari