Skip to contentSkip to conversation

kernel BUG at fs/direct-io.c:916!

DISCUSSION Ralf Hildebrandt 3 participants 12 messages Mar 26, 2006 – Mar 30, 2006 3 messages not in the archive

lore
Share & tools
lore
#0 lore raw

Today I wanted to run xfs_fsr on my laptop. I got:

------------[ cut here ]------------
kernel BUG at fs/direct-io.c:916!
invalid opcode: 0000 [#1]
PREEMPT 
last sysfs file: /devices/pci0000:00/0000:00:13.0/usb1/idProduct
Modules linked in: tda9887 tuner rtc rtc_dev rtc_ds1672 rtc_m48t86 rtc_pcf8563 rtc_rs5c372 rtc_sysfs rtc_x1205 rtc_core rtc_lib thermal fan button processor ac battery af_packet ide_scsi scsi_mod saa7134_dvb mt352 saa7134 compat_ioctl32 v4l2_common v4l1_compat ir_kbd_i2c ir_common videodev video_buf_dvb dvb_core video_buf nxt200x dvb_pll tda1004x i2c_core usbhid usbmouse tsdev pcmcia firmware_class yenta_socket psmouse evdev rt2500 rsrc_nonstatic pcmcia_core 8139too ehci_hcd ohci_hcd usbcore snd_atiixp_modem snd_atiixp snd_ac97_codec ide_cd ati_agp agpgart snd_ac97_bus snd_pcm snd_timer snd soundcore snd_page_alloc cdrom unix
CPU:    0
EIP:    0060:[<c0182645>]    Not tainted VLI
EFLAGS: 00210246   (2.6.16-mm1 #1) 

EIP is at __blockdev_direct_IO+0xe75/0xed5
eax: 00000007 ebx: dd029254 ecx: 00000009 edx: 00000000
esi: 00000000 edi: 00000000 ebp: dd029200 esp: d52e1c98
ds: 007b es: 007b ss: 0068
Process xfs_fsr (pid: 22319, threadinfo=d52e1000 task=c9278580)

Stack: <0>00000001 006e78ea 174a9067 c014a4bd 00002c00 dd029254 00011200 ddf394c0 
       c7bc425c d52e1ef8 00000001 01bd4040 00000009 00000009 c14f601c 00003000 
       00000000 00000000 00000000 c14f2fa0 00000009 00200286 00000008 00000001 
Call Trace:
 <c014a4bd> __handle_mm_fault+0x77d/0x850   <c01f760a> xfs_vm_direct_IO+0xda/0x100
 <c01f7a40> xfs_get_blocks_direct+0x0/0x50   <c01f7170> xfs_end_io_direct+0x0/0x90
 <c0175729> touch_atime+0x59/0xc0   <c013cd27> generic_file_direct_IO+0x87/0x130
 <c013ce40> generic_file_direct_write+0x70/0x1b0   <c0175659> file_update_time+0x39/0xb0
 <c01ff906> xfs_write+0x4a6/0xd40   <c013c520> file_read_actor+0x0/0xe0
 <c01fb887> xfs_file_aio_write+0x87/0xb0   <c015b594> do_sync_write+0xc4/0x120
 <c012e5b0> autoremove_wake_function+0x0/0x50   <c0205826> sys_shmdt+0x6/0x1a0
 <c01fba15> xfs_file_ioctl+0x35/0x70   <c015baa3> vfs_write+0xa3/0x160
 <c015b4d0> do_sync_write+0x0/0x120   <c015c451> sys_write+0x41/0x70
 <c0102e93> sysenter_past_esp+0x54/0x75  
Code: ff ff 8b 84 24 80 00 00 00 bb f1 ff ff ff e8 73 21 fc ff 8b 75 2c 8b 55 34 e9 65 f8 ff ff e8 63 44 17 00 8d 76 00 e9 e4 f9 ff ff <0f> 0b 94 03 55 01 31 c0 8d 76 00 e9 dd fb ff ff e8 46 44 17 00 
# uname -a
Linux knarzkiste 2.6.16-mm1 #1 PREEMPT Fri Mar 24 19:01:24 CET 2006  i686 GNU/Linux
Signature · 12 lines
_________________________________________________

  Charité - Universitätsmedizin Berlin
_________________________________________________

  Ralf Hildebrandt
   i.A. Geschäftsbereich Informationsmanagement
   Campus Benjamin Franklin
   Hindenburgdamm 30 | Berlin
   Tel. +49 30 450 570155 | Fax +49 30 450 570962
   [email protected]
   http://www.charite.de
#1 lore raw

* Ralf Hildebrandt <[email protected]>:

Quoted 1 line

Today I wanted to run xfs_fsr on my laptop. I got:

It's a 2.6.16-mm1 issue, since 2.6.16-git11 does not exhibit that failure.

Quoted 2 lines
# uname -a
Linux knarzkiste 2.6.16-mm1 #1 PREEMPT Fri Mar 24 19:01:24 CET 2006  i686 GNU/Linux
Signature · 4 lines
Ralf Hildebrandt (i.A. des IT-Zentrums)         [email protected]
Charite - Universitätsmedizin Berlin            Tel.  +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin    Fax.  +49 (0)30-450 570-962
IT-Zentrum Standort CBF                 send no mail to [email protected]
#2 lore raw
Quoted 7 lines · Ralf Hildebrandt

* Ralf Hildebrandt <[email protected]>:

Quoted 1 line

Today I wanted to run xfs_fsr on my laptop. I got:

It's a 2.6.16-mm1 issue, since 2.6.16-git11 does not exhibit that failure.

Quoted 1 line

Linux knarzkiste 2.6.16-mm1 #1 PREEMPT Fri Mar 24 19:01:24 CET 2006 i686 GNU/Linux

Hmm, there were XFS patches in -mm last week, but they also got merged to mainline last week, not clear whether your git kernel had those changes or not. I think there's probably some direct I/O (generic) changes in -mm too based on list traffic from the last couple of weeks (I'm an -mm lamer, sorry, couldn't easily tell you exactly what patches those might be) - could you retry with todays git snapshot and see if mainline is affected? Else we'll need to find and analyse any -mm fs/direct-io.c patches.

cheers.

Signature · 1 line
Nathan
#4 lore raw
In reply to a message not in the archive

* Nathan Scott <[email protected]>:

Quoted 8 lines

Hmm, there were XFS patches in -mm last week, but they also got merged to mainline last week, not clear whether your git kernel had those changes or not. I think there's probably some direct I/O (generic) changes in -mm too based on list traffic from the last couple of weeks (I'm an -mm lamer, sorry, couldn't easily tell you exactly what patches those might be) - could you retry with todays git snapshot and see if mainline is affected? Else we'll need to find and analyse any -mm fs/direct-io.c patches.

2.6.16-git12 also fails utterly:

Quoted 29 lines
Mar 27 00:51:55 knarzkiste kernel: ------------[ cut here ]------------
Mar 27 00:51:55 knarzkiste kernel: kernel BUG at fs/direct-io.c:916!
Mar 27 00:51:55 knarzkiste kernel: invalid opcode: 0000 [#1]
Mar 27 00:51:55 knarzkiste kernel: PREEMPT
Mar 27 00:51:56 knarzkiste kernel: Modules linked in: thermal fan button ac battery af_packet ide_scsi sata_sil libata scsi_mod eeprom powernow_k8 freq_table processor saa7134_dvb mt352 video_buf_dvb dvb_core nxt200x dvb_pll tda1004x usbhid usbmouse tda9887 tuner saa7134 video_buf compat_ioctl32 v4l2_common v4l1_compat ir_kbd_i2c ir_common videodev tsdev pcmcia firmware_class 8250_pci ehci_hcd ohci_hcd evdev 8139too 8250 serial_core psmouse yenta_socket rsrc_nonstatic pcmcia_core ide_cd usbcore ati_agp agpgart snd_atiixp_modem snd_atiixp snd_ac97_codec snd_ac97_bus snd_pcm snd_timer snd soundcore snd_page_alloc cdrom unix
Mar 27 00:51:56 knarzkiste kernel: CPU:    0
Mar 27 00:51:56 knarzkiste kernel: EIP:    0060:[__blockdev_direct_IO+3701/3797]    Not tainted VLI
Mar 27 00:51:56 knarzkiste kernel: EFLAGS: 00210246   (2.6.16-git12 #1)
Mar 27 00:51:56 knarzkiste kernel: EIP is at __blockdev_direct_IO+0xe75/0xed5
Mar 27 00:51:56 knarzkiste kernel: eax: 00000008   ebx: dcc7d454   ecx: 00000009   edx: 00000000
Mar 27 00:51:56 knarzkiste kernel: esi: 00000000   edi: 00000000   ebp: dcc7d400   esp: dd5fec98
Mar 27 00:51:56 knarzkiste kernel: ds: 007b   es: 007b   ss: 0068
Mar 27 00:51:56 knarzkiste kernel: Process xfs_fsr (pid: 4828, threadinfo=dd5fe000 task=dd11f560)
Mar 27 00:51:56 knarzkiste kernel: Stack: <0>00000001 022faf40 0f978067 c0148ebd 007d9200 dcc7d454 d1d00bd0 c01422d6
Mar 27 00:51:56 knarzkiste kernel:        d1d0099c dd5feef8 00000001 016bf3c0 00000009 00000009 dd61d900 007da000
Mar 27 00:51:56 knarzkiste kernel:        00000000 00000000 00000000 00000000 00000009 00200286 00000008 00000000
Mar 27 00:51:56 knarzkiste kernel: Call Trace:
Mar 27 00:51:56 knarzkiste kernel:  <c0148ebd> __handle_mm_fault+0x77d/0x850   <c01422d6> __do_page_cache_readahead+0xc6/0x290
Mar 27 00:51:56 knarzkiste kernel:  <c01f3baa> xfs_vm_direct_IO+0xda/0x100   <c01f3fe0> xfs_get_blocks_direct+0x0/0x50
Mar 27 00:51:56 knarzkiste kernel:  <c01f3710> xfs_end_io_direct+0x0/0x90   <c0173019> touch_atime+0x59/0xc0
Mar 27 00:51:56 knarzkiste kernel:  <c013bc87> generic_file_direct_IO+0x87/0x130   <c013bda0> generic_file_direct_write+0x70/0x1b0
Mar 27 00:51:56 knarzkiste kernel:  <c0172f49> file_update_time+0x39/0xb0   <c01fbda6> xfs_write+0x4a6/0xd40
Mar 27 00:51:56 knarzkiste kernel:  <c013b3b0> file_read_actor+0x0/0xe0   <c01f7d27> xfs_file_aio_write+0x87/0xb0
Mar 27 00:51:56 knarzkiste kernel:  <c01596f4> do_sync_write+0xc4/0x120   <c01591d4> generic_file_llseek+0x34/0xe0
Mar 27 00:51:56 knarzkiste kernel:  <c012de90> autoremove_wake_function+0x0/0x50   <c0205826> crypt+0x136/0x210
Mar 27 00:51:56 knarzkiste kernel:  <c01f7eb5> xfs_file_ioctl+0x35/0x70   <c0159c03> vfs_write+0xa3/0x160
Mar 27 00:51:56 knarzkiste kernel:  <c0159630> do_sync_write+0x0/0x120   <c015a5b1> sys_write+0x41/0x70
Mar 27 00:51:56 knarzkiste kernel:  <c0102e93> sysenter_past_esp+0x54/0x75
Mar 27 00:51:56 knarzkiste kernel: Code: ff ff 8b 84 24 80 00 00 00 bb f1 ff ff ff e8 23 39 fc ff 8b 75 2c 8b 55 34 e9 65 f8 ff ff e8 83 32 17 00 8d 76 00 e9 e4 f9 ff ff <0f> 0b 94 03 1a a5 30 c0 8d 76 00 e9 dd fb ff ff e8 66 32 17 00
Signature · 4 lines
Ralf Hildebrandt (i.A. des IT-Zentrums)         [email protected]
Charite - Universitätsmedizin Berlin            Tel.  +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin    Fax.  +49 (0)30-450 570-962
IT-Zentrum Standort CBF                 send no mail to [email protected]
#5 lore raw
Quoted 11 lines · Ralf Hildebrandt

* Nathan Scott <[email protected]>:

Quoted 8 lines

Hmm, there were XFS patches in -mm last week, but they also got merged to mainline last week, not clear whether your git kernel had those changes or not. I think there's probably some direct I/O (generic) changes in -mm too based on list traffic from the last couple of weeks (I'm an -mm lamer, sorry, couldn't easily tell you exactly what patches those might be) - could you retry with todays git snapshot and see if mainline is affected? Else we'll need to find and analyse any -mm fs/direct-io.c patches.

Quoted 1 line

2.6.16-git12 also fails utterly:

Could you find the inode number where its failing (fsr -v -d) and send me the xfs_bmap output for that file - use find -inum to get from an inum to a path for bmap.

thanks.

Signature · 1 line
Nathan
#8 lore raw
In reply to a message not in the archive
Quoted 15 lines · Ralf Hildebrandt

* Nathan Scott <[email protected]>:

Quoted 14 lines · Ralf Hildebrandt
Quoted 12 lines

* Nathan Scott <[email protected]>:

Quoted 8 lines

Hmm, there were XFS patches in -mm last week, but they also got merged to mainline last week, not clear whether your git kernel had those changes or not. I think there's probably some direct I/O (generic) changes in -mm too based on list traffic from the last couple of weeks (I'm an -mm lamer, sorry, couldn't easily tell you exactly what patches those might be) - could you retry with todays git snapshot and see if mainline is affected? Else we'll need to find and analyse any -mm fs/direct-io.c patches.

2.6.16-git12 also fails utterly:

Quoted 8 lines
Quoted 5 lines

Could you also try reverting this patch:

http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=1d8fa7a2b9a39d18727acc5c468e870df606c852

and let me know if the problem still happens?

Reverting this particular patch does ELIMINATE the problem. Excellent!

OK, I think I see whats gone wrong here now. Ralf, could you try the patch below and check that it fixes your test case?

Badari, it looks like a regression from the "remove ->get_blocks() support" patch - can you look over the fix below and confirm/deny please?

I'm definately seeing block mapping requests that are smaller than the filesystem block size coming into XFS from direct-io.c - and it looks like that eventually blows up in do_direct_IO if dio_remainder becomes set and we could only map one block (if dio->blocks_available was 1 after get_more_blocks). We'll reduce that to zero right at the end of the branch that calls get_more_blocks in do_direct_IO... and mayhem ensues further on.

I have a couple of other .17 changes pending, if you could ACK this I'll get it merged in for ya.

cheers.

Signature · 41 lines
Nathan


Index: xfs-linux-2.6/fs/direct-io.c
===================================================================
--- xfs-linux-2.6.orig/fs/direct-io.c
+++ xfs-linux-2.6/fs/direct-io.c
@@ -524,8 +524,6 @@ static int get_more_blocks(struct dio *d
 	 */
 	ret = dio->page_errors;
 	if (ret == 0) {
-		map_bh->b_state = 0;
-		map_bh->b_size = 0;
 		BUG_ON(dio->block_in_file >= dio->final_block_in_request);
 		fs_startblk = dio->block_in_file >> dio->blkfactor;
 		dio_count = dio->final_block_in_request - dio->block_in_file;
@@ -534,6 +532,9 @@ static int get_more_blocks(struct dio *d
 		if (dio_count & blkmask)	
 			fs_count++;
 
+		map_bh->b_state = 0;
+		map_bh->b_size = fs_count << dio->inode->i_blkbits;
+
 		create = dio->rw == WRITE;
 		if (dio->lock_type == DIO_LOCKING) {
 			if (dio->block_in_file < (i_size_read(dio->inode) >>
@@ -542,13 +543,13 @@ static int get_more_blocks(struct dio *d
 		} else if (dio->lock_type == DIO_NO_LOCKING) {
 			create = 0;
 		}
+
 		/*
 		 * For writes inside i_size we forbid block creations: only
 		 * overwrites are permitted.  We fall back to buffered writes
 		 * at a higher level for inside-i_size block-instantiating
 		 * writes.
 		 */
-		map_bh->b_size = fs_count << dio->blkbits;
 		ret = (*dio->get_block)(dio->inode, fs_startblk,
 						map_bh, create);
 	}
#9 lore raw

* Nathan Scott <[email protected]>:

Quoted 2 lines

OK, I think I see whats gone wrong here now. Ralf, could you try the patch below and check that it fixes your test case?

The patch is against what? -git12? 2.6.16?

Signature · 12 lines
_________________________________________________

  Charité - Universitätsmedizin Berlin
_________________________________________________

  Ralf Hildebrandt
   i.A. Geschäftsbereich Informationsmanagement
   Campus Benjamin Franklin
   Hindenburgdamm 30 | Berlin
   Tel. +49 30 450 570155 | Fax +49 30 450 570962
   [email protected]
   http://www.charite.de
#12 lore raw
In reply to Nathan Scott
Quoted 40 lines · Nathan Scott
Quoted 15 lines · Ralf Hildebrandt

* Nathan Scott <[email protected]>:

Quoted 14 lines · Ralf Hildebrandt
Quoted 12 lines

* Nathan Scott <[email protected]>:

Quoted 8 lines

Hmm, there were XFS patches in -mm last week, but they also got merged to mainline last week, not clear whether your git kernel had those changes or not. I think there's probably some direct I/O (generic) changes in -mm too based on list traffic from the last couple of weeks (I'm an -mm lamer, sorry, couldn't easily tell you exactly what patches those might be) - could you retry with todays git snapshot and see if mainline is affected? Else we'll need to find and analyse any -mm fs/direct-io.c patches.

2.6.16-git12 also fails utterly:

Quoted 8 lines
Quoted 5 lines

Could you also try reverting this patch:

http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=1d8fa7a2b9a39d18727acc5c468e870df606c852

and let me know if the problem still happens?

Reverting this particular patch does ELIMINATE the problem. Excellent!

OK, I think I see whats gone wrong here now. Ralf, could you try the patch below and check that it fixes your test case?

Badari, it looks like a regression from the "remove ->get_blocks() support" patch - can you look over the fix below and confirm/deny please?

I'm definately seeing block mapping requests that are smaller than the filesystem block size coming into XFS from direct-io.c - and it looks like that eventually blows up in do_direct_IO if dio_remainder becomes set and we could only map one block (if dio->blocks_available was 1 after get_more_blocks). We'll reduce that to zero right at the end of the branch that calls get_more_blocks in do_direct_IO... and mayhem ensues further on.

Quoted 5 lines

I have a couple of other .17 changes pending, if you could ACK this I'll get it merged in for ya.

cheers.

Nathan,

Thanks for working this out. You may want to add a description to the patch. Like:

"inode->i_blkbits should be used instead of dio->blkbits, as it may not indicate the filesystem block size all the time".

Thanks,
Badari

#10 lore raw
In reply to Ralf Hildebrandt
Quoted 6 lines · Ralf Hildebrandt

* Nathan Scott <[email protected]>:

Quoted 2 lines

OK, I think I see whats gone wrong here now. Ralf, could you try the patch below and check that it fixes your test case?

The patch is against what? -git12? 2.6.16?

Should apply cleanly to the current git tree (did yesterday, anyway).

cheers.

Signature · 1 line
Nathan
#13 lore raw
In reply to Badari Pulavarty
Quoted 5 lines · Badari Pulavarty

Thanks for working this out. You may want to add a description to the patch. Like:

"inode->i_blkbits should be used instead of dio->blkbits, as it may not indicate the filesystem block size all the time".

Will do, thanks. Oh, another thing - what is the situation where a NULL bdev would be passed into __blockdev_direct_IO? All the filesystems seem to pass i_sb->s_bdev, so I guess it must be blkdev_direct_IO - can I_BDEV(inode) ever be NULL on a block device inode (doesn't sound right)? If it cannot, I suppose we should remove those NULL bdev checks too...

cheers.

Signature · 41 lines
Nathan


Index: xfs-linux-2.6/fs/direct-io.c
===================================================================
--- xfs-linux-2.6.orig/fs/direct-io.c
+++ xfs-linux-2.6/fs/direct-io.c
@@ -1186,8 +1186,8 @@ __blockdev_direct_IO(int rw, struct kioc
 	size_t size;
 	unsigned long addr;
 	unsigned blkbits = inode->i_blkbits;
-	unsigned bdev_blkbits = 0;
 	unsigned blocksize_mask = (1 << blkbits) - 1;
+	unsigned bdev_blkbits = blksize_bits(bdev_hardsect_size(bdev));
 	ssize_t retval = -EINVAL;
 	loff_t end = offset;
 	struct dio *dio;
@@ -1197,12 +1197,8 @@ __blockdev_direct_IO(int rw, struct kioc
 	if (rw & WRITE)
 		current->flags |= PF_SYNCWRITE;
 
-	if (bdev)
-		bdev_blkbits = blksize_bits(bdev_hardsect_size(bdev));
-
 	if (offset & blocksize_mask) {
-		if (bdev)
-			 blkbits = bdev_blkbits;
+		blkbits = bdev_blkbits;
 		blocksize_mask = (1 << blkbits) - 1;
 		if (offset & blocksize_mask)
 			goto out;
@@ -1214,8 +1210,7 @@ __blockdev_direct_IO(int rw, struct kioc
 		size = iov[seg].iov_len;
 		end += size;
 		if ((addr & blocksize_mask) || (size & blocksize_mask))  {
-			if (bdev)
-				 blkbits = bdev_blkbits;
+			blkbits = bdev_blkbits;
 			blocksize_mask = (1 << blkbits) - 1;
 			if ((addr & blocksize_mask) || (size & blocksize_mask))  
 				goto out;
#11 lore raw
In reply to Nathan Scott

* Nathan Scott <[email protected]>:

Quoted 9 lines · Ralf Hildebrandt
Quoted 6 lines

* Nathan Scott <[email protected]>:

Quoted 2 lines

OK, I think I see whats gone wrong here now. Ralf, could you try the patch below and check that it fixes your test case?

The patch is against what? -git12? 2.6.16?

Should apply cleanly to the current git tree (did yesterday, anyway).

Alas, it fixes the problem (in -mm2, that is). Thanks!

Signature · 12 lines
_________________________________________________

  Charité - Universitätsmedizin Berlin
_________________________________________________

  Ralf Hildebrandt
   i.A. Geschäftsbereich Informationsmanagement
   Campus Benjamin Franklin
   Hindenburgdamm 30 | Berlin
   Tel. +49 30 450 570155 | Fax +49 30 450 570962
   [email protected]
   http://www.charite.de
#14 lore raw
In reply to Nathan Scott
Quoted 9 lines · Nathan Scott
Quoted 6 lines · Badari Pulavarty

Thanks for working this out. You may want to add a description to the patch. Like:

"inode->i_blkbits should be used instead of dio->blkbits, as it may not indicate the filesystem block size all the time".

Quoted 9 lines

Will do, thanks. Oh, another thing - what is the situation where a NULL bdev would be passed into __blockdev_direct_IO? All the filesystems seem to pass i_sb->s_bdev, so I guess it must be blkdev_direct_IO - can I_BDEV(inode) ever be NULL on a block device inode (doesn't sound right)? If it cannot, I suppose we should remove those NULL bdev checks too...

cheers.

I can't think of a case, where we would end up getting b_dev = NULL in direct IO code.

Thanks,
Badari

Quoted 2 lines