Skip to contentSkip to conversation

2.6.20.3: kernel BUG at mm/slab.c:597 try#2

DISCUSSION Andreas Steinmetz 6 participants 16 messages Mar 19, 2007

Subsystems (2)
SCSI SCSI tape driver
lore
Share & tools
lore

Andreas Steinmetz

author
#0 lore raw

As posted to lkml and linux-scsi on 2007-03-15 without reply, see
http://marc.info/?l=linux-kernel&m=117395128412313&w=2 for original post:

It is not so nice when one can write backup tapes but the tapes cannot be read. I don't know if memory management or the st driver is the culprit, but this is a not so nice situation.

I can't even say if the tapes are written correctly as I can't read them (one does not reboot production machines back to 2.4.x just to try to read a backup tape - I don't have 2.6.x older than 2.6.20 on these machines).

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#1 lore raw
Quoted 2 lines · On Mon, 19 Mar 2007 01:34:22 +0100 Andreas Steinmetz <[email protected]> wrote:

As posted to lkml and linux-scsi on 2007-03-15 without reply, see
http://marc.info/?l=linux-kernel&m=117395128412313&w=2 for original post:

Repeatable oops in our most recently released kernel, nobody bothers to reply.

Quoted 8 lines

It is not so nice when one can write backup tapes but the tapes cannot be read. I don't know if memory management or the st driver is the culprit, but this is a not so nice situation.

I can't even say if the tapes are written correctly as I can't read them (one does not reboot production machines back to 2.4.x just to try to read a backup tape - I don't have 2.6.x older than 2.6.20 on these machines).

        BUG_ON(!PageSlab(page));

that's seriously screwed up. Do you have CONFIG_DEBUG_SLAB enabled? If not, please enable it and retest.

#2 lore raw
Quoted 4 lines · Andrew Morton
        BUG_ON(!PageSlab(page));

that's seriously screwed up. Do you have CONFIG_DEBUG_SLAB enabled? If not, please enable it and retest.

This is scary. Looking at disassembly of the OOPS:

Disassembly of section .text:

00000000 <.text>:

  0:   5f                      pop    %edi
  1:   c3                      ret
  2:   57                      push   %edi
  3:   89 c1                   mov    %eax,%ecx
  5:   89 d7                   mov    %edx,%edi
  7:   8d 92 00 00 00 40       lea    0x40000000(%edx),%edx
  d:   56                      push   %esi
  e:   c1 ea 0c                shr    $0xc,%edx
 11:   53                      push   %ebx
 12:   c1 e2 05                shl    $0x5,%edx
 15:   03 15 40 5d 5a c0       add    0xc05a5d40,%edx

At this point, edx has the result of virt_to_page().

 1b:   8b 02                   mov    (%edx),%eax
 1d:   f6 c4 40                test   $0x40,%ah
 20:   74 03                   je     0x25

If it's a compound page, look up the real page from ->private.

22: 8b 52 0c mov 0xc(%edx),%edx

Now, reload page flags.

25: 8b 02 mov (%edx),%eax

And test...

 27:   a8 80                   test   $0x80,%al
 29:   75 04                   jne    0x2f
 2b:   0f 0b                   ud2a
 2d:   eb fe                   jmp    0x2d
 2f:   39 4a 18                cmp    %ecx,0x18(%edx)

[snip, snip]

EIP is at kmem_cache_free+0x29/0x5a
eax: c1800000 ebx: f0ae12c0 ecx: c18f73c0 edx: c1800000
esi: c1919de0 edi: 00000000 ebp: 00001000 esp: f1fe7e14
ds: 007b es: 007b ss: 0068

But somehow eax and edx have the same value 0xc1800000 here. Hmm?

Pekka

#3 lore raw
Quoted 6 lines · Pekka Enberg

EIP is at kmem_cache_free+0x29/0x5a
eax: c1800000 ebx: f0ae12c0 ecx: c18f73c0 edx: c1800000
esi: c1919de0 edi: 00000000 ebp: 00001000 esp: f1fe7e14
ds: 007b es: 007b ss: 0068

But somehow eax and edx have the same value 0xc1800000 here. Hmm?

Aah, but if you look at contents of the stack:

Stack: f0ae12c0 c1919de0 ffffffea c0137f97 00000000 f0ae12c0 c1919e20 c0168d45
      f0ae12c0 00001000 c0168fb9 c02a77e3 00001000 00000000 00000000 00000000
      00000000 c17bb6e0 00001000 00000000 f1b38be8 00000003 f54ac050 c1b9d6e8
Call Trace:
[<c0137f97>] mempool_free+0x48/0x4c
[<c0168d45>] bio_free+0x21/0x2c
[<c0168fb9>] bio_put+0x22/0x23

You can see that mempool_free is passing a NULL pointer to kmem_cache_free() which doesn't handle it properly. The NULL pointer comes from bio_free() where ->bi_io_vec is NULL because nr_iovecs passed to bio_alloc_bioset() was zero.

The question is, why is nr_pages zero in scsi_req_map_sg()?

#4 lore raw
Quoted 6 lines · Pekka Enberg

You can see that mempool_free is passing a NULL pointer to kmem_cache_free() which doesn't handle it properly. The NULL pointer comes from bio_free() where ->bi_io_vec is NULL because nr_iovecs passed to bio_alloc_bioset() was zero.

The question is, why is nr_pages zero in scsi_req_map_sg()?

Note that the following patch I posted only addresses the part where slab is clearly failing here:

http://lkml.org/lkml/2007/3/19/42

So, while it should fix the oops, there might be a bug lurking in the SCSI or block layer still.

#5 lore raw
In reply to Andreas Steinmetz
Quoted 11 lines · Andreas Steinmetz

As posted to lkml and linux-scsi on 2007-03-15 without reply, see
http://marc.info/?l=linux-kernel&m=117395128412313&w=2 for original post:

It is not so nice when one can write backup tapes but the tapes cannot be read. I don't know if memory management or the st driver is the culprit, but this is a not so nice situation.

I can't even say if the tapes are written correctly as I can't read them (one does not reboot production machines back to 2.4.x just to try to read a backup tape - I don't have 2.6.x older than 2.6.20 on these machines).

Could you try this patch
http://marc.info/?l=linux-scsi&m=116464965414878&w=2
I thought st was modified to not send offsets in the last elements but it looks like it wasn't.

#6 lore raw
Quoted 9 lines · Mike Christie
Quoted 4 lines

I can't even say if the tapes are written correctly as I can't read them (one does not reboot production machines back to 2.4.x just to try to read a backup tape - I don't have 2.6.x older than 2.6.20 on these machines).

Could you try this patch
http://marc.info/?l=linux-scsi&m=116464965414878&w=2
I thought st was modified to not send offsets in the last elements but it looks like it wasn't.

Actually, there are two patches in the email referred to. If the analysis that we're passing NULL to mempool_free is correct, it should be the second one that fixes the problem (the one that checks bio->bi_io_vec before freeing it). Which would mean we have a nr_vecs==0 bio generated by the tar somehow.

James

#7 lore raw
Quoted 10 lines · James Bottomley
Quoted 8 lines · Mike Christie
Quoted 4 lines

I can't even say if the tapes are written correctly as I can't read them (one does not reboot production machines back to 2.4.x just to try to read a backup tape - I don't have 2.6.x older than 2.6.20 on these machines).

Could you try this patch
http://marc.info/?l=linux-scsi&m=116464965414878&w=2
I thought st was modified to not send offsets in the last elements but it looks like it wasn't.

Quoted 6 lines

Actually, there are two patches in the email referred to. If the analysis that we're passing NULL to mempool_free is correct, it should be the second one that fixes the problem (the one that checks bio->bi_io_vec before freeing it). Which would mean we have a nr_vecs==0 bio generated by the tar somehow.

I think we might only need the first patch if the problem is similar to what the lsi guys were seeing. I thought the problem is that we are not estimating how large the transfer is correctly because we do not take into account offsets at the end. This results in nr_vecs being zero when it should be a valid value. I thought Kai's patch:
http://bugzilla.kernel.org/show_bug.cgi?id=7919
http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commitdiff;h=9abe16c670bd3d4ab5519257514f9f291383d104
fixed the problem on st's side, but I guess not so you are probably right.

Here is a patch that dumps the sgl we are getting from st so we can see for sure what we are getting and can decide if we need the first patch, second patch or both.

drivers/scsi/scsi_lib.c +4 -0
index 5f95570..81005aa 100644
@@ -306,6 +306,10 @@ static int scsi_req_map_sg(struct reques
 	struct bio *bio = NULL;
 	int i, err, nr_vecs = 0;
 
+	for (i = 0; i < nsegs; i++)
+		printk(KERN_INFO "sg length %u offset %u\n", sgl[i].length,
+			sgl[i].offset);
+
 	for (i = 0; i < nsegs; i++) {
 		page = sgl[i].page;
 		off = sgl[i].offset;
#8 lore raw
Quoted 17 lines · Mike Christie
Quoted 9 lines · James Bottomley
Quoted 8 lines · Mike Christie
Quoted 4 lines

I can't even say if the tapes are written correctly as I can't read them (one does not reboot production machines back to 2.4.x just to try to read a backup tape - I don't have 2.6.x older than 2.6.20 on these machines).

Could you try this patch
http://marc.info/?l=linux-scsi&m=116464965414878&w=2
I thought st was modified to not send offsets in the last elements but it looks like it wasn't.

Quoted 6 lines

Actually, there are two patches in the email referred to. If the analysis that we're passing NULL to mempool_free is correct, it should be the second one that fixes the problem (the one that checks bio->bi_io_vec before freeing it). Which would mean we have a nr_vecs==0 bio generated by the tar somehow.

Quoted 8 lines

I think we might only need the first patch if the problem is similar to what the lsi guys were seeing. I thought the problem is that we are not estimating how large the transfer is correctly because we do not take into account offsets at the end. This results in nr_vecs being zero when it should be a valid value. I thought Kai's patch:
http://bugzilla.kernel.org/show_bug.cgi?id=7919
http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commitdiff;h=9abe16c670bd3d4ab5519257514f9f291383d104
fixed the problem on st's side,

Oh, I noticed that the subject for the mail references 2.6.30.3 and the patch for st in the bugzilla did not make into 2.6.20 and is not in .3. Could we try the st patch in the bugzilla first?

#13 lore raw
In reply to James Bottomley
Quoted 11 lines · James Bottomley
Quoted 9 lines · Mike Christie
Quoted 4 lines

I can't even say if the tapes are written correctly as I can't read them (one does not reboot production machines back to 2.4.x just to try to read a backup tape - I don't have 2.6.x older than 2.6.20 on these machines).

Could you try this patch
http://marc.info/?l=linux-scsi&m=116464965414878&w=2
I thought st was modified to not send offsets in the last elements but it looks like it wasn't.

Quoted 7 lines

Actually, there are two patches in the email referred to. If the analysis that we're passing NULL to mempool_free is correct, it should be the second one that fixes the problem (the one that checks bio->bi_io_vec before freeing it). Which would mean we have a nr_vecs==0 bio generated by the tar somehow.

James

James, could this also be the cause of a tar based backup going crazy and thinking all data is new under any 2.6.21-rc* kernel I've tested so far with amanda, which in my case uses tar? I've tried the fedora patched tar-1.15-1, and one I hand built right after 1.15-1 came out over a year ago, and they both do it, but only when booted to a 2.6.21-rc* kernel.

This obviously will be a show-stopper, either for amanda (and by inference, any app that uses tar), or for the migration of an amanda users machinery to a 2.6.21 kernel.

Quoted 6 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 5 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
fractal radiation jamming the backbone
#14 lore raw
Quoted 9 lines · Gene Heskett

James, could this also be the cause of a tar based backup going crazy and thinking all data is new under any 2.6.21-rc* kernel I've tested so far with amanda, which in my case uses tar? I've tried the fedora patched tar-1.15-1, and one I hand built right after 1.15-1 came out over a year ago, and they both do it, but only when booted to a 2.6.21-rc* kernel.

This obviously will be a show-stopper, either for amanda (and by inference, any app that uses tar), or for the migration of an amanda users machinery to a 2.6.21 kernel.

Er, I don't think so .. that sounds like mtime miscompare, which is either a problem with the filesystem or a problem with the way mtime is stored in the tar archive.

James

author
#9 lore raw
In reply to Mike Christie
Quoted 26 lines · Mike Christie
Quoted 16 lines · Mike Christie
Quoted 9 lines · James Bottomley
Quoted 8 lines · Mike Christie
Quoted 4 lines

I can't even say if the tapes are written correctly as I can't read them (one does not reboot production machines back to 2.4.x just to try to read a backup tape - I don't have 2.6.x older than 2.6.20 on these machines).

Could you try this patch
http://marc.info/?l=linux-scsi&m=116464965414878&w=2
I thought st was modified to not send offsets in the last elements but it looks like it wasn't.

Quoted 6 lines

Actually, there are two patches in the email referred to. If the analysis that we're passing NULL to mempool_free is correct, it should be the second one that fixes the problem (the one that checks bio->bi_io_vec before freeing it). Which would mean we have a nr_vecs==0 bio generated by the tar somehow.

Quoted 8 lines

I think we might only need the first patch if the problem is similar to what the lsi guys were seeing. I thought the problem is that we are not estimating how large the transfer is correctly because we do not take into account offsets at the end. This results in nr_vecs being zero when it should be a valid value. I thought Kai's patch:
http://bugzilla.kernel.org/show_bug.cgi?id=7919
http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commitdiff;h=9abe16c670bd3d4ab5519257514f9f291383d104
fixed the problem on st's side,

Quoted 3 lines

Oh, I noticed that the subject for the mail references 2.6.30.3 and the patch for st in the bugzilla did not make into 2.6.20 and is not in .3. Could we try the st patch in the bugzilla first?

Ok, the st patch from bugzilla solves the problem (tested on both affected machines).

Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
author
#12 lore raw
In reply to Mike Christie
Quoted 26 lines · Mike Christie
Quoted 9 lines · James Bottomley
Quoted 8 lines · Mike Christie
Quoted 4 lines

I can't even say if the tapes are written correctly as I can't read them (one does not reboot production machines back to 2.4.x just to try to read a backup tape - I don't have 2.6.x older than 2.6.20 on these machines).

Could you try this patch
http://marc.info/?l=linux-scsi&m=116464965414878&w=2
I thought st was modified to not send offsets in the last elements but it looks like it wasn't.

Quoted 6 lines

Actually, there are two patches in the email referred to. If the analysis that we're passing NULL to mempool_free is correct, it should be the second one that fixes the problem (the one that checks bio->bi_io_vec before freeing it). Which would mean we have a nr_vecs==0 bio generated by the tar somehow.

I think we might only need the first patch if the problem is similar to what the lsi guys were seeing. I thought the problem is that we are not estimating how large the transfer is correctly because we do not take into account offsets at the end. This results in nr_vecs being zero when it should be a valid value. I thought Kai's patch:
http://bugzilla.kernel.org/show_bug.cgi?id=7919
http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commitdiff;h=9abe16c670bd3d4ab5519257514f9f291383d104
fixed the problem on st's side, but I guess not so you are probably right.

Quoted 4 lines

Here is a patch that dumps the sgl we are getting from st so we can see for sure what we are getting and can decide if we need the first patch, second patch or both.

Here's the patch output:

sg length 6 offset 0
sg length 12 offset 0
sg length 4096 offset 0
sg length 4096 offset 0
sg length 2048 offset 0

Please note (as replied in the other mail) that the bugzilla patch solves the problem.

Quoted 18 lines

------------------------------------------------------------------------

drivers/scsi/scsi_lib.c +4 -0
index 5f95570..81005aa 100644
@@ -306,6 +306,10 @@ static int scsi_req_map_sg(struct reques
 	struct bio *bio = NULL;
 	int i, err, nr_vecs = 0;
 
+	for (i = 0; i < nsegs; i++)
+		printk(KERN_INFO "sg length %u offset %u\n", sgl[i].length,
+			sgl[i].offset);
+
 	for (i = 0; i < nsegs; i++) {
 		page = sgl[i].page;
 		off = sgl[i].offset;
Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]
#15 lore raw
In reply to James Bottomley
Quoted 12 lines · James Bottomley
Quoted 10 lines · Gene Heskett

James, could this also be the cause of a tar based backup going crazy and thinking all data is new under any 2.6.21-rc* kernel I've tested so far with amanda, which in my case uses tar? I've tried the fedora patched tar-1.15-1, and one I hand built right after 1.15-1 came out over a year ago, and they both do it, but only when booted to a 2.6.21-rc* kernel.

This obviously will be a show-stopper, either for amanda (and by inference, any app that uses tar), or for the migration of an amanda users machinery to a 2.6.21 kernel.

Quoted 5 lines

Er, I don't think so .. that sounds like mtime miscompare, which is either a problem with the filesystem or a problem with the way mtime is stored in the tar archive.

James

Well, since the times reported by ls -l --full-time are sane [root@coyote pix]# ls -l --full-time
total 924784
-rw-r--r-- 1 root root 985324 2002-06-09 18:14:54.000000000 -0400
0203.jpg
[... the rest of a 100k listing, booted to 2.6.20.3-rdsl-0.31]

And:
[root@coyote pix]# ls -l --full-time
total 924784
-rw-r--r-- 1 root root 985324 2002-06-09 18:14:54.000000000 -0400
0203.jpg

booted to 2.6.21-rc4

allthough the fractional second is a string of .000000000, even when booted to a tar-unfriendly kernel, then it would tend to point at tar, but two differently built versions of tar have been confirmed as miss-behaving in the presence of a kernel in the 2.6.21 series so far, all of them.

I'm going to reboot twice more tonight, once to verify that the output of an ls -l --full-time is as I said above, I'll save this and do that again and clip it in after a reboot to 2.6.21-rc4, and once to 2.6.20.4-rc1 to see if by chance one of those patches is the guilty party. I'll leave the latter running tonight for the amanda run & see what falls out.

Signature · 6 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Try to relax and enjoy the crisis.
		-- Ashleigh Brilliant
#10 lore raw
In reply to Andreas Steinmetz
Quoted 27 lines · On Tue, 20 Mar 2007 00:25:02 +0100 Andreas Steinmetz <[email protected]> wrote:
Quoted 26 lines · Mike Christie
Quoted 24 lines · Mike Christie
Quoted 9 lines · James Bottomley
Quoted 8 lines · Mike Christie
Quoted 4 lines

I can't even say if the tapes are written correctly as I can't read them (one does not reboot production machines back to 2.4.x just to try to read a backup tape - I don't have 2.6.x older than 2.6.20 on these machines).

Could you try this patch
http://marc.info/?l=linux-scsi&m=116464965414878&w=2
I thought st was modified to not send offsets in the last elements but it looks like it wasn't.

Quoted 6 lines

Actually, there are two patches in the email referred to. If the analysis that we're passing NULL to mempool_free is correct, it should be the second one that fixes the problem (the one that checks bio->bi_io_vec before freeing it). Which would mean we have a nr_vecs==0 bio generated by the tar somehow.

I think we might only need the first patch if the problem is similar to what the lsi guys were seeing. I thought the problem is that we are not estimating how large the transfer is correctly because we do not take into account offsets at the end. This results in nr_vecs being zero when it should be a valid value. I thought Kai's patch:
http://bugzilla.kernel.org/show_bug.cgi?id=7919
http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commitdiff;h=9abe16c670bd3d4ab5519257514f9f291383d104
fixed the problem on st's side,

Quoted 6 lines
Quoted 3 lines

Oh, I noticed that the subject for the mail references 2.6.30.3 and the patch for st in the bugzilla did not make into 2.6.20 and is not in .3. Could we try the st patch in the bugzilla first?

Ok, the st patch from bugzilla solves the problem (tested on both affected machines).

If you're referring to the below patch then it's already in mainline, and has been for a month.

Have you tested 2.6.21-rc4? If not, please do so.

Perhaps we should merge this into 2.6.20.x?

commit 9abe16c670bd3d4ab5519257514f9f291383d104

Author: Kai Makisara <[email protected]>
Date:   Sat Feb 3 13:21:29 2007 +0200

[SCSI] st: fix Tape dies if wrong block size used, bug 7919

On Thu, 1 Feb 2007, Andrew Morton wrote:
> On Thu, 1 Feb 2007 15:34:29 -0800
> [email protected] wrote:
>
> > http://bugzilla.kernel.org/show_bug.cgi?id=7919
> >
> > Summary: Tape dies if wrong block size used
> > Kernel Version: 2.6.20-rc5
> > Status: NEW
> > Severity: normal
> > Owner: [email protected]
> > Submitter: [email protected]
> >
> >
> > Most recent kernel where this bug did *NOT* occur: 2.6.17.14
> >
> > Other Kernels Tested and Results:
> >
> > OK 2.6.15.7
> > OK 2.6.16.37
> > OK 2.6.17.14
> > BAD 2.6.18.6
> > BAD 2.6.18-1.2869.fc6
> > BAD 2.6.19.2 +
> > BAD 2.6.20-rc5
> >
> > NOTE: 2.6.18-1.2869.fc6 is a Fedora modified kernel, all others are from kernel.org > >
...
> > Steps to reproduce:
> > Get a Adaptec AHA-2940U/UW/D / AIC-7881U card and a tape drive,
> > install a recent kernel
> > set the tape block size - mt setblk 4096
> > read from or write to tape using wrong block size - tar -b 7 -cvf /dev/tape foo > >
Write does not trigger this bug because the driver refuses in fixed block mode writes that are not a multiple of the block size. Read does trigger it in my system.

The bug is not associated with any specific HBA. st tries to do direct i/o in fixed block mode with reads that are not a multiple of tape block size.

The patch in this message fixes the st problem by switching to using the driver buffer up to the next close of the device file in fixed block mode if the user asks for a read like this.

I don't know why the bug has surfaced only after 2.6.17 although the st problem is old. There may be another bug in the block subsystem and this patch works around it. However, the patch fixes a problem in st and in this way it is a valid fix.

This patch may also fix the bug 7900.

The patch compiles and is lightly tested.

Signed-off-by Kai Makisara <[email protected]>
Signed-off-by James Bottomley <[email protected]>
drivers/scsi/st.c +13 -10
index e016e09..fba8b20 100644
@@ -9,7 +9,7 @@
    Steve Hirsch, Andreas Koppenh"ofer, Michael Leodolter, Eyal Lebedinsky,
    Michael Schaefer, J"org Weule, and Eric Youngdale.
 
-   Copyright 1992 - 2006 Kai Makisara
+   Copyright 1992 - 2007 Kai Makisara
    email [email protected]
 
    Some small formal changes - aeb, 950809

@@ -17,7 +17,7 @@
    Last modified: 18-JAN-1998 Richard Gooch <[email protected]> Devfs support
  */
 
-static const char *verstr = "20061107";
+static const char *verstr = "20070203";
 
 #include <linux/module.h>
 

@@ -1168,6 +1168,7 @@ static int st_open(struct inode *inode, 
 		STps = &(STp->ps[i]);
 		STps->rw = ST_IDLE;
 	}
+	STp->try_dio_now = STp->try_dio;
 	STp->recover_count = 0;
 	DEB( STp->nbr_waits = STp->nbr_finished = 0;
 	     STp->nbr_requests = STp->nbr_dio = STp->nbr_pages = STp->nbr_combinable = 0; )

@@ -1400,9 +1401,9 @@ static int setup_buffering(struct scsi_t
 	struct st_buffer *STbp = STp->buffer;
 
 	if (is_read)
-		i = STp->try_dio && try_rdio;
+		i = STp->try_dio_now && try_rdio;
 	else
-		i = STp->try_dio && try_wdio;
+		i = STp->try_dio_now && try_wdio;
 
 	if (i && ((unsigned long)buf & queue_dma_alignment(
 					STp->device->request_queue)) == 0) {

@@ -1599,7 +1600,7 @@ st_write(struct file *filp, const char _
 			STm->do_async_writes && STps->eof < ST_EOM_OK;
 
 		if (STp->block_size != 0 && STm->do_buffer_writes &&
-		    !(STp->try_dio && try_wdio) && STps->eof < ST_EOM_OK &&
+		    !(STp->try_dio_now && try_wdio) && STps->eof < ST_EOM_OK &&
 		    STbp->buffer_bytes < STbp->buffer_size) {
 			STp->dirty = 1;
 			/* Don't write a buffer that is not full enough. */

@@ -1769,7 +1770,7 @@ static long read_tape(struct scsi_tape *
 	if (STp->block_size == 0)
 		blks = bytes = count;
 	else {
-		if (!(STp->try_dio && try_rdio) && STm->do_read_ahead) {
+		if (!(STp->try_dio_now && try_rdio) && STm->do_read_ahead) {
 			blks = (STp->buffer)->buffer_blocks;
 			bytes = blks * STp->block_size;
 		} else {

@@ -1948,10 +1949,12 @@ st_read(struct file *filp, char __user *
 		goto out;
 
 	STm = &(STp->modes[STp->current_mode]);
-	if (!(STm->do_read_ahead) && STp->block_size != 0 &&
-	    (count % STp->block_size) != 0) {
-		retval = (-EINVAL);	/* Read must be integral number of blocks */
-		goto out;
+	if (STp->block_size != 0 && (count % STp->block_size) != 0) {
+		if (!STm->do_read_ahead) {
+			retval = (-EINVAL);	/* Read must be integral number of blocks */
+			goto out;
+		}
+		STp->try_dio_now = 0;  /* Direct i/o can't handle split blocks */
 	}
 
 	STps = &(STp->ps[STp->partition]);
drivers/scsi/st.h +2 -1
index 05a5cae..50f3deb 100644
@@ -117,7 +117,8 @@ struct scsi_tape {
 	unsigned char cln_sense_value;
 	unsigned char cln_sense_mask;
 	unsigned char use_pf;			/* Set Page Format bit in all mode selects? */
-	unsigned char try_dio;			/* try direct i/o? */
+	unsigned char try_dio;			/* try direct i/o in general? */
+	unsigned char try_dio_now;		/* try direct i/o before next close? */
 	unsigned char c_algo;			/* compression algorithm */
 	unsigned char pos_unknown;			/* after reset position unknown */
 	int tape_type;
author
#11 lore raw
Quoted 32 lines · Andrew Morton
Quoted 29 lines · On Tue, 20 Mar 2007 00:25:02 +0100 Andreas Steinmetz <[email protected]> wrote:
Quoted 28 lines · Mike Christie
Quoted 16 lines · Mike Christie
Quoted 9 lines · James Bottomley
Quoted 8 lines · Mike Christie
Quoted 4 lines

I can't even say if the tapes are written correctly as I can't read them (one does not reboot production machines back to 2.4.x just to try to read a backup tape - I don't have 2.6.x older than 2.6.20 on these machines).

Could you try this patch
http://marc.info/?l=linux-scsi&m=116464965414878&w=2
I thought st was modified to not send offsets in the last elements but it looks like it wasn't.

Quoted 6 lines

Actually, there are two patches in the email referred to. If the analysis that we're passing NULL to mempool_free is correct, it should be the second one that fixes the problem (the one that checks bio->bi_io_vec before freeing it). Which would mean we have a nr_vecs==0 bio generated by the tar somehow.

Quoted 8 lines

I think we might only need the first patch if the problem is similar to what the lsi guys were seeing. I thought the problem is that we are not estimating how large the transfer is correctly because we do not take into account offsets at the end. This results in nr_vecs being zero when it should be a valid value. I thought Kai's patch:
http://bugzilla.kernel.org/show_bug.cgi?id=7919
http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commitdiff;h=9abe16c670bd3d4ab5519257514f9f291383d104
fixed the problem on st's side,

Oh, I noticed that the subject for the mail references 2.6.30.3 and the patch for st in the bugzilla did not make into 2.6.20 and is not in .3. Could we try the st patch in the bugzilla first?

Quoted 7 lines
Quoted 2 lines

Ok, the st patch from bugzilla solves the problem (tested on both affected machines).

If you're referring to the below patch then it's already in mainline, and has been for a month.

Yes, that's the patch I'm referring to.

Quoted 2 lines

Have you tested 2.6.21-rc4? If not, please do so.

Sorry, this is not possible on these machines. They are production servers and every problem on them that cannot be easily solved via remote access is a 40km (one way) drive in the middle of the night.

Quoted 2 lines

Perhaps we should merge this into 2.6.20.x?

I would suggest so.

Quoted 158 lines

commit 9abe16c670bd3d4ab5519257514f9f291383d104

Author: Kai Makisara <[email protected]>
Date:   Sat Feb 3 13:21:29 2007 +0200

[SCSI] st: fix Tape dies if wrong block size used, bug 7919

On Thu, 1 Feb 2007, Andrew Morton wrote:
> On Thu, 1 Feb 2007 15:34:29 -0800
> [email protected] wrote:
>
> > http://bugzilla.kernel.org/show_bug.cgi?id=7919
> >
> > Summary: Tape dies if wrong block size used
> > Kernel Version: 2.6.20-rc5
> > Status: NEW
> > Severity: normal
> > Owner: [email protected]
> > Submitter: [email protected]
> >
> >
> > Most recent kernel where this bug did *NOT* occur: 2.6.17.14
> >
> > Other Kernels Tested and Results:
> >
> > OK 2.6.15.7
> > OK 2.6.16.37
> > OK 2.6.17.14
> > BAD 2.6.18.6
> > BAD 2.6.18-1.2869.fc6
> > BAD 2.6.19.2 +
> > BAD 2.6.20-rc5
> >
> > NOTE: 2.6.18-1.2869.fc6 is a Fedora modified kernel, all others are from kernel.org > >
...
> > Steps to reproduce:
> > Get a Adaptec AHA-2940U/UW/D / AIC-7881U card and a tape drive,
> > install a recent kernel
> > set the tape block size - mt setblk 4096
> > read from or write to tape using wrong block size - tar -b 7 -cvf /dev/tape foo > >
Write does not trigger this bug because the driver refuses in fixed block mode writes that are not a multiple of the block size. Read does trigger it in my system.

The bug is not associated with any specific HBA. st tries to do direct i/o in fixed block mode with reads that are not a multiple of tape block size.

The patch in this message fixes the st problem by switching to using the driver buffer up to the next close of the device file in fixed block mode if the user asks for a read like this.

I don't know why the bug has surfaced only after 2.6.17 although the st problem is old. There may be another bug in the block subsystem and this patch works around it. However, the patch fixes a problem in st and in this way it is a valid fix.

This patch may also fix the bug 7900.

The patch compiles and is lightly tested.

Signed-off-by Kai Makisara <[email protected]>
Signed-off-by James Bottomley <[email protected]>
drivers/scsi/st.c +13 -10
index e016e09..fba8b20 100644
@@ -9,7 +9,7 @@
    Steve Hirsch, Andreas Koppenh"ofer, Michael Leodolter, Eyal Lebedinsky,
    Michael Schaefer, J"org Weule, and Eric Youngdale.
 
-   Copyright 1992 - 2006 Kai Makisara
+   Copyright 1992 - 2007 Kai Makisara
    email [email protected]
 
    Some small formal changes - aeb, 950809

@@ -17,7 +17,7 @@
    Last modified: 18-JAN-1998 Richard Gooch <[email protected]> Devfs support
  */
 
-static const char *verstr = "20061107";
+static const char *verstr = "20070203";
 
 #include <linux/module.h>
 

@@ -1168,6 +1168,7 @@ static int st_open(struct inode *inode, 
 		STps = &(STp->ps[i]);
 		STps->rw = ST_IDLE;
 	}
+	STp->try_dio_now = STp->try_dio;
 	STp->recover_count = 0;
 	DEB( STp->nbr_waits = STp->nbr_finished = 0;
 	     STp->nbr_requests = STp->nbr_dio = STp->nbr_pages = STp->nbr_combinable = 0; )

@@ -1400,9 +1401,9 @@ static int setup_buffering(struct scsi_t
 	struct st_buffer *STbp = STp->buffer;
 
 	if (is_read)
-		i = STp->try_dio && try_rdio;
+		i = STp->try_dio_now && try_rdio;
 	else
-		i = STp->try_dio && try_wdio;
+		i = STp->try_dio_now && try_wdio;
 
 	if (i && ((unsigned long)buf & queue_dma_alignment(
 					STp->device->request_queue)) == 0) {

@@ -1599,7 +1600,7 @@ st_write(struct file *filp, const char _
 			STm->do_async_writes && STps->eof < ST_EOM_OK;
 
 		if (STp->block_size != 0 && STm->do_buffer_writes &&
-		    !(STp->try_dio && try_wdio) && STps->eof < ST_EOM_OK &&
+		    !(STp->try_dio_now && try_wdio) && STps->eof < ST_EOM_OK &&
 		    STbp->buffer_bytes < STbp->buffer_size) {
 			STp->dirty = 1;
 			/* Don't write a buffer that is not full enough. */

@@ -1769,7 +1770,7 @@ static long read_tape(struct scsi_tape *
 	if (STp->block_size == 0)
 		blks = bytes = count;
 	else {
-		if (!(STp->try_dio && try_rdio) && STm->do_read_ahead) {
+		if (!(STp->try_dio_now && try_rdio) && STm->do_read_ahead) {
 			blks = (STp->buffer)->buffer_blocks;
 			bytes = blks * STp->block_size;
 		} else {

@@ -1948,10 +1949,12 @@ st_read(struct file *filp, char __user *
 		goto out;
 
 	STm = &(STp->modes[STp->current_mode]);
-	if (!(STm->do_read_ahead) && STp->block_size != 0 &&
-	    (count % STp->block_size) != 0) {
-		retval = (-EINVAL);	/* Read must be integral number of blocks */
-		goto out;
+	if (STp->block_size != 0 && (count % STp->block_size) != 0) {
+		if (!STm->do_read_ahead) {
+			retval = (-EINVAL);	/* Read must be integral number of blocks */
+			goto out;
+		}
+		STp->try_dio_now = 0;  /* Direct i/o can't handle split blocks */
 	}
 
 	STps = &(STp->ps[STp->partition]);
drivers/scsi/st.h +2 -1
index 05a5cae..50f3deb 100644
@@ -117,7 +117,8 @@ struct scsi_tape {
 	unsigned char cln_sense_value;
 	unsigned char cln_sense_mask;
 	unsigned char use_pf;			/* Set Page Format bit in all mode selects? */
-	unsigned char try_dio;			/* try direct i/o? */
+	unsigned char try_dio;			/* try direct i/o in general? */
+	unsigned char try_dio_now;		/* try direct i/o before next close? */
 	unsigned char c_algo;			/* compression algorithm */
 	unsigned char pos_unknown;			/* after reset position unknown */
 	int tape_type;
Signature · 1 line
Andreas Steinmetz                       SPAMmers use [email protected]