Skip to contentSkip to conversation

BUG() in asm/pci.h:142 with 2.4.13

DISCUSSION Christian Hammers 5 participants 14 messages Oct 25, 2001 – Oct 30, 2001

lore
Share & tools
lore
#0 lore raw

Hello

My system crashed several times now with 2.4.11-pre6 and 2.4.13 (pre6 because it was the first one I got that fixed some 2GB RAM memory allocation bug).

2.4.13 was the easiest one to reproduce: when starting the tape backup to a HP DDS3/DAT Streamer (C1537A) via a Adaptec SCSI Controller (Adaptec 7892A in /proc/pci) on a Gigabyte GA-6VTXD Dual Motherboard with two PIII and 2GB of RAM it crashed immediately with the error attached below. The machine was under "stresstest-simulation" load at this time.

The tape_backup.pl uses the "mt" and "cpio" commands to access /dev/nst0.

Maybe worth noting is, that the system crashed another time yesterday after replacing the external SCSI RAID Chassis/Controller (not the disks in it) and just this moment with another message (see below).

Any help or hints appreciated!
[please keep me Cc'ed as I'm not subscribed to this list]

bye,

-christian-

kernel: kernel BUG at /usr/local/src/kernel/linux-2.4.13/include/asm/pci.h:142!
kernel: invalid operand: 0000
kernel: CPU:    1
kernel: EIP:    0010:[ahc_linux_run_device_queue+899/2144]    Not tainted
kernel: EFLAGS: 00010082
kernel: eax: 00000048   ebx: f7bb5650   ecx: c0275a88   edx: 00010071
kernel: esi: c5915a30   edi: 00000000   ebp: c5915a30   esp: e9ae3e14
kernel: ds: 0018   es: 0018   ss: 0018
kernel: Process tape_backup.pl (pid: 4366, stackpage=e9ae3000)
kernel: Stack: c024e100 0000008e f7bbec00 e9ae3e6c 00000000 00000000 f5358de0 0000000e 
kernel:        f7bbec10 00000007 00000007 401af000 41ffffff 00000004 c5915600 c01b0e09  
kernel:        f7bbec00 c301fee0 00000202 d35ce1d4 c5915600 f7bbfa20 00000096 c01a5f76     
kernel: Call Trace: [ahc_linux_queue+361/424] [scsi_dispatch_cmd+354/632] [scsi_done+0/200] [scsi_request_fn+752/820] [__scsi_insert_special+110/128]  
kernel:    [scsi_insert_special_req+26/32] [scsi_do_req+284/324] [<f8a8940b>] [<f8a89240>] [<f8a8aad1>] [sys_write+143/196] 
kernel:    [system_call+51/56] 

kernel:
kernel: Code: 0f 0b eb 18 90 83 7e 04 00 75 14 68 90 00 00 00 68 00 e1 24

#
# The output from the other SCSI crash. This came from remote syslogging

# and console.
#

kernel: scsi0:0:0:0: Attempting to queue an ABORT message
kernel: (scsi0:A:0:0): Queuing a recovery SCB
kernel: scsi0:0:0:0: Device is disconnected, re-queuing SCB
kernel: Recovery code sleeping
kernel: (scsi0:A:0:0): Abort Tag Message Sent
kernel: (scsi0:A:0:0): SCB 153 - Abort Tag Completed.
kernel: Recovery SCB completes
kernel: Recovery code awake
kernel: aic7xxx_abort returns 8194
kernel: scsi0:0:0:0: Attempting to queue an ABORT message

Some more debugging help:

mtv-server:/usr/local/src/kernel/linux-2.4.13/include/asm# lspci    
00:00.0 Host bridge: VIA Technologies, Inc. VT82C691 [Apollo PRO] (rev c4)
00:01.0 PCI bridge: VIA Technologies, Inc. VT82C598 [Apollo MVP3 AGP]
00:07.0 ISA bridge: VIA Technologies, Inc. VT82C686 [Apollo Super] (rev 40)
00:07.1 IDE interface: VIA Technologies, Inc. VT82C586 IDE [Apollo] (rev
06)
00:07.2 USB Controller: VIA Technologies, Inc. VT82C586B USB (rev 1a)
00:07.3 USB Controller: VIA Technologies, Inc. VT82C586B USB (rev 1a)
00:07.4 SMBus: VIA Technologies, Inc. VT82C686 [Apollo Super ACPI] (rev 40)
00:0a.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone]
(rev 30)
00:0c.0 SCSI storage controller: Adaptec 7892A (rev 02)
01:00.0 VGA compatible controller: ATI Technologies Inc Rage XL AGP (rev
27)
mtv-server:~$ cat /proc/scsi/scsi 
Attached devices: 
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: easyRAID Model:  U3              Rev: 0001
  Type:   Direct-Access                    ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 02 Lun: 00
  Vendor: HP       Model: C1537A           Rev: L708
  Type:   Sequential-Access                ANSI SCSI revision: 02
Signature · 3 lines
Christian Hammers    WESTEND GmbH - Aachen und Dueren     Tel 0241/701333-0
[email protected]     Internet & Security for Professionals    Fax 0241/911879
           WESTEND ist CISCO Systems Partner - Premium Certified
#2 lore raw
In reply to Christian Hammers
Quoted 12 lines · Christian Hammers

Hello

My system crashed several times now with 2.4.11-pre6 and 2.4.13 (pre6 because it was the first one I got that fixed some 2GB RAM memory allocation bug).

2.4.13 was the easiest one to reproduce: when starting the tape backup to a HP DDS3/DAT Streamer (C1537A) via a Adaptec SCSI Controller (Adaptec 7892A in /proc/pci) on a Gigabyte GA-6VTXD Dual Motherboard with two PIII and 2GB of RAM it crashed immediately with the error attached below. The machine was under "stresstest-simulation" load at this time.

Quoted 5 lines

The tape_backup.pl uses the "mt" and "cpio" commands to access /dev/nst0.

Maybe worth noting is, that the system crashed another time yesterday after replacing the external SCSI RAID Chassis/Controller (not the disks in it) and just this moment with another message (see below).

Could you try this patch and see if it fixes the pci.h BUG at least?

Signature · 1 line
Jens Axboe
scsi/scsi_merge.c renamed from scsi/scsi_merge.c~ +1 -0
@@ -943,6 +943,7 @@
 		}
 		count++;
 		sgpnt[count - 1].address = bh->b_data;
+		sgpnt[count - 1].page = NULL;
 		sgpnt[count - 1].length += bh->b_size;
 		if (!dma_host) {
 			SCpnt->request_bufflen += bh->b_size;
#7 lore raw
In reply to Christian Hammers

Hello

Quoted 7 lines · Jens Axboe
Quoted 5 lines

2.4.13 was the easiest one to reproduce: when starting the tape backup to a HP DDS3/DAT Streamer (C1537A) via a Adaptec SCSI Controller (Adaptec 7892A in /proc/pci) on a Gigabyte GA-6VTXD Dual Motherboard with two PIII and 2GB of RAM it crashed immediately with the error attached below. The machine was under "stresstest-simulation" load at this time.

Could you try this patch and see if it fixes the pci.h BUG at least?

This patch did not prevent the crash. Again immediately after rewinding the tape when it began to write. I'll try now the 2.4.12-ac6... and it works.

Quoted 1 line

Jens Axboe

bye,

-christian- (happy about Alan having forked the kernel tree once ago..)

Signature · 3 lines
Christian Hammers    WESTEND GmbH - Aachen und Dueren     Tel 0241/701333-0
[email protected]     Internet & Security for Professionals    Fax 0241/911879
           WESTEND ist CISCO Systems Partner - Premium Certified
#8 lore raw
In reply to Christian Hammers
Quoted 9 lines · Christian Hammers

Hello

Quoted 5 lines · Jens Axboe
Quoted 5 lines

2.4.13 was the easiest one to reproduce: when starting the tape backup to a HP DDS3/DAT Streamer (C1537A) via a Adaptec SCSI Controller (Adaptec 7892A in /proc/pci) on a Gigabyte GA-6VTXD Dual Motherboard with two PIII and 2GB of RAM it crashed immediately with the error attached below. The machine was under "stresstest-simulation" load at this time.

Quoted 3 lines
Quoted 1 line

Could you try this patch and see if it fixes the pci.h BUG at least?

This patch did not prevent the crash. Again immediately after rewinding the tape when it began to write. I'll try now the 2.4.12-ac6... and it works.

Ok, someone else is meddling with the scatterlist then. I'll take a 2nd look.

Signature · 1 line
Jens Axboe
#11 lore raw
In reply to Christian Hammers

Hello

Quoted 4 lines · Jens Axboe
Quoted 2 lines

This patch did not prevent the crash. Again immediately after rewinding the tape when it began to write. I'll try now the 2.4.12-ac6... and it works.

Ok, someone else is meddling with the scatterlist then. I'll take a 2nd look.

The 2.4.12-ac6 crashed, too, when I killed the dd and cpio processes with SIGKILL. I got the extra scsi queue debug information on the console but it was too much to write down. I now have a serial connection to another computer and did "ln /dev/ttyS1 /dev/console" in the hope to be able to save you all kernel output.

-christian-

Signature · 3 lines
Christian Hammers    WESTEND GmbH - Aachen und Dueren     Tel 0241/701333-0
[email protected]     Internet & Security for Professionals    Fax 0241/911879
           WESTEND ist CISCO Systems Partner - Premium Certified
#12 lore raw
In reply to Christian Hammers

Hello

Now it crashed again when writing the tape but after writing 1GB to it. This time I could capture the output of the extra new-queue debugging output that I enabled in the kernel configuration (this time 2.4.12-ac6).

For the linux-scsi guys: if you need more information please take a look at my previous posts to linux-kernel (same subject) or contact me directly.

bye,

-christian-

Quoted 5 lines · Christian Hammers

2.4.13 was the easiest one to reproduce: when starting the tape backup to a HP DDS3/DAT Streamer (C1537A) via a Adaptec SCSI Controller (Adaptec 7892A in /proc/pci) on a Gigabyte GA-6VTXD Dual Motherboard with two PIII and 2GB of RAM it crashed immediately with the error attached below. The machine was under "stresstest-simulation" load at this time.

#

# console dump via minicom and serial line
#
 /USR/SBIN/CRON[10591]: (root) CMD
(/usr/local/maint/watchdog)
 kernel: scsi0:0:0:0: Attempting to queue an
ABORT message
 kernel: scsi0: Dumping Card State while idle, at
SEQADDR 0x8
 kernel: ACCUM = 0x0, SINDEX = 0x20, DINDEX =
0xe4, ARG_2 = 0x0
 kernel: HCNT = 0x0
 kernel: SCSISEQ = 0x12, SBLKCTL = 0xa
 kernel:  DFCNTRL = 0x0, DFSTATUS = 0x89
 kernel: LASTPHASE = 0x1, SCSISIGI = 0x0,
SXFRCTL0 = 0x80
 kernel: SSTAT0 = 0x0, SSTAT1 = 0x8
 kernel: SCSIPHASE = 0x0
 kernel: STACK == 0x3, 0x108, 0x160, 0x0
 kernel: SCB count = 248
 kernel: Kernel NEXTQSCB = 62
 kernel: Card NEXTQSCB = 62
 kernel: QINFIFO entries: 
 kernel: Waiting Queue entries: 
 kernel: Disconnected Queue entries: 6:150 12:210
2:178 29:28 23:181 13:61 24:7 
 kernel: QOUTFIFO entries: 
 kernel: Sequencer Free SCB List: 22 26 9 4 11 19
10 16 28 1 8 5 27 7 20 31 0 30 
25 17 21 3 14 15 18 
 kernel: Pending list: 150, 210, 178, 28, 181,
61, 7
 kernel: Kernel Free SCB list: 32 77 79 149 198
171 223 152 140 105 189 151 78 66
 199 88 6 224 138 177 67 84 194 191 23 246 215 24 160 185 225 230 93 174 49
241 110 2 20 147 170 240 33 59 
243 99 54 175 19 176 18 192 76 100 190 238 108 8 159 208 207 60 242 217 56
221 1 17 213 92 127 70 162 74 19
7 142 239 196 82 124 29 235 134 232 123 179 218 139 211 117 3 119 57 219
125 122 209 101 44 155 45 39 212 1
28 233 202 158 91 187 46 0 180 182 201 109 118 228 131 12 4 112 229 200 236
173 132 247 97 186 148 55 216 1
33 144 113 231 30 63 37 137 206 156 83 146 135 141 161 64 165 98 35 234 166
81 9 10 214 43 58 111 71 115 10
6 85 183 72 11 204 172 157 130 47 154 188 226 90 220 96 107 27 227 145 40
87 22 94 129 48 205 65 120 73 163
 69 26 41 86 103 68 169 53 5 237 167 42 51 195 15 38 80 13 168 21 89 52 16
114 50 193 36 136 75 25 34 95 14
 153 203 126 222 116 31 143 104 164 121 102 184 245 244 
 kernel: DevQ(0:0:0): 0 waiting
 kernel: DevQ(0:2:0): 0 waiting
 kernel: (scsi0:A:0:0): Queuing a recovery SCB
 kernel: scsi0:0:0:0: Device is disconnected,
re-queuing SCB
 kernel: Recovery code sleeping
 kernel: (scsi0:A:0:0): Abort Tag Message Sent
 kernel: (scsi0:A:0:0): SCB 7 - Abort Tag
Completed.
 kernel: Recovery SCB completes
 kernel: Recovery code awake
 kernel: aic7xxx_abort returns 0x2002
 sendmail[349]: rejecting connections on daemon
MTA: load average: 83
<80>xüxÀ<80>xÀ<80>x<ð<80>xx<xþ<80><80><80>øx<øxÀ<80>xü<80>x<ðøxüøxÀøx<80><80><80><80><80>xÀøø<80><80><80>
Signature · 3 lines
Christian Hammers    WESTEND GmbH - Aachen und Dueren     Tel 0241/701333-0
[email protected]     Internet & Security for Professionals    Fax 0241/911879
           WESTEND ist CISCO Systems Partner - Premium Certified
#9 lore raw

SCSI tape crashes (was Re: BUG() in asm/pci.h:142 with 2.4.13)

In reply to Jens Axboe

From: Jens Axboe <[email protected]>
Date: Thu, 25 Oct 2001 19:32:48 +0200

Quoted 2 lines · Christian Hammers

This patch did not prevent the crash. Again immediately after rewinding the tape when it began to write. I'll try now the 2.4.12-ac6... and it works.

Ok, someone else is meddling with the scatterlist then. I'll take a 2nd look.

Can people try out this patch? I believe this will fix the bug.

scsi/st.c renamed from scsi/st.c.~1~ +3 -0
@@ -3233,6 +3233,7 @@
 				break;
 			}
 		}
+		tb->sg[0].page = NULL;
 		if (tb->sg[segs].address == NULL) {
 			kfree(tb);
 			tb = NULL;
@@ -3264,6 +3265,7 @@
 					tb = NULL;
 					break;
 				}
+				tb->sg[segs].page = NULL;
 				tb->sg[segs].length = b_size;
 				got += b_size;
 				segs++;
@@ -3337,6 +3339,7 @@
 			normalize_buffer(STbuffer);
 			return FALSE;
 		}
+		STbuffer->sg[segs].page = NULL;
 		STbuffer->sg[segs].length = b_size;
 		STbuffer->sg_segs += 1;
 		got += b_size;
#3 lore raw

Re: SCSI tape crashes (was Re: BUG() in asm/pci.h:142 with 2.4.13)

In reply to Jens Axboe

David,

Is this waht's causing the earlier bug I reported in 2.4.10? If so where is this patch so I can see if it fixes the problem.

Thanks,

Jeff

Quoted 43 lines · David S. Miller

From: Jens Axboe <[email protected]>
Date: Thu, 25 Oct 2001 19:32:48 +0200

Quoted 2 lines · Christian Hammers

This patch did not prevent the crash. Again immediately after rewinding the tape when it began to write. I'll try now the 2.4.12-ac6... and it works.

Ok, someone else is meddling with the scatterlist then. I'll take a 2nd look.

Can people try out this patch? I believe this will fix the bug.

scsi/st.c renamed from scsi/st.c.~1~ +3 -1
@@ -3233,6 +3233,7 @@
 				break;
 			}
 		}
+		tb->sg[0].page = NULL;
 		if (tb->sg[segs].address == NULL) {
 			kfree(tb);
 			tb = NULL;
@@ -3264,6 +3265,7 @@
 					tb = NULL;
 					break;
 				}
+				tb->sg[segs].page = NULL;
 				tb->sg[segs].length = b_size;
 				got += b_size;
 				segs++;
@@ -3337,6 +3339,7 @@
 			normalize_buffer(STbuffer);
 			return FALSE;
 		}
+		STbuffer->sg[segs].page = NULL;
 		STbuffer->sg[segs].length = b_size;
 		STbuffer->sg_segs += 1;
 		got += b_size;
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#4 lore raw

Re: SCSI tape crashes

From: "Jeff V. Merkey" <[email protected]>
Date: Thu, 25 Oct 2001 19:26:48 -0700

Is this waht's causing the earlier bug I reported in 2.4.10? If so where is this patch so I can see if it fixes the problem.

The patch was in the email, can't you read? :-)
(You even quoted the patch in your reply!)

Anyways, my patch isn't relevant to your problem since the bug I am fixing only can exist in 2.4.13 and later kernels. Sorry.

scsi/st.c

Quoted 4 lines · David S. Miller

Can people try out this patch? I believe this will fix the bug.

scsi/st.c renamed from scsi/st.c.~1~ +0 -0

Franks a lot,
David S. Miller
[email protected]

#5 lore raw

Re: SCSI tape crashes (was Re: BUG() in asm/pci.h:142 with 2.4.13)

In reply to Jens Axboe

Hi

Some addition: the kernel (at least the 2.4.11-pre6 worked well with the tape streamer before the day I replaced the external RAID chassis (broken display) with a new one. My personal guess in this case is that the new RAID has a different firmware and maybe a bug that triggers the crash- condition whenever a second device (here the scsi tape) tries to use the bus, too.

Would this scenario fit into your idea of the bug?

bye,

-christian-

Quoted 8 lines · Jeff V. Merkey
Quoted 8 lines

Ok, someone else is meddling with the scatterlist then. I'll take a 2nd look.

Can people try out this patch? I believe this will fix the bug.

scsi/st.c renamed from scsi/st.c.~1~ +0 -0
@@ -3233,6 +3233,7 @@
Signature · 3 lines
Christian Hammers    WESTEND GmbH - Aachen und Dueren     Tel 0241/701333-0
[email protected]     Internet & Security for Professionals    Fax 0241/911879
           WESTEND ist CISCO Systems Partner - Premium Certified
#10 lore raw

Re: SCSI tape crashes

In reply to Jens Axboe

David,

Thanks. I'll wait to hear from Kai since it looks like something related to his code. I did find the patch, but was using mutt, and you were correct, it was attached at the end of the email.

:-)

Jeff

Quoted 23 lines · David S. Miller

From: "Jeff V. Merkey" <[email protected]>
Date: Thu, 25 Oct 2001 19:26:48 -0700

Is this waht's causing the earlier bug I reported in 2.4.10? If so where is this patch so I can see if it fixes the problem.

The patch was in the email, can't you read? :-)
(You even quoted the patch in your reply!)

Anyways, my patch isn't relevant to your problem since the bug I am fixing only can exist in 2.4.13 and later kernels. Sorry.

   On Thu, Oct 25, 2001 at 05:25:41PM -0700, David S. Miller wrote:
   > Can people try out this patch?  I believe this will fix the bug.
   > 
   > --- drivers/scsi/st.c.~1~	Sun Oct 21 02:47:53 2001
   > +++ drivers/scsi/st.c	Thu Oct 25 17:23:45 2001

Franks a lot,
David S. Miller
[email protected]

#6 lore raw

Re: SCSI tape crashes (was Re: BUG() in asm/pci.h:142 with 2.4.13)

In reply to Jens Axboe
Quoted 9 lines · David S. Miller

From: Jens Axboe <[email protected]>
Date: Thu, 25 Oct 2001 19:32:48 +0200

Quoted 4 lines · Christian Hammers

This patch did not prevent the crash. Again immediately after rewinding the
tape when it began to write. I'll try now the 2.4.12-ac6... and it works.

Quoted 4 lines

Ok, someone else is meddling with the scatterlist then. I'll take a 2nd look.

Can people try out this patch? I believe this will fix the bug.

Yessiree, that fixed the scsi tape lockups we had in 2.4.13.

Many thanks,

--Pete [email protected]

Quoted 26 lines
scsi/st.c renamed from scsi/st.c.~1~ +3 -0
@@ -3233,6 +3233,7 @@
 				break;
 			}
 		}
+		tb->sg[0].page = NULL;
 		if (tb->sg[segs].address == NULL) {
 			kfree(tb);
 			tb = NULL;
@@ -3264,6 +3265,7 @@
 					tb = NULL;
 					break;
 				}
+				tb->sg[segs].page = NULL;
 				tb->sg[segs].length = b_size;
 				got += b_size;
 				segs++;
@@ -3337,6 +3339,7 @@
 			normalize_buffer(STbuffer);
 			return FALSE;
 		}
+		STbuffer->sg[segs].page = NULL;
 		STbuffer->sg[segs].length = b_size;
 		STbuffer->sg_segs += 1;
 		got += b_size;
#13 lore raw

Re: BUG() in asm/pci.h:142 with 2.4.13 (cause found!)

In reply to Christian Hammers

Hello

The cause for my problems with crashing kernels when accessing the tape drive were differences between the original external RAID that belongs to the machine and a temporarily, nearly equal, RAID that we attached while the original was send away for repair.
The temp. RAID was Ultra3 160 and /proc/scsi/aic7xxx/0 showed me a cur. speed of 160 while the "old" and working one only did Ultra2 with 80Mbit/s.

I don't know if it's a hardware incompatibility or if the Linux kernel drivers cannot handle this specific case.

The problem exists in 2.4.11-pre6, 2.4.13, 2.4.12-ac6 and 2.4.13 with patched from axboa(?) and D. Miller.

bye,

-christian-

On Thu, Oct 25, 2001 at 12:07:01PM +0200, Christian Hammers wrote:
...

Quoted 5 lines

2.4.13 was the easiest one to reproduce: when starting the tape backup to a HP DDS3/DAT Streamer (C1537A) via a Adaptec SCSI Controller (Adaptec 7892A in /proc/pci) on a Gigabyte GA-6VTXD Dual Motherboard with two PIII and 2GB of RAM it crashed immediately with the error attached below. The machine was under "stresstest-simulation" load at this time.

...

Quoted 1 line

kernel: kernel BUG at /usr/local/src/kernel/linux-2.4.13/include/asm/pci.h:142!

...

Quoted 3 lines

kernel: scsi0:0:0:0: Attempting to queue an ABORT message
kernel: (scsi0:A:0:0): Queuing a recovery SCB
kernel: scsi0:0:0:0: Device is disconnected, re-queuing SCB

...

Signature · 3 lines
Christian Hammers    WESTEND GmbH - Aachen und Dueren     Tel 0241/701333-0
[email protected]     Internet & Security for Professionals    Fax 0241/911879
           WESTEND ist CISCO Systems Partner - Premium Certified