Skip to contentSkip to conversation

Slab corruption in 2.6.16-rc5-mm2

PATCH In review Jesper Juhl 11 participants 49 messages Mar 6, 2006 – Mar 9, 2006

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Just found the following in dmesg :

Slab corruption: start=f72948a0, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270) 000: 70 00 05 00 00 00 00 0a 00 00 00 00 24 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f7294854, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c0173923>](real_lookup+0x93/0xe0)
000: 6c 69 62 6b 64 65 69 6e 69 74 5f 6b 69 6f 5f 68 010: 74 74 70 5f 63 61 63 68 65 5f 63 6c 65 61 6e 65 Next obj: start=f72948ec, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c01c7c80>](ext3_init_block_alloc_info+0x20/0x70) 000: 10 cf ce f7 00 00 00 00 00 00 00 00 00 00 00 00 010: 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Slab corruption: start=f70aeab4, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270) 000: 70 00 02 00 00 00 00 0a 00 00 00 00 3a 01 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f70aea68, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c023d6df>](init_dev+0x5cf/0x630)
000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Next obj: start=f70aeb00, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c0173923>](real_lookup+0x93/0xe0)
000: 6c 69 62 62 6f 6f 73 74 5f 70 72 67 5f 65 78 65 010: 63 5f 6d 6f 6e 69 74 6f 72 2d 67 63 63 2d 31 5f

Machine is running 2.6.16-rc5-mm2 :

$ uname -a
Linux dragon 2.6.16-rc5-mm2 #1 SMP PREEMPT Mon Mar 6 00:06:54 CET 2006 i686 athlon-4 i386 GNU/Linux

CPU is a dualcore Athlon X2 4400+

Kernel is 32bit, build for i386.

The machine has 2GB of RAM.

Let me know what additional info would be useful, if any.

If patches need testing then just send them my way.

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#1 lore raw
Quoted 10 lines · Jesper Juhl

Slab corruption: start=f72948a0, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270) 000: 70 00 05 00 00 00 00 0a 00 00 00 00 24 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f7294854, len=64
Redzone: 0x170fc2a5/0x170fc2a5.

Quoted 8 lines

Last user: [<c0173923>](real_lookup+0x93/0xe0)
000: 6c 69 62 6b 64 65 69 6e 69 74 5f 6b 69 6f 5f 68 010: 74 74 70 5f 63 61 63 68 65 5f 63 6c 65 61 6e 65 Next obj: start=f72948ec, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c01c7c80>](ext3_init_block_alloc_info+0x20/0x70) 000: 10 cf ce f7 00 00 00 00 00 00 00 00 00 00 00 00 010: 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Ok, this is interesting because the previous and the next objects look fine, which implies that it's not something else that overwrote it.

Perhaps more importantly, the actual corrupted object _should_ contain the POISON_FREE bytes, but doesn't. In fact, what it _does_ contain is apparently perfectly valid SCSI sense request data (if I read it right, it's ASC/ASCQ of 24/00 means "Invalid field in cdb").

So that "last user: sr_do_ioctl" thing actually matches what the contents are.

Your other smal corruption:

Quoted 9 lines

Slab corruption: start=f70aeab4, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270) 000: 70 00 02 00 00 00 00 0a 00 00 00 00 3a 01 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f70aea68, len=64
Redzone: 0x170fc2a5/0x170fc2a5.

Quoted 8 lines

Last user: [<c023d6df>](init_dev+0x5cf/0x630) 000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Next obj: start=f70aeb00, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c0173923>](real_lookup+0x93/0xe0) 000: 6c 69 62 62 6f 6f 73 74 5f 70 72 67 5f 65 78 65 010: 63 5f 6d 6f 6e 69 74 6f 72 2d 67 63 63 2d 31 5f

is exactly the same thing. The objects around it look ok. The object that should be poisoned again looks like it contains normal request-sense data (this time 3a/01: "Medium not present - tray closed", if I read it right).

So it really looks like in this case, the sr_do_ioctl() function free'd the sense key object, but that the sense data was actually written _after_ the object was free'd.

That in turn would imply that scsi_execute() returned before the SCSI command had actually completed. Now, scsi_execute() just calls to the generic block device layer, which in turn just submits it, and then waits for its completion, so if it returns too early, that means that it got _completed_ too early by the driver.

Alternatively, it got re-tried. The reason I mention that is that we have commit 17e01f216b611fc46956dcd9063aec4de75991e3, which changes scsi_execute() to add retries.

I wonder if something does a "complete(rq->waiting)" while the thing is still retrying? In general, I do not believe that we should retry special commands that have been initiated by a user, we should return the error. But I haven't thought this through.

Anyway, Jesper, I see two potential reasons for this bug:

- total and utter slab confusion (the slab layer returned the same slab allocation twice to two different callers). I consider this pretty unlikely, because it's such a _major_ failure of the slab code, and the slab code hasn't changed that much, but I mention it just in case.

- SCSI layer breakage. It might well be the low-level driver completing a request too early, or it migth be the re-trying. If it's the re-trying, you could try just reverting that commit I pointed to (ie if you're a git user, just do "git revert 17e01f21", otherwise you'd need to look it up from gitweb and un-apply the patch)

Regardless, Jesper, it would be great to hear _what_ strange CDROM device you have that would implied in sr_ioctl.c - is it USB, SATA or something else?

James, Mike, can you double-check the retries? In particular, it's _wrong_ to retry after you've already marked a command completed with "complete(rq->waiting)", so if that happens somewhere, things are really broken.

Linus

#2 lore raw
Quoted 1 line · Linus Torvalds

<...snip...>

Quoted 7 lines

Anyway, Jesper, I see two potential reasons for this bug:

- total and utter slab confusion (the slab layer returned the same slab allocation twice to two different callers). I consider this pretty unlikely, because it's such a _major_ failure of the slab code, and the slab code hasn't changed that much, but I mention it just in case.

Quoted 6 lines

- SCSI layer breakage. It might well be the low-level driver completing a request too early, or it migth be the re-trying. If it's the re-trying, you could try just reverting that commit I pointed to (ie if you're a git user, just do "git revert 17e01f21", otherwise you'd need to look it up from gitweb and un-apply the patch)

Not a git user (I need to become one but haven't found the time to read up on it yet), but no problem, I'll dig out the patch and try reverting it. Luckily it seems this is pretty repeatable on every boot, I find it in the logs instantly after logging in and launching a shell on my KDE desktop and running dmesg - I'll do a few more reboots to make sure it *really* is reproducible before reverting the patch so we can be sure if it fixes the problem or not.

Btw, the messages turn out slightly different on each boot, here are the ones from this current boot of my box:

Slab corruption: start=f72b6b98, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270) 000: 70 00 02 00 00 00 00 0a 00 00 00 00 3a 01 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f72b6b4c, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01813e6>](free_fdtable_rcu+0x66/0x150) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Next obj: start=f72b6be4, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<00000000>](_stext+0x3feffd68/0x8) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Slab corruption: start=f72b6b98, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270) 000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f72b6b4c, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01813e6>](free_fdtable_rcu+0x66/0x150) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Next obj: start=f72b6be4, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<00000000>](_stext+0x3feffd68/0x8) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Slab corruption: start=f72b6b98, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01d3769>](ext3_clear_inode+0x29/0x40) 000: 70 00 05 00 00 00 00 0a 00 00 00 00 24 00 00 00 010: 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Prev obj: start=f72b6b4c, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01813e6>](free_fdtable_rcu+0x66/0x150) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Next obj: start=f72b6be4, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<00000000>](_stext+0x3feffd68/0x8) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b

Would gathering more of these help you out?

Quoted 4 lines

Regardless, Jesper, it would be great to hear _what_ strange CDROM device you have that would implied in sr_ioctl.c - is it USB, SATA or something else?

I have no USB, SATA or similar devices in the box, only a floppy drive, a SCSI harddisk, a SCSI CD writer and a SCSI DVD-ROM.
Here are some details :

$ cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 04 Lun: 00
  Vendor: PIONEER  Model: DVD-ROM DVD-305  Rev: 1.03
  Type:   CD-ROM                           ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 05 Lun: 00
  Vendor: PLEXTOR  Model: CD-R   PX-W1210S Rev: 1.01
  Type:   CD-ROM                           ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 06 Lun: 00
  Vendor: IBM      Model: DDYS-T36950N     Rev: S96H
  Type:   Direct-Access                    ANSI SCSI revision: 03
Quoted 1 line

From dmesg :

scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0

        <Adaptec 29160N Ultra160 SCSI adapter>
        aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs

Vendor: PIONEER Model: DVD-ROM DVD-305 Rev: 1.03
Type: CD-ROM ANSI SCSI revision: 02
target0:0:4: Beginning Domain Validation
target0:0:4: FAST-20 SCSI 20.0 MB/s ST (50 ns, offset 16) target0:0:4: Domain Validation skipping write tests target0:0:4: Ending Domain Validation
Vendor: PLEXTOR Model: CD-R PX-W1210S Rev: 1.01
Type: CD-ROM ANSI SCSI revision: 02
target0:0:5: Beginning Domain Validation
target0:0:5: FAST-20 SCSI 20.0 MB/s ST (50 ns, offset 16) target0:0:5: Domain Validation skipping write tests target0:0:5: Ending Domain Validation
Vendor: IBM Model: DDYS-T36950N Rev: S96H
Type: Direct-Access ANSI SCSI revision: 03
scsi0:A:6:0: Tagged Queuing enabled. Depth 200
target0:0:6: Beginning Domain Validation
target0:0:6: wide asynchronous
target0:0:6: FAST-80 WIDE SCSI 160.0 MB/s DT (12.5 ns, offset 63) target0:0:6: Ending Domain Validation
SCSI device sda: 71687340 512-byte hdwr sectors (36704 MB)
sda: Write Protect is off
sda: Mode Sense: cb 00 00 08
SCSI device sda: drive cache: write back
SCSI device sda: 71687340 512-byte hdwr sectors (36704 MB)
sda: Write Protect is off
sda: Mode Sense: cb 00 00 08
SCSI device sda: drive cache: write back
sda: sda1 sda2 sda3 sda4
sd 0:0:6:0: Attached scsi disk sda
sr0: scsi3-mmc drive: 16x/40x cd/rw xa/form2 cdda tray Uniform CD-ROM driver Revision: 3.20
sr 0:0:4:0: Attached scsi CD-ROM sr0
sr1: scsi3-mmc drive: 32x/32x writer cd/rw xa/form2 cdda tray sr 0:0:5:0: Attached scsi CD-ROM sr1
sr 0:0:4:0: Attached scsi generic sg0 type 5
sr 0:0:5:0: Attached scsi generic sg1 type 5
sd 0:0:6:0: Attached scsi generic sg2 type 0

# lspci -vvx
00:00.0 Host bridge: ALi Corporation M1695 K8 Northbridge [PCI Express and HyperTransport]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0
        Capabilities: [40] #08 [0060]
        Capabilities: [5c] #08 [a800]
        Capabilities: [68] #08 [9000]
        Capabilities: [74] #08 [8000]
        Capabilities: [7c] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable-
                Address: 00000000fee00000  Data: 0000
00: b9 10 95 16 07 00 10 00 00 00 00 06 00 00 00 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00

00:01.0 PCI bridge: ALi Corporation: Unknown device 524b (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, cache line size 10
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Memory behind bridge: ff200000-ff2fffff
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable-
Address: 00000000fee00000 Data: 0000
Capabilities: [58] #10 [0141]
Capabilities: [7c] #08 [a800]
Capabilities: [88] #08 [8825]
00: b9 10 4b 52 06 01 10 00 00 00 04 06 10 00 01 00
10: 00 00 00 00 00 00 00 00 00 01 01 00 f0 00 00 00
20: 20 ff 20 ff f0 ff 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 03 00

00:02.0 PCI bridge: ALi Corporation: Unknown device 524c (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, cache line size 10
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
Memory behind bridge: ff300000-ff3fffff
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable-
Address: 00000000fee00000 Data: 0000
Capabilities: [58] #10 [0141]
Capabilities: [7c] #08 [a800]
Capabilities: [88] #08 [8825]
00: b9 10 4c 52 06 01 10 00 00 00 04 06 10 00 01 00
10: 00 00 00 00 00 00 00 00 00 02 02 00 f0 00 00 00
20: 30 ff 30 ff f0 ff 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 0b 01 03 00

00:04.0 Host bridge: ALi Corporation M1689 K8 Northbridge [Super K8 Single Chip]

        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0
        Region 0: Memory at dc000000 (32-bit, prefetchable) [size=64M]
        Capabilities: [40] #08 [0024]
        Capabilities: [60] #08 [8038]
        Capabilities: [80] AGP version 3.0
                Status: RQ=28 Iso- ArqSz=0 Cal=0 SBA+ ITACoh- GART64- HTrans- 64bit- FW- AGP3- Rate=x1,x2,x4
                Command: RQ=1 ArqSz=0 Cal=0 SBA- AGP- GART64- 64bit- FW- Rate=<none>

00: b9 10 89 16 06 01 10 00 00 00 00 06 00 00 00 00 10: 08 00 00 dc 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00

00:05.0 PCI bridge: ALi Corporation AGP8X Controller (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Bus: primary=00, secondary=03, subordinate=03, sec-latency=64
Memory behind bridge: ff400000-ff4fffff
Prefetchable memory behind bridge: c7f00000-d7efffff
BridgeCtl: Parity+ SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
00: b9 10 46 52 07 01 20 00 00 00 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 03 03 40 f0 00 20 22
20: 40 ff 40 ff f0 c7 e0 d7 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0b 00

00:06.0 PCI bridge: ALi Corporation M5249 HTT to PCI Bridge (prog-if 01 [Subtractive decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Bus: primary=00, secondary=04, subordinate=04, sec-latency=32
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: ff500000-ff5fffff
Prefetchable memory behind bridge: 88000000-880fffff
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
00: b9 10 49 52 07 01 00 00 00 01 04 06 00 00 01 00
10: 00 00 00 00 00 00 00 00 00 04 04 20 d0 d0 00 22
20: 50 ff 50 ff 00 88 00 88 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 00

00:07.0 ISA bridge: ALi Corporation M1563 HyperTransport South Bridge (rev 70)
Subsystem: ASRock Incorporation: Unknown device 1563
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0 (250ns min, 6000ns max)
00: b9 10 63 15 0f 00 00 02 70 00 01 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 49 18 63 15
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 18

00:07.1 Bridge: ALi Corporation M7101 Power Management Controller [PMU]
Subsystem: ASRock Incorporation: Unknown device 7101
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00: b9 10 01 71 00 00 00 02 00 00 80 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 49 18 01 71
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:11.0 Ethernet controller: ALi Corporation M5263 Ethernet Controller (rev 40)
Subsystem: ASRock Incorporation: Unknown device 5263
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (5000ns min, 10000ns max), cache line size 08
Interrupt: pin A routed to IRQ 10
Region 0: I/O ports at e800 [size=256]
Region 1: Memory at ff6ffc00 (32-bit, non-prefetchable) [size=256]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00: b9 10 63 52 07 01 10 02 40 00 00 02 08 20 00 00
10: 01 e8 00 00 00 fc 6f ff 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 49 18 63 52
30: 00 00 00 00 50 00 00 00 00 00 00 00 0a 01 14 28

00:12.0 IDE interface: ALi Corporation M5229 IDE (rev c7) (prog-if 8a [Master SecP PriP])
Subsystem: ASRock Incorporation: Unknown device 5229
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32
Interrupt: pin A routed to IRQ 0
Region 0: I/O ports at <ignored>
Region 1: I/O ports at <ignored>
Region 2: I/O ports at <ignored>
Region 3: I/O ports at <ignored>
Region 4: I/O ports at ff00 [size=16]
00: b9 10 29 52 05 00 a0 02 c7 8a 01 01 00 20 00 00
10: f1 01 00 00 f5 03 00 00 71 01 00 00 75 03 00 00
20: 01 ff 00 00 00 00 00 00 00 00 00 00 49 18 29 52
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00

00:13.0 USB Controller: ALi Corporation USB 1.1 Controller (rev 03) (prog-if 10 [OHCI])
Subsystem: ASRock Incorporation: Unknown device 5237
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (20000ns max), cache line size 10
Interrupt: pin A routed to IRQ 11
Region 0: Memory at ff6fe000 (32-bit, non-prefetchable) [size=4K]
00: b9 10 37 52 17 01 a8 02 03 10 03 0c 10 20 80 00
10: 00 e0 6f ff 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 49 18 37 52
30: 00 00 00 00 00 00 00 00 00 00 00 00 0b 01 00 50

00:13.1 USB Controller: ALi Corporation USB 1.1 Controller (rev 03) (prog-if 10 [OHCI])
Subsystem: ASRock Incorporation: Unknown device 5237
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (20000ns max), cache line size 10
Interrupt: pin B routed to IRQ 3
Region 0: Memory at ff6fd000 (32-bit, non-prefetchable) [size=4K]
00: b9 10 37 52 17 01 a8 02 03 10 03 0c 10 20 80 00
10: 00 d0 6f ff 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 49 18 37 52
30: 00 00 00 00 00 00 00 00 00 00 00 00 03 02 00 50

00:13.2 USB Controller: ALi Corporation USB 1.1 Controller (rev 03) (prog-if 10 [OHCI])
Subsystem: ASRock Incorporation: Unknown device 5237
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (20000ns max), cache line size 10
Interrupt: pin C routed to IRQ 11
Region 0: Memory at ff6fc000 (32-bit, non-prefetchable) [size=4K]
00: b9 10 37 52 17 01 a8 02 03 10 03 0c 10 20 80 00
10: 00 c0 6f ff 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 49 18 37 52
30: 00 00 00 00 00 00 00 00 00 00 00 00 0b 03 00 50

00:13.3 USB Controller: ALi Corporation USB 2.0 Controller (rev 01) (prog-if 20 [EHCI])
Subsystem: ASRock Incorporation: Unknown device 5239
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (4000ns min, 8000ns max), cache line size 10
Interrupt: pin D routed to IRQ 5
Region 0: Memory at ff6ff800 (32-bit, non-prefetchable) [size=256]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] #0a [2090]
00: b9 10 39 52 16 01 b0 02 01 20 03 0c 10 20 80 00
10: 00 f8 6f ff 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 49 18 39 52
30: 00 00 00 00 50 00 00 00 00 00 00 00 05 04 10 20

00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Capabilities: [80] #08 [2101]
00: 22 10 00 11 00 00 10 00 00 00 00 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00

00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00: 22 10 01 11 00 00 00 00 00 00 00 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00: 22 10 02 11 00 00 00 00 00 00 00 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00: 22 10 03 11 00 00 00 00 00 00 00 06 00 00 80 00
10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

03:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA Parhelia AGP (rev 03) (prog-if 00 [VGA])
Subsystem: Matrox Graphics, Inc. Parhelia 128Mb
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (4000ns min, 8000ns max), cache line size 10
Interrupt: pin A routed to IRQ 5
Region 0: Memory at c8000000 (32-bit, prefetchable) [size=128M]
Region 1: Memory at ff4fe000 (32-bit, non-prefetchable) [size=8K]
Expansion ROM at ff4c0000 [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [f0] AGP version 2.0
Status: RQ=32 Iso- ArqSz=0 Cal=0 SBA+ ITACoh- GART64- HTrans- 64bit- FW+ AGP3- Rate=x1,x2,x4
Command: RQ=1 ArqSz=0 Cal=0 SBA- AGP- GART64- 64bit- FW- Rate=<none>
00: 2b 10 27 05 07 00 b0 02 03 00 00 03 10 20 00 00
10: 08 00 00 c8 00 e0 4f ff 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 2b 10 40 08
30: 00 00 4c ff dc 00 00 00 00 00 00 00 05 01 10 20

04:05.0 Multimedia audio controller: Creative Labs SB Live! EMU10k1 (rev 0a)
Subsystem: Creative Labs SBLive! 5.1 eMicro 28028
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (500ns min, 5000ns max)
Interrupt: pin A routed to IRQ 20
Region 0: I/O ports at d880 [size=32]
Capabilities: [dc] Power Management version 1
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00: 02 11 02 00 05 01 90 02 0a 00 01 04 00 20 80 00
10: 81 d8 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 02 11 67 80
30: 00 00 00 00 dc 00 00 00 00 00 00 00 0b 01 02 14

04:05.1 Input device controller: Creative Labs SB Live! MIDI/Game Port (rev 0a)
Subsystem: Creative Labs Gameport Joystick
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32
Region 0: I/O ports at dc00 [size=8]
Capabilities: [dc] Power Management version 1
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00: 02 11 02 70 05 01 90 02 0a 00 80 09 00 20 80 00
10: 01 dc 00 00 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 02 11 20 00
30: 00 00 00 00 dc 00 00 00 00 00 00 00 00 00 00 00

04:06.0 SCSI storage controller: Adaptec AIC-7892A U160/m (rev 02)
Subsystem: Adaptec 29160N Ultra160 SCSI Controller
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (10000ns min, 6250ns max), cache line size 10
Interrupt: pin A routed to IRQ 19
BIST result: 00
Region 0: I/O ports at d400 [disabled] [size=256]
Region 1: Memory at ff5ff000 (64-bit, non-prefetchable) [size=4K]
Expansion ROM at 88000000 [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00: 05 90 80 00 16 01 b0 02 02 00 00 01 10 20 00 80
10: 01 d4 00 00 04 f0 5f ff 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 05 90 a0 62
30: 00 00 5c ff dc 00 00 00 00 00 00 00 03 01 28 19

04:07.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 42)
Subsystem: D-Link System Inc DFE-530TX rev B
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (750ns min, 2000ns max), cache line size 10
Interrupt: pin A routed to IRQ 18
Region 0: I/O ports at d000 [size=256]
Region 1: Memory at ff5fec00 (32-bit, non-prefetchable) [size=256]
Expansion ROM at 88020000 [disabled] [size=64K]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00: 06 11 65 30 17 01 10 02 42 00 00 02 10 20 00 00
10: 01 d0 00 00 00 ec 5f ff 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 11 01 14
30: 00 00 ff ff 40 00 00 00 00 00 00 00 02 01 03 08

/Jesper

#47 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

James, Mike, can you double-check the retries? In particular, it's _wrong_ to retry after you've already marked a command completed with "complete(rq->waiting)", so if that happens somewhere, things are really broken.

I am looking into it. I think it has something to do with the request getting completed too early or maybe something crazy like twice. This looks like a similar problem that was reported to linux-scsi where for some tape setup the request's bio gets freed twice.

#48 lore raw
Quoted 7 lines · Mike Christie
Quoted 3 lines · Linus Torvalds

James, Mike, can you double-check the retries? In particular, it's _wrong_ to retry after you've already marked a command completed with "complete(rq->waiting)", so if that happens somewhere, things are really broken.

I am looking into it. I think it has something to do with the request getting completed too early or maybe something crazy like twice. This looks like a similar problem that was reported to linux-scsi

Oh yeah here is that thread

http://marc.theaimsgroup.com/?l=linux-scsi&m=114127615918030&w=2

#3 lore raw
In reply to Jesper Juhl
Quoted 3 lines · Jesper Juhl

Not a git user (I need to become one but haven't found the time to read up on it yet), but no problem, I'll dig out the patch and try reverting it.

It's attached here.

NOTE! I'm not at all sure it's the re-try logic. It could be something else. Anything that completes the request before it's actually totally done - or possibly re-uses the sense data for something else would be wrong and buggy.

Quoted 10 lines

Btw, the messages turn out slightly different on each boot, here are the ones from this current boot of my box:

Slab corruption: start=f72b6b98, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270) 000: 70 00 02 00 00 00 00 0a 00 00 00 00 3a 01 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Ok, same deal. "Medium not present - tray closed" sense data.

Quoted 7 lines

Slab corruption: start=f72b6b98, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270) 000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Hmm. Totally empty sense data? Strange.

Quoted 5 lines

Slab corruption: start=f72b6b98, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01d3769>](ext3_clear_inode+0x29/0x40) 000: 70 00 05 00 00 00 00 0a 00 00 00 00 24 00 00 00 010: 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b

This is different. But it looks similar. It looks like the thing was actually re-allocated for something else (posix acl data?) but then overwritten. However, the overwritten data does look like SCSI sense information again ("Invalid field in cdb"), so I think it's the same thing despite the fact that it had gotten re-allocated for something else.

Quoted 1 line

Would gathering more of these help you out?

It's always interesting when trying to find the pattern, but I think the pattern is already pretty clear. sr_do_ioctl() seems to be the thing, and sense data is written too late.

Quoted 2 lines

I have no USB, SATA or similar devices in the box, only a floppy drive, a SCSI harddisk, a SCSI CD writer and a SCSI DVD-ROM.

Well, the fact that you have a CDSI CD-writer and a SCSI DVD-ROM explains the thing, so that's all good.

Quoted 3 lines

scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0

        <Adaptec 29160N Ultra160 SCSI adapter>
        aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs

So it's either an aic7xxx bug, or it's generic SCSI.

Considering that we've had other slab corruption issues (the reason I was looking closely at yours), generic SCSI isn't out of the question.

If you were a git user, doing a bisection run would be useful since you seem to be able to recreate it at will. Oh, well. Testign that one patch would still help.

Linus

diff-tree 17e01f216b611fc46956dcd9063aec4de75991e3 (from 6e68af666f5336254b5715dca591026b7324499a)

Author: Mike Christie <[email protected]>
Date:   Fri Nov 11 05:31:37 2005 -0600

[SCSI] add retries field to request for REQ_BLOCK_PC use

For tape we need to control the retries. This patch adds a retries counter on the request for REQ_BLOCK_PC commands originating from scsi_execute* to use. REQ_BLOCK_PC commands comming from the block layer SG_IO path continue to use the retires set in the ULD init_command. (scsi_execute* does not set the gendisk so we do not execute the init_command in that path).

drivers/scsi/scsi_lib.c +3 -1
index eb0cfbf..365843a 100644
@@ -259,6 +259,7 @@ int scsi_execute(struct scsi_device *sde
 	memcpy(req->cmd, cmd, req->cmd_len);
 	req->sense = sense;
 	req->sense_len = 0;
+	req->retries = retries;
 	req->timeout = timeout;
 	req->flags |= flags | REQ_BLOCK_PC | REQ_SPECIAL | REQ_QUIET;
 
@@ -472,6 +473,7 @@ int scsi_execute_async(struct scsi_devic
 	req->sense = sioc->sense;
 	req->sense_len = 0;
 	req->timeout = timeout;
+	req->retries = retries;
 	req->flags |= REQ_BLOCK_PC | REQ_QUIET;
 	req->end_io_data = sioc;
 
@@ -1393,7 +1395,7 @@ static int scsi_prep_fn(struct request_q
 				cmd->sc_data_direction = DMA_NONE;
 			
 			cmd->transfersize = req->data_len;
-			cmd->allowed = 3;
+			cmd->allowed = req->retries;
 			cmd->timeout_per_command = req->timeout;
 			cmd->done = scsi_generic_done;
 		}
include/linux/blkdev.h +1 -0
index 9a68716..509e9a0 100644
@@ -184,6 +184,7 @@ struct request {
 	void *sense;
 
 	unsigned int timeout;
+	int retries;
 
 	/*
 	 * For Power Management requests
#4 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Jesper Juhl

Not a git user (I need to become one but haven't found the time to read up on it yet), but no problem, I'll dig out the patch and try reverting it.

It's attached here.

Thanks.

Quoted 5 lines

NOTE! I'm not at all sure it's the re-try logic. It could be something else. Anything that completes the request before it's actually totally done - or possibly re-uses the sense data for something else would be wrong and buggy.

Ohh well, let's work on the assumption that it is the re-try logic first, then try something else if it turns out it isn't. I have no problem testing a bunch of patches if needed.

<...>

Quoted 2 lines

This is different. But it looks similar. It looks like the thing was actually re-allocated for something else (posix acl data?) but then

I doubt it's POSIX ACL data :

juhl@dragon:~/download/kernel/linux-2.6.16-rc5-mm2$ grep -i ACL .config
# CONFIG_FS_POSIX_ACL is not set
Quoted 10 lines

overwritten. However, the overwritten data does look like SCSI sense information again ("Invalid field in cdb"), so I think it's the same thing despite the fact that it had gotten re-allocated for something else.

Quoted 1 line

Would gathering more of these help you out?

It's always interesting when trying to find the pattern, but I think the pattern is already pretty clear. sr_do_ioctl() seems to be the thing, and sense data is written too late.

Ok, it's reproducible on demand (at least I've now reproduced it on 6 more boots), so if you need any more just let me know and I'll gather a few.

Quoted 10 lines
Quoted 2 lines

I have no USB, SATA or similar devices in the box, only a floppy drive, a SCSI harddisk, a SCSI CD writer and a SCSI DVD-ROM.

Well, the fact that you have a CDSI CD-writer and a SCSI DVD-ROM explains the thing, so that's all good.

Quoted 3 lines

scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0

        <Adaptec 29160N Ultra160 SCSI adapter>
        aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
Quoted 6 lines

So it's either an aic7xxx bug, or it's generic SCSI.

Considering that we've had other slab corruption issues (the reason I was looking closely at yours), generic SCSI isn't out of the question.

If you were a git user, doing a bisection run would be useful since you

Well, now is probably as good a time as any for becoming a git user, tracking my own patches as individual plain-text files is getting un-managable and as you say, bisection would be useful to be able to do. I'll dig up some git docs and start reading.

Quoted 3 lines

seem to be able to recreate it at will. Oh, well. Testign that one patch would still help.

Hmm, that patch does not apply to the 2.6.16-rc5-mm2 kernel :

patching file drivers/scsi/scsi_lib.c
Hunk #1 succeeded at 260 (offset 1 line).
Hunk #2 FAILED at 473.
Hunk #3 FAILED at 1394.
2 out of 3 hunks FAILED -- saving rejects to file drivers/scsi/scsi_lib.c.rej patching file include/linux/blkdev.h
Hunk #1 succeeded at 190 (offset 6 lines).

I'll go see if the problem also exists in mainline - will report on that shortly.

/Jesper

#5 lore raw
Quoted 7 lines · Jesper Juhl
Quoted 4 lines · Linus Torvalds

seem to be able to recreate it at will. Oh, well. Testign that one patch would still help.

Hmm, that patch does not apply to the 2.6.16-rc5-mm2 kernel :

I fixed it up by hand.
Building a new kernel at the moment - results in a short while.

Quoted 4 lines

I'll go see if the problem also exists in mainline - will report on that shortly.

I'll still do this. Just downloading 2.6.16-rc5-git8 as we speak.

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#6 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds

So it's either an aic7xxx bug, or it's generic SCSI.

Considering that we've had other slab corruption issues (the reason I was looking closely at yours), generic SCSI isn't out of the question.

If you were a git user, doing a bisection run would be useful since you seem to be able to recreate it at will. Oh, well. Testign that one patch would still help.

Hmm.. This appended patch may or may not help.

It overwrites the SCSI command "req" pointer when the request has been done. The request cannot be used afterwards, so anybody accessing it would be a bug. I think.

HOWEVER. I noticed something else strange. Your slab corruption report says

	Slab corruption: start=f72948a0, len=64
	Redzone: 0x5a2cf071/0x5a2cf071.
	Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270)
	...

and the scary thing is that "len=64".

The thing is, SCSI uses "SCSI_SENSE_BUFFERSIZE" to determine the maximum sense size to copy, and what do we have, if not

include/scsi/scsi_cmnd.h:#define SCSI_SENSE_BUFFERSIZE 96

ie a 64-byte buffer is simply TOO DAMN SMALL!

Now, the thing is, the 64 comes from "sizeof(struct request_sense)", which is what "struct packet_command *" uses. We can change that sizeof() to just use SCSI_SENSE_BUFFERSIZE, but that still makes me worry about somebody else having allocated a "packed_command->sense" using just the same 64-byte "struct request_sense".

Can a SCSI sense-buffer really be 96? If so, why is "struct request_sense" the wrong size? This looks likea really nasty bug.

Linus

----

drivers/scsi/scsi_lib.c +3 -0
index 701a328..296ac8e 100644
@@ -788,6 +788,9 @@ static struct scsi_cmnd *scsi_end_reques
 		}
 	}
 
+	/* Poison the request pointer: it is done and no longer exists after this */
+	cmd->request = (void *) 0x5a5a5a5a;
+
 	add_disk_randomness(req->rq_disk);
 
 	spin_lock_irqsave(q->queue_lock, flags);
#7 lore raw
Quoted 12 lines · Linus Torvalds
Quoted 9 lines · Linus Torvalds

So it's either an aic7xxx bug, or it's generic SCSI.

Considering that we've had other slab corruption issues (the reason I was looking closely at yours), generic SCSI isn't out of the question.

If you were a git user, doing a bisection run would be useful since you seem to be able to recreate it at will. Oh, well. Testign that one patch would still help.

Since the patch you sent me didn't apply cleanly to the mm kernel I made the changes by hand. This is what I ended up with (should be the same end result as what you intended as far as I can see) :

drivers/scsi/scsi_lib.c +1 -3
@@ -260,7 +260,6 @@ int scsi_execute(struct scsi_device *sde
 	memcpy(req->cmd, cmd, req->cmd_len);
 	req->sense = sense;
 	req->sense_len = 0;
-	req->retries = retries;
 	req->timeout = timeout;
 	req->flags |= flags | REQ_BLOCK_PC | REQ_SPECIAL | REQ_QUIET;
 
@@ -478,7 +477,6 @@ int scsi_execute_async(struct scsi_devic
 	req->sense = sioc->sense;
 	req->sense_len = 0;
 	req->timeout = timeout;
-	req->retries = retries;
 	req->end_io_data = sioc;
 
 	sioc->data = privdata;
@@ -1240,7 +1238,7 @@ static void scsi_setup_blk_pc_cmnd(struc
 		cmd->sc_data_direction = DMA_FROM_DEVICE;
 	
 	cmd->transfersize = req->data_len;
-	cmd->allowed = req->retries;
+	cmd->allowed = 3;
 	cmd->timeout_per_command = req->timeout;
 	cmd->done = scsi_blk_pc_done;
 }
block/scsi_ioctl.c +0 -3
@@ -314,8 +314,6 @@ static int sg_io(struct file *file, requ
 	if (!rq->timeout)
 		rq->timeout = BLK_DEFAULT_TIMEOUT;
 
-	rq->retries = 0;
-
 	start_time = jiffies;
 
 	/* ignore return value. All information is passed back to caller
@@ -433,7 +431,6 @@ static int sg_scsi_ioctl(struct file *fi
 	rq->data = buffer;
 	rq->data_len = bytes;
 	rq->flags |= REQ_BLOCK_PC;
-	rq->retries = 0;
 
 	blk_execute_rq(q, bd_disk, rq, 0);
 	err = rq->errors & 0xff;	/* only 8 bit SCSI status */
include/linux/blkdev.h +0 -1
@@ -190,7 +190,6 @@ struct request {
 	void *sense;
 
 	unsigned int timeout;
-	int retries;
 
 	/*
 	 * For Power Management requests

Unfortunately that didn't fix it. After booting the patched kernel I found this in dmesg :

Slab corruption: start=f77d768c, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934db>](sr_do_ioctl+0x11b/0x270)
000: 70 00 02 00 00 00 00 0a 00 00 00 00 3a 01 00 00
010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Prev obj: start=f77d7640, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c023d64d>](init_dev+0x55d/0x630)
000: 00 01 00 00 05 00 00 00 bf 00 00 00 3b 8a 00 00
010: 00 03 1c 7f 15 04 00 01 00 11 13 1a 00 12 0f 17
Next obj: start=f77d76d8, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c01c7c80>](ext3_init_block_alloc_info+0x20/0x70) 000: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a
010: 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Slab corruption: start=f7001640, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934db>](sr_do_ioctl+0x11b/0x270)
000: 70 00 05 00 00 00 00 0a 00 00 00 00 24 00 00 00
010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Prev obj: start=f70015f4, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<f8bdf124>](__snd_util_mem_alloc+0x74/0x80 [snd_util_mem]) 000: 00 10 00 00 00 00 00 00 b0 75 7d f7 50 33 bd f5
010: 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
Next obj: start=f700168c, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c0173923>](real_lookup+0x93/0xe0)
000: 6c 69 62 62 6f 6f 73 74 5f 75 6e 69 74 5f 74 65
010: 73 74 5f 66 72 61 6d 65 77 6f 72 6b 2d 67 63 63

Quoted 2 lines

Hmm.. This appended patch may or may not help.

I'll give it a spin.

Quoted 4 lines

It overwrites the SCSI command "req" pointer when the request has been done. The request cannot be used afterwards, so anybody accessing it would be a bug. I think.

Let's see what happens.
I've applied it on top of the changes mentioned above - let me know if that's wrong.

Quoted 13 lines

HOWEVER. I noticed something else strange. Your slab corruption report says

	Slab corruption: start=f72948a0, len=64
	Redzone: 0x5a2cf071/0x5a2cf071.
	Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270)
	...

and the scary thing is that "len=64".

The thing is, SCSI uses "SCSI_SENSE_BUFFERSIZE" to determine the maximum sense size to copy, and what do we have, if not

Quoted 7 lines

include/scsi/scsi_cmnd.h:#define SCSI_SENSE_BUFFERSIZE 96

ie a 64-byte buffer is simply TOO DAMN SMALL!

Now, the thing is, the 64 comes from "sizeof(struct request_sense)", which is what "struct packet_command *" uses. We can change that sizeof() to just use SCSI_SENSE_BUFFERSIZE, but that still makes me worry about

I'll try that after booting with the patch you just supplied and let you know the results shortly.

/Jesper

#8 lore raw
Quoted 76 lines · Jesper Juhl
Quoted 12 lines · Linus Torvalds
Quoted 9 lines · Linus Torvalds

So it's either an aic7xxx bug, or it's generic SCSI.

Considering that we've had other slab corruption issues (the reason I was looking closely at yours), generic SCSI isn't out of the question.

If you were a git user, doing a bisection run would be useful since you seem to be able to recreate it at will. Oh, well. Testign that one patch would still help.

Since the patch you sent me didn't apply cleanly to the mm kernel I made the changes by hand. This is what I ended up with (should be the same end result as what you intended as far as I can see) :

drivers/scsi/scsi_lib.c +1 -3
@@ -260,7 +260,6 @@ int scsi_execute(struct scsi_device *sde
 	memcpy(req->cmd, cmd, req->cmd_len);
 	req->sense = sense;
 	req->sense_len = 0;
-	req->retries = retries;
 	req->timeout = timeout;
 	req->flags |= flags | REQ_BLOCK_PC | REQ_SPECIAL | REQ_QUIET;
 
@@ -478,7 +477,6 @@ int scsi_execute_async(struct scsi_devic
 	req->sense = sioc->sense;
 	req->sense_len = 0;
 	req->timeout = timeout;
-	req->retries = retries;
 	req->end_io_data = sioc;
 
 	sioc->data = privdata;
@@ -1240,7 +1238,7 @@ static void scsi_setup_blk_pc_cmnd(struc
 		cmd->sc_data_direction = DMA_FROM_DEVICE;
 
 	cmd->transfersize = req->data_len;
-	cmd->allowed = req->retries;
+	cmd->allowed = 3;
 	cmd->timeout_per_command = req->timeout;
 	cmd->done = scsi_blk_pc_done;
 }
block/scsi_ioctl.c +0 -3
@@ -314,8 +314,6 @@ static int sg_io(struct file *file, requ
 	if (!rq->timeout)
 		rq->timeout = BLK_DEFAULT_TIMEOUT;
 
-	rq->retries = 0;
-
 	start_time = jiffies;
 
 	/* ignore return value. All information is passed back to caller
@@ -433,7 +431,6 @@ static int sg_scsi_ioctl(struct file *fi
 	rq->data = buffer;
 	rq->data_len = bytes;
 	rq->flags |= REQ_BLOCK_PC;
-	rq->retries = 0;
 
 	blk_execute_rq(q, bd_disk, rq, 0);
 	err = rq->errors & 0xff;	/* only 8 bit SCSI status */
include/linux/blkdev.h +0 -1
@@ -190,7 +190,6 @@ struct request {
 	void *sense;
 
 	unsigned int timeout;
-	int retries;
 
 	/*
 	 * For Power Management requests
Quoted 2 lines

Unfortunately that didn't fix it. After booting the patched kernel I found this in dmesg :

I don't see how it could be, honestly, we would gladly oops in locally close places if that was the case. If you disable slab debug/poison, do you get a nice NULL pointer dereference instead? There have been some reports on a NULL queue for sr devices as of lately, I wonder if some SCSI change recently was broken.

Tejun, I seem to recall you looking at this, but I can't seem to locate the thread. Did anything come of it?

Signature · 1 line
Jens Axboe
#9 lore raw
Quoted 8 lines · Jens Axboe

I don't see how it could be, honestly, we would gladly oops in locally close places if that was the case. If you disable slab debug/poison, do you get a nice NULL pointer dereference instead? There have been some reports on a NULL queue for sr devices as of lately, I wonder if some SCSI change recently was broken.

Tejun, I seem to recall you looking at this, but I can't seem to locate the thread. Did anything come of it?

This is the one:

http://marc.theaimsgroup.com/?l=linux-kernel&m=114041855331295&w=2

Also an -mm report, btw. Does this reproduce with 2.6.16-rcX latest?

Signature · 1 line
Jens Axboe
#20 lore raw
In reply to Linus Torvalds
Quoted 12 lines · Linus Torvalds
Quoted 9 lines · Linus Torvalds

So it's either an aic7xxx bug, or it's generic SCSI.

Considering that we've had other slab corruption issues (the reason I was looking closely at yours), generic SCSI isn't out of the question.

If you were a git user, doing a bisection run would be useful since you seem to be able to recreate it at will. Oh, well. Testign that one patch would still help.

Quoted 6 lines

Hmm.. This appended patch may or may not help.

It overwrites the SCSI command "req" pointer when the request has been done. The request cannot be used afterwards, so anybody accessing it would be a bug. I think.

With the retry code removed and your req poisoning patch on top I just got this :

Slab corruption: start=f727c5a8, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934db>](sr_do_ioctl+0x11b/0x270) 000: 70 00 02 00 00 00 00 0a 00 00 00 00 3a 01 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f727c55c, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01813ee>](free_fdtable_rcu+0x6e/0x150) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Next obj: start=f727c5f4, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01813ee>](free_fdtable_rcu+0x6e/0x150) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Slab corruption: start=f727c5a8, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934db>](sr_do_ioctl+0x11b/0x270) 000: 70 00 05 00 00 00 00 0a 00 00 00 00 24 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f727c55c, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01813ee>](free_fdtable_rcu+0x6e/0x150) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Next obj: start=f727c5f4, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01813ee>](free_fdtable_rcu+0x6e/0x150) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b

and another, probably unrelated, thing I just noticed in my dmesg output:

initcall at 0xc0428240: init_hpet_clocksource+0x0/0x90(): returned with error code -19

Quoted 13 lines

HOWEVER. I noticed something else strange. Your slab corruption report says

	Slab corruption: start=f72948a0, len=64
	Redzone: 0x5a2cf071/0x5a2cf071.
	Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270)
	...

and the scary thing is that "len=64".

The thing is, SCSI uses "SCSI_SENSE_BUFFERSIZE" to determine the maximum sense size to copy, and what do we have, if not

Quoted 7 lines

include/scsi/scsi_cmnd.h:#define SCSI_SENSE_BUFFERSIZE 96

ie a 64-byte buffer is simply TOO DAMN SMALL!

Now, the thing is, the 64 comes from "sizeof(struct request_sense)", which is what "struct packet_command *" uses. We can change that sizeof() to just use SCSI_SENSE_BUFFERSIZE, but that still makes me worry about

Building a kernel with that change on top of the other ones atm.

/ Jesper

#21 lore raw
Quoted 2 lines · Jesper Juhl
Quoted 1 line · Linus Torvalds

<...snip...>

Quoted 7 lines
Quoted 7 lines

and the scary thing is that "len=64".

The thing is, SCSI uses "SCSI_SENSE_BUFFERSIZE" to determine the maximum sense size to copy, and what do we have, if not

include/scsi/scsi_cmnd.h:#define SCSI_SENSE_BUFFERSIZE 96

Quoted 8 lines
Quoted 5 lines

ie a 64-byte buffer is simply TOO DAMN SMALL!

Now, the thing is, the 64 comes from "sizeof(struct request_sense)", which is what "struct packet_command *" uses. We can change that sizeof() to just use SCSI_SENSE_BUFFERSIZE, but that still makes me worry about

Building a kernel with that change on top of the other ones atm.

Changing the sizeof() to SCSI_SENSE_BUFFERSIZE doesn't fix it :

Slab corruption: start=f79da5a8, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934db>](sr_do_ioctl+0x11b/0x270) 000: 70 00 02 00 00 00 00 0a 00 00 00 00 3a 01 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f79da55c, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c0158918>](__vmalloc_node+0x68/0x80) 000: d0 1e 1e c3 18 1f 1e c3 60 1f 1e c3 a8 1f 1e c3 010: f0 1f 1e c3 38 20 1e c3 80 20 1e c3 c8 20 1e c3 Next obj: start=f79da5f4, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c0173923>](real_lookup+0x93/0xe0) 000: 6c 69 62 62 6f 6f 73 74 5f 70 72 67 5f 65 78 65 010: 63 5f 6d 6f 6e 69 74 6f 72 2d 67 63 63 2d 6d 74 Slab corruption: start=f79da5a8, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934db>](sr_do_ioctl+0x11b/0x270) 000: 70 00 05 00 00 00 00 0a 00 00 00 00 24 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f79da55c, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c0158918>](__vmalloc_node+0x68/0x80) 000: d0 1e 1e c3 18 1f 1e c3 60 1f 1e c3 a8 1f 1e c3 010: f0 1f 1e c3 38 20 1e c3 80 20 1e c3 c8 20 1e c3 Next obj: start=f79da5f4, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c0173923>](real_lookup+0x93/0xe0) 000: 6c 69 62 62 6f 6f 73 74 5f 70 72 67 5f 65 78 65 010: 63 5f 6d 6f 6e 69 74 6f 72 2d 67 63 63 2d 6d 74

I'll now go test the things Jens suggested. Expect more feedback shortly.

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#22 lore raw
Quoted 17 lines · Jesper Juhl
Quoted 2 lines · Jesper Juhl
Quoted 1 line · Linus Torvalds

<...snip...>

Quoted 13 lines
Quoted 12 lines

and the scary thing is that "len=64".

The thing is, SCSI uses "SCSI_SENSE_BUFFERSIZE" to determine the maximum sense size to copy, and what do we have, if not

include/scsi/scsi_cmnd.h:#define SCSI_SENSE_BUFFERSIZE 96

ie a 64-byte buffer is simply TOO DAMN SMALL!

Now, the thing is, the 64 comes from "sizeof(struct request_sense)", which is what "struct packet_command *" uses. We can change that sizeof() to just use SCSI_SENSE_BUFFERSIZE, but that still makes me worry about

Quoted 5 lines
Quoted 2 lines

Building a kernel with that change on top of the other ones atm.

Changing the sizeof() to SCSI_SENSE_BUFFERSIZE doesn't fix it :

Slab corruption: start=f79da5a8, len=64

Hmm, is it just me or should that len= have read len=96 ???

This is the change I made :

--- linux-2.6.16-rc5-mm2/block/scsi_ioctl.c~ 2006-03-06
21:43:56.000000000 +0100
+++ linux-2.6.16-rc5-mm2/block/scsi_ioctl.c 2006-03-06
21:43:56.000000000 +0100
@@ -568,7 +568,7 @@ int scsi_cmd_ioctl(struct file *file, st

                        hdr.dxferp = cgc.buffer;
                        hdr.sbp = cgc.sense;
                        if (hdr.sbp)

- hdr.mx_sb_len = sizeof(struct request_sense);
+ hdr.mx_sb_len = SCSI_SENSE_BUFFERSIZE;
hdr.timeout = cgc.timeout; hdr.cmdp = ((struct cdrom_generic_command

__user*) arg)->cmd;
                        hdr.cmd_len = sizeof(cgc.cmd);

did I mess up?

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#23 lore raw
Quoted 19 lines · Jesper Juhl

Hmm, is it just me or should that len= have read len=96 ???

This is the change I made :

--- linux-2.6.16-rc5-mm2/block/scsi_ioctl.c~ 2006-03-06
21:43:56.000000000 +0100
+++ linux-2.6.16-rc5-mm2/block/scsi_ioctl.c 2006-03-06
21:43:56.000000000 +0100

(quoted diff) +1 -1
@@ -568,7 +568,7 @@ int scsi_cmd_ioctl(struct file *file, st
                        hdr.dxferp = cgc.buffer;
                        hdr.sbp = cgc.sense;
                        if (hdr.sbp)
-                               hdr.mx_sb_len = sizeof(struct request_sense);
+                               hdr.mx_sb_len = SCSI_SENSE_BUFFERSIZE;
                        hdr.timeout = cgc.timeout;
                        hdr.cmdp = ((struct cdrom_generic_command __user*) arg)->cmd;
                        hdr.cmd_len = sizeof(cgc.cmd);
Quoted 1 line

did I mess up?

That's not the one to change. It's the one in "sr_do_ioctl()", where it uses "sizeof(*sense)".

Linus

----

drivers/scsi/sr_ioctl.c +1 -1
index 5d02ff4..b65462f 100644
@@ -192,7 +192,7 @@ int sr_do_ioctl(Scsi_CD *cd, struct pack
 	SDev = cd->device;
 
 	if (!sense) {
-		sense = kmalloc(sizeof(*sense), GFP_KERNEL);
+		sense = kmalloc(SCSI_SENSE_BUFFERSIZE, GFP_KERNEL);
 		if (!sense) {
 			err = -ENOMEM;
 			goto out;
#10 lore raw
In reply to Jens Axboe

On 3/6/06, Jens Axboe <[email protected]> wrote:
<snip>

Quoted 7 lines

This is the one:

http://marc.theaimsgroup.com/?l=linux-kernel&m=114041855331295&w=2

Also an -mm report, btw. Does this reproduce with 2.6.16-rcX latest?

I just build and booted a2.6.16-rc5-git8 with the same config that I used for the -mm kernel and the problem did not manifest itself there. So it seems that mainline is fine but we need to find the bug before it propagates from mm to mainline.

I'll test a -mm kernel without slab debug/poison now to see it it goes Oops.

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#24 lore raw
In reply to Linus Torvalds
Quoted 22 lines · Linus Torvalds
Quoted 19 lines · Jesper Juhl

Hmm, is it just me or should that len= have read len=96 ???

This is the change I made :

--- linux-2.6.16-rc5-mm2/block/scsi_ioctl.c~ 2006-03-06
21:43:56.000000000 +0100
+++ linux-2.6.16-rc5-mm2/block/scsi_ioctl.c 2006-03-06
21:43:56.000000000 +0100

(quoted diff) +1 -1
@@ -568,7 +568,7 @@ int scsi_cmd_ioctl(struct file *file, st
                        hdr.dxferp = cgc.buffer;
                        hdr.sbp = cgc.sense;
                        if (hdr.sbp)
-                               hdr.mx_sb_len = sizeof(struct request_sense);
+                               hdr.mx_sb_len = SCSI_SENSE_BUFFERSIZE;
                        hdr.timeout = cgc.timeout;
                        hdr.cmdp = ((struct cdrom_generic_command __user*) arg)->cmd;
                        hdr.cmd_len = sizeof(cgc.cmd);
Quoted 5 lines
Quoted 1 line

did I mess up?

That's not the one to change. It's the one in "sr_do_ioctl()", where it uses "sizeof(*sense)".

Ahh, so I did mess up - whoops - I just grep'ed for "sizeof(struct request_sense)" :-(

I'll try it again (with the correct change) in a moment, after I've tested Jens's "does no slab poison/debug make it go Oops" question...

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#11 lore raw
In reply to Jens Axboe

On 3/6/06, Jens Axboe <[email protected]> wrote:
[...snip...]

Quoted 7 lines

I don't see how it could be, honestly, we would gladly oops in locally close places if that was the case. If you disable slab debug/poison, do you get a nice NULL pointer dereference instead? There have been some reports on a NULL queue for sr devices as of lately, I wonder if some SCSI change recently was broken.

I just build and booted a plain 2.6.16-rc5-mm2 with the same config I've used previously, but with the following options disabled :

CONFIG_DEBUG_SLAB
CONFIG_PAGE_OWNER
CONFIG_DEBUG_VM
CONFIG_DEBUG_PAGEALLOC

The resulting kernel boots and runs just fine (no Oops) and leaves nothing in dmesg.
So, without the debugging options it appears to the user that everything is OK - nasty.

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#25 lore raw
In reply to Linus Torvalds
Quoted 1 line · Linus Torvalds

[.snip.]

Quoted 13 lines

That's not the one to change. It's the one in "sr_do_ioctl()", where it uses "sizeof(*sense)".

Linus

----

drivers/scsi/sr_ioctl.c +0 -0
index 5d02ff4..b65462f 100644

drivers/scsi/sr_ioctl.c int sr_do_ioctl(Scsi_CD *cd, struct pack · line 198

Quoted 8 lines
drivers/scsi/sr_ioctl.c +1 -1
@@ -192,7 +192,7 @@ int sr_do_ioctl(Scsi_CD *cd, struct pack
        SDev = cd->device;
 
        if (!sense) {
-               sense = kmalloc(sizeof(*sense), GFP_KERNEL);
+               sense = kmalloc(SCSI_SENSE_BUFFERSIZE, GFP_KERNEL);
                if (!sense) {
                        err = -ENOMEM;
                        goto out;

Ok, booting a plain 2.6.16-rc5-mm2 kernel with the above being the only change made results in this :

Slab corruption: start=f4f6a11c, len=128
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270) 000: 70 00 02 00 00 00 00 0a 00 00 00 00 3a 01 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f4f6a090, len=128
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c01f4a26>](alloc_as_io_context+0x16/0xd0) 000: 01 00 00 00 00 00 00 00 ad 4e ad de ff ff ff ff 010: ff ff ff ff b0 49 1f c0 c0 49 1f c0 07 00 00 00 Next obj: start=f4f6a1a8, len=128
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c01f4a26>](alloc_as_io_context+0x16/0xd0) 000: 01 00 00 00 00 00 00 00 ad 4e ad de ff ff ff ff 010: ff ff ff ff b0 49 1f c0 c0 49 1f c0 07 00 00 00 Slab corruption: start=f4f6a11c, len=128
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934eb>](sr_do_ioctl+0x11b/0x270) 000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f4f6a090, len=128
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c01f4a26>](alloc_as_io_context+0x16/0xd0) 000: 01 00 00 00 00 00 00 00 ad 4e ad de ff ff ff ff 010: ff ff ff ff b0 49 1f c0 c0 49 1f c0 07 00 00 00 Next obj: start=f4f6a1a8, len=128
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c01f4a26>](alloc_as_io_context+0x16/0xd0) 000: 01 00 00 00 00 00 00 00 ad 4e ad de ff ff ff ff 010: ff ff ff ff b0 49 1f c0 c0 49 1f c0 07 00 00 00

Where do we go from here ?

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#12 lore raw
In reply to Jesper Juhl
Quoted 9 lines · Jesper Juhl

CONFIG_DEBUG_SLAB
CONFIG_PAGE_OWNER
CONFIG_DEBUG_VM
CONFIG_DEBUG_PAGEALLOC

The resulting kernel boots and runs just fine (no Oops) and leaves nothing in dmesg.
So, without the debugging options it appears to the user that everything is OK - nasty.

DEBUG_PAGEALLOC in particular is *fantastic* at making bugs hide. I've lost many an hour trying to pin bugs down due to that.

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#13 lore raw
Quoted 7 lines · Dave Jones
Quoted 5 lines · Jesper Juhl

CONFIG_DEBUG_SLAB
CONFIG_PAGE_OWNER
CONFIG_DEBUG_VM
CONFIG_DEBUG_PAGEALLOC

Quoted 8 lines
Quoted 4 lines

The resulting kernel boots and runs just fine (no Oops) and leaves nothing in dmesg.
So, without the debugging options it appears to the user that everything is OK - nasty.

DEBUG_PAGEALLOC in particular is *fantastic* at making bugs hide. I've lost many an hour trying to pin bugs down due to that.

Well, in this case, turning the option *off* hides the bug ;)

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#29 lore raw
In reply to Jesper Juhl
Quoted 3 lines · Jesper Juhl

Ok, booting a plain 2.6.16-rc5-mm2 kernel with the above being the only change made results in this :

Yeah. I'm not surprised. A real mode-sense shouldn't be even 64 bytes, much less 96, so it shouldn't have overflowed, and we had no indication of the corruption spreading past the one allocation anyway.

It did/does seem a bug, though, so worth checking.

So onward in our tireless battle. Does this patch make any difference for you? It does two things:

- it clears the "->sense" buffer in blk_end_sync_rq() (since it won't be valid any more: the request is gone)
- it adds a BUG_ON() if we appear to have already done the sense fill on SCSI IO completion, and do it again.

Now, I've not tried either of these, and the BUG_ON() in particular might be a false positive itself, but it might be worth testing.

Linus

---

block/ll_rw_blk.c +1 -0
index 03d9c82..4351d34 100644
@@ -2637,6 +2637,7 @@ void blk_end_sync_rq(struct request *rq,
 	struct completion *waiting = rq->waiting;
 
 	rq->waiting = NULL;
+	rq->sense = NULL;
 	__blk_put_request(rq->q, rq);
 
 	/*
drivers/scsi/scsi_lib.c +4 -0
index 701a328..2b60769 100644
@@ -961,6 +961,10 @@ void scsi_io_completion(struct scsi_cmnd
 		if (result) {
 			clear_errors = 0;
 			if (sense_valid && req->sense) {
+
+				/* Have we already filled the sense buffer? */
+				BUG_ON(req->sense_len);
+
 				/*
 				 * SG_IO wants current and deferred errors
 				 */
#28 lore raw
In reply to Jesper Juhl
Quoted 6 lines · Jesper Juhl
Quoted 5 lines · Andrew Morton
Quoted 3 lines · Jesper Juhl

Where do we go from here ?

Quoted 9 lines
Quoted 6 lines

If you can test just

        2.6.16-rc5 + linus.patch + git-scsi-misc.patch

then we'd have a clearer idea.

Sure, I'll get right on it.
I'll post the results in 15min or so.

Ok, a plain 2.6.15-rc5 + linus.patch + git-scsi-misc.patch results in this :

Slab corruption: start=f4812d14, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c028f61b>](sr_do_ioctl+0x11b/0x270) 000: 70 00 02 00 00 00 00 0a 00 00 00 00 3a 01 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f4812cc8, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<00000000>](_stext+0x3feffd68/0x8) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Next obj: start=f4812d60, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c02367ef>](init_dev+0x5cf/0x630) 000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Slab corruption: start=f4812d14, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c028f61b>](sr_do_ioctl+0x11b/0x270) 000: 70 00 05 00 00 00 00 0a 00 00 00 00 24 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f4812cc8, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<00000000>](_stext+0x3feffd68/0x8) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Next obj: start=f4812d60, len=64
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c02367ef>](init_dev+0x5cf/0x630) 000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#30 lore raw
In reply to Linus Torvalds

Ok,
I have a new favorite suspect.

It is this one: commit 4d268eba1187ef66844a6a33b9431e5d0dadd4ad:

[PATCH] slab: extract slab order calculation to separate function

This patch moves the ugly loop that determines the 'optimal' size (page order) of cache slabs from kmem_cache_create() to a separate function and cleans it up a bit.

Thanks to Matthew Wilcox for the help with this patch.

and I think it may be broken.

In particular, as far as I can tell, that

+ /* More than offslab_limit objects will cause problems */
+ if (flags & CFLGS_OFF_SLAB && cachep->num > offslab_limit)
+ break;

has been incorrectly translated for several reasons:

- we shouldn't check "cachep->num > offslab_limit". We should check just "num > offslab_limit" (cachep->num is the _previous_ number we tested).

- when we do "break", we've already incremented "gfporder", and we should correct for that.

Now, maybe I'm just off my rocker again (I've certainly been batting 0.000 so far, even if I think I've been finding real bugs). So who knows. But I get the feeling that that patch is broken.

Either revert it, or try this (TOTALLY UNTESTED!!!) patch..

And hey, maybe I'm just crazy.

Linus

----

mm/slab.c +6 -9
index 2b0b151..1cca41d 100644
@@ -1628,25 +1628,22 @@ static inline size_t calculate_slab_orde
 			size_t size, size_t align, unsigned long flags)
 {
 	size_t left_over = 0;
+	int gfporder;
 
-	for (;; cachep->gfporder++) {
+	for (gfporder = 0 ; gfporder < MAX_GFP_ORDER; gfporder++) {
 		unsigned int num;
 		size_t remainder;
 
-		if (cachep->gfporder > MAX_GFP_ORDER) {
-			cachep->num = 0;
-			break;
-		}
-
-		cache_estimate(cachep->gfporder, size, align, flags,
-			       &remainder, &num);
+		cache_estimate(gfporder, size, align, flags, &remainder, &num);
 		if (!num)
 			continue;
+
 		/* More than offslab_limit objects will cause problems */
-		if (flags & CFLGS_OFF_SLAB && cachep->num > offslab_limit)
+		if ((flags & CFLGS_OFF_SLAB) && num > offslab_limit)
 			break;
 
 		cachep->num = num;
+		cachep->gfporder = gfporder;
 		left_over = remainder;
 
 		/*
#46 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds
Quoted 1 line · Jesper Juhl
Quoted 9 lines
Quoted 2 lines

Ok, booting a plain 2.6.16-rc5-mm2 kernel with the above being the only change made results in this :

Yeah. I'm not surprised. A real mode-sense shouldn't be even 64 bytes, much less 96, so it shouldn't have overflowed, and we had no indication of the corruption spreading past the one allocation anyway.

It did/does seem a bug, though, so worth checking.

Well, hopefully the SCSI people can take a look at that as a seperate issue...

Quoted 11 lines

So onward in our tireless battle. Does this patch make any difference for you? It does two things:

- it clears the "->sense" buffer in blk_end_sync_rq() (since it won't be valid any more: the request is gone)
- it adds a BUG_ON() if we appear to have already done the sense fill on SCSI IO completion, and do it again.

Now, I've not tried either of these, and the BUG_ON() in particular might be a false positive itself, but it might be worth testing.

Unfortunately this doesn't seem to make a difference :

Slab corruption: start=f70c0770, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934fb>](sr_do_ioctl+0x11b/0x270) 000: 70 00 02 00 00 00 00 0a 00 00 00 00 3a 01 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f70c0724, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01813e6>](free_fdtable_rcu+0x66/0x150) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Next obj: start=f70c07bc, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01813ee>](free_fdtable_rcu+0x6e/0x150) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Slab corruption: start=f70c0770, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02934fb>](sr_do_ioctl+0x11b/0x270) 000: 70 00 05 00 00 00 00 0a 00 00 00 00 24 00 00 00 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Prev obj: start=f70c0724, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01813e6>](free_fdtable_rcu+0x66/0x150) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Next obj: start=f70c07bc, len=64
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c01813ee>](free_fdtable_rcu+0x6e/0x150) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#31 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

Ok,
I have a new favorite suspect.

Heh, you are coming up with stuff to test faster than I can build & boot kernels ;-)
Which is good, we'll get to the bottom of this all the faster :)

Quoted 2 lines

It is this one: commit 4d268eba1187ef66844a6a33b9431e5d0dadd4ad:

[--snip--]

Quoted 7 lines

Now, maybe I'm just off my rocker again (I've certainly been batting 0.000 so far, even if I think I've been finding real bugs). So who knows. But I get the feeling that that patch is broken.

Either revert it, or try this (TOTALLY UNTESTED!!!) patch..

Hmm, that patch doesn't apply at all to 2.6.16-rc5-mm2 :/

patching file mm/slab.c
Hunk #1 FAILED at 1628.
1 out of 1 hunk FAILED -- saving rejects to file mm/slab.c.rej

$ cat mm/slab.c.rej
***************
*** 1628,1649 ****
  			size_t size, size_t align, unsigned long flags)
  {
  	size_t left_over = 0;
- 	int gfporder;

- for (gfporder = 0 ; gfporder < MAX_GFP_ORDER; gfporder++) {
unsigned int num;
size_t remainder;

- cache_estimate(gfporder, size, align, flags, &remainder, &num);
if (!num)
continue;
-
/* More than offslab_limit objects will cause problems */
- if ((flags & CFLGS_OFF_SLAB) && num > offslab_limit)
break;

cachep->num = num;
- cachep->gfporder = gfporder;
left_over = remainder;

/*
--- 1628,1652 ----
size_t size, size_t align, unsigned long flags)
{
size_t left_over = 0;

+ for (;; cachep->gfporder++) {
unsigned int num;
size_t remainder;

+ if (cachep->gfporder > MAX_GFP_ORDER) {
+ cachep->num = 0;
+ break;
+ }
+
+ cache_estimate(cachep->gfporder, size, align, flags,
+ &remainder, &num);
if (!num)
continue;
/* More than offslab_limit objects will cause problems */
+ if (flags & CFLGS_OFF_SLAB && cachep->num > offslab_limit)
break;

  		cachep->num = num;
  		left_over = remainder;

/*

Quoted 2 lines

And hey, maybe I'm just crazy.

Somehow I don't think that's the core problem here ;)

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#32 lore raw
In reply to Linus Torvalds
Quoted 2 lines · Linus Torvalds

Either revert it, or try this (TOTALLY UNTESTED!!!) patch..

Don't even bother with the untested patch.

mm/slab.c static inline size_t calculate_slab_orde · line 1633

Quoted 1 line
mm/slab.c +1 -0
+	for (gfporder = 0 ; gfporder < MAX_GFP_ORDER; gfporder++) {

At a minimum, this "<" needs to be "<=".

After that, it might even work. Not that I can convince me that the test for "offslab_limit" ever even triggers, so..

Linus

#33 lore raw
Quoted 6 lines · Linus Torvalds
Quoted 2 lines · Linus Torvalds

Either revert it, or try this (TOTALLY UNTESTED!!!) patch..

Quoted 9 lines

Don't even bother with the untested patch.

Quoted 1 line

+ for (gfporder = 0 ; gfporder < MAX_GFP_ORDER; gfporder++) {

At a minimum, this "<" needs to be "<=".

After that, it might even work. Not that I can convince me that the test for "offslab_limit" ever even triggers, so..

Ehh, it's getting pretty clear that you are looking at 2.6.16-rc5-git<latest> and I'm using -mm here, since that code is not present in mm/slab.c in 2.6.16-rc5-mm2 in anything near that form.

And since 2.6.16-rc5-git8 is not experiencing problems I'd suggest you perhaps instead take a look at what's in -mm... That's where we need to work (it seems) to find the bug...

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#34 lore raw
Quoted 4 lines · Jesper Juhl

And since 2.6.16-rc5-git8 is not experiencing problems I'd suggest you
perhaps instead take a look at what's in -mm... That's where we need to work (it seems) to find the bug...

Yes, it's very probably something in git-scsi-misc.

 drivers/block/cciss.c               |    3 
 drivers/message/fusion/Kconfig      |    1 
 drivers/message/fusion/mptbase.c    |   72 -
 drivers/message/fusion/mptbase.h    |   14 
 drivers/message/fusion/mptfc.c      |    4 
 drivers/message/fusion/mptlan.c     |    5 
 drivers/message/fusion/mptsas.c     |  196 ++
 drivers/message/fusion/mptscsih.c   | 2402 +-----------------------------------
 drivers/message/fusion/mptscsih.h   |   11 
 drivers/message/fusion/mptspi.c     |  733 ++++++++++
 drivers/scsi/53c700.c               |   18 
 drivers/scsi/aacraid/aacraid.h      |    5 
 drivers/scsi/aacraid/comminit.c     |    1 
 drivers/scsi/aacraid/commsup.c      |   14 
 drivers/scsi/aacraid/linit.c        |   14 
 drivers/scsi/aha152x.c              |    7 
 drivers/scsi/aic7xxx/aic79xx_core.c |   24 
 drivers/scsi/aic7xxx/aic7xxx_core.c |   24 
 drivers/scsi/aic7xxx/aic7xxx_osm.c  |   45 
 drivers/scsi/aic7xxx/aic7xxx_osm.h  |    5 
 drivers/scsi/hosts.c                |    3 
 drivers/scsi/ipr.c                  |  109 +
 drivers/scsi/ips.c                  |    2 
 drivers/scsi/jazz_esp.c             |   19 
 drivers/scsi/lpfc/lpfc.h            |   40 
 drivers/scsi/lpfc/lpfc_attr.c       |  162 +-
 drivers/scsi/lpfc/lpfc_crtn.h       |   28 
 drivers/scsi/lpfc/lpfc_ct.c         |   74 -
 drivers/scsi/lpfc/lpfc_disc.h       |   19 
 drivers/scsi/lpfc/lpfc_els.c        |  772 +++++++----
 drivers/scsi/lpfc/lpfc_hbadisc.c    |  492 +++----
 drivers/scsi/lpfc/lpfc_hw.h         |   65 
 drivers/scsi/lpfc/lpfc_init.c       |  265 ++-
 drivers/scsi/lpfc/lpfc_mbox.c       |   33 
 drivers/scsi/lpfc/lpfc_nportdisc.c  |  374 +++--
 drivers/scsi/lpfc/lpfc_scsi.c       |   25 
 drivers/scsi/lpfc/lpfc_scsi.h       |    5 
 drivers/scsi/lpfc/lpfc_sli.c        |  385 +++--
 drivers/scsi/lpfc/lpfc_sli.h        |    5 
 drivers/scsi/lpfc/lpfc_version.h    |    6 
 drivers/scsi/ncr53c8xx.c            |  127 -
 drivers/scsi/ncr53c8xx.h            |   37 
 drivers/scsi/osst.c                 |   24 
 drivers/scsi/qla2xxx/qla_def.h      |    6 
 drivers/scsi/qla2xxx/qla_gbl.h      |    2 
 drivers/scsi/qla2xxx/qla_isr.c      |    7 
 drivers/scsi/qla2xxx/qla_mbx.c      |    4 
 drivers/scsi/qla2xxx/qla_os.c       |   89 -
 drivers/scsi/qla2xxx/qla_sup.c      |    4 
 drivers/scsi/scsi.c                 |    6 
 drivers/scsi/scsi_debug.c           |    9 
 drivers/scsi/scsi_ioctl.c           |    3 
 drivers/scsi/scsi_lib.c             |   76 -
 drivers/scsi/scsi_scan.c            |  100 -
 drivers/scsi/scsi_sysfs.c           |    4 
 drivers/scsi/scsi_transport_fc.c    |    9 
 drivers/scsi/scsi_transport_iscsi.c |    3 
 drivers/scsi/scsi_transport_sas.c   |  258 +++
 drivers/scsi/scsi_transport_spi.c   |   83 -
 drivers/scsi/sd.c                   |   11 
 drivers/scsi/sg.c                   |   16 
 drivers/scsi/sr.c                   |    5 
 drivers/scsi/sr_ioctl.c             |    6 
 drivers/scsi/st.c                   |   29 
 drivers/scsi/sym53c8xx_2/sym_hipd.c |   53 
 include/linux/workqueue.h           |    6 
 include/scsi/scsi.h                 |    2 
 include/scsi/scsi_cmnd.h            |   20 
 include/scsi/scsi_device.h          |   16 
 include/scsi/scsi_transport_sas.h   |   22 
 include/scsi/scsi_transport_spi.h   |    4 
 kernel/workqueue.c                  |   29 
 72 files changed, 3444 insertions(+), 4107 deletions(-)
#35 lore raw
Quoted 8 lines · Andrew Morton
Quoted 4 lines · Jesper Juhl

And since 2.6.16-rc5-git8 is not experiencing problems I'd suggest you
perhaps instead take a look at what's in -mm... That's where we need to work (it seems) to find the bug...

Yes, it's very probably something in git-scsi-misc.

I would say that's correct. I just build 2.6.16-rc5-mm2 with just git-scsi-misc.patch reverted, and that makes the problem go away.

So now the big question is; what part(s) of git-scsi-misc is broken?

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#36 lore raw
Quoted 4 lines · Jesper Juhl
Quoted 2 lines · Andrew Morton
Quoted 1 line · Jesper Juhl
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

And since 2.6.16-rc5-git8 is not experiencing problems I'd suggest you
perhaps instead take a look at what's in -mm... That's where we need to work (it seems) to find the bug...

Yes, it's very probably something in git-scsi-misc.

I would say that's correct. I just build 2.6.16-rc5-mm2 with just git-scsi-misc.patch reverted, and that makes the problem go away.

Ok. I was kind of hoping that it was just a more reliable case of the corruption that Andrew had been seeing too (which seems to be hard to trigger in mainline too, but might exist there).

Quoted 1 line

So now the big question is; what part(s) of git-scsi-misc is broken?

Well, its origin is actually a git tree, so you could try the "git bisect" approach using the

	git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi-misc-2.6.git

tree that the patch comes from..

Linus

#37 lore raw
Quoted 20 lines · Linus Torvalds
Quoted 4 lines · Jesper Juhl
Quoted 2 lines · Andrew Morton
Quoted 1 line · Jesper Juhl
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

And since 2.6.16-rc5-git8 is not experiencing problems I'd suggest you
perhaps instead take a look at what's in -mm... That's where we need to work (it seems) to find the bug...

Yes, it's very probably something in git-scsi-misc.

I would say that's correct. I just build 2.6.16-rc5-mm2 with just git-scsi-misc.patch reverted, and that makes the problem go away.

Ok. I was kind of hoping that it was just a more reliable case of the corruption that Andrew had been seeing too (which seems to be hard to trigger in mainline too, but might exist there).

Quoted 9 lines
Quoted 1 line

So now the big question is; what part(s) of git-scsi-misc is broken?

Well, its origin is actually a git tree, so you could try the "git bisect" approach using the

        git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi-misc-2.6.git

tree that the patch comes from..

I'll give that a go tomorrow - right now I need to get some sleep. If there are other things to try, then just drop me a mail and I'll test it tomorrow.

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#38 lore raw
In reply to Jesper Juhl
Quoted 8 lines · Jesper Juhl
Quoted 6 lines · Andrew Morton
Quoted 3 lines · Jesper Juhl

And since 2.6.16-rc5-git8 is not experiencing problems I'd suggest you perhaps instead take a look at what's in -mm... That's where we need to work (it seems) to find the bug...

Quoted 8 lines
Quoted 2 lines

Yes, it's very probably something in git-scsi-misc.

I would say that's correct. I just build 2.6.16-rc5-mm2 with just git-scsi-misc.patch reverted, and that makes the problem go away.

So now the big question is; what part(s) of git-scsi-misc is broken?

Is it relate to this change?

drivers/scsi/sr_ioctl.c +2 -4
index 5d02ff4..03fbc4b 100644
@@ -44,11 +44,10 @@ static int sr_read_tochdr(struct cdrom_d
        int result;
        unsigned char *buffer;
 
-       buffer = kmalloc(32, GFP_KERNEL | SR_GFP_DMA(cd));
+       buffer = kzalloc(32, GFP_KERNEL | SR_GFP_DMA(cd));
        if (!buffer)
                return -ENOMEM;
 
-       memset(&cgc, 0, sizeof(struct packet_command));         cgc.timeout = IOCTL_TIMEOUT;
        cgc.cmd[0] = GPCMD_READ_TOC_PMA_ATIP;         cgc.cmd[8] = 12;                /* LSB of length */
@@ -74,11 +73,10 @@ static int sr_read_tocentry(struct cdrom
        int result;
        unsigned char *buffer;
 
-       buffer = kmalloc(32, GFP_KERNEL | SR_GFP_DMA(cd));
+       buffer = kzalloc(32, GFP_KERNEL | SR_GFP_DMA(cd));
        if (!buffer)
                return -ENOMEM;
 
-       memset(&cgc, 0, sizeof(struct packet_command));
 
 
 

When someone converted the *buffer* allocation to kzalloc they also removed the the memset for the *packet_cmmand* struct.

The

memset(&cgc, 0, sizeof(struct packet_command));

should be added back I think.

#39 lore raw
Quoted 6 lines · Mike Christie
drivers/scsi/sr_ioctl.c +1 -2
-       buffer = kmalloc(32, GFP_KERNEL | SR_GFP_DMA(cd));
+       buffer = kzalloc(32, GFP_KERNEL | SR_GFP_DMA(cd));
        if (!buffer)
                return -ENOMEM;
 
-       memset(&cgc, 0, sizeof(struct packet_command));

...

Quoted 7 lines
drivers/scsi/sr_ioctl.c +1 -2
-       buffer = kmalloc(32, GFP_KERNEL | SR_GFP_DMA(cd));
+       buffer = kzalloc(32, GFP_KERNEL | SR_GFP_DMA(cd));
        if (!buffer)
                return -ENOMEM;
 
-       memset(&cgc, 0, sizeof(struct packet_command));
Quoted 8 lines

When someone converted the *buffer* allocation to kzalloc they also removed the the memset for the *packet_cmmand* struct.

The

memset(&cgc, 0, sizeof(struct packet_command));

should be added back I think.

Good eyes. I bet that's it.

Linus

#42 lore raw

PATCH 1/1 slab: fix offslab_limit in calculate_slab_order (Was: Slab corruption in 2.6.16-rc5-mm2)

In reply to Linus Torvalds
Quoted 14 lines · Linus Torvalds

In particular, as far as I can tell, that

(quoted diff) +3 -0
+               /* More than offslab_limit objects will cause problems */
+               if (flags & CFLGS_OFF_SLAB && cachep->num > offslab_limit)
+                       break;

has been incorrectly translated for several reasons:

- we shouldn't check "cachep->num > offslab_limit". We should check just "num > offslab_limit" (cachep->num is the _previous_ number we tested).

- when we do "break", we've already incremented "gfporder", and we should correct for that.

Quoted 7 lines

Now, maybe I'm just off my rocker again (I've certainly been batting 0.000 so far, even if I think I've been finding real bugs). So who knows. But I get the feeling that that patch is broken.

Either revert it, or try this (TOTALLY UNTESTED!!!) patch..

And hey, maybe I'm just crazy.

No you're not, it's broken. However, I think you're forgetting to reset cachep->num when we go over MAX_GFP_ORDER, no? This patch boots although I don't hit offslab limit.

Pekka

[PATCH] slab: fix offslab_limit in calculate_slab_order

From: Linus Torvalds <[email protected]>

The commit "[PATCH] slab: extract slab order calculation to separate function" broke offslab_limit handling:

- We should check for num instead of cachep->num because the latter is the number of objects for the _previous_ gfp order.

- After hitting the offslab_limit, we need to correct gfporder and calculate left_over and cachep->num for that before exiting. We do this by keeping current state in local variables and previous state in cachep.

Linus spotted the problem and wrote the patch. I fixed gfporder resetting when we go over MAX_GFP_ORDER and tested it with UM.

---

mm/slab.c +9 -9
index add05d8..fe63479 100644
@@ -1628,25 +1628,22 @@ static inline size_t calculate_slab_orde
 			size_t size, size_t align, unsigned long flags)
 {
 	size_t left_over = 0;
+	int gfporder;
 
-	for (;; cachep->gfporder++) {
+	for (gfporder = 0 ; gfporder <= MAX_GFP_ORDER; gfporder++) {
 		unsigned int num;
 		size_t remainder;
 
-		if (cachep->gfporder > MAX_GFP_ORDER) {
-			cachep->num = 0;
-			break;
-		}
-
-		cache_estimate(cachep->gfporder, size, align, flags,
-			       &remainder, &num);
+		cache_estimate(gfporder, size, align, flags, &remainder, &num);
 		if (!num)
 			continue;
+
 		/* More than offslab_limit objects will cause problems */
-		if (flags & CFLGS_OFF_SLAB && cachep->num > offslab_limit)
+		if ((flags & CFLGS_OFF_SLAB) && num > offslab_limit)
 			break;
 
 		cachep->num = num;
+		cachep->gfporder = gfporder;
 		left_over = remainder;
 
 		/*
@@ -1660,6 +1657,9 @@ static inline size_t calculate_slab_orde
 			/* Acceptable internal fragmentation */
 			break;
 	}
+	if (cachep->gfporder > MAX_GFP_ORDER)
+		cachep->num = 0;
+
 	return left_over;
 }
 
#43 lore raw

Re: [PATCH] slab: fix offslab_limit in calculate_slab_order (Was: Slab corruption in 2.6.16-rc5-mm2)

Quoted 3 lines · Pekka J Enberg

No you're not, it's broken. However, I think you're forgetting to reset cachep->num when we go over MAX_GFP_ORDER, no?

No, we only ever set "cachep->num" for something that we've decided is valid.

"gfporder" can never be > MAX_GFP_ORDER inside the loop, because we just iterate between 0..MAX_GFP_ORDER.

Linus

#40 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds
Quoted 1 line

should be added back I think.

Good eyes. I bet that's it.

Yes, well done. Do we have confirmation yet that reversing this fixes the bug?

I think a full reversal is in order, since buffer is a quantity being written to, there's no point in zeroing it.

[Note to self: must do better in checking janitors patches]

James

#44 lore raw

Re: [PATCH] slab: fix offslab_limit in calculate_slab_order (Was: Slab corruption in 2.6.16-rc5-mm2)

In reply to Linus Torvalds
Quoted 2 lines · Pekka J Enberg
Quoted 2 lines

No you're not, it's broken. However, I think you're forgetting to reset cachep->num when we go over MAX_GFP_ORDER, no?

Quoted 5 lines · Linus Torvalds

No, we only ever set "cachep->num" for something that we've decided is valid.

"gfporder" can never be > MAX_GFP_ORDER inside the loop, because we just iterate between 0..MAX_GFP_ORDER.

I don't think that's true. We set cachep->num to something we think is valid but check for internal fragmentation later. So I think we can get out of the loop with cachep->num initialized to non-zero but gfporder set to MAX_GFP_ORDER, no? Or did I forget to take my medicine this morning...?

Pekka

#41 lore raw
In reply to James Bottomley
Quoted 8 lines · James Bottomley
Quoted 3 lines · Linus Torvalds
Quoted 1 line

should be added back I think.

Good eyes. I bet that's it.

Yes, well done. Do we have confirmation yet that reversing this fixes the bug?

I just tried reverting that bit only from 2.6.16-rc5-mm2, and it does indeed fix the problem.

Thanks for spotting that Mike.

--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

#45 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds

has been incorrectly translated for several reasons:

- we shouldn't check "cachep->num > offslab_limit". We should check just "num > offslab_limit" (cachep->num is the _previous_ number we tested).

- when we do "break", we've already incremented "gfporder", and we should correct for that.

Now, maybe I'm just off my rocker again (I've certainly been batting 0.000 so far, even if I think I've been finding real bugs). So who knows. But I get the feeling that that patch is broken.

I thought stumbling over bugs while looking for other things was part of the new development model ;-)

Quoted 4 lines

Either revert it, or try this (TOTALLY UNTESTED!!!) patch..

And hey, maybe I'm just crazy.

Being crazy and being right are not mutually exclusize.

Signature · 3 lines
   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me
#14 lore raw
In reply to Dave Jones
Quoted 7 lines · Dave Jones
Quoted 5 lines · Jesper Juhl

CONFIG_DEBUG_SLAB
CONFIG_PAGE_OWNER
CONFIG_DEBUG_VM
CONFIG_DEBUG_PAGEALLOC

Quoted 7 lines
Quoted 4 lines

The resulting kernel boots and runs just fine (no Oops) and leaves nothing in dmesg.
So, without the debugging options it appears to the user that everything is OK - nasty.

DEBUG_PAGEALLOC in particular is *fantastic* at making bugs hide. I've lost many an hour trying to pin bugs down due to that.

Is this backwards? We're saying DEBUG_PAGEALLOC is bad?

OK, what I'm going to try to do, given the recent comments re CONFIG_DEBUG_SLAB and also PAGEALLOC is to arrange with Andy to run a debug kernel as well as a normal kernel for every test, and then we can publish the results on http://test.kernel.org. For now it'll be a seperate matrix, until I work out how to fold the 3d cube nicely into 2d - I know I have to do that anyway, so no big deal.

Do we NOT want to have DEBUG_SLAB and DEBUG_PAGEALLOC both enabled? Running multiple permutations is going to get really painful on the systems involved. Any other requests for what gets enabled (I really want to just stick to one 'debug' setup if possible).

I have no idea why I didn't do this a year ago <slaps self>.

M.

#15 lore raw
Quoted 13 lines
Quoted 2 lines

DEBUG_PAGEALLOC in particular is *fantastic* at making bugs hide. I've lost many an hour trying to pin bugs down due to that.

Is this backwards? We're saying DEBUG_PAGEALLOC is bad?

OK, what I'm going to try to do, given the recent comments re CONFIG_DEBUG_SLAB and also PAGEALLOC is to arrange with Andy to run a debug kernel as well as a normal kernel for every test, and then we can publish the results on http://test.kernel.org. For now it'll be a seperate matrix, until I work out how to fold the 3d cube nicely into 2d - I know I have to do that anyway, so no big deal.

Quoted 6 lines

Do we NOT want to have DEBUG_SLAB and DEBUG_PAGEALLOC both enabled? Running multiple permutations is going to get really painful on the systems involved. Any other requests for what gets enabled (I really want to just stick to one 'debug' setup if possible).

I have no idea why I didn't do this a year ago <slaps self>.

Oh, and I ported CONFIG_DEBUG_PAGEALLOC to x86_64 last week. Will send out the patch later today.

M.

#16 lore raw
In reply to Martin J. Bligh
Quoted 4 lines · Martin J. Bligh

Do we NOT want to have DEBUG_SLAB and DEBUG_PAGEALLOC both enabled? Running multiple permutations is going to get really painful on the systems involved. Any other requests for what gets enabled (I really want to just stick to one 'debug' setup if possible).

Debug kernels are incredibly slow, making hitting certain races next to impossible. By all means non-DEBUG kernels should definately be getting tested.

-ben

Signature · 2 lines
"Time is of no importance, Mr. President, only life is important."
Don't Email: <[email protected]>.
#17 lore raw
Quoted 9 lines · Benjamin LaHaise
Quoted 4 lines · Martin J. Bligh

Do we NOT want to have DEBUG_SLAB and DEBUG_PAGEALLOC both enabled? Running multiple permutations is going to get really painful on the systems involved. Any other requests for what gets enabled (I really want to just stick to one 'debug' setup if possible).

Debug kernels are incredibly slow, making hitting certain races next to impossible. By all means non-DEBUG kernels should definately be getting tested.

It'd still run a totally non-debug kernel as well - I want that for the perf tests etc anyway. I guess the question is whether the debug kernels should have most of the DEBUG_* turned on, or just a select few.

M.

#18 lore raw
In reply to Martin J. Bligh
Quoted 5 lines · Martin J. Bligh
Quoted 3 lines

DEBUG_PAGEALLOC in particular is *fantastic* at making bugs hide. I've lost many an hour trying to pin bugs down due to that.

Is this backwards? We're saying DEBUG_PAGEALLOC is bad?

DEBUG_PAGEALLOC is great for finding the really stupid kinds of bugs, and it's definitely worth doing every once in a while.

However, DEBUG_PAGEALLOC makes many things orders of magnitude slower, and it eats memory like mad (because it turns some slabs into whole pages - but it still doesn't help small allocation debugging that much). So unlike DEBUG_SLAB, it's not reasonable to have it on all the time.

IOW, DEBUG_SLAB is something that a distro kernel can reasonably enable for users by default (I think fedora-devel does, for example). In contrast, DEBUG_PAGEALLOC is more of a "useful for special cases" thing, where you want to validate that there's nothing _obviously_ bad going on.

Quoted 1 line

Do we NOT want to have DEBUG_SLAB and DEBUG_PAGEALLOC both enabled?

I suspect that once DEBUG_PAGEALLOC is on, whether you do DEBUG_SLAB or not is a toss-up. The interesting cases tend to be

- neither: usable for benchmarking
- DEBUG_SLAB: perfectly usable for normal work
- DEBUG_PAGEALLOC (with or without DEBUG_SLAB): debugging tool only

At least that's my opinion, maybe others have other experiences.

Linus

#19 lore raw
Quoted 2 lines · Linus Torvalds

IOW, DEBUG_SLAB is something that a distro kernel can reasonably enable for users by default (I think fedora-devel does, for example).

Correct.

Quoted 5 lines

- neither: usable for benchmarking
- DEBUG_SLAB: perfectly usable for normal work
- DEBUG_PAGEALLOC (with or without DEBUG_SLAB): debugging tool only

At least that's my opinion, maybe others have other experiences.

That's pretty much my experience. I get people screaming at me when I enable PAGEALLOC for a day or so in the Fedora-devel kernel if I'm chasing something :)

Dave

Signature · 1 line
http://www.codemonkey.org.uk