Skip to contentSkip to conversation

cdrom: use kmalloced buffers instead of buffers on stack

DISCUSSION Jeff Garzik 5 participants 10 messages Apr 22, 2008 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 15 lines · Linux Kernel Mailing List
Gitweb:     http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=22a9189fd073db3d03a4cf8b8c098aa207602de1
Commit:     22a9189fd073db3d03a4cf8b8c098aa207602de1
Parent:     0a0c4114df4a6903bccb65b06cabb6ddc968f877
Author:     Thomas Bogendoerfer <[email protected]>
AuthorDate: Wed Mar 26 12:09:38 2008 +0100
Committer:  Jens Axboe <[email protected]>
CommitDate: Mon Apr 21 09:50:08 2008 +0200

cdrom: use kmalloced buffers instead of buffers on stack

If cdrom commands are issued to a scsi drive in most cases the buffer will be filled via dma. This leads to bad stack corruption on non coherent platforms, because the buffers are neither cache line aligned nor is the size a multiple of the cache line size. Using kmalloced buffers avoids this.

Quoted 5 lines
---
 drivers/cdrom/cdrom.c |  274 ++++++++++++++++++++++++++++++++-----------------
 1 files changed, 181 insertions(+), 93 deletions(-)

Eh... AFAICS this is only really useful in two of the cases converted.

For all the other cases (<= 32 bytes), it is _far_ less complex, far less code to simply communicate the additional alignment requirements to the compiler.

What about __attribute__ __aligned__? Was that tried?

Jeff

#2 lore raw

From: Jeff Garzik <[email protected]>
Date: Mon, 21 Apr 2008 22:01:26 -0400

Quoted 7 lines

Eh... AFAICS this is only really useful in two of the cases converted.

For all the other cases (<= 32 bytes), it is _far_ less complex, far less code to simply communicate the additional alignment requirements to the compiler.

What about __attribute__ __aligned__? Was that tried?

On some platforms the stack is virtually mapped instead of from kmalloc or get_free_pages memory.

DMA on stack buffers is really not feasible.

#3 lore raw
In reply to Jeff Garzik
Quoted 17 lines · Jeff Garzik
Quoted 10 lines

cdrom: use kmalloced buffers instead of buffers on stack

If cdrom commands are issued to a scsi drive in most cases the buffer will be
filled via dma. This leads to bad stack corruption on non coherent platforms,
because the buffers are neither cache line aligned nor is the size a multiple
of the cache line size. Using kmalloced buffers avoids this.

Quoted 6 lines

---
drivers/cdrom/cdrom.c | 274
++++++++++++++++++++++++++++++++-----------------

1 files changed, 181 insertions(+), 93 deletions(-)
Quoted 7 lines

Eh... AFAICS this is only really useful in two of the cases converted.

For all the other cases (<= 32 bytes), it is _far_ less complex, far less code to simply communicate the additional alignment requirements to the compiler.

What about __attribute__ __aligned__? Was that tried?

I used that while narrowing down the bug. But not only the alignment is important, but also size needs to be a multiple of the cache line size. Which means it needs to be 128 bytes for most SGI machines. That and the following in DMA-mapping.txt

"This rule also means that you may use neither kernel image addresses (items in data/text/bss segments), nor module image addresses, nor stack addresses for DMA."

let me choose the kmalloc() solution.

Thomas.

Signature · 2 lines
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]
#4 lore raw
Quoted 31 lines · Thomas Bogendoerfer
Quoted 17 lines · Jeff Garzik
Quoted 10 lines

cdrom: use kmalloced buffers instead of buffers on stack

If cdrom commands are issued to a scsi drive in most cases the buffer will be
filled via dma. This leads to bad stack corruption on non coherent platforms,
because the buffers are neither cache line aligned nor is the size a multiple
of the cache line size. Using kmalloced buffers avoids this.

Quoted 6 lines

---
drivers/cdrom/cdrom.c | 274
++++++++++++++++++++++++++++++++-----------------

1 files changed, 181 insertions(+), 93 deletions(-)
Quoted 7 lines

Eh... AFAICS this is only really useful in two of the cases converted.

For all the other cases (<= 32 bytes), it is _far_ less complex, far less code to simply communicate the additional alignment requirements to the compiler.

What about __attribute__ __aligned__? Was that tried?

I used that while narrowing down the bug. But not only the alignment is important, but also size needs to be a multiple of the cache line size. Which means it needs to be 128 bytes for most SGI machines. That and the following in DMA-mapping.txt

Quoted 5 lines

"This rule also means that you may use neither kernel image addresses (items in data/text/bss segments), nor module image addresses, nor stack addresses for DMA."

let me choose the kmalloc() solution.

Which is good, the patch has been due for years :-)

Signature · 1 line
Jens Axboe
#5 lore raw
In reply to Thomas Bogendoerfer
Quoted 31 lines · On Tue, 22 Apr 2008 07:48:58 +0200 [email protected] (Thomas…
Quoted 17 lines · Jeff Garzik
Quoted 10 lines

cdrom: use kmalloced buffers instead of buffers on stack

If cdrom commands are issued to a scsi drive in most cases the buffer will be
filled via dma. This leads to bad stack corruption on non coherent platforms,
because the buffers are neither cache line aligned nor is the size a multiple
of the cache line size. Using kmalloced buffers avoids this.

Quoted 6 lines

---
drivers/cdrom/cdrom.c | 274
++++++++++++++++++++++++++++++++-----------------

1 files changed, 181 insertions(+), 93 deletions(-)
Quoted 7 lines

Eh... AFAICS this is only really useful in two of the cases converted.

For all the other cases (<= 32 bytes), it is _far_ less complex, far less code to simply communicate the additional alignment requirements to the compiler.

What about __attribute__ __aligned__? Was that tried?

I used that while narrowing down the bug. But not only the alignment is important, but also size needs to be a multiple of the cache line size. Which means it needs to be 128 bytes for most SGI machines. That and the following in DMA-mapping.txt

Quoted 5 lines

"This rule also means that you may use neither kernel image addresses (items in data/text/bss segments), nor module image addresses, nor stack addresses for DMA."

let me choose the kmalloc() solution.

Can we advertise such architecture's dma restrictions? For example, if we can update dma_pad_mask and dma_alignment in request_queue, blk_rq_map_kern uses a proper bounce buffer for such architectures. Then we can avoid putting extra complexity in uppper drivers such as cdrom.c

#6 lore raw
Quoted 38 lines · FUJITA Tomonori
Quoted 35 lines · On Tue, 22 Apr 2008 07:48:58 +0200 [email protected] (Thomas…
Quoted 17 lines · Jeff Garzik
Quoted 10 lines

cdrom: use kmalloced buffers instead of buffers on stack

If cdrom commands are issued to a scsi drive in most cases the buffer will be
filled via dma. This leads to bad stack corruption on non coherent platforms,
because the buffers are neither cache line aligned nor is the size a multiple
of the cache line size. Using kmalloced buffers avoids this.

Quoted 6 lines

---
drivers/cdrom/cdrom.c | 274
++++++++++++++++++++++++++++++++-----------------

1 files changed, 181 insertions(+), 93 deletions(-)
Quoted 7 lines

Eh... AFAICS this is only really useful in two of the cases converted.

For all the other cases (<= 32 bytes), it is _far_ less complex, far less code to simply communicate the additional alignment requirements to the compiler.

What about __attribute__ __aligned__? Was that tried?

I used that while narrowing down the bug. But not only the alignment is important, but also size needs to be a multiple of the cache line size. Which means it needs to be 128 bytes for most SGI machines. That and the following in DMA-mapping.txt

"This rule also means that you may use neither kernel image addresses (items in data/text/bss segments), nor module image addresses, nor stack addresses for DMA."

Quoted 7 lines
Quoted 1 line

let me choose the kmalloc() solution.

Can we advertise such architecture's dma restrictions? For example, if we can update dma_pad_mask and dma_alignment in request_queue, blk_rq_map_kern uses a proper bounce buffer for such architectures. Then we can avoid putting extra complexity in uppper drivers such as cdrom.c

That would work fine, if cdrom was then also updated to get rid of ->generic_packet() and use the regular queue transport instead. Which would be a VERY nice thing to do anyway, so I'd welcome the effort :-)

Signature · 1 line
Jens Axboe
#7 lore raw
Quoted 47 lines · On Tue, 22 Apr 2008 13:55:19 +0200 Jens Axboe <[email protected]> wrote:
Quoted 38 lines · FUJITA Tomonori
Quoted 35 lines · On Tue, 22 Apr 2008 07:48:58 +0200 [email protected] (Thomas…
Quoted 17 lines · Jeff Garzik
Quoted 10 lines

cdrom: use kmalloced buffers instead of buffers on stack

If cdrom commands are issued to a scsi drive in most cases the buffer will be
filled via dma. This leads to bad stack corruption on non coherent platforms,
because the buffers are neither cache line aligned nor is the size a multiple
of the cache line size. Using kmalloced buffers avoids this.

Quoted 6 lines

---
drivers/cdrom/cdrom.c | 274
++++++++++++++++++++++++++++++++-----------------

1 files changed, 181 insertions(+), 93 deletions(-)
Quoted 7 lines

Eh... AFAICS this is only really useful in two of the cases converted.

For all the other cases (<= 32 bytes), it is _far_ less complex, far less code to simply communicate the additional alignment requirements to the compiler.

What about __attribute__ __aligned__? Was that tried?

I used that while narrowing down the bug. But not only the alignment is important, but also size needs to be a multiple of the cache line size. Which means it needs to be 128 bytes for most SGI machines. That and the following in DMA-mapping.txt

"This rule also means that you may use neither kernel image addresses (items in data/text/bss segments), nor module image addresses, nor stack addresses for DMA."

Quoted 7 lines
Quoted 1 line

let me choose the kmalloc() solution.

Can we advertise such architecture's dma restrictions? For example, if we can update dma_pad_mask and dma_alignment in request_queue, blk_rq_map_kern uses a proper bounce buffer for such architectures. Then we can avoid putting extra complexity in uppper drivers such as cdrom.c

Quoted 3 lines

That would work fine, if cdrom was then also updated to get rid of ->generic_packet() and use the regular queue transport instead. Which would be a VERY nice thing to do anyway, so I'd welcome the effort :-)

sr_packet calls ioctl so it works for SCSI at least?

#8 lore raw
Quoted 50 lines · FUJITA Tomonori
Quoted 47 lines · On Tue, 22 Apr 2008 13:55:19 +0200 Jens Axboe <[email protected]> wrote:
Quoted 45 lines · FUJITA Tomonori
Quoted 31 lines · On Tue, 22 Apr 2008 07:48:58 +0200 [email protected] (Thomas…
Quoted 17 lines · Jeff Garzik
Quoted 10 lines

cdrom: use kmalloced buffers instead of buffers on stack

If cdrom commands are issued to a scsi drive in most cases the buffer will be
filled via dma. This leads to bad stack corruption on non coherent platforms,
because the buffers are neither cache line aligned nor is the size a multiple
of the cache line size. Using kmalloced buffers avoids this.

Quoted 6 lines

---
drivers/cdrom/cdrom.c | 274
++++++++++++++++++++++++++++++++-----------------

1 files changed, 181 insertions(+), 93 deletions(-)
Quoted 7 lines

Eh... AFAICS this is only really useful in two of the cases converted.

For all the other cases (<= 32 bytes), it is _far_ less complex, far less code to simply communicate the additional alignment requirements to the compiler.

What about __attribute__ __aligned__? Was that tried?

I used that while narrowing down the bug. But not only the alignment is important, but also size needs to be a multiple of the cache line size. Which means it needs to be 128 bytes for most SGI machines. That and the following in DMA-mapping.txt

Quoted 5 lines

"This rule also means that you may use neither kernel image addresses (items in data/text/bss segments), nor module image addresses, nor stack addresses for DMA."

let me choose the kmalloc() solution.

Can we advertise such architecture's dma restrictions? For example, if we can update dma_pad_mask and dma_alignment in request_queue, blk_rq_map_kern uses a proper bounce buffer for such architectures. Then we can avoid putting extra complexity in uppper drivers such as cdrom.c

Quoted 5 lines
Quoted 3 lines

That would work fine, if cdrom was then also updated to get rid of ->generic_packet() and use the regular queue transport instead. Which would be a VERY nice thing to do anyway, so I'd welcome the effort :-)

sr_packet calls ioctl so it works for SCSI at least?

Yup, it'll work fine for sr currently, but not ide-cd. But we should just get rid of ->generic_packet() and prepare a request in cdrom.c instead.

Signature · 1 line
Jens Axboe
#9 lore raw
Quoted 55 lines · On Tue, 22 Apr 2008 14:04:38 +0200 Jens Axboe <[email protected]> wrote:
Quoted 54 lines · FUJITA Tomonori
Quoted 50 lines · On Tue, 22 Apr 2008 13:55:19 +0200 Jens Axboe <[email protected]> wrote:
Quoted 38 lines · FUJITA Tomonori
Quoted 35 lines · On Tue, 22 Apr 2008 07:48:58 +0200 [email protected] (Thomas…
Quoted 17 lines · Jeff Garzik
Quoted 10 lines

cdrom: use kmalloced buffers instead of buffers on stack

If cdrom commands are issued to a scsi drive in most cases the buffer will be
filled via dma. This leads to bad stack corruption on non coherent platforms,
because the buffers are neither cache line aligned nor is the size a multiple
of the cache line size. Using kmalloced buffers avoids this.

Quoted 6 lines

---
drivers/cdrom/cdrom.c | 274
++++++++++++++++++++++++++++++++-----------------

1 files changed, 181 insertions(+), 93 deletions(-)
Quoted 7 lines

Eh... AFAICS this is only really useful in two of the cases converted.

For all the other cases (<= 32 bytes), it is _far_ less complex, far less code to simply communicate the additional alignment requirements to the compiler.

What about __attribute__ __aligned__? Was that tried?

I used that while narrowing down the bug. But not only the alignment is important, but also size needs to be a multiple of the cache line size. Which means it needs to be 128 bytes for most SGI machines. That and the following in DMA-mapping.txt

"This rule also means that you may use neither kernel image addresses (items in data/text/bss segments), nor module image addresses, nor stack addresses for DMA."

Quoted 7 lines
Quoted 1 line

let me choose the kmalloc() solution.

Can we advertise such architecture's dma restrictions? For example, if we can update dma_pad_mask and dma_alignment in request_queue, blk_rq_map_kern uses a proper bounce buffer for such architectures. Then we can avoid putting extra complexity in uppper drivers such as cdrom.c

That would work fine, if cdrom was then also updated to get rid of ->generic_packet() and use the regular queue transport instead. Which would be a VERY nice thing to do anyway, so I'd welcome the effort :-)

Quoted 5 lines
Quoted 1 line

sr_packet calls ioctl so it works for SCSI at least?

Yup, it'll work fine for sr currently, but not ide-cd. But we should just get rid of ->generic_packet() and prepare a request in cdrom.c instead.

Ok, I'll see how things work after finishing the work to remove the request structure on the stack.

So are you ok with a patch to make blk_rq_map_kern handle dma padding and alignment propely?

http://marc.info/?l=linux-scsi&m=120860454709078&w=2

#10 lore raw
Quoted 64 lines · FUJITA Tomonori
Quoted 55 lines · On Tue, 22 Apr 2008 14:04:38 +0200 Jens Axboe <[email protected]> wrote:
Quoted 54 lines · FUJITA Tomonori
Quoted 50 lines · On Tue, 22 Apr 2008 13:55:19 +0200 Jens Axboe <[email protected]> wrote:
Quoted 38 lines · FUJITA Tomonori
Quoted 35 lines · On Tue, 22 Apr 2008 07:48:58 +0200 [email protected] (Thomas…
Quoted 17 lines · Jeff Garzik
Quoted 10 lines

cdrom: use kmalloced buffers instead of buffers on stack

If cdrom commands are issued to a scsi drive in most cases the buffer will be
filled via dma. This leads to bad stack corruption on non coherent platforms,
because the buffers are neither cache line aligned nor is the size a multiple
of the cache line size. Using kmalloced buffers avoids this.

Quoted 6 lines

---
drivers/cdrom/cdrom.c | 274
++++++++++++++++++++++++++++++++-----------------

1 files changed, 181 insertions(+), 93 deletions(-)
Quoted 7 lines

Eh... AFAICS this is only really useful in two of the cases converted.

For all the other cases (<= 32 bytes), it is _far_ less complex, far less code to simply communicate the additional alignment requirements to the compiler.

What about __attribute__ __aligned__? Was that tried?

I used that while narrowing down the bug. But not only the alignment is important, but also size needs to be a multiple of the cache line size. Which means it needs to be 128 bytes for most SGI machines. That and the following in DMA-mapping.txt

"This rule also means that you may use neither kernel image addresses (items in data/text/bss segments), nor module image addresses, nor stack addresses for DMA."

Quoted 7 lines
Quoted 1 line

let me choose the kmalloc() solution.

Can we advertise such architecture's dma restrictions? For example, if we can update dma_pad_mask and dma_alignment in request_queue, blk_rq_map_kern uses a proper bounce buffer for such architectures. Then we can avoid putting extra complexity in uppper drivers such as cdrom.c

That would work fine, if cdrom was then also updated to get rid of ->generic_packet() and use the regular queue transport instead. Which would be a VERY nice thing to do anyway, so I'd welcome the effort :-)

Quoted 5 lines
Quoted 1 line

sr_packet calls ioctl so it works for SCSI at least?

Yup, it'll work fine for sr currently, but not ide-cd. But we should just get rid of ->generic_packet() and prepare a request in cdrom.c instead.

Quoted 7 lines

Ok, I'll see how things work after finishing the work to remove the request structure on the stack.

So are you ok with a patch to make blk_rq_map_kern handle dma padding and alignment propely?

http://marc.info/?l=linux-scsi&m=120860454709078&w=2

I'm getting to that patch, from a cursory look it seems fine.

Signature · 1 line
Jens Axboe