DMA memory limitation?

Until recently, on IA-64, pci_alloc_consistent() given a NULL pci_dev would fault. It's been fixed in at least the most recent IA-64 patch. pci_map_single() and pci_map_sg() still have the same problem, as they dereference pci_dev w/o checking for NULL first.

Matt Domsch
Sr. Software Engineer
Dell Linux Solutions
www.dell.com/linux
#2 Linux Server provider with 17% in the US and 14% Worldwide (IDC)!
#3 Unix provider with 18% in the US (Dataquest)!

Is there a patch that fixes the pci_alloc_consistent for older versions of the IA-64 kernels? To do 64bit DMA in my driver I need to pass pci_alloc_consistent pci_dev pointer of NULL.

I'm interested in getting this to work on the older IA64 kernels that are currently on distribution CDs.

From: [email protected]
To: [email protected], [email protected]
Cc: [email protected], [email protected],
[email protected],
[email protected],
[email protected], [email protected]
Date: Fri, 6 Jul 2001 10:56:47 -0500
Subject: [Linux-ia64] RE: DMA memory limitation?

> The important thing is that pci_alloc_consistent and the
other PCI DMA
> functions work as advertised on IA64. If you pass NULL to > pci_alloc_consistent, IA64 should give you an ISA DMA-able

> address. If
> you don't, you get a 32-bit PCI DMA address. Use of
GFP_DMA is a
> arch-specific detail, so don't let me confuse you there.

Until recently, on IA-64, pci_alloc_consistent() given a
NULL pci_dev would
fault. It's been fixed in at least the most recent IA-64
patch.
pci_map_single() and pci_map_sg() still have the same
problem, as they
dereference pci_dev w/o checking for NULL first.

--
Matt Domsch
Sr. Software Engineer
Dell Linux Solutions
www.dell.com/linux
#2 Linux Server provider with 17% in the US and 14%
Worldwide (IDC)!
#3 Unix provider with 18% in the US (Dataquest)!

You're not going to find such a thing, and it wouldn't make sense even if you did. See Jeff Garzik's note again:

Never have I seen a using NULL pci_dev pointer give you a 64-bit DMA address. Per Jeff's comment, it should give you an address with a mask of 24-bits (within ISA space, i.e. <16MB), but since you can't add in any ISA cards on IA-64, this wouldn't make much sense either.

At present, the DMA address is entirely dependent upon pci_dev->dma_mask, which by default is a 32-bit mask, and which you can set to be a 64-bit mask. If you need a 64-bit DMA address, create a new pci_dev structure, set the mask to ~0L and pass that in. (This may cause other problems, I can't say for certain.)

Thanks,
Matt

--
Matt Domsch
Sr. Software Engineer
Dell Linux Solutions
www.dell.com/linux
#2 Linux Server provider with 17% in the US and 14% Worldwide (IDC)! #3 Unix provider with 18% in the US (Dataquest)!

Sorry, just to clarify...
My device has the limitation that the command buffers I send must be in 32 bit space, but the IO can be in 64 bit. But there is only ONE dma_mask for the device. The way I figured out how to do 64 bit DMA (from reading the information in the DMA-mapping.txt) was to set my DMA mask of the pci_dev for my device to allow 64 bit DMA, but when I allocated my command buffers( with pci_alloc_consistent) send a pci_dev of NULL. So I would get the default of memory that was 24bit addressable.

This Seg faulted in earlier versions of the IA64 port, but works great in the latest versions. But now I have been asked if I can get this to work with the kernels that are released IA64 CDs.

So, I was wondering if there was already a patch that would fix this one problem with the some of the older trees.

-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Monday, July 09, 2001 9:47 AM
To: White, Charles
Cc: [email protected]
Subject: RE: [Linux-ia64] RE: DMA memory limitation?

> Is there a patch that fixes the pci_alloc_consistent for > older versions of
> the IA-64 kernels? To do 64bit DMA in my driver I need to
pass
> pci_alloc_consistent pci_dev pointer of NULL.

You're not going to find such a thing, and it wouldn't make
sense even if
you did. See Jeff Garzik's note again:

>> The important thing is that pci_alloc_consistent and the
other PCI DMA
>> functions work as advertised on IA64. If you pass NULL
to
>> pci_alloc_consistent, IA64 should give you an ISA
DMA-able
>> address. If you don't, you get a 32-bit PCI DMA address.

Never have I seen a using NULL pci_dev pointer give you a
64-bit DMA
address. Per Jeff's comment, it should give you an address
with a mask of
24-bits (within ISA space, i.e. <16MB), but since you can't
add in any ISA
cards on IA-64, this wouldn't make much sense either.

At present, the DMA address is entirely dependent upon
pci_dev->dma_mask,
which by default is a 32-bit mask, and which you can set to
be a 64-bit
mask. If you need a 64-bit DMA address, create a new
pci_dev structure, set
the mask to ~0L and pass that in. (This may cause other
problems, I can't
say for certain.)

Thanks,
Matt

--
Matt Domsch
Sr. Software Engineer
Dell Linux Solutions
www.dell.com/linux
#2 Linux Server provider with 17% in the US and 14%
Worldwide (IDC)!
#3 Unix provider with 18% in the US (Dataquest)!

Ahh, yes. The AMI MegaRAID driver/controllers have the same requirements. megaraid solved it as:

pci_find_device() finds their device, they get a pci_dev back which has a 32-bit dma_mask.
pci_alloc_consistent() the mailbox (this must be in 32-bit space). if it's a 64-bit-capable controller, set pci_dev->dma_mask = ~0L. pci_alloc_consistent() the command buffers and scatter-gather lists (these can be in 64-bit space).

Could something similar work for you?

Thanks,
Matt

--
Matt Domsch
Sr. Software Engineer
Dell Linux Solutions
www.dell.com/linux
#2 Linux Server provider with 17% in the US and 14% Worldwide (IDC)! #3 Unix provider with 18% in the US (Dataquest)!

Yes, I thought about doing it that way, and it COULD be done, but right now the driver allocates command buffer for IOCTLs as needed.

So the pci_dev set to NULL allows to continue to do that...

-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Monday, July 09, 2001 10:20 AM
To: White, Charles
Cc: [email protected]
Subject: RE: [Linux-ia64] RE: DMA memory limitation?

> My device has the limitation that the command buffers I
send
> must be in 32
> bit space, but the IO can be in 64 bit. But there is only

> ONE dma_mask for the device.

Ahh, yes. The AMI MegaRAID driver/controllers have the same
requirements.
megaraid solved it as:

pci_find_device() finds their device, they get a pci_dev
back which has a

		32-bit dma_mask.
		pci_alloc_consistent() the mailbox (this must be in 32-bit
space).
		if it's a 64-bit-capable controller, set pci_dev->dma_mask ~0L.
		pci_alloc_consistent() the command buffers and

scatter-gather lists (these
can be in 64-bit space).

Could something similar work for you?

Thanks,
Matt

--
Matt Domsch
Sr. Software Engineer
Dell Linux Solutions
www.dell.com/linux
#2 Linux Server provider with 17% in the US and 14%
Worldwide (IDC)!
#3 Unix provider with 18% in the US (Dataquest)!

Matt> Never have I seen a using NULL pci_dev pointer give you a 64-bit Matt> DMA address. Per Jeff's comment, it should give you an address Matt> with a mask of 24-bits (within ISA space, i.e. <16MB), but since Matt> you can't add in any ISA cards on IA-64, this wouldn't make much Matt> sense either.

Well but we don't have any ISA busses on the ia64 boxes. The only reason to return a 24 bit mask would be for broken devices that require a 31 or 29 bit or similar mask, but one should set that via the dma_mask interface anyway.

Jes