Skip to contentSkip to conversation

Workaround hardware bug addressing physical memory

DISCUSSION Unai Uribarri 2 participants 5 messages Jul 15, 2010 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Thanks.
----- "Marin Mitov" <[email protected]> wrote:

| On Wednesday, July 14, 2010 08:06:49 pm you wrote:
| > ----- "Marin Mitov" <[email protected]> wrote:
| > 
| > | Hi,
| > | 
| > | This is pci driver. You can set dma mask:
| > | 
| > | dma_set_coheren_mask(pdev, DMA_BIT_MASK(31)) 
| > | 
| > | All further alloc_coherent() should be from the region 0-2GB.
| > | 
| > 
| > But I'm using a 64 bit operating system with 32GB of RAM. It's a
| > pity to be unable to use 4GB-32GB range because the 2-4GB range is
| > unusable. So I've written this code to skip invalid areas. Do you
| > think this code could be useful for other drivers?
| 
| Let me summarize if I have correctly understood what you do.
| 
| First, your hardware has problems when the physical (bus) address
| is out of the 0-2GB region, so you cannot use buffers that are out 
| of this range in any case. And the defect is in the peripheral, not in
| the bridge between it and the memory.

The hardware works correctly for physical address in the ranges 0 to 2GB AND 4GB to 32GB. Physical address in the 2-4GB range are read correctly by the device. But when the device tries to write to them it issues invalid PCIe transaction headers: it tries to access such addresses using a 64-bit transactions when the PCI Express standard mandates to use 32-bit transactions for memory addresses below 4GB. Some bridges accept such invalid transactions, but Intel 5500 chipset rejects them.

I'm allocating 256MB of RAM for I/O buffers; I'm fear that restricting all the allocations to the first 2GB of memory will put too much pressure in that zone of memory. But restricting it to 4GB and above will be okay.

Is there any way to restrict to memory address above 4GB?

| Second, you do allocations and check if dma_addr_t is in this range.
| If it is, you keep it for use. If not, you put it in a list and
| allocate a new one.
| When you allocate sufficiently memory in 0-2GB region, you deallocate
| 
| the memory (kept in the list) that is out of the region. Right?
| 

Yes, except that the validator function is:

static int validate_dma(void *ptr, struct device *dev, size_t size, dma_addr_t handle)

{
        return handle < 0x80000000U || handle > 0xFFFFFFFFU;
}

Thanks.

#2 lore raw
Quoted 27 lines · Unai Uribarri

Thanks.
----- "Marin Mitov" <[email protected]> wrote:

| On Wednesday, July 14, 2010 08:06:49 pm you wrote:
| > ----- "Marin Mitov" <[email protected]> wrote:
| > 
| > | Hi,
| > | 
| > | This is pci driver. You can set dma mask:
| > | 
| > | dma_set_coheren_mask(pdev, DMA_BIT_MASK(31)) 
| > | 
| > | All further alloc_coherent() should be from the region 0-2GB.
| > | 
| > 
| > But I'm using a 64 bit operating system with 32GB of RAM. It's a
| > pity to be unable to use 4GB-32GB range because the 2-4GB range is
| > unusable. So I've written this code to skip invalid areas. Do you
| > think this code could be useful for other drivers?
| 
| Let me summarize if I have correctly understood what you do.
| 
| First, your hardware has problems when the physical (bus) address
| is out of the 0-2GB region, so you cannot use buffers that are out 
| of this range in any case. And the defect is in the peripheral, not in
| the bridge between it and the memory.
Quoted 2 lines

The hardware works correctly for physical address in the ranges 0 to 2GB AND 4GB to 32GB.

Sorry, I have missed that point.

Quoted 7 lines

Physical address in the 2-4GB range are read correctly
by the device. But when the device tries to write to them it issues invalid PCIe transaction headers: it tries to access such addresses using a 64-bit transactions when the PCI Express standard mandates to use 32-bit transactions for memory addresses below 4GB. Some bridges accept such invalid transactions, but Intel 5500 chipset rejects them.

Quoted 5 lines

I'm allocating 256MB of RAM for I/O buffers; I'm fear that restricting all the allocations to the first 2GB of memory will put too much pressure in that zone of memory. But restricting it to 4GB and above will be okay.

Is there any way to restrict to memory address above 4GB?

I am thinking if your BIOS could make a memory holl between 2-4GB remapping this memory above 4GB? On my (relatively old machine) the memory hall is restricted to 512MB in the region (4GB-512MB) - 4GB. AGP apertires could be up to 2GB (according to specs), or using kernel parameters (if exist) to map the memory (excluding this 2-4GB region).

Quoted 19 lines

| Second, you do allocations and check if dma_addr_t is in this range.
| If it is, you keep it for use. If not, you put it in a list and
| allocate a new one.
| When you allocate sufficiently memory in 0-2GB region, you deallocate
| 
| the memory (kept in the list) that is out of the region. Right?
| 

Yes, except that the validator function is:

static int validate_dma(void *ptr, struct device *dev, size_t size, dma_addr_t handle)

{
        return handle < 0x80000000U || handle > 0xFFFFFFFFU;
}

Thanks.

#3 lore raw
In reply to Unai Uribarri
Quoted 35 lines · Unai Uribarri

Thanks.
----- "Marin Mitov" <[email protected]> wrote:

| On Wednesday, July 14, 2010 08:06:49 pm you wrote:
| > ----- "Marin Mitov" <[email protected]> wrote:
| > 
| > | Hi,
| > | 
| > | This is pci driver. You can set dma mask:
| > | 
| > | dma_set_coheren_mask(pdev, DMA_BIT_MASK(31)) 
| > | 
| > | All further alloc_coherent() should be from the region 0-2GB.
| > | 
| > 
| > But I'm using a 64 bit operating system with 32GB of RAM. It's a
| > pity to be unable to use 4GB-32GB range because the 2-4GB range is
| > unusable. So I've written this code to skip invalid areas. Do you
| > think this code could be useful for other drivers?
| 
| Let me summarize if I have correctly understood what you do.
| 
| First, your hardware has problems when the physical (bus) address
| is out of the 0-2GB region, so you cannot use buffers that are out 
| of this range in any case. And the defect is in the peripheral, not in
| the bridge between it and the memory.

The hardware works correctly for physical address in the ranges 0 to 2GB AND 4GB to 32GB. Physical address in the 2-4GB range are read correctly by the device. But when the device tries to write to them it issues invalid PCIe transaction headers: it tries to access such addresses using a 64-bit transactions when the PCI Express standard mandates to use 32-bit transactions for memory addresses below 4GB. Some bridges accept such invalid transactions, but Intel 5500 chipset rejects them.

Quoted 5 lines

I'm allocating 256MB of RAM for I/O buffers; I'm fear that restricting all the allocations to the first 2GB of memory will put too much pressure in that zone of memory. But restricting it to 4GB and above will be okay.

Is there any way to restrict to memory address above 4GB?

Would this work for you?
From: Documentation/kernel-parameters.txt:

	memmap=nn[KMG]$ss[KMG]
			[KNL,ACPI] Mark specific memory as reserved.
			Region of memory to be used, from ss to ss+nn.
			Example: Exclude memory from 0x18690000-0x1869ffff
			         memmap=64K$0x18690000
			         or
			         memmap=0x10000$0x18690000

Marin Mitov

#4 lore raw

----- "Marin Mitov" <[email protected]> wrote:

| On 15.7.2010, Unai Uribarri wrote:
| > Thanks.
| > ----- "Marin Mitov" <[email protected]> wrote:
| > 
| > | On Wednesday, July 14, 2010 08:06:49 pm you wrote:
| > | > ----- "Marin Mitov" <[email protected]> wrote:
| > | > 
| > | > | Hi,
| > | > | 
| > | > | This is pci driver. You can set dma mask:
| > | > | 
| > | > | dma_set_coheren_mask(pdev, DMA_BIT_MASK(31)) 
| > | > | 
| > | > | All further alloc_coherent() should be from the region 0-2GB.
| > | > | 
| > | > 
| > | > But I'm using a 64 bit operating system with 32GB of RAM. It's
| a
| > | > pity to be unable to use 4GB-32GB range because the 2-4GB range
| is
| > | > unusable. So I've written this code to skip invalid areas. Do
| you
| > | > think this code could be useful for other drivers?
| > | 
| > | Let me summarize if I have correctly understood what you do.
| > | 
| > | First, your hardware has problems when the physical (bus) address
| > | is out of the 0-2GB region, so you cannot use buffers that are out
| 
| > | of this range in any case. And the defect is in the peripheral,
| not in
| > | the bridge between it and the memory.
| > 
| > The hardware works correctly for physical address in the ranges 0 to
| 2GB
| > AND 4GB to 32GB. Physical address in the 2-4GB range are read
| correctly
| > by the device. But when the device tries to write to them it issues
| 
| > invalid PCIe transaction headers: it tries to access such addresses
| using
| > a 64-bit transactions when the PCI Express standard mandates to use
| 32-bit
| > transactions for memory addresses below 4GB. Some bridges accept
| such
| > invalid transactions, but Intel 5500 chipset rejects them.
| > 
| > I'm allocating 256MB of RAM for I/O buffers; I'm fear that
| restricting all
| > the allocations to the first 2GB of memory will put too much
| pressure in
| > that zone of memory. But restricting it to 4GB and above will be
| okay. 
| > 
| > Is there any way to restrict to memory address above 4GB?
| 
| Would this work for you?
| From: Documentation/kernel-parameters.txt:
| 
| 	memmap=nn[KMG]$ss[KMG]
| 			[KNL,ACPI] Mark specific memory as reserved.
| 			Region of memory to be used, from ss to ss+nn.
| 			Example: Exclude memory from 0x18690000-0x1869ffff
| 			         memmap=64K$0x18690000
| 			         or
| 			         memmap=0x10000$0x18690000

I think that linking in a list the unusable memory and freeing it after completing all the allocations is a better solution.

Thanks.

#5 lore raw
Quoted 70 lines · Unai Uribarri

----- "Marin Mitov" <[email protected]> wrote:

| On 15.7.2010, Unai Uribarri wrote:
| > Thanks.
| > ----- "Marin Mitov" <[email protected]> wrote:
| > 
| > | On Wednesday, July 14, 2010 08:06:49 pm you wrote:
| > | > ----- "Marin Mitov" <[email protected]> wrote:
| > | > 
| > | > | Hi,
| > | > | 
| > | > | This is pci driver. You can set dma mask:
| > | > | 
| > | > | dma_set_coheren_mask(pdev, DMA_BIT_MASK(31)) 
| > | > | 
| > | > | All further alloc_coherent() should be from the region 0-2GB.
| > | > | 
| > | > 
| > | > But I'm using a 64 bit operating system with 32GB of RAM. It's
| a
| > | > pity to be unable to use 4GB-32GB range because the 2-4GB range
| is
| > | > unusable. So I've written this code to skip invalid areas. Do
| you
| > | > think this code could be useful for other drivers?
| > | 
| > | Let me summarize if I have correctly understood what you do.
| > | 
| > | First, your hardware has problems when the physical (bus) address
| > | is out of the 0-2GB region, so you cannot use buffers that are out
| 
| > | of this range in any case. And the defect is in the peripheral,
| not in
| > | the bridge between it and the memory.
| > 
| > The hardware works correctly for physical address in the ranges 0 to
| 2GB
| > AND 4GB to 32GB. Physical address in the 2-4GB range are read
| correctly
| > by the device. But when the device tries to write to them it issues
| 
| > invalid PCIe transaction headers: it tries to access such addresses
| using
| > a 64-bit transactions when the PCI Express standard mandates to use
| 32-bit
| > transactions for memory addresses below 4GB. Some bridges accept
| such
| > invalid transactions, but Intel 5500 chipset rejects them.
| > 
| > I'm allocating 256MB of RAM for I/O buffers; I'm fear that
| restricting all
| > the allocations to the first 2GB of memory will put too much
| pressure in
| > that zone of memory. But restricting it to 4GB and above will be
| okay. 
| > 
| > Is there any way to restrict to memory address above 4GB?
| 
| Would this work for you?
| From: Documentation/kernel-parameters.txt:
| 
| 	memmap=nn[KMG]$ss[KMG]
| 			[KNL,ACPI] Mark specific memory as reserved.
| 			Region of memory to be used, from ss to ss+nn.
| 			Example: Exclude memory from 0x18690000-0x1869ffff
| 			         memmap=64K$0x18690000
| 			         or
| 			         memmap=0x10000$0x18690000
Quoted 1 line

I think that linking in a list the unusable memory and freeing it after completing all the allocations is a better solution.

Yes, I agree.

Quoted 3 lines

Thanks.