Skip to contentSkip to conversation

firewire: Enable physical DMA above 4GB

PATCH In review Peter Hurley 3 participants 11 messages Mar 26, 2013 – Mar 29, 2013 Firewire

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 firewire: Enable physical DMA above 4GB

Quadlet reads to memory above 4GB is painfully slow when serviced by the AR DMA context. In addition, the CPU(s) may be locked-up, preventing any transfer at all.

Write the PhyUpperBound register with the end-of-memory value. If end-of-memory is beyond the OHCI limit of 0x0000ffff00000000, clamp to that value.

---
 drivers/firewire/ohci.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/firewire/ohci.c +5 -1
index 044ace3..b4135a5 100644
@@ -2249,6 +2249,7 @@ static int ohci_enable(struct fw_card *card,
 	struct pci_dev *dev = to_pci_dev(card->device);
 	u32 lps, version, irqs;
 	int i, ret;
+	u32 phys_upper;
 
 	if (software_reset(ohci)) {
 		dev_err(card->device, "failed to reset ohci card\n");
@@ -2323,7 +2324,10 @@ static int ohci_enable(struct fw_card *card,
 	reg_write(ohci, OHCI1394_FairnessControl, 0);
 	card->priority_budget_implemented = ohci->pri_req_max != 0;
 
-	reg_write(ohci, OHCI1394_PhyUpperBound, 0x00010000);
+	phys_upper = min(0xffff0000ULL,
+			 (dma_get_required_mask(card->device) >> 16) + 1);
+	reg_write(ohci, OHCI1394_PhyUpperBound, max(phys_upper, 0x00010000U));
+
 	reg_write(ohci, OHCI1394_IntEventClear, ~0);
 	reg_write(ohci, OHCI1394_IntMaskClear, ~0);
 
Signature · 1 line
1.8.1.2
#1 lore raw
Quoted 3 lines · Peter Hurley

Quadlet reads to memory above 4GB is painfully slow when serviced by the AR DMA context. In addition, the CPU(s) may be locked-up, preventing any transfer at all.

Using physical DMA prevents the use of that address space for software address handlers, so you have adjust the low_memory_region start in core-transaction.c.

Quoted 3 lines

Write the PhyUpperBound register with the end-of-memory value. If end-of-memory is beyond the OHCI limit of 0x0000ffff00000000, clamp to that value.

You will have to lower this limit; there are protcols that assume that addresses like 0xecc000000000 are available for software.

Regards,
Clemens

#2 lore raw
Quoted 8 lines · Clemens Ladisch
Quoted 3 lines · Peter Hurley

Quadlet reads to memory above 4GB is painfully slow when serviced by the AR DMA context. In addition, the CPU(s) may be locked-up, preventing any transfer at all.

Using physical DMA prevents the use of that address space for software address handlers, so you have adjust the low_memory_region start in core-transaction.c.

Right, thanks for pointing that out.

Quoted 6 lines
Quoted 3 lines

Write the PhyUpperBound register with the end-of-memory value. If end-of-memory is beyond the OHCI limit of 0x0000ffff00000000, clamp to that value.

You will have to lower this limit; there are protcols that assume that addresses like 0xecc000000000 are available for software.

Maybe 0x0000e80000000000 is a better maximum upper bound?

Regards,
Peter Hurley

#3 lore raw
Quoted 10 lines · Peter Hurley
Quoted 7 lines · Clemens Ladisch
Quoted 3 lines · Peter Hurley

Write the PhyUpperBound register with the end-of-memory value. If end-of-memory is beyond the OHCI limit of 0x0000ffff00000000, clamp to that value.

You will have to lower this limit; there are protcols that assume that addresses like 0xecc000000000 are available for software.

Maybe 0x0000e80000000000 is a better maximum upper bound?

Why the space of 0x04c000000000? ;-)

There's also 0xc007dedadada (although this address isn't used on the PC side). Maybe we should just round down to 0x800000000000 and revisit the question when there are 256 TB memory machines. (Directly accessing the full 256 TB will not be possible in any case.)

Regards,
Clemens

#4 lore raw
In reply to Peter Hurley
Quoted 27 lines · On Mar 26 Peter Hurley wrote:

Quadlet reads to memory above 4GB is painfully slow when serviced by the AR DMA context. In addition, the CPU(s) may be locked-up, preventing any transfer at all.

Write the PhyUpperBound register with the end-of-memory value. If end-of-memory is beyond the OHCI limit of 0x0000ffff00000000, clamp to that value.

---
 drivers/firewire/ohci.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/firewire/ohci.c +1 -0
index 044ace3..b4135a5 100644
@@ -2249,6 +2249,7 @@ static int ohci_enable(struct fw_card *card,
 	struct pci_dev *dev = to_pci_dev(card->device);
 	u32 lps, version, irqs;
 	int i, ret;
+	u32 phys_upper;
 
 	if (software_reset(ohci)) {
 		dev_err(card->device, "failed to reset ohci card\n");
@@ -2323,7 +2324,10 @@ static int ohci_enable(struct fw_card *card,
 	reg_write(ohci, OHCI1394_FairnessControl, 0);

drivers/firewire/ohci.c static int ohci_enable(struct fw_card *card, · line 2333

Quoted 10 lines
drivers/firewire/ohci.c +4 -1
 	card->priority_budget_implemented = ohci->pri_req_max != 0;
 
-	reg_write(ohci, OHCI1394_PhyUpperBound, 0x00010000);
+	phys_upper = min(0xffff0000ULL,
+			 (dma_get_required_mask(card->device) >> 16) + 1);
+	reg_write(ohci, OHCI1394_PhyUpperBound, max(phys_upper, 0x00010000U));
+
 	reg_write(ohci, OHCI1394_IntEventClear, ~0);
 	reg_write(ohci, OHCI1394_IntMaskClear, ~0);
 

What Clemens said.

Also: By far most OHCI-1394 chips do not implement PhyUpperBound, i.e. ignore any writes to PhyUpperBound, return 0 when PhyUpperBound is read, and keep the boundary between physical response and AR response at 4 GB, as described in the spec.

It has been a long time though since I last checked whether PhyUpperBound is implemented; maybe it has become more widespread than it was back then.

Or maybe it hasn't: All OHCI-1394 chips that ever came to market are 32 bit chips anyway. So the few rare ones that do support PhyUpperBound larger than 4 GB cannot in fact use it.

Or am I severely behind the times about this?

Signature · 3 lines
Stefan Richter
-=====-===-= --== ==-=-
http://arcgraph.de/sr/
#5 lore raw
Quoted 30 lines · Stefan Richter
Quoted 29 lines · On Mar 26 Peter Hurley wrote:

Quadlet reads to memory above 4GB is painfully slow when serviced by the AR DMA context. In addition, the CPU(s) may be locked-up, preventing any transfer at all.

Write the PhyUpperBound register with the end-of-memory value. If end-of-memory is beyond the OHCI limit of 0x0000ffff00000000, clamp to that value.

---
 drivers/firewire/ohci.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/firewire/ohci.c +1 -0
index 044ace3..b4135a5 100644
@@ -2249,6 +2249,7 @@ static int ohci_enable(struct fw_card *card,
 	struct pci_dev *dev = to_pci_dev(card->device);
 	u32 lps, version, irqs;
 	int i, ret;
+	u32 phys_upper;
 
 	if (software_reset(ohci)) {
 		dev_err(card->device, "failed to reset ohci card\n");
@@ -2323,7 +2324,10 @@ static int ohci_enable(struct fw_card *card,
 	reg_write(ohci, OHCI1394_FairnessControl, 0);
 	card->priority_budget_implemented = ohci->pri_req_max != 0;
 
Quoted 10 lines

drivers/firewire/ohci.c static int ohci_enable(struct fw_card *card, · line 2333

Quoted 8 lines
drivers/firewire/ohci.c +4 -1
-	reg_write(ohci, OHCI1394_PhyUpperBound, 0x00010000);
+	phys_upper = min(0xffff0000ULL,
+			 (dma_get_required_mask(card->device) >> 16) + 1);
+	reg_write(ohci, OHCI1394_PhyUpperBound, max(phys_upper, 0x00010000U));
+
 	reg_write(ohci, OHCI1394_IntEventClear, ~0);
 	reg_write(ohci, OHCI1394_IntMaskClear, ~0);
 

What Clemens said.

That's what I'm testing right now for v2 -- a fixed phys-AR boundary of 0x800000000000ULL.

Quoted 5 lines

Also: By far most OHCI-1394 chips do not implement PhyUpperBound, i.e. ignore any writes to PhyUpperBound, return 0 when PhyUpperBound is read, and keep the boundary between physical response and AR response at 4 GB, as described in the spec.

Quoted 8 lines

It has been a long time though since I last checked whether PhyUpperBound is implemented; maybe it has become more widespread than it was back then.

Or maybe it hasn't: All OHCI-1394 chips that ever came to market are 32 bit chips anyway. So the few rare ones that do support PhyUpperBound larger than 4 GB cannot in fact use it.

Or am I severely behind the times about this?

The FW643e-2 is natively PCIe (not behind a bridge) and supports phys DMA past 4GB (the datasheet says all 48 bits but I can only test it out to 10GB).

I thought the FW643e was as well? You'll have to test that out :)

Regards,
Peter Hurley

#6 lore raw
Quoted 8 lines · On Mar 26 Peter Hurley wrote:
Quoted 7 lines · Stefan Richter

It has been a long time though since I last checked whether PhyUpperBound is implemented; maybe it has become more widespread than it was back then.

Or maybe it hasn't: All OHCI-1394 chips that ever came to market are 32 bit chips anyway. So the few rare ones that do support PhyUpperBound larger than 4 GB cannot in fact use it.

Quoted 7 lines
Quoted 1 line

Or am I severely behind the times about this?

The FW643e-2 is natively PCIe (not behind a bridge) and supports phys DMA past 4GB (the datasheet says all 48 bits but I can only test it out to 10GB).

I thought the FW643e was as well? You'll have to test that out :)

OK, will do.

Signature · 3 lines
Stefan Richter
-=====-===-= --== ==-=-
http://arcgraph.de/sr/
#7 lore raw
Quoted 11 lines · On Mar 26 Stefan Richter wrote:
Quoted 10 lines · On Mar 26 Peter Hurley wrote:
Quoted 8 lines · Stefan Richter

It has been a long time though since I last checked whether PhyUpperBound is implemented; maybe it has become more widespread than it was back then.

Or maybe it hasn't: All OHCI-1394 chips that ever came to market are 32 bit chips anyway. So the few rare ones that do support PhyUpperBound larger than 4 GB cannot in fact use it.

Or am I severely behind the times about this?

Quoted 7 lines
Quoted 5 lines

The FW643e-2 is natively PCIe (not behind a bridge) and supports phys DMA past 4GB (the datasheet says all 48 bits but I can only test it out to 10GB).

I thought the FW643e was as well? You'll have to test that out :)

OK, will do.

Does lspci or something similar show which PCI devices are capable of 64 bit wide addressing?

Signature · 3 lines
Stefan Richter
-=====-===-= --== ===-=
http://arcgraph.de/sr/
#8 lore raw
Quoted 16 lines · Stefan Richter
Quoted 15 lines · On Mar 26 Stefan Richter wrote:
Quoted 14 lines · On Mar 26 Peter Hurley wrote:
Quoted 8 lines · Stefan Richter

It has been a long time though since I last checked whether PhyUpperBound is implemented; maybe it has become more widespread than it was back then.

Or maybe it hasn't: All OHCI-1394 chips that ever came to market are 32 bit chips anyway. So the few rare ones that do support PhyUpperBound larger than 4 GB cannot in fact use it.

Or am I severely behind the times about this?

The FW643e-2 is natively PCIe (not behind a bridge) and supports phys DMA past 4GB (the datasheet says all 48 bits but I can only test it out to 10GB).

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

I thought the FW643e was as well? You'll have to test that out :)

OK, will do.

Does lspci or something similar show which PCI devices are capable of 64 bit wide addressing?

Not definitively.

Usually (but not always), if the host registers are 64-bit addressable, then the device supports DAC. But that's not something that can be relied on programmatically.

For example, lspci on FW643e-2:
06:00.0 FireWire (IEEE 1394): LSI Corporation FW643 [TrueFire] PCIe 1394b Controller (rev 08) (prog-if 10 [OHCI])
Subsystem: LSI Corporation FW643 [TrueFire] PCIe 1394b Controller
Flags: bus master, fast devsel, latency 0, IRQ 89
==> Memory at dbeff000 (64-bit, non-prefetchable) [size=4K]
Capabilities: <access denied>
Kernel driver in use: firewire_ohci
Kernel modules: firewire-ohci

By contrast, lspci on FW323 on same machine:
07:06.0 FireWire (IEEE 1394): LSI Corporation FW322/323 [TrueFire] 1394a Controller (rev 70) (prog-if 10 [OHCI])
Subsystem: Dell Device 5811
Flags: bus master, medium devsel, latency 64, IRQ 30
==> Memory at dbbff000 (32-bit, non-prefetchable) [size=4K]
Capabilities: <access denied>
Kernel driver in use: firewire_ohci
Kernel modules: firewire-ohci

#9 lore raw
Quoted 8 lines · Peter Hurley
Quoted 7 lines · Stefan Richter
Quoted 6 lines · On Mar 26 Peter Hurley wrote:
Quoted 5 lines

The FW643e-2 is natively PCIe (not behind a bridge) and supports phys DMA past 4GB (the datasheet says all 48 bits but I can only test it out to 10GB).

I thought the FW643e was as well? You'll have to test that out :)

Quoted 7 lines
Quoted 2 lines

Does lspci or something similar show which PCI devices are capable of 64 bit wide addressing?

Not definitively.

Usually (but not always), if the host registers are 64-bit addressable, then the device supports DAC.

DACs are a feature of conventional PCI.

All PCI Express devices are 64-bit addressable. In the lspci output, PCIe devices have the "Express" capability.

However, whether a device can *generate* 64-bit DMA requests is completely device-specific.

Regards,
Clemens

#10 lore raw
Quoted 14 lines · Clemens Ladisch
Quoted 13 lines · Peter Hurley
Quoted 9 lines · Stefan Richter
Quoted 6 lines · On Mar 26 Peter Hurley wrote:
Quoted 5 lines

The FW643e-2 is natively PCIe (not behind a bridge) and supports phys DMA past 4GB (the datasheet says all 48 bits but I can only test it out to 10GB).

I thought the FW643e was as well? You'll have to test that out :)

Does lspci or something similar show which PCI devices are capable of 64 bit wide addressing?

Not definitively.

Quoted 7 lines
Quoted 2 lines

Usually (but not always), if the host registers are 64-bit addressable, then the device supports DAC.

DACs are a feature of conventional PCI.

All PCI Express devices are 64-bit addressable. In the lspci output, PCIe devices have the "Express" capability.

The "device" could be sitting behind a PCIe-PCI bridge that _only_ supports the host window for 64-bit. All other 64-bit decodes could return garbage.

Quoted 2 lines

However, whether a device can *generate* 64-bit DMA requests is completely device-specific.