Skip to contentSkip to conversation

New firewire stack - updated patches

PATCH In review Kristian Høgsberg 3 participants 11 messages Dec 20, 2006 – Dec 21, 2006

lore
Share & tools
lore
#0 lore raw

cover New firewire stack - updated patches

Hi,

Here's a new set of patches for the new firewire stack. The changes since the last set of patches address the issues that were raised on the list and can be reviewed in detail here:

http://gitweb.freedesktop.org/?p=users/krh/juju.git

but to sum up the changes:

- Got rid of bitfields.

- Tested on ppc, ppc64 x86-64 and x86.

- ioctl interface tested on 32-bit userspace / 64-bit kernels.

- ASCIIfied sources.

- Incorporated Jeff Garziks comments.

- Updated to work with the new workqueue API changes.

- Moved subsystem to drivers/firewire from drivers/fw.

plus a number of bug fixes.

As mentioned last time, the stack still lacks isochronous receive functionality to be on par with the old stack, feature-wise. This is the one remaining piece of feature work kernel-side. When that is done, I have a couple of TODO items in user space:

- Make a libraw1394 compatibility library

- Port libdv1394 to new isochronous API.

which will allow us to move most user space applications to the new stack. That is, even if the new stack provides a new interface for asynchronous and isochronous IO, a lot of applications can still work since the changes are isolated to a couple of libraries. This is still in development and is being discussed on the linux1394-devel list. It will likely require a few changes kernel side in the stack as we figure out how to do this.

It is still work in progress, but at least now it should work across all architectures and endianesses.

Happy Holidays,
Kristian

#1 lore raw

Kristian Høgsberg wrote:
...

Quoted 10 lines

to sum up the changes:

- Got rid of bitfields.

- Tested on ppc, ppc64 x86-64 and x86.

- ioctl interface tested on 32-bit userspace / 64-bit kernels.

- ASCIIfied sources.

Quoted 7 lines

- Incorporated Jeff Garziks comments.

- Updated to work with the new workqueue API changes.

- Moved subsystem to drivers/firewire from drivers/fw.

plus a number of bug fixes.

Congrats. WRT the 1st, 3rd, and 5th item you are now ahead of mainline's stack. :-)

Quoted 4 lines

As mentioned last time, the stack still lacks isochronous receive functionality to be on par with the old stack, feature-wise. This is the one remaining piece of feature work kernel-side. When that is done, I have a couple of TODO items in user space:

Actually there are also eth1394 and pcilynx to be pulled over. Eth1394 should be quite easy to do for anybody after iso reception is settled in your stack. Pcilynx could follow depending on developer interest. It's increasingly rare hardware and the few old machines which have it can be cheaply upgraded to OHCI (which performs better for SBP-2 anyway).

Quoted 1 line

- Make a libraw1394 compatibility library

Consider using libraw1394 right from the start of this porting project. If there is only one libraw1394 (which works with raw1394 and with fw-device-cdev), enthusiasts might have an easier time to test your stack.

Quoted 4 lines

- Port libdv1394 to new isochronous API.

which will allow us to move most user space applications to the new stack.

...

Signature · 3 lines
Stefan Richter
-=====-=-==- ==-- =-=--
http://arcgraph.de/sr/
#2 lore raw
Quoted 14 lines · Stefan Richter

Kristian Høgsberg wrote:
...

Quoted 12 lines

to sum up the changes:

- Got rid of bitfields.

- Tested on ppc, ppc64 x86-64 and x86.

- ioctl interface tested on 32-bit userspace / 64-bit kernels.

- ASCIIfied sources.

- Incorporated Jeff Garziks comments.

Quoted 8 lines
Quoted 5 lines

- Updated to work with the new workqueue API changes.

- Moved subsystem to drivers/firewire from drivers/fw.

plus a number of bug fixes.

Congrats. WRT the 1st, 3rd, and 5th item you are now ahead of mainline's stack. :-)

Hehe, yeah, I saw the big FIXME in raw1394.c about compat_ioctl. But raw1394.c shouldn't be that hard to fix, the ioctl structs are using explicitly sized types and u64 for userspace pointers, the one problem I see is that some of the 64-bit fields aren't 64-bit aligend.

Quoted 10 lines
Quoted 4 lines

As mentioned last time, the stack still lacks isochronous receive functionality to be on par with the old stack, feature-wise. This is the one remaining piece of feature work kernel-side. When that is done, I have a couple of TODO items in user space:

Actually there are also eth1394 and pcilynx to be pulled over. Eth1394 should be quite easy to do for anybody after iso reception is settled in your stack. Pcilynx could follow depending on developer interest. It's increasingly rare hardware and the few old machines which have it can be cheaply upgraded to OHCI (which performs better for SBP-2 anyway).

Well... I don't think eth1394 was ever used much and it's not something I plan to port over. The only thing I've ever heard people say about it is that it's annoying because it screws up their network interface ordering. And Windows Vista will drop the IP over 1394 functionality, which is another data point about how widely used this standard is.

I'm not planning to port the pcilynx driver either. I think it's a sore point for the old stack as it is - nobody uses it or tests it and it's continously bit-rotting. So I'd rather not support it. However, it can perform as well as an OHCI card for SBP-2. If you set up a self-modifying DMA program it can read and write system memory without CPU intervention too.

Quoted 5 lines
Quoted 1 line

- Make a libraw1394 compatibility library

Consider using libraw1394 right from the start of this porting project. If there is only one libraw1394 (which works with raw1394 and with fw-device-cdev), enthusiasts might have an easier time to test your stack.

Hmm, maybe. There is going to be completely different code paths for each API entry point and not a lot of code sharing. But there is definitely some merit to only having one library, and if it could detect the kernel interface automatically and just work that would be even better.

cheers,
Kristian

#3 lore raw
Quoted 9 lines · Kristian Høgsberg
Quoted 5 lines · Stefan Richter

Actually there are also eth1394 and pcilynx to be pulled over. Eth1394 should be quite easy to do for anybody after iso reception is settled in your stack. Pcilynx could follow depending on developer interest. It's increasingly rare hardware and the few old machines which have it can be cheaply upgraded to OHCI (which performs better for SBP-2 anyway).

Well... I don't think eth1394 was ever used much and it's not something I plan to port over.

It is used, even though it is not very robust because it is not actively maintained (yet). If your stack will shape up to become a potential replacement of mainline's stack, I'm sure _someone_ will do the port.

Quoted 3 lines

The only thing I've ever heard people say about it
is that it's annoying because it screws up their network interface ordering.

Yeah, the same way hot-pluggable SCSI devices screw up device naming of

Quoted 2 lines

And Windows Vista will drop the IP over 1394 functionality, which is another data point about how widely used this standard is.

If we cared what Windows supports or does not support, we would have gap count optimization by now, but no support of IEEE 1394b-2002.

Quoted 3 lines

I'm not planning to port the pcilynx driver either. I think it's a sore point for the old stack as it is - nobody uses it or tests it and it's continously bit-rotting. So I'd rather not support it.

Here I agree.

Quoted 4 lines

However, it can
perform as well as an OHCI card for SBP-2. If you set up a self-modifying DMA program it can read and write system memory without CPU intervention too.

OK, I didn't know that although I suspected something like this might be possible. Of course this remains a potential feature as long as there is no manpower to actually implement it. (Nor is there a userbase to speak of to appreciate such an effort.)

Quoted 11 lines
Quoted 6 lines
Quoted 1 line

- Make a libraw1394 compatibility library

Consider using libraw1394 right from the start of this porting project. If there is only one libraw1394 (which works with raw1394 and with fw-device-cdev), enthusiasts might have an easier time to test your stack.

Hmm, maybe. There is going to be completely different code paths for each API entry point and not a lot of code sharing. But there is definitely some merit to only having one library, and if it could detect the kernel interface automatically and just work that would be even better.

Certainly, there won't be any benefit WRT code sharing in the library. It's more about deployment.

Signature · 3 lines
Stefan Richter
-=====-=-==- ==-- =-=--
http://arcgraph.de/sr/
#4 lore raw

Stefan Richter wrote:
...

Quoted 5 lines
Quoted 2 lines

And Windows Vista will drop the IP over 1394 functionality, which is another data point about how widely used this standard is.

If we cared what Windows supports or does not support, we would have gap count optimization by now, but no support of IEEE 1394b-2002.

Yeah, my point wasn't that we need to drop it because Windows dropped it, it was just a data point backing up the claim that IP 1394 isn't very widely used.

And as for gap count optimization, I just added that to my git repo. I thought about adding it before sending out these patches, but I was running late on getting them out -- I ended up spending too much time chasing down a stupid spinlock recursion. It is pretty simple to implement in the new stack, since we have all the topology information available. I did a quick, unscientific benchmark (md5summing 10 32M files) and the optimization is definitely noticable. This is a setup where the box and the disk are both connected to a hub so the max hops is 2, so we're using gap count 4:

	real    0m10.021s
	user    0m1.435s
	sys     0m0.356s

compared to no optimization, ie gap count 63:

	real    0m14.537s
	user    0m1.390s
	sys     0m0.402s

Though I see that Mac OS X uses a more conservative setting for a similiar topology, so maybe we need to add a bit or "margin" to the numbers from the table from 1394.

Quoted 11 lines
Quoted 3 lines

I'm not planning to port the pcilynx driver either. I think it's a sore point for the old stack as it is - nobody uses it or tests it and it's continously bit-rotting. So I'd rather not support it.

Here I agree.

Quoted 4 lines

However, it can
perform as well as an OHCI card for SBP-2. If you set up a self-modifying DMA program it can read and write system memory without CPU intervention too.

Quoted 4 lines

OK, I didn't know that although I suspected something like this might be possible. Of course this remains a potential feature as long as there is no manpower to actually implement it. (Nor is there a userbase to speak of to appreciate such an effort.)

Exactly. It's a cool hack (it's mentioned briefly in appendix E.1 of the PCILynx functional specification) and it would be fun to make it work, but I don't really see a userbase here. And if somebody has a PCILynx card and want to use the new stack, I'll trade them for a OHCI controller :) I have a much more useful way to put PCILynx cards to work using my firewire sniffer (http://bitplanet.net/nosy).

cheers,
Kristian

#5 lore raw
Quoted 1 line · Kristian Høgsberg

And as for gap count optimization, I just added that to my git repo.

What can I say.

...

Quoted 3 lines

and the optimization is definitely noticable. This is a setup where the box and the disk are both connected to a hub so the max hops is 2, so we're using gap count 4:

...

Quoted 3 lines

Though I see that Mac OS X uses a more conservative setting for a similiar topology, so maybe we need to add a bit or "margin" to the numbers from the table from 1394.

The table in IEEE 1394-1995 is not entirely safe. Use the one from IEEE 1394a-2000:

Hops  GC     Hops  GC     Hops  GC     Hops  GC     Hops  GC
   1   5        6  16       11  29       16  43       21  57
   2   7        7  18       12  32       17  46       22  59
   3   8        8  21       13  35       18  48       23  62
   4  10        9  24       14  37       19  51
   5  13       10  26       15  40       20  54

(It's certainly not necessary to optimize for more than ~8 hops.)

But note, this does not work anymore as soon as there is an 1394b node in the mix --- at least with one or more 1394b repeater node, I'm not sure about 1394b leaf nodes. Because of the potentially much larger repeater delays of 1394b PHYs, the only suitable method to determine a working least gap count of such setups is round-trip delay measurement with ping packets. But a good compromise would be to run table-based gap count optimization for 1394a environments and no optimization for 1394b or mixed environments. (Even though the latter would also clearly benefit from it.)

Signature · 3 lines
Stefan Richter
-=====-=-==- ==-- =-=--
http://arcgraph.de/sr/
#6 lore raw
Quoted 7 lines · I wrote:

Because of the potentially much larger
repeater delays of 1394b PHYs, the only suitable method to determine a working least gap count of such setups is round-trip delay measurement with ping packets. But a good compromise would be to run table-based gap count optimization for 1394a environments and no optimization for 1394b or mixed environments. (Even though the latter would also clearly benefit from it.)

               ^^  "it" == "a reduced gap count"
Signature · 3 lines
Stefan Richter
-=====-=-==- ==-- =-=--
http://arcgraph.de/sr/
#7 lore raw
In reply to Stefan Richter
Quoted 1 line · I wrote:

Kristian Høgsberg wrote:

[eth1394]

Quoted 4 lines
Quoted 2 lines

The only thing I've ever heard people say about it is that it's annoying because it screws up their network interface ordering.

Yeah, the same way hot-pluggable SCSI devices screw up device naming of

(I forgot to complete the post.) ... fixed SCSI devices and among themselves. That's old and continues to become apparent with more and more types of hardware. The solution is old too; distributors should be aware of it.

That said, the way how eth1394 is married with ieee1394 could stand improvement. There should be an option (and it should be the default) to let eth1394 load its ROM entries == don't automatically modprobe eth1394 as soon as the low level set up a host adapter.

Signature · 3 lines
Stefan Richter
-=====-=-==- ==-- =-=--
http://arcgraph.de/sr/
#8 lore raw
In reply to Kristian Høgsberg
Quoted 5 lines · Kristian Høgsberg

Hi,

Here's a new set of patches for the new firewire stack. The changes since the last set of patches address the issues that were raised on the list and can be reviewed in detail here:

.. for some reason I didn't get patch 3/4 and 4/4 on the linux1394-devel list, so I'll reply to this one.

I would suggest a reordering of the interrupt flag checks. Currently the interrupts that are least likely to occur are checked first. I don't see why. I would reorder the check such that ISO interrupts are checked first, as they have the most stringent timing constraints and are most likely to occur (when using ISO traffic).

After processing the ISO interrupts (and maybe the Async ones), a bypass could be inserted to exit the interrupt handler when there are no other interrupts to be handled. All other interrupts are to report relatively rare events or errors (error handling still to be added I assume). The effective run-length of the interrupt handler would be shorter using such a bypass, especially in the case where there is a lot of ISO traffic.

I'm looking forward to your ISO implementation, and I hope to incorporate it into FreeBob really fast.

Pieter

#9 lore raw
Quoted 9 lines · Pieter Palmers
Quoted 5 lines · Kristian Høgsberg

Hi,

Here's a new set of patches for the new firewire stack. The changes since the last set of patches address the issues that were raised on the list and can be reviewed in detail here:

.. for some reason I didn't get patch 3/4 and 4/4 on the linux1394-devel list, so I'll reply to this one.

I would suggest a reordering of the interrupt flag checks. Currently the interrupts that are least likely to occur are checked first. I don't see why. I would reorder the check such that ISO interrupts are checked first, as they have the most stringent timing constraints and are most likely to occur (when using ISO traffic).

All the interrupt handler does is schedule tasklets and they are all handled before returning to userspace. So not matter how you order them it's going to take the same time. If you want to defer handling of async traffic to after your userspace handler has run, you need to schedule a work_struct for handling the async events.

Having said that, I doubt that the latency between iso interrupt and user space handler imposed by the irq handler and the tasklets will be a problem. All the async tasklets do is copy the data out of the DMA buffers, possibly lookup a corresponding request and eventually call complete() on some struct completion. The worst case is the bus reset tasklet which does the topology walk, but even that is pretty fast.

Quoted 3 lines

After processing the ISO interrupts (and maybe the Async ones), a bypass could be inserted to exit the interrupt handler when there are no other interrupts to be handled. All other interrupts are to report relatively rare events or errors (error handling still to be added I assume). The effective run-length of the interrupt handler would be shorter using such a bypass, especially in the case where there is a lot of ISO traffic.

I'm looking forward to your ISO implementation, and I hope to incorporate it into FreeBob really fast.

Sounds great, I'll get to the isochronous receive feature as soon as possible.
I'm open to changing the interrupt handling if we can acheive lower latency,
but we need to be able to measure it before we start making changes.

cheers,
Kristian

#10 lore raw
In reply to Kristian Høgsberg
Quoted 1 line · Kristian Høgsberg

Here's a new set of patches for the new firewire stack.

...

Quoted 2 lines

It is still work in progress, but at least now it should work across all architectures and endianesses.

Committed to linux1394-2.6.git.

BTW, I prepended "ieee1394:" to the titles of most of the commits to this tree. From now on I will always do this on commits affecting mainline's FireWire stack, and prepend "firewire:" to the titles of commits affecting the JUJU stack. It's redundant but IMO helpful when reading changelogs.

Signature · 3 lines
Stefan Richter
-=====-=-==- ==-- =-=-=
http://arcgraph.de/sr/