Request for comments on dejittering app
Hello,
I'd like to describe what I'm trying to do, so people can comment.
Basically I'm just transfering data between two systems over a network (possibly the Internet, but not necessarily).
(Systems A and B are synchronized by NTP.)
System A receives data from an input device on the PCI bus. This input device provides data at constant bit rate R, meaning if the driver requests B bytes, the DMA will take approximately B/R seconds.
I wrote a 'send' (user space) app on system A which requests 1316 bytes from the input device, adds an RTP header, and sends this data in a UDP datagram to system B. The RTP header provides, notably, a sequence number and a timestamp.
The datagrams travel over several networks, a few get lost, some arrive at system B, not necessarily in the order they were sent, and definitely not every B/R seconds.
I wrote a 'recv' (user space) app on system B whose job is to receive the packets from system A and re-send them (either to an output device on the PCI bus, or to another system on the LAN) after having removed the network jitter.
Here's my implementation:
Note: Packet P{i} contains a timestamp T{i}
buffer N packets
grab the current (absolute, CLOCK_REALTIME) time H{0} send P0
for i=1 to infinity
{
recv() a few packets (e.g. 3 at most) compute H{i} = H{i-1} + (T{i} - T{i-1}) sleep until H{i}
send packet P_i
}
The packets were timestamped on system A, thus for every i, T{i} - T{i-1} = ~B/R Therefore, most of the network jitter will be removed.
Any comments so far?
Now all this seems nice in theory, but I'd also like to make it work in practice! (And I'm having some problems.)
R is typically 38 Mbit/s => B/R is typically 277 µs
Obviously, I'm going to need high-resolution timers if I want to sleep for so small an interval.
As far as I understand, each packet received will generate one IRQ, and each write to the PCI output device will generate one IRQ. This means 3600 IRQs per second from the NIC, and 3600 IRQs per second from the PCI device. Is that reasonable? (Considering a 1267 MHz P3 with no IO-APIC.)
It is easy to reduce the number of IRQs from the PCI device by grouping several packets for a single write. It might be worthwhile.
One problem I have is that the PCI device's driver blocks until the device has acknowledged the data, and the write operation sometimes blocks for 200, 300, even 400 µs (I have not been able to tell why).
I might not need -rt, if I'm willing to handle several packets every time I wake up?
buffer N packets
grab the current (absolute) time H{0}
send P0
for i=1 to infinity
{
sleep for at least X ms
recv() some packets (how many? as much as I can?) grab the current time Hnow
while H{i} < Hnow send P{i}
}
(Rough pseudo-code.)
I'm eager to hear anyone's comments and suggestions.
Regards.
Quoted 2 lines · John Sigler
As far as I understand, each packet received will generate one IRQ
This depends on the driver implementation. If it uses the optional NAPI interface the first interrupt received will simply schedule a netif related thread to poll the device, disable future rx-interrupts for that NIC, and return.
The packet scheduler thread will then poll your device according to its configured policy with a specific "budget" of packets. The device can netif_rx as many packets from the device in that one invocation until one of two criteria is met: The budget is exhausted, or the rx-ring is exhausted.
If the rx-ring is exhausted, rx-interrupts are re-enabled and the return code tells the netif_rx scheduler to deschedule this device. If the budget is exhausted before the ring, the return code indicates that we still have work to do and interrupts remain off. The scheduler will re-poll your device at some point in the future.
There are other nitty gritty details here, but long story short you generally will not have a 1:1 of interrupts to rx-packets on a busy link. This is important from both a latency and DOS perspective because interrupt storms are mitigated. So if your driver supports NAPI, I suggest you use it. If it doesn't, it might be worth changing it.
I know you had other aspects to your system, but this may help in at least one dimension.
-Greg
--
Quoted 1 line · John Sigler
Hello,
Hi John,
Quoted 6 lines
I'd like to describe what I'm trying to do, so people can comment.
Any comments so far?
Well, some NICs buffer writes, and some even buffer receives. So your NIC may be buffering underneath you too.
Quoted 4 lines
Now all this seems nice in theory, but I'd also like to make it work in practice! (And I'm having some problems.)
Quoted 8 lines
R is typically 38 Mbit/s => B/R is typically 277 µs Obviously, I'm going to need high-resolution timers if I want to sleep for so small an interval.
As far as I understand, each packet received will generate one IRQ, and each write to the PCI output device will generate one IRQ. This means 3600 IRQs per second from the NIC, and 3600 IRQs per second from the PCI device. Is that reasonable? (Considering a 1267 MHz P3 with no IO-APIC.)
As mentioned above, the NIC may buffer too, so it may not be sending the packets out immediately. Usually, a driver would send out several packets before sending a transmit IRQ. In fact, that transmit IRQ is usually so to tell the driver that it can write more after a certain threshold has been reached. (It's been several years since I wrote a NIC driver, so I'm going off of my old memory ;-)
Quoted 3 lines
It is easy to reduce the number of IRQs from the PCI device by grouping several packets for a single write. It might be worthwhile.
May not be needed if the driver already does it.
Quoted 4 lines
One problem I have is that the PCI device's driver blocks until the device has acknowledged the data, and the write operation sometimes blocks for 200, 300, even 400 µs (I have not been able to tell why).
Not sure what you mean by saying the driver "blocks". Is it a sleeping thread (in -rt) or from an interrupt? Can you be more specific here.
Quoted 3 lines
I might not need -rt, if I'm willing to handle several packets every time I wake up?
For any reaction time less than 100 us (or maybe even higher), you will need -rt.
-- Steve
Quoted 6 lines · Steven Rostedt
Quoted 1 line · John Sigler
Any comments so far?
Well, some NICs buffer writes, and some even buffer receives. So your NIC may be buffering underneath you too.
Now that you mention it, I did notice that /proc/interrupts did not report the same numbers whether I send 1 packet every X ms OR N packets every N*X ms.
By the way, what does /proc/interrupts report? The number of IRQs that were actually serviced by the OS?
Quoted 12 lines
Quoted 11 lines
Now all this seems nice in theory, but I'd also like to make it work in practice! (And I'm having some problems.)
R is typically 38 Mbit/s => B/R is typically 277 µs Obviously, I'm going to need high-resolution timers if I want to sleep for so small an interval.
As far as I understand, each packet received will generate one IRQ, and each write to the PCI output device will generate one IRQ. This means 3600 IRQs per second from the NIC, and 3600 IRQs per second from the PCI device. Is that reasonable? (Considering a 1267 MHz P3 with no IO-APIC.)
Quoted 6 lines
As mentioned above, the NIC may buffer too, so it may not be sending the packets out immediately. Usually, a driver would send out several packets before sending a transmit IRQ. In fact, that transmit IRQ is usually so to tell the driver that it can write more after a certain threshold has been reached. (It's been several years since I wrote a NIC driver, so I'm going off of my old memory ;-)
Is /proc/interrupts a good way to tell whether the NIC is grouping transmits before generating an IRQ?
Quoted 11 lines
Quoted 2 lines
It is easy to reduce the number of IRQs from the PCI device by grouping several packets for a single write. It might be worthwhile.
May not be needed if the driver already does it.
Quoted 3 lines
One problem I have is that the PCI device's driver blocks until the device has acknowledged the data, and the write operation sometimes blocks for 200, 300, even 400 µs (I have not been able to tell why).
Not sure what you mean by saying the driver "blocks". Is it a sleeping thread (in -rt) or from an interrupt? Can you be more specific here.
(I may have several misconceptions.) In the case of network drivers, when a user space process calls send, the driver copies the user data, and schedules the operation to happen "sometime later".
But it "feels" like my PCI device's driver didn't copy the user space data, and instead programmed the DMA directly from the user space buffer, which means it has to put the user space process to sleep until the operation is complete, which sometimes takes 100s of µs.
Or perhaps, there are only a few buffers (I've read something about ping pong DMA in the data sheet) in the driver, and because I'm writing only small amounts of data, they are sometimes all full, because some DMA operations are taking longer than usual.
The driver source code is available here:
http://dektec.com/Products/LinuxSDK/Downloads/LinuxSDK.zip
I will take a hard look at it when I come back at the office.
Quoted 5 lines
Quoted 2 lines
I might not need -rt, if I'm willing to handle several packets every time I wake up?
For any reaction time less than 100 us (or maybe even higher), you will need -rt.
I'm still unsure whether there is a place in my app where I need a quick reaction time. What's your opinion?
Regards.