Skip to contentSkip to conversation

Fighting out-of-order reception with RPS?

DISCUSSION Oliver Hartkopp 5 participants 16 messages Jul 8, 2015 – Jul 14, 2015

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

I'm picking up the request 'Setting RPS affinities from network driver' from Sunil Kovvuri http://marc.info/?t=142424023500001&r=1&w=2 as I assume to have the same issue here.

When receiving CAN frames from a specific CAN network interface (e.g. can0) the frames are sporadically out-of-order on SMP systems like my Core i7 laptop with 4 CPUs. This out-of-order reception kills reliable communication e.g. for CAN transport protocols.

First approach was to set the smp_affinity for the USB adapter on irq 28 with:

	echo 1 > /proc/irq/28/smp_affinity

This worked in my case but it looks wrong to pin the USB host adapter to a single CPU and has to be done by hand depending on where the CAN interfaces are attached to the system.

Next idea was to use RPS after reading Documentation/networking/scaling.txt

As the only relevant flow identifiction is the number of the incoming CAN interface I added

	skb_set_hash(skb, dev->ifindex, PKT_HASH_TYPE_L2);

when creating CAN skbs with alloc_can_skb() in drivers/net/can/dev.c

After enabling RPS for my four CPUs with

echo f > /sys/class/net/can0/queues/rx-0/rps_cpus

I had no more out-of-order frames in my system :-)

My two questions:

1. Is there any better solution to meet the described requirements?
2. If not: How can enable this RPS solution by default for CAN interfaces?

Best regards,
Oliver

#1 lore raw
Quoted 28 lines · Oliver Hartkopp

I'm picking up the request 'Setting RPS affinities from network driver' from Sunil Kovvuri http://marc.info/?t=142424023500001&r=1&w=2 as I assume to have the same issue here.

When receiving CAN frames from a specific CAN network interface (e.g. can0) the frames are sporadically out-of-order on SMP systems like my Core i7 laptop with 4 CPUs. This out-of-order reception kills reliable communication e.g. for CAN transport protocols.

First approach was to set the smp_affinity for the USB adapter on irq 28 with:

        echo 1 > /proc/irq/28/smp_affinity

This worked in my case but it looks wrong to pin the USB host adapter to a single CPU and has to be done by hand depending on where the CAN interfaces are attached to the system.

Next idea was to use RPS after reading Documentation/networking/scaling.txt

As the only relevant flow identifiction is the number of the incoming CAN interface I added

        skb_set_hash(skb, dev->ifindex, PKT_HASH_TYPE_L2);

when creating CAN skbs with alloc_can_skb() in drivers/net/can/dev.c

After enabling RPS for my four CPUs with

Quoted 7 lines

echo f > /sys/class/net/can0/queues/rx-0/rps_cpus

I had no more out-of-order frames in my system :-)

My two questions:

1. Is there any better solution to meet the described requirements?

I would suggest that you look into how there are OOO packets in the first place. Even if the interrupts is allowed to happen on different CPUs by sm_affinity, NAPI execution should be serialized for the device so that OOO shouldn't happen. The result of your RPS setting should be all packets go to the same queue, this shouldn't normally affect the ordering. Looking at drivers/net/can there are apparently several variants of the driver. Do you know which one you're running?

Tom

Quoted 9 lines

2. If not: How can enable this RPS solution by default for CAN interfaces?

Best regards,
Oliver

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 1 line · Tom Herbert

On Wed, Jul 8, 2015 at 10:49 AM, Oliver Hartkopp <[email protected]> wrote:

(..)

Quoted 6 lines
Quoted 6 lines

When receiving CAN frames from a specific CAN network interface (e.g. can0) the frames are sporadically out-of-order on SMP systems like my Core i7 laptop with 4 CPUs. This out-of-order reception kills reliable communication e.g. for CAN transport protocols.

First approach was to set the smp_affinity for the USB adapter on irq 28 with:

(..)

Quoted 1 line
Quoted 1 line

Next idea was to use RPS after reading Documentation/networking/scaling.txt

(..)

Quoted 5 lines
Quoted 4 lines

My two questions:

1. Is there any better solution to meet the described requirements?

Quoted 7 lines

I would suggest that you look into how there are OOO packets in the first place. Even if the interrupts is allowed to happen on different CPUs by sm_affinity, NAPI execution should be serialized for the device so that OOO shouldn't happen. The result of your RPS setting should be all packets go to the same queue, this shouldn't normally affect the ordering. Looking at drivers/net/can there are apparently several variants of the driver. Do you know which one you're running?

I have two CAN hardware interfaces I can test together with a SMP system:

1. PCAN-USB using the driver at drivers/net/can/usb/peak_usb/
2. PCAN Compact PCIe using drivers/net/can/sja1000/(peak_pci.c / sja1000.c)

Both drivers do not use NAPI. The just follow the way

interrupt -> alloc_skb() -> fill skb -> netif_rx(skb)

I'm usually testing with the USB adapters as the PCIe setup is not very handy.

Best regards,
Oliver

Quoted 2 lines
Quoted 2 lines

2. If not: How can enable this RPS solution by default for CAN interfaces?

#15 lore raw
Quoted 5 lines · Holger Schurig
Quoted 2 lines

and has to be done by hand depending on where the CAN interfaces are attached to the system.

No ... udev rules rock! :-)

Yeah. But it can not be the approach to fix a known problem in the kernel by urging people to make workarounds in userspace. For my testing I did it by hand. But in real life the kernel has to do 'the right thing' by having wise defaults without interaction.

Regards,
Oliver

#3 lore raw
In reply to Oliver Hartkopp
Quoted 46 lines · Oliver Hartkopp
Quoted 3 lines · Tom Herbert

On Wed, Jul 8, 2015 at 10:49 AM, Oliver Hartkopp <[email protected]> wrote:

(..)

Quoted 11 lines
Quoted 11 lines

When receiving CAN frames from a specific CAN network interface (e.g. can0)
the frames are sporadically out-of-order on SMP systems like my Core i7 laptop
with 4 CPUs. This out-of-order reception kills reliable communication e.g. for
CAN transport protocols.

First approach was to set the smp_affinity for the USB adapter on irq 28 with:

(..)

Quoted 3 lines
Quoted 3 lines

Next idea was to use RPS after reading
Documentation/networking/scaling.txt

(..)

Quoted 7 lines
Quoted 5 lines

My two questions:

1. Is there any better solution to meet the described requirements?

Quoted 7 lines

I would suggest that you look into how there are OOO packets in the first place. Even if the interrupts is allowed to happen on different CPUs by sm_affinity, NAPI execution should be serialized for the device so that OOO shouldn't happen. The result of your RPS setting should be all packets go to the same queue, this shouldn't normally affect the ordering. Looking at drivers/net/can there are apparently several variants of the driver. Do you know which one you're running?

I have two CAN hardware interfaces I can test together with a SMP system:

1. PCAN-USB using the driver at drivers/net/can/usb/peak_usb/
2. PCAN Compact PCIe using drivers/net/can/sja1000/(peak_pci.c / sja1000.c)

Quoted 7 lines

Both drivers do not use NAPI. The just follow the way

interrupt -> alloc_skb() -> fill skb -> netif_rx(skb)

I'm usually testing with the USB adapters as the PCIe setup is not very handy.

Okay, I see what is happening. In netif_rx when RPS is not enabled that packet is queued to the backlog queue for the local CPU. Since you're doing round robin on the interrupts then OOO packets can be a result. Unfortunately, this is the expected behavior. The correct kernel fix would be to move to these drivers to use NAPI. RPS eliminates the OOO, but if there is no ability to derive a flow hash from packets everything will wind up one queue without load balancing. Besides that, automatically setting RPS in drivers is a difficult proposition since there is no definitively "correct" way to do that in an arbitrary configuration.

Tom

Quoted 9 lines

Best regards,
Oliver

Quoted 3 lines
Quoted 3 lines

2. If not: How can enable this RPS solution by default for CAN interfaces?

#4 lore raw
Quoted 5 lines · Tom Herbert
Quoted 4 lines · Oliver Hartkopp

Both drivers do not use NAPI. The just follow the way

interrupt -> alloc_skb() -> fill skb -> netif_rx(skb)

Quoted 8 lines
Quoted 3 lines

I'm usually testing with the USB adapters as the PCIe setup is not very handy.

Okay, I see what is happening. In netif_rx when RPS is not enabled that packet is queued to the backlog queue for the local CPU. Since you're doing round robin on the interrupts then OOO packets can be a result. Unfortunately, this is the expected behavior. The correct kernel fix would be to move to these drivers to use NAPI.

Hm. Doesn't sound like a good solution when there's a difference between NAPI and non-NAPI drivers in matters of OOO, right?

Quoted 3 lines

RPS
eliminates the OOO, but if there is no ability to derive a flow hash from packets everything will wind up one queue without load balancing.

Correct.

That's why I added

	skb_set_hash(skb, dev->ifindex, PKT_HASH_TYPE_L2);

in my driver, because the only relevant flow identifiction is the number of the incoming CAN interface.

Quoted 3 lines

Besides that, automatically setting RPS in drivers is a difficult proposition since there is no definitively "correct" way to do that in an arbitrary configuration.

What about checking in netif_rx() if the non-NAPI driver has set a hash (aka the driver is OOO sensitive)?
And if so we could automatically set rps_cpus for this interface in a way that all CPUs are enabled to take skbs following the hash.

Best regards,
Oliver

#5 lore raw
Quoted 9 lines · Oliver Hartkopp
Quoted 8 lines · Tom Herbert
Quoted 7 lines · Oliver Hartkopp

Both drivers do not use NAPI. The just follow the way

interrupt -> alloc_skb() -> fill skb -> netif_rx(skb)

I'm usually testing with the USB adapters as the PCIe setup is not very handy.

Quoted 8 lines
Quoted 5 lines

Okay, I see what is happening. In netif_rx when RPS is not enabled that packet is queued to the backlog queue for the local CPU. Since you're doing round robin on the interrupts then OOO packets can be a result. Unfortunately, this is the expected behavior. The correct kernel fix would be to move to these drivers to use NAPI.

Hm. Doesn't sound like a good solution when there's a difference between NAPI and non-NAPI drivers in matters of OOO, right?

Isn't OOO a problem for you ? Then you either have to :

1) Use a single CPU to handle IRQ from the device
2) Use NAPI

Quoted 14 lines
Quoted 4 lines

RPS
eliminates the OOO, but if there is no ability to derive a flow hash from packets everything will wind up one queue without load balancing.

Correct.

That's why I added

	skb_set_hash(skb, dev->ifindex, PKT_HASH_TYPE_L2);

in my driver, because the only relevant flow identifiction is the number of the incoming CAN interface.

Quoted 8 lines
Quoted 3 lines

Besides that, automatically setting RPS in drivers is a difficult proposition since there is no definitively "correct" way to do that in an arbitrary configuration.

What about checking in netif_rx() if the non-NAPI driver has set a hash (aka the driver is OOO sensitive)?
And if so we could automatically set rps_cpus for this interface in a way that all CPUs are enabled to take skbs following the hash.

Wow, netif_rx() is packet processing fast path, certainly not the place to add controlling path decisions.

Please convert your driver to NAPI. You might then even benefit from GRO.

#6 lore raw

Hello Eric,

Quoted 10 lines · Eric Dumazet
Quoted 2 lines · Oliver Hartkopp

Hm. Doesn't sound like a good solution when there's a difference between NAPI and non-NAPI drivers in matters of OOO, right?

Isn't OOO a problem for you ? Then you either have to :

1) Use a single CPU to handle IRQ from the device
2) Use NAPI

See below ...

Quoted 7 lines
Quoted 4 lines

What about checking in netif_rx() if the non-NAPI driver has set a hash (aka the driver is OOO sensitive)?
And if so we could automatically set rps_cpus for this interface in a way that all CPUs are enabled to take skbs following the hash.

Wow, netif_rx() is packet processing fast path, certainly not the place to add controlling path decisions.

My only requirement is to be able to pick CAN frames (contained in skbs) from the socket in the same order they have been received.

Quoted 2 lines

Please convert your driver to NAPI. You might then even benefit from GRO.

Just some remarks about CAN and CAN frames as you suggest GRO which is completely pointless for CAN.

CAN frames have a 11 or 29 bit CAN Identifier (no MAC but content addressing) and 0 to 64 bytes of payload. Therefore the MTU for CAN interfaces is 16 or 72 byte (see struct can(fd)_frame). Each skbuff contains a single CAN frame.

There are CAN controllers which have a FIFO for up to 32 CAN frames, e.g. flexcan.c which also implements NAPI. Others (e.g. sja1000.c) don't have any FIFO and the reading of the CAN frame from the memory mapped registers needs to be processed in the irq context instantly. So 'fast path' netif_rx() is reasonable, right?

So why is it not possible to pass netif_rx() skbs from a specific CAN network interface to whatever queue where they are processed in order?

E.g. with

	skb_set_hash(skb, dev->ifindex, PKT_HASH_TYPE_L2);

and

echo f > /sys/class/net/can0/queues/rx-0/rps_cpus

I get properly ordered CAN frames - even with netif_rx() processed skbs. I just want to have this stuff to be enabled by default for CAN interfaces to kill the OOO frame issue.

Regards,
Oliver

maintainer
#7 lore raw

From: Oliver Hartkopp <[email protected]>
Date: Sun, 12 Jul 2015 21:15:36 +0200

Quoted 2 lines

Just some remarks about CAN and CAN frames as you suggest GRO which is completely pointless for CAN.

GRO may be pointless for CAN, but NAPI _definitely_ is useful for every single network device, period.

So you should do NAPI for reasons outside of packet receive ordering, and in return you'll have your packet ordering problem solved as well.

I really am stumped as to why you are avoiding NAPI so vehemently.

#8 lore raw
In reply to Oliver Hartkopp
Quoted 11 lines · Oliver Hartkopp

E.g. with

	skb_set_hash(skb, dev->ifindex, PKT_HASH_TYPE_L2);

and

echo f > /sys/class/net/can0/queues/rx-0/rps_cpus

I get properly ordered CAN frames - even with netif_rx() processed skbs. I just want to have this stuff to be enabled by default for CAN interfaces to kill the OOO frame issue.

I doubt your skb_set_hash() makes any difference.

RPS prefers a L4 hash anyway (skb_get_hash()), so flow dissection happens.

#13 lore raw
In reply to Oliver Hartkopp
Quoted 50 lines · Oliver Hartkopp

Hello Eric,

Quoted 10 lines · Eric Dumazet
Quoted 2 lines · Oliver Hartkopp

Hm. Doesn't sound like a good solution when there's a difference between NAPI and non-NAPI drivers in matters of OOO, right?

Isn't OOO a problem for you ? Then you either have to :

1) Use a single CPU to handle IRQ from the device
2) Use NAPI

See below ...

Quoted 7 lines
Quoted 4 lines

What about checking in netif_rx() if the non-NAPI driver has set a hash (aka the driver is OOO sensitive)?
And if so we could automatically set rps_cpus for this interface in a way that all CPUs are enabled to take skbs following the hash.

Wow, netif_rx() is packet processing fast path, certainly not the place to add controlling path decisions.

My only requirement is to be able to pick CAN frames (contained in skbs) from the socket in the same order they have been received.

Quoted 2 lines

Please convert your driver to NAPI. You might then even benefit from GRO.

Just some remarks about CAN and CAN frames as you suggest GRO which is completely pointless for CAN.

CAN frames have a 11 or 29 bit CAN Identifier (no MAC but content addressing) and 0 to 64 bytes of payload. Therefore the MTU for CAN interfaces is 16 or 72 byte (see struct can(fd)_frame). Each skbuff contains a single CAN frame.

There are CAN controllers which have a FIFO for up to 32 CAN frames, e.g. flexcan.c which also implements NAPI. Others (e.g. sja1000.c) don't have any FIFO and the reading of the CAN frame from the memory mapped registers needs to be processed in the irq context instantly. So 'fast path' netif_rx() is reasonable, right?

So why is it not possible to pass netif_rx() skbs from a specific CAN network interface to whatever queue where they are processed in order?

E.g. with

        skb_set_hash(skb, dev->ifindex, PKT_HASH_TYPE_L2);
Quoted 8 lines

and

echo f > /sys/class/net/can0/queues/rx-0/rps_cpus

I get properly ordered CAN frames - even with netif_rx() processed skbs. I just want to have this stuff to be enabled by default for CAN interfaces to kill the OOO frame issue.

If you really must process the CAN FIFO in the hard interrupt then create a private sk_buf queue. In the interrupt, dequeue from FIFO and enqueue on the sk_buf queue. Then schedule NAPI, and when that runs process the sk_buf queue calling call netif_receive_skb for each enqueued skb. Pretty simple actually :-)

Quoted 3 lines

Regards,
Oliver

#9 lore raw
In reply to Eric Dumazet
Quoted 10 lines · Eric Dumazet
Quoted 8 lines · Oliver Hartkopp

E.g. with

	skb_set_hash(skb, dev->ifindex, PKT_HASH_TYPE_L2);

and

echo f > /sys/class/net/can0/queues/rx-0/rps_cpus

Quoted 9 lines
Quoted 3 lines

I get properly ordered CAN frames - even with netif_rx() processed skbs. I just want to have this stuff to be enabled by default for CAN interfaces to kill the OOO frame issue.

I doubt your skb_set_hash() makes any difference.

RPS prefers a L4 hash anyway (skb_get_hash()), so flow dissection happens.

Please take a look into netif_rx_internal() in net/core/dev.c

http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/net/core/dev.c?id=v4.2-rc1#n3486

with CONFIG_RPS netif_rx() takes care about the rps cpu and puts the skb into the correct hash specific queue.

As we usually have several PF_CAN sockets which get CAN frames from a specific CAN interface it makes no sense to enqueue packets into queues sorted by receiving sockets or L4 hash (we don't have L4 addressing on CAN).

The skb_set_hash(skb, dev->ifindex, PKT_HASH_TYPE_L2) makes sure that the skbs from a specific CAN netdev are always processed in the same queue.

When this is not wanted in 'fastpath netif_rx()' why is the CONFIG_RPS section in there?

What is the advantage of implementing NAPI and a 'private sk_buf queue' suggested by Tom in http://marc.info/?l=linux-can&m=143681458003381&w=2 to set the hash as shown and enable rps_cpus?

The latter just looks just like a complexity boost to have a functionality that already exists in netif_rx(). I just want to understand it.

Regards,
Oliver

#10 lore raw
Quoted 42 lines · Oliver Hartkopp
Quoted 18 lines · Eric Dumazet
Quoted 6 lines · Oliver Hartkopp

E.g. with

        skb_set_hash(skb, dev->ifindex, PKT_HASH_TYPE_L2);

and

Quoted 7 lines

echo f > /sys/class/net/can0/queues/rx-0/rps_cpus

I get properly ordered CAN frames - even with netif_rx() processed skbs. I
just want to have this stuff to be enabled by default for CAN interfaces to
kill the OOO frame issue.

Quoted 5 lines

I doubt your skb_set_hash() makes any difference.

RPS prefers a L4 hash anyway (skb_get_hash()), so flow dissection happens.

Please take a look into netif_rx_internal() in net/core/dev.c

http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/net/core/dev.c?id=v4.2-rc1#n3486

with CONFIG_RPS netif_rx() takes care about the rps cpu and puts the skb into the correct hash specific queue.

As we usually have several PF_CAN sockets which get CAN frames from a specific CAN interface it makes no sense to enqueue packets into queues sorted by receiving sockets or L4 hash (we don't have L4 addressing on CAN).

The skb_set_hash(skb, dev->ifindex, PKT_HASH_TYPE_L2) makes sure that the skbs from a specific CAN netdev are always processed in the same queue.

When this is not wanted in 'fastpath netif_rx()' why is the CONFIG_RPS section in there?

Quoted 7 lines

What is the advantage of implementing NAPI and a 'private sk_buf queue' suggested by Tom in http://marc.info/?l=linux-can&m=143681458003381&w=2 to set the hash as shown and enable rps_cpus?

The latter just looks just like a complexity boost to have a functionality that already exists in netif_rx(). I just want to understand it.

netif_rx allows OOO packets and the way you're fixing that with RPS and setting a "flow hash" are nothing more than hacks. Sorry, but there is no way we are going to change RPS just to satisfy this your case. If you want to do this right and get something in the kernel, then implement NAPI for CAN drivers as has been suggested now by three very experienced developers. This solves the your OOO problem and moves drivers to NAPI which is the greatly preferred interface. I really don't see why there needs to be any more discussion on this.

Tom

Quoted 4 lines

Regards,
Oliver

maintainer
#11 lore raw

From: Tom Herbert <[email protected]>
Date: Tue, 14 Jul 2015 11:02:16 -0700

Quoted 3 lines

then implement NAPI for CAN drivers as has been suggested now by three very experienced developers. This solves the your OOO problem and moves drivers to NAPI which is the greatly preferred interface.

+1

+1

+1

#12 lore raw
Quoted 7 lines · David Miller

From: Tom Herbert <[email protected]>
Date: Tue, 14 Jul 2015 11:02:16 -0700

Quoted 3 lines

then implement NAPI for CAN drivers as has been suggested now by three very experienced developers. This solves the your OOO problem and moves drivers to NAPI which is the greatly preferred interface.

Quoted 5 lines

+1

+1

+1

ok got it :-)