macvtap performance regression (bisected) between 3.13 and 3.14-rc1

Vlad,

commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
macvtap: Add support of packet capture on macvtap device.

causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card.
Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec

Latency seems to be unchanges (uperf 1byte ping pong).

According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 40000 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed).

Any ideas?

Christian

Quoted 18 lines · Christian Borntraeger

Vlad,

commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
macvtap: Add support of packet capture on macvtap device.

causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card.
Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec

Latency seems to be unchanges (uperf 1byte ping pong).

According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 40000 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed).

Quoted 1 line

Any ideas?

I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput.

Quoted 19 lines · Vlad Yasevich

Quoted 18 lines · Christian Borntraeger

Vlad,

commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
macvtap: Add support of packet capture on macvtap device.

causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card.
Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec

Latency seems to be unchanges (uperf 1byte ping pong).

According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 40000 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed).

Quoted 4 lines

Quoted 1 line

Any ideas?

I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput.

Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests.

Maybe you remember, we had a similar situation with commit 3e4f8b787370978733ca6cae452720a4f0c296b8 (macvtap: Perform GSO on forwarding path), the setup is basically the same.

Christian

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 20 lines · Christian Borntraeger

Quoted 19 lines · Vlad Yasevich

Quoted 18 lines · Christian Borntraeger

Vlad,

commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
macvtap: Add support of packet capture on macvtap device.

causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card.
Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec

Latency seems to be unchanges (uperf 1byte ping pong).

According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 40000 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed).

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

Any ideas?

I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput.

Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests.

Could you post 'ethtool -k' output for both lower-level device and the macvtap device?

Thanks
-vlad

Quoted 12 lines

Maybe you remember, we had a similar situation with commit 3e4f8b787370978733ca6cae452720a4f0c296b8 (macvtap: Perform GSO on forwarding path), the setup is basically the same.

Christian

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 30 lines · Vlad Yasevich

Quoted 20 lines · Christian Borntraeger

Quoted 19 lines · Vlad Yasevich

Quoted 18 lines · Christian Borntraeger

Vlad,

commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
macvtap: Add support of packet capture on macvtap device.

causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card.
Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec

Latency seems to be unchanges (uperf 1byte ping pong).

According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 40000 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed).

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

Any ideas?

I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput.

Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests.

Quoted 6 lines

Could you post 'ethtool -k' output for both lower-level device and the macvtap device?

Thanks
-vlad

Ok. I think I see what's happening. Since you turn off offloads on lower device, that's propagated to macvlan device. As a result, when when we call dev_queue_xmit on the vlan->dev, we end up segmenting since lower level says it does support segmentation.

One way to fix this is to never disable offloads on macvlan. macvlan will always try to use __dev_queue_xmit() with it's lower device, so any segmentation can happen there.

Quoted 43 lines · Vlad Yasevich

Quoted 30 lines · Vlad Yasevich

Quoted 20 lines · Christian Borntraeger

Quoted 19 lines · Vlad Yasevich

Quoted 18 lines · Christian Borntraeger

Vlad,

commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
macvtap: Add support of packet capture on macvtap device.

causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card.
Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec

Latency seems to be unchanges (uperf 1byte ping pong).

According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 40000 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed).

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

Any ideas?

I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput.

Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests.

Quoted 6 lines

Could you post 'ethtool -k' output for both lower-level device and the macvtap device?

Thanks
-vlad

Ok. I think I see what's happening. Since you turn off offloads on lower device, that's propagated to macvlan device. As a result, when when we call dev_queue_xmit on the vlan->dev, we end up segmenting since lower level says it does support segmentation.

Quoted 3 lines

One way to fix this is to never disable offloads on macvlan. macvlan will always try to use __dev_queue_xmit() with it's lower device, so any segmentation can happen there.

If you have anything that I should test, let me know.

Christian

Quoted 25 lines · Christian Borntraeger

Quoted 23 lines · Vlad Yasevich

Quoted 11 lines · Christian Borntraeger

Vlad,

commit 6acf54f1cf0a6747bac9fea26f34cfc5a9029523
macvtap: Add support of packet capture on macvtap device.

causes a performance regression for iperf traffic between two KVM guests on my s390 system. Both guests are connected via two macvtaps on the same OSA network card.
Before that patch I get ~20 Gbit/sec between two guests, afterwards I get ~4Gbit/sec

Quoted 8 lines

Latency seems to be unchanges (uperf 1byte ping pong).

According to ifconfig in the guest, I have ~ 1500 bytes per packet with this patch and ~ 40000 bytes without. So for some reason this patch causes the network stack to do segmentation. (the guest kernel stays the same, only host kernel is changed).

Any ideas?

I am looking. It shouldn't cause addition segmentations and when I ran netperf on the code I didn't see any difference in the throughput.

Quoted 3 lines

Dont know if the different bytes/packets ratio is really the reason or just a side effect. As a hint: the underlying network device does not support segmentation, but this should not matter for traffic between to guests.

Could you post 'ethtool -k' output for both lower-level device and the macvtap device?