Strange performance issue when using two devices at once

Hi everyone,

I have two RT5370 devices connected to the same access point. Both devices are very slow, but the instant I disconnect one device, the other speeds up by a factor of 10.

The really strange part is that one device will perform slowly even if the other device is basically idle! I've confirmed this with a packet sniffer.

I've been trying to do some debugging, and I've found that when both devices are connected to the access point, they report a large number of duplicate frames. I added some debug output
in ieee80211_rx_h_check_dup() to confirm that this only happens while both devices are connected. The packet sniffer also shows a large number of retries while this is occurring.

Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.

I did post about this previously on this mailing list (RT5370 performance issues), but I thought I'd post again with this new information and more descriptive title. I'm a little bit stuck on this for a while now, so any ideas are much appreciated.

Thanks!

Marlon

Quoted 3 lines · Marlon Smith

Hi everyone,

Quoted 3 lines

I have two RT5370 devices connected to the same access point. Both devices are very slow, but the instant I disconnect one device, the other speeds up by a factor of 10.

Out of curiosity, are both of the RT5370 used on the same client device?

Quoted 2 lines

Did you check that they have unique MAC addresses? Thanks,

Ben

Quoted 30 lines

Quoted 1 line

The really strange part is that one device will perform slowly even if
the other device is basically idle! I've confirmed this with a packet
sniffer.

I've been trying to do some debugging, and I've found that when both
devices are connected to the access point, they report a large number
of duplicate frames. I added some debug output
in ieee80211_rx_h_check_dup() to confirm that this only happens while
both devices are connected. The packet sniffer also shows a large number of retries while this is occurring.

Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.

I did post about this previously on this mailing list (RT5370 performance issues), but I thought I'd post again with this new information and more descriptive title. I'm a little bit stuck on this
for a while now, so any ideas are much appreciated.

Thanks!

Marlon

Quoted 4 lines · Ben Greear

Quoted 3 lines · Marlon Smith

Hi everyone,

Quoted 38 lines

Quoted 3 lines

I have two RT5370 devices connected to the same access point. Both devices are very slow, but the instant I disconnect one device, the other speeds up by a factor of 10.

Out of curiosity, are both of the RT5370 used on the same client device?

Quoted 2 lines

Did you check that they have unique MAC addresses? Thanks,

Ben

Quoted 30 lines

Quoted 1 line

The really strange part is that one device will perform slowly even if
the other device is basically idle! I've confirmed this with a packet
sniffer.

I've been trying to do some debugging, and I've found that when both
devices are connected to the access point, they report a large number
of duplicate frames. I added some debug output
in ieee80211_rx_h_check_dup() to confirm that this only happens while
both devices are connected. The packet sniffer also shows a large number of retries while this is occurring.

Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.

I did post about this previously on this mailing list (RT5370 performance issues), but I thought I'd post again with this new information and more descriptive title. I'm a little bit stuck on this
for a while now, so any ideas are much appreciated.

Thanks!

Marlon

They are on separate devices, although the mac addresses are close. 70:F1:1C:2E:AF:B4 and
70:F1:1C:2E:AF:B6.

However, I have a third device 70:F1:1C:2E:AF:BB which performs well and does not affect the performance of the other two.

Quoted 53 lines · Marlon Smith

Quoted 11 lines · Ben Greear

Quoted 9 lines · Marlon Smith

Hi everyone,

I have two RT5370 devices connected to the same access point. Both
devices are very slow, but the instant I disconnect one device, the
other speeds up by a factor of 10.

Quoted 40 lines

Out of curiosity, are both of the RT5370 used on the same client device?

Quoted 3 lines

Did you check that they have unique MAC addresses? Thanks,

Ben

Quoted 34 lines

Quoted 3 lines

The really strange part is that one device will perform slowly even
if
the other device is basically idle! I've confirmed this with a packet
sniffer.

I've been trying to do some debugging, and I've found that when both
devices are connected to the access point, they report a large number
of duplicate frames. I added some debug output
in ieee80211_rx_h_check_dup() to confirm that this only happens while
both devices are connected. The packet sniffer also shows a large
number of retries while this is occurring.

Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.

I did post about this previously on this mailing list (RT5370 performance issues), but I thought I'd post again with this new information and more descriptive title. I'm a little bit stuck on
this
for a while now, so any ideas are much appreciated.

Thanks!

Marlon

Quoted 8 lines

They are on separate devices, although the mac addresses are close. 70:F1:1C:2E:AF:B4 and
70:F1:1C:2E:AF:B6.

However, I have a third device 70:F1:1C:2E:AF:BB which performs well
and does not affect the performance of the other two.

Have you tried a different AP?

And also tried using the exact same MAC addresses configured on a different
radio (like ath9k)?

Thanks,
Ben

Quoted 68 lines · Ben Greear

Quoted 53 lines · Marlon Smith

Quoted 11 lines · Ben Greear

Quoted 9 lines · Marlon Smith

Hi everyone,

I have two RT5370 devices connected to the same access point. Both
devices are very slow, but the instant I disconnect one device, the
other speeds up by a factor of 10.

Quoted 40 lines

Out of curiosity, are both of the RT5370 used on the same client device?

Quoted 3 lines

Did you check that they have unique MAC addresses? Thanks,

Ben

Quoted 34 lines

Quoted 3 lines

The really strange part is that one device will perform slowly even
if
the other device is basically idle! I've confirmed this with a packet
sniffer.

I've been trying to do some debugging, and I've found that when both
devices are connected to the access point, they report a large number
of duplicate frames. I added some debug output
in ieee80211_rx_h_check_dup() to confirm that this only happens while
both devices are connected. The packet sniffer also shows a large
number of retries while this is occurring.

Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.

I did post about this previously on this mailing list (RT5370 performance issues), but I thought I'd post again with this new information and more descriptive title. I'm a little bit stuck on
this
for a while now, so any ideas are much appreciated.

Thanks!

Marlon

Quoted 8 lines

They are on separate devices, although the mac addresses are close. 70:F1:1C:2E:AF:B4 and
70:F1:1C:2E:AF:B6.

However, I have a third device 70:F1:1C:2E:AF:BB which performs well
and does not affect the performance of the other two.

Have you tried a different AP?

And also tried using the exact same MAC addresses configured on a different
radio (like ath9k)?

Quoted 4 lines

Thanks,
Ben

I have tried a different access point in a different environment but no luck. I'll see if I can configure my laptop to use one of the problematic devices' mac address.

Quoted 90 lines · Marlon Smith

Quoted 88 lines · Ben Greear

Quoted 71 lines · Marlon Smith

Quoted 15 lines · Ben Greear

Quoted 5 lines · Marlon Smith

Hi everyone,

Quoted 7 lines

I have two RT5370 devices connected to the same access point.
Both
devices are very slow, but the instant I disconnect one device,
the
other speeds up by a factor of 10.

Quoted 53 lines

Out of curiosity, are both of the RT5370 used on the same client
device?

Quoted 4 lines

Did you check that they have unique MAC addresses? Thanks,

Ben

Quoted 45 lines

Quoted 5 lines

The really strange part is that one device will perform slowly
even
if
the other device is basically idle! I've confirmed this with a
packet
sniffer.

I've been trying to do some debugging, and I've found that when
both
devices are connected to the access point, they report a large
number
of duplicate frames. I added some debug output
in ieee80211_rx_h_check_dup() to confirm that this only happens
while
both devices are connected. The packet sniffer also shows a large
number of retries while this is occurring.

Using backports 5.3-rc4 for this, but also tested on 4.14- rc2.

I did post about this previously on this mailing list (RT5370
performance issues), but I thought I'd post again with this new
information and more descriptive title. I'm a little bit stuck
on
this
for a while now, so any ideas are much appreciated.

Thanks!

Marlon

Quoted 9 lines

They are on separate devices, although the mac addresses are close.
70:F1:1C:2E:AF:B4 and
70:F1:1C:2E:AF:B6.

However, I have a third device 70:F1:1C:2E:AF:BB which performs well
and does not affect the performance of the other two.

Have you tried a different AP?

And also tried using the exact same MAC addresses configured on a different
radio (like ath9k)?

Quoted 8 lines

Quoted 4 lines

Thanks,
Ben

I have tried a different access point in a different environment but no
luck. I'll see if I can configure my laptop to use one of the problematic devices' mac address.

It might be tricky to determine, but if you can notice whether one of your station devices
is (block-)acking the other's frames, that would be a good clue that it is a station
side bug.  A carefully inspected sniff, especially if you can put sniffer near one station
and far from the other and so use RSSI as a sorting factor, should allow you to determine
this.

Thanks,
Ben

Quoted 90 lines · Marlon Smith

Quoted 88 lines · Ben Greear

Quoted 71 lines · Marlon Smith

Quoted 15 lines · Ben Greear

Quoted 5 lines · Marlon Smith

Hi everyone,

Quoted 7 lines

I have two RT5370 devices connected to the same access point.
Both
devices are very slow, but the instant I disconnect one device,
the
other speeds up by a factor of 10.

Quoted 53 lines

Out of curiosity, are both of the RT5370 used on the same client
device?

Quoted 4 lines

Did you check that they have unique MAC addresses? Thanks,

Ben

Quoted 45 lines

Quoted 5 lines

The really strange part is that one device will perform slowly
even
if
the other device is basically idle! I've confirmed this with a
packet
sniffer.

I've been trying to do some debugging, and I've found that when
both
devices are connected to the access point, they report a large
number
of duplicate frames. I added some debug output
in ieee80211_rx_h_check_dup() to confirm that this only happens
while
both devices are connected. The packet sniffer also shows a large
number of retries while this is occurring.

Using backports 5.3-rc4 for this, but also tested on 4.14- rc2.

I did post about this previously on this mailing list (RT5370
performance issues), but I thought I'd post again with this new
information and more descriptive title. I'm a little bit stuck
on
this
for a while now, so any ideas are much appreciated.

Thanks!

Marlon

Quoted 9 lines

They are on separate devices, although the mac addresses are close.
70:F1:1C:2E:AF:B4 and
70:F1:1C:2E:AF:B6.

However, I have a third device 70:F1:1C:2E:AF:BB which performs well
and does not affect the performance of the other two.

Have you tried a different AP?

And also tried using the exact same MAC addresses configured on a different
radio (like ath9k)?

Quoted 8 lines

Quoted 4 lines

Thanks,
Ben

I have tried a different access point in a different environment but no
luck. I'll see if I can configure my laptop to use one of the problematic devices' mac address.

It might be tricky to determine, but if you can notice whether one of your station devices
is (block-)acking the other's frames, that would be a good clue that it is a station
side bug.  A carefully inspected sniff, especially if you can put sniffer near one station
and far from the other and so use RSSI as a sorting factor, should allow you to determine
this.

Thanks,
Ben

So I did a few tests and got some pretty interesting results.

With the two devices connected to the access point and performing slowly, I changed the MAC address of one of the devices (Just from a B6 to a BA). Immediately, both devices began working properly!

Back to the original MAC address, I looked at my sniffer, and I do not see any evidence of one device acking another device's frames. With one device sitting idle and the other transferring data, I see high numbers of retransmitted frames, and then an eventual ack from the correct device, but very little activity from the idle device. While this is happening, the device driver indicates that it is dropping lots of duplicate received packets.

Quoted 8 lines · Marlon Smith

So I did a few tests and got some pretty interesting results.

With the two devices connected to the access point and performing slowly, I changed the MAC address of one of the devices (Just from a B6
to a BA). Immediately, both devices began working properly!

Quoted 11 lines

Back to the original MAC address, I looked at my sniffer, and I do not
see any evidence of one device acking another device's frames. With one
device sitting idle and the other transferring data, I see high numbers
of retransmitted frames, and then an eventual ack from the correct device, but very little activity from the idle device. While this is
happening, the device driver indicates that it is dropping lots of duplicate received packets.

Here's the cause and an approach for fixing it:

The MAC_BSSID_DW1 register configures the Multi-BSSID mode. It can be configured for 1, 2, 4 or 8 BSSIDs. This controls what MAC addresses the
chip will respond to. The driver always sets it up for 8 BSSIDs.

The problem here is that the chip has a hardcoded pattern for how those
BSSIDs are allocated:
Depending on the number of BSSIDs, the lower 1, 2 or 3 bits are used to
encode the BSS index. This means that when 8 BSSIDs are enabled, the lower 3 bits are effectively masked out on the MAC address filter that
decides which frames to ACK. Because your MAC addresses are so close together, both devices try to ACK the same frames at the same time. This
will not be visible in a sniffer, since they will be transmitting at the
same time.

I'd say the first step in fixing this is to make the driver set MAC_BSSID_DW1_BSS_ID_MASK based on the MAC addresses of configured virtual interfaces. For older chips, this MAC address pattern cannot be
fixed.

However, on at least RT3883, RT3352 and RT5350 (and thus most likely also RT5370), you can set BIT 21 in that register, which changes the pattern. With that bit set, the second interface only sets the local bit
of the MAC address, and from the third interface on it adds (idx- 1)<<2
to the first byte. That gets rid of the MAC address conflicts.

Hopefully someone familiar with the rt2x00 code will read this and write
some patches :)

- Felix

-----Message d'origine-----
De : [email protected] [mailto:[email protected]] De la part de Marlon Smith
Envoyé : mercredi 29 janvier 2020 19:40
À : [email protected]
Objet : Strange performance issue when using two devices at once

Hi everyone,

I have two RT5370 devices connected to the same access point. Both devices are very slow, but the instant I disconnect one device, the other speeds up by a factor of 10.

The really strange part is that one device will perform slowly even if the other device is basically idle! I've confirmed this with a packet sniffer.

I've been trying to do some debugging, and I've found that when both devices are connected to the access point, they report a large number of duplicate frames. I added some debug output
in ieee80211_rx_h_check_dup() to confirm that this only happens while both devices are connected. The packet sniffer also shows a large number of retries while this is occurring.

Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.

I did post about this previously on this mailing list (RT5370 performance issues), but I thought I'd post again with this new information and more descriptive title. I'm a little bit stuck on this for a while now, so any ideas are much appreciated.

Thanks!

Marlon

Maybe you need to put some distance between the 3 devices. We experienced this kind of problem on radios close to each other (around 10cm).
It was due to radio signals inducing themselves in the PCB of another nearby radio, causing the other radio to badly transmit and receive.

Quoted 42 lines

-----Message d'origine-----
De : [email protected] [mailto:linux-wireless-ow [email protected]] De la part de
Marlon Smith
Envoyé : mercredi 29 janvier 2020 19:40
À : [email protected]
Objet : Strange performance issue when using two devices at once

Hi everyone,

I have two RT5370 devices connected to the same access point. Both devices are very slow, but the instant I disconnect one device, the other speeds up by a factor of 10.

The really strange part is that one device will perform slowly even if
the other device is basically idle! I've confirmed this with a packet
sniffer.

I've been trying to do some debugging, and I've found that when both
devices are connected to the access point, they report a large number
of duplicate frames. I added some debug output
in ieee80211_rx_h_check_dup() to confirm that this only happens while
both devices are connected. The packet sniffer also shows a large number of retries while this is occurring.

Using backports 5.3-rc4 for this, but also tested on 4.14-rc2.

I did post about this previously on this mailing list (RT5370 performance issues), but I thought I'd post again with this new information and more descriptive title. I'm a little bit stuck on this
for a while now, so any ideas are much appreciated.

Thanks!

Marlon