PPC440GX ethernet oddities
I'm having a slightly strange behavior with PPC440GX ethernet, I'm looking for a little advice where I can poke around to see what's going on.
I have a custom 440GX board, I use the two RGMII gigabit interfaces to two Vistesse PHYs. This works nicely.
The board has a large FPGA signal processor that is DMA'ing data into main memory, the PPC sends data from main memory out the ethernet interfaces. This all works well. For testing purposes I'm DMA'ing a pseudo random sequence at 80MB/s, sending this over ethernet on a TCP socket to a server machine and checking the sequence at the receiving end. So far so good. Runs for days on several prototype machines.
As part of the DMA diagnostic program I keep track of the maximum occupied capacity of the main memory ring buffer holding data from my FPGA device driver. This lets me keep track of how close I get to a buffer overflow seeing as I'm running the gigabit ethernet port close to the edge at 80MB/s. The ring buffer will typically reach a maximum level of 512kB. This is how far the network connection gets behind the realtime DMA from the FPGAs.
Here's the weird part. On one of the four prototype boxes, if I plug the second ethernet port into gigabit switch and get a link light (2nd interface is not enabled under linux), the DMA behavior will change and I can see the ring buffer get as large as 25MB (up from 512kB!)
Only one of my four boxes shows this strange behavior, and only when the second ethernet port is connected to an ethernet switch. Everything still works properly, my 80MB/s pseudo random sequence is still generated by the FPGAs and checked by a server on the other end of the network connection. I let the ring buffer get as large as 64MB before failing, but the large ring buffer says that the network connection sometimes gets as much as 25MB behind the FPGA DMA, or 25/80 = 0.3125 seconds, which seems kind of crazy.
I look at "ifconfig" (busybox ifconfig) and I see no errors on the ethernet interface. I'm guessing there might be some design problem or maybe just a problem with this one particular board that is causing errors that occasionally slows down the TCP connection, perhaps crosstalk between the two RGMII interfaces or maybe some interaction between the magnetics on the two ports, but I can't figure out where to look to measure errors on the physical ethernet interface.
Can someone give me a hint about where to look for this problem? This is a 2.6.15 kernel.
Thanks for reading, I went on a bit long...
jeff
Quoted 5 lines · Jeff Mock
I'm having a slightly strange behavior with PPC440GX ethernet, I'm looking for a little advice where I can poke around to see what's going on.
I have a custom 440GX board, I use the two RGMII gigabit interfaces to two Vistesse PHYs. This works nicely.
What is the CPU clock speed?
Quoted 6 lines
The board has a large FPGA signal processor that is DMA'ing data into main memory, the PPC sends data from main memory out the ethernet interfaces. This all works well. For testing purposes I'm DMA'ing a pseudo random sequence at 80MB/s, sending this over ethernet on a TCP socket to a server machine and checking the sequence at the receiving end.
Could you elaborate a little here. Is it a user-space program or some kernel mode code which does the copying?
[snip]
Quoted 8 lines
Only one of my four boxes shows this strange behavior, and only when the second ethernet port is connected to an ethernet switch. Everything still works properly, my 80MB/s pseudo random sequence is still generated by the FPGAs and checked by a server on the other end of the network connection. I let the ring buffer get as large as 64MB before failing, but the large ring buffer says that the network connection sometimes gets as much as 25MB behind the FPGA DMA, or 25/80 = 0.3125 seconds, which seems kind of crazy.
Well, 300ms doesn't look particularly crazy to me given a data stream and the fact you are using non-realtime OSes on both ends.
Quoted 3 lines
I look at "ifconfig" (busybox ifconfig) and I see no errors on the ethernet interface.
Try ethtool -S, EMAC driver supports it.
Quoted 9 lines
I'm guessing there might be some design problem or
maybe just a problem with this one particular board that is causing errors that occasionally slows down the TCP connection, perhaps crosstalk between the two RGMII interfaces or maybe some interaction between the magnetics on the two ports, but I can't figure out where to look to measure errors on the physical ethernet interface.Can someone give me a hint about where to look for this problem? This is a 2.6.15 kernel.
There are some stuff you can do:
1) Try another GigE switch, better if from the different vendor
2) Try another peer system you are streaming your data to (better with different GigE NIC)
3) Check if hw flow-control is enabled and try disabling it on both ends
4) Play with socket buffer sizes on both ends
5) And the final, but the most useful one - get a realtime capture of the TCP stream when your ring buffer starts growing. The best way is to get GigE switch which supports port mirroring or use special hw network analyzer.
Signature · 1 line
Eugene
Quoted 9 lines · Eugene Surovegin
Quoted 5 lines · Jeff Mock
I'm having a slightly strange behavior with PPC440GX ethernet, I'm looking for a little advice where I can poke around to see what's going on.
I have a custom 440GX board, I use the two RGMII gigabit interfaces to two Vistesse PHYs. This works nicely.
What is the CPU clock speed?
They are 800MHz rev.F 440GX's.
Quoted 10 lines
Quoted 6 lines
The board has a large FPGA signal processor that is DMA'ing data into main memory, the PPC sends data from main memory out the ethernet interfaces. This all works well. For testing purposes I'm DMA'ing a pseudo random sequence at 80MB/s, sending this over ethernet on a TCP socket to a server machine and checking the sequence at the receiving end.
Could you elaborate a little here. Is it a user-space program or some kernel mode code which does the copying?
The device driver for the FPGA allocates 64MB of memory at boot time and uses this as a ring buffer. The user process mmap()s the entire buffer. When the user process does a read it blocks until data is available and the device driver returns pointers into the ring buffer for the currently available data. The user process passes pointers to the ring buffer to the write() used to send data out the network connection.
This isn't terribly memory efficient, but the embedded 440GX main purpose in life is to move data from the FPGAs across the network. At 80MB/s the 440GX CPU seems to be about 40% utilized.
Quoted 4 lines
Well, 300ms doesn't look particularly crazy to me given a data stream and the fact you are using non-realtime OSes on both ends.
I'm starting to think that you might be right about that. The odd thing that caused me to seek community help is that the ring buffer only fills to about 512kB under normal circumstances, but fills to 25MB if I connect the second Gb ethernet port of the 440GX.
Quoted 2 lines
Try ethtool -S, EMAC driver supports it.
This is a great idea, I didn't realize that ethtool supports the IBM EMAC driver. I suspected that I would find some electrical problem that was causing additional packet errors when the second ethernet port is connected. A ran about 2TB last night, ethtool -S on the PPC says:
# ./ethtool -S eth0
NIC statistics:
rx_packets: 683430611
rx_bytes: 45405076316
tx_packets: 1331225622
tx_bytes: 2005276384145
rx_packets_csum: 683428670
tx_packets_csum: 1331224150
tx_undo: 0
rx_dropped_stack: 80
rx_dropped_oom: 0
rx_dropped_error: 0
[lots more 0's...]The number of checksum errors seems okay and doesn't really change when I plug in the second network connection, but the ring buffer usage still increases dramatically.
Thanks for the other suggestions. I have a friend at a 'real' company where I might be able to borrow a network analyzer.
jeff
Quoted 6 lines · Jeff Mock
Quoted 4 lines · Eugene Surovegin
Quoted 3 lines · Jeff Mock
I'm having a slightly strange behavior with PPC440GX ethernet, I'm looking for a little advice where I can poke around to see what's going on.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I have a custom 440GX board, I use the two RGMII gigabit interfaces to two Vistesse PHYs. This works nicely.
What is the CPU clock speed?
They are 800MHz rev.F 440GX's.
OK, this explains why you can get 80MB/s over Linux sockets :)
Quoted 19 lines
Quoted 11 lines
Quoted 7 lines
The board has a large FPGA signal processor that is DMA'ing data into main memory, the PPC sends data from main memory out the ethernet interfaces. This all works well. For testing purposes I'm DMA'ing a pseudo random sequence at 80MB/s, sending this over ethernet on a TCP socket to a server machine and checking the sequence at the receiving end.
Could you elaborate a little here. Is it a user-space program or some kernel mode code which does the copying?
The device driver for the FPGA allocates 64MB of memory at boot time and uses this as a ring buffer. The user process mmap()s the entire buffer. When the user process does a read it blocks until data is available and the device driver returns pointers into the ring buffer for the currently available data. The user process passes pointers to the ring buffer to the
write()used to send data out the network connection.
Quoted 3 lines
This isn't terribly memory efficient, but the embedded 440GX main purpose in life is to move data from the FPGAs across the network. At 80MB/s the 440GX CPU seems to be about 40% utilized.
Yeah, you probably can save one memory copy by using kernel mode code, but 40% seems OK if this is the only thing 440GX has to do.
Quoted 9 lines
Quoted 4 lines
Well, 300ms doesn't look particularly crazy to me given a data stream and the fact you are using non-realtime OSes on both ends.
I'm starting to think that you might be right about that. The odd thing that caused me to seek community help is that the ring buffer only fills to about 512kB under normal circumstances, but fills to 25MB if I connect the second Gb ethernet port of the 440GX.
Although it's interesting to figure out what caused this change in behaviour, you have to keep in mind that TCP isn't a real-time protocol and it can create some jitter. You have to be prepared to buffer upto several seconds of your data when using TCP on local LAN and more if using intranet/Internet.
Quoted 24 lines
Quoted 4 lines
Try ethtool -S, EMAC driver supports it.
This is a great idea, I didn't realize that ethtool supports the IBM EMAC driver. I suspected that I would find some electrical problem that was causing additional packet errors when the second ethernet port is connected. A ran about 2TB last night, ethtool -S on the PPC says:
# ./ethtool -S eth0 NIC statistics: rx_packets: 683430611 rx_bytes: 45405076316 tx_packets: 1331225622 tx_bytes: 2005276384145 rx_packets_csum: 683428670 tx_packets_csum: 1331224150 tx_undo: 0 rx_dropped_stack: 80 rx_dropped_oom: 0 rx_dropped_error: 0 [lots more 0's...]
Quoted 3 lines
The number of checksum errors seems okay and doesn't really change when I plug in the second network connection, but the ring buffer usage still increases dramatically.
Stats look OK. BTW, tx/rx_packets_csum is not a number of checksum errors, that's a number of packets where driver used hw checksum acceleration :).
Signature · 1 line
Eugene
Quoted 15 lines · Eugene Surovegin
Quoted 14 lines · Jeff Mock
# ./ethtool -S eth0 NIC statistics: rx_packets: 683430611 rx_bytes: 45405076316 tx_packets: 1331225622 tx_bytes: 2005276384145 rx_packets_csum: 683428670 tx_packets_csum: 1331224150 tx_undo: 0 rx_dropped_stack: 80 rx_dropped_oom: 0 rx_dropped_error: 0 [lots more 0's...]
Quoted 8 lines
Quoted 3 lines
The number of checksum errors seems okay and doesn't really change when I plug in the second network connection, but the ring buffer usage still increases dramatically.
Stats look OK. BTW, tx/rx_packets_csum is not a number of checksum errors, that's a number of packets where driver used hw checksum acceleration :).
Ah, silly me. Can you tell me where I can find the number of received packets with bad ethernet checksums? I'm still thinking that I might have a little hardware problem that is increasing the bit error rate when I use both ports, ultimately increasing my buffer utilization.
thanks,
jeff
Quoted 27 lines · Jeff Mock
Quoted 15 lines · Eugene Surovegin
Quoted 14 lines · Jeff Mock
# ./ethtool -S eth0 NIC statistics: rx_packets: 683430611 rx_bytes: 45405076316 tx_packets: 1331225622 tx_bytes: 2005276384145 rx_packets_csum: 683428670 tx_packets_csum: 1331224150 tx_undo: 0 rx_dropped_stack: 80 rx_dropped_oom: 0 rx_dropped_error: 0 [lots more 0's...]Quoted 8 lines
Quoted 3 lines
The number of checksum errors seems okay and doesn't really change when I plug in the second network connection, but the ring buffer usage still increases dramatically.
Stats look OK. BTW, tx/rx_packets_csum is not a number of checksum errors, that's a number of packets where driver used hw checksum acceleration :).
Quoted 4 lines
Ah, silly me. Can you tell me where I can find the number of received packets with bad ethernet checksums? I'm still thinking that I might have a little hardware problem that is increasing the bit error rate when I use both ports, ultimately increasing my buffer utilization.
Packets with bad ethernet checksums are dropped by the MAC before being handed off to MAL. You can disable this behavior by setting RFP bit in EMACx_RMR register. In this case I think IP stack will drop these packets because IP and/or TCP checksum will fail. Use netstat -s to check network stack statistics.
Here is untested patch which enables RFP bit. It's against recent-ish kernel.org git tree:
drivers/net/ibm_emac/ibm_emac_core.c +1 -1
@@ -372,7 +372,7 @@ static void emac_hash_mc(struct ocp_enetstatic inline u32 emac_iff2rmr (struct net_device *ndev) {u32 r =EMAC_RMR_SP |EMAC_RMR_SFCS |EMAC_RMR_IAE |EMAC_RMR_BAE | -EMAC_RMR_BASE ; +EMAC_RMR_BASE |EMAC_RMR_RFP ;if (ndev->flags &IFF_PROMISC ) r |=EMAC_RMR_PME ;
Signature · 1 line
Eugene
Quoted 8 lines · In message <[email protected]> you wrote:
Here's the weird part. On one of the four prototype boxes, if I plug the second ethernet port into gigabit switch and get a link light (2nd interface is not enabled under linux), the DMA behavior will change and I can see the ring buffer get as large as 25MB (up from 512kB!)
Only one of my four boxes shows this strange behavior, and only when the second ethernet port is connected to an ethernet switch. Everything
Are you using exactly the same switch, and the same port on the switch, and the same network cable with all of your boxen?
Best regards,
Wolfgang Denk
Signature · 4 lines
Software Engineering: Embedded and Realtime Systems, Embedded Linux Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] The universe does not have laws - it has habits, and habits can be broken.
Quoted 6 lines · Wolfgang Denk
Quoted 5 lines · In message <[email protected]> you wrote:
Here's the weird part. On one of the four prototype boxes, if I plug the second ethernet port into gigabit switch and get a link light (2nd interface is not enabled under linux), the DMA behavior will change and I can see the ring buffer get as large as 25MB (up from 512kB!)
Quoted 6 lines
Quoted 2 lines
Only one of my four boxes shows this strange behavior, and only when the second ethernet port is connected to an ethernet switch. Everything
Are you using exactly the same switch, and the same port on the switch, and the same network cable with all of your boxen?
Yeah, I'm using the same switch, cables, and server machines for these tests. I've done quite a lot of fiddling with servers, cables, and switches looking for some differential behavior, but the only thing I find is that with second ethernet port unconnected, my ring buffer will consistently only fill to 512kB, but with the second ethernet port connect to get a link light (disabled under linux), the ring buffer will get as full as 25MB over time.
jeff