Skip to contentSkip to conversation

pmac_zilog debugging ...

DISCUSSION Kevin Diggs 3 participants 12 messages Nov 7, 2008 – Nov 17, 2008

lore
Share & tools
lore
#0 lore raw

Hi,

If I use a command similar to:

pppd ttyS0 1200 satellites: netmask 255.255.255.0 lock crtscts mru 1064 noauth debug kdebug 7 logfile /tmp/pppd.log local

to connect an 8600 to a laptop via ppp the link will lock up in short order from "payloaded" pings. Any advice on how to figure out where it is locking up? This command works fine to connect two x86 laptops. At 1200 it does take a while for an xterm to show up, though.

kevin

#1 lore raw
Quoted 4 lines · Kevin Diggs

to connect an 8600 to a laptop via ppp the link will lock up in short order from "payloaded" pings. Any advice on how to figure out where it is locking up? This command works fine to connect two x86 laptops. At 1200 it does take a while for an xterm to show up, though.

Flow control might be busted. You may need to revert the polarity of the CTS line sampling. Let me know what you find out.

Cheers,
Ben.

#2 lore raw
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 4 lines · Kevin Diggs

to connect an 8600 to a laptop via ppp the link will lock up in short order from "payloaded" pings. Any advice on how to figure out where it is locking up? This command works fine to connect two x86 laptops. At 1200 it does take a while for an xterm to show up, though.

Quoted 6 lines

Flow control might be busted. You may need to revert the polarity of the CTS line sampling. Let me know what you find out.

Cheers,
Ben.

I put the YDL 2.1 disk with it's 2.4.31 kernel back in the beast. With the macserial driver ppp works flawlessly. I can connect to my ISP using one of those Global Village modems. I can also torture a Nec VersaM100 laptop at 115200.

Using 2.6.27 and pmac_zilog I can't even run:

xdpyinfo -display pm8600bs:0

after telnetting into afore mentioned abused laptop.

There are some debug macros that are in pmac_zilog. I have also added a few new ones in an attempt to see where it is locking up. The following lines are the result of the DEBUG_HARD macro (interrupt debug):

pmac_zilog 0.00013000:ch-b: irq, r3: 2
pmac_zilog 0.00013000:ch-b: irq done.
pmac_zilog 0.00013000:ch-b: irq, r3: 4
pmac_zilog 0.00013000:ch-b: pmz_receive_chars() -> 1 (loops 1) pmac_zilog 0.00013000:ch-b: irq done.
pmac_zilog 0.00013000:ch-b: irq, r3: 0
pmac_zilog 0.00013000:ch-b: irq done.
pmac_zilog 0.00013000:ch-b: irq, r3: 2
pmac_zilog 0.00013000:ch-b: irq done.

I think r3 is the interrupt status register. Note the '0' for the next to last interrupt. I think this is a pattern: a 0 (???), a 2 (tx), then locked.

Here are some metrics.

2.4.31 macserial @ 115,200:

pppstats: 47,389,911 bytes in? 46,540 packets; 2,925,937 bytes out?

45,831 rxdma interrupts
658,825 SCC interrupts

2.6.27 pmac_zilog @ 4800:

pppstats: 10,653 bytes in? 145 packets; 13,089 bytes out? 195 packets /proc/tty/driver/ttyPZ: tx: 6,227; rx: 5,986 (!= pppstats???)

12,206 PowerMac Zilog interrupts

Interrupt load is higher without the DMA support.

Is it possible that this hardware was not meant to be used without the DMA (i.e. it does not work quite right?)?

kevin

#3 lore raw
Quoted 6 lines · Kevin Diggs

12,206 PowerMac Zilog interrupts

Interrupt load is higher without the DMA support.

Is it possible that this hardware was not meant to be used without the DMA (i.e. it does not work quite right?)?

Well, the HW Rx buffer is only 3 bytes so if you have high interrupt latencies you are more likely to loose data...

Now, as I said, have you looked at flow control ? It's a likely cause of problems and it's possible that pmac_zilog doesn't do it the way macserial did...

Regarding DMA, it's possible to implement, though there were interesting issues with the way it was done in macserial, it should be done differently in pmac_zilog.

I think the only approach that really works properly (though it's fugly) is what Apple does in OSX I think, which is to have a DMA descriptor per input byte (no need for a huge DMA buffer anyway).

Cheers,
Ben.

#4 lore raw
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 2 lines · Kevin Diggs

12,206 PowerMac Zilog interrupts

Quoted 9 lines
Quoted 4 lines

Interrupt load is higher without the DMA support.

Is it possible that this hardware was not meant to be used without the DMA (i.e. it does not work quite right?)?

Well, the HW Rx buffer is only 3 bytes so if you have high interrupt latencies you are more likely to loose data...

These are not real 8530s any more, right? How certain are we of this? Is it possible that there is a larger buffer when used with the DMA capability ... somehow?

Quoted 4 lines

Now, as I said, have you looked at flow control ? It's a likely cause of problems and it's possible that pmac_zilog doesn't do it the way macserial did...

I tried to put some debug statements where the flow lines are managed. I could have goofed it up. They never produce any output. The latest attempt used nortscts which should have disabled flow control. That coupled with the fact that a 250 MHz 750GX is talking to a 486dx4 at 1200 - 9600 baud I would have thought would reduce the chance the PowerMac would fall behind?

Quoted 8 lines

Regarding DMA, it's possible to implement, though there were interesting issues with the way it was done in macserial, it should be done differently in pmac_zilog.

I think the only approach that really works properly (though it's fugly) is what Apple does in OSX I think, which is to have a DMA descriptor per input byte (no need for a huge DMA buffer anyway).

You would need to explain to me the advantage of doing DMA in this case???

Quoted 4 lines

Cheers,
Ben.

#5 lore raw
Quoted 12 lines · Kevin Diggs
Quoted 11 lines · Benjamin Herrenschmidt
Quoted 6 lines · Kevin Diggs

12,206 PowerMac Zilog interrupts

Interrupt load is higher without the DMA support.

Is it possible that this hardware was not meant to be used without the DMA (i.e. it does not work quite right?)?

Quoted 6 lines
Quoted 3 lines

Well, the HW Rx buffer is only 3 bytes so if you have high interrupt latencies you are more likely to loose data...

These are not real 8530s any more, right? How certain are we of this? Is it possible that there is a larger buffer when used with the DMA capability ... somehow?

Well, the main thing is that when using DMA, it doesn't need to wait for the kernel to come fetch the bytes, and thus the only latency that matters if the DMA list is appropriately provisioned is the bus latency, which is much less likely to be an issue even with a small buffer.

It's definitely not a "basic" 8530, it's an ESCC but I don't think the base rx buffer in polled mode is any bigger (I may be wrong).

Quoted 6 lines

I tried to put some debug statements where the flow lines are managed. I could have goofed it up. They never produce any output. The latest attempt used nortscts which should have disabled flow control. That coupled with the fact that a 250 MHz 750GX is talking to a 486dx4 at 1200 - 9600 baud I would have thought would reduce the chance the PowerMac would fall behind?

Have you disabled flow control both with the old macserial -and- pmac_zilog and still experiencing the same problems ? (ie one works and the other one doesn't ?)

Quoted 1 line

You would need to explain to me the advantage of doing DMA in this case???

Well, if I setup for example 128 DMA descriptors for 1 byte each, then the chip will be able to DMA up to 128 bytes without CPU intervention, thus is a -lot- less likely to overflow it's fifo. It's essentially a way to have the DMA engine operate as an external FIFO. As the CPU fetches the bytes, it can recycle the descriptors at the end of the list effectively acting as some kind of ring buffer.

We can more easily do DMA for Tx but while this can improve performances and lower interrupt usage, it is not a correctness issue in the sense that Tx isn't -losing- data today, it's Rx that is a potential problem.

Cheers,
Ben.

#6 lore raw
Quoted 25 lines · Benjamin Herrenschmidt
Quoted 14 lines · Kevin Diggs
Quoted 12 lines · Benjamin Herrenschmidt
Quoted 6 lines · Kevin Diggs

12,206 PowerMac Zilog interrupts

Interrupt load is higher without the DMA support.

Is it possible that this hardware was not meant to be used without the DMA (i.e. it does not work quite right?)?

Quoted 7 lines
Quoted 3 lines

Well, the HW Rx buffer is only 3 bytes so if you have high interrupt latencies you are more likely to loose data...

These are not real 8530s any more, right? How certain are we of this? Is it possible that there is a larger buffer when used with the DMA capability ... somehow?

Quoted 5 lines

Well, the main thing is that when using DMA, it doesn't need to wait for the kernel to come fetch the bytes, and thus the only latency that matters if the DMA list is appropriately provisioned is the bus latency, which is much less likely to be an issue even with a small buffer.

"... if the DMA list is appropriately provisioned ..." ???

Quoted 3 lines

It's definitely not a "basic" 8530, it's an ESCC but I don't think the base rx buffer in polled mode is any bigger (I may be wrong).

Any idea how we might find out?

Quoted 9 lines
Quoted 7 lines

I tried to put some debug statements where the flow lines are managed. I could have goofed it up. They never produce any output. The latest attempt used nortscts which should have disabled flow control. That coupled with the fact that a 250 MHz 750GX is talking to a 486dx4 at 1200 - 9600 baud I would have thought would reduce the chance the PowerMac would fall behind?

Quoted 4 lines

Have you disabled flow control both with the old macserial -and- pmac_zilog and still experiencing the same problems ? (ie one works and the other one doesn't ?)

I reinstalled the disk with 2.4.31 and reran the tests. With "nortscts" added to the pppd options macserial still works fine. Even at 115,200 it seemed fine (I expected to see some type of packet errors via pppstats or ifconfig). I did not, however, verify that pppd actually does something with this option.

I can't get pmac_zilog to work even at 1200 baud. That is pretty slow.

Quoted 11 lines
Quoted 2 lines

You would need to explain to me the advantage of doing DMA in this case???

Well, if I setup for example 128 DMA descriptors for 1 byte each, then the chip will be able to DMA up to 128 bytes without CPU intervention, thus is a -lot- less likely to overflow it's fifo. It's essentially a way to have the DMA engine operate as an external FIFO. As the CPU fetches the bytes, it can recycle the descriptors at the end of the list effectively acting as some kind of ring buffer.

Quoted 4 lines

We can more easily do DMA for Tx but while this can improve performances and lower interrupt usage, it is not a correctness issue in the sense that Tx isn't -losing- data today, it's Rx that is a potential problem.

Ah! I think I get it. I was thinking that cpu intervention would be required after each byte. But the descriptors are more like linked commands for the DMA hardware. So, I'm on board with this approach. Since I don't really know what I am doing, how do you recommend I proceed?

Would it be correct to say that DMA would also free the cpu from doing io accesses which are MUCH slower than normal memory acdcesses?

Quoted 5 lines

Cheers,
Ben.

#7 lore raw
Quoted 7 lines
Quoted 6 lines

Well, the main thing is that when using DMA, it doesn't need to wait for the kernel to come fetch the bytes, and thus the only latency that matters if the DMA list is appropriately provisioned is the bus latency, which is much less likely to be an issue even with a small buffer.

"... if the DMA list is appropriately provisioned ..." ???

Well, if you have some descriptors to available buffers :-) If you let that run out too, then the chip will miss

Quoted 4 lines
Quoted 3 lines

It's definitely not a "basic" 8530, it's an ESCC but I don't think the base rx buffer in polled mode is any bigger (I may be wrong).

Any idea how we might find out?

Nope. Trial and error ? :-)

Quoted 6 lines

I reinstalled the disk with 2.4.31 and reran the tests. With "nortscts" added to the pppd options macserial still works fine. Even at 115,200 it seemed fine (I expected to see some type of packet errors via pppstats or ifconfig). I did not, however, verify that pppd actually does something with this option.

I can't get pmac_zilog to work even at 1200 baud. That is pretty slow.

That's definitely strange. I would expect the kernel to be able to get interrupts fast enough to service a 1200 bauds serial port. Maybe there's something else wrong, or an other driver causing undue interrupt latencies....

Out of curiosity, check that IDE properly unmasks interrupts (hdparm -u1 /dev/hda).

Quoted 3 lines

Ah! I think I get it. I was thinking that cpu intervention would be required after each byte. But the descriptors are more like linked commands for the DMA hardware.

Yes.

Quoted 2 lines

So, I'm on board with this approach. Since I don't really know what I am
doing, how do you recommend I proceed?

Google for a document called MacTech.pdf which contains various documentations for bits of the ancestor of the IO chip in your machine, along with a description of the DBDMA engine :-) Something else you can do is to look at how it's properly used by other drivers such as bmac and look at some of the darwin source code for reference on how the HW works.

Quoted 2 lines

Would it be correct to say that DMA would also free the cpu from doing io accesses which are MUCH slower than normal memory acdcesses?

To a certain extent yes.

Cheers,
Ben.

#8 lore raw
Quoted 7 lines · Benjamin Herrenschmidt

That's definitely strange. I would expect the kernel to be able to get interrupts fast enough to service a 1200 bauds serial port. Maybe there's something else wrong, or an other driver causing undue interrupt latencies....

As far as I can see the system is NOT busy. I see no evidence of excessive interrupt loading. It does have an Adaptec 2940 u2w SCSI card, an ATI video card, and a USB/firewire card. The SCSI card has some disks on it. The other two cards are unused. I guess, in theory, something in my 2.6.27 kernel could be causing one of the two unused cards to throw spurious interrupts?

I still think the hardware is mis-behaving.

Quoted 3 lines

Out of curiosity, check that IDE properly unmasks interrupts (hdparm -u1 /dev/hda).

This is an 8600. It is SCSI only (the onboard controller is the MESH).

Quoted 5 lines
Quoted 3 lines

So, I'm on board with this approach. Since I don't really know what I am doing, how do you recommend I proceed?

Quoted 7 lines

Google for a document called MacTech.pdf which contains various documentations for bits of the ancestor of the IO chip in your machine, along with a description of the DBDMA engine :-) Something else you can do is to look at how it's properly used by other drivers such as bmac and look at some of the darwin source code for reference on how the HW works.

where might one find older Darwin source?

Quoted 5 lines

Cheers,
Ben.

_______________________________________________

#9 lore raw
Quoted 8 lines · Kevin Diggs
Quoted 7 lines · Benjamin Herrenschmidt

That's definitely strange. I would expect the kernel to be able to get interrupts fast enough to service a 1200 bauds serial port. Maybe there's something else wrong, or an other driver causing undue interrupt latencies....

Quoted 7 lines

As far as I can see the system is NOT busy. I see no evidence of excessive interrupt loading. It does have an Adaptec 2940 u2w SCSI card, an ATI video card, and a USB/firewire card. The SCSI card has some disks on it. The other two cards are unused. I guess, in theory, something in my 2.6.27 kernel could be causing one of the two unused cards to throw spurious interrupts?

I still think the hardware is mis-behaving.

That's strange. Maybe one of the drivers is occasionally hogging interrutps. Well, there may also be a bug in the code :-)

One thing you can try is to disable DMA in macserial (shouldn't be hard to hack) and see if it degrades the same way.

Quoted 4 lines
Quoted 3 lines

Out of curiosity, check that IDE properly unmasks interrupts (hdparm -u1 /dev/hda).

This is an 8600. It is SCSI only (the onboard controller is the MESH).

Ah yes.

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

So, I'm on board with this approach. Since I don't really know what I am doing, how do you recommend I proceed?

Quoted 8 lines
Quoted 7 lines

Google for a document called MacTech.pdf which contains various documentations for bits of the ancestor of the IO chip in your machine, along with a description of the DBDMA engine :-) Something else you can do is to look at how it's properly used by other drivers such as bmac and look at some of the darwin source code for reference on how the HW works.

where might one find older Darwin source?

Apple still has most of them back to 10.0 and even the recent ones still have an SCC serial driver afaik.

Ben.

Quoted 10 lines
Quoted 5 lines

Cheers,
Ben.

_______________________________________________

_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc-dev

#10 lore raw
In reply to Kevin Diggs
Quoted 7 lines · Kevin Diggs

pppd ttyS0 1200 satellites: netmask 255.255.255.0 lock crtscts mru 1064 noauth debug kdebug 7 logfile /tmp/pppd.log local

to connect an 8600 to a laptop via ppp the link will lock up in short order from "payloaded" pings. Any advice on how to figure out where it is locking up? This command works fine to connect two x86 laptops. At 1200 it does take a while for an xterm to show up, though.

Try it without the crtscts (actually, use nocrtscts instead of crtscts). That will tell us whether it is hardware flow control causing problems. IIRC, those mac serial ports didn't have all of RTS, CTS, CD and DTR, but I don't recall which one(s) were missing.

Also make sure you don't have the xonxoff option.

Paul.

#11 lore raw
Quoted 10 lines · Paul Mackerras
Quoted 7 lines · Kevin Diggs

pppd ttyS0 1200 satellites: netmask 255.255.255.0 lock crtscts mru 1064 noauth debug kdebug 7 logfile /tmp/pppd.log local

to connect an 8600 to a laptop via ppp the link will lock up in short order from "payloaded" pings. Any advice on how to figure out where it is locking up? This command works fine to connect two x86 laptops. At 1200 it does take a while for an xterm to show up, though.

Quoted 6 lines

Try it without the crtscts (actually, use nocrtscts instead of crtscts). That will tell us whether it is hardware flow control causing problems. IIRC, those mac serial ports didn't have all of RTS, CTS, CD and DTR, but I don't recall which one(s) were missing.

Also make sure you don't have the xonxoff option.

iirc, RTS is missing and on MacOS we typically have an option to use DTR instead. There's also some issues with CTS polarity which is different between the actual serial ports, vs. some internal modems or something around those lines.

Ben