tricky challenge for getting round level-driven interrupt problem: help!
hi, please kindly respond cc to me because i am subscribed to the lists for post-only-and-view-archives-on-demand-to-minimise-overload purposes.
i have a particularly acute and knotty computing problem involving a stupid hardware design fault in a cirrus logic "maverick" EDB 7134 ARM processor (max 90Mhz).
it only does level-based interrupts and i need to create a driver that does two-way 8-bit data communication.
i would genuinely appreciate some advice from people with more experience than i on how to go about getting round this stupid hardware design - in order to make a RELIABLE, non-race-conditioned kernel driver.
hardware
--------
connected to the ARM's 8-bit port is a 6Mhz p16f877a PIC processor.
connected to the serial and other ports of the PIC processor are various peripherals, including an LCD, a GPS satellite receiver, an accelerometer and the battery level / charger detector.
on the PIC is some assembler code that:
--------------------------------------
* merges GPS, Accelerometer, battery and charger information into a data stream that goes out the 8-bit port of the PIC and in on the ARM's 8-bit port.
* receives instructions from the ARM down the same 8-bit port, in a custom-designed length-encoded data stream that tells the PIC what to put on the LCD, and where.
interrupts are generated as follows:
------------------------------------
* on the PIC, a "read" interrupt can be asserted to the ARM. it's xxxxing level-based.
* on the PIC, a "write" interrupt can be asserted to the ARM. it's xxxxing level-based.
* on the ARM, a SINGLE interrupt can be generated to the PIC by EITHER reading OR writing to the PIC's 8-bit port.
unlike the stupid xxxxing ARM, it's edge triggered (thank god).
interrupts are cleared as follows:
---------------------------------
* on the PIC, in the [single!] ISR routine, if the PIC knows that it was doing a read, it resets the "read" interrupt flag to the ARM.
[the problem is obvious: a level-based interrupt could fail
to be acknowledged, could be masked out and accidentally regenerated, and we are into "nightmare" scenario time]
* on the PIC, in the [single!] ISR routine, if the PIC knows that it was doing a read, it resets the "read" interrupt flag to the ARM.
[the problem is obvious: a level-based interrupt could fail
to be acknowledged, could be masked out and accidentally regenerated, and we are into "nightmare" scenario time]
* on the ARM, in the READ isr routine, the PIC "reads" the byte, THIS ACT generates an interrupt to the PIC, which the ARM then acknowledges by waiting - in a tight loop - for the PIC to clear the "read" interrupt flag.
[this is the old code btw, not the new code: for brevity
i have not described exactly how bad the code is]
* on the ARM, in the WRITE isr routine, similar situation as for "read".
the protocol that i designed to overcome the race condition nightmare is as follows:
* ARM and PIC ****MUST**** exchange read and write bytes, interleaved.
* if the ARM does not have anything to write [to the LCD] at the time that a read is to be carried out, it sends a "dummy" or "null" encoded data stream indicating to the PIC that it is receiving data of zero length.
the sole purpose of initiating this "i-am-sending-you-zero-bytes" dummy stream of bytes is to keep the read-write-read-write.... cycle going.
* if the PIC does not have anything to be read [from the GPS and other peripherals] then it sends "0xff" instead. the ARM receives this non-ascii byte and knows that it must throw it away.
the sole purpose of sending this "non-ascii" byte is to keep the read-write-read-write... cycle going.
* if there is both read and write data to be exchanged, everything is hunky-dory.
* if there is no data to be exchanged, everything is hunky-dory.
here's where i have got to, and where i am stuck:
----------------------------------------------------
on the ARM, i have cut/paste the code from sonypi.c to create a poll_wait "read" driver. it is pretty much exactly the same structure as sonypi.c, as far as "read" is concerned.
so, when reading, I/O is in non-blocking mode, the queue is empty, a WAITQUEUE is woken up, present kernel process is made "TASK_INTERRUPTIBLE", schedule()d, a read interrupt is [eventually] generated and read data added to queue, then pic_misc_poll() detects data now present, and "WAKES UP" the waiting blocking process.
everything hunky-dory.
here's where i have got to, and where i am stuck:
----------------------------------------------------
i don't know how to "wake up" an equivalent write process.
because there isn't one.
the situation where there is simultaneous blocking-read and blocking-write _could_ be covered by duplicating the sonypi.c code _again_ for writing.
that leaves the situations where there is ONLY read occurring.
bearing in mind that reads must be interleaved with writes, and those reads must be intitiated from inside the "read" interrupt service routine.... what the hell do i do?
i think i have a clue in that the "old" pic_write() code would unmask the Write Interrupt, and a "write" interrupt would immediately occur - then once we'd done, Write Interrupts would be re-masked again.
any advice really appreciated because this is one mean comp-sci classic that really only _actually_ occurs in real life when there's a fuckup in the hardware design and you don't have any choice but to make it work :)
cheers,
l.
Signature · 3 lines
-- <a href="http://lkcl.net">http://lkcl.net</a> --
Quoted 2 lines · Luke Kenneth Casson Leighton
it only does level-based interrupts and i need to create a driver that does two-way 8-bit data communication.
Level triggered is generally considered a feature by people less than 200 years old 8)
Quoted 4 lines
i would genuinely appreciate some advice from people with more experience than i on how to go about getting round this stupid hardware design - in order to make a RELIABLE, non-race-conditioned kernel driver.
Assuming you are using the core Linux IRQ code then you shouldn't have any great problem. The basic rules with level triggered IRQ are that
- You must ack the IRQ on the device to clear it either specifically or as part of some other process.
- The device must raise the IRQ again if the irq condition is present *at* or after the point of the IRQ ack (or in many implementations 'just before' in fact)
the core Linux code deals with masking on the controller to ensure IRQ's dont get called recursively, and ensuring an IRQ gets rechecked/recalled on the unmask/exit path if it is still asserted.
So an implementation is much like a serial port fifo, assert the IRQ until the PC has fetched all the bits. In fact the IRQ line is essentially "NOT(fifo_empty)" and rechecked each time a byte is fetched.
*WAY* simpler than nasty edge triggered stuff 8)
Alan
alan, thanks for responding directly.
the internet is down at the moment (at least, bt's bit is) so your continued direct responses, if you have the time, greatly appreciated.
i can't get http or ssh access to _anywhere_, but bizarrely, smtp is getting through.
[sod bt. royally.]
Quoted 6 lines · Alan Cox
Quoted 2 lines · Luke Kenneth Casson Leighton
it only does level-based interrupts and i need to create a driver that does two-way 8-bit data communication.
Level triggered is generally considered a feature by people less than 200 years old 8)
lots of experience, then, you'd say? my experience with doing this kind of stuff is kinda limited (doesn't stop me trying) so your response is really appreciated.
Quoted 6 lines
Quoted 4 lines
i would genuinely appreciate some advice from people with more experience than i on how to go about getting round this stupid hardware design - in order to make a RELIABLE, non-race-conditioned kernel driver.
Assuming you are using the core Linux IRQ code
yep.
Quoted 9 lines
then you shouldn't have
any great problem. The basic rules with level triggered IRQ are that- You must ack the IRQ on the device to clear it either specifically or as part of some other process.
- The device must raise the IRQ again if the irq condition is present *at* or after the point of the IRQ ack (or in many implementations 'just before' in fact)
Quoted 7 lines
the core Linux code deals with masking on the controller to ensure IRQ's dont get called recursively, and ensuring an IRQ gets rechecked/recalled on the unmask/exit path if it is still asserted.
So an implementation is much like a serial port fifo, assert the IRQ until the PC has fetched all the bits. In fact the IRQ line is essentially "
NOT(fifo_empty)" and rechecked each time a byte is fetched.
okay.
Quoted 1 line
*WAY* simpler than nasty edge triggered stuff 8)
bleh :)
okay.
i believe i get it: you raise a level-triggered interrupt which _stays_ raised until such time as your fifo is empty.
the original protocol that was written [by somebody even less experienced than i] was designed the way it is because the PIC chip that's connected to the ARM only has *one* interrupt.
that interrupt is triggered:
* on a port read (there are four 8-bit ports!)
* on a port write
* by the serial port
* by the timer
[skip this bit if you're busy, it's just background info]
so there's now a _thousand_ lines of hand-written assembly code which took well over a year to write and debug on the PIC, that implements the state-machine to deal with the GPS on the serial port, interleaving the "device status" info into the GPS data stream, getting the GPS data stream over the 8-bit port and receiving LCD commands over the 8-bit port, interpreting them and then feeding the LCD.
okay.
so.
assuming that i can handle the pic having only a single interrupt, then does the process go something like this (for read, e.g.):
1) the ARM unmasks read interrupts when it's ready.
2) when the PIC has data to be read (which is only ever a single byte), it asserts the level-triggered read interrupt to the ARM.
3) the ARM receives the interrupt, and reads the byte. the _act_ of reading the byte causes an edge-triggered interrupt to the PIC.
5) the first thing the PIC does in its [one] ISR is to de-assert the read interrupt. why? because there is only a one byte buffer.
we have an additional step in the chain - step 4 - which is:
4) in the ISR, the ARM goes into a tight loop checking that the PIC has cleared the level-triggered interrupt.
the reason for checking this is so that when we come out of the ISR on the ARM, the PIC is *SOOO* slow - it's only 6mhz after all - that sometimes (frequently, in fact - about 1 in every 50 times) it hasn't got round to clearing the level-driven interrupt by the time we come out of the ARM ISR (!)
the problem we're having is actually worse than that.
sometimes something goes screwy and _despite_ having this loop-check, we get duplicated characters.
sometimes, even worse, we lose sync, and the ARM sits there in the loop...
i was getting _so_ fed up with this that i was thinking that there was a problem with the approach being taken.
hence the redesign to do alternate read-write-read-write, and making reads exclusive of writes, etc.
awful. just awful :)
... so - in your opinion, alan, is the old approach we had actually _on_ the right lines?
also, are you going to ukuug in august, it being _in_ aberystwyth and all :)
l.
Quoted 2 lines · Luke Kenneth Casson Leighton
i believe i get it: you raise a level-triggered interrupt which _stays_ raised until such time as your fifo is empty.
Bingo. It only goes away when the chip really has nothing left to say.
Quoted 3 lines
all - that sometimes (frequently, in fact - about 1 in every 50 times) it hasn't got round to clearing the level-driven interrupt by the time we come out of the ARM ISR (!)
So you'll poll again and find there is no pending work to do.
Quoted 2 lines
hence the redesign to do alternate read-write-read-write, and making reads exclusive of writes, etc.
and maybe even turn the IRQ off and use a timer if its slow and not sensitive to latency.. ?
Quoted 2 lines
... so - in your opinion, alan, is the old approach we had actually _on_ the right lines?
level triggered IRQ does sort of expect the other end responds promptly to be efficient as opposed to merely reliable.
Quoted 2 lines
also, are you going to ukuug in august, it being _in_ aberystwyth and all :)
Its not in Aberystwyth, but I might be. Its in Swansea 8)
Quoted 11 lines · Alan Cox
Quoted 2 lines · Luke Kenneth Casson Leighton
i believe i get it: you raise a level-triggered interrupt which _stays_ raised until such time as your fifo is empty.
Bingo. It only goes away when the chip really has nothing left to say.
Quoted 3 lines
all - that sometimes (frequently, in fact - about 1 in every 50 times) it hasn't got round to clearing the level-driven interrupt by the time we come out of the ARM ISR (!)
So you'll poll again and find there is no pending work to do.
oh.
ah.
wait - this might be the bit i don't get
when you say poll again, do you mean poll again inside the ISR?
so:
* we do the read (which creates the interrupt to the PIC) and then sit there polling the level-driven interrupt status register
* the PIC, having no more bytes to write, clears the ARM level-driven interrupt.
* the ARM detects the change of the level-driven interrupt status register
* the ARM pauses for, oh, i dunno... mmm... 6mhz equals 166ns so we call udelay(10) or something ridiculously long...
* the ARM then checks the level-driven interrupt status register AGAIN, and if it's clear, goes "oh, we ain't gonna get no more data".
something like that? [am trying it now, anyway - on the basis that it can't hurt :)]
i'm not being deliberately thick - honest - i've just never had to do this sort of thing before.
btw alan where would you recommend i read up on this type of thing? [_please_ don't say "on the inside of my skull" i'll be tempted to go get a saw and a spoon i've _so_ got to get this to work...]
Quoted 5 lines
Quoted 2 lines
hence the redesign to do alternate read-write-read-write, and making reads exclusive of writes, etc.
and maybe even turn the IRQ off and use a timer if its slow and not sensitive to latency.. ?
tried that last month [i'm not sending to lkml about this as a first resort - honest!]
the baud rate from the GPS is 4800 baud, so that's 600 chars/sec.
the jiffies timer on the ARM is ... er... 250? per second?
so ... hey, yeh, that would explain why i saw about 1 character in 3 :)
Quoted 5 lines
Quoted 2 lines
... so - in your opinion, alan, is the old approach we had actually _on_ the right lines?
level triggered IRQ does sort of expect the other end responds promptly to be efficient as opposed to merely reliable.
... and a 6mhz processor vs a 90mhz processor...
Quoted 4 lines
Quoted 2 lines
also, are you going to ukuug in august, it being _in_ aberystwyth and all :)
Its not in Aberystwyth, but I might be. Its in Swansea 8)
ahhh :)
glad i checked then, 'cos up until 30 seconds ago i was gonna drive to aber.
*wonders why the hell i've been let loose on this project...*
Quoted 5 lines · Alan Cox
Quoted 2 lines
hence the redesign to do alternate read-write-read-write, and making reads exclusive of writes, etc.
and maybe even turn the IRQ off and use a timer if its slow and not sensitive to latency.. ?
good suggestion...
been there, tried that [i really _am_ sending to lkml as last resort, not first!]
jiffies equals approx 250? per second?
baud rate from PIC is determined by GPS - 4800 baud - approx 600 per second.
so that'd explain why i only got one character every 3.
*cold sweat*.
i could always use the FIQ, which will be running off the back of the Audio DAC/ADC's interrupts, 8khz....
*shudder*...
Quoted 10 lines · Alan Cox
Quoted 2 lines · Luke Kenneth Casson Leighton
i believe i get it: you raise a level-triggered interrupt which _stays_ raised until such time as your fifo is empty.
Bingo. It only goes away when the chip really has nothing left to say.
Quoted 3 lines
all - that sometimes (frequently, in fact - about 1 in every 50 times) it hasn't got round to clearing the level-driven interrupt by the time we come out of the ARM ISR (!)
Quoted 7 lines
So you'll poll again and find there is no pending work to do.
Quoted 2 lines
hence the redesign to do alternate read-write-read-write, and making reads exclusive of writes, etc.
and maybe even turn the IRQ off and use a timer if its slow and not sensitive to latency.. ?
okay.
you will be pleased to know that the switch-off-irqs-in-the-isr and then switch-it-back-on-in-the-FIQ approach works.
*gibber*
basically i am doing this:
* in the ARM interrupt read routine, first thing i do is mask read interrupts.
* i then set up the ARM port for "read", including disabling the keyboard which shares some of the port lines.
* i then read the byte (which causes an edge-triggered interrupt to the PIC, which then sets about clearing the interrupt).
* i then re-enable the keyboard.
* at the end of the routine, i then set a flag.
some time later, within 1/8000th of a second, this flag is checked. if it's set, read interrupts are unmasked.
... btw when you say "using the linux interrupt management" routines, what exactly do you mean?
at the moment (because that's what was done when i stepped in to sort out this code) i am directly accessing the INTMR1 and INTSR1 registers - _not_ calling enable_irq() or disable_irq() or other)
(and in the FIQ handler of course i _can't_ call those functions).
is that a problem?
l.
hi: just a fyi.
i do not know, cannot tell, and do not care [ItNowWorks(tm)] why what i have done manages to GetItToWork(tm) - i am merely providing you with information such that _other people_ may dissect this issue, may look at this in six months/years and go "oh yeah, maybe i should try that too".
in the write communications, i have REMOVED the use of interrupts and am PURELY using the INTMR1 bit for polling purposes (!!!)
the code goes something like this:
int pic_misc_write( ..... char *data, int len)
{
for (i = 0; i < data; i++)
{
pic_queue_byte(data[i]);
pic_writable_int(0, 0, NULL);
}
}ha ha, i hear you say, very funny.
well, in pic_writable_int - which USED to be called as an interrupt handler, i have this:
irq_return_t pic_writable_int(int irq, ....)
{
/* check that the PIC is ready for us to proceed */
while (!(clps_readl(INTMR) & 0x40)) {
}
...
...
/* place the byte on the
clps_writeb(pic_get_queued_byte(), PORT_A_DATA_REGISTER);
/* this generates an interrupt to the PIC */
clps_writeb(0x...., PORT_A_SOMETHING_REGISTER);
/* ... which we sit there twiddling our thumbs waiting for
it to be cleared
*/
while ((clps_readl(INTMR) & 0x40)) {
}
}ironically, it's _still_ not 100% reliable: about 1 in every 400 bytes gets corrupted.
but that's probably due to bugs in the PIC assembly code, which i can live with / get these guys to fix, or it's to do with the setup timing of the port directions - again, i can deal with that.
... the question is - what the _hell_ is going on with the linux 2.6 level-based interrupt handling / my-braindead-way-of-using-them that causes the above approach to be [virtually] 100% reliable as compared to a 95%-reliable cock-up?
l.
Quoted 140 lines · Luke Kenneth Casson Leighton
alan, thanks for responding directly.
the internet is down at the moment (at least, bt's bit is) so your continued direct responses, if you have the time, greatly appreciated.
i can't get http or ssh access to _anywhere_, but bizarrely, smtp is getting through.
[sod bt. royally.]
Quoted 6 lines · Alan Cox
Quoted 2 lines · Luke Kenneth Casson Leighton
it only does level-based interrupts and i need to create a driver that does two-way 8-bit data communication.
Level triggered is generally considered a feature by people less than 200 years old 8)
lots of experience, then, you'd say? my experience with doing this kind of stuff is kinda limited (doesn't stop me trying) so your response is really appreciated.
Quoted 6 lines
Quoted 4 lines
i would genuinely appreciate some advice from people with more experience than i on how to go about getting round this stupid hardware design - in order to make a RELIABLE, non-race-conditioned kernel driver.
Assuming you are using the core Linux IRQ code
yep.
Quoted 9 lines
then you shouldn't have
any great problem. The basic rules with level triggered IRQ are that- You must ack the IRQ on the device to clear it either specifically or as part of some other process.
- The device must raise the IRQ again if the irq condition is present *at* or after the point of the IRQ ack (or in many implementations 'just before' in fact)Quoted 7 lines
the core Linux code deals with masking on the controller to ensure IRQ's dont get called recursively, and ensuring an IRQ gets rechecked/recalled on the unmask/exit path if it is still asserted.
So an implementation is much like a serial port fifo, assert the IRQ until the PC has fetched all the bits. In fact the IRQ line is essentially "
NOT(fifo_empty)" and rechecked each time a byte is fetched.okay.
Quoted 1 line
*WAY* simpler than nasty edge triggered stuff 8)
bleh :)
okay.
i believe i get it: you raise a level-triggered interrupt which _stays_ raised until such time as your fifo is empty.
the original protocol that was written [by somebody even less experienced than i] was designed the way it is because the PIC chip that's connected to the ARM only has *one* interrupt.
that interrupt is triggered:
* on a port read (there are four 8-bit ports!)
* on a port write
* by the serial port
* by the timer[skip this bit if you're busy, it's just background info]
so there's now a _thousand_ lines of hand-written assembly code which took well over a year to write and debug on the PIC, that implements the state-machine to deal with the GPS on the serial port, interleaving the "device status" info into the GPS data stream, getting the GPS data stream over the 8-bit port and receiving LCD commands over the 8-bit port, interpreting them and then feeding the LCD.
okay.
so.
assuming that i can handle the pic having only a single interrupt, then does the process go something like this (for read, e.g.):
1) the ARM unmasks read interrupts when it's ready.
2) when the PIC has data to be read (which is only ever a single byte), it asserts the level-triggered read interrupt to the ARM.
3) the ARM receives the interrupt, and reads the byte. the _act_ of reading the byte causes an edge-triggered interrupt to the PIC.
5) the first thing the PIC does in its [one] ISR is to de-assert the read interrupt. why? because there is only a one byte buffer.
we have an additional step in the chain - step 4 - which is:
4) in the ISR, the ARM goes into a tight loop checking that the PIC has cleared the level-triggered interrupt.
the reason for checking this is so that when we come out of the ISR on the ARM, the PIC is *SOOO* slow - it's only 6mhz after all - that sometimes (frequently, in fact - about 1 in every 50 times) it hasn't got round to clearing the level-driven interrupt by the time we come out of the ARM ISR (!)
the problem we're having is actually worse than that.
sometimes something goes screwy and _despite_ having this loop-check, we get duplicated characters.
sometimes, even worse, we lose sync, and the ARM sits there in the loop...
i was getting _so_ fed up with this that i was thinking that there was a problem with the approach being taken.
hence the redesign to do alternate read-write-read-write, and making reads exclusive of writes, etc.
awful. just awful :)
... so - in your opinion, alan, is the old approach we had actually _on_ the right lines?
also, are you going to ukuug in august, it being _in_ aberystwyth and all :)
l.
Signature · 3 lines
-- <a href="http://lkcl.net">http://lkcl.net</a> --