RFC: I2C bus fault recovery and I2C reset
Hi all,
I was debugging an I2C bus connected to a i2c-imx peripheral as master, with several slaves connected to it, when I realized that this driver (and many (all?) others) cannot recover from a bus fault in a graceful manner. If, for instance, one slave device misses one (or more) clock pulses for whatever reason during a slave->master transmission (read), during a 0-data bit, this slave may eventually keep the SDA line active in low-state. Most I2C master peripherals, and particularly i2c-imx will not be able to continue operating. Any operation will just timeout with a "busy bus" error. The simplest and most often used way of recovering from such a situation is "resetting" the I2C bus, by toggling SCL a few times (maximum 9) until SDA is released again. After that a START sequence can successfully reset the state of any slave device.
One can argue whether it may or may not be accepted that this happens under normal circumstances, but it definitely can happen at any moment (heavy EMC interference, bad bus design, long bus, misbehaving slave... you name it), and IMHO a linux-driver should always have the ability to try to recover gracefully from such an event. Whether the system this bus takes part of can tolerate such a situation or not is not up to the driver to decide either... it should just try to recover.
This issue seems to have been discussed before in this thread:
http://article.gmane.org/gmane.linux.drivers.i2c/3010
The proposed solution back then was to issue a reset sequence "by hand" via a sysfs interface. This may be useful for debugging, but IMHO an I2C driver needs to do this automatically.
For many peripherals in order to support this, a special function would be needed, that reconfigures the SDA/SCL pins as GPIO and manually toggles SCL a few times. This would probably need to be implemented in board-support-/platform code...?
In my specific situation, there was no way of recovering other than power-cycling the device, which is completely unacceptable, specially for an industrial control system. A temporary bus-lockup with automatic recovery via a proper I2C bus reset OTOH, wouldn't have any significant impact even if occurring sporadically.
Individually resetting I2C slaves is also not a real solution because it may not be possible to determine which is the I2C slave that misbehaved.
Any idea on how to solve this problem?
Should each driver implement support for it and implement optional callback functions in platform-data?
Best regards,
Signature · 2 lines
David Jander Protonic Holland.
Am 24.11.2011 12:02, schrieb David Jander:
Quoted 24 lines
Hi all,
I was debugging an I2C bus connected to a i2c-imx peripheral as master, with several slaves connected to it, when I realized that this driver (and many (all?) others) cannot recover from a bus fault in a graceful manner. If, for instance, one slave device misses one (or more) clock pulses for whatever reason during a
slave->mastertransmission (read), during a 0-data bit, this slave may eventually keep the SDA line active in low-state. Most I2C master peripherals, and particularly i2c-imx will not be able to continue operating. Any operation will just timeout with a "busy bus" error. The simplest and most often used way of recovering from such a situation is "resetting" the I2C bus, by toggling SCL a few times (maximum 9) until SDA is released again. After that a START sequence can successfully reset the state of any slave device.One can argue whether it may or may not be accepted that this happens under normal circumstances, but it definitely can happen at any moment (heavy EMC interference, bad bus design, long bus, misbehaving slave... you name it), and IMHO a linux-driver should always have the ability to try to recover gracefully from such an event. Whether the system this bus takes part of can tolerate such a situation or not is not up to the driver to decide either... it should just try to recover.
Quoted 7 lines
This issue seems to have been discussed before in this thread:
http://article.gmane.org/gmane.linux.drivers.i2c/3010
The proposed solution back then was to issue a reset sequence "by hand" via a sysfs interface. This may be useful for debugging, but IMHO an I2C driver needs to do this automatically.
ACK
Quoted 4 lines
For many peripherals in order to support this, a special function would be needed, that reconfigures the SDA/SCL pins as GPIO and manually toggles SCL a few times. This would probably need to be implemented in board-support-/platform code...?
Needs to be part of recover function which in turn is part of driver code.
Quoted 7 lines
In my specific situation, there was no way of recovering other than power-cycling the device, which is completely unacceptable, specially for an industrial control system. A temporary bus-lockup with automatic recovery via a proper I2C bus reset OTOH, wouldn't have any significant impact even if occurring sporadically.
Individually resetting I2C slaves is also not a real solution because it may not be possible to determine which is the I2C slave that misbehaved.
Most I2C slaves haven't got any reset line.
Quoted 3 lines
Any idea on how to solve this problem?
Should each driver implement support for it and implement optional callback functions in platform-data?
IMHO this typically is adapter driver's job. It strongly depends on particular H/W whether controller can return information on busy/blocked bus and whether it is able to manually toggle the clock line. On single master systems, the driver code should automatically try to recover when not being able to send start flag. On multi master systems the situation is more complex.
JM2C
Signature · 2 lines
KR Michael
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Micheal,
Quoted 30 lines · On Fri, 25 Nov 2011 11:27:44 +0100 Michael Lawnick…
Am 24.11.2011 12:02, schrieb David Jander:
Quoted 21 lines
I was debugging an I2C bus connected to a i2c-imx peripheral as master, with several slaves connected to it, when I realized that this driver (and many (all?) others) cannot recover from a bus fault in a graceful manner. If, for instance, one slave device misses one (or more) clock pulses for whatever reason during a
slave->mastertransmission (read), during a 0-data bit, this slave may eventually keep the SDA line active in low-state. Most I2C master peripherals, and particularly i2c-imx will not be able to continue operating. Any operation will just timeout with a "busy bus" error. The simplest and most often used way of recovering from such a situation is "resetting" the I2C bus, by toggling SCL a few times (maximum 9) until SDA is released again. After that a START sequence can successfully reset the state of any slave device.One can argue whether it may or may not be accepted that this happens under normal circumstances, but it definitely can happen at any moment (heavy EMC interference, bad bus design, long bus, misbehaving slave... you name it), and IMHO a linux-driver should always have the ability to try to recover gracefully from such an event. Whether the system this bus takes part of can tolerate such a situation or not is not up to the driver to decide either... it should just try to recover.
Quoted 7 lines
This issue seems to have been discussed before in this thread:
http://article.gmane.org/gmane.linux.drivers.i2c/3010
The proposed solution back then was to issue a reset sequence "by hand" via a sysfs interface. This may be useful for debugging, but IMHO an I2C driver needs to do this automatically.
Quoted 8 lines
ACK
Quoted 4 lines
For many peripherals in order to support this, a special function would be needed, that reconfigures the SDA/SCL pins as GPIO and manually toggles SCL a few times. This would probably need to be implemented in board-support-/platform code...?
Needs to be part of recover function which in turn is part of driver code.
In the case of the i.MX I2C peripheral, and probably in the case of a few others, there is no way of doing this, except for switching I2C i/o pins to GPIO via the iomux and toggling the GPIO pin that corresponds to SCL "by hand", while watching the GPIO pin that corresponds to SDA.
I know of no standard kind of IOMUX framework in the kernel that could help doing this in a generic way.... Grant?
Due to this, it can become fairly complicated if one wants to do this entirely in the driver. IMHO, probably the easiest way of implementing this would be via platform/board specific functions that are called via optional function-pointers in the platform-data. I don't really like that solution, so I hope someone can come up with a better one....
Quoted 9 lines
Quoted 7 lines
In my specific situation, there was no way of recovering other than power-cycling the device, which is completely unacceptable, specially for an industrial control system. A temporary bus-lockup with automatic recovery via a proper I2C bus reset OTOH, wouldn't have any significant impact even if occurring sporadically.
Individually resetting I2C slaves is also not a real solution because it may not be possible to determine which is the I2C slave that misbehaved.Most I2C slaves haven't got any reset line.
Even worse.... that means the bus will never come back, even if you reset the machine!!! Only a power-cycle would save you.
Quoted 10 lines
Quoted 3 lines
Any idea on how to solve this problem?
Should each driver implement support for it and implement optional callback functions in platform-data?IMHO this typically is adapter driver's job. It strongly depends on particular H/W whether controller can return information on busy/blocked bus and whether it is able to manually toggle the clock line. On single master systems, the driver code should automatically try to recover when not being able to send start flag. On multi master systems the situation is more complex.
I agree. There might be a few platforms where there is no solution to this, other than hardwiring a separate GPIO line to SCL...
Best regards,
Signature · 2 lines
David Jander Protonic Holland.
Am 28.11.2011 08:48, schrieb David Jander:
Quoted 11 lines
Hi Micheal,
Quoted 6 lines · On Fri, 25 Nov 2011 11:27:44 +0100 Michael Lawnick…
Am 24.11.2011 12:02, schrieb David Jander:
Quoted 4 lines
For many peripherals in order to support this, a special function would be needed, that reconfigures the SDA/SCL pins as GPIO and manually toggles SCL a few times. This would probably need to be implemented in board-support-/platform code...?
Quoted 6 lines
Quoted 1 line
Needs to be part of recover function which in turn is part of driver code.
In the case of the i.MX I2C peripheral, and probably in the case of a few others, there is no way of doing this, except for switching I2C i/o pins to GPIO via the iomux and toggling the GPIO pin that corresponds to SCL "by hand", while watching the GPIO pin that corresponds to SDA.
So only one problem up to here: may the i2c adapter code have reserved access to iomux? If its the only user -> move control into adpater driver, reserve the H/W-access and you are done. If not, then you have a shared device -> make a driver for iomux registers that serializes access, possibly with reservation functions, export them and reference from adapter code.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
In my specific situation, there was no way of recovering other than power-cycling the device, which is completely unacceptable, specially for an industrial control system. A temporary bus-lockup with automatic recovery via a proper I2C bus reset OTOH, wouldn't have any significant impact even if occurring sporadically.
Individually resetting I2C slaves is also not a real solution because it may not be possible to determine which is the I2C slave that misbehaved.
Quoted 5 lines
Quoted 1 line
Most I2C slaves haven't got any reset line.
Even worse.... that means the bus will never come back, even if you reset the machine!!! Only a power-cycle would save you.
Correct.
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
Any idea on how to solve this problem?
Should each driver implement support for it and implement optional callback functions in platform-data?IMHO this typically is adapter driver's job. It strongly depends on particular H/W whether controller can return information on busy/blocked bus and whether it is able to manually toggle the clock line. On single master systems, the driver code should automatically try to recover when not being able to send start flag. On multi master systems the situation is more complex.
Quoted 2 lines
I agree. There might be a few platforms where there is no solution to this, other than hardwiring a separate GPIO line to SCL...
or by wiring Vcc of unresetable I2C devices to a controllable on-board power supply/relays.
Signature · 2 lines
KR Michael
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 19 lines · On Mon, 28 Nov 2011 10:51:59 +0100 Michael Lawnick…
Am 28.11.2011 08:48, schrieb David Jander:
Quoted 11 lines
Hi Micheal,
Quoted 6 lines · On Fri, 25 Nov 2011 11:27:44 +0100 Michael Lawnick…
Am 24.11.2011 12:02, schrieb David Jander:
Quoted 4 lines
For many peripherals in order to support this, a special function would be needed, that reconfigures the SDA/SCL pins as GPIO and manually toggles SCL a few times. This would probably need to be implemented in board-support-/platform code...?
Quoted 6 lines
Quoted 1 line
Needs to be part of recover function which in turn is part of driver code.
In the case of the i.MX I2C peripheral, and probably in the case of a few others, there is no way of doing this, except for switching I2C i/o pins to GPIO via the iomux and toggling the GPIO pin that corresponds to SCL "by hand", while watching the GPIO pin that corresponds to SDA.
Quoted 6 lines
So only one problem up to here: may the i2c adapter code have reserved access to iomux? If its the only user -> move control into adpater driver, reserve the H/W-access and you are done. If not, then you have a shared device -> make a driver for iomux registers that serializes access, possibly with reservation functions, export them and reference from adapter code.
I don't think IOMUX should ever be accessed directly within a driver. Besides, the imx-i2c.c peripheral is found in many different chips that have the same I2C controller, but different IOMUX and GPIO peripherals.
I think, what we are missing is probably a generic IOMUX framework for linux, that can deal with changing functions of I/O pins.
Until we have such a framework, we probably must do with platform-data function-pointers.... :-(
I would like to know if anyone disagrees with the fact that I2C bus fault recovery and reset should be done by the driver. If no one disagrees, I will try to add support for this to the imx-i2c.c driver.
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
In my specific situation, there was no way of recovering other than power-cycling the device, which is completely unacceptable, specially for an industrial control system. A temporary bus-lockup with automatic recovery via a proper I2C bus reset OTOH, wouldn't have any significant impact even if occurring sporadically.
Individually resetting I2C slaves is also not a real solution because it may not be possible to determine which is the I2C slave that misbehaved.Quoted 5 lines
Quoted 1 line
Most I2C slaves haven't got any reset line.
Even worse.... that means the bus will never come back, even if you reset the machine!!! Only a power-cycle would save you.
Correct.
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
Any idea on how to solve this problem?
Should each driver implement support for it and implement optional callback functions in platform-data?IMHO this typically is adapter driver's job. It strongly depends on particular H/W whether controller can return information on busy/blocked bus and whether it is able to manually toggle the clock line. On single master systems, the driver code should automatically try to recover when not being able to send start flag. On multi master systems the situation is more complex.
Quoted 5 lines
Quoted 2 lines
I agree. There might be a few platforms where there is no solution to this, other than hardwiring a separate GPIO line to SCL...
or by wiring Vcc of unresetable I2C devices to a controllable on-board power supply/relays.
Yes, but that would be more or less like having a reset-pin on the device... :-)
Best regards,
Signature · 2 lines
David Jander Protonic Holland.