Skip to contentSkip to conversation

Enable BCM2835 mailbox support

RFC PATCH New Craig McGeachie 3 participants 5 messages Sep 13, 2013 – Oct 2, 2013

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/3 Enable BCM2835 mailbox support

Cherry-picked selection of commits by Lubomir Rintel [1], Suman Anna and Jassi Brar [2] on which to base the implementation.

[1] http://lists.infradead.org/pipermail/linux-rpi-kernel/2013-April/000528.html
[2] http://lists.infradead.org/pipermail/linux-rpi-kernel/2013-May/000546.html

---

Documentation/devicetree/bindings/mailbox/brcm,bcm2835-mbox.txt new file +15 -0
new file mode 100644
index 0000000..75dbb89
@@ -0,0 +1,15 @@
+Broadcom BCM2835 VideoCore mailbox IPC
+
+Required properties:
+
+- compatible : should be "brcm,bcm2835-mbox"
+- reg : Specifies base physical address and size of the registers.
+- interrupts : the interrupt number
+
+Example:
+
+mailbox {
+	compatible = "brcm,bcm2835-mbox";
+	reg = <0x7e00b800 0x400>;
+	interrupts = <0 1>;
+};
arch/arm/boot/dts/bcm2835.dtsi +5 -0
index 1e12aef..4fc8a68 100644
@@ -29,6 +29,12 @@
 			#interrupt-cells = <2>;
 		};
 +		mailbox {
+			compatible = "brcm,bcm2835-mbox";
+			reg = <0x7e00b880 0x40>;
+			interrupts = <0 1>;
+		};
+
 		watchdog {
 			compatible = "brcm,bcm2835-pm-wdt";
 			reg = <0x7e100000 0x28>;
arch/arm/mach-bcm2835/Makefile +1 -1
index 4c3892f..934ab42 100644
@@ -1 +1 @@
-obj-y += bcm2835.o
+obj-y += bcm2835.o mailbox.o
arch/arm/mach-bcm2835/mailbox.c new file +278 -0

278 lines not on this page · raw message

drivers/mailbox/Makefile +4 -0
index e0facb3..2fa343a 100644
@@ -1,3 +1,7 @@
+# Generic MAILBOX API
+
+obj-$(CONFIG_MAILBOX)		+= mailbox.o
+
 obj-$(CONFIG_PL320_MBOX)	+= pl320-ipc.o
  obj-$(CONFIG_OMAP_MBOX)		+= omap-mailbox.o
drivers/mailbox/mailbox.c new file +474 -0

474 lines not on this page · raw message

drivers/mailbox/mailbox_internal.h new file +77 -0

77 lines not on this page · raw message

include/linux/mailbox.h +14 -13
index 5161f63..232e2c4 100644
@@ -1,17 +1,17 @@
 /*
- * This program is free software; you can redistribute it and/or modify it
- * under the terms and conditions of the GNU General Public License,
- * version 2, as published by the Free Software Foundation.
- *
- * This program is distributed in the hope it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
- * more details.
- *
- * You should have received a copy of the GNU General Public License along with
- * this program.  If not, see <http://www.gnu.org/licenses/>.
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
  */
 -int pl320_ipc_transmit(u32 *data);
-int pl320_ipc_register_notifier(struct notifier_block *nb);
-int pl320_ipc_unregister_notifier(struct notifier_block *nb);
+#ifndef __MAILBOX_H
+#define __MAILBOX_H
+
+enum xfer_result {
+	XFER_OK = 0,
+	XFER_ERR,
+};
+
+typedef unsigned request_token_t;
+
+#endif /* __MAILBOX_H */
include/linux/mailbox_client.h new file +87 -0

87 lines not on this page · raw message

include/linux/mailbox_controller.h new file +102 -0

102 lines not on this page · raw message

#1 lore raw

RFC PATCH 1/3 Enable BCM2835 mailbox support

Quoted 9 lines · Craig McGeachie

Cherry-picked selection of commits by Lubomir Rintel [1], Suman Anna and Jassi Brar [2] on which to base the implementation.

[1] http://lists.infradead.org/pipermail/linux-rpi-kernel/2013-April/000528.html
[2] http://lists.infradead.org/pipermail/linux-rpi-kernel/2013-May/000546.html

There are two problems with the mailbox framework that I can find. The first is in ipc_send_message. On the BCM2835, the mailbox send result doesn't trigger an interrupt, but the pending reply does. When ipc_request_channel is called without blocking, the flow of events looks like this:

[    0.521312] req - cf9845c0: 00000020 00000000 00030006 00000008
[    0.527245] req - cf9845d0: 00000008 00000000 00000000 00000000
[    0.533248] bcm2835-mbox 2000b880.mailbox: Request 0x0F9845C8
[    0.539007] bcm2835-mbox 2000b880.mailbox: Reply 0x0F9845C8
[    0.544582] bcm2835_thermal thermal.1: rxcb #1 @0f9845c0
[    0.549939] bcm2835-mbox 2000b880.mailbox: Polling
[    0.554773] bcm2835_thermal thermal.1: txcb #1 @0f9845c0
[    0.560102] rep - cf9845c0: 00000020 80000000 00030006 00000008
[    0.566062] rep - cf9845d0: 80000008 00000000 000078da 00000000

I've never seen the transmit callback come first.

When ipc_request_channel is called with blocking, then the best result I've had a null pointer causing a kernel panic. Most of the time, the system just deadlocks. The fix is:

drivers/mailbox/mailbox.c +1 -1
index c4ef608..08fae83 100644
@@ -202,6 +202,7 @@ request_token_t ipc_send_message(void *channel, void *mssg)
        if (!t)
                pr_err("Try increasing MBOX_TX_QUEUE_LEN\n");
 
+       init_completion(&chan->tx_complete);
        _msg_submit(chan);
 
        if (chan->txdone_method == TXDONE_BY_POLL)
@@ -209,7 +210,6 @@ request_token_t ipc_send_message(void *channel, void *mssg)
 
        if (chan->tx_block && chan->active_req) {
                int ret;
-               init_completion(&chan->tx_complete);
                ret = wait_for_completion_timeout(&chan->tx_complete,
                        chan->tx_tout);
                if (ret == 0) {

When the init_completion is done after the _msg_submit, it might be executed after the complete in tx_tick. And for the BCM2835, it's pretty reliable that this will happen. When this happens, the calls are:
1. complete
2. init_completion
3. wait_for_completion_timeout

The complete call is missed.

With this fix in place, the flow of events now looks like:

[    0.521214] req - cf9845c0: 00000020 00000000 00030006 00000008
[    0.527149] req - cf9845d0: 00000008 00000000 00000000 00000000
[    0.533149] bcm2835-mbox 2000b880.mailbox: Request 0x0F9845C8
[    0.538908] bcm2835-mbox 2000b880.mailbox: Reply 0x0F9845C8
[    0.544482] bcm2835_thermal thermal.1: rxcb #1 @0f9845c0
[    0.549840] bcm2835-mbox 2000b880.mailbox: Polling
[    0.554686] rep - cf9845c0: 00000020 80000000 00030006 00000008
[    0.560648] rep - cf9845d0: 80000008 00000000 0000835c 00000000

Just a comment about this common mailbox API with respect to the BCM2835. It makes more sense for the BCM285 to always send messages with a blocking transmit. The mailbox send is a single 32 bit IO register shared for all channels (the channel number is encoded into the send data). BCM2835 drivers would only want to use the mailbox API with an effective transmit queue depth of 1. This is fine. From what I can see, IPC calls via the mailbox should be very infrequent. Mostly only to set up some other inter-processor communication mechanism, or to query remote processor properties at system start or on behalf of a user process.

The second problem is in ipc_request_channel

list_for_each_entry(chan, &con->channels, node) {
	if (!strcmp(con_name + len + 1, chan->name)
			&& !chan->assigned) {
		spin_lock_irqsave(&chan->lock, flags);
[...snip...]
		chan->assigned = true;
[...snip...]
		spin_unlock_irqrestore(&chan->lock, flags);
		ret = 1;
		break;
	}
}

I think this is a race condition. I can see no reason why two or more threads couldn't proceed through the if condition before the spin lock is acquired. If that happens, then after the first thread has claimed the channel by setting assigned true and released the spin lock, the second thread will proceed, and overwrite the channel state with its own values. Then wackiness ensues.

The fix would be to move the lock acquisition to before the if, but getting the unlock correct will become messy.

Cheers,
Craig.

#2 lore raw

RFC PATCH 1/3 Enable BCM2835 mailbox support

Quoted 63 lines · Craig McGeachie
Quoted 9 lines · Craig McGeachie

Cherry-picked selection of commits by Lubomir Rintel [1], Suman Anna and Jassi Brar [2] on which to base the implementation.

[1] http://lists.infradead.org/pipermail/linux-rpi-kernel/2013-April/000528.html
[2] http://lists.infradead.org/pipermail/linux-rpi-kernel/2013-May/000546.html

There are two problems with the mailbox framework that I can find. The first is in ipc_send_message. On the BCM2835, the mailbox send result doesn't trigger an interrupt, but the pending reply does. When ipc_request_channel is called without blocking, the flow of events looks like this:

[    0.521312] req - cf9845c0: 00000020 00000000 00030006 00000008
[    0.527245] req - cf9845d0: 00000008 00000000 00000000 00000000
[    0.533248] bcm2835-mbox 2000b880.mailbox: Request 0x0F9845C8
[    0.539007] bcm2835-mbox 2000b880.mailbox: Reply 0x0F9845C8
[    0.544582] bcm2835_thermal thermal.1: rxcb #1 @0f9845c0
[    0.549939] bcm2835-mbox 2000b880.mailbox: Polling
[    0.554773] bcm2835_thermal thermal.1: txcb #1 @0f9845c0
[    0.560102] rep - cf9845c0: 00000020 80000000 00030006 00000008
[    0.566062] rep - cf9845d0: 80000008 00000000 000078da 00000000

I've never seen the transmit callback come first.

When ipc_request_channel is called with blocking, then the best result I've had a null pointer causing a kernel panic. Most of the time, the system just deadlocks. The fix is:

drivers/mailbox/mailbox.c +1 -1
index c4ef608..08fae83 100644
@@ -202,6 +202,7 @@ request_token_t ipc_send_message(void *channel, void *mssg)
        if (!t)
                pr_err("Try increasing MBOX_TX_QUEUE_LEN\n");
 
+       init_completion(&chan->tx_complete);
        _msg_submit(chan);
 
        if (chan->txdone_method == TXDONE_BY_POLL)
@@ -209,7 +210,6 @@ request_token_t ipc_send_message(void *channel, void *mssg)
 
        if (chan->tx_block && chan->active_req) {
                int ret;
-               init_completion(&chan->tx_complete);
                ret = wait_for_completion_timeout(&chan->tx_complete,
                        chan->tx_tout);
                if (ret == 0) {

When the init_completion is done after the _msg_submit, it might be executed after the complete in tx_tick. And for the BCM2835, it's pretty reliable that this will happen. When this happens, the calls are:
1. complete
2. init_completion
3. wait_for_completion_timeout

The complete call is missed.

With this fix in place, the flow of events now looks like:

Quoted 9 lines
[    0.521214] req - cf9845c0: 00000020 00000000 00030006 00000008
[    0.527149] req - cf9845d0: 00000008 00000000 00000000 00000000
[    0.533149] bcm2835-mbox 2000b880.mailbox: Request 0x0F9845C8
[    0.538908] bcm2835-mbox 2000b880.mailbox: Reply 0x0F9845C8
[    0.544482] bcm2835_thermal thermal.1: rxcb #1 @0f9845c0
[    0.549840] bcm2835-mbox 2000b880.mailbox: Polling
[    0.554686] rep - cf9845c0: 00000020 80000000 00030006 00000008
[    0.560648] rep - cf9845d0: 80000008 00000000 0000835c 00000000

LF.Tan already provided the same patch (and another) a few weeks ago. I have it locally but it should be pushed soon.

commit 28785f7c4af687026e601e0a7328414de43c84b3

Author: LeyFoon Tan <[email protected]>
Date:   Tue Aug 13 10:40:21 2013 +0530

mailbox: Fix deleteing poll timer

Try to delete the timer only if it was init/used.

drivers/mailbox/mailbox.c +2 -1
index 0883496..0888be5 100644
@@ -469,7 +469,8 @@ void ipc_links_unregister(struct ipc_controller *ipc)
        list_for_each_entry(chan, &con->channels, node)
                ipc_free_channel((void *)chan);
 
-       del_timer_sync(&con->poll);
+       if (ipc->txdone_poll)
+               del_timer_sync(&con->poll);
 
        kfree(con);
 }

commit 8456ca4665da4a5bac3163eb1ddd98bcccd690c2

Author: LeyFoon Tan <[email protected]>
Date:   Mon Aug 12 22:02:10 2013 +0530

mailbox: Fix TX completion init

For fast TX the complete could be called before being initialized as follows
ipc_send_message --> poll_txdone --> tx_tick -->
complete(&chan->tx_complete)

Init the completion early enough to fix the race.

drivers/mailbox/mailbox.c +3 -1
index c4ef608..0883496 100644
@@ -198,6 +198,9 @@ request_token_t ipc_send_message(void *channel, void *mssg)
        if (!chan || !chan->assigned)
                return 0;
 
+       if (chan->tx_block)
+               init_completion(&chan->tx_complete);
+
        t = _add_to_rbuf(chan, mssg);
        if (!t)
                pr_err("Try increasing MBOX_TX_QUEUE_LEN\n");
@@ -209,7 +212,6 @@ request_token_t ipc_send_message(void *channel, void *mssg)
 
        if (chan->tx_block && chan->active_req) {
                int ret;
-               init_completion(&chan->tx_complete);
                ret = wait_for_completion_timeout(&chan->tx_complete,
                        chan->tx_tout);
                if (ret == 0) {
Quoted 33 lines

Just a comment about this common mailbox API with respect to the BCM2835. It makes more sense for the BCM285 to always send messages with a blocking transmit. The mailbox send is a single 32 bit IO register shared for all channels (the channel number is encoded into the send data). BCM2835 drivers would only want to use the mailbox API with an effective transmit queue depth of 1. This is fine. From what I can see, IPC calls via the mailbox should be very infrequent. Mostly only to set up some other inter-processor communication mechanism, or to query remote processor properties at system start or on behalf of a user process.

The second problem is in ipc_request_channel

list_for_each_entry(chan, &con->channels, node) {
        if (!strcmp(con_name + len + 1, chan->name)
                        && !chan->assigned) {
                spin_lock_irqsave(&chan->lock, flags);
[...snip...]
                chan->assigned = true;
[...snip...]
                spin_unlock_irqrestore(&chan->lock, flags);
                ret = 1;
                break;
        }
}

I think this is a race condition. I can see no reason why two or more threads couldn't proceed through the if condition before the spin lock is acquired. If that happens, then after the first thread has claimed the channel by setting assigned true and released the spin lock, the second thread will proceed, and overwrite the channel state with its own values. Then wackiness ensues.

Quoted 3 lines

The fix would be to move the lock acquisition to before the if, but getting the unlock correct will become messy.

Thanks for pointing out... the following fix should work.

drivers/mailbox/mailbox.c +7 -5
index 0888be5..25e9924 100644
@@ -261,9 +261,10 @@ void *ipc_request_channel(struct ipc_client *cl)
 
        ret = 0;
        list_for_each_entry(chan, &con->channels, node) {
-               if (!strcmp(con_name + len + 1, chan->name)
-                               && !chan->assigned) {
-                       spin_lock_irqsave(&chan->lock, flags);
+               if (strcmp(con_name + len + 1, chan->name))
+                       continue;
+               spin_lock_irqsave(&chan->lock, flags);
+               if (!chan->assigned) {
                        chan->msg_free = 0;
                        chan->msg_count = 0;
                        chan->active_req = NULL;
@@ -279,10 +280,11 @@ void *ipc_request_channel(struct ipc_client *cl)
                        if (chan->txdone_method == TXDONE_BY_POLL
                                        && cl->knows_txdone)
                                chan->txdone_method |= TXDONE_BY_ACK;
-                       spin_unlock_irqrestore(&chan->lock, flags);
                        ret = 1;
-                       break;
                }
+               spin_unlock_irqrestore(&chan->lock, flags);
+               if (ret)
+                       break;
        }
 
        if (!ret) {
#3 lore raw

RFC PATCH 1/3 Enable BCM2835 mailbox support

In reply to Craig McGeachie
Quoted 9 lines · Craig McGeachie

Cherry-picked selection of commits by Lubomir Rintel [1], Suman Anna and Jassi Brar [2] on which to base the implementation.

[1] http://lists.infradead.org/pipermail/linux-rpi-kernel/2013-April/000528.html
[2] http://lists.infradead.org/pipermail/linux-rpi-kernel/2013-May/000546.html

This needs a proper patch description. It should describe what the mailbox HW is, what services it provides, etc.

There shouldn't be any text after the tag paragraph (i.e. the s-o-b lines) and the --- separator.

Are you able to generate patches using "git format-patch". That would included diffstat in the patch summary, which can be useful.

I'd prefer to see patch 1 and 2 squashed together, since the new mailbox driver should just be implemented in one shot, rather than added with one implementation, then reworked to be different in a separate patch.

That said, the patches need to be split up based on subsystem divisions. The squashed patch 1 and 2, and the separate patch 3, should be split into:

a) A patch that adds the driver. This would touch just drivers/mailbox/... (or drivers/thermal/...) and Documentation/devicetree/bindings/...

b) A patch that adds the necessary entries to *.dts/*.dtsi

c) A patch that enables any new entries in bcm2835_defconfig.

In my initial round of review, I'll only discuss the DT bindings, and only in response to patch 2 where they're complete rather than here. Once the bindings are finalized and the driver adjusted to implement the final bindings, then I'll look at the driver code.

Note that all new DT bindings should be CC'd to the DT bindings maintainers; see the following in MAINTAINERS:

OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS

M:      Rob Herring <[email protected]>
M:      Pawel Moll <[email protected]>
M:      Mark Rutland <[email protected]>
M:      Stephen Warren <[email protected]>
M:      Ian Campbell <[email protected]>
L:      devicetree at vger.kernel.org
S:      Maintained
F:      Documentation/devicetree/
F:      arch/*/boot/dts/
F:      include/dt-bindings/

arch/arm/boot/dts/bcm2835.dtsi line 30

Quoted 7 lines
arch/arm/boot/dts/bcm2835.dtsi +0 -0
index 1e12aef..4fc8a68 100644
@@ -29,6 +29,12 @@
 			#interrupt-cells = <2>;
 		};

There needs to be a blank line here.

arch/arm/boot/dts/bcm2835.dtsi line 31

Quoted 1 line
arch/arm/boot/dts/bcm2835.dtsi +0 -0
 +		mailbox {

Patch corruption? + is indented!

#4 lore raw

RFC PATCH 1/3 Enable BCM2835 mailbox support

Quoted 11 lines · Stephen Warren
Quoted 9 lines · Craig McGeachie

Cherry-picked selection of commits by Lubomir Rintel [1], Suman Anna and Jassi Brar [2] on which to base the implementation.

[1] http://lists.infradead.org/pipermail/linux-rpi-kernel/2013-April/000528.html
[2] http://lists.infradead.org/pipermail/linux-rpi-kernel/2013-May/000546.html

Quoted 8 lines

This needs a proper patch description. It should describe what the mailbox HW is, what services it provides, etc.

There shouldn't be any text after the tag paragraph (i.e. the s-o-b lines) and the --- separator.

Are you able to generate patches using "git format-patch". That would included diffstat in the patch summary, which can be useful.

I can do all that. I'm rather new to all of this, and didn't get git format-patch and git send-email sorted out until after this. You really don't want to know what I did to get this sent.

Quoted 11 lines

I'd prefer to see patch 1 and 2 squashed together, since the new mailbox driver should just be implemented in one shot, rather than added with one implementation, then reworked to be different in a separate patch.

That said, the patches need to be split up based on subsystem divisions. The squashed patch 1 and 2, and the separate patch 3, should be split into:

a) A patch that adds the driver. This would touch just drivers/mailbox/... (or drivers/thermal/...) and Documentation/devicetree/bindings/...

Quoted 8 lines

b) A patch that adds the necessary entries to *.dts/*.dtsi

c) A patch that enables any new entries in bcm2835_defconfig.

In my initial round of review, I'll only discuss the DT bindings, and only in response to patch 2 where they're complete rather than here. Once the bindings are finalized and the driver adjusted to implement the final bindings, then I'll look at the driver code.

I'll look at restructuring the patches. Are you really sure about drivers/mailbox/* and drivers/thermal/* as one patch?

The serious concern that I have is that Suman Anna and Jassi Brar are still working on the core mailbox subsystem. I've just captured their work to present a working patch that will build and show the mailbox working. That is why I marked the series as RFC. I'm not happy about submitting someone else's work that I know is still evolving. Especially when I know that there are a couple of fixes that Jassi has made but not published yet.

For all that, I want mailbox support for the BCM2835 as soon as possible. I see its lack as a serious blocker. The video driver needs it to get the videocore to map an agreed chunk of memory for a framebuffer. The DMA controller could use it to find which DMA channels are available for use by the ARM core. Power management of whole devices uses it (DMA can turn individual channels on and off but that's with iomem register). There may be more, but that's all I've researched. Anything I can do to hasten the general availability of mailbox support ... well, anything that doesn't involve stepping on toes.

Quoted 4 lines

Note that all new DT bindings should be CC'd to the DT bindings maintainers; see the following in MAINTAINERS:

OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS

Quoted 10 lines
M:      Rob Herring <[email protected]>
M:      Pawel Moll <[email protected]>
M:      Mark Rutland <[email protected]>
M:      Stephen Warren <[email protected]>
M:      Ian Campbell <[email protected]>
L:      devicetree at vger.kernel.org
S:      Maintained
F:      Documentation/devicetree/
F:      arch/*/boot/dts/
F:      include/dt-bindings/

Do you want me to CC this interim discussion as well, or wait until there is a more accpetable patch put together.

Quoted 3 lines
Quoted 1 line
  +		mailbox {

Patch corruption? + is indented!

As I said. Poor tool use.

Cheers,
Craig.