Skip to contentSkip to conversation

Inter-processor Mailboxes Drivers

RFC Meador Inge 8 participants 19 messages Feb 11, 2011 – Feb 16, 2011

lore
Share & tools
lore
#0 lore raw

Hi All,

I am currently working on building AMP systems using OpenMCAPI
(https://bitbucket.org/hollisb/openmcapi/wiki/Home) as the inter-processor communication mechanism. With OpenMCAPI we, of course, need a way to send messages to various cores. On some Freescale PPC platforms (e.g. P1022DS, MPC8572DS), we have been using message registers to do this work. Recently, I was looking at the OMAP4 mailboxes to gear up for moving into ARM based platforms.

With that, I noticed 'arch/arm/plat-omap/mailbox.c'. This is very specific to the OMAP4 boards. I am looking at designing a new set of drivers to expose a mailbox service to userspace that will be used for inter-processor communication. This would entail the traditional generic/specific driver split:

1. Hardware specific bits somewhere under '.../arch/*'. Drivers for the MPIC message registers on Power and OMAP4 mailboxes, for example.
2. A higher level driver under '.../drivers/mailbox/*'. That the pieces in (1) would register with. This piece would expose the main kernel API.
3. Userspace interfaces for accessing the mailboxes. A '/dev/mailbox1', '/dev/mailbox2', etc... mapping, for example.

Now I have the following questions:

1. Do others see value in this?
2. Does something like this already exist?
3. Is someone else already working on this?

Any feedback will be greatly appreciated.

Signature · 2 lines
Meador Inge     | meador_inge AT mentor.com
Mentor Embedded | http://www.mentor.com/embedded-software
#1 lore raw

Hi,

Quoted 4 lines · Meador Inge

? ?1. Hardware specific bits somewhere under '.../arch/*'. ?Drivers ? ? ? for the MPIC message registers on Power and OMAP4 mailboxes, for ? ? ? example.

Yes; this can help.

Quoted 3 lines

? ?2. A higher level driver under '.../drivers/mailbox/*'. ?That the ? ? ? pieces in (1) would register with. ?This piece would expose the ? ? ? main kernel API.

A lot of mailboxes are too platform specific with regards to the communication with the main CPU and probably it depends on the mailbox too; you can find polling and
interrupt supported mailbox support at times on the same platform. APIs should probably be
generic enough to be able to operate in any context.

Quoted 3 lines

Now I have the following questions:

? ?1. Do others see value in this?

At least I would like this; I wanted to generalize such mailbox IPCs right from the
day when I was working on one, but coudnt really work on that.

Quoted 1 line

? ?2. Does something like this already exist?

Not generic as you say; but apart from the OMAP platforms, you could refer to arch/arm/mach-ux500/prcmu for a mailbox based IPC on the U8500 platform.

Quoted 1 line

? ?3. Is someone else already working on this?

Not sure of that too :), but I am CCing Linus W, the maintainer of U8500 if he thinks it is a good idea to come up with a mailbox IPC framework

Cheers!

Signature · 3 lines
---------
The views expressed in this email are personal and do not necessarily
echo my employers.
#2 lore raw

2011/2/12 Sundar <[email protected]>:

Quoted 9 lines

At least I would like this; I wanted to generalize such mailbox IPCs right from the day when I was working on one, but coudnt really work on that.

Quoted 1 line

? ?2. Does something like this already exist?

Not generic as you say; but apart from the OMAP platforms, you could refer to arch/arm/mach-ux500/prcmu for a mailbox based IPC on the U8500 platform.

We also have this thing:
arch/arm/mach-ux500/mbox-db5500.c

It's another mailbox driver, this one talks to the modem in the U5500 (basically a physical transport for the CAIF protocol). (For the U8500 I think modem IPC is instead handled with a high-speed hardware FIFO, a bit different.)

Quoted 5 lines
Quoted 1 line

? ?3. Is someone else already working on this?

Not sure of that too :), but I am CCing Linus W, the maintainer of U8500 if he thinks it is a good idea to come up with a mailbox IPC framework

I don't know too much about the subject actually, I've not been deeply into any such code. I don't think anyone is working on something general from ST-Ericsson or Linaro.

Recently I saw that Texas Instruments are posting a "hardware spinlock" framework though, this would be on a related tone, but I think it's for shared data structures (control path) rather than buffer passing (data path). I'm guessing this works like that one CPU gets to spin waiting for another one to release the lock.

Given that we may have a framework for hardware spinlock and that we don't want to stockpile drivers into arch/* or drivers/misc/* I would say it's intuitively a good idea, but the question is what data types you would pass in? In arch/arm/mach-ux500/include/mach/mbox-db5500.h we have a struct like this:

/**
* struct mbox - Mailbox instance struct

  * @list:              Linked list head.
  * @pdev:              Pointer to device struct.
  * @cb:                Callback function. Will be called
  *                     when new data is received.
  * @client_data:       Clients private data. Will be sent back
  *                     in the callback function.
  * @virtbase_peer:     Virtual address for outgoing mailbox.
  * @virtbase_local:    Virtual address for incoming mailbox.
  * @buffer:            Then internal queue for outgoing messages.
  * @name:              Name of this mailbox.
  * @buffer_available:  Completion variable to achieve "blocking send".
  *                     This variable will be signaled when there is
  *                     internal buffer space available.
  * @client_blocked:    To keep track if any client is currently
  *                     blocked.
  * @lock:              Spinlock to protect this mailbox instance.
  * @write_index:       Index in internal buffer to write to.
  * @read_index:        Index in internal buffer to read from.
  * @allocated:         Indicates whether this particular mailbox
  *                     id has been allocated by someone.
  */
struct mbox {
        struct list_head list;
        struct platform_device *pdev;
        mbox_recv_cb_t *cb;
        void *client_data;
        void __iomem *virtbase_peer;
        void __iomem *virtbase_local;
        u32 buffer[MBOX_BUF_SIZE];
        char name[MBOX_NAME_SIZE];
        struct completion buffer_available;
        u8 client_blocked;
        spinlock_t lock;
        u8 write_index;
        u8 read_index;
        bool allocated;
};

Compare OMAPs mailboxes in
arch/arm/plat-omap/include/plat/mailbox.h:

typedef u32 mbox_msg_t;

truct omap_mbox_ops {
        omap_mbox_type_t        type;
        int             (*startup)(struct omap_mbox *mbox);
        void            (*shutdown)(struct omap_mbox *mbox);
        /* fifo */
        mbox_msg_t      (*fifo_read)(struct omap_mbox *mbox);
        void            (*fifo_write)(struct omap_mbox *mbox, mbox_msg_t msg);
        int             (*fifo_empty)(struct omap_mbox *mbox);
        int             (*fifo_full)(struct omap_mbox *mbox);
        /* irq */
        void            (*enable_irq)(struct omap_mbox *mbox,
                                                omap_mbox_irq_t irq);
        void            (*disable_irq)(struct omap_mbox *mbox,
                                                omap_mbox_irq_t irq);
        void            (*ack_irq)(struct omap_mbox *mbox, omap_mbox_irq_t irq);
        int             (*is_irq)(struct omap_mbox *mbox, omap_mbox_irq_t irq);
        /* ctx */
        void            (*save_ctx)(struct omap_mbox *mbox);
        void            (*restore_ctx)(struct omap_mbox *mbox);
};
struct omap_mbox_queue {
        spinlock_t              lock;
        struct kfifo            fifo;
        struct work_struct      work;
        struct tasklet_struct   tasklet;
        struct omap_mbox        *mbox;
        bool full;
};
struct omap_mbox {
        char                    *name;
        unsigned int            irq;
        struct omap_mbox_queue  *txq, *rxq;
        struct omap_mbox_ops    *ops;
        struct device           *dev;
        void                    *priv;
        int                     use_count;
        struct blocking_notifier_head   notifier;
};

Some of this may be generalized? I dunno, they look quite different but maybe queueing etc can actually be made general enough to form a framework.

Yours,
Linus Walleij

#8 lore raw
In reply to Meador Inge

2011/2/11 Meador Inge <[email protected]>:

Quoted 6 lines

?This would entail the traditional
generic/specific driver split:

? ?1. Hardware specific bits somewhere under '.../arch/*'. ?Drivers ? ? ? for the MPIC message registers on Power and OMAP4 mailboxes, for ? ? ? example.

Having any drivers under arch/* is no good tradition IMO. Better move the whole shebang down to drivers/mailbox so that the subsystem maintainer get the complete overview of her/his driver family.

Quoted 3 lines

? ?2. A higher level driver under '.../drivers/mailbox/*'. ?That the ? ? ? pieces in (1) would register with. ?This piece would expose the ? ? ? main kernel API.

Cool...

Quoted 2 lines

? ?3. Userspace interfaces for accessing the mailboxes. ?A ? ? ? '/dev/mailbox1', '/dev/mailbox2', etc... mapping, for example.

What kind of business does userspace have with directly using mailboxes? Enlighten me so I get it... in our system these are used by protocols, such as net/caif/* thru drivers/net/caif/*, and we have similar kernelspace functionality for Phonet.

CAIF and Phonet on the other hand, have custom openings down to the thing that exists on the other end of the mailbox. Most of these systems tend to talk some funny protocol that is often better handled by the kernel than by any userspace.

So is this for the situation when you have no intermediate protocol between your userpace and the other CPU's subsystem? Or are you thinking about handling that protocol in userspace? That is generally not such a good idea for efficiency reasons.

Yoursm
Linus Walleij

#11 lore raw
In reply to Meador Inge

Hi Meador,

From: ext Meador Inge <[email protected]>
Subject: [RFC] Inter-processor Mailboxes Drivers
Date: Fri, 11 Feb 2011 15:19:51 -0600

Quoted 26 lines

Hi All,

I am currently working on building AMP systems using OpenMCAPI
(https://bitbucket.org/hollisb/openmcapi/wiki/Home) as the inter-processor communication mechanism. With OpenMCAPI we, of course,
need a way to send messages to various cores. On some Freescale PPC platforms (e.g. P1022DS, MPC8572DS), we have been using message registers to do this work. Recently, I was looking at the OMAP4 mailboxes to gear up for moving into ARM based platforms.

With that, I noticed 'arch/arm/plat-omap/mailbox.c'. This is very specific to the OMAP4 boards. I am looking at designing a new set of drivers to expose a mailbox service to userspace that will be used for inter-processor communication. This would entail the traditional generic/specific driver split:

1. Hardware specific bits somewhere under '.../arch/*'. Drivers for the MPIC message registers on Power and OMAP4 mailboxes, for example.
2. A higher level driver under '.../drivers/mailbox/*'. That the pieces in (1) would register with. This piece would expose the main kernel API.
3. Userspace interfaces for accessing the mailboxes. A '/dev/mailbox1', '/dev/mailbox2', etc... mapping, for example.

Quoted 3 lines

Now I have the following questions:

1. Do others see value in this?

Yes.

I discussed this with TI(Hari) long time ago, but it didn't proceed with some reason, not techinical one.

Quoted 4 lines

2. Does something like this already exist?
3. Is someone else already working on this?

Any feedback will be greatly appreciated.

Now the basic concpet can be devided into 3 parts, (1)HW dependent,
(2) generic driver, (3) character device interface. I guess that it might be good to have one _pseudo_ H/W instance, then easier for verification without real H/W and firmwares, across multiplatforms.

#3 lore raw
In reply to Linus Walleij

From: ext Linus Walleij <[email protected]>
Subject: Re: [RFC] Inter-processor Mailboxes Drivers
Date: Sun, 13 Feb 2011 22:16:12 +0100

Quoted 130 lines

2011/2/12 Sundar <[email protected]>:

Quoted 9 lines

At least I would like this; I wanted to generalize such mailbox IPCs right from the day when I was working on one, but coudnt really work on that.

Quoted 1 line

? ?2. Does something like this already exist?

Not generic as you say; but apart from the OMAP platforms, you could refer to arch/arm/mach-ux500/prcmu for a mailbox based IPC on the U8500 platform.

We also have this thing:
arch/arm/mach-ux500/mbox-db5500.c

It's another mailbox driver, this one talks to the modem in the U5500 (basically a physical transport for the CAIF protocol). (For the U8500 I think modem IPC is instead handled with a high-speed hardware FIFO, a bit different.)

Quoted 5 lines
Quoted 1 line

? ?3. Is someone else already working on this?

Not sure of that too :), but I am CCing Linus W, the maintainer of U8500 if he thinks it is a good idea to come up with a mailbox IPC framework

I don't know too much about the subject actually, I've not been deeply into any such code. I don't think anyone is working on something general from ST-Ericsson or Linaro.

Recently I saw that Texas Instruments are posting a "hardware spinlock" framework though, this would be on a related tone, but I think it's for shared data structures (control path) rather than buffer passing (data path). I'm guessing this works like that one CPU gets to spin waiting for another one to release the lock.

Given that we may have a framework for hardware spinlock and that we don't want to stockpile drivers into arch/* or drivers/misc/* I would say it's intuitively a good idea, but the question is what data types you would pass in? In arch/arm/mach-ux500/include/mach/mbox-db5500.h we have a struct like this:

/**
* struct mbox - Mailbox instance struct

  * @list:              Linked list head.
  * @pdev:              Pointer to device struct.
  * @cb:                Callback function. Will be called
  *                     when new data is received.
  * @client_data:       Clients private data. Will be sent back
  *                     in the callback function.
  * @virtbase_peer:     Virtual address for outgoing mailbox.
  * @virtbase_local:    Virtual address for incoming mailbox.
  * @buffer:            Then internal queue for outgoing messages.
  * @name:              Name of this mailbox.
  * @buffer_available:  Completion variable to achieve "blocking send".
  *                     This variable will be signaled when there is
  *                     internal buffer space available.
  * @client_blocked:    To keep track if any client is currently
  *                     blocked.
  * @lock:              Spinlock to protect this mailbox instance.
  * @write_index:       Index in internal buffer to write to.
  * @read_index:        Index in internal buffer to read from.
  * @allocated:         Indicates whether this particular mailbox
  *                     id has been allocated by someone.
  */
struct mbox {
        struct list_head list;
        struct platform_device *pdev;
        mbox_recv_cb_t *cb;
        void *client_data;
        void __iomem *virtbase_peer;
        void __iomem *virtbase_local;
        u32 buffer[MBOX_BUF_SIZE];
        char name[MBOX_NAME_SIZE];
        struct completion buffer_available;
        u8 client_blocked;
        spinlock_t lock;
        u8 write_index;
        u8 read_index;
        bool allocated;
};

Compare OMAPs mailboxes in
arch/arm/plat-omap/include/plat/mailbox.h:

typedef u32 mbox_msg_t;

truct omap_mbox_ops {
        omap_mbox_type_t        type;
        int             (*startup)(struct omap_mbox *mbox);
        void            (*shutdown)(struct omap_mbox *mbox);
        /* fifo */
        mbox_msg_t      (*fifo_read)(struct omap_mbox *mbox);
        void            (*fifo_write)(struct omap_mbox *mbox, mbox_msg_t msg);
        int             (*fifo_empty)(struct omap_mbox *mbox);
        int             (*fifo_full)(struct omap_mbox *mbox);
        /* irq */
        void            (*enable_irq)(struct omap_mbox *mbox,
                                                omap_mbox_irq_t irq);
        void            (*disable_irq)(struct omap_mbox *mbox,
                                                omap_mbox_irq_t irq);
        void            (*ack_irq)(struct omap_mbox *mbox, omap_mbox_irq_t irq);
        int             (*is_irq)(struct omap_mbox *mbox, omap_mbox_irq_t irq);
        /* ctx */
        void            (*save_ctx)(struct omap_mbox *mbox);
        void            (*restore_ctx)(struct omap_mbox *mbox);
};
struct omap_mbox_queue {
        spinlock_t              lock;
        struct kfifo            fifo;
        struct work_struct      work;
        struct tasklet_struct   tasklet;
        struct omap_mbox        *mbox;
        bool full;
};
struct omap_mbox {
        char                    *name;
        unsigned int            irq;
        struct omap_mbox_queue  *txq, *rxq;
        struct omap_mbox_ops    *ops;
        struct device           *dev;
        void                    *priv;
        int                     use_count;
        struct blocking_notifier_head   notifier;
};
Quoted 3 lines

Some of this may be generalized? I dunno, they look quite different but maybe queueing etc can actually be made general enough to form a framework.

OMAP mailbox is the interrupt driven 32bit unit H/W FIFO to other cores.

"struct omap_mbox_ops" was provided mainly to absorb the difference between OMAP1 and OMAP2+ from H/W POV. So generally the layer could be:

	-----------------------
	character device driver
	-----------------------
	generic mailbox driver
	-----------------------
	   H/W registration
	-----------------------

In OMAP case, in addition to the above, it could be exceptionally:

	-----------------------
	character device driver
	-----------------------
	generic mailbox driver
	-----------------------
	   H/W registration
	-----------------------
	   OMAP 1  | OMAP2+
	-----------------------

So "character device driver"(interface) and "generic mailbox driver"(queuing) may be able to abstructed/generalized.

#4 lore raw
Quoted 2 lines · Hiroshi DOYU

OMAP mailbox is the interrupt driven 32bit unit H/W FIFO to other cores.

How is it used? Is it a low-traffic (like single 32bit words etc) signal control-path link while the actual high-throughput data-path is done with shared memory? (That is how the db5500 mbox works anyways.)

Linus Walleij

#5 lore raw

From: ext Linus Walleij <[email protected]>
Subject: Re: [RFC] Inter-processor Mailboxes Drivers
Date: Mon, 14 Feb 2011 09:39:32 +0100

Quoted 8 lines · Hiroshi DOYU
Quoted 3 lines

OMAP mailbox is the interrupt driven 32bit unit H/W FIFO to other cores.

How is it used? Is it a low-traffic (like single 32bit words etc) signal control-path link while the actual high-throughput data-path is done with shared memory? (That is how the db5500 mbox works anyways.)

Yes, maybe quite similar.

maibox is not single 32 bit but is 32 bit x 4(or 8?) slots fifo, IIRC, and mainly it is used as notification between cores. And big amount of data is transftered with sahred memory, which has been mapped onto the virtual address space of the other side of core, in advance.

For example, typical usage of DSP, mp3 decoding,

1, ARM maps 2 shared memory area(input/output) onto DSP virtual
address space.
2, ARM fills mp3 data in input buffer.
3, ARM notifies DSP that data is ready in input buffer. 4, DSP decodes input data and put output data on output buffer. 5, DSP notifies that output buffer is ready.

Roughly something like the above. DSP S/W is multi-tasking, though.

Does db5500 use IOMMU for mapping shared memories?

#6 lore raw
Quoted 1 line · Hiroshi DOYU

Does db5500 use IOMMU for mapping shared memories?

Nope, it's a fixed physical allocation from the modem side of the world.

Yours,
Linus Walleij

#7 lore raw
In reply to Hiroshi DOYU

From: Hiroshi DOYU <[email protected]>
Subject: Re: [RFC] Inter-processor Mailboxes Drivers
Date: Mon, 14 Feb 2011 10:55:53 +0200 (EET)

Quoted 22 lines

From: ext Linus Walleij <[email protected]>
Subject: Re: [RFC] Inter-processor Mailboxes Drivers
Date: Mon, 14 Feb 2011 09:39:32 +0100

Quoted 8 lines · Hiroshi DOYU
Quoted 3 lines

OMAP mailbox is the interrupt driven 32bit unit H/W FIFO to other cores.

How is it used? Is it a low-traffic (like single 32bit words etc) signal control-path link while the actual high-throughput data-path is done with shared memory? (That is how the db5500 mbox works anyways.)

Yes, maybe quite similar.

maibox is not single 32 bit but is 32 bit x 4(or 8?) slots fifo, IIRC, and mainly it is used as notification between cores. And big amount of data is transftered with sahred memory, which has been mapped onto the virtual address space of the other side of core, in advance.

For example, typical usage of DSP, mp3 decoding,

Quoted 8 lines

1, ARM maps 2 shared memory area(input/output) onto DSP virtual
address space.
2, ARM fills mp3 data in input buffer.
3, ARM notifies DSP that data is ready in input buffer. 4, DSP decodes input data and put output data on output buffer. 5, DSP notifies that output buffer is ready.

Roughly something like the above. DSP S/W is multi-tasking, though.

Here, ARM side process talks to DSP site process in their own way, and also there are another cores, talking with their own protocols. So I think that at least, protocol part should be pluggable anyway although it doesn't have to be always in userland.

#12 lore raw
In reply to Meador Inge
Quoted 8 lines · Meador Inge

1. Hardware specific bits somewhere under '.../arch/*'. Drivers for the MPIC message registers on Power and OMAP4 mailboxes, for example.
2. A higher level driver under '.../drivers/mailbox/*'. That the pieces in (1) would register with. This piece would expose the main kernel API.
3. Userspace interfaces for accessing the mailboxes. A '/dev/mailbox1', '/dev/mailbox2', etc... mapping, for example.

How about using virtio for all of this and having the mailbox as a notification/message passing driver for the virtio backend? There are already virtio console and network drivers that could be useful for the userspace part of it. drivers/virtio/virtio_ring.c might be a good starting point if you thought there was some mileage in this approach.

Jamie

#13 lore raw
Quoted 10 lines · Jamie Iles
Quoted 8 lines · Meador Inge

? ? 1. Hardware specific bits somewhere under '.../arch/*'. ?Drivers ? ? ? ?for the MPIC message registers on Power and OMAP4 mailboxes, for ? ? ? ?example.
? ? 2. A higher level driver under '.../drivers/mailbox/*'. ?That the ? ? ? ?pieces in (1) would register with. ?This piece would expose the ? ? ? ?main kernel API.
? ? 3. Userspace interfaces for accessing the mailboxes. ?A ? ? ? ?'/dev/mailbox1', '/dev/mailbox2', etc... mapping, for example.

Quoted 2 lines

How about using virtio for all of this and having the mailbox as a notification/message passing driver for the virtio backend?

This is exactly what we are doing now, and it looks promising. expect patches soon.

#14 lore raw
Quoted 11 lines · Ohad Ben-Cohen
Quoted 10 lines · Jamie Iles
Quoted 8 lines · Meador Inge

? ? 1. Hardware specific bits somewhere under '.../arch/*'. ?Drivers ? ? ? ?for the MPIC message registers on Power and OMAP4 mailboxes, for ? ? ? ?example.
? ? 2. A higher level driver under '.../drivers/mailbox/*'. ?That the ? ? ? ?pieces in (1) would register with. ?This piece would expose the ? ? ? ?main kernel API.
? ? 3. Userspace interfaces for accessing the mailboxes. ?A ? ? ? ?'/dev/mailbox1', '/dev/mailbox2', etc... mapping, for example.

Quoted 5 lines
Quoted 2 lines

How about using virtio for all of this and having the mailbox as a notification/message passing driver for the virtio backend?

This is exactly what we are doing now, and it looks promising. expect patches soon.

I'll be happy to examine the feasibility of doing a port to mpc83xx as soon as I see the code. :-) I have been using the message registers to create a software "network card" over PCI (between a host system and an mpc83xx in a PCI slot). I have wanted to use virtio for this task for a long time.

I think a uniform interface for the mailbox registers would be a very useful API.

Thanks,
Ira

#9 lore raw
In reply to Linus Walleij
Quoted 12 lines · Linus Walleij
Quoted 2 lines
Quoted 2 lines

3. Userspace interfaces for accessing the mailboxes. A '/dev/mailbox1', '/dev/mailbox2', etc... mapping, for example.

What kind of business does userspace have with directly using mailboxes? Enlighten me so I get it... in our system these are used by protocols, such as net/caif/* thru drivers/net/caif/*, and we have similar kernelspace functionality for Phonet.

CAIF and Phonet on the other hand, have custom openings down to the thing that exists on the other end of the mailbox. Most of these systems tend to talk some funny protocol that is often better handled by the kernel than by any userspace.

Quoted 5 lines

So is this for the situation when you have no intermediate protocol between your userpace and the other CPU's subsystem? Or are you thinking about handling that protocol in userspace? That is generally not such a good idea for efficiency reasons.

OpenMCAPI (http://openmcapi.org) implements the MCAPI specification, which is a simple application-level communication API that uses shared memory. The API could be layered over any protocol, but was more or less designed for simple shared-memory systems, e.g. fixed topology, no retransmission, etc.

Currently, we implement almost all of this as a shared library, plus a very small kernel driver. The only requirements on the kernel are to allow userspace to map the shared memory area, and provide an IPI mechanism (and allow the process to sleep while waiting). Applications sync with each other using normal atomic memory operations.

We're now trying to optimize the transfer of scalars on platforms that provide mailboxes (beyond simple IPIs), which is why we're looking at defining a user-facing API to such hardware.

I'll add that we haven't done serious optimization yet, but the numbers we do have seem reasonable. What are the "efficiency" issues you're worried about?

Hollis Blanchard
Mentor Graphics, Embedded Systems Division

#15 lore raw
In reply to Jamie Iles
Quoted 10 lines · Jamie Iles
Quoted 8 lines · Meador Inge

1. Hardware specific bits somewhere under '.../arch/*'. Drivers for the MPIC message registers on Power and OMAP4 mailboxes, for example.
2. A higher level driver under '.../drivers/mailbox/*'. That the pieces in (1) would register with. This piece would expose the main kernel API.
3. Userspace interfaces for accessing the mailboxes. A '/dev/mailbox1', '/dev/mailbox2', etc... mapping, for example.

Quoted 5 lines

How about using virtio for all of this and having the mailbox as a notification/message passing driver for the virtio backend? There are already virtio console and network drivers that could be useful for the userspace part of it. drivers/virtio/virtio_ring.c might be a good starting point if you thought there was some mileage in this approach.

To be honest, I am not that familiar with 'virtio', but I will take a look. Thanks for the pointer. Maybe Hollis can speak to this idea more.

Quoted 2 lines

Jamie

Signature · 2 lines
Meador Inge     | meador_inge AT mentor.com
Mentor Embedded | http://www.mentor.com/embedded-software
#16 lore raw
Quoted 12 lines · Meador Inge
Quoted 11 lines · Jamie Iles
Quoted 9 lines · Meador Inge

1. Hardware specific bits somewhere under '.../arch/*'. Drivers for the MPIC message registers on Power and OMAP4 mailboxes,
for
example.
2. A higher level driver under '.../drivers/mailbox/*'. That the pieces in (1) would register with. This piece would expose the main kernel API.
3. Userspace interfaces for accessing the mailboxes. A '/dev/mailbox1', '/dev/mailbox2', etc... mapping, for example.

Quoted 8 lines
Quoted 5 lines

How about using virtio for all of this and having the mailbox as a notification/message passing driver for the virtio backend? There are already virtio console and network drivers that could be useful for the userspace part of it. drivers/virtio/virtio_ring.c might be a good starting point if you thought there was some mileage in this approach.

To be honest, I am not that familiar with 'virtio', but I will take a look. Thanks for the pointer. Maybe Hollis can speak to this idea more.

My opinion is that virtio is (over?) complicated.

I've looked into it in the past, and I'm definitely open to using it if somebody can demonstrate how easy it is, but adopting it wouldn't have helped OpenMCAPI with our use cases, and would have incurred extra pain, so we didn't.

Hollis Blanchard
Mentor Graphics, Embedded Systems Division

#17 lore raw

From: "ext Blanchard, Hollis" <[email protected]>
Subject: Re: [RFC] Inter-processor Mailboxes Drivers
Date: Tue, 15 Feb 2011 15:38:25 -0800

Quoted 19 lines · Meador Inge
Quoted 18 lines · Jamie Iles
Quoted 16 lines · Meador Inge
Quoted 9 lines

1. Hardware specific bits somewhere under '.../arch/*'. Drivers for the MPIC message registers on Power and OMAP4 mailboxes,
for
example.
2. A higher level driver under '.../drivers/mailbox/*'. That the pieces in (1) would register with. This piece would expose the main kernel API.
3. Userspace interfaces for accessing the mailboxes. A '/dev/mailbox1', '/dev/mailbox2', etc... mapping, for example.

How about using virtio for all of this and having the mailbox as a notification/message passing driver for the virtio backend? There are already virtio console and network drivers that could be useful for the userspace part of it. drivers/virtio/virtio_ring.c might be a good starting point if you thought there was some mileage in this approach.

Quoted 3 lines
Quoted 2 lines

To be honest, I am not that familiar with 'virtio', but I will take a look. Thanks for the pointer. Maybe Hollis can speak to this idea more.

My opinion is that virtio is (over?) complicated.

Considering the case of omap mailbox H/W, it is just a simple one way 4 slot x 32bit H/W FIFO, I also may think that this may be a bit too much...

Quoted 8 lines

I've looked into it in the past, and I'm definitely open to using it if somebody can demonstrate how easy it is, but adopting it wouldn't have helped OpenMCAPI with our use cases, and would have incurred extra pain, so we didn't.

Hollis Blanchard
Mentor Graphics, Embedded Systems Division

#18 lore raw
Quoted 21 lines · Hiroshi DOYU

From: "ext Blanchard, Hollis"<[email protected]>
Subject: Re: [RFC] Inter-processor Mailboxes Drivers
Date: Tue, 15 Feb 2011 15:38:25 -0800

Quoted 17 lines · Meador Inge
Quoted 16 lines · Jamie Iles
Quoted 15 lines · Meador Inge
Quoted 9 lines

1. Hardware specific bits somewhere under '.../arch/*'. Drivers for the MPIC message registers on Power and OMAP4 mailboxes,
for
example.
2. A higher level driver under '.../drivers/mailbox/*'. That the pieces in (1) would register with. This piece would expose the main kernel API.
3. Userspace interfaces for accessing the mailboxes. A '/dev/mailbox1', '/dev/mailbox2', etc... mapping, for example.

How about using virtio for all of this and having the mailbox as a notification/message passing driver for the virtio backend? There are already virtio console and network drivers that could be useful for the userspace part of it. drivers/virtio/virtio_ring.c might be a good starting point if you thought there was some mileage in this approach.

Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

To be honest, I am not that familiar with 'virtio', but I will take a look. Thanks for the pointer. Maybe Hollis can speak to this idea more.

My opinion is that virtio is (over?) complicated.

Considering the case of omap mailbox H/W, it is just a simple one way 4 slot x 32bit H/W FIFO, I also may think that this may be a bit too much...

I think the proposal is to implement a virtio link using the OMAP mailboxes as the interrupt mechanism, and shared memory to carry the data and descriptor rings.

Hollis Blanchard
Mentor Graphics, Embedded Systems Division

#10 lore raw
In reply to Blanchard, Hollis

2011/2/15 Blanchard, Hollis <[email protected]>:

Quoted 5 lines

OpenMCAPI (http://openmcapi.org) implements the MCAPI specification, which is a simple application-level communication API that uses shared memory. The API could be layered over any protocol, but was more or less designed for simple shared-memory systems, e.g. fixed topology, no retransmission, etc.

Cool...

Quoted 5 lines

Currently, we implement almost all of this as a shared library, plus a very small kernel driver. The only requirements on the kernel are to allow userspace to map the shared memory area, and provide an IPI mechanism (and allow the process to sleep while waiting). Applications sync with each other using normal atomic memory operations.

Can't this real small kernel driver take care of the mailbox business as well?

It seems a bit backward if you have say /dev/mcapi0, /dev/mcapi1 etc (or however you expose this to userspace) and /dev/mailbox0 /dev/mailbox1 etc on top of that. One device node per communication channel instead of this would certainly be nicer? Then you would have some ioctl() on the /dev/mcapi0 etc node to trigger the transport and need not worry that it's a mailbox doing the sync.

What I'm after is that whatever datapath you have should include the control mechanism, now it's like you're opening two interfaces into the kernel, one for mapping in data pages, one for synchronizing the transfers, or am I getting things wrong?

I think nominally all mailbox users would be in-kernel like the MCAPI driver, so they don't need a userspace interface, to me it feels like say /dev/mutex0, /dev/mutex1 for some other shared memory opening into the kernel (such as the framebuffer), and that would look a bit funny.

Quoted 3 lines

I'll add that we haven't done serious optimization yet, but the numbers we do have seem reasonable. What are the "efficiency" issues you're worried about?

For huge data flows I think you may get into trouble, needing things like queueing, descriptor pools etc. But if you're convinced this will work, do go ahead.

Linus Walleij