Skip to contentSkip to conversation

RFC: Low-latency SDIO

PATCH In review Christer Weinigel 2 participants 6 messages Sep 30, 2008 – Oct 2, 2008 MMC, SD and SDIO

lore
Share & tools
lore
#0 lore raw

cover RFC: Low-latency SDIO

Hi Pierre and anyone else who is interested,

Here is a cleaned up patch that implements the low-latency SDIO stuff I mailed about a couple of weeks ago. Basically the patches adds asynchronous SDIO operations and makes it possible to register a "hard" SDIO interrupt handler which will be called directly from interrupt context (mmc_signal_sdio_irq).

So, tell me what you think. :-)

/Christer

Signature · 3 lines
"Just how much can I get away with and still go to heaven?"

Christer Weinigel <[email protected]>  http://www.weinigel.se
#1 lore raw

PATCH 1/3 RFC: Low-latency SDIO, add asynchronous SDIO operations

To improve the latency of the SDIO operations, I've add asynchronous versions of sdio operations that split a blocking sdio call into a _start call which takes a callback and a _result function which gets the result from the last operation. The callback is called directly from interrupt context so this avoids the scheduling overhead from waking up the calling thread all the time. This makes a noticeable difference in latency and CPU load on slow embedded systems.

To begin with the patch extracts the guts of mmc_io_rw_direct into two functions, mmc_io_rw_direct_start and mmc_io_rw_direct_result which can be called separately. mmc_io_rw_direct is still there but delegates most of its work to the above functions, and becomes:

    mmc_io_rw_direct_start(..., mmc_io_done, &complete);
    wait_for_completion(&complete);
    return mmc_io_rw_direct_result(card, out);

To support the above mmc_start_request has to be exported and since the function can be called directly I moved the host->claimed check into that function.

A similar refactoring is done for most sdio_ops functions, e.g. for sdio_readb I've added sdio_readb_start and sdio_readb_result while keeping the original function unmodified.

A lot of state had to be moved from local variables into the mmc_card structure.

With the above changes, a driver which used to run in a worker thread and did a blocking call to sdio_readb:

    void foo() 
    {
        value = sdio_readb(func, addr, &err);
    }

can now be writen as:

    void foo_done(struct sdio_func *func);

    void foo()
    {
         sdio_readb_start(func, addr, foo_done);
    }

    void foo_done(struct sdio_func *func)
    {
        value = sdio_readb_result(func, &err);
    }

This is a lot more complex, but the advantage is that since it won't sleep, all of this can be done directly from interrupt context which can give much better latency and a bit lower CPU load by avoiding the thread scheduling. Non latency sensitive applications can still use the blocking calls of course.

This work is done for my employer, CSR.

/Christer

drivers/mmc/core/sdio_ops.c +105 -57
Index: linux-2.6.26.2/drivers/mmc/core/sdio_ops.c
===================================================================
@@ -67,110 +67,158 @@ int mmc_send_io_op_cond(struct mmc_host 
 	return err;
 }
 
-int mmc_io_rw_direct(struct mmc_card *card, int write, unsigned fn,
-	unsigned addr, u8 in, u8* out)
+static void mmc_io_done(struct mmc_request *mrq)
 {
-	struct mmc_command cmd;
-	int err;
+	complete(mrq->done_data);
+}
 
+void mmc_io_rw_direct_start(struct mmc_card *card, int write, unsigned fn,
+			    unsigned addr, u8 in, bool want_out,
+			    void (*done)(struct mmc_request *),
+			    void *done_data)
+{
 	BUG_ON(!card);
 	BUG_ON(fn > 7);
 
-	memset(&cmd, 0, sizeof(struct mmc_command));
+	memset(&card->cmd, 0, sizeof(struct mmc_command));
+
+	card->cmd.opcode = SD_IO_RW_DIRECT;
+	card->cmd.arg = write ? 0x80000000 : 0x00000000;
+	card->cmd.arg |= fn << 28;
+	card->cmd.arg |= (write && want_out) ? 0x08000000 : 0x00000000;
+	card->cmd.arg |= addr << 9;
+	card->cmd.arg |= in;
+	card->cmd.flags = MMC_RSP_SPI_R5 | MMC_RSP_R5 | MMC_CMD_AC;
+
+	memset(card->cmd.resp, 0, sizeof(card->cmd.resp));
+	card->cmd.retries = 0;
+
+	memset(&card->mrq, 0, sizeof(struct mmc_request));
+	card->mrq.cmd = &card->cmd;
+	card->cmd.data = NULL;
+
+	card->mrq.done_data = done_data;
+	card->mrq.done = done;
+
+	mmc_start_request(card->host, &card->mrq);
+}
 
-	cmd.opcode = SD_IO_RW_DIRECT;
-	cmd.arg = write ? 0x80000000 : 0x00000000;
-	cmd.arg |= fn << 28;
-	cmd.arg |= (write && out) ? 0x08000000 : 0x00000000;
-	cmd.arg |= addr << 9;
-	cmd.arg |= in;
-	cmd.flags = MMC_RSP_SPI_R5 | MMC_RSP_R5 | MMC_CMD_AC;
-
-	err = mmc_wait_for_cmd(card->host, &cmd, 0);
-	if (err)
-		return err;
+int mmc_io_rw_direct_result(struct mmc_card *card, u8 *out)
+{
+	if (card->cmd.error)
+		return card->cmd.error;
 
 	if (mmc_host_is_spi(card->host)) {
 		/* host driver already reported errors */
 	} else {
-		if (cmd.resp[0] & R5_ERROR)
+		if (card->cmd.resp[0] & R5_ERROR)
 			return -EIO;
-		if (cmd.resp[0] & R5_FUNCTION_NUMBER)
+		if (card->cmd.resp[0] & R5_FUNCTION_NUMBER)
 			return -EINVAL;
-		if (cmd.resp[0] & R5_OUT_OF_RANGE)
+		if (card->cmd.resp[0] & R5_OUT_OF_RANGE)
 			return -ERANGE;
 	}
 
 	if (out) {
 		if (mmc_host_is_spi(card->host))
-			*out = (cmd.resp[0] >> 8) & 0xFF;
+			*out = (card->cmd.resp[0] >> 8) & 0xFF;
 		else
-			*out = cmd.resp[0] & 0xFF;
+			*out = card->cmd.resp[0] & 0xFF;
 	}
 
 	return 0;
 }
 
-int mmc_io_rw_extended(struct mmc_card *card, int write, unsigned fn,
-	unsigned addr, int incr_addr, u8 *buf, unsigned blocks, unsigned blksz)
+int mmc_io_rw_direct(struct mmc_card *card, int write, unsigned fn,
+	unsigned addr, u8 in, u8 *out)
 {
-	struct mmc_request mrq;
-	struct mmc_command cmd;
-	struct mmc_data data;
-	struct scatterlist sg;
+	DECLARE_COMPLETION_ONSTACK(complete);
 
+	mmc_io_rw_direct_start(card, write, fn, addr, in, out,
+			       mmc_io_done, &complete);
+
+	wait_for_completion(&complete);
+
+	return mmc_io_rw_direct_result(card, out);
+}
+
+void mmc_io_rw_extended_start(struct mmc_card *card, int write, unsigned fn,
+	unsigned addr, int incr_addr, u8 *buf, unsigned blocks, unsigned blksz,
+	void (*done)(struct mmc_request *), void *done_data)
+{
 	BUG_ON(!card);
 	BUG_ON(fn > 7);
 	BUG_ON(blocks == 1 && blksz > 512);
 	WARN_ON(blocks == 0);
 	WARN_ON(blksz == 0);
 
-	memset(&mrq, 0, sizeof(struct mmc_request));
-	memset(&cmd, 0, sizeof(struct mmc_command));
-	memset(&data, 0, sizeof(struct mmc_data));
-
-	mrq.cmd = &cmd;
-	mrq.data = &data;
-
-	cmd.opcode = SD_IO_RW_EXTENDED;
-	cmd.arg = write ? 0x80000000 : 0x00000000;
-	cmd.arg |= fn << 28;
-	cmd.arg |= incr_addr ? 0x04000000 : 0x00000000;
-	cmd.arg |= addr << 9;
+	memset(&card->mrq, 0, sizeof(struct mmc_request));
+	memset(&card->cmd, 0, sizeof(struct mmc_command));
+	memset(&card->data, 0, sizeof(struct mmc_data));
+
+	card->mrq.cmd = &card->cmd;
+	card->mrq.data = &card->data;
+
+	card->cmd.opcode = SD_IO_RW_EXTENDED;
+	card->cmd.arg = write ? 0x80000000 : 0x00000000;
+	card->cmd.arg |= fn << 28;
+	card->cmd.arg |= incr_addr ? 0x04000000 : 0x00000000;
+	card->cmd.arg |= addr << 9;
 	if (blocks == 1 && blksz <= 512)
-		cmd.arg |= (blksz == 512) ? 0 : blksz;	/* byte mode */
+		card->cmd.arg |= (blksz == 512) ? 0 : blksz;	/* byte mode */
 	else
-		cmd.arg |= 0x08000000 | blocks;		/* block mode */
-	cmd.flags = MMC_RSP_SPI_R5 | MMC_RSP_R5 | MMC_CMD_ADTC;
+		card->cmd.arg |= 0x08000000 | blocks;		/* block mode */
+	card->cmd.flags = MMC_RSP_SPI_R5 | MMC_RSP_R5 | MMC_CMD_ADTC;
 
-	data.blksz = blksz;
-	data.blocks = blocks;
-	data.flags = write ? MMC_DATA_WRITE : MMC_DATA_READ;
-	data.sg = &sg;
-	data.sg_len = 1;
+	card->data.blksz = blksz;
+	card->data.blocks = blocks;
+	card->data.flags = write ? MMC_DATA_WRITE : MMC_DATA_READ;
+	card->data.sg = &card->sg;
+	card->data.sg_len = 1;
 
-	sg_init_one(&sg, buf, blksz * blocks);
+	sg_init_one(&card->sg, buf, blksz * blocks);
 
-	mmc_set_data_timeout(&data, card);
+	mmc_set_data_timeout(&card->data, card);
 
-	mmc_wait_for_req(card->host, &mrq);
+	card->mrq.done_data = done_data;
+	card->mrq.done = done;
 
-	if (cmd.error)
-		return cmd.error;
-	if (data.error)
-		return data.error;
+	mmc_start_request(card->host, &card->mrq);
+}
+
+// TODO this can be merged with mmc_io_rw_direct_result
+int mmc_io_rw_extended_result(struct mmc_card *card)
+{
+	if (card->cmd.error)
+		return card->cmd.error;
+	if (card->data.error)
+		return card->data.error;
 
 	if (mmc_host_is_spi(card->host)) {
 		/* host driver already reported errors */
 	} else {
-		if (cmd.resp[0] & R5_ERROR)
+		if (card->cmd.resp[0] & R5_ERROR)
 			return -EIO;
-		if (cmd.resp[0] & R5_FUNCTION_NUMBER)
+		if (card->cmd.resp[0] & R5_FUNCTION_NUMBER)
 			return -EINVAL;
-		if (cmd.resp[0] & R5_OUT_OF_RANGE)
+		if (card->cmd.resp[0] & R5_OUT_OF_RANGE)
 			return -ERANGE;
 	}
 
 	return 0;
 }
 
+int mmc_io_rw_extended(struct mmc_card *card, int write, unsigned fn,
+	unsigned addr, int incr_addr, u8 *buf, unsigned blocks, unsigned blksz)
+{
+	DECLARE_COMPLETION_ONSTACK(complete);
+
+	mmc_io_rw_extended_start(card, write, fn, addr, incr_addr,
+				 buf, blocks, blksz,
+				 mmc_io_done, &complete);
+
+	wait_for_completion(&complete);
+
+	return mmc_io_rw_extended_result(card);
+}
+
drivers/mmc/core/core.c +4 -5
Index: linux-2.6.26.2/drivers/mmc/core/core.c
===================================================================
@@ -113,16 +113,16 @@ void mmc_request_done(struct mmc_host *h
 			mrq->done(mrq);
 	}
 }
-
 EXPORT_SYMBOL(mmc_request_done);
 
-static void
-mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
+void mmc_start_request(struct mmc_host *host, struct mmc_request *mrq)
 {
 #ifdef CONFIG_MMC_DEBUG
 	unsigned int i, sz;
 #endif
 
+	WARN_ON(!host->claimed);
+
 	pr_debug("%s: starting CMD%u arg %08x flags %08x\n",
 		 mmc_hostname(host), mrq->cmd->opcode,
 		 mrq->cmd->arg, mrq->cmd->flags);
@@ -172,6 +172,7 @@ mmc_start_request(struct mmc_host *host,
 	}
 	host->ops->request(host, mrq);
 }
+EXPORT_SYMBOL(mmc_start_request);
 
 static void mmc_wait_done(struct mmc_request *mrq)
 {
@@ -215,8 +216,6 @@ int mmc_wait_for_cmd(struct mmc_host *ho
 {
 	struct mmc_request mrq;
 
-	WARN_ON(!host->claimed);
-
 	memset(&mrq, 0, sizeof(struct mmc_request));
 
 	memset(cmd->resp, 0, sizeof(cmd->resp));
drivers/mmc/core/sdio_io.c +224 -0

248 lines not on this page · raw message

drivers/mmc/core/sdio_ops.h +9 -0
Index: linux-2.6.26.2/drivers/mmc/core/sdio_ops.h
===================================================================
@@ -15,8 +15,17 @@
 int mmc_send_io_op_cond(struct mmc_host *host, u32 ocr, u32 *rocr);
 int mmc_io_rw_direct(struct mmc_card *card, int write, unsigned fn,
 	unsigned addr, u8 in, u8* out);
+void mmc_io_rw_direct_start(struct mmc_card *card, int write, unsigned fn,
+			    unsigned addr, u8 in, bool want_out,
+			    void (*done)(struct mmc_request *),
+			    void *done_data);
+int mmc_io_rw_direct_result(struct mmc_card *card, u8 *out);
 int mmc_io_rw_extended(struct mmc_card *card, int write, unsigned fn,
 	unsigned addr, int incr_addr, u8 *buf, unsigned blocks, unsigned blksz);
+void mmc_io_rw_extended_start(struct mmc_card *card, int write, unsigned fn,
+	unsigned addr, int incr_addr, u8 *buf, unsigned blocks, unsigned blksz,
+	void (*done)(struct mmc_request *), void *done_data);
+int mmc_io_rw_extended_result(struct mmc_card *card);
 
 #endif
 
include/linux/mmc/core.h +1 -0
Index: linux-2.6.26.2/include/linux/mmc/core.h
===================================================================
@@ -129,6 +129,7 @@ struct mmc_request {
 struct mmc_host;
 struct mmc_card;
 
+extern void mmc_start_request(struct mmc_host *host, struct mmc_request *mrq);
 extern void mmc_wait_for_req(struct mmc_host *, struct mmc_request *);
 extern int mmc_wait_for_cmd(struct mmc_host *, struct mmc_command *, int);
 extern int mmc_wait_for_app_cmd(struct mmc_host *, struct mmc_card *,
include/linux/mmc/sdio_func.h +12 -0
Index: linux-2.6.26.2/include/linux/mmc/sdio_func.h
===================================================================
@@ -122,18 +122,30 @@ extern int sdio_release_irq(struct sdio_
 
 extern unsigned char sdio_readb(struct sdio_func *func,
 	unsigned int addr, int *err_ret);
+extern void sdio_readb_start(struct sdio_func *func, unsigned int addr,
+	void (*done)(struct sdio_func *func));
+extern unsigned char sdio_readb_result(struct sdio_func *func, int *err_ret);
 extern unsigned short sdio_readw(struct sdio_func *func,
 	unsigned int addr, int *err_ret);
 extern unsigned long sdio_readl(struct sdio_func *func,
 	unsigned int addr, int *err_ret);
 
+extern void sdio_block_io_result(struct sdio_func *func, int *err_ret);
+
 extern int sdio_memcpy_fromio(struct sdio_func *func, void *dst,
 	unsigned int addr, int count);
 extern int sdio_readsb(struct sdio_func *func, void *dst,
 	unsigned int addr, int count);
+extern void sdio_readsb_start(struct sdio_func *func, void *dst,
+	unsigned int addr, int count,
+	void (*done)(struct sdio_func *func));
 
 extern void sdio_writeb(struct sdio_func *func, unsigned char b,
 	unsigned int addr, int *err_ret);
+extern void sdio_writeb_start(struct sdio_func *func,
+	unsigned char b, unsigned int addr,
+	void (*done)(struct sdio_func *func));
+void sdio_writeb_result(struct sdio_func *func, int *err_ret);
 extern void sdio_writew(struct sdio_func *func, unsigned short b,
 	unsigned int addr, int *err_ret);
 extern void sdio_writel(struct sdio_func *func, unsigned long b,
include/linux/mmc/card.h +16 -0
Index: linux-2.6.26.2/include/linux/mmc/card.h
===================================================================
@@ -10,6 +10,8 @@
 #ifndef LINUX_MMC_CARD_H
 #define LINUX_MMC_CARD_H
 
+#include <linux/scatterlist.h>
+
 #include <linux/mmc/core.h>
 
 struct mmc_cid {
@@ -111,6 +113,20 @@ struct mmc_card {
 	unsigned		num_info;	/* number of info strings */
 	const char		**info;		/* info strings */
 	struct sdio_func_tuple	*tuples;	/* unknown common tuples */
+
+	/* state used internally by asynchronous operations */
+	struct mmc_command	cmd;		/* current command */
+	struct mmc_request	mrq;		/* current request */
+	struct mmc_data		data;		/* data for current
+						 * request */
+	struct scatterlist	sg;		/*  */
+	void			(*done)(struct sdio_func *func); /* called when request done */
+	unsigned		remainder;	/* amount of data left for block operation */
+	int			write;		/* true if this is a write operation */
+	unsigned		addr;		/* address for operation */
+	int			incr_addr;	/* true if a block operation should increment the address */
+	u8			*buf;		/* the buffer for a block operation */
+	unsigned		size;		/* the size of a block operation */
 };
 
 #define mmc_card_mmc(c)		((c)->type == MMC_TYPE_MMC)
Signature · 3 lines
"Just how much can I get away with and still go to heaven?"

Christer Weinigel <[email protected]>  http://www.weinigel.se
#2 lore raw

PATCH 2/3 RFC: Low-latency SDIO, convert sdio_irq_thread to workqueue.

In reply to Christer Weinigel

Change sdio_irq_thread into a workqueue. The polling logic is moved to delayed_work which runs periodically.

There are two major changes in behaviour from the thread version:

First, mmc_signal_sdio_irq will use an asynchronous operation to read SDIO_CCCR_INTx, and the asynchronous callback then schedules sdio_irq_work to do the processing. Since this function can be called from interrupt context, the request function in the SDIO host driver can no longer sleep, something that it could get away with before when all calls into the SDIO layer were made from a thread context.

Second, I disable interrupts in mmc_claim_host (and enable them from mmc_release_host). Since the first thing mmc_signal_sdio_irq does is mmc_try_claim_host, it can't do anything if someone else has claimed the host anyway. I don't think this should break anything.

Existing SDIO function drivers should still work as before and should not notice any difference.

This work is done for my employer, CSR.

drivers/mmc/core/sdio_irq.c +132 -89
Index: linux-2.6.26.2/drivers/mmc/core/sdio_irq.c
===================================================================
@@ -28,18 +28,10 @@
 static int process_sdio_pending_irqs(struct mmc_card *card)
 {
 	int i, ret, count;
-	unsigned char pending;
-
-	ret = mmc_io_rw_direct(card, 0, 0, SDIO_CCCR_INTx, 0, &pending);
-	if (ret) {
-		printk(KERN_DEBUG "%s: error %d reading SDIO_CCCR_INTx\n",
-		       mmc_card_id(card), ret);
-		return ret;
-	}
 
 	count = 0;
 	for (i = 1; i <= 7; i++) {
-		if (pending & (1 << i)) {
+		if (card->pending_irqs & (1 << i)) {
 			struct sdio_func *func = card->sdio_func[i - 1];
 			if (!func) {
 				printk(KERN_WARNING "%s: pending IRQ for "
@@ -47,6 +39,7 @@ static int process_sdio_pending_irqs(str
 					mmc_card_id(card));
 				ret = -EINVAL;
 			} else if (func->irq_handler) {
+				card->pending_irqs &= ~(1 << i);
 				func->irq_handler(func);
 				count++;
 			} else {
@@ -63,86 +56,29 @@ static int process_sdio_pending_irqs(str
 	return ret;
 }
 
-static int sdio_irq_thread(void *_host)
+/* Set realtime priority on the workqueue thread. */
+static void sdio_irq_sched_work(struct work_struct *work)
 {
-	struct mmc_host *host = _host;
 	struct sched_param param = { .sched_priority = 1 };
-	unsigned long period, idle_period;
-	int ret;
 
 	sched_setscheduler(current, SCHED_FIFO, &param);
+}
 
-	/*
-	 * We want to allow for SDIO cards to work even on non SDIO
-	 * aware hosts.  One thing that non SDIO host cannot do is
-	 * asynchronous notification of pending SDIO card interrupts
-	 * hence we poll for them in that case.
-	 */
-	idle_period = msecs_to_jiffies(10);
-	period = (host->caps & MMC_CAP_SDIO_IRQ) ?
-		MAX_SCHEDULE_TIMEOUT : idle_period;
-
-	pr_debug("%s: IRQ thread started (poll period = %lu jiffies)\n",
-		 mmc_hostname(host), period);
-
-	do {
-		/*
-		 * We claim the host here on drivers behalf for a couple
-		 * reasons:
-		 *
-		 * 1) it is already needed to retrieve the CCCR_INTx;
-		 * 2) we want the driver(s) to clear the IRQ condition ASAP;
-		 * 3) we need to control the abort condition locally.
-		 *
-		 * Just like traditional hard IRQ handlers, we expect SDIO
-		 * IRQ handlers to be quick and to the point, so that the
-		 * holding of the host lock does not cover too much work
-		 * that doesn't require that lock to be held.
-		 */
-		ret = __mmc_claim_host(host, &host->sdio_irq_thread_abort);
-		if (ret)
-			break;
-		ret = process_sdio_pending_irqs(host->card);
-		mmc_release_host(host);
-
-		/*
-		 * Give other threads a chance to run in the presence of
-		 * errors.  FIXME: determine if due to card removal and
-		 * possibly exit this thread if so.
-		 */
-		if (ret < 0)
-			ssleep(1);
-
-		/*
-		 * Adaptive polling frequency based on the assumption
-		 * that an interrupt will be closely followed by more.
-		 * This has a substantial benefit for network devices.
-		 */
-		if (!(host->caps & MMC_CAP_SDIO_IRQ)) {
-			if (ret > 0)
-				period /= 2;
-			else {
-				period++;
-				if (period > idle_period)
-					period = idle_period;
-			}
-		}
+static void sdio_irq_work(struct work_struct *work)
+{
+	struct mmc_host *host = container_of(work, struct mmc_host, irq_work);
 
-		set_current_state(TASK_INTERRUPTIBLE);
-		if (host->caps & MMC_CAP_SDIO_IRQ)
-			host->ops->enable_sdio_irq(host, 1);
-		if (!kthread_should_stop())
-			schedule_timeout(period);
-		set_current_state(TASK_RUNNING);
-	} while (!kthread_should_stop());
+	process_sdio_pending_irqs(host->card);
 
-	if (host->caps & MMC_CAP_SDIO_IRQ)
-		host->ops->enable_sdio_irq(host, 0);
+	mmc_release_host(host);
+}
 
-	pr_debug("%s: IRQ thread exiting with code %d\n",
-		 mmc_hostname(host), ret);
+static void sdio_irq_poll_work(struct work_struct *work)
+{
+	struct mmc_host *host = container_of(work, struct mmc_host,
+					     irq_poll_work.work);
 
-	return ret;
+	mmc_signal_sdio_irq(host);
 }
 
 static int sdio_card_irq_get(struct mmc_card *card)
@@ -152,14 +88,21 @@ static int sdio_card_irq_get(struct mmc_
 	WARN_ON(!host->claimed);
 
 	if (!host->sdio_irqs++) {
-		atomic_set(&host->sdio_irq_thread_abort, 0);
-		host->sdio_irq_thread =
-			kthread_run(sdio_irq_thread, host, "ksdiorqd");
-		if (IS_ERR(host->sdio_irq_thread)) {
-			int err = PTR_ERR(host->sdio_irq_thread);
+		host->irq_workqueue =
+			create_singlethread_workqueue("ksdiorqd");
+		if (!host->irq_workqueue) {
 			host->sdio_irqs--;
-			return err;
+			return -ENOMEM;
 		}
+
+		INIT_WORK(&host->irq_sched_work, sdio_irq_sched_work);
+		INIT_WORK(&host->irq_work, sdio_irq_work);
+		INIT_DELAYED_WORK(&host->irq_poll_work,
+				  sdio_irq_poll_work);
+
+		queue_work(host->irq_workqueue, &host->irq_sched_work);
+
+		host->irq_poll_period = msecs_to_jiffies(10);
 	}
 
 	return 0;
@@ -172,14 +115,114 @@ static int sdio_card_irq_put(struct mmc_
 	WARN_ON(!host->claimed);
 	BUG_ON(host->sdio_irqs < 1);
 
-	if (!--host->sdio_irqs) {
-		atomic_set(&host->sdio_irq_thread_abort, 1);
-		kthread_stop(host->sdio_irq_thread);
+	if (!--host->sdio_irqs)
+		destroy_workqueue(host->irq_workqueue);
+
+	return 0;
+}
+
+void sdio_enable_irq(struct mmc_host *host)
+{
+	if (host->sdio_irqs) {
+		if (host->caps & MMC_CAP_SDIO_IRQ)
+			host->ops->enable_sdio_irq(host, 1);
+		else {
+			queue_delayed_work(host->irq_workqueue,
+					   &host->irq_poll_work,
+					   host->irq_poll_period);
+		}
+	}
+}
+
+void sdio_disable_irq(struct mmc_host *host)
+{
+	if (host->sdio_irqs) {
+		if (host->caps & MMC_CAP_SDIO_IRQ)
+			host->ops->enable_sdio_irq(host, 0);
+		else
+			cancel_delayed_work(&host->irq_poll_work);
+	}
+}
+
+int sdio_kick_irq(struct mmc_host *host)
+{
+	if (!host->card)
+		return 0;
+
+	if (host->sdio_irqs && host->card->pending_irqs) {
+		queue_work(host->irq_workqueue, &host->irq_work);
+		return 1;
 	}
 
 	return 0;
 }
 
+static void sdio_irq_pending_done(struct mmc_request *mrq);
+
+/**
+ *	mmc_signal_sdio_irq - signal a SDIO IRQ
+ *	@host: SDIO host
+ *
+ *	This function should be called by the MMC host driver when
+ *	it has detected a SDIO IRQ.
+ */
+void mmc_signal_sdio_irq(struct mmc_host *host)
+{
+	/*
+	 * We claim the host here on drivers behalf and don't release
+	 * it until all interrupts are processed.
+	 */
+
+	if (!mmc_try_claim_host(host)) {
+		/* this should only happen if the MMC host
+		 * driver signals SDIO interrupts even if they
+		 * are disabled  */
+		pr_debug("%s: card already claimed\n", __func__);
+		return;
+	}
+
+	mmc_io_rw_direct_start(host->card, 0, 0, SDIO_CCCR_INTx, 0, 1,
+			       sdio_irq_pending_done, host);
+}
+EXPORT_SYMBOL_GPL(mmc_signal_sdio_irq);
+
+/* Called when the asynchronous operation has fetched the pending irqs */
+static void sdio_irq_pending_done(struct mmc_request *mrq)
+{
+	struct mmc_host *host = mrq->done_data;
+	int ret;
+
+	ret = mmc_io_rw_direct_result(host->card, &host->card->pending_irqs);
+	if (ret) {
+		pr_warning("%s: error %d reading SDIO_CCCR_INTx\n",
+			   mmc_card_id(host->card), ret);
+		host->card->pending_irqs = 0;
+		mmc_release_host(host);
+		return;
+	}
+
+	if (!(host->caps & MMC_CAP_SDIO_IRQ)) {
+		unsigned long idle_period = msecs_to_jiffies(100);
+
+		/*
+		 * Adaptive polling frequency based on the assumption
+		 * that an interrupt will be closely followed by more.
+		 * This has a substantial benefit for network devices.
+		 */
+		if (host->card->pending_irqs)
+			host->irq_poll_period /= 2;
+		else {
+			host->irq_poll_period++;
+			if (host->irq_poll_period > idle_period)
+				host->irq_poll_period = idle_period;
+		}
+	}
+
+	/* If pending_irqs is set this this will kick the SDIO irq
+	 * handling, otherwise we'll just release the host. */
+	mmc_release_host(host);
+}
+
 /**
  *	sdio_claim_irq - claim the IRQ for a SDIO function
  *	@func: SDIO function
include/linux/mmc/host.h +6 -8
Index: linux-2.6.26.2/include/linux/mmc/host.h
===================================================================
@@ -128,8 +128,11 @@ struct mmc_host {
 	unsigned int		bus_refs;	/* reference counter */
 
 	unsigned int		sdio_irqs;
-	struct task_struct	*sdio_irq_thread;
-	atomic_t		sdio_irq_thread_abort;
+	struct workqueue_struct *irq_workqueue;
+	struct delayed_work     irq_poll_work;
+	unsigned long           irq_poll_period;
+	struct work_struct      irq_work;
+	struct work_struct      irq_sched_work;
 
 #ifdef CONFIG_LEDS_TRIGGERS
 	struct led_trigger	*led;		/* activity led */
@@ -159,12 +162,7 @@ extern int mmc_resume_host(struct mmc_ho
 
 extern void mmc_detect_change(struct mmc_host *, unsigned long delay);
 extern void mmc_request_done(struct mmc_host *, struct mmc_request *);
-
-static inline void mmc_signal_sdio_irq(struct mmc_host *host)
-{
-	host->ops->enable_sdio_irq(host, 0);
-	wake_up_process(host->sdio_irq_thread);
-}
+extern void mmc_signal_sdio_irq(struct mmc_host *host);
 
 #endif
 
drivers/mmc/core/core.c +30 -0
Index: linux-2.6.26.2/drivers/mmc/core/core.c
===================================================================
@@ -34,6 +34,7 @@
 #include "mmc_ops.h"
 #include "sd_ops.h"
 #include "sdio_ops.h"
+#include "sdio_irq.h"
 
 static struct workqueue_struct *workqueue;
 
@@ -294,6 +295,30 @@ void mmc_set_data_timeout(struct mmc_dat
 EXPORT_SYMBOL(mmc_set_data_timeout);
 
 /**
+ *	mmc_try_claim_host - exclusively claim a host
+ *	@host: mmc host to claim
+ *
+ *	Claim a host for a set of operations.  Returns true with the
+ *	lock held if it acquired the lock or false if it failed.
+ */
+int mmc_try_claim_host(struct mmc_host *host)
+{
+	unsigned long flags;
+	int ret = 0;
+
+	spin_lock_irqsave(&host->lock, flags);
+	if (!host->claimed) {
+		host->claimed = 1;
+		sdio_disable_irq(host);
+		ret = 1;
+	}
+	spin_unlock_irqrestore(&host->lock, flags);
+
+	return ret;
+}
+EXPORT_SYMBOL(mmc_try_claim_host);
+
+/**
  *	__mmc_claim_host - exclusively claim a host
  *	@host: mmc host to claim
  *	@abort: whether or not the operation should be aborted
@@ -322,6 +347,7 @@ int __mmc_claim_host(struct mmc_host *ho
 		schedule();
 		spin_lock_irqsave(&host->lock, flags);
 	}
+	sdio_disable_irq(host);
 	set_current_state(TASK_RUNNING);
 	if (!stop)
 		host->claimed = 1;
@@ -347,7 +373,11 @@ void mmc_release_host(struct mmc_host *h
 
 	WARN_ON(!host->claimed);
 
+	if (sdio_kick_irq(host))
+		return;
+
 	spin_lock_irqsave(&host->lock, flags);
+	sdio_enable_irq(host);
 	host->claimed = 0;
 	spin_unlock_irqrestore(&host->lock, flags);
 
drivers/mmc/core/sdio_irq.h new file +17 -0
Index: linux-2.6.26.2/drivers/mmc/core/sdio_irq.h
===================================================================
@@ -0,0 +1,17 @@
+/*
+ *  linux/drivers/mmc/core/sdio_irq.h
+ *
+ *  Copyright (C) 2008 CSR
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+#ifndef _MMC_CORE_SDIO_IRQ_H
+#define _MMC_CORE_SDIO_IRQ_H
+
+void sdio_enable_irq(struct mmc_host *host);
+void sdio_disable_irq(struct mmc_host *host);
+bool sdio_kick_irq(struct mmc_host *host);
+
+#endif
include/linux/mmc/card.h +2 -0
Index: linux-2.6.26.2/include/linux/mmc/card.h
===================================================================
@@ -127,6 +127,8 @@ struct mmc_card {
 	int			incr_addr;	/* true if a block operation should increment the address */
 	u8			*buf;		/* the buffer for a block operation */
 	unsigned		size;		/* the size of a block operation */
+
+	unsigned char		pending_irqs;	/* pending SDIO interrupts */
 };
 
 #define mmc_card_mmc(c)		((c)->type == MMC_TYPE_MMC)
include/linux/mmc/core.h +1 -0
Index: linux-2.6.26.2/include/linux/mmc/core.h
===================================================================
@@ -137,6 +137,7 @@ extern int mmc_wait_for_app_cmd(struct m
 
 extern void mmc_set_data_timeout(struct mmc_data *, const struct mmc_card *);
 
+extern int mmc_try_claim_host(struct mmc_host *host);
 extern int __mmc_claim_host(struct mmc_host *host, atomic_t *abort);
 extern void mmc_release_host(struct mmc_host *host);
 
Signature · 3 lines
"Just how much can I get away with and still go to heaven?"

Christer Weinigel <[email protected]>  http://www.weinigel.se
#3 lore raw

PATCH 3/3 RFC: Low-latency SDIO, add hard interrupt handlers

In reply to Christer Weinigel

Finally, the goal of all these changes, add a hard SDIO interrupt handler which will be called directly from interrupt context.

Nothing really tricky here, the patch adds a new function sdio_claim_hard_irq and moves the common code from sdio_hard_irq into __sdio_claim_irq. It also adds some code that calls the hard interrupt handler directly from interrupt context.

One very important thing here is that since the hard interrupt handler will return immediately, it has to call sdio_release_host when it is done. The call to sdio_kick_irq do the right thing, either process the next hard interrupt, wake up the soft interrupt workqueue or return 0 which really releases the host.

This work is done for my employer, CSR.

include/linux/mmc/sdio_func.h +6 -1
Index: linux-2.6.26.2/include/linux/mmc/sdio_func.h
===================================================================
@@ -15,6 +15,8 @@
 #include <linux/device.h>
 #include <linux/mod_devicetable.h>
 
+#define SDIO_HAVE_HARD_IRQ 1
+
 struct mmc_card;
 struct sdio_func;
 
@@ -36,7 +38,8 @@ struct sdio_func_tuple {
 struct sdio_func {
 	struct mmc_card		*card;		/* the card this device belongs to */
 	struct device		dev;		/* the device */
-	sdio_irq_handler_t	*irq_handler;	/* IRQ callback */
+	sdio_irq_handler_t	*irq_handler;	/* IRQ thread callback */
+	sdio_irq_handler_t	*hard_irq_handler; /* hard IRQ callback */
 	unsigned int		num;		/* function number */
 
 	unsigned char		class;		/* standard interface class */
@@ -110,6 +113,8 @@ extern void sdio_unregister_driver(struc
  * SDIO I/O operations
  */
 extern void sdio_claim_host(struct sdio_func *func);
+extern int sdio_claim_hard_irq(struct sdio_func *func,
+			       sdio_irq_handler_t *handler);
 extern void sdio_release_host(struct sdio_func *func);
 
 extern int sdio_enable_func(struct sdio_func *func);
drivers/mmc/core/sdio_irq.c +71 -15

138 lines not on this page · raw message

Signature · 3 lines
"Just how much can I get away with and still go to heaven?"

Christer Weinigel <[email protected]>  http://www.weinigel.se
#4 lore raw
In reply to Christer Weinigel
Quoted 2 lines · On Tue, 30 Sep 2008 12:23:13 +0200 Christer Weinigel <[email protected]>…

Hi Pierre and anyone else who is interested,

Hi Christer,

Quoted 8 lines

Here is a cleaned up patch that implements the low-latency SDIO stuff I mailed about a couple of weeks ago. Basically the patches adds asynchronous SDIO operations and makes it possible to register a "hard" SDIO interrupt handler which will be called directly from interrupt context (mmc_signal_sdio_irq).

So, tell me what you think. :-)

Did you see the response I sent to the previous thread?

Rgds

Signature · 9 lines
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  rdesktop, core developer          http://www.rdesktop.org

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.
#5 lore raw
Quoted 9 lines · Pierre Ossman
Quoted 5 lines · On Tue, 30 Sep 2008 12:23:13 +0200 Christer Weinigel <[email protected]>…

Here is a cleaned up patch that implements the low-latency SDIO stuff I mailed about a couple of weeks ago. Basically the patches adds asynchronous SDIO operations and makes it possible to register a "hard" SDIO interrupt handler which will be called directly from interrupt context (mmc_signal_sdio_irq).

Did you see the response I sent to the previous thread?

Yes, i think so. My previous post was mostly handwaving and some claims of a performance improvement, I wanted to clean up my low latency patches first and show a concrete implementation of what I was thinking about before asking for more feedback.

This patch mostly adds things, the asynchronous API is available for for those that need it, but old drivers will still work as before. There is a difference for host drivers, drivers that sleep in their request function will not work, but I believe request shouldn't do that anyway.

Anyway, is this the response you're thinking of:

Quoted 12 lines · Pierre Ossman

The latency improvement is indeed impressive, but I am not convinced it is worth it. An asynchronous API is much more complex and difficult to work with (not to mention reading and trying to make sense of existing code), and SDIO is not even an asynchronous bus to begin with.

I do like the idea of reducing latencies (and generally improving the performance of the MMC stack). The primary reason I haven't done anything myself is lack of stuff to test and proper instrumentation.

There are really two issues here, which aren't necessarily related; actual interrupt latency and command completion latency.

Quoted 8 lines

The main culprit in your case is the command completion one. Perhaps there is some other way of solving that inside the MMC core? E.g. we could spin instead of sleeping while we wait for the request to complete. Most drivers never use process context to handle a request anyway. We would need to determine when the request is small enough (all non-busy, non-data commands?) and that the CPU is slow enough. I also saw something about a new trigger interface that could make this efficient.

There are actually three issues I think, interrupt latency, command completion latency, and CPU load.

Using a busy-wait for command completion would both reduce the load and latency (because there will be no interrupt or wakeup overhead). But it only helps for quick commands, for any commands that transfer data, we'll have to use an interrupt and get the wakeup latencies. And for me
any latency, initial interrupt latency or latency after the data
transfer, are killers, so that's what I'm trying to solve.

I did google a bit for a Linux trigger interface and for some reason (I can't remember exactly why now), I did not think that it would be suitable for what I want to do.

So I still would like to get something like this low-latency thing into the kernel.

/Christer