mmc: mmc: Add Software Command Queuing
Subsystems (2)
v9 cover mmc: mmc: Add Software Command Queuing
Hi
Here is an updated version of the Software Command Queuing patches, re-based on next, with some changes - refer changes in V9 below. The patches have been re-ordered to put tidy-ups first. The first 4 patches could be considered, but especially the first 2.
Performance results (not updated since V5):
Results can vary from run to run, but here are some results showing 1, 2 or 4 processes with 4k and 32k record sizes. They show up to 40% improvement in read performance when there are multiple processes.
iozone -s 8192k -r 4k -i 0 -i 1 -i 2 -i 8 -I -t 1 -F /mnt/mmc/iozone1.tmp
Children see throughput for 1 initial writers = 27909.87 kB/sec 24204.14 kB/sec -13.28 % Children see throughput for 1 rewriters = 28839.28 kB/sec 25531.92 kB/sec -11.47 % Children see throughput for 1 readers = 25889.65 kB/sec 24883.23 kB/sec -3.89 % Children see throughput for 1 re-readers = 25558.23 kB/sec 24679.89 kB/sec -3.44 % Children see throughput for 1 random readers = 25571.48 kB/sec 24689.52 kB/sec -3.45 % Children see throughput for 1 mixed workload = 25758.59 kB/sec 24487.52 kB/sec -4.93 % Children see throughput for 1 random writers = 24787.51 kB/sec 19368.99 kB/sec -21.86 %
iozone -s 8192k -r 32k -i 0 -i 1 -i 2 -i 8 -I -t 1 -F /mnt/mmc/iozone1.tmp
Children see throughput for 1 initial writers = 91344.61 kB/sec 102008.56 kB/sec 11.67 % Children see throughput for 1 rewriters = 87932.36 kB/sec 96630.44 kB/sec 9.89 % Children see throughput for 1 readers = 134879.82 kB/sec 110292.79 kB/sec -18.23 % Children see throughput for 1 re-readers = 147632.13 kB/sec 109053.33 kB/sec -26.13 % Children see throughput for 1 random readers = 93547.37 kB/sec 112225.50 kB/sec 19.97 % Children see throughput for 1 mixed workload = 93560.04 kB/sec 110515.21 kB/sec 18.12 % Children see throughput for 1 random writers = 92841.84 kB/sec 81153.81 kB/sec -12.59 %
iozone -s 8192k -r 4k -i 0 -i 1 -i 2 -i 8 -I -t 2 -F /mnt/mmc/iozone1.tmp /mnt/mmc/iozone2.tmp
Children see throughput for 2 initial writers = 31145.43 kB/sec 33771.25 kB/sec 8.43 % Children see throughput for 2 rewriters = 30592.57 kB/sec 35916.46 kB/sec 17.40 % Children see throughput for 2 readers = 31669.83 kB/sec 37460.13 kB/sec 18.28 % Children see throughput for 2 re-readers = 32079.94 kB/sec 37373.33 kB/sec 16.50 % Children see throughput for 2 random readers = 27731.19 kB/sec 37601.65 kB/sec 35.59 % Children see throughput for 2 mixed workload = 13927.50 kB/sec 14617.06 kB/sec 4.95 % Children see throughput for 2 random writers = 31250.00 kB/sec 33106.72 kB/sec 5.94 %
iozone -s 8192k -r 32k -i 0 -i 1 -i 2 -i 8 -I -t 2 -F /mnt/mmc/iozone1.tmp /mnt/mmc/iozone2.tmp
Children see throughput for 2 initial writers = 123255.84 kB/sec 131252.22 kB/sec 6.49 % Children see throughput for 2 rewriters = 115234.91 kB/sec 107225.74 kB/sec -6.95 % Children see throughput for 2 readers = 128921.86 kB/sec 148562.71 kB/sec 15.23 % Children see throughput for 2 re-readers = 127815.24 kB/sec 149304.32 kB/sec 16.81 % Children see throughput for 2 random readers = 125600.46 kB/sec 148406.56 kB/sec 18.16 % Children see throughput for 2 mixed workload = 44006.94 kB/sec 50937.36 kB/sec 15.75 % Children see throughput for 2 random writers = 120623.95 kB/sec 103969.05 kB/sec -13.81 %
iozone -s 8192k -r 4k -i 0 -i 1 -i 2 -i 8 -I -t 4 -F /mnt/mmc/iozone1.tmp /mnt/mmc/iozone2.tmp /mnt/mmc/iozone3.tmp /mnt/mmc/iozone4.tmp
Children see throughput for 4 initial writers = 24100.96 kB/sec 33336.58 kB/sec 38.32 % Children see throughput for 4 rewriters = 31650.20 kB/sec 33091.53 kB/sec 4.55 % Children see throughput for 4 readers = 33276.92 kB/sec 41799.89 kB/sec 25.61 % Children see throughput for 4 re-readers = 31786.96 kB/sec 41501.74 kB/sec 30.56 % Children see throughput for 4 random readers = 31991.65 kB/sec 40973.93 kB/sec 28.08 % Children see throughput for 4 mixed workload = 15804.80 kB/sec 13581.32 kB/sec -14.07 % Children see throughput for 4 random writers = 31231.42 kB/sec 34537.03 kB/sec 10.58 %
iozone -s 8192k -r 32k -i 0 -i 1 -i 2 -i 8 -I -t 4 -F /mnt/mmc/iozone1.tmp /mnt/mmc/iozone2.tmp /mnt/mmc/iozone3.tmp /mnt/mmc/iozone4.tmp
Children see throughput for 4 initial writers = 116567.42 kB/sec 119280.35 kB/sec 2.33 % Children see throughput for 4 rewriters = 115010.96 kB/sec 120864.34 kB/sec 5.09 % Children see throughput for 4 readers = 130700.29 kB/sec 177834.21 kB/sec 36.06 % Children see throughput for 4 re-readers = 125392.58 kB/sec 175975.28 kB/sec 40.34 % Children see throughput for 4 random readers = 132194.57 kB/sec 176630.46 kB/sec 33.61 % Children see throughput for 4 mixed workload = 56464.98 kB/sec 54140.61 kB/sec -4.12 % Children see throughput for 4 random writers = 109128.36 kB/sec 85359.80 kB/sec -21.78 %
The current block driver supports 2 requests on the go at a time. Patches 3 - 8 make preparations for an arbitrary sized queue. Patches 9 - 12 introduce Command Queue definitions and helpers. Patches 13 - 16 complete the job of making the block driver use a queue. Patches 17 - 20 finally add Software Command Queuing. Most of the Software Command Queuing functionality is added in patch 19.
As noted below, the patches can also be found here:
http://git.infradead.org/users/ahunter/linux-sdhci.git/shortlog/refs/heads/swcmdq
which also includes a debug-only patch to help debug stuck queues:
mmc: block: Add debugfs state file for debugging stuck queues
Changes in V9:
Re-based on next, dropping patches that have been applied:
mmc: block: Restore line inadvertently removed with packed commands
mmc: block: Fix 4K native sector check
mmc: queue: Fix queue thread wake-up
mmc: queue: Factor out mmc_queue_alloc_bounce_bufs()
mmc: queue: Factor out mmc_queue_alloc_bounce_sgs()
mmc: queue: Factor out mmc_queue_alloc_sgs()
mmc: queue: Factor out mmc_queue_reqs_free_bufs()
mmc: queue: Introduce queue depth and use it to allocate and free
mmc: mmc: Add Command Queue definitions
mmc: queue: Share mmc request array between partitions
Change to allocate / free from mmc_blk_probe / mmc_blk_remove instead of using a reference count.
mmc: block: Add Software Command Queuing
Adjusted due to change in "mmc: queue: Share mmc request array between partitions"
Changes in V8:
Re-based on next, dropping references to Packed Commands.
mmc: block: Restore line inadvertently removed with packed commands
New patch
mmc: block: Fix 4K native sector check
Moved to be the 2nd patch
Added Reviewed-by: Linus Walleij
mmc: queue: Fix queue thread wake-up
Added Reviewed-by: Linus Walleij <[email protected]> Added Reviewed-by: Harjani Ritesh <[email protected]>
mmc: queue: Factor out mmc_queue_alloc_bounce_bufs()
Added Reviewed-by: Linus Walleij <[email protected]> Added Reviewed-by: Harjani Ritesh <[email protected]>
mmc: queue: Factor out mmc_queue_alloc_bounce_sgs()
Added Reviewed-by: Linus Walleij <[email protected]> Added Reviewed-by: Harjani Ritesh <[email protected]>
mmc: queue: Factor out mmc_queue_alloc_sgs()
Added Reviewed-by: Linus Walleij <[email protected]> Added Reviewed-by: Harjani Ritesh <[email protected]>
mmc: queue: Factor out mmc_queue_reqs_free_bufs()
Added Reviewed-by: Linus Walleij <[email protected]> Added Reviewed-by: Harjani Ritesh <[email protected]>
mmc: queue: Introduce queue depth
Drop chunk referring to mmc_packed_init().
Combined into new patch "mmc: queue: Introduce queue depth and use it to allocate and free"
mmc: queue: Use queue depth to allocate and free
Combined into new patch "mmc: queue: Introduce queue depth and use it to allocate and free"
mmc: queue: Allocate queue of size qdepth
Combined into new patch "mmc: queue: Introduce queue depth and use it to allocate and free"
mmc: queue: Introduce queue depth and use it to allocate and free
New patch from combining 3 patches above.
mmc: mmc: Add Command Queue definitions
Add comment about excluding qdepths of 1 or 2.
mmc: mmc: Add functions to enable / disable the Command Queue
Change mmc_cmdq_switch() 'enable' parameter from 'int' to 'bool'.
mmc: mmc_test: Disable Command Queue while mmc_test is used
Added Reviewed-by: Harjani Ritesh <[email protected]>
mmc: block: Disable Command Queue while RPMB is used
As per Ritesh, assign 'ret' to 0 and return 'ret'. Added Reviewed-by: Harjani Ritesh <[email protected]>
mmc: core: Do not prepare a new request twice
Added Reviewed-by: Harjani Ritesh <[email protected]>
mmc: core: Export mmc_retune_hold() and mmc_retune_release()
Added Reviewed-by: Harjani Ritesh <[email protected]>
mmc: block: Factor out mmc_blk_requeue()
Dropped because it only affected "packed commands" code.
mmc: block: Use local var for mqrq_cur
Dropped chucks referring to Packed Commands. Added Reviewed-by: Linus Walleij <[email protected]>
mmc: block: Pass mqrq to mmc_blk_prep_packed_list()
Dropped because it only affected "packed commands" code.
mmc: block: Introduce queue semantics
Add mmc_blk_requeue() lost from dropped patches.
mmc: queue: Share mmc request array between partitions
Dropped chucks referring to Packed Commands. Change mqrq_ref_cnt from 'int' to 'unsigned int' Add a comment about synchronisation of mqrq_ref_cnt.
mmc: mmc: Enable Software Command Queuing
Get rid of MMC_CAP_SWCMDQ and use MMC_CAP_CMD_DURING_TFR instead
mmc: sdhci-pci: Enable Software Command Queuing for some Intel controllers
Dropped because MMC_CAP_SWCMDQ removed.
mmc: sdhci-acpi: Enable Software Command Queuing for some Intel controllers
Dropped because MMC_CAP_SWCMDQ removed.
Changes in V7:
Re-based on next.
mmc: mmc: Add Command Queue definitions
Remove cmdq_en flag and add Linus Walleij's Reviewed-by.
mmc: mmc: Add functions to enable / disable the Command
Add cmdq_en flag.
Changes in V6:
mmc: core: Do not prepare a new request twice
Ensure struct mmc_async_req is always initialized to zero
Changes in V5:
Patches 1-5 dropped because they have been applied.
Re-based on next.
Fixed use of blk_end_request_cur() when it should have been blk_end_request_all() to error out requests during error recovery.
Fixed unpaired retune_hold / retune_release in the error recovery path.
Changes in V4:
Re-based on next + v4.8-rc2 + "block: Fix secure erase" patch
Changes in V3:
Patches 1-25 dropped because they have been applied.
Re-based on next.
mmc: queue: Allocate queue of size qdepth
Free queue during cleanup
mmc: mmc: Add Command Queue definitions
Add cmdq_en to mmc-dev-attrs.txt documentation
mmc: queue: Share mmc request array between partitions
New patch
Changes in V2:
Added 5 patches already sent here:
http://marc.info/?l=linux-mmc&m=146712062816835
Added 3 more new patches:
mmc: sdhci-pci: Do not runtime suspend at the end of sdhci_pci_probe()
mmc: sdhci: Avoid STOP cmd triggering warning in sdhci_send_command()
mmc: sdhci: sdhci_execute_tuning() must delete timer
Carried forward the V2 fix to:
mmc: mmc_test: Disable Command Queue while mmc_test is used
Also reset the cmd circuit for data timeout if it is processing the data cmd, in patch:
mmc: sdhci: Do not reset cmd or data circuits that are in use
There wasn't much comment on the RFC so there have been few changes. Venu Byravarasu commented that it may be more efficient to use Software Command Queuing only when there is more than 1 request queued - it isn't obvious how well that would work in practice, but it could be added later if it could be shown to be beneficial.
Original Cover Letter:
Chuanxiao Dong sent some patches last year relating to eMMC 5.1 Software Command Queuing. He did not follow-up but I have contacted him and he says it is OK if I take over upstreaming the patches.
eMMC Command Queuing is a feature added in version 5.1. The card maintains a queue of up to 32 data transfers. Commands CMD45/CMD45 are sent to queue up transfers in advance, and then one of the transfers is selected to "execute" by CMD46/CMD47 at which point data transfer actually begins.
The advantage of command queuing is that the card can prepare for transfers in advance. That makes a big difference in the case of random reads because the card can start reading into its cache in advance.
A v5.1 host controller can manage the command queue itself, but it is also possible for software to manage the queue using an non-v5.1 host controller
- that is what Software Command Queuing is.
Refer to the JEDEC (http://www.jedec.org/) eMMC v5.1 Specification for more information about Command Queuing.
While these patches are heavily based on Dong's patches, there are some changes:
SDHCI has been amended to support commands during transfer. That is a generic change added in patches 1 - 5. [Those patches have now been applied] In principle, that would also support SDIO's CMD52 during data transfer.
The original approach added multiple commands into the same request for sending CMD44, CMD45 and CMD13. That is not strictly necessary and has been omitted for now.
The original approach also called blk_end_request() from the mrq->done() function, which means the upper layers learnt of completed requests slightly earlier. That is not strictly related to Software Command Queuing and is something that could potentially be done for all data requests. That has been omitted for now.
The current block driver supports 2 requests on the go at a time. Patches 1 - 8 make preparations for an arbitrary sized queue. Patches 9 - 12 introduce Command Queue definitions and helpers. Patches 13 - 19 complete the job of making the block driver use a queue. Patches 20 - 23 finally add Software Command Queuing, and 24 - 25 enable it for Intel eMMC controllers. Most of the Software Command Queuing functionality is added in patch 22.
The patches can also be found here:
http://git.infradead.org/users/ahunter/linux-sdhci.git/shortlog/refs/heads/swcmdq
The patches have only had basic testing so far. Ad-hoc testing shows a degradation in sequential read performance of about 10% but an increase in throughput for mixed workload of multiple processes of about 90%. The reduction in sequential performance is due to the need to read the Queue Status register between each transfer.
These patches should not conflict with Hardware Command Queuing which handles the queue in a completely different way and thus does not need to share code with Software Command Queuing. The exceptions being the Command Queue definitions and queue allocation which should be able to be used.
Adrian Hunter (11):
mmc: block: Use local var for mqrq_cur
mmc: queue: Share mmc request array between partitions
mmc: block: Introduce queue semantics
mmc: core: Do not prepare a new request twice
mmc: mmc: Add functions to enable / disable the Command Queue
mmc: mmc_test: Disable Command Queue while mmc_test is used
mmc: block: Disable Command Queue while RPMB is used
mmc: core: Export mmc_retune_hold() and mmc_retune_release()
mmc: queue: Add a function to control wake-up on new requests
mmc: block: Add Software Command Queuing
mmc: mmc: Enable Software Command Queuing
Documentation/mmc/mmc-dev-attrs.txt | 1 + drivers/mmc/card/block.c | 706 ++++++++++++++++++++++++++++++++++-- drivers/mmc/card/mmc_test.c | 21 +- drivers/mmc/card/queue.c | 331 ++++++++++------- drivers/mmc/card/queue.h | 25 +- drivers/mmc/core/core.c | 12 +- drivers/mmc/core/host.c | 2 + drivers/mmc/core/host.h | 2 - drivers/mmc/core/mmc.c | 26 +- drivers/mmc/core/mmc_ops.c | 28 ++ include/linux/mmc/card.h | 7 + include/linux/mmc/core.h | 6 + include/linux/mmc/host.h | 1 + 13 files changed, 999 insertions(+), 169 deletions(-)
Regards
Adrian
PATCH v9 1/11 mmc: block: Use local var for mqrq_cur
A subsequent patch will remove 'mq->mqrq_cur'. Prepare for that by assigning it to a local variable.
--- drivers/mmc/card/block.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
drivers/mmc/card/block.c +6 -5
@@ -1600,6 +1600,7 @@ static int mmc_blk_issue_rw_rq(struct mmc_queue *mq, struct request *rqc)struct mmc_blk_request *brq;int ret =1 , disable_multi =0 , retry =0 , type, retune_retry_done =0 ;enum mmc_blk_status status; +struct mmc_queue_req *mqrq_cur = mq->mqrq_cur;struct mmc_queue_req *mq_rq;struct request *req;struct mmc_async_req *areq;
@@ -1617,14 +1618,14 @@ static int mmc_blk_issue_rw_rq(struct mmc_queue *mq, struct request *rqc) !IS_ALIGNED(blk_rq_sectors(rqc), 8)) {pr_err ("%s: Transfer size is not 4KB sector size aligned\n" , rqc->rq_disk->disk_name); - mq_rq = mq->mqrq_cur; + mq_rq = mqrq_cur; req = rqc; rqc =NULL ; goto cmd_abort; } -mmc_blk_rw_rq_prep (mq->mqrq_cur, card,0 , mq); - areq = &mq->mqrq_cur->mmc_active; +mmc_blk_rw_rq_prep (mqrq_cur, card,0 , mq); + areq = &mqrq_cur->mmc_active; }else areq =NULL ; areq =mmc_start_req (card->host, areq, &status);
@@ -1744,9 +1745,9 @@ static int mmc_blk_issue_rw_rq(struct mmc_queue *mq, struct request *rqc) rqc->cmd_flags |=REQ_QUIET ;blk_end_request_all (rqc, -EIO ); }else { -mmc_blk_rw_rq_prep (mq->mqrq_cur, card,0 , mq); +mmc_blk_rw_rq_prep (mqrq_cur, card,0 , mq);mmc_start_req (card->host, - &mq->mqrq_cur->mmc_active,NULL ); + &mqrq_cur->mmc_active,NULL ); } }
Signature · 1 line
1.9.1
PATCH v9 2/11 mmc: queue: Share mmc request array between partitions
In reply to Adrian HuntereMMC can have multiple internal partitions that are represented as separate disks / queues. However switching between partitions is only done when the queue is empty. Consequently the array of mmc requests that are queued can be shared between partitions saving memory.
Keep a pointer to the mmc request queue on the card, and use that instead of allocating a new one for each partition.
--- drivers/mmc/card/block.c | 11 ++- drivers/mmc/card/queue.c | 237 ++++++++++++++++++++++++++++------------------- drivers/mmc/card/queue.h | 2 + include/linux/mmc/card.h | 5 + 4 files changed, 157 insertions(+), 98 deletions(-)
drivers/mmc/card/block.c +10 -1
@@ -2172,6 +2172,7 @@ static int mmc_blk_probe(struct mmc_card *card) {struct mmc_blk_data *md, *part_md;char cap_str[10 ]; +int ret;/* * Check that the card supports the command class(es) we need.
@@ -2181,9 +2182,15 @@ static int mmc_blk_probe(struct mmc_card *card)mmc_fixup_device (card, blk_fixups); + ret =mmc_queue_alloc_shared_queue (card); +if (ret) +return ret; + md =mmc_blk_alloc (card); -if (IS_ERR (md)) +if (IS_ERR (md)) { +mmc_queue_free_shared_queue (card);return PTR_ERR (md); + }string_get_size ((u64 )get_capacity (md->disk),512 ,STRING_UNITS_2 , cap_str,sizeof (cap_str));
@@ -2221,6 +2228,7 @@ static int mmc_blk_probe(struct mmc_card *card) out:mmc_blk_remove_parts (card, md);mmc_blk_remove_req (md); +mmc_queue_free_shared_queue (card);return 0 ; }
@@ -2238,6 +2246,7 @@ static void mmc_blk_remove(struct mmc_card *card)pm_runtime_put_noidle (&card->dev);mmc_blk_remove_req (md);dev_set_drvdata (&card->dev,NULL ); +mmc_queue_free_shared_queue (card); }static int _mmc_blk_suspend (struct mmc_card *card)
drivers/mmc/card/queue.c +140 -97
320 lines not on this page · raw message
drivers/mmc/card/queue.h +2 -0
@@ -48,6 +48,8 @@ struct mmc_queue {int qdepth; }; +extern int mmc_queue_alloc_shared_queue (struct mmc_card *card); +extern void mmc_queue_free_shared_queue (struct mmc_card *card);extern int mmc_init_queue (struct mmc_queue *,struct mmc_card *,spinlock_t *,const char *);extern void mmc_cleanup_queue (struct mmc_queue *);
include/linux/mmc/card.h +5 -0
@@ -206,6 +206,7 @@ struct sdio_cis {struct mmc_ios ;struct sdio_func ;struct sdio_func_tuple ; +struct mmc_queue_req ;#define SDIO_MAX_FUNCS 7
@@ -306,6 +307,10 @@ struct mmc_card {struct dentry *debugfs_root;struct mmc_part part[MMC_NUM_PHY_PARTITION ];/* physical partitions */ unsigned int nr_parts; + +struct mmc_queue_req *mqrq;/* Shared queue structure */ +unsigned int bouncesz;/* Bounce buffer size */ +int qdepth;/* Shared queue depth */ };/*
Signature · 1 line
1.9.1
PATCH v9 3/11 mmc: block: Introduce queue semantics
In reply to Adrian HunterChange from viewing the requests in progress as 'current' and 'previous', to viewing them as a queue. The current request is allocated to the first free slot. The presence of incomplete requests is determined from the count (mq->qcnt) of entries in the queue. Non-read-write requests (i.e. discards and flushes) are not added to the queue at all and require no special handling. Also no special handling is needed for the MMC_BLK_NEW_REQUEST case.
As well as allowing an arbitrarily sized queue, the queue thread function is significantly simpler.
--- drivers/mmc/card/block.c | 49 +++++++++++++++++++------------- drivers/mmc/card/queue.c | 74 ++++++++++++++++++++++++++++++------------------ drivers/mmc/card/queue.h | 10 +++++-- 3 files changed, 84 insertions(+), 49 deletions(-)
drivers/mmc/card/block.c +30 -19
@@ -129,6 +129,13 @@ static inline int mmc_blk_part_switch(struct mmc_card *card, struct mmc_blk_data *md);static int get_card_status (struct mmc_card *card,u32 *status,int retries); +static void mmc_blk_requeue (struct request_queue *q,struct request *req) +{ +spin_lock_irq (q->queue_lock); +blk_requeue_request (q, req); +spin_unlock_irq (q->queue_lock); +} +static struct mmc_blk_data *mmc_blk_get (struct gendisk *disk) {struct mmc_blk_data *md;
@@ -1600,12 +1607,21 @@ static int mmc_blk_issue_rw_rq(struct mmc_queue *mq, struct request *rqc)struct mmc_blk_request *brq;int ret =1 , disable_multi =0 , retry =0 , type, retune_retry_done =0 ;enum mmc_blk_status status; -struct mmc_queue_req *mqrq_cur = mq->mqrq_cur; +struct mmc_queue_req *mqrq_cur =NULL ;struct mmc_queue_req *mq_rq;struct request *req;struct mmc_async_req *areq; -if (!rqc && !mq->mqrq_prev->req) +if (rqc) { + mqrq_cur =mmc_queue_req_find (mq, rqc); +if (!mqrq_cur) { +WARN_ON (1 ); +mmc_blk_requeue (mq->queue, rqc); + rqc =NULL ; + } + } + +if (!mq->qcnt)return 0 ; do {
@@ -1629,11 +1645,8 @@ static int mmc_blk_issue_rw_rq(struct mmc_queue *mq, struct request *rqc) }else areq =NULL ; areq =mmc_start_req (card->host, areq, &status); -if (!areq) { -if (status ==MMC_BLK_NEW_REQUEST ) - mq->flags |=MMC_QUEUE_NEW_REQUEST ; +if (!areq)return 0 ; - } mq_rq = container_of(areq, struct mmc_queue_req, mmc_active); brq = &mq_rq->brq;
@@ -1730,6 +1743,8 @@ static int mmc_blk_issue_rw_rq(struct mmc_queue *mq, struct request *rqc) } }while (ret); +mmc_queue_req_free (mq, mq_rq); +return 1 ; cmd_abort:
@@ -1744,6 +1759,7 @@ static int mmc_blk_issue_rw_rq(struct mmc_queue *mq, struct request *rqc)if (mmc_card_removed (card)) { rqc->cmd_flags |=REQ_QUIET ;blk_end_request_all (rqc, -EIO ); +mmc_queue_req_free (mq, mqrq_cur); }else {mmc_blk_rw_rq_prep (mqrq_cur, card,0 , mq); mmc_start_req(card->host,
@@ -1751,6 +1767,8 @@ static int mmc_blk_issue_rw_rq(struct mmc_queue *mq, struct request *rqc) } } +mmc_queue_req_free (mq, mq_rq); +return 0 ; }
@@ -1759,9 +1777,8 @@ int mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req)int ret;struct mmc_blk_data *md = mq->blkdata;struct mmc_card *card = md->queue.card; -bool req_is_special =mmc_req_is_special (req); -if (req && !mq->mqrq_prev->req) +if (req && !mq->qcnt)/* claim host only for the first request */ mmc_get_card (card);
@@ -1774,20 +1791,19 @@ int mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req) goto out; } - mq->flags &= ~MMC_QUEUE_NEW_REQUEST ;if (req &&req_op (req) ==REQ_OP_DISCARD ) {/* complete ongoing async transfer before issuing discard */ -if (card->host->areq) +if (mq->qcnt)mmc_blk_issue_rw_rq (mq,NULL ); ret =mmc_blk_issue_discard_rq (mq, req); }else if (req &&req_op (req) ==REQ_OP_SECURE_ERASE ) {/* complete ongoing async transfer before issuing secure erase*/ -if (card->host->areq) +if (mq->qcnt)mmc_blk_issue_rw_rq (mq,NULL ); ret =mmc_blk_issue_secdiscard_rq (mq, req); }else if (req &&req_op (req) ==REQ_OP_FLUSH ) {/* complete ongoing async transfer before issuing flush */ -if (card->host->areq) +if (mq->qcnt)mmc_blk_issue_rw_rq (mq,NULL ); ret =mmc_blk_issue_flush (mq, req); }else {
@@ -1795,13 +1811,8 @@ int mmc_blk_issue_rq(struct mmc_queue *mq, struct request *req) } out: -if ((!req && !(mq->flags &MMC_QUEUE_NEW_REQUEST )) || req_is_special) -/* -* Release host when there are no more requests -* and after special request(discard, flush) is done. -* In case sepecial request, there is no reentry to -* the 'mmc_blk_issue_rq' with 'mqrq_prev->req'. -*/ +/* Release host when there are no more requests */ +if (!mq->qcnt)mmc_put_card (card);return ret; }
drivers/mmc/card/queue.c +47 -27
119 lines not on this page · raw message
drivers/mmc/card/queue.h +7 -3
@@ -30,6 +30,7 @@ struct mmc_queue_req {struct scatterlist *bounce_sg;unsigned int bounce_sg_len;struct mmc_async_req mmc_active; +int task_id; };struct mmc_queue {
@@ -38,14 +39,13 @@ struct mmc_queue {struct semaphore thread_sem;unsigned int flags;#define MMC_QUEUE_SUSPENDED (1 << 0) -#define MMC_QUEUE_NEW_REQUEST (1 << 1)bool asleep;struct mmc_blk_data *blkdata;struct request_queue *queue;struct mmc_queue_req *mqrq; -struct mmc_queue_req *mqrq_cur; -struct mmc_queue_req *mqrq_prev;int qdepth; +int qcnt; +unsigned long qslots; };extern int mmc_queue_alloc_shared_queue (struct mmc_card *card);
@@ -63,4 +63,8 @@ extern unsigned int mmc_queue_map_sg(struct mmc_queue *,extern int mmc_access_rpmb (struct mmc_queue *); +extern struct mmc_queue_req *mmc_queue_req_find (struct mmc_queue *, +struct request *); +extern void mmc_queue_req_free (struct mmc_queue *,struct mmc_queue_req *); +#endif
Signature · 1 line
1.9.1
PATCH v9 4/11 mmc: core: Do not prepare a new request twice
In reply to Adrian Huntermmc_start_req() assumes it is never called with the new request already prepared. That is true if the queue consists of only 2 requests, but is not true for a longer queue. e.g. mmc_start_req() has a current and previous request but still exits to queue a new request if the queue size is greater than 2. In that case, when mmc_start_req() is called again, the current request will have been prepared already. Fix by flagging if the request has been prepared.
That also means ensuring that struct mmc_async_req is always initialized to zero, which wasn't the case in mmc_test.
--- drivers/mmc/card/mmc_test.c | 8 ++++---- drivers/mmc/core/core.c | 12 +++++++++--- include/linux/mmc/host.h | 1 + 3 files changed, 14 insertions(+), 7 deletions(-)
drivers/mmc/card/mmc_test.c +4 -4
@@ -830,7 +830,10 @@ static int mmc_test_nonblock_transfer(struct mmc_test_card *test,struct mmc_command stop2;struct mmc_data data2; -struct mmc_test_async_req test_areq[2 ]; +struct mmc_test_async_req test_areq[2 ] = { + { .test = test }, + { .test = test }, + };struct mmc_async_req *done_areq;struct mmc_async_req *cur_areq = &test_areq[0 ].areq;struct mmc_async_req *other_areq = &test_areq[1 ].areq;
@@ -838,9 +841,6 @@ static int mmc_test_nonblock_transfer(struct mmc_test_card *test,int i;int ret =RESULT_OK ; - test_areq[0 ].test = test; - test_areq[1 ].test = test; -mmc_test_nonblock_reset (&mrq1, &cmd1, &stop1, &data1);mmc_test_nonblock_reset (&mrq2, &cmd2, &stop2, &data2);
drivers/mmc/core/core.c +9 -3
@@ -656,8 +656,10 @@ struct mmc_async_req *mmc_start_req(struct mmc_host *host,struct mmc_async_req *data = host->areq;/* Prepare a new request */ -if (areq) +if (areq && !areq->pre_req_done) { + areq->pre_req_done = true;mmc_pre_req (host, areq->mrq); + }if (host->areq) { status =mmc_wait_for_data_req_done (host, host->areq->mrq, areq);
@@ -693,12 +695,16 @@ struct mmc_async_req *mmc_start_req(struct mmc_host *host,if (status ==MMC_BLK_SUCCESS && areq) start_err =__mmc_start_data_req (host, areq->mrq); -if (host->areq) +if (host->areq) { + host->areq->pre_req_done = false;mmc_post_req (host, host->areq->mrq,0 ); + }/* Cancel a prepared request if it was not started. */ -if ((status !=MMC_BLK_SUCCESS || start_err) && areq) +if ((status !=MMC_BLK_SUCCESS || start_err) && areq) { + areq->pre_req_done = false;mmc_post_req (host, areq->mrq, -EINVAL ); + }if (status !=MMC_BLK_SUCCESS ) host->areq =NULL ;
include/linux/mmc/host.h +1 -0
@@ -173,6 +173,7 @@ struct mmc_async_req {* Returns 0 if success otherwise non zero. */ enum mmc_blk_status (*err_check)(struct mmc_card *,struct mmc_async_req *); +bool pre_req_done; };/**
Signature · 1 line
1.9.1
PATCH v9 5/11 mmc: mmc: Add functions to enable / disable the Command Queue
In reply to Adrian HunterAdd helper functions to enable or disable the Command Queue.
--- Documentation/mmc/mmc-dev-attrs.txt | 1 + drivers/mmc/core/mmc.c | 2 ++ drivers/mmc/core/mmc_ops.c | 28 ++++++++++++++++++++++++++++ include/linux/mmc/card.h | 1 + include/linux/mmc/core.h | 2 ++ 5 files changed, 34 insertions(+)
Documentation/mmc/mmc-dev-attrs.txt +1 -0
@@ -30,6 +30,7 @@ All attributes are read-only. rel_sectors Reliable write sector count ocr Operation Conditions Register dsr Driver Stage Register + cmdq_en Command Queue enabled: 1 => enabled, 0 => not enabled Note on Erase Size and Preferred Erase Size:
drivers/mmc/core/mmc.c +2 -0
@@ -771,6 +771,7 @@ static int mmc_compare_ext_csds(struct mmc_card *card, unsigned bus_width)MMC_DEV_ATTR (raw_rpmb_size_mult,"%#x\n" , card->ext_csd.raw_rpmb_size_mult);MMC_DEV_ATTR (rel_sectors,"%#x\n" , card->ext_csd.rel_sectors);MMC_DEV_ATTR (ocr,"%08x\n" , card->ocr); +MMC_DEV_ATTR (cmdq_en,"%d\n" , card->ext_csd.cmdq_en); static ssize_t mmc_fwrev_show(struct device *dev, struct device_attribute *attr,
@@ -824,6 +825,7 @@ static ssize_t mmc_dsr_show(struct device *dev, &dev_attr_rel_sectors.attr, &dev_attr_ocr.attr, &dev_attr_dsr.attr, + &dev_attr_cmdq_en.attr,NULL , };ATTRIBUTE_GROUPS (mmc_std);
drivers/mmc/core/mmc_ops.c +28 -0
@@ -839,3 +839,31 @@ int mmc_can_ext_csd(struct mmc_card *card) {return (card && card->csd.mmca_vsn >CSD_SPEC_VER_3 ); } + +static int mmc_cmdq_switch (struct mmc_card *card,bool enable) +{ +u8 val = enable ?EXT_CSD_CMDQ_MODE_ENABLED :0 ; +int err; + +if (!card->ext_csd.cmdq_support) +return -EOPNOTSUPP ; + + err =mmc_switch (card,EXT_CSD_CMD_SET_NORMAL ,EXT_CSD_CMDQ_MODE_EN , + val, card->ext_csd.generic_cmd6_time); +if (!err) + card->ext_csd.cmdq_en = enable; + +return err; +} + +int mmc_cmdq_enable (struct mmc_card *card) +{ +return mmc_cmdq_switch (card, true); +} +EXPORT_SYMBOL_GPL (mmc_cmdq_enable); + +int mmc_cmdq_disable (struct mmc_card *card) +{ +return mmc_cmdq_switch (card, false); +} +EXPORT_SYMBOL_GPL (mmc_cmdq_disable);
include/linux/mmc/card.h +1 -0
@@ -89,6 +89,7 @@ struct mmc_ext_csd {unsigned int boot_ro_lock;/* ro lock support */ bool boot_ro_lockable;bool ffu_capable;/* Firmware upgrade support */ +bool cmdq_en;/* Command Queue enabled */ bool cmdq_support;/* Command Queue supported */ unsigned int cmdq_depth;/* Command Queue depth */ #define MMC_FIRMWARE_LEN 8
include/linux/mmc/core.h +2 -0
@@ -178,6 +178,8 @@ extern int mmc_wait_for_app_cmd(struct mmc_host *, struct mmc_card *,extern int mmc_send_tuning (struct mmc_host *host,u32 opcode,int *cmd_error);extern int mmc_abort_tuning (struct mmc_host *host,u32 opcode);extern int mmc_get_ext_csd (struct mmc_card *card,u8 **new_ext_csd); +extern int mmc_cmdq_enable (struct mmc_card *card); +extern int mmc_cmdq_disable (struct mmc_card *card);#define MMC_ERASE_ARG 0x00000000#define MMC_SECURE_ERASE_ARG 0x80000000
Signature · 1 line
1.9.1
PATCH v9 6/11 mmc: mmc_test: Disable Command Queue while mmc_test is used
In reply to Adrian HunterNormal read and write commands may not be used while the command queue is enabled. Disable the Command Queue when mmc_test is probed and re-enable it when it is removed.
--- drivers/mmc/card/mmc_test.c | 13 +++++++++++++ drivers/mmc/core/mmc.c | 7 +++++++ include/linux/mmc/card.h | 1 + 3 files changed, 21 insertions(+)
drivers/mmc/card/mmc_test.c +13 -0
@@ -3268,6 +3268,14 @@ static int mmc_test_probe(struct mmc_card *card)if (ret)return ret; +if (card->ext_csd.cmdq_en) { +mmc_claim_host (card->host); + ret =mmc_cmdq_disable (card); +mmc_release_host (card->host); +if (ret) +return ret; + } +dev_info (&card->dev,"Card claimed for testing.\n" );return 0 ;
@@ -3275,6 +3283,11 @@ static int mmc_test_probe(struct mmc_card *card)static void mmc_test_remove (struct mmc_card *card) { +if (card->reenable_cmdq) { +mmc_claim_host (card->host); +mmc_cmdq_enable (card); +mmc_release_host (card->host); + }mmc_test_free_result (card);mmc_test_free_dbgfs_file (card); }
drivers/mmc/core/mmc.c +7 -0
@@ -1766,6 +1766,13 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr, }/* +* In some cases (e.g. RPMB or mmc_test), the Command Queue must be +* disabled for a time, so a flag is needed to indicate to re-enable the +* Command Queue. +*/ + card->reenable_cmdq = card->ext_csd.cmdq_en; + +/* * The mandatory minimum values are defined for packed command. * read: 5, write: 3 */
include/linux/mmc/card.h +1 -0
@@ -274,6 +274,7 @@ struct mmc_card {#define MMC_QUIRK_TRIM_BROKEN (1<<12)/* Skip trim */ #define MMC_QUIRK_BROKEN_HPI (1<<13)/* Disable broken HPI support */ +bool reenable_cmdq;/* Re-enable Command Queue */ unsigned int erase_size;/* erase size in sectors */ unsigned int erase_shift;/* if erase unit is power 2 */
Signature · 1 line
1.9.1
PATCH v9 7/11 mmc: block: Disable Command Queue while RPMB is used
In reply to Adrian HunterRPMB does not allow Command Queue commands. Disable and re-enable the Command Queue when switching.
Note that the driver only switches partitions when the queue is empty.
--- drivers/mmc/card/block.c | 46 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 8 deletions(-)
drivers/mmc/card/block.c +38 -8
@@ -732,10 +732,41 @@ static int mmc_blk_compat_ioctl(struct block_device *bdev, fmode_t mode,#endif }; +static int mmc_blk_part_switch_pre (struct mmc_card *card, +unsigned int part_type) +{ +int ret =0 ; + +if (part_type ==EXT_CSD_PART_CONFIG_ACC_RPMB ) { +if (card->ext_csd.cmdq_en) { + ret =mmc_cmdq_disable (card); +if (ret) +return ret; + } +mmc_retune_pause (card->host); + } + +return ret; +} + +static int mmc_blk_part_switch_post (struct mmc_card *card, +unsigned int part_type) +{ +int ret =0 ; + +if (part_type ==EXT_CSD_PART_CONFIG_ACC_RPMB ) { +mmc_retune_unpause (card->host); +if (card->reenable_cmdq && !card->ext_csd.cmdq_en) + ret =mmc_cmdq_enable (card); + } + +return ret; +} +static inline int mmc_blk_part_switch (struct mmc_card *card,struct mmc_blk_data *md) { -int ret; +int ret =0 ;struct mmc_blk_data *main_md =dev_get_drvdata (&card->dev);if (main_md->part_curr == md->part_type)
@@ -744,8 +775,9 @@ static inline int mmc_blk_part_switch(struct mmc_card *card,if (mmc_card_mmc (card)) {u8 part_config = card->ext_csd.part_config; -if (md->part_type ==EXT_CSD_PART_CONFIG_ACC_RPMB ) -mmc_retune_pause (card->host); + ret =mmc_blk_part_switch_pre (card, md->part_type); +if (ret) +return ret; part_config &= ~EXT_CSD_PART_CONFIG_ACC_MASK ; part_config |= md->part_type;
@@ -754,19 +786,17 @@ static inline int mmc_blk_part_switch(struct mmc_card *card,EXT_CSD_PART_CONFIG , part_config, card->ext_csd.part_time);if (ret) { -if (md->part_type ==EXT_CSD_PART_CONFIG_ACC_RPMB ) -mmc_retune_unpause (card->host); +mmc_blk_part_switch_post (card, md->part_type);return ret; } card->ext_csd.part_config = part_config; -if (main_md->part_curr ==EXT_CSD_PART_CONFIG_ACC_RPMB ) -mmc_retune_unpause (card->host); + ret =mmc_blk_part_switch_post (card, main_md->part_curr); } main_md->part_curr = md->part_type; -return 0 ; +return ret; }static u32 mmc_sd_num_wr_blocks (struct mmc_card *card)
Signature · 1 line
1.9.1
PATCH v9 8/11 mmc: core: Export mmc_retune_hold() and mmc_retune_release()
In reply to Adrian HunterRe-tuning can only be done when the Command Queue is empty, when means holding and releasing re-tuning from the block driver, so export those functions.
--- drivers/mmc/core/host.c | 2 ++ drivers/mmc/core/host.h | 2 -- include/linux/mmc/core.h | 3 +++ 3 files changed, 5 insertions(+), 2 deletions(-)
drivers/mmc/core/host.c +2 -0
@@ -112,6 +112,7 @@ void mmc_retune_hold(struct mmc_host *host) host->retune_now =1 ; host->hold_retune +=1 ; } +EXPORT_SYMBOL (mmc_retune_hold);void mmc_retune_release (struct mmc_host *host) {
@@ -120,6 +121,7 @@ void mmc_retune_release(struct mmc_host *host)else WARN_ON (1 ); } +EXPORT_SYMBOL (mmc_retune_release);int mmc_retune (struct mmc_host *host) {
drivers/mmc/core/host.h +0 -2
@@ -17,8 +17,6 @@void mmc_retune_enable (struct mmc_host *host);void mmc_retune_disable (struct mmc_host *host); -void mmc_retune_hold (struct mmc_host *host); -void mmc_retune_release (struct mmc_host *host);int mmc_retune (struct mmc_host *host);#endif
include/linux/mmc/core.h +3 -0
@@ -221,6 +221,9 @@ extern int mmc_set_blockcount(struct mmc_card *card, unsigned int blockcount,extern int mmc_detect_card_removed (struct mmc_host *host); +extern void mmc_retune_hold (struct mmc_host *host); +extern void mmc_retune_release (struct mmc_host *host); +/** * mmc_claim_host - exclusively claim a host * @host: mmc host to claim
Signature · 1 line
1.9.1
PATCH v9 9/11 mmc: queue: Add a function to control wake-up on new requests
In reply to Adrian HunterAdd a function to control wake-up on new requests. This will enable Software Command Queuing to choose whether or not to queue new requests immediately or wait for the current task to complete.
--- drivers/mmc/card/queue.c | 16 ++++++++++++++++ drivers/mmc/card/queue.h | 2 ++ 2 files changed, 18 insertions(+)
drivers/mmc/card/queue.c +16 -0
@@ -97,6 +97,7 @@ static int mmc_queue_thread(void *d) cntx->is_waiting_last_req = false; cntx->is_new_req = false;if (!req) { + mq->is_new_req = false;/* * Dispatch queue is empty so set flags for * mmc_request_fn() to wake us up.
@@ -147,6 +148,8 @@ static void mmc_request_fn(struct request_queue *q)return ; } + mq->is_new_req = true; + cntx = &mq->card->host->context_info;if (cntx->is_waiting_last_req) {
@@ -158,6 +161,19 @@ static void mmc_request_fn(struct request_queue *q)wake_up_process (mq->thread); } +void mmc_queue_set_wake (struct mmc_queue *mq,bool wake_me) +{ +struct mmc_context_info *cntx = &mq->card->host->context_info; +struct request_queue *q = mq->queue; + +if (cntx->is_waiting_last_req != wake_me) { +spin_lock_irq (q->queue_lock); + cntx->is_waiting_last_req = wake_me; + cntx->is_new_req = wake_me && mq->is_new_req; +spin_unlock_irq (q->queue_lock); + } +} +static struct scatterlist *mmc_alloc_sg (int sg_len) {struct scatterlist *sg;
drivers/mmc/card/queue.h +2 -0
@@ -40,6 +40,7 @@ struct mmc_queue {unsigned int flags;#define MMC_QUEUE_SUSPENDED (1 << 0)bool asleep; +bool is_new_req;struct mmc_blk_data *blkdata;struct request_queue *queue;struct mmc_queue_req *mqrq;
@@ -66,5 +67,6 @@ extern unsigned int mmc_queue_map_sg(struct mmc_queue *,extern struct mmc_queue_req *mmc_queue_req_find (struct mmc_queue *,struct request *);extern void mmc_queue_req_free (struct mmc_queue *,struct mmc_queue_req *); +extern void mmc_queue_set_wake (struct mmc_queue *,bool );#endif
Signature · 1 line
1.9.1
PATCH v9 10/11 mmc: block: Add Software Command Queuing
In reply to Adrian HuntereMMC Command Queuing is a feature added in version 5.1. The card maintains a queue of up to 32 data transfers. Commands CMD44/CMD45 are sent to queue up transfers in advance, and then one of the transfers is selected to "execute" by CMD46/CMD47 at which point data transfer actually begins.
The advantage of command queuing is that the card can prepare for transfers in advance. That makes a big difference in the case of random reads because the card can start reading into its cache in advance.
A v5.1 host controller can manage the command queue itself, but it is also possible for software to manage the queue using an non-v5.1 host controller
- that is what Software Command Queuing is.
Refer to the JEDEC (http://www.jedec.org/) eMMC v5.1 Specification for more information about Command Queuing.
Two important aspects of Command Queuing that affect the implementation are:
- only read/write requests are queued
- the queue must be empty to send other commands, including re-tuning
To support Software Command Queuing a separate function is provided to issue read/write requests (i.e. mmc_swcmdq_issue_rw_rq()) and the mmc_blk_request structure amended to cater for additional commands CMD44 and CMD45. There is a separate function (mmc_swcmdq_prep()) to prepare the needed commands, but transfers are started by mmc_start_req() like normal.
mmc_swcmdq_issue_rw_rq() enqueues the new request and then executes tasks until the queue is empty or mmc_swcmdq_execute() asks for a new request. This puts mmc_swcmdq_execute() in control of the decision whether to queue more requests or wait for the active one.
Recovery is invoked if anything goes wrong. Recovery has 2 options:
1. Discard the queue and re-queue all requests. If that fails, fall back to option 2.
2. Reset and re-queue all requests. If that fails, error out all the requests.
In either case, re-tuning will be done if needed after the queue becomes empty because re-tuning is released at that point.
--- drivers/mmc/card/block.c | 591 ++++++++++++++++++++++++++++++++++++++++++++++- drivers/mmc/card/queue.c | 10 +- drivers/mmc/card/queue.h | 11 +- include/linux/mmc/core.h | 1 + 4 files changed, 610 insertions(+), 3 deletions(-)
drivers/mmc/card/block.c +590 -1
@@ -108,6 +108,7 @@ struct mmc_blk_data {#define MMC_BLK_WRITE BIT(1)#define MMC_BLK_DISCARD BIT(2)#define MMC_BLK_SECDISCARD BIT(3) +#define MMC_BLK_SWCMDQ BIT(4)/* * Only set in main mmc_blk_data associated
@@ -1630,7 +1631,584 @@ static int mmc_blk_cmd_err(struct mmc_blk_data *md, struct mmc_card *card,return ret; } -static int mmc_blk_issue_rw_rq (struct mmc_queue *mq,struct request *rqc) +static enum mmc_blk_status mmc_swcmdq_err_check (struct mmc_card *card, +struct mmc_async_req *areq) +{ + struct mmc_queue_req *mqrq = container_of(areq, struct mmc_queue_req, + mmc_active); +struct mmc_blk_request *brq = &mqrq->brq; +struct request *req = mqrq->req; +int err; + + err = brq->data.error; +/* In the case of data errors, send stop */ +if (err) +mmc_wait_for_cmd (card->host, &brq->stop,0 ); +else + err = brq->cmd.error; + +/* In the case of CRC errors when re-tuning is needed, retry */ +if (err == -EILSEQ && card->host->need_retune) +return MMC_BLK_RETRY ; + +/* For other errors abort */ +if (err) +return MMC_BLK_ABORT ; + +if (blk_rq_bytes (req) != brq->data.bytes_xfered) +return MMC_BLK_PARTIAL ; + +return MMC_BLK_SUCCESS ; +} + +static void mmc_swcmdq_prep (struct mmc_queue *mq,struct mmc_queue_req *mqrq) +{ +struct mmc_blk_data *md = mq->blkdata; +struct mmc_card *card = md->queue.card; +struct mmc_blk_request *brq = &mqrq->brq; +struct request *req = mqrq->req; +bool do_data_tag; + +/* +* Reliable writes are used to implement Forced Unit Access and +* are supported only on MMCs. +*/ +bool do_rel_wr = (req->cmd_flags &REQ_FUA ) && + (rq_data_dir (req) ==WRITE ) && + (md->flags &MMC_BLK_REL_WR ); + +memset (brq,0 ,sizeof (struct mmc_blk_request )); + brq->mrq.cmd = &brq->cmd; + brq->mrq.data = &brq->data; + brq->mrq.cap_cmd_during_tfr = true; + +if (rq_data_dir (req) ==READ ) { + brq->cmd.opcode =MMC_EXECUTE_READ_TASK ; + brq->data.flags =MMC_DATA_READ ; + brq->stop.flags =MMC_RSP_R1 |MMC_CMD_AC ; + }else { + brq->cmd.opcode =MMC_EXECUTE_WRITE_TASK ; + brq->data.flags =MMC_DATA_WRITE ; + brq->stop.flags =MMC_RSP_R1B |MMC_CMD_AC ; + } + brq->cmd.arg = mqrq->task_id <<16 ; + brq->cmd.flags =MMC_RSP_R1 |MMC_CMD_ADTC ; + + brq->data.blksz =512 ; + brq->data.blocks =blk_rq_sectors (req); + + brq->stop.opcode =MMC_STOP_TRANSMISSION ; + brq->stop.arg =0 ; + +/* +* The block layer doesn't support all sector count +* restrictions, so we need to be prepared for too big +* requests. +*/ +if (brq->data.blocks > card->host->max_blk_count) + brq->data.blocks = card->host->max_blk_count; + +if (do_rel_wr) +mmc_apply_rel_rw (brq, card, req); + +/* +* Data tag is used only during writing meta data to speed +* up write and any subsequent read of this meta data +*/ + do_data_tag = (card->ext_csd.data_tag_unit_size) && + (req->cmd_flags &REQ_META ) && + (rq_data_dir (req) ==WRITE ) && + ((brq->data.blocks * brq->data.blksz) >= + card->ext_csd.data_tag_unit_size); + + brq->cmd44.opcode =MMC_QUE_TASK_PARAMS ; + brq->cmd44.arg = brq->data.blocks | + (do_rel_wr ? (1 <<31 ) :0 ) | + ((rq_data_dir (req) ==READ ) ? (1 <<30 ) :0 ) | + (do_data_tag ? (1 <<29 ) :0 ) | + mqrq->task_id <<16 ; + brq->cmd44.flags =MMC_RSP_R1 |MMC_CMD_AC ; + + brq->cmd45.opcode =MMC_QUE_TASK_ADDR ; + brq->cmd45.arg =blk_rq_pos (req); + +mmc_set_data_timeout (&brq->data, card); + + brq->data.sg = mqrq->sg; + brq->data.sg_len =mmc_queue_map_sg (mq, mqrq); + +/* +* Adjust the sg list so it is the same size as the +* request. +*/ +if (brq->data.blocks !=blk_rq_sectors (req)) { +int i, data_size = brq->data.blocks <<9 ; +struct scatterlist *sg; + +for_each_sg (brq->data.sg, sg, brq->data.sg_len, i) { + data_size -= sg->length; +if (data_size <=0 ) { + sg->length += data_size; + i++; +break ; + } + } + brq->data.sg_len = i; + } + + mqrq->mmc_active.mrq = &brq->mrq; + mqrq->mmc_active.err_check = mmc_swcmdq_err_check; + +mmc_queue_bounce_pre (mqrq); +} + +static int mmc_swcmdq_blk_err (struct mmc_card *card,int err) +{ +/* Re-try after CRC errors when re-tuning is needed */ +if (err == -EILSEQ && card->host->need_retune) +return MMC_BLK_RETRY ; + +if (err) +return MMC_BLK_ABORT ; + +return 0 ; +} + +#define SWCMDQ_ENQUEUE_ERR ( \ + R1_OUT_OF_RANGE | \ + R1_ADDRESS_ERROR | \ + R1_BLOCK_LEN_ERROR | \ + R1_WP_VIOLATION | \ + R1_CC_ERROR | \ + R1_ERROR | \ + R1_COM_CRC_ERROR | \ + R1_ILLEGAL_COMMAND) + +static int __mmc_swcmdq_enqueue (struct mmc_queue *mq, +struct mmc_queue_req *mqrq) +{ +struct mmc_card *card = mq->card; +int err; + +mmc_swcmdq_prep (mq, mqrq); + + err =mmc_wait_for_cmd (card->host, &mqrq->brq.cmd44,0 ); +if (err) + goto out; + + err =mmc_wait_for_cmd (card->host, &mqrq->brq.cmd45,0 ); +if (err) + goto out; + +/* +* Don't assume the task is queued if there are any error bits set in +* the response. +*/ +if (mqrq->brq.cmd45.resp[0 ] &SWCMDQ_ENQUEUE_ERR ) +return MMC_BLK_ABORT ; +out: +return mmc_swcmdq_blk_err (card, err); +} + +static int mmc_swcmdq_enqueue (struct mmc_queue *mq,struct request *req) +{ +struct mmc_queue_req *mqrq; + + mqrq =mmc_queue_req_find (mq, req); +if (!mqrq) { +WARN_ON (1 ); +mmc_blk_requeue (mq->queue, req); +return 0 ; + } + +/* Need to hold re-tuning so long as the queue is not empty */ +if (mq->qcnt ==1 ) +mmc_retune_hold (mq->card->host); + +return __mmc_swcmdq_enqueue (mq, mqrq); +} + +static struct mmc_async_req *mmc_swcmdq_next (struct mmc_queue *mq) +{ +int i =__ffs (mq->qsr); + +__clear_bit (i, &mq->qsr); + +if (i >= mq->qdepth) +return NULL ; + +return &mq->mqrq[i].mmc_active; +} + +static int mmc_get_qsr (struct mmc_card *card,u32 *qsr) +{ +struct mmc_command cmd = {0 }; +int err, retries =3 ; + + cmd.opcode =MMC_SEND_STATUS ; + cmd.arg = card->rca <<16 |1 <<15 ; + cmd.flags =MMC_RSP_R1 |MMC_CMD_AC ; + err =mmc_wait_for_cmd (card->host, &cmd, retries); +if (err) +return err; + + *qsr = cmd.resp[0 ]; + +return 0 ; +} + +static int mmc_await_qsr (struct mmc_card *card,u32 *qsr) +{ +unsigned long timeout; +u32 status =0 ; +int err; + + timeout = jiffies +msecs_to_jiffies (10 *1000 ); +while (!status) { + err =mmc_get_qsr (card, &status); +if (err) +return err; +if (time_after (jiffies, timeout)) { +pr_err ("%s: Card stuck with queued tasks\n" , +mmc_hostname (card->host)); +return -ETIMEDOUT ; + } + } + + *qsr = status; + +return 0 ; +} + +static int mmc_swcmdq_await_qsr (struct mmc_queue *mq,struct mmc_card *card, +bool wait) +{ +struct mmc_queue_req *mqrq; +u32 qsr; +int err; + +if (wait) + err =mmc_await_qsr (card, &qsr); +else + err =mmc_get_qsr (card, &qsr); +if (err) + goto out_err; + + mq->qsr = qsr; + +if (card->host->areq) { +/* +* The active request remains in the QSR until completed. Remove +* it so that mq->qsr only contains ones that are ready but not +* executed. +*/ + mqrq = container_of(card->host->areq, struct mmc_queue_req, + mmc_active); +__clear_bit (mqrq->task_id, &mq->qsr); + } + +if (mq->qsr) + mq->qsr_err = false; +out_err: +if (err) { +/* Don't repeatedly retry if no progress is made */ +if (mq->qsr_err) +return MMC_BLK_ABORT ; + mq->qsr_err = true; + } + +return mmc_swcmdq_blk_err (card, err); +} + +static int mmc_swcmdq_execute (struct mmc_queue *mq,bool flush,bool requeuing, +bool new_req) +{ +struct mmc_card *card = mq->card; +struct mmc_async_req *next =NULL , *prev; +struct mmc_blk_request *brq; +struct mmc_queue_req *mqrq; +enum mmc_blk_status status; +struct request *req; +int active = card->host->areq ?1 :0 ; +int ret; + +if (mq->prepared_areq) { +/* +* A request that has been prepared before (i.e. passed to +* mmc_start_req()) but not started because another new request +* turned up. +*/ + next = mq->prepared_areq; + }else if (requeuing) { +/* Just finish the active request */ + next =NULL ; + }else if (mq->qsr) { +/* Get the next task from the Queue Status Register */ + next =mmc_swcmdq_next (mq); + }else if (mq->qcnt > active) { +/* +* There are queued tasks so read the Queue Status Register to +* see if any are ready. Wait for a ready task only if there is +* no active request and no new request. +*/ + ret =mmc_swcmdq_await_qsr (mq, card, !active && !new_req); +if (ret) +return ret; +if (mq->qsr) + next =mmc_swcmdq_next (mq); + } + +if (next) { +/* +* Don't wake for a new request when waiting for the active +* request if there is another request ready to start. +*/ +if (active) +mmc_queue_set_wake (mq, false); + }else { +if (!active) +return 0 ; +/* +* Don't wake for a new request when flushing or the queue is +* full. +*/ +if (flush || mq->qcnt == mq->qdepth) +mmc_queue_set_wake (mq, false); +else +mmc_queue_set_wake (mq, true); + } + + prev =mmc_start_req (card->host, next, &status); + +if (status ==MMC_BLK_NEW_REQUEST ) { + mq->prepared_areq = next; +return status; + } + + mq->prepared_areq =NULL ; + +if (!prev) +return 0 ; + + mqrq = container_of(prev, struct mmc_queue_req, mmc_active); + brq = &mqrq->brq; + req = mqrq->req; + +mmc_queue_bounce_post (mqrq); + +switch (status) { +case MMC_BLK_SUCCESS : +case MMC_BLK_PARTIAL : +case MMC_BLK_SUCCESS_ERR : +mmc_blk_reset_success (mq->blkdata,MMC_BLK_SWCMDQ ); + ret =blk_end_request (req,0 , brq->data.bytes_xfered); +if (ret) { +if (!requeuing) +return __mmc_swcmdq_enqueue (mq, mqrq); +return 0 ; + } +break ; +case MMC_BLK_NEW_REQUEST : +return status; +default : +if (mqrq->retry_cnt++) { +blk_end_request_all (req, -EIO ); +break ; + } +return status; + } + +mmc_queue_req_free (mq, mqrq); + +/* Release re-tuning when queue is empty */ +if (!mq->qcnt) +mmc_retune_release (card->host); + +return 0 ; +} + +static enum mmc_blk_status mmc_swcmdq_requeue_check (struct mmc_card *card, +struct mmc_async_req *areq) +{ +enum mmc_blk_status ret =mmc_swcmdq_err_check (card, areq); + +/* +* In the case of success, prevent mmc_start_req() from starting +* another request by returning MMC_BLK_SUCCESS_ERR. +*/ +return ret ==MMC_BLK_SUCCESS ?MMC_BLK_SUCCESS_ERR : ret; +} + +static int mmc_swcmdq_await_active (struct mmc_queue *mq) +{ +struct mmc_async_req *areq = mq->card->host->areq; +int err; + +if (!areq) +return 0 ; + + areq->err_check = mmc_swcmdq_requeue_check; + + err =mmc_swcmdq_execute (mq, true, true, false); + +/* The request will be requeued anyway, so ignore 'retry' */ +if (err ==MMC_BLK_RETRY ) + err =0 ; + +return err; +} + +static int mmc_swcmdq_discard_queue (struct mmc_queue *mq) +{ +struct mmc_command cmd = {0 }; + +if (!mq->qcnt) +return 0 ; + + mq->qsr =0 ; + + cmd.opcode =MMC_CMDQ_TASK_MGMT ; + cmd.arg =1 ;/* Discard entire queue */ + cmd.flags =MMC_RSP_R1B |MMC_CMD_AC ; +/* This is for recovery and the response is not needed, so ignore CRC */ + cmd.flags &= ~MMC_RSP_CRC ; + +return mmc_wait_for_cmd (mq->card->host, &cmd,0 ); +} + +static int __mmc_swcmdq_requeue (struct mmc_queue *mq) +{ +unsigned long i, qslots = mq->qslots; +int err; + +if (qslots) { +/* Cause re-tuning before next command, if needed */ +mmc_retune_release (mq->card->host); +mmc_retune_hold (mq->card->host); + } + +while (qslots) { + i =__ffs (qslots); + err =__mmc_swcmdq_enqueue (mq, &mq->mqrq[i]); +if (err) +return err; +__clear_bit (i, &qslots); + } + +return 0 ; +} + +static void __mmc_swcmdq_error_out (struct mmc_queue *mq) +{ +unsigned long i, qslots = mq->qslots; +struct request *req; + +if (qslots) +mmc_retune_release (mq->card->host); + +while (qslots) { + i =__ffs (qslots); + req = mq->mqrq[i].req; +blk_end_request_all (req, -EIO ); + mq->mqrq[i].req =NULL ; +__clear_bit (i, &qslots); + } + + mq->qslots =0 ; + mq->qcnt =0 ; +} + +static int mmc_swcmdq_requeue (struct mmc_queue *mq) +{ +int err; + +/* Wait for active request */ + err =mmc_swcmdq_await_active (mq); +if (err) +return err; + + err =mmc_swcmdq_discard_queue (mq); +if (err) +return err; + +return __mmc_swcmdq_requeue (mq); +} + +static void mmc_swcmdq_reset (struct mmc_queue *mq) +{ +/* Wait for active request ignoring errors */ +mmc_swcmdq_await_active (mq); + +/* Ensure the queue is discarded */ +mmc_swcmdq_discard_queue (mq); + +/* Reset and requeue else error out all requests */ +if (mmc_blk_reset (mq->blkdata, mq->card->host,MMC_BLK_SWCMDQ ) || +__mmc_swcmdq_requeue (mq)) +__mmc_swcmdq_error_out (mq); +} + +/* +* Recovery has 2 options: +* 1. Discard the queue and re-queue all requests. If that fails, fall back to +* option 2. +* 2. Reset and re-queue all requests. If that fails, error out all the +* requests. +* In either case, re-tuning will be done if needed after the queue becomes +* empty because re-tuning is released at that point. +*/ +static void mmc_swcmdq_recovery (struct mmc_queue *mq,int err) +{ +switch (err) { +case MMC_BLK_RETRY : + err =mmc_swcmdq_requeue (mq); +if (!err) +break ; +/* Fall through */ +default : +mmc_swcmdq_reset (mq); + } +} + +static int mmc_swcmdq_issue_rw_rq (struct mmc_queue *mq,struct request *req) +{ +struct mmc_context_info *cntx = &mq->card->host->context_info; +bool flush = !req && !cntx->is_waiting_last_req; +int err; + +/* Enqueue new requests */ +if (req) { + err =mmc_swcmdq_enqueue (mq, req); +if (err) +mmc_swcmdq_recovery (mq, err); + } + +/* +* Keep executing queued requests until the queue is empty or +* mmc_swcmdq_execute() asks for new requests by returning +* MMC_BLK_NEW_REQUEST. +*/ +while (mq->qcnt) { +/* +* Re-tuning can only be done when the queue is empty. Recovery +* for MMC_BLK_RETRY will discard the queue and re-queue all +* requests. At the point the queue is empty, re-tuning is +* released and will be done automatically before the next +* mmc_request. +*/ +if (mq->card->host->need_retune) +mmc_swcmdq_recovery (mq,MMC_BLK_RETRY ); + err =mmc_swcmdq_execute (mq, flush, false, !!req); +if (err ==MMC_BLK_NEW_REQUEST ) +return 0 ; +if (err) +mmc_swcmdq_recovery (mq, err); + } + +return 0 ; +} + +static int __mmc_blk_issue_rw_rq (struct mmc_queue *mq,struct request *rqc) {struct mmc_blk_data *md = mq->blkdata;struct mmc_card *card = md->queue.card;
@@ -1802,6 +2380,17 @@ static int mmc_blk_issue_rw_rq(struct mmc_queue *mq, struct request *rqc)return 0 ; } +static int mmc_blk_issue_rw_rq (struct mmc_queue *mq,struct request *req) +{ +struct mmc_blk_data *md = mq->blkdata; +struct mmc_card *card = md->queue.card; + +if (card->ext_csd.cmdq_en) +return mmc_swcmdq_issue_rw_rq (mq, req); +else +return __mmc_blk_issue_rw_rq (mq, req); +} +int mmc_blk_issue_rq (struct mmc_queue *mq,struct request *req) {int ret;
drivers/mmc/card/queue.c +9 -1
@@ -64,6 +64,7 @@ struct mmc_queue_req *mmc_queue_req_find(struct mmc_queue *mq, mqrq->req = req; mq->qcnt +=1 ;__set_bit (mqrq->task_id, &mq->qslots); + mqrq->retry_cnt =0 ;return mqrq; }
@@ -377,7 +378,14 @@ static int __mmc_queue_alloc_shared_queue(struct mmc_card *card, int qdepth)int mmc_queue_alloc_shared_queue (struct mmc_card *card) { -return __mmc_queue_alloc_shared_queue (card,2 ); +int qdepth; + +if (card->ext_csd.cmdq_en) + qdepth = card->ext_csd.cmdq_depth; +else + qdepth =2 ; + +return __mmc_queue_alloc_shared_queue (card, qdepth); }/**
drivers/mmc/card/queue.h +10 -1
@@ -15,9 +15,13 @@ static inline bool mmc_req_is_special(struct request *req)struct mmc_blk_request {struct mmc_request mrq; -struct mmc_command sbc; +union { +struct mmc_command sbc; +struct mmc_command cmd44; + };struct mmc_command cmd;struct mmc_command stop; +struct mmc_command cmd45;struct mmc_data data;int retune_retry_done; };
@@ -31,6 +35,7 @@ struct mmc_queue_req {unsigned int bounce_sg_len;struct mmc_async_req mmc_active;int task_id; +unsigned int retry_cnt; };struct mmc_queue {
@@ -47,6 +52,10 @@ struct mmc_queue {int qdepth;int qcnt;unsigned long qslots; +/* Following are defined for Software Command Queuing */ +unsigned long qsr; +struct mmc_async_req *prepared_areq; +bool qsr_err; };extern int mmc_queue_alloc_shared_queue (struct mmc_card *card);
include/linux/mmc/core.h +1 -0
@@ -25,6 +25,7 @@ enum mmc_blk_status {MMC_BLK_ECC_ERR ,MMC_BLK_NOMEDIUM ,MMC_BLK_NEW_REQUEST , + MMC_BLK_SUCCESS_ERR, /* Success but prevent starting another request */ }; struct mmc_command {
Signature · 1 line
1.9.1
PATCH v9 11/11 mmc: mmc: Enable Software Command Queuing
In reply to Adrian HunterEnable the Command Queue if the host controller supports Software Command Queuing. It is not compatible with Packed Commands, so do not enable that at the same time.
--- drivers/mmc/core/mmc.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
drivers/mmc/core/mmc.c +16 -1
@@ -1765,6 +1765,20 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr, } } +/* Enable Command Queue if supported */ + card->ext_csd.cmdq_en = false; +if (card->ext_csd.cmdq_support && host->caps &MMC_CAP_CMD_DURING_TFR ) { + err =mmc_cmdq_enable (card); +if (err && err != -EBADMSG ) + goto free_card; +if (err) { +pr_warn ("%s: Enabling CMDQ failed\n" , +mmc_hostname (card->host)); + card->ext_csd.cmdq_support = false; + card->ext_csd.cmdq_depth =0 ; + err =0 ; + } + }/* * In some cases (e.g. RPMB or mmc_test), the Command Queue must be * disabled for a time, so a flag is needed to indicate to re-enable the
@@ -1778,7 +1792,8 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr,*/ if (card->ext_csd.max_packed_writes >=3 && card->ext_csd.max_packed_reads >=5 && - host->caps2 &MMC_CAP2_PACKED_CMD ) { + host->caps2 &MMC_CAP2_PACKED_CMD && + !card->ext_csd.cmdq_en) { err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL, EXT_CSD_EXP_EVENTS_CTRL, EXT_CSD_PACKED_EVENT_EN,
Signature · 1 line
1.9.1