Subsystems (6)
Jens Axboe
maintainerHi,
I've been working (on and off) on blk-mq for a while now, and I think it's finally time to polish it off and get it posted for review with inclusion for 3.13. It's fairly well contained and these core bits should not impact any drivers. There's a series of prep patches in front of the meat patch.
You can also find this series here:
git://git.kernel.dk/linux-block.git blk-mq/core
and there's a separate repo with some driver updates here:
git://git.kernel.dk/linux-block.git blk-mq/drivers
which is layered on top of the core bits.
See patch 10 for a description of blk-mq itself and the reasoning behind this new queueing model. There's also been a paper published on results run on the code last year. Note that this is somewhat out of date as the code has moved along since, most notably it should be much faster now. The paper can be found here:
http://kernel.dk/systor13-final18.pdf
and was presented at Systor 2013 earlier this year.
Jens Axboe
maintainerPATCH 1/11 Export __smp_call_function_single()
The blk-mq core and the blk-mq null driver uses it.
--- kernel/smp.c | 1 + 1 file changed, 1 insertion(+)
kernel/smp.c +1 -0
@@ -340,6 +340,7 @@ void __smp_call_function_single(int cpu, struct call_single_data *csd, }put_cpu (); } +EXPORT_SYMBOL (__smp_call_function_single);/** * smp_call_function_many(): Run a function on a set of other CPUs.
Signature · 1 line
1.8.1.2
Re: [PATCH 01/11] Export __smp_call_function_single()
Quoted 1 line · Jens Axboe
The blk-mq core and the blk-mq null driver uses it.
Can't say I really like exporting it as all users will need a major ifdef mess. If at all we'd expose the higher level add to llist and ipi abstraction that both blk-mq and the null_blk driver use.
For now I'd just suggest removing the NULL_IRQ_SOFTIRQ mode from null_blk and decouple that from blk-mq itself. I'll try to cook up such an abstraction in the meantime.
Jens Axboe
maintainerRe: [PATCH 01/11] Export __smp_call_function_single()
Quoted 10 lines · Christoph Hellwig
Quoted 1 line · Jens Axboe
The blk-mq core and the blk-mq null driver uses it.
Can't say I really like exporting it as all users will need a major ifdef mess. If at all we'd expose the higher level add to llist and ipi abstraction that both blk-mq and the
null_blkdriver use.For now I'd just suggest removing the
NULL_IRQ_SOFTIRQmode fromnull_blkand decouple that from blk-mq itself. I'll try to cook up such an abstraction in the meantime.
Agree, lets get a proper interface exported for this from kernel/smp.c instead. I've killed this patch and the softirq mode in null_blk for now.
Signature · 1 line
Jens Axboe
Christoph Hellwig
Reviewed-byRe: [PATCH 01/11] Export __smp_call_function_single()
Quoted 8 lines · Jens Axboe
Quoted 7 lines · Christoph Hellwig
Quoted 1 line · Jens Axboe
The blk-mq core and the blk-mq null driver uses it.
Can't say I really like exporting it as all users will need a major ifdef mess. If at all we'd expose the higher level add to llist and ipi abstraction that both blk-mq and the
null_blkdriver use.
Quoted 7 lines
Quoted 3 lines
For now I'd just suggest removing the
NULL_IRQ_SOFTIRQmode fromnull_blkand decouple that from blk-mq itself. I'll try to cook up such an abstraction in the meantime.Agree, lets get a proper interface exported for this from
kernel/smp.cinstead. I've killed this patch and the softirq mode innull_blkfor now.
Turns out the proper interface is __smp_call_function_single as long as we make sure it's always available. I've got a series that allows us to use it directly without the llist wrapping, but it probably won't be ready for the next merge window.
So I'll take my earlier objection back, instead please add a:
Re: [PATCH 01/11] Export __smp_call_function_single()
In reply to Jens AxboeQuoted 13 lines · Jens Axboe
The blk-mq core and the blk-mq null driver uses it.
Cc Ingo Molnar <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- kernel/smp.c | 1 + 1 file changed, 1 insertion(+)kernel/smp.c +0
-0
kernel/smp.c void __smp_call_function_single(int cpu, struct call_single_data *csd, · line 343
Quoted 4 lines
kernel/smp.c +1
-0@@ -340,6 +340,7 @@ void __smp_call_function_single(int cpu, struct call_single_data *csd, }put_cpu (); } +EXPORT_SYMBOL (__smp_call_function_single);
I agree with exporting it to in-kernel users, but I'd prefer a _GPL export for newly exported symbols that are for in-kernel use.
Thanks,
Ingo
Jens Axboe
maintainerRe: [PATCH 01/11] Export __smp_call_function_single()
Quoted 11 lines · Ingo Molnar
Quoted 8 lines · Jens Axboe
The blk-mq core and the blk-mq null driver uses it.
Cc Ingo Molnar <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- kernel/smp.c | 1 + 1 file changed, 1 insertion(+)
Quoted 12 lines
kernel/smp.c void __smp_call_function_single(int cpu, struct call_single_data *csd, · line 343
Quoted 9 lines
kernel/smp.c +1
-0@@ -340,6 +340,7 @@ void __smp_call_function_single(int cpu, struct call_single_data *csd, }put_cpu (); } +EXPORT_SYMBOL (__smp_call_function_single);I agree with exporting it to in-kernel users, but I'd prefer a
_GPLexport for newly exported symbols that are for in-kernel use.
Agree, it should be _GPL. I will update the patch. Can I add your acked-by?
Signature · 1 line
Jens Axboe
Ingo Molnar
Acked-byRe: [PATCH 01/11] Export __smp_call_function_single()
Quoted 25 lines · Jens Axboe
Quoted 11 lines · Ingo Molnar
Quoted 8 lines · Jens Axboe
The blk-mq core and the blk-mq null driver uses it.
Cc Ingo Molnar <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- kernel/smp.c | 1 + 1 file changed, 1 insertion(+)Quoted 12 lines
kernel/smp.c void __smp_call_function_single(int cpu, struct call_single_data *csd, · line 343
Quoted 9 lines
kernel/smp.c +1
-0@@ -340,6 +340,7 @@ void __smp_call_function_single(int cpu, struct call_single_data *csd, }put_cpu (); } +EXPORT_SYMBOL (__smp_call_function_single);I agree with exporting it to in-kernel users, but I'd prefer a
_GPLexport for newly exported symbols that are for in-kernel use.
Quoted 2 lines
Agree, it should be
_GPL. I will update the patch. Can I add your acked-by?
Sure:
Thanks,
Ingo
Jens Axboe
maintainerPATCH 2/11 smp: don't warn about csd->flags having CSD_FLAG_LOCK cleared for !wait
In reply to Jens Axboeblk-mq reuses the request potentially immediately, since the most cache hot is always given out first. This means that rq->csd could be reused between csd->func() being called and csd_unlock() being called. This isn't a problem, since we never use wait == 1 for the smp call function. Add CSD_FLAG_WAIT to be able to tell the difference, retaining the warning for other cases.
--- kernel/smp.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/smp.c +5 -1
@@ -18,6 +18,7 @@#ifdef CONFIG_USE_GENERIC_SMP_HELPERS enum {CSD_FLAG_LOCK =0x01 , +CSD_FLAG_WAIT =0x02 , }; struct call_function_data {
@@ -124,7 +125,7 @@ static void csd_lock(struct call_single_data *csd)static void csd_unlock (struct call_single_data *csd) { -WARN_ON (!(csd->flags &CSD_FLAG_LOCK )); +WARN_ON ((csd->flags &CSD_FLAG_WAIT ) && !(csd->flags &CSD_FLAG_LOCK ));/* * ensure we're all done before releasing data:
@@ -146,6 +147,9 @@ void generic_exec_single(int cpu, struct call_single_data *csd, int wait)unsigned long flags;int ipi; +if (wait) + csd->flags |=CSD_FLAG_WAIT ; +raw_spin_lock_irqsave (&dst->lock, flags); ipi =list_empty (&dst->list);list_add_tail (&csd->list, &dst->list);
Signature · 1 line
1.8.1.2
Re: [PATCH 02/11] smp: don't warn about csd->flags having CSD_FLAG_LOCK cleared for !wait
Quoted 9 lines · Jens Axboe
blk-mq reuses the request potentially immediately, since the most cache hot is always given out first. This means that
rq->csdcould be reused betweencsd->func()being called andcsd_unlock()being called. This isn't a problem, since we never use wait == 1 for the smp call function. AddCSD_FLAG_WAITto be able to tell the difference, retaining the warning for other cases.Cc Ingo Molnar <[email protected]>Signed-off-by Jens Axboe <[email protected]>
Looks fine, but I'd be tempted to just delete the WARN_ON and add a comment similar to the commit message there.
Jens Axboe
maintainerPATCH 3/11 block: make rq->cmd_flags be 64-bit
In reply to Jens AxboeWe have officially run out of flags in a 32-bit space. Extend it to 64-bit even on 32-bit archs.
--- block/blk-core.c | 4 +-- drivers/block/floppy.c | 4 +-- drivers/scsi/sd.c | 2 +- include/linux/blk_types.h | 66 +++++++++++++++++++++++------------------------ include/linux/blkdev.h | 4 +-- 5 files changed, 40 insertions(+), 40 deletions(-)
block/blk-core.c +2 -2
@@ -174,9 +174,9 @@ void blk_dump_rq_flags(struct request *rq, char *msg) {int bit; -printk (KERN_INFO "%s: dev %s: type=%x, flags=%x\n" , msg, +printk (KERN_INFO "%s: dev %s: type=%x, flags=%llx\n" , msg, rq->rq_disk ? rq->rq_disk->disk_name :"?" , rq->cmd_type, - rq->cmd_flags); + (unsigned long long ) rq->cmd_flags); printk(KERN_INFO " sector %llu, nr/cnr %u/%u\n", (unsigned long long)blk_rq_pos(rq),
drivers/block/floppy.c +2 -2
@@ -2886,9 +2886,9 @@ static void do_fd_request(struct request_queue *q)return ;if (WARN (atomic_read (&usage_count) ==0 , -"warning: usage count=0, current_req=%p sect=%ld type=%x flags=%x\n" , +"warning: usage count=0, current_req=%p sect=%ld type=%x flags=%llx\n" , current_req, (long )blk_rq_pos (current_req), current_req->cmd_type, - current_req->cmd_flags)) + (unsigned long long ) current_req->cmd_flags))return ;if (test_and_set_bit (0 , &fdc_busy)) {
drivers/scsi/sd.c +1 -1
@@ -1002,7 +1002,7 @@ static int sd_prep_fn(struct request_queue *q, struct request *rq) SCpnt->cmnd[0 ] =READ_6 ; SCpnt->sc_data_direction =DMA_FROM_DEVICE ; }else { -scmd_printk (KERN_ERR , SCpnt,"Unknown command %x\n" , rq->cmd_flags); +scmd_printk (KERN_ERR , SCpnt,"Unknown command %llx\n" , (unsigned long long ) rq->cmd_flags); goto out; }
include/linux/blk_types.h +33 -33
77 lines not on this page · raw message
include/linux/blkdev.h +2 -2
@@ -99,7 +99,7 @@ struct request {struct request_queue *q; -unsigned int cmd_flags; +u64 cmd_flags;enum rq_cmd_type_bits cmd_type;unsigned long atomic_flags;
@@ -570,7 +570,7 @@ static inline void queue_flag_clear(unsigned int flag, struct request_queue *q)#define list_entry_rq (ptr) list_entry((ptr), struct request, queuelist) -#define rq_data_dir (rq) ((rq)->cmd_flags & 1) +#define rq_data_dir (rq) (((rq)->cmd_flags & 1) != 0)static inline unsigned int blk_queue_cluster (struct request_queue *q) {
Signature · 1 line
1.8.1.2
Re: [PATCH 03/11] block: make rq->cmd_flags be 64-bit
Quoted 4 lines · Jens Axboe
We have officially run out of flags in a 32-bit space. Extend it to 64-bit even on 32-bit archs.
Signed-off-by Jens Axboe <[email protected]>
Looks good,
Jens Axboe
maintainerPATCH 4/11 block: remove request ref_count
In reply to Jens AxboeFrom: Christoph Hellwig <[email protected]>
This reference count has been around since before git history, but the only place where it's used is in blk_execute_rq, and ther it is entirely useless as it is incremented before submitting the request and decremented in the end_io handler before waking up the submitter thread.
--- block/blk-core.c | 3 --- block/blk-exec.c | 7 ------- include/linux/blkdev.h | 2 -- 3 files changed, 12 deletions(-)
block/blk-core.c +0 -3
@@ -145,7 +145,6 @@ void blk_rq_init(struct request_queue *q, struct request *rq) rq->cmd = rq->__cmd; rq->cmd_len =BLK_MAX_CDB ; rq->tag =-1 ; - rq->ref_count =1 ; rq->start_time = jiffies;set_start_time_ns (rq); rq->part =NULL ;
@@ -1272,8 +1271,6 @@ void __blk_put_request(struct request_queue *q, struct request *req) {if (unlikely (!q))return ; -if (unlikely (--req->ref_count)) -return ;blk_pm_put_request (req);
block/blk-exec.c +0 -7
@@ -24,7 +24,6 @@ static void blk_end_sync_rq(struct request *rq, int error)struct completion *waiting = rq->end_io_data; rq->end_io_data =NULL ; -__blk_put_request (rq->q, rq);/* * complete last, if this is a stack request the process (and thus
@@ -103,12 +102,6 @@ int blk_execute_rq(struct request_queue *q, struct gendisk *bd_disk,int err =0 ;unsigned long hang_check; -/* -* we need an extra reference to the request, so we can look at -* it after io completion -*/ - rq->ref_count++; -if (!rq->sense) {memset (sense,0 ,sizeof (sense)); rq->sense = sense;
include/linux/blkdev.h +0 -2
@@ -160,8 +160,6 @@ struct request {unsigned short ioprio; -int ref_count; -void *special;/* opaque pointer available for LLD use */ char *buffer;/* kaddr of the current segment if available */
Signature · 1 line
1.8.1.2
Jens Axboe
maintainerPATCH 5/11 direct-io: only inc/dec inode->i_dio_count for file systems
In reply to Jens AxboeWe don't need truncate protection for block devices, so add a flag bypassing this cache line dirtying twice for every IO. This easily contributes to 5-10% of the CPU time on high IOPS O_DIRECT testing.
--- fs/block_dev.c | 3 ++- fs/direct-io.c | 14 +++++++++++++- fs/inode.c | 7 ++++++- include/linux/fs.h | 4 ++++ 4 files changed, 25 insertions(+), 3 deletions(-)
fs/block_dev.c +2 -1
@@ -172,7 +172,8 @@ blkdev_direct_IO(int rw, struct kiocb *iocb, const struct iovec *iov,struct inode *inode = file->f_mapping->host;return __blockdev_direct_IO (rw, iocb, inode,I_BDEV (inode), iov, offset, - nr_segs, blkdev_get_block,NULL ,NULL ,0 ); + nr_segs, blkdev_get_block,NULL ,NULL , +DIO_IGNORE_TRUNCATE ); }int __sync_blockdev (struct block_device *bdev,int wait)
fs/direct-io.c +13 -1
@@ -265,7 +265,11 @@ static ssize_t dio_complete(struct dio *dio, loff_t offset, ssize_t ret,if (dio->end_io && dio->result) dio->end_io (dio->iocb, offset, transferred, dio->private); -inode_dio_done (dio->inode); +if (dio->flags &DIO_IGNORE_TRUNCATE ) +__inode_dio_done (dio->inode); +else +inode_dio_done (dio->inode); +if (is_async) {if (dio->rw &WRITE ) {int err;
@@ -1194,6 +1198,14 @@ do_blockdev_direct_IO(int rw, struct kiocb *iocb, struct inode *inode, }/* +* Will be decremented at I/O completion time. For a block device +* we don't need to protect against truncate, so don't increment +* the inode direct IO count. +*/ +if (!(dio->flags &DIO_IGNORE_TRUNCATE )) +atomic_inc (&inode->i_dio_count); + +/* * For file extending writes updating i_size before data * writeouts complete can expose uninitialized blocks. So * even for AIO, we need to wait for i/o to complete before
fs/inode.c +6 -1
@@ -1854,6 +1854,11 @@ void inode_dio_wait(struct inode *inode) }EXPORT_SYMBOL (inode_dio_wait); +void __inode_dio_done (struct inode *inode) +{ +wake_up_bit (&inode->i_state, __I_DIO_WAKEUP); +} +/* * inode_dio_done - signal finish of a direct I/O requests * @inode: inode the direct I/O happens on
@@ -1864,6 +1869,6 @@ EXPORT_SYMBOL(inode_dio_wait);void inode_dio_done (struct inode *inode) {if (atomic_dec_and_test (&inode->i_dio_count)) -wake_up_bit (&inode->i_state, __I_DIO_WAKEUP); +__inode_dio_done (inode); }EXPORT_SYMBOL (inode_dio_done);
include/linux/fs.h +4 -0
@@ -2468,6 +2468,9 @@ enum {/* filesystem does not support filling holes */ DIO_SKIP_HOLES =0x02 , + +/* inode/fs/bdev does not need truncate protection */ + DIO_IGNORE_TRUNCATE = 0x04, };void dio_end_io (struct bio *bio,int error);
@@ -2488,6 +2491,7 @@ static inline ssize_t blockdev_direct_IO(int rw, struct kiocb *iocb,#endif void inode_dio_wait (struct inode *inode); +void __inode_dio_done (struct inode *inode);void inode_dio_done (struct inode *inode);extern const struct file_operations generic_ro_fops;
Signature · 1 line
1.8.1.2
Jens Axboe
maintainerRe: [PATCH 05/11] direct-io: only inc/dec inode->i_dio_count for file systems
Quoted 3 lines · Jens Axboe
We don't need truncate protection for block devices, so add a flag bypassing this cache line dirtying twice for every IO. This easily contributes to 5-10% of the CPU time on high IOPS
O_DIRECTtesting.
There's a merge error in this patch, please find the correct one below.
fs/block_dev.c +2 -1
@@ -172,7 +172,8 @@ blkdev_direct_IO(int rw, struct kiocb *iocb, const struct iovec *iov,struct inode *inode = file->f_mapping->host;return __blockdev_direct_IO (rw, iocb, inode,I_BDEV (inode), iov, offset, - nr_segs, blkdev_get_block,NULL ,NULL ,0 ); + nr_segs, blkdev_get_block,NULL ,NULL , +DIO_IGNORE_TRUNCATE ); }int __sync_blockdev (struct block_device *bdev,int wait)
fs/direct-io.c +10 -3
@@ -265,7 +265,11 @@ static ssize_t dio_complete(struct dio *dio, loff_t offset, ssize_t ret,if (dio->end_io && dio->result) dio->end_io (dio->iocb, offset, transferred, dio->private); -inode_dio_done (dio->inode); +if (dio->flags &DIO_IGNORE_TRUNCATE ) +__inode_dio_done (dio->inode); +else +inode_dio_done (dio->inode); +if (is_async) {if (dio->rw &WRITE ) {int err;
@@ -1223,9 +1227,12 @@ do_blockdev_direct_IO(int rw, struct kiocb *iocb, struct inode *inode, }/* -* Will be decremented at I/O completion time. +* Will be decremented at I/O completion time. For a block device +* we don't need to protect against truncate, so don't increment +* the inode direct IO count. */ -atomic_inc (&inode->i_dio_count); +if (!(dio->flags &DIO_IGNORE_TRUNCATE )) +atomic_inc (&inode->i_dio_count); retval =0 ; sdio.blkbits = blkbits;
fs/inode.c +6 -1
@@ -1854,6 +1854,11 @@ void inode_dio_wait(struct inode *inode) }EXPORT_SYMBOL (inode_dio_wait); +void __inode_dio_done (struct inode *inode) +{ +wake_up_bit (&inode->i_state, __I_DIO_WAKEUP); +} +/* * inode_dio_done - signal finish of a direct I/O requests * @inode: inode the direct I/O happens on
@@ -1864,6 +1869,6 @@ EXPORT_SYMBOL(inode_dio_wait);void inode_dio_done (struct inode *inode) {if (atomic_dec_and_test (&inode->i_dio_count)) -wake_up_bit (&inode->i_state, __I_DIO_WAKEUP); +__inode_dio_done (inode); }EXPORT_SYMBOL (inode_dio_done);
include/linux/fs.h +4 -0
@@ -2468,6 +2468,9 @@ enum {/* filesystem does not support filling holes */ DIO_SKIP_HOLES =0x02 , + +/* inode/fs/bdev does not need truncate protection */ + DIO_IGNORE_TRUNCATE = 0x04, };void dio_end_io (struct bio *bio,int error);
@@ -2488,6 +2491,7 @@ static inline ssize_t blockdev_direct_IO(int rw, struct kiocb *iocb,#endif void inode_dio_wait (struct inode *inode); +void __inode_dio_done (struct inode *inode);void inode_dio_done (struct inode *inode);extern const struct file_operations generic_ro_fops;
Signature · 1 line
Jens Axboe
Re: [PATCH 05/11] direct-io: only inc/dec inode->i_dio_count for file systems
Quoted 6 lines · Jens Axboe
Quoted 3 lines · Jens Axboe
We don't need truncate protection for block devices, so add a flag bypassing this cache line dirtying twice for every IO. This easily contributes to 5-10% of the CPU time on high IOPS
O_DIRECTtesting.There's a merge error in this patch, please find the correct one below.
Btw, I think this patch should not go in with this series but rather via the VFS tree as it's not related to the blk-mq infrastructure at all.
Jens Axboe
maintainerRe: [PATCH 05/11] direct-io: only inc/dec inode->i_dio_count for file systems
Quoted 10 lines · Christoph Hellwig
Quoted 6 lines · Jens Axboe
Quoted 3 lines · Jens Axboe
We don't need truncate protection for block devices, so add a flag bypassing this cache line dirtying twice for every IO. This easily contributes to 5-10% of the CPU time on high IOPS
O_DIRECTtesting.There's a merge error in this patch, please find the correct one below.
Btw, I think this patch should not go in with this series but rather via the VFS tree as it's not related to the blk-mq infrastructure at all.
That'd be fine with me, if Al could pick it up if he agrees with it. Feel free to add an acked/review-by :-)
Signature · 1 line
Jens Axboe
Re: [PATCH 05/11] direct-io: only inc/dec inode->i_dio_count for file systems
Quoted 7 lines · Jens Axboe
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Jens Axboe
Quoted 3 lines · Jens Axboe
We don't need truncate protection for block devices, so add a flag bypassing this cache line dirtying twice for every IO. This easily contributes to 5-10% of the CPU time on high IOPS
O_DIRECTtesting.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
There's a merge error in this patch, please find the correct one below.
Btw, I think this patch should not go in with this series but rather via the VFS tree as it's not related to the blk-mq infrastructure at all.
That'd be fine with me, if Al could pick it up if he agrees with it. Feel free to add an acked/review-by :-)
This patch never got picked up. Can it be?
Jens Axboe
maintainerPATCH 6/11 percpu_counter: make APIs irq safe
In reply to Jens AxboeFrom: Shaohua Li <[email protected]>
In my usage, sometimes the percpu APIs are called with irq locked, sometimes not. lockdep complains there is potential deadlock. Let's always use percpucounter lock in irq safe way. There should be no performance penality, as all those are slow code path.
--- lib/percpu_counter.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
lib/percpu_counter.c +9 -6
@@ -60,14 +60,15 @@ static inline void debug_percpu_counter_deactivate(struct percpu_counter *fbc)void percpu_counter_set (struct percpu_counter *fbc,s64 amount) {int cpu; +unsigned long flags; -raw_spin_lock (&fbc->lock); +raw_spin_lock_irqsave (&fbc->lock, flags);for_each_possible_cpu (cpu) {s32 *pcount =per_cpu_ptr (fbc->counters, cpu); *pcount =0 ; } fbc->count = amount; -raw_spin_unlock (&fbc->lock); +raw_spin_unlock_irqrestore (&fbc->lock, flags); }EXPORT_SYMBOL (percpu_counter_set);
@@ -78,9 +79,10 @@ void __percpu_counter_add(struct percpu_counter *fbc, s64 amount, s32 batch)preempt_disable (); count =__this_cpu_read (*fbc->counters) + amount;if (count >= batch || count <= -batch) { -raw_spin_lock (&fbc->lock); +unsigned long flags; +raw_spin_lock_irqsave (&fbc->lock, flags); fbc->count += count; -raw_spin_unlock (&fbc->lock); +raw_spin_unlock_irqrestore (&fbc->lock, flags);__this_cpu_write (*fbc->counters,0 ); }else {__this_cpu_write (*fbc->counters, count);
@@ -97,14 +99,15 @@ s64 __percpu_counter_sum(struct percpu_counter *fbc) {s64 ret;int cpu; +unsigned long flags; -raw_spin_lock (&fbc->lock); +raw_spin_lock_irqsave (&fbc->lock, flags); ret = fbc->count;for_each_online_cpu (cpu) {s32 *pcount =per_cpu_ptr (fbc->counters, cpu); ret += *pcount; } -raw_spin_unlock (&fbc->lock); +raw_spin_unlock_irqrestore (&fbc->lock, flags);return ret; }EXPORT_SYMBOL (__percpu_counter_sum);
Signature · 1 line
1.8.1.2
Jens Axboe
maintainerPATCH 7/11 percpu_ida: make percpu_ida percpu size/batch configurable
In reply to Jens AxboeFrom: Shaohua Li <[email protected]>
Make percpu_ida percpu size/batch configurable. The block-mq-tag will use it.
After block-mq uses percpu_ida to manage tags, performance is improved. My test is done in a 2 sockets machine, 12 process cross the 2 sockets. So if there is lock contention or ipi, should be stressed heavily. Testing is done for null-blk.
hw_queue_depth nopatch iops patch iops
64 ~800k/s ~1470k/s
2048 ~4470k/s ~4340k/s
--- include/linux/percpu_ida.h | 18 +++++++++++++++++- lib/percpu_ida.c | 28 +++++++++++----------------- 2 files changed, 28 insertions(+), 18 deletions(-)
include/linux/percpu_ida.h +17 -1
@@ -16,6 +16,8 @@ struct percpu_ida {* percpu_ida_init() */ unsigned nr_tags; +unsigned percpu_max_size; +unsigned percpu_batch_size;struct percpu_ida_cpu __percpu *tag_cpu;
@@ -51,10 +53,24 @@ struct percpu_ida { } ____cacheline_aligned_in_smp; }; +/* +* Number of tags we move between the percpu freelist and the global freelist at +* a time +*/ +#define IDA_DEFAULT_PCPU_BATCH_MOVE 32U +/* Max size of percpu freelist, */ +#define IDA_DEFAULT_PCPU_SIZE ((IDA_DEFAULT_PCPU_BATCH_MOVE * 3) / 2) +int percpu_ida_alloc (struct percpu_ida *pool,gfp_t gfp);void percpu_ida_free (struct percpu_ida *pool,unsigned tag);void percpu_ida_destroy (struct percpu_ida *pool); -int percpu_ida_init (struct percpu_ida *pool,unsigned long nr_tags); +int __percpu_ida_init (struct percpu_ida *pool,unsigned long nr_tags, +unsigned long max_size,unsigned long batch_size); +static inline int percpu_ida_init (struct percpu_ida *pool,unsigned long nr_tags) +{ +return __percpu_ida_init (pool, nr_tags,IDA_DEFAULT_PCPU_SIZE , +IDA_DEFAULT_PCPU_BATCH_MOVE ); +}#endif /* __PERCPU_IDA_H__ */
lib/percpu_ida.c +11 -17
@@ -30,15 +30,6 @@#include <linux/spinlock.h> #include <linux/percpu_ida.h> -/* -* Number of tags we move between the percpu freelist and the global freelist at -* a time -*/ -#define IDA_PCPU_BATCH_MOVE 32U - -/* Max size of percpu freelist, */ -#define IDA_PCPU_SIZE ((IDA_PCPU_BATCH_MOVE * 3) / 2) -struct percpu_ida_cpu {/* * Even though this is percpu, we need a lock for tag stealing by remote
@@ -78,7 +69,7 @@ static inline void steal_tags(struct percpu_ida *pool,struct percpu_ida_cpu *remote;for (cpus_have_tags =cpumask_weight (&pool->cpus_have_tags); - cpus_have_tags *IDA_PCPU_SIZE > pool->nr_tags /2 ; + cpus_have_tags * pool->percpu_max_size > pool->nr_tags /2 ; cpus_have_tags--) { cpu =cpumask_next (cpu, &pool->cpus_have_tags);
@@ -123,7 +114,7 @@ static inline void alloc_global_tags(struct percpu_ida *pool, {move_tags (tags->freelist, &tags->nr_free, pool->freelist, &pool->nr_free, -min (pool->nr_free,IDA_PCPU_BATCH_MOVE )); +min (pool->nr_free, pool->percpu_batch_size)); } static inline unsigned alloc_local_tag(struct percpu_ida *pool,
@@ -245,17 +236,17 @@ void percpu_ida_free(struct percpu_ida *pool, unsigned tag)wake_up (&pool->wait); } -if (nr_free ==IDA_PCPU_SIZE ) { +if (nr_free == pool->percpu_max_size) {spin_lock (&pool->lock);/* * Global lock held and irqs disabled, don't need percpu * lock */ -if (tags->nr_free ==IDA_PCPU_SIZE ) { +if (tags->nr_free == pool->percpu_max_size) {move_tags (pool->freelist, &pool->nr_free, tags->freelist, &tags->nr_free, -IDA_PCPU_BATCH_MOVE ); + pool->percpu_batch_size);wake_up (&pool->wait); }
@@ -292,7 +283,8 @@ EXPORT_SYMBOL_GPL(percpu_ida_destroy);* Allocation is percpu, but sharding is limited by nr_tags - for best * performance, the workload should not span more cpus than nr_tags / 128. */ -int percpu_ida_init (struct percpu_ida *pool,unsigned long nr_tags) +int __percpu_ida_init (struct percpu_ida *pool,unsigned long nr_tags, +unsigned long max_size,unsigned long batch_size) {unsigned i, cpu, order;
@@ -301,6 +293,8 @@ int percpu_ida_init(struct percpu_ida *pool, unsigned long nr_tags)init_waitqueue_head (&pool->wait);spin_lock_init (&pool->lock); pool->nr_tags = nr_tags; + pool->percpu_max_size = max_size; + pool->percpu_batch_size = batch_size;/* Guard against overflow */ if (nr_tags > (unsigned )INT_MAX +1 ) {
@@ -319,7 +313,7 @@ int percpu_ida_init(struct percpu_ida *pool, unsigned long nr_tags) pool->nr_free = nr_tags; pool->tag_cpu =__alloc_percpu (sizeof (struct percpu_ida_cpu ) + -IDA_PCPU_SIZE *sizeof (unsigned ), + pool->percpu_max_size *sizeof (unsigned ),sizeof (unsigned ));if (!pool->tag_cpu) goto err;
@@ -332,4 +326,4 @@ err:percpu_ida_destroy (pool);return -ENOMEM ; } -EXPORT_SYMBOL_GPL (percpu_ida_init); +EXPORT_SYMBOL_GPL (__percpu_ida_init);
Signature · 1 line
1.8.1.2
Jens Axboe
maintainerPATCH 8/11 percpu_ida: add percpu_ida_for_each_free
In reply to Jens AxboeFrom: Shaohua Li <[email protected]>
Add a new API to iterate free ids. blk-mq-tag will use it.
Note, this doesn't guarantee to iterate all free ids restrictly. Caller should be aware of this. blk-mq uses it to do sanity check for request timedout, so can tolerate the limitation.
--- include/linux/percpu_ida.h | 4 ++++ lib/percpu_ida.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+)
include/linux/percpu_ida.h +4 -0
@@ -73,4 +73,8 @@ static inline int percpu_ida_init(struct percpu_ida *pool, unsigned long nr_tags IDA_DEFAULT_PCPU_BATCH_MOVE); } +typedef int (*percpu_ida_cb )(unsigned ,void *); +int percpu_ida_for_each_free (struct percpu_ida *pool,percpu_ida_cb fn, +void *data); +#endif /* __PERCPU_IDA_H__ */
lib/percpu_ida.c +44 -0
47 lines not on this page · raw message
Signature · 1 line
1.8.1.2
Jens Axboe
maintainerPATCH 9/11 percpu_ida: add an API to return free tags
In reply to Jens AxboeFrom: Shaohua Li <[email protected]>
Add an API to return free tags, blk-mq-tag will use it.
Note, this just returns a snapshot of free tags number. blk-mq-tag has two usages of it. One is for info output for diagnosis. The other is to quickly check if there are free tags for request dispatch checking. Neither requires very precise.
--- include/linux/percpu_ida.h | 1 + lib/percpu_ida.c | 17 +++++++++++++++++ 2 files changed, 18 insertions(+)
include/linux/percpu_ida.h +1 -0
@@ -77,4 +77,5 @@ typedef int (*percpu_ida_cb)(unsigned, void *);int percpu_ida_for_each_free (struct percpu_ida *pool,percpu_ida_cb fn,void *data); +unsigned percpu_ida_free_tags (struct percpu_ida *pool,int cpu);#endif /* __PERCPU_IDA_H__ */
lib/percpu_ida.c +17 -0
@@ -371,3 +371,20 @@ out:return err; }EXPORT_SYMBOL_GPL (percpu_ida_for_each_free); + +/** +* percpu_ida_free_tags - return free tags number of a specific cpu or global pool +* @pool: pool related +* @cpu: specific cpu or global pool if @cpu == nr_cpu_ids +* +* Note: this just returns a snapshot of free tags number. +*/ +unsigned percpu_ida_free_tags (struct percpu_ida *pool,int cpu) +{ +struct percpu_ida_cpu *remote; +if (cpu == nr_cpu_ids) +return pool->nr_free; + remote =per_cpu_ptr (pool->tag_cpu, cpu); +return remote->nr_free; +} +EXPORT_SYMBOL_GPL (percpu_ida_free_tags);
Signature · 1 line
1.8.1.2
Jens Axboe
maintainerPATCH 10/11 blk-mq: new multi-queue block IO queueing mechanism
In reply to Jens AxboeLinux currently has two models for block devices:
- The classic request_fn based approach, where drivers use struct request units for IO. The block layer provides various helper functionalities to let drivers share code, things like tag management, timeout handling, queueing, etc.
- The "stacked" approach, where a driver squeezes in between the block layer and IO submitter. Since this bypasses the IO stack, driver generally have to manage everything themselves.
With drivers being written for new high IOPS devices, the classic request_fn based driver doesn't work well enough. The design dates back to when both SMP and high IOPS was rare. It has problems with scaling to bigger machines, and runs into scaling issues even on smaller machines when you have IOPS in the hundreds of thousands per device.
The stacked approach is then most often selected as the model for the driver. But this means that everybody has to re-invent everything, and along with that we get all the problems again that the shared approach solved.
This commit introduces blk-mq, block multi queue support. The design is centered around per-cpu queues for queueing IO, which then funnel down into x number of hardware submission queues. We might have a 1:1 mapping between the two, or it might be an N:M mapping. That all depends on what the hardware supports.
blk-mq provides various helper functions, which include:
- Scalable support for request tagging. Most devices need to be able to uniquely identify a request both in the driver and to the hardware. The tagging uses per-cpu caches for freed tags, to enable cache hot reuse.
- Timeout handling without tracking request on a per-device basis. Basically the driver should be able to get a notification, if a request happens to fail.
- Optional support for non 1:1 mappings between issue and submission queues. blk-mq can redirect IO completions to the desired location.
- Support for per-request payloads. Drivers almost always need to associate a request structure with some driver private command structure. Drivers can tell blk-mq this at init time, and then any request handed to the driver will have the required size of memory associated with it.
- Support for merging of IO, and plugging. The stacked model gets neither of these. Even for high IOPS devices, merging sequential IO reduces per-command overhead and thus increases bandwidth.
For now, this is provided as a potential 3rd queueing model, with the hope being that, as it matures, it can replace both the classic and stacked model. That would get us back to having just 1 real model for block devices, leaving the stacked approach to dm/md devices (as it was originally intended).
Contributions in this patch from the following people:
Shaohua Li <[email protected]>
Alexander Gordeev <[email protected]>
Christoph Hellwig <[email protected]>
Mike Christie <[email protected]>
Matias Bjorling <[email protected]>
Jeff Moyer <[email protected]>
--- block/Makefile | 5 +- block/blk-core.c | 142 +++-- block/blk-exec.c | 7 + block/blk-flush.c | 154 ++++- block/blk-mq-cpu.c | 93 +++ block/blk-mq-cpumap.c | 108 ++++ block/blk-mq-sysfs.c | 384 ++++++++++++ block/blk-mq-tag.c | 204 +++++++ block/blk-mq-tag.h | 27 + block/blk-mq.c | 1480 +++++++++++++++++++++++++++++++++++++++++++++ block/blk-mq.h | 52 ++ block/blk-sysfs.c | 13 + block/blk-timeout.c | 73 ++- block/blk.h | 17 + include/linux/bio.h | 2 + include/linux/blk-mq.h | 182 ++++++ include/linux/blk_types.h | 2 + include/linux/blkdev.h | 54 +- 18 files changed, 2890 insertions(+), 109 deletions(-) create mode 100644 block/blk-mq-cpu.c create mode 100644 block/blk-mq-cpumap.c create mode 100644 block/blk-mq-sysfs.c create mode 100644 block/blk-mq-tag.c create mode 100644 block/blk-mq-tag.h create mode 100644 block/blk-mq.c create mode 100644 block/blk-mq.h create mode 100644 include/linux/blk-mq.h
block/Makefile +3 -2
@@ -5,8 +5,9 @@ obj-$(CONFIG_BLOCK) := elevator.o blk-core.o blk-tag.o blk-sysfs.o \ blk-flush.o blk-settings.o blk-ioc.o blk-map.o \ blk-exec.o blk-merge.o blk-softirq.o blk-timeout.o \ - blk-iopoll.o blk-lib.o ioctl.o genhd.o scsi_ioctl.o \ - partition-generic.o partitions/ + blk-iopoll.o blk-lib.o blk-mq.o blk-mq-tag.o \ + blk-mq-sysfs.o blk-mq-cpu.o blk-mq-cpumap.o ioctl.o \ + genhd.o scsi_ioctl.o partition-generic.o partitions/ obj-$(CONFIG_BLK_DEV_BSG) += bsg.o obj-$(CONFIG_BLK_DEV_BSGLIB) += bsg-lib.o
block/blk-core.c +84 -58
285 lines not on this page · raw message
block/blk-exec.c +7 -0
@@ -5,6 +5,7 @@#include <linux/module.h> #include <linux/bio.h> #include <linux/blkdev.h> +#include <linux/blk-mq.h> #include <linux/sched/sysctl.h> #include "blk.h"
@@ -58,6 +59,12 @@ void blk_execute_rq_nowait(struct request_queue *q, struct gendisk *bd_disk, rq->rq_disk = bd_disk; rq->end_io = done; + +if (q->mq_ops) { +blk_mq_insert_request (q, rq, true); +return ; + } +/* * need to check this before __blk_run_queue(), because rq can * be freed before that returns.
block/blk-flush.c +139 -15
271 lines not on this page · raw message
block/blk-mq-cpu.c new file +93 -0
93 lines not on this page · raw message
block/blk-mq-cpumap.c new file +108 -0
108 lines not on this page · raw message
block/blk-mq-sysfs.c new file +384 -0
384 lines not on this page · raw message
block/blk-mq-tag.c new file +204 -0
204 lines not on this page · raw message
block/blk-mq-tag.h new file +27 -0
@@ -0,0 +1,27 @@ +#ifndef INT_BLK_MQ_TAG_H +#define INT_BLK_MQ_TAG_H + +struct blk_mq_tags ; + +extern struct blk_mq_tags *blk_mq_init_tags (unsigned int nr_tags,unsigned int reserved_tags,int node); +extern void blk_mq_free_tags (struct blk_mq_tags *tags); + +extern unsigned int blk_mq_get_tag (struct blk_mq_tags *tags,gfp_t gfp,bool reserved); +extern void blk_mq_wait_for_tags (struct blk_mq_tags *tags); +extern void blk_mq_put_tag (struct blk_mq_tags *tags,unsigned int tag); +extern void blk_mq_tag_busy_iter (struct blk_mq_tags *tags,void (*fn)(void *data,unsigned long *),void *data); +extern bool blk_mq_has_free_tags (struct blk_mq_tags *tags); +extern ssize_t blk_mq_tag_sysfs_show (struct blk_mq_tags *tags,char *page); + +enum { +BLK_MQ_TAG_CACHE_MIN =1 , +BLK_MQ_TAG_CACHE_MAX =64 , +}; + +enum { +BLK_MQ_TAG_FAIL =-1U , +BLK_MQ_TAG_MIN =BLK_MQ_TAG_CACHE_MIN , +BLK_MQ_TAG_MAX =BLK_MQ_TAG_FAIL -1 , +}; + +#endif
block/blk-mq.c new file +1480 -0
1,480 lines not on this page · raw message
block/blk-mq.h new file +52 -0
52 lines not on this page · raw message
block/blk-sysfs.c +13 -0
@@ -7,6 +7,7 @@#include <linux/bio.h> #include <linux/blkdev.h> #include <linux/blktrace_api.h> +#include <linux/blk-mq.h> #include "blk.h" #include "blk-cgroup.h"
@@ -542,6 +543,11 @@ static void blk_release_queue(struct kobject *kobj)if (q->queue_tags)__blk_queue_free_tags (q); +percpu_counter_destroy (&q->mq_usage_counter); + +if (q->mq_ops) +blk_mq_free_queue (q); +blk_trace_shutdown (q);bdi_destroy (&q->backing_dev_info);
@@ -575,6 +581,7 @@ int blk_register_queue(struct gendisk *disk)* bypass from queue allocation. */ blk_queue_bypass_end (q); +queue_flag_set_unlocked (QUEUE_FLAG_INIT_DONE , q); ret =blk_trace_init_sysfs (dev);if (ret)
@@ -588,6 +595,9 @@ int blk_register_queue(struct gendisk *disk)kobject_uevent (&q->kobj,KOBJ_ADD ); +if (q->mq_ops) +blk_mq_register_disk (disk); +if (!q->request_fn)return 0 ;
@@ -610,6 +620,9 @@ void blk_unregister_queue(struct gendisk *disk)if (WARN_ON (!q))return ; +if (q->mq_ops) +blk_mq_unregister_disk (disk); +if (q->request_fn)elv_unregister_queue (q);
block/blk-timeout.c +46 -27
23 lines not on this page · raw message
include/linux/blk-mq.h new file +182 -0
182 lines not on this page · raw message
include/linux/blkdev.h +47 -7
54 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
1.8.1.2
Part of the diff is not shown here. View the raw message.
Re: [PATCH 10/11] blk-mq: new multi-queue block IO queueing mechanism
block/blk-mq-sysfs.c line 291
Quoted 4 lines · Jens Axboe
block/blk-mq-sysfs.c new file +4
-0+static struct blk_mq_hw_ctx_sysfs_entry blk_mq_hw_sysfs_ipi = { + .attr = {.name ="ipi_redirect" , .mode =S_IRUGO |S_IWUGO }, + .show = blk_mq_hw_sysfs_ipi_show, + .store = blk_mq_hw_sysfs_ipi_store,
Is having this world writable a good idea ?
Dave
Jens Axboe
maintainerRe: [PATCH 10/11] blk-mq: new multi-queue block IO queueing mechanism
Quoted 7 lines · Dave Jones
block/blk-mq-sysfs.c line 291
Quoted 4 lines · Jens Axboe
block/blk-mq-sysfs.c new file +4
-0+static struct blk_mq_hw_ctx_sysfs_entry blk_mq_hw_sysfs_ipi = { + .attr = {.name ="ipi_redirect" , .mode =S_IRUGO |S_IWUGO }, + .show = blk_mq_hw_sysfs_ipi_show, + .store = blk_mq_hw_sysfs_ipi_store,Is having this world writable a good idea ?
Good catch, that's a typo (should be S_IWUSR). Will fix up.
Signature · 1 line
Jens Axboe
Re: [PATCH 10/11] blk-mq: new multi-queue block IO queueing mechanism
In reply to Jens AxboeLooking forward to get this in and have a reasonale API for modern block drivers instead of using the bio API.
Jens Axboe
maintainerPATCH 11/11 null_blk: multi queue aware block test driver
In reply to Jens AxboeA driver that simply completes IO it receives, it does no transfers. Written to fascilitate testing of the blk-mq code. It supports various module options to use either bio queueing, rq queueing, or mq mode.
--- drivers/block/Kconfig | 3 + drivers/block/Makefile | 1 + drivers/block/null_blk.c | 638 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 642 insertions(+) create mode 100644 drivers/block/null_blk.c
drivers/block/Kconfig +3 -0
@@ -15,6 +15,9 @@ menuconfig BLK_DEV if BLK_DEV +config BLK_DEV_NULL_BLK + tristate "Null test block driver" + config BLK_DEV_FD tristate "Normal floppy disk support" depends on ARCH_MAY_HAVE_PC_FDC
drivers/block/Makefile +1 -0
@@ -41,6 +41,7 @@ obj-$(CONFIG_BLK_DEV_RBD) += rbd.o obj-$(CONFIG_BLK_DEV_PCIESSD_MTIP32XX) += mtip32xx/ obj-$(CONFIG_BLK_DEV_RSXX) += rsxx/ +obj-$(CONFIG_BLK_DEV_NULL_BLK) += null_blk.o nvme-y := nvme-core.o nvme-scsi.o swim_mod-y := swim.o swim_asm.o
drivers/block/null_blk.c new file +638 -0
638 lines not on this page · raw message
Signature · 1 line
1.8.1.2