Introduce my_cpu_ptr/__my_cpu_ptr
Subsystems (10)
cover Introduce my_cpu_ptr/__my_cpu_ptr
The problem with per_cpu_ptr(x, smp_processor_id) is that it requires an array lookup to find the offset for the cpu. Processors typically have the offset for the current cpu area already in some kind of (arch dependent) efficiently accessible register or memory location.
We can use the per cpu offset instead of doing the array lookup to speed up the determination of the address of the percpu variable. This is particularly significant because these lookups occur in performance critical paths of the core kernel.
This optimization is a prerequiste to the introduction of per processor atomic operations for the core code. Atomic per processor operations implicitly do the offset calculation to the current per cpu area in a single instruction. The locations touched by this patchset are potential candidates for atomic per cpu operations.
The patchset does not cover arch code nor exotic cases where the current cpu is determined before the call into a function.
PATCH 1/5 Introduce my_cpu_ptr()
my_cpu_ptr(xx) = per_cpu_ptr(xx, smp_processor_id).
The problem with per_cpu_ptr(x, smp_processor_id) is that it requires an array lookup to find the offset for the cpu. Processors typically have the offset for the current cpu area in some kind of (arch dependent) efficiently accessible register or memory location.
We can use that instead of doing the array lookup to speed up the determination of the addressof the percpu variable. This is particularly significant because these lookups occur in performance critical paths of the core kernel.
This optimization is a prerequiste to the introduction of per processor atomic operations for the core code. Atomic per processor operations implicitly do the offset calculation to the current per cpu area in a single instruction. All the locations touched by this patchset are potential candidates for atomic per cpu operations.
my_cpu_ptr comes in two flavors. The preemption context matters since we are referring the the currently executing processor. In many cases we must insure that the processor does not change while a code segment is executed.
__my_cpu_ptr -> Do not check for preemption contextmy_cpu_ptr -> Check preemption context
--- include/linux/percpu.h | 11 +++++++++++ 1 file changed, 11 insertions(+)
include/linux/percpu.h +11 -0
@@ -78,6 +78,9 @@ extern ssize_t __init pcpu_embed_first_c*/ #define per_cpu_ptr (ptr, cpu) SHIFT_PERCPU_PTR((ptr), per_cpu_offset((cpu))) +#define my_cpu_ptr (ptr) SHIFT_PERCPU_PTR(ptr, my_cpu_offset) +#define __my_cpu_ptr (ptr) SHIFT_PERCPU_PTR(ptr, __my_cpu_offset) +extern void *__alloc_reserved_percpu (size_t size,size_t align);#else /* CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */
@@ -94,6 +97,12 @@ struct percpu_data { (__typeof__ (ptr ))__p->ptrs[(cpu)]; \ }) +#define my_cpu_ptr (ptr) \ + per_cpu_ptr(ptr, smp_processor_id()) + +#define __my_cpu_ptr (ptr) \ + per_cpu_ptr(ptr, raw_smp_processor_id()) +#endif /* CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */ extern void *__alloc_percpu (size_t size,size_t align);
@@ -102,6 +111,8 @@ extern void free_percpu(void *__pdata);#else /* CONFIG_SMP */ #define per_cpu_ptr (ptr, cpu) ({ (void)(cpu); (ptr); }) +#define my_cpu_ptr (ptr) per_cpu_ptr(ptr, 0) +#define __my_cpu_ptr (ptr) my_cpu_ptr(ptr)static inline void *__alloc_percpu (size_t size,size_t align) {
Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()
Quoted 1 line · On Thu, 28 May 2009 03:16:59 am [email protected] wrote:
my_cpu_ptr(xx)=per_cpu_ptr(xx, smp_processor_id).
I had this implemented earlier as as get_cpu_ptr()/__get_cpu_ptr(), to match get_cpu_var() / __get_cpu_var().
But other than that nomenclature quibble, it looks fine!
Thanks,
Rusty.
Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()
Quoted 5 lines · Rusty Russell
Quoted 1 line · On Thu, 28 May 2009 03:16:59 am [email protected] wrote:
my_cpu_ptr(xx)=per_cpu_ptr(xx, smp_processor_id).I had this implemented earlier as as get_cpu_ptr()/__get_cpu_ptr(), to match
get_cpu_var()/__get_cpu_var().
Have not seen it but it would be a bit confusing since we already have get_cpu* which must be paired with put_cpu* because of the refcount taking (get_cpu_var and get_cpu). get_cpu_ptr() would not have to be paired.
Better use a different name.
my_cpu_ptr came from my_cpu_offset:
#define my_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, my_cpu_offset) #define __my_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, __my_cpu_offset)
Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()
Quoted 11 lines · Christoph Lameter
Quoted 5 lines · Rusty Russell
Quoted 1 line · On Thu, 28 May 2009 03:16:59 am [email protected] wrote:
my_cpu_ptr(xx)=per_cpu_ptr(xx, smp_processor_id).I had this implemented earlier as as get_cpu_ptr()/__get_cpu_ptr(), to match
get_cpu_var()/__get_cpu_var().Have not seen it but it would be a bit confusing since we already have get_cpu* which must be paired with put_cpu* because of the refcount taking (
get_cpu_varandget_cpu).get_cpu_ptr()would not have to be paired.
To clarify, get_cpu_ptr() would be paired with put_cpu_ptr(). __get_cpu_ptr() would be the "raw" one:
#define get_cpu_ptr(xx) per_cpu_ptr(xx, get_cpu()) #define __get_cpu_ptr(xx) per_cpu_ptr(xx, smp_processor_id())
Quoted 3 lines
Better use a different name.
my_cpu_ptrcame frommy_cpu_offset:
Yep, but that's not as widely exposed as get_cpu & get_cpu_var.
Cheers,
Rusty.
Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()
Quoted 10 lines · Rusty Russell
Quoted 4 lines
Have not seen it but it would be a bit confusing since we already have get_cpu* which must be paired with put_cpu* because of the refcount taking (
get_cpu_varandget_cpu).get_cpu_ptr()would not have to be paired.To clarify,
get_cpu_ptr()would be paired withput_cpu_ptr().__get_cpu_ptr()would be the "raw" one:#define
get_cpu_ptr(xx)per_cpu_ptr(xx, get_cpu())#define__get_cpu_ptr(xx)per_cpu_ptr(xx, smp_processor_id())
Hmmm.. That would be a major change in semantics. I tend to think about these ops more as atomic like operations rather than preempt sections. The operation to obtain a the current instance for a given processor is an operation done in a given preemption context. The get_cpu_ptr() approach establishes the preemption context during the same operation.
That would mean use
*get_cpu_ptr(per_cpu_ptr1)++ *__get_cpu_ptr(per_cpu_ptr2)++ put_cpu_ptr()
instead of
preempt_disable() *__this_cpu_ptr(per_cpu_ptr1)++ *__this_cpu_ptr(per_cpu_ptr2)++ preempt_enable().
And for context in which we know that preemption is off
*__get_cpu_ptr(per_cpu_ptr1)++ *__get_cpu_ptr(per_cpu_ptr2)++
vs
*__this_cpu_ptr(per_cpu_ptr2)++ *__this_cpu_ptr(per_cpu_ptr2)++
So its the same
How would that look for atomic per cpu ops?
get_cpu_ptr_inc(per_cpu_ptr1); __get_cpu_ptr_inc(per_cpu_ptr2) put_cpu_ptr()
vs.
this_cpu_inc(per_cpu_ptr1) this_cpu_inc(per_cpu_ptr2)
and known preemption off context
__get_cpu_ptr_inc(per_cpu_ptr1) __get_cpu_ptr_inc(per_cpu_ptr2)
vs
__this_cpu_inc(percpu_ptr1) __this_cpu_inc(percpu_ptr2)
Same again
Seems to be added complexity to also push a change of preemption into these operations. I'd rather keep the preempt status unchanged by these operations in particular because some counters may need interrupt protection on some platforms and not on others. We will likely need some additional operations like
this_cpu_inc_irqsafe( ....)
that will require an irq off / on on some platforms and be a simple increment on others.
Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()
Quoted 6 lines · Christoph Lameter
Quoted 5 lines · Rusty Russell
Quoted 4 lines
Have not seen it but it would be a bit confusing since we already have get_cpu* which must be paired with put_cpu* because of the refcount taking (
get_cpu_varandget_cpu).get_cpu_ptr()would not have to be paired.
Quoted 7 lines
Quoted 5 lines
To clarify,
get_cpu_ptr()would be paired withput_cpu_ptr().__get_cpu_ptr()would be the "raw" one:#define
get_cpu_ptr(xx)per_cpu_ptr(xx, get_cpu())#define__get_cpu_ptr(xx)per_cpu_ptr(xx, smp_processor_id())Hmmm.. That would be a major change in semantics.
It's exactly like get_cpu_var. For better or worse, let's not invent YA new convention.
Quoted 10 lines
How would that look for atomic per cpu ops?
get_cpu_ptr_inc(per_cpu_ptr1); __get_cpu_ptr_inc(per_cpu_ptr2) put_cpu_ptr()vs.
this_cpu_inc(per_cpu_ptr1) this_cpu_inc(per_cpu_ptr2)
Well, get_* doesn't really make sense for any function which doesn't return a value.
So that name question doesn't really have a clear convention answer: we could re-use cpu_local_inc() since I think we decided to kill local_t. I slightly prefer it over "this_cpu_*" since we're not actually doing anything to the cpu itself, but I don't think anyone will get too confused and think that after this executes their CPU will be stepping 11. :)
Thanks,
Rusty.
Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()
Quoted 4 lines · Rusty Russell
Quoted 1 line
Hmmm.. That would be a major change in semantics.
It's exactly like
get_cpu_var. For better or worse, let's not invent YA new convention.
It hink we are doing that if we go the get_cpu_ptr line. this_cpu_ptr starts from per_cpu_ptr() and its straightforward as a special case of per_cpu_ptr.
Quoted 5 lines
So that name question doesn't really have a clear convention answer: we could re-use
cpu_local_inc()since I think we decided to killlocal_t. I slightly prefer it over "this_cpu_*" since we're not actually doing anything to the cpu itself, but I don't think anyone will get too confused and think that after this executes their CPU will be stepping 11. :)
local is loaded with meaning by glibc where it means thread local. The variables here are cpu specific, not task specific.
Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()
In reply to Rusty RussellQuoted 7 lines · Rusty Russell
Quoted 1 line · On Thu, 28 May 2009 03:16:59 am [email protected] wrote:
my_cpu_ptr(xx)=per_cpu_ptr(xx, smp_processor_id).I had this implemented earlier as as get_cpu_ptr()/__get_cpu_ptr(), to match
get_cpu_var()/__get_cpu_var().But other than that nomenclature quibble, it looks fine!
my_ seems a very odd naming. We have a lot of this_cpu naming for the current cpu i nthe ctree, so I would suggest sticking to that.
Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()
Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Rusty Russell
Quoted 1 line · On Thu, 28 May 2009 03:16:59 am [email protected] wrote:
my_cpu_ptr(xx)=per_cpu_ptr(xx, smp_processor_id).I had this implemented earlier as as get_cpu_ptr()/__get_cpu_ptr(), to match
get_cpu_var()/__get_cpu_var().But other than that nomenclature quibble, it looks fine!
my_seems a very odd naming. We have a lot ofthis_cpunaming for the current cpu i nthe ctree, so I would suggest sticking to that.
So this_cpu is taken. I used THIS_CPU in earlier version but got complaints about uppercase use. Is
this_cpu_ptr()
and
__this_cpu_ptr()
ok?
Tejun Heo
maintainerRe: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()
Quoted 15 lines · Christoph Lameter
Quoted 9 lines · Christoph Hellwig
Quoted 6 lines · Rusty Russell
Quoted 1 line · On Thu, 28 May 2009 03:16:59 am [email protected] wrote:
my_cpu_ptr(xx)=per_cpu_ptr(xx, smp_processor_id).I had this implemented earlier as as get_cpu_ptr()/__get_cpu_ptr(), to match
get_cpu_var()/__get_cpu_var().But other than that nomenclature quibble, it looks fine!
my_seems a very odd naming. We have a lot ofthis_cpunaming for the current cpu i nthe ctree, so I would suggest sticking to that.So
this_cpuis taken. I usedTHIS_CPUin earlier version but got complaints about uppercase use. Is
Quoted 7 lines
this_cpu_ptr()and
__this_cpu_ptr()ok?
Yeap, those look fine to me. Sans the renaming, the series generally looks good but I didn't review each conversion.
For the first patch, Acked-by: Tejun Heo <[email protected]>
Thanks.
Signature · 1 line
tejun
PATCH 2/5 Straight transformations
In reply to [email protected]Use my_cpu_ptr and __my_cpu_ptr in locations where straight transformations are possible because per_cpu_ptr is used with either smp_processor_id() or raw_smp_processor_id().
--- drivers/infiniband/hw/ehca/ehca_irq.c | 3 +-- drivers/net/chelsio/sge.c | 5 ++--- drivers/net/loopback.c | 2 +- fs/ext4/mballoc.c | 2 +- include/net/neighbour.h | 2 +- include/net/netfilter/nf_conntrack.h | 4 ++-- include/net/netfilter/nf_conntrack_ecache.h | 2 +- include/net/snmp.h | 4 ++-- 8 files changed, 11 insertions(+), 13 deletions(-)
drivers/net/chelsio/sge.c +2 -3
@@ -1378,7 +1378,7 @@ static void sge_rx(struct sge *sge, stru }__skb_pull (skb,sizeof (*p)); - st =per_cpu_ptr (sge->port_stats[p->iff],smp_processor_id ()); + st =my_cpu_ptr (sge->port_stats[p->iff]); skb->protocol =eth_type_trans (skb, adapter->port[p->iff].dev); if ((adapter->flags & RX_CSUM_ENABLED) && p->csum == 0xffff &&
@@ -1780,8 +1780,7 @@ int t1_start_xmit(struct sk_buff *skb, s {struct adapter *adapter = dev->ml_priv;struct sge *sge = adapter->sge; -struct sge_port_stats *st =per_cpu_ptr (sge->port_stats[dev->if_port], -smp_processor_id ()); +struct sge_port_stats *st =my_cpu_ptr (sge->port_stats[dev->if_port]);struct cpl_tx_pkt *cpl;struct sk_buff *orig_skb = skb;int ret;
drivers/net/loopback.c +1 -1
@@ -78,7 +78,7 @@ static int loopback_xmit(struct sk_buff/* it's OK to use per_cpu_ptr() because BHs are off */ pcpu_lstats = dev->ml_priv; - lb_stats =per_cpu_ptr (pcpu_lstats,smp_processor_id ()); + lb_stats =my_cpu_ptr (pcpu_lstats); lb_stats->bytes += skb->len; lb_stats->packets++;
fs/ext4/mballoc.c +1 -1
@@ -4210,7 +4210,7 @@ static void ext4_mb_group_or_file(struct* per cpu locality group is to reduce the contention between block * request from multiple CPUs. */ - ac->ac_lg =per_cpu_ptr (sbi->s_locality_groups,raw_smp_processor_id ()); + ac->ac_lg =__my_cpu_ptr (sbi->s_locality_groups);/* we're going to use group allocation */ ac->ac_flags |=EXT4_MB_HINT_GROUP_ALLOC ;
include/net/neighbour.h +1 -1
@@ -92,7 +92,7 @@ struct neigh_statistics#define NEIGH_CACHE_STAT_INC (tbl, field) \ do { \ preempt_disable(); \ - (per_cpu_ptr((tbl)->stats, smp_processor_id())->field)++; \ + (my_cpu_ptr((tbl)->stats)->field)++; \ preempt_enable(); \ } while (0)
include/net/netfilter/nf_conntrack.h +2 -2
@@ -292,11 +292,11 @@ extern unsigned int nf_conntrack_htable_extern unsigned int nf_conntrack_max;#define NF_CT_STAT_INC (net, count) \ - (per_cpu_ptr((net)->ct.stat, raw_smp_processor_id())->count++) + (__my_cpu_ptr((net)->ct.stat)->count++)#define NF_CT_STAT_INC_ATOMIC (net, count) \ do { \ local_bh_disable(); \ - per_cpu_ptr((net)->ct.stat, raw_smp_processor_id())->count++; \ + __my_cpu_ptr((net)->ct.stat)->count++; \ local_bh_enable(); \ } while (0)
include/net/netfilter/nf_conntrack_ecache.h +1 -1
@@ -39,7 +39,7 @@ nf_conntrack_event_cache(enum ip_conntrastruct nf_conntrack_ecache *ecache;local_bh_disable (); - ecache =per_cpu_ptr (net->ct.ecache,raw_smp_processor_id ()); + ecache =__my_cpu_ptr (net->ct.ecache);if (ct != ecache->ct)__nf_ct_event_cache_init (ct); ecache->events |= event;
drivers/infiniband/hw/ehca/ehca_irq.c +1 -2
@@ -827,8 +827,7 @@ static void __cpuinit take_over_work(str cq = list_entry(cct->cq_list.next, struct ehca_cq, entry);list_del (&cq->entry); -__queue_comp_task (cq,per_cpu_ptr (pool->cpu_comp_tasks, -smp_processor_id ())); +__queue_comp_task (cq,my_cpu_ptr (pool->cpu_comp_tasks)); }spin_unlock_irqrestore (&cct->task_lock, flags_cct);
include/net/snmp.h +2 -2
@@ -137,7 +137,7 @@ struct linux_xfrm_mib {#define SNMP_STAT_USRPTR (name) (name[1])#define SNMP_INC_STATS_BH (mib, field) \ - (per_cpu_ptr(mib[0], raw_smp_processor_id())->mibs[field]++) + (__my_cpu_ptr(mib[0])->mibs[field]++)#define SNMP_INC_STATS_USER (mib, field) \ do { \ per_cpu_ptr(mib[1], get_cpu())->mibs[field]++; \
@@ -154,7 +154,7 @@ struct linux_xfrm_mib {put_cpu (); \ } while (0)#define SNMP_ADD_STATS_BH (mib, field, addend) \ - (per_cpu_ptr(mib[0], raw_smp_processor_id())->mibs[field] += addend) + (__my_cpu_ptr(mib[0])->mibs[field] += addend)#define SNMP_ADD_STATS_USER (mib, field, addend) \ do { \ per_cpu_ptr(mib[1], get_cpu())->mibs[field] += addend; \
PATCH 3/5 Elimninate get/put_cpu
In reply to [email protected]There are cases where we can use my_cpu_ptr and as the result of using my_cpu_ptr we no longer need to determine the current executing cpu.
In those places no get/put_cpu combination is needed anymore. The local cpu variable can be eliminated.
Preemption still needs to be disabled and enabled since the modifications of the per cpu variables is not atomic. There may be multiple per cpu variables modified and those must all be from the same processor.
--- drivers/dma/dmaengine.c | 34 +++++++++++++++------------------- drivers/net/veth.c | 7 +++---- fs/nfs/iostat.h | 21 +++++++++------------ include/net/snmp.h | 31 +++++++++++++++++-------------- 4 files changed, 44 insertions(+), 49 deletions(-)
drivers/dma/dmaengine.c +15 -19
@@ -327,11 +327,10 @@ arch_initcall(dma_channel_table_init);struct dma_chan *dma_find_channel (enum dma_transaction_type tx_type) {struct dma_chan *chan; -int cpu; - cpu =get_cpu (); - chan =per_cpu_ptr (channel_table[tx_type], cpu)->chan; -put_cpu (); +preempt_disable () + chan = my_cpu_ptr(channel_table[tx_type])->chan; +preempt_enable ();return chan; }
@@ -803,7 +802,6 @@ dma_async_memcpy_buf_to_buf(struct dma_cstruct dma_async_tx_descriptor *tx;dma_addr_t dma_dest, dma_src;dma_cookie_t cookie; -int cpu;unsigned long flags; dma_src =dma_map_single (dev->dev, src, len,DMA_TO_DEVICE );
@@ -822,10 +820,10 @@ dma_async_memcpy_buf_to_buf(struct dma_c tx->callback =NULL ; cookie = tx->tx_submit (tx); - cpu =get_cpu (); -per_cpu_ptr (chan->local, cpu)->bytes_transferred += len; -per_cpu_ptr (chan->local, cpu)->memcpy_count++; -put_cpu (); +preempt_disable (); +my_cpu_ptr (chan->local)->bytes_transferred += len; +my_cpu_ptr (chan->local)->memcpy_count++; +preempt_enable ();return cookie; }
@@ -852,7 +850,6 @@ dma_async_memcpy_buf_to_pg(struct dma_chstruct dma_async_tx_descriptor *tx;dma_addr_t dma_dest, dma_src;dma_cookie_t cookie; -int cpu;unsigned long flags; dma_src =dma_map_single (dev->dev, kdata, len,DMA_TO_DEVICE );
@@ -869,10 +866,10 @@ dma_async_memcpy_buf_to_pg(struct dma_ch tx->callback =NULL ; cookie = tx->tx_submit (tx); - cpu =get_cpu (); -per_cpu_ptr (chan->local, cpu)->bytes_transferred += len; -per_cpu_ptr (chan->local, cpu)->memcpy_count++; -put_cpu (); +preempt_disable (); +my_cpu_ptr (chan->local)->bytes_transferred += len; +my_cpu_ptr (chan->local)->memcpy_count++; +preempt_enable ();return cookie; }
@@ -901,7 +898,6 @@ dma_async_memcpy_pg_to_pg(struct dma_chastruct dma_async_tx_descriptor *tx;dma_addr_t dma_dest, dma_src;dma_cookie_t cookie; -int cpu;unsigned long flags; dma_src =dma_map_page (dev->dev, src_pg, src_off, len,DMA_TO_DEVICE );
@@ -919,10 +915,10 @@ dma_async_memcpy_pg_to_pg(struct dma_cha tx->callback =NULL ; cookie = tx->tx_submit (tx); - cpu =get_cpu (); -per_cpu_ptr (chan->local, cpu)->bytes_transferred += len; -per_cpu_ptr (chan->local, cpu)->memcpy_count++; -put_cpu (); +preempt_disable (); +my_cpu_ptr (chan->local)->bytes_transferred += len; +my_cpu_ptr (chan->local)->memcpy_count++; +preempt_disable ();return cookie; }
drivers/net/veth.c +3 -4
@@ -153,7 +153,7 @@ static int veth_xmit(struct sk_buff *skbstruct net_device *rcv =NULL ;struct veth_priv *priv, *rcv_priv;struct veth_net_stats *stats, *rcv_stats; -int length, cpu; +int length;skb_orphan (skb);
@@ -161,9 +161,8 @@ static int veth_xmit(struct sk_buff *skb rcv = priv->peer; rcv_priv =netdev_priv (rcv); - cpu =smp_processor_id (); - stats =per_cpu_ptr (priv->stats, cpu); - rcv_stats =per_cpu_ptr (rcv_priv->stats, cpu); + stats =my_cpu_ptr (priv->stats); + rcv_stats =my_cpu_ptr (rcv_priv->stats);if (!(rcv->flags &IFF_UP )) goto tx_drop;
fs/nfs/iostat.h +9 -12
@@ -26,12 +26,11 @@ static inline void nfs_inc_server_stats( enum nfs_stat_eventcounters stat) {struct nfs_iostats *iostats; -int cpu; - cpu =get_cpu (); - iostats =per_cpu_ptr (server->io_stats, cpu); +preempt_disable (); + iostats =my_cpu_ptr (server->io_stats); iostats->events[stat]++; -put_cpu_no_resched (); +preempt_enable_no_resched (); } static inline void nfs_inc_stats(const struct inode *inode,
@@ -45,12 +44,11 @@ static inline void nfs_add_server_stats( unsigned long addend) {struct nfs_iostats *iostats; -int cpu; - cpu =get_cpu (); - iostats =per_cpu_ptr (server->io_stats, cpu); +preempt_disable (); + iostats =my_cpu_ptr (server->io_stats); iostats->bytes[stat] += addend; -put_cpu_no_resched (); +preempt_enable_no_resched (); } static inline void nfs_add_stats(const struct inode *inode,
@@ -66,12 +64,11 @@ static inline void nfs_add_fscache_stats unsigned long addend) {struct nfs_iostats *iostats; -int cpu; - cpu =get_cpu (); - iostats =per_cpu_ptr (NFS_SERVER (inode)->io_stats, cpu); +preempt_disable (); + iostats =my_cpu_ptr (NFS_SERVER (inode)->io_stats); iostats->fscache[stat] += addend; -put_cpu_no_resched (); +preempt_enable_no_resched (); }#endif
include/net/snmp.h +17 -14
@@ -139,26 +139,29 @@ struct linux_xfrm_mib {#define SNMP_INC_STATS_BH (mib, field) \ (__my_cpu_ptr(mib[0])->mibs[field]++)#define SNMP_INC_STATS_USER (mib, field) \ - do { \ - per_cpu_ptr(mib[1], get_cpu())->mibs[field]++; \ - put_cpu(); \ - } while (0) + do { \ + preempt_disable(); \ + my_cpu_ptr(mib[1])->mibs[field]++; \ + preempt_enable(); \ + } while (0)#define SNMP_INC_STATS (mib, field) \ - do { \ - per_cpu_ptr(mib[!in_softirq()], get_cpu())->mibs[field]++; \ - put_cpu(); \ - } while (0) + do { \ + preempt_disable(); \ + my_cpu_ptr(mib[!in_softirq()])->mibs[field]++; \ + preempt_enable(); \ + } while (0)#define SNMP_DEC_STATS (mib, field) \ do { \ - per_cpu_ptr(mib[!in_softirq()], get_cpu())->mibs[field]--; \ - put_cpu(); \ + preempt_disable(); \ + my_cpu_ptr(mib[!in_softirq()])->mibs[field]--; \ + preempt_enable(); \ } while (0)#define SNMP_ADD_STATS_BH (mib, field, addend) \ (__my_cpu_ptr(mib[0])->mibs[field] += addend)#define SNMP_ADD_STATS_USER (mib, field, addend) \ - do { \ - per_cpu_ptr(mib[1], get_cpu())->mibs[field] += addend; \ - put_cpu(); \ + do { \ + preempt_disable(); \ + my_cpu_ptr(mib[1])->mibs[field] += addend; \ + preempt_enable(); \ } while (0) -#endif
Re: [my_cpu_ptr 3/5] Elimninate get/put_cpu
dmaengine.c was not compiled.... Fixes:
--- drivers/dma/dmaengine.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/dma/dmaengine.c +2 -2
@@ -328,7 +328,7 @@ struct dma_chan *dma_find_channel(enum d {struct dma_chan *chan; -preempt_disable () +preempt_disable (); chan =my_cpu_ptr (channel_table[tx_type])->chan;preempt_enable ();
@@ -918,7 +918,7 @@ dma_async_memcpy_pg_to_pg(struct dma_chapreempt_disable ();my_cpu_ptr (chan->local)->bytes_transferred += len;my_cpu_ptr (chan->local)->memcpy_count++; -preempt_disable (); +preempt_enable ();return cookie; }
PATCH 4/5 sda_icsb_modify_counters() does not need a "cpu" variable
In reply to [email protected]The xfs_icsb_modify_counters() function no longer needs the cpu variable if we use my_cpu_ptr() and we can get rid of get/put_cpu().
--- fs/xfs/xfs_mount.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
fs/xfs/xfs_mount.c +6 -6
@@ -2320,12 +2320,12 @@ xfs_icsb_modify_counters( {xfs_icsb_cnts_t *icsbp;long long lcounter;/* long counter for 64 bit fields */ -int cpu, ret =0 ; +int ret =0 ;might_sleep (); again: - cpu =get_cpu (); - icsbp = (xfs_icsb_cnts_t *)per_cpu_ptr (mp->m_sb_cnts, cpu); +preempt_disable (); + icsbp = (xfs_icsb_cnts_t *)my_cpu_ptr (mp->m_sb_cnts);/* * if the counter is disabled, go to slow path
@@ -2369,11 +2369,11 @@ again:break ; }xfs_icsb_unlock_cntr (icsbp); -put_cpu (); +preempt_enable ();return 0 ; slow_path: -put_cpu (); +preempt_enable ();/* * serialise with a mutex so we don't burn lots of cpu on
@@ -2421,7 +2421,7 @@ slow_path: balance_counter:xfs_icsb_unlock_cntr (icsbp); -put_cpu (); +preempt_enable ();/* * We may have multiple threads here if multiple per-cpu
Re: [my_cpu_ptr 4/5] sda_icsb_modify_counters() does not need a "cpu" variable
Quoted 6 lines · cl
The
xfs_icsb_modify_counters()function no longer needs the cpu variable if we usemy_cpu_ptr()and we can get rid of get/put_cpu().Cc Christoph Hellwig <[email protected]>Cc Olaf Weber <[email protected]>Signed-off-by Christoph Lameter <[email protected]>
Looks fine to me.
Signature · 4 lines
Olaf Weber SGI Phone: +31(0)30-6696752
Veldzigt 2b Fax: +31(0)30-6696799
Technical Lead 3454 PW de Meern Vnet: 955-7151
Storage Software The Netherlands Email: [email protected]PATCH 5/5 Use my_cpu_ptr in crypto subsystem
In reply to [email protected]Just a slight optimization that removes one array lookup. The processor number is needed for other things as well so the get/put_cpu cannot be removed.
--- crypto/cryptd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
crypto/cryptd.c +1 -1
@@ -93,7 +93,7 @@ static int cryptd_enqueue_request(structstruct cryptd_cpu_queue *cpu_queue; cpu =get_cpu (); - cpu_queue =per_cpu_ptr (queue->cpu_queue, cpu); + cpu_queue =my_cpu_ptr (queue->cpu_queue); err =crypto_enqueue_request (&cpu_queue->queue, request);queue_work_on (cpu, kcrypto_wq, &cpu_queue->work);put_cpu ();