Skip to contentSkip to conversation

Introduce my_cpu_ptr/__my_cpu_ptr

PATCH In review [email protected] 5 participants 17 messages May 27, 2009 – Jun 3, 2009

Subsystems (10)
lore
Share & tools
lore
#0 lore raw

cover Introduce my_cpu_ptr/__my_cpu_ptr

The problem with per_cpu_ptr(x, smp_processor_id) is that it requires an array lookup to find the offset for the cpu. Processors typically have the offset for the current cpu area already in some kind of (arch dependent) efficiently accessible register or memory location.

We can use the per cpu offset instead of doing the array lookup to speed up the determination of the address of the percpu variable. This is particularly significant because these lookups occur in performance critical paths of the core kernel.

This optimization is a prerequiste to the introduction of per processor atomic operations for the core code. Atomic per processor operations implicitly do the offset calculation to the current per cpu area in a single instruction. The locations touched by this patchset are potential candidates for atomic per cpu operations.

The patchset does not cover arch code nor exotic cases where the current cpu is determined before the call into a function.

#1 lore raw

PATCH 1/5 Introduce my_cpu_ptr()

my_cpu_ptr(xx) = per_cpu_ptr(xx, smp_processor_id).

The problem with per_cpu_ptr(x, smp_processor_id) is that it requires an array lookup to find the offset for the cpu. Processors typically have the offset for the current cpu area in some kind of (arch dependent) efficiently accessible register or memory location.

We can use that instead of doing the array lookup to speed up the determination of the addressof the percpu variable. This is particularly significant because these lookups occur in performance critical paths of the core kernel.

This optimization is a prerequiste to the introduction of per processor atomic operations for the core code. Atomic per processor operations implicitly do the offset calculation to the current per cpu area in a single instruction. All the locations touched by this patchset are potential candidates for atomic per cpu operations.

my_cpu_ptr comes in two flavors. The preemption context matters since we are referring the the currently executing processor. In many cases we must insure that the processor does not change while a code segment is executed.

__my_cpu_ptr -> Do not check for preemption context
my_cpu_ptr -> Check preemption context

Cc David Howells <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Rusty Russell <[email protected]>
Cc Eric Dumazet <[email protected]>
---
 include/linux/percpu.h |   11 +++++++++++
 1 file changed, 11 insertions(+)
include/linux/percpu.h +11 -0
Index: linux-2.6/include/linux/percpu.h
===================================================================
@@ -78,6 +78,9 @@ extern ssize_t __init pcpu_embed_first_c
  */
 #define per_cpu_ptr(ptr, cpu)	SHIFT_PERCPU_PTR((ptr), per_cpu_offset((cpu)))
 
+#define my_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, my_cpu_offset)
+#define __my_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, __my_cpu_offset)
+
 extern void *__alloc_reserved_percpu(size_t size, size_t align);
 
 #else /* CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */
@@ -94,6 +97,12 @@ struct percpu_data {
         (__typeof__(ptr))__p->ptrs[(cpu)];				\
 })
 
+#define my_cpu_ptr(ptr) \
+		per_cpu_ptr(ptr, smp_processor_id())
+
+#define __my_cpu_ptr(ptr) \
+		per_cpu_ptr(ptr, raw_smp_processor_id())
+
 #endif /* CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */
 
 extern void *__alloc_percpu(size_t size, size_t align);
@@ -102,6 +111,8 @@ extern void free_percpu(void *__pdata);
 #else /* CONFIG_SMP */
 
 #define per_cpu_ptr(ptr, cpu) ({ (void)(cpu); (ptr); })
+#define my_cpu_ptr(ptr) per_cpu_ptr(ptr, 0)
+#define __my_cpu_ptr(ptr) my_cpu_ptr(ptr)
 
 static inline void *__alloc_percpu(size_t size, size_t align)
 {
#2 lore raw

Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()

Quoted 1 line · On Thu, 28 May 2009 03:16:59 am [email protected] wrote:

my_cpu_ptr(xx) = per_cpu_ptr(xx, smp_processor_id).

I had this implemented earlier as as get_cpu_ptr()/__get_cpu_ptr(), to match get_cpu_var() / __get_cpu_var().

But other than that nomenclature quibble, it looks fine!

Thanks,
Rusty.

#3 lore raw

Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()

Quoted 5 lines · Rusty Russell
Quoted 1 line · On Thu, 28 May 2009 03:16:59 am [email protected] wrote:

my_cpu_ptr(xx) = per_cpu_ptr(xx, smp_processor_id).

I had this implemented earlier as as get_cpu_ptr()/__get_cpu_ptr(), to match get_cpu_var() / __get_cpu_var().

Have not seen it but it would be a bit confusing since we already have get_cpu* which must be paired with put_cpu* because of the refcount taking (get_cpu_var and get_cpu). get_cpu_ptr() would not have to be paired.

Better use a different name.

my_cpu_ptr came from my_cpu_offset:

#define my_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, my_cpu_offset) #define __my_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, __my_cpu_offset)

#4 lore raw

Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()

Quoted 11 lines · Christoph Lameter
Quoted 5 lines · Rusty Russell
Quoted 1 line · On Thu, 28 May 2009 03:16:59 am [email protected] wrote:

my_cpu_ptr(xx) = per_cpu_ptr(xx, smp_processor_id).

I had this implemented earlier as as get_cpu_ptr()/__get_cpu_ptr(), to match get_cpu_var() / __get_cpu_var().

Have not seen it but it would be a bit confusing since we already have get_cpu* which must be paired with put_cpu* because of the refcount taking (get_cpu_var and get_cpu). get_cpu_ptr() would not have to be paired.

To clarify, get_cpu_ptr() would be paired with put_cpu_ptr(). __get_cpu_ptr() would be the "raw" one:

#define get_cpu_ptr(xx) per_cpu_ptr(xx, get_cpu()) #define __get_cpu_ptr(xx) per_cpu_ptr(xx, smp_processor_id())

Quoted 3 lines

Better use a different name.

my_cpu_ptr came from my_cpu_offset:

Yep, but that's not as widely exposed as get_cpu & get_cpu_var.

Cheers,
Rusty.

#5 lore raw

Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()

Quoted 10 lines · Rusty Russell
Quoted 4 lines

Have not seen it but it would be a bit confusing since we already have get_cpu* which must be paired with put_cpu* because of the refcount taking (get_cpu_var and get_cpu). get_cpu_ptr() would not have to be paired.

To clarify, get_cpu_ptr() would be paired with put_cpu_ptr(). __get_cpu_ptr() would be the "raw" one:

#define get_cpu_ptr(xx) per_cpu_ptr(xx, get_cpu()) #define __get_cpu_ptr(xx) per_cpu_ptr(xx, smp_processor_id())

Hmmm.. That would be a major change in semantics. I tend to think about these ops more as atomic like operations rather than preempt sections. The operation to obtain a the current instance for a given processor is an operation done in a given preemption context. The get_cpu_ptr() approach establishes the preemption context during the same operation.

That would mean use

	*get_cpu_ptr(per_cpu_ptr1)++
	*__get_cpu_ptr(per_cpu_ptr2)++
	put_cpu_ptr()

instead of

	preempt_disable()
	*__this_cpu_ptr(per_cpu_ptr1)++
	*__this_cpu_ptr(per_cpu_ptr2)++
	preempt_enable().

And for context in which we know that preemption is off

	*__get_cpu_ptr(per_cpu_ptr1)++
	*__get_cpu_ptr(per_cpu_ptr2)++

vs

	*__this_cpu_ptr(per_cpu_ptr2)++
	*__this_cpu_ptr(per_cpu_ptr2)++

So its the same

How would that look for atomic per cpu ops?

	get_cpu_ptr_inc(per_cpu_ptr1);
	__get_cpu_ptr_inc(per_cpu_ptr2)
	put_cpu_ptr()

vs.

	this_cpu_inc(per_cpu_ptr1)
	this_cpu_inc(per_cpu_ptr2)

and known preemption off context

	__get_cpu_ptr_inc(per_cpu_ptr1)
	__get_cpu_ptr_inc(per_cpu_ptr2)

vs

	__this_cpu_inc(percpu_ptr1)
	__this_cpu_inc(percpu_ptr2)

Same again

Seems to be added complexity to also push a change of preemption into these operations. I'd rather keep the preempt status unchanged by these operations in particular because some counters may need interrupt protection on some platforms and not on others. We will likely need some additional operations like

	this_cpu_inc_irqsafe( ....)

that will require an irq off / on on some platforms and be a simple increment on others.

#6 lore raw

Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()

Quoted 6 lines · Christoph Lameter
Quoted 5 lines · Rusty Russell
Quoted 4 lines

Have not seen it but it would be a bit confusing since we already have get_cpu* which must be paired with put_cpu* because of the refcount taking (get_cpu_var and get_cpu). get_cpu_ptr() would not have to be paired.

Quoted 7 lines
Quoted 5 lines

To clarify, get_cpu_ptr() would be paired with put_cpu_ptr(). __get_cpu_ptr() would be the "raw" one:

#define get_cpu_ptr(xx) per_cpu_ptr(xx, get_cpu()) #define __get_cpu_ptr(xx) per_cpu_ptr(xx, smp_processor_id())

Hmmm.. That would be a major change in semantics.

It's exactly like get_cpu_var. For better or worse, let's not invent YA new convention.

Quoted 10 lines

How would that look for atomic per cpu ops?

	get_cpu_ptr_inc(per_cpu_ptr1);
	__get_cpu_ptr_inc(per_cpu_ptr2)
	put_cpu_ptr()

vs.

	this_cpu_inc(per_cpu_ptr1)
	this_cpu_inc(per_cpu_ptr2)

Well, get_* doesn't really make sense for any function which doesn't return a value.

So that name question doesn't really have a clear convention answer: we could re-use cpu_local_inc() since I think we decided to kill local_t. I slightly prefer it over "this_cpu_*" since we're not actually doing anything to the cpu itself, but I don't think anyone will get too confused and think that after this executes their CPU will be stepping 11. :)

Thanks,
Rusty.

#7 lore raw

Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()

Quoted 4 lines · Rusty Russell
Quoted 1 line

Hmmm.. That would be a major change in semantics.

It's exactly like get_cpu_var. For better or worse, let's not invent YA new convention.

It hink we are doing that if we go the get_cpu_ptr line. this_cpu_ptr starts from per_cpu_ptr() and its straightforward as a special case of per_cpu_ptr.

Quoted 5 lines

So that name question doesn't really have a clear convention answer: we could re-use cpu_local_inc() since I think we decided to kill local_t. I slightly prefer it over "this_cpu_*" since we're not actually doing anything to the cpu itself, but I don't think anyone will get too confused and think that after this executes their CPU will be stepping 11. :)

local is loaded with meaning by glibc where it means thread local. The variables here are cpu specific, not task specific.

#8 lore raw

Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()

In reply to Rusty Russell
Quoted 7 lines · Rusty Russell
Quoted 1 line · On Thu, 28 May 2009 03:16:59 am [email protected] wrote:

my_cpu_ptr(xx) = per_cpu_ptr(xx, smp_processor_id).

I had this implemented earlier as as get_cpu_ptr()/__get_cpu_ptr(), to match get_cpu_var() / __get_cpu_var().

But other than that nomenclature quibble, it looks fine!

my_ seems a very odd naming. We have a lot of this_cpu naming for the current cpu i nthe ctree, so I would suggest sticking to that.

#9 lore raw

Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()

Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Rusty Russell
Quoted 1 line · On Thu, 28 May 2009 03:16:59 am [email protected] wrote:

my_cpu_ptr(xx) = per_cpu_ptr(xx, smp_processor_id).

I had this implemented earlier as as get_cpu_ptr()/__get_cpu_ptr(), to match get_cpu_var() / __get_cpu_var().

But other than that nomenclature quibble, it looks fine!

my_ seems a very odd naming. We have a lot of this_cpu naming for the current cpu i nthe ctree, so I would suggest sticking to that.

So this_cpu is taken. I used THIS_CPU in earlier version but got complaints about uppercase use. Is

this_cpu_ptr()

and

__this_cpu_ptr()

ok?

maintainer
#10 lore raw

Re: [my_cpu_ptr 1/5] Introduce my_cpu_ptr()

Quoted 15 lines · Christoph Lameter
Quoted 9 lines · Christoph Hellwig
Quoted 6 lines · Rusty Russell
Quoted 1 line · On Thu, 28 May 2009 03:16:59 am [email protected] wrote:

my_cpu_ptr(xx) = per_cpu_ptr(xx, smp_processor_id).

I had this implemented earlier as as get_cpu_ptr()/__get_cpu_ptr(), to match get_cpu_var() / __get_cpu_var().

But other than that nomenclature quibble, it looks fine!

my_ seems a very odd naming. We have a lot of this_cpu naming for the current cpu i nthe ctree, so I would suggest sticking to that.

So this_cpu is taken. I used THIS_CPU in earlier version but got complaints about uppercase use. Is

Quoted 7 lines

this_cpu_ptr()

and

__this_cpu_ptr()

ok?

Yeap, those look fine to me. Sans the renaming, the series generally looks good but I didn't review each conversion.

For the first patch, Acked-by: Tejun Heo <[email protected]>

Thanks.

Signature · 1 line
tejun
#11 lore raw

PATCH 2/5 Straight transformations

In reply to [email protected]

Use my_cpu_ptr and __my_cpu_ptr in locations where straight transformations are possible because per_cpu_ptr is used with either smp_processor_id() or raw_smp_processor_id().

Cc David Howells <[email protected]>
Cc Tejun Heo <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Rusty Russell <[email protected]>
Cc Eric Dumazet <[email protected]>
---
 drivers/infiniband/hw/ehca/ehca_irq.c       |    3 +--
 drivers/net/chelsio/sge.c                   |    5 ++---
 drivers/net/loopback.c                      |    2 +-
 fs/ext4/mballoc.c                           |    2 +-
 include/net/neighbour.h                     |    2 +-
 include/net/netfilter/nf_conntrack.h        |    4 ++--
 include/net/netfilter/nf_conntrack_ecache.h |    2 +-
 include/net/snmp.h                          |    4 ++--
 8 files changed, 11 insertions(+), 13 deletions(-)
drivers/net/chelsio/sge.c +2 -3
Index: linux-2.6/drivers/net/chelsio/sge.c
===================================================================
@@ -1378,7 +1378,7 @@ static void sge_rx(struct sge *sge, stru
 	}
 	__skb_pull(skb, sizeof(*p));
 
-	st = per_cpu_ptr(sge->port_stats[p->iff], smp_processor_id());
+	st = my_cpu_ptr(sge->port_stats[p->iff]);
 
 	skb->protocol = eth_type_trans(skb, adapter->port[p->iff].dev);
 	if ((adapter->flags & RX_CSUM_ENABLED) && p->csum == 0xffff &&
@@ -1780,8 +1780,7 @@ int t1_start_xmit(struct sk_buff *skb, s
 {
 	struct adapter *adapter = dev->ml_priv;
 	struct sge *sge = adapter->sge;
-	struct sge_port_stats *st = per_cpu_ptr(sge->port_stats[dev->if_port],
-						smp_processor_id());
+	struct sge_port_stats *st = my_cpu_ptr(sge->port_stats[dev->if_port]);
 	struct cpl_tx_pkt *cpl;
 	struct sk_buff *orig_skb = skb;
 	int ret;
drivers/net/loopback.c +1 -1
Index: linux-2.6/drivers/net/loopback.c
===================================================================
@@ -78,7 +78,7 @@ static int loopback_xmit(struct sk_buff 
 
 	/* it's OK to use per_cpu_ptr() because BHs are off */
 	pcpu_lstats = dev->ml_priv;
-	lb_stats = per_cpu_ptr(pcpu_lstats, smp_processor_id());
+	lb_stats = my_cpu_ptr(pcpu_lstats);
 	lb_stats->bytes += skb->len;
 	lb_stats->packets++;
 
fs/ext4/mballoc.c +1 -1
Index: linux-2.6/fs/ext4/mballoc.c
===================================================================
@@ -4210,7 +4210,7 @@ static void ext4_mb_group_or_file(struct
 	 * per cpu locality group is to reduce the contention between block
 	 * request from multiple CPUs.
 	 */
-	ac->ac_lg = per_cpu_ptr(sbi->s_locality_groups, raw_smp_processor_id());
+	ac->ac_lg = __my_cpu_ptr(sbi->s_locality_groups);
 
 	/* we're going to use group allocation */
 	ac->ac_flags |= EXT4_MB_HINT_GROUP_ALLOC;
include/net/neighbour.h +1 -1
Index: linux-2.6/include/net/neighbour.h
===================================================================
@@ -92,7 +92,7 @@ struct neigh_statistics
 #define NEIGH_CACHE_STAT_INC(tbl, field)				\
 	do {								\
 		preempt_disable();					\
-		(per_cpu_ptr((tbl)->stats, smp_processor_id())->field)++; \
+		(my_cpu_ptr((tbl)->stats)->field)++;			\
 		preempt_enable();					\
 	} while (0)
 
include/net/netfilter/nf_conntrack.h +2 -2
Index: linux-2.6/include/net/netfilter/nf_conntrack.h
===================================================================
@@ -292,11 +292,11 @@ extern unsigned int nf_conntrack_htable_
 extern unsigned int nf_conntrack_max;
 
 #define NF_CT_STAT_INC(net, count)	\
-	(per_cpu_ptr((net)->ct.stat, raw_smp_processor_id())->count++)
+	(__my_cpu_ptr((net)->ct.stat)->count++)
 #define NF_CT_STAT_INC_ATOMIC(net, count)		\
 do {							\
 	local_bh_disable();				\
-	per_cpu_ptr((net)->ct.stat, raw_smp_processor_id())->count++;	\
+	__my_cpu_ptr((net)->ct.stat)->count++;		\
 	local_bh_enable();				\
 } while (0)
 
include/net/netfilter/nf_conntrack_ecache.h +1 -1
Index: linux-2.6/include/net/netfilter/nf_conntrack_ecache.h
===================================================================
@@ -39,7 +39,7 @@ nf_conntrack_event_cache(enum ip_conntra
 	struct nf_conntrack_ecache *ecache;
 
 	local_bh_disable();
-	ecache = per_cpu_ptr(net->ct.ecache, raw_smp_processor_id());
+	ecache = __my_cpu_ptr(net->ct.ecache);
 	if (ct != ecache->ct)
 		__nf_ct_event_cache_init(ct);
 	ecache->events |= event;
drivers/infiniband/hw/ehca/ehca_irq.c +1 -2
Index: linux-2.6/drivers/infiniband/hw/ehca/ehca_irq.c
===================================================================
@@ -827,8 +827,7 @@ static void __cpuinit take_over_work(str
 		cq = list_entry(cct->cq_list.next, struct ehca_cq, entry);
 
 		list_del(&cq->entry);
-		__queue_comp_task(cq, per_cpu_ptr(pool->cpu_comp_tasks,
-						  smp_processor_id()));
+		__queue_comp_task(cq, my_cpu_ptr(pool->cpu_comp_tasks));
 	}
 
 	spin_unlock_irqrestore(&cct->task_lock, flags_cct);
include/net/snmp.h +2 -2
Index: linux-2.6/include/net/snmp.h
===================================================================
@@ -137,7 +137,7 @@ struct linux_xfrm_mib {
 #define SNMP_STAT_USRPTR(name)	(name[1])
 
 #define SNMP_INC_STATS_BH(mib, field) 	\
-	(per_cpu_ptr(mib[0], raw_smp_processor_id())->mibs[field]++)
+	(__my_cpu_ptr(mib[0])->mibs[field]++)
 #define SNMP_INC_STATS_USER(mib, field) \
 	do { \
 		per_cpu_ptr(mib[1], get_cpu())->mibs[field]++; \
@@ -154,7 +154,7 @@ struct linux_xfrm_mib {
 		put_cpu(); \
 	} while (0)
 #define SNMP_ADD_STATS_BH(mib, field, addend) 	\
-	(per_cpu_ptr(mib[0], raw_smp_processor_id())->mibs[field] += addend)
+	(__my_cpu_ptr(mib[0])->mibs[field] += addend)
 #define SNMP_ADD_STATS_USER(mib, field, addend) 	\
 	do { \
 		per_cpu_ptr(mib[1], get_cpu())->mibs[field] += addend; \
#12 lore raw

PATCH 3/5 Elimninate get/put_cpu

In reply to [email protected]

There are cases where we can use my_cpu_ptr and as the result of using my_cpu_ptr we no longer need to determine the current executing cpu.

In those places no get/put_cpu combination is needed anymore. The local cpu variable can be eliminated.

Preemption still needs to be disabled and enabled since the modifications of the per cpu variables is not atomic. There may be multiple per cpu variables modified and those must all be from the same processor.

Cc Dan Williams <[email protected]>
Cc Eric Biederman <[email protected]>
Cc Stephen Hemminger <[email protected]>
Cc Trond Myklebust <[email protected]>
Cc Herbert Xu <[email protected]>
Cc David L Stevens <[email protected]>
---
 drivers/dma/dmaengine.c |   34 +++++++++++++++-------------------
 drivers/net/veth.c      |    7 +++----
 fs/nfs/iostat.h         |   21 +++++++++------------
 include/net/snmp.h      |   31 +++++++++++++++++--------------
 4 files changed, 44 insertions(+), 49 deletions(-)
drivers/dma/dmaengine.c +15 -19
Index: linux-2.6/drivers/dma/dmaengine.c
===================================================================
@@ -327,11 +327,10 @@ arch_initcall(dma_channel_table_init);
 struct dma_chan *dma_find_channel(enum dma_transaction_type tx_type)
 {
 	struct dma_chan *chan;
-	int cpu;
 
-	cpu = get_cpu();
-	chan = per_cpu_ptr(channel_table[tx_type], cpu)->chan;
-	put_cpu();
+	preempt_disable()
+	chan = my_cpu_ptr(channel_table[tx_type])->chan;
+	preempt_enable();
 
 	return chan;
 }
@@ -803,7 +802,6 @@ dma_async_memcpy_buf_to_buf(struct dma_c
 	struct dma_async_tx_descriptor *tx;
 	dma_addr_t dma_dest, dma_src;
 	dma_cookie_t cookie;
-	int cpu;
 	unsigned long flags;
 
 	dma_src = dma_map_single(dev->dev, src, len, DMA_TO_DEVICE);
@@ -822,10 +820,10 @@ dma_async_memcpy_buf_to_buf(struct dma_c
 	tx->callback = NULL;
 	cookie = tx->tx_submit(tx);
 
-	cpu = get_cpu();
-	per_cpu_ptr(chan->local, cpu)->bytes_transferred += len;
-	per_cpu_ptr(chan->local, cpu)->memcpy_count++;
-	put_cpu();
+	preempt_disable();
+	my_cpu_ptr(chan->local)->bytes_transferred += len;
+	my_cpu_ptr(chan->local)->memcpy_count++;
+	preempt_enable();
 
 	return cookie;
 }
@@ -852,7 +850,6 @@ dma_async_memcpy_buf_to_pg(struct dma_ch
 	struct dma_async_tx_descriptor *tx;
 	dma_addr_t dma_dest, dma_src;
 	dma_cookie_t cookie;
-	int cpu;
 	unsigned long flags;
 
 	dma_src = dma_map_single(dev->dev, kdata, len, DMA_TO_DEVICE);
@@ -869,10 +866,10 @@ dma_async_memcpy_buf_to_pg(struct dma_ch
 	tx->callback = NULL;
 	cookie = tx->tx_submit(tx);
 
-	cpu = get_cpu();
-	per_cpu_ptr(chan->local, cpu)->bytes_transferred += len;
-	per_cpu_ptr(chan->local, cpu)->memcpy_count++;
-	put_cpu();
+	preempt_disable();
+	my_cpu_ptr(chan->local)->bytes_transferred += len;
+	my_cpu_ptr(chan->local)->memcpy_count++;
+	preempt_enable();
 
 	return cookie;
 }
@@ -901,7 +898,6 @@ dma_async_memcpy_pg_to_pg(struct dma_cha
 	struct dma_async_tx_descriptor *tx;
 	dma_addr_t dma_dest, dma_src;
 	dma_cookie_t cookie;
-	int cpu;
 	unsigned long flags;
 
 	dma_src = dma_map_page(dev->dev, src_pg, src_off, len, DMA_TO_DEVICE);
@@ -919,10 +915,10 @@ dma_async_memcpy_pg_to_pg(struct dma_cha
 	tx->callback = NULL;
 	cookie = tx->tx_submit(tx);
 
-	cpu = get_cpu();
-	per_cpu_ptr(chan->local, cpu)->bytes_transferred += len;
-	per_cpu_ptr(chan->local, cpu)->memcpy_count++;
-	put_cpu();
+	preempt_disable();
+	my_cpu_ptr(chan->local)->bytes_transferred += len;
+	my_cpu_ptr(chan->local)->memcpy_count++;
+	preempt_disable();
 
 	return cookie;
 }
drivers/net/veth.c +3 -4
Index: linux-2.6/drivers/net/veth.c
===================================================================
@@ -153,7 +153,7 @@ static int veth_xmit(struct sk_buff *skb
 	struct net_device *rcv = NULL;
 	struct veth_priv *priv, *rcv_priv;
 	struct veth_net_stats *stats, *rcv_stats;
-	int length, cpu;
+	int length;
 
 	skb_orphan(skb);
 
@@ -161,9 +161,8 @@ static int veth_xmit(struct sk_buff *skb
 	rcv = priv->peer;
 	rcv_priv = netdev_priv(rcv);
 
-	cpu = smp_processor_id();
-	stats = per_cpu_ptr(priv->stats, cpu);
-	rcv_stats = per_cpu_ptr(rcv_priv->stats, cpu);
+	stats = my_cpu_ptr(priv->stats);
+	rcv_stats = my_cpu_ptr(rcv_priv->stats);
 
 	if (!(rcv->flags & IFF_UP))
 		goto tx_drop;
fs/nfs/iostat.h +9 -12
Index: linux-2.6/fs/nfs/iostat.h
===================================================================
@@ -26,12 +26,11 @@ static inline void nfs_inc_server_stats(
 					enum nfs_stat_eventcounters stat)
 {
 	struct nfs_iostats *iostats;
-	int cpu;
 
-	cpu = get_cpu();
-	iostats = per_cpu_ptr(server->io_stats, cpu);
+	preempt_disable();
+	iostats = my_cpu_ptr(server->io_stats);
 	iostats->events[stat]++;
-	put_cpu_no_resched();
+	preempt_enable_no_resched();
 }
 
 static inline void nfs_inc_stats(const struct inode *inode,
@@ -45,12 +44,11 @@ static inline void nfs_add_server_stats(
 					unsigned long addend)
 {
 	struct nfs_iostats *iostats;
-	int cpu;
 
-	cpu = get_cpu();
-	iostats = per_cpu_ptr(server->io_stats, cpu);
+	preempt_disable();
+	iostats = my_cpu_ptr(server->io_stats);
 	iostats->bytes[stat] += addend;
-	put_cpu_no_resched();
+	preempt_enable_no_resched();
 }
 
 static inline void nfs_add_stats(const struct inode *inode,
@@ -66,12 +64,11 @@ static inline void nfs_add_fscache_stats
 					 unsigned long addend)
 {
 	struct nfs_iostats *iostats;
-	int cpu;
 
-	cpu = get_cpu();
-	iostats = per_cpu_ptr(NFS_SERVER(inode)->io_stats, cpu);
+	preempt_disable();
+	iostats = my_cpu_ptr(NFS_SERVER(inode)->io_stats);
 	iostats->fscache[stat] += addend;
-	put_cpu_no_resched();
+	preempt_enable_no_resched();
 }
 #endif
 
include/net/snmp.h +17 -14
Index: linux-2.6/include/net/snmp.h
===================================================================
@@ -139,26 +139,29 @@ struct linux_xfrm_mib {
 #define SNMP_INC_STATS_BH(mib, field) 	\
 	(__my_cpu_ptr(mib[0])->mibs[field]++)
 #define SNMP_INC_STATS_USER(mib, field) \
-	do { \
-		per_cpu_ptr(mib[1], get_cpu())->mibs[field]++; \
-		put_cpu(); \
-	} while (0)
+ 	do { \
+		preempt_disable();				\
+		my_cpu_ptr(mib[1])->mibs[field]++;		\
+		preempt_enable();				\
+  	} while (0)
 #define SNMP_INC_STATS(mib, field) 	\
-	do { \
-		per_cpu_ptr(mib[!in_softirq()], get_cpu())->mibs[field]++; \
-		put_cpu(); \
-	} while (0)
+ 	do { \
+		preempt_disable();				\
+		my_cpu_ptr(mib[!in_softirq()])->mibs[field]++;	\
+		preempt_enable(); \
+ 	} while (0)
 #define SNMP_DEC_STATS(mib, field) 	\
 	do { \
-		per_cpu_ptr(mib[!in_softirq()], get_cpu())->mibs[field]--; \
-		put_cpu(); \
+		preempt_disable();				\
+		my_cpu_ptr(mib[!in_softirq()])->mibs[field]--;	\
+		preempt_enable();				\
 	} while (0)
 #define SNMP_ADD_STATS_BH(mib, field, addend) 	\
 	(__my_cpu_ptr(mib[0])->mibs[field] += addend)
 #define SNMP_ADD_STATS_USER(mib, field, addend) 	\
-	do { \
-		per_cpu_ptr(mib[1], get_cpu())->mibs[field] += addend; \
-		put_cpu(); \
+ 	do { \
+		preempt_disable();				\
+		my_cpu_ptr(mib[1])->mibs[field] += addend;	\
+		preempt_enable();				\
 	} while (0)
-
 #endif
#13 lore raw

Re: [my_cpu_ptr 3/5] Elimninate get/put_cpu

dmaengine.c was not compiled.... Fixes:

---
 drivers/dma/dmaengine.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/dma/dmaengine.c +2 -2
Index: linux-2.6/drivers/dma/dmaengine.c
===================================================================
@@ -328,7 +328,7 @@ struct dma_chan *dma_find_channel(enum d
 {
 	struct dma_chan *chan;
 
-	preempt_disable()
+	preempt_disable();
 	chan = my_cpu_ptr(channel_table[tx_type])->chan;
 	preempt_enable();
 
@@ -918,7 +918,7 @@ dma_async_memcpy_pg_to_pg(struct dma_cha
 	preempt_disable();
 	my_cpu_ptr(chan->local)->bytes_transferred += len;
 	my_cpu_ptr(chan->local)->memcpy_count++;
-	preempt_disable();
+	preempt_enable();
 
 	return cookie;
 }
#14 lore raw

PATCH 4/5 sda_icsb_modify_counters() does not need a "cpu" variable

In reply to [email protected]

The xfs_icsb_modify_counters() function no longer needs the cpu variable if we use my_cpu_ptr() and we can get rid of get/put_cpu().

Cc Christoph Hellwig <[email protected]>
Cc Olaf Weber <[email protected]>
---
 fs/xfs/xfs_mount.c |   12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
fs/xfs/xfs_mount.c +6 -6
Index: linux-2.6/fs/xfs/xfs_mount.c
===================================================================
@@ -2320,12 +2320,12 @@ xfs_icsb_modify_counters(
 {
 	xfs_icsb_cnts_t	*icsbp;
 	long long	lcounter;	/* long counter for 64 bit fields */
-	int		cpu, ret = 0;
+	int		ret = 0;
 
 	might_sleep();
 again:
-	cpu = get_cpu();
-	icsbp = (xfs_icsb_cnts_t *)per_cpu_ptr(mp->m_sb_cnts, cpu);
+	preempt_disable();
+	icsbp = (xfs_icsb_cnts_t *)my_cpu_ptr(mp->m_sb_cnts);
 
 	/*
 	 * if the counter is disabled, go to slow path
@@ -2369,11 +2369,11 @@ again:
 		break;
 	}
 	xfs_icsb_unlock_cntr(icsbp);
-	put_cpu();
+	preempt_enable();
 	return 0;
 
 slow_path:
-	put_cpu();
+	preempt_enable();
 
 	/*
 	 * serialise with a mutex so we don't burn lots of cpu on
@@ -2421,7 +2421,7 @@ slow_path:
 
 balance_counter:
 	xfs_icsb_unlock_cntr(icsbp);
-	put_cpu();
+	preempt_enable();
 
 	/*
 	 * We may have multiple threads here if multiple per-cpu
Acked-by on v1 4/5
#15 lore raw

Re: [my_cpu_ptr 4/5] sda_icsb_modify_counters() does not need a "cpu" variable

Quoted 6 lines · cl

The xfs_icsb_modify_counters() function no longer needs the cpu variable if we use my_cpu_ptr() and we can get rid of get/put_cpu().

Cc Christoph Hellwig <[email protected]>
Cc Olaf Weber <[email protected]>

Looks fine to me.

Signature · 4 lines
Olaf Weber                 SGI               Phone:  +31(0)30-6696752
                           Veldzigt 2b       Fax:    +31(0)30-6696799
Technical Lead             3454 PW de Meern  Vnet:   955-7151
Storage Software           The Netherlands   Email:  [email protected]
#16 lore raw

PATCH 5/5 Use my_cpu_ptr in crypto subsystem

In reply to [email protected]

Just a slight optimization that removes one array lookup. The processor number is needed for other things as well so the get/put_cpu cannot be removed.

---
 crypto/cryptd.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
crypto/cryptd.c +1 -1
Index: linux-2.6/crypto/cryptd.c
===================================================================
@@ -93,7 +93,7 @@ static int cryptd_enqueue_request(struct
 	struct cryptd_cpu_queue *cpu_queue;
 
 	cpu = get_cpu();
-	cpu_queue = per_cpu_ptr(queue->cpu_queue, cpu);
+	cpu_queue = my_cpu_ptr(queue->cpu_queue);
 	err = crypto_enqueue_request(&cpu_queue->queue, request);
 	queue_work_on(cpu, kcrypto_wq, &cpu_queue->work);
 	put_cpu();