Skip to contentSkip to conversation

ehea: Optional TX/RX path optimized for SMP

PATCH In review Jan-Bernd Themann 4 participants 6 messages Feb 23, 2007 – Mar 3, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ehea: Optional TX/RX path optimized for SMP

Hi!

This patch introduces an optional alternative receive processing functionality (enabled via module load parameter). The ehea adapter can sort TCP traffic to multiple receive queues to be processed by the driver in parallel on multiple CPUs. The hardware always puts packets for an individual tcp stream on the same queue. As the current NAPI interface does not allow to handle parallel receive threads for a single adapter (processing on multiple CPUs in parallel) this patch uses tasklets with a simple fairness algorithm instead. On the send side we also take advantage of ehea's multiple send queue capabilites. A simple hash function in combination with the LL_TX attribute allows to process tx-packets on multiple CPUs on different queues. The hash function is needed to guarantee proper TCP packet ordering. This alternative packet processing functionality leads to significant performance improvements with ehea.

Are there any concerns about this approach?

Regards,
Jan-Bernd

---

drivers/net/ehea/ehea.h +7 -1
diff -Nurp -X dontdiff linux-2.6.21-rc1/drivers/net/ehea/ehea.h patched_kernel/drivers/net/ehea/ehea.h
@@ -39,7 +39,7 @@
 #include <asm/io.h>
 
 #define DRV_NAME	"ehea"
-#define DRV_VERSION	"EHEA_0048"
+#define DRV_VERSION	"EHEA_0049"
 
 #define EHEA_MSG_DEFAULT (NETIF_MSG_LINK | NETIF_MSG_TIMER \
 	| NETIF_MSG_RX_ERR | NETIF_MSG_TX_ERR)
@@ -375,8 +375,12 @@ struct ehea_port_res {
 	struct tasklet_struct send_comp_task;
 	spinlock_t recv_lock;
 	struct port_state p_state;
+	struct tasklet_struct recv_task;
 	u64 rx_packets;
 	u32 poll_counter;
+	u32 spoll_counter;
+	u32 rtcount;
+	u32 stcount;
 };
 
 
@@ -416,7 +420,9 @@ struct ehea_port {
 	char int_aff_name[EHEA_IRQ_NAME_SIZE];
 	int allmulti;			 /* Indicates IFF_ALLMULTI state */
 	int promisc;		 	 /* Indicates IFF_PROMISC state */
+	int num_tx_qps;
 	int num_add_tx_qps;
+	int num_mcs;
 	int resets;
 	u64 mac_addr;
 	u32 logical_port_id;
drivers/net/ehea/ehea_main.c +188 -44

394 lines not on this page · raw message

drivers/net/ehea/ehea_qmr.h +6 -1
diff -Nurp -X dontdiff linux-2.6.21-rc1/drivers/net/ehea/ehea_qmr.h patched_kernel/drivers/net/ehea/ehea_qmr.h
@@ -320,6 +320,11 @@ static inline struct ehea_cqe *ehea_poll
 	return hw_qeit_get_valid(queue);
 }
 
+static inline void ehea_inc_cq(struct ehea_cq *cq)
+{
+	hw_qeit_inc(&cq->hw_queue);
+}
+
 static inline void ehea_inc_rq1(struct ehea_qp *qp)
 {
 	hw_qeit_inc(&qp->hw_rqueue1);
@@ -327,7 +332,7 @@ static inline void ehea_inc_rq1(struct e
 
 static inline struct ehea_cqe *ehea_poll_cq(struct ehea_cq *my_cq)
 {
-	return hw_qeit_get_inc_valid(&my_cq->hw_queue);
+	return hw_qeit_get_valid(&my_cq->hw_queue);
 }
 
 #define EHEA_CQ_REGISTER_ORIG 0
#1 lore raw
Quoted 6 lines

This patch introduces an optional alternative receive processing functionality (enabled via module load parameter). The ehea adapter can sort TCP traffic to multiple receive queues to be processed by the driver in parallel on multiple CPUs. The hardware always puts packets for an individual tcp stream on the same queue. As the current NAPI interface does not allow to handle parallel receive

Quoted 8 lines

threads for a single adapter (processing on multiple CPUs in parallel) this patch uses tasklets with a simple fairness algorithm instead. On the send side we also take advantage of ehea's multiple send queue capabilites. A simple hash function in combination with the LL_TX attribute allows to process tx-packets on multiple CPUs on different queues. The hash function is needed to guarantee proper TCP packet ordering. This alternative packet processing functionality leads to significant performance improvements with ehea.

Why make this a module option that the user has to set? Are there any circumstances when someone wouldn't want "significant performance improvements?" If this approach is just better, then it should just replace the old code.

Also, as far as the approach of using tasklets, I think it would be better to use the "fake netdev" approach to continue to use NAPI. Basically you create a pseudo-netdev for each receive queue and have NAPI handle the polling for you -- you could look for drivers/net/cxgb3 for an example of this.

- R.

#2 lore raw

Hi

Quoted 6 lines

Also, as far as the approach of using tasklets, I think it would be better to use the "fake netdev" approach to continue to use NAPI. Basically you create a pseudo-netdev for each receive queue and have NAPI handle the polling for you -- you could look for drivers/net/cxgb3 for an example of this.

Thanks for pointing us to this solution. We are now building a NAPI version that makes use of these pseudo-netdev. The fairness amongst other netdevices should be better this way.

Quoted 6 lines

Why make this a module option that the user has to set? Are there any circumstances when someone wouldn't want "significant performance improvements?" If this approach is just better, then it should just replace the old code.

We'll change the default behaviour to multi queue, but we'd like to keep the option to run in a single queue mode for debug and backward compabilty.

Thanks,

Jan-Bernd & Christoph R.

#4 lore raw
Quoted 6 lines · Andi Kleen
Quoted 2 lines · Jan-Bernd Themann

Are there any concerns about this approach?

Yes. You should fix the NAPI code instead of trying to work around it.

NAPI is being fixed but the fix will take time to get in. In the meantime, the solution to get something working is the workaround.

Ben.

#5 lore raw
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 6 lines · Andi Kleen
Quoted 2 lines · Jan-Bernd Themann

Are there any concerns about this approach?

Yes. You should fix the NAPI code instead of trying to work around it.

NAPI is being fixed but the fix will take time to get in. In the meantime, the solution to get something working is the workaround

If it works right now with just a little less efficiency there is no pressing need to do workarounds until the real solution.

Besides I doubt that patch would have made .21 anyways and in .22 you might already have multiqueue NAPI.

-Andi