ovs-dpdk: placing the metadata
Hi,
I've noticed in lib/netdev-dpdk.c that __rte_pktmbuf_init() stores the packet metadata right after "struct rte_mbuf", and before the buffer data:
/* start of buffer is just after mbuf structure */
m->buf_addr = (char *)m + sizeof(struct dp_packet);(struct dp_packet has the rte_mbuf as first member if DPDK enabled)
However, lib/librte_mbuf/rte_mbuf.h seems to codify that the buffer should start right after the rte_mbuf:
/**
* Given the buf_addr returns the pointer to corresponding mbuf. */
#define RTE_MBUF_FROM_BADDR(ba) (((struct rte_mbuf *)(ba)) - 1)
/**
* Given the pointer to mbuf returns an address where it's buf_addr
* should point to.
*/
#define RTE_MBUF_TO_BADDR(mb) (((struct rte_mbuf *)(mb)) + 1)
These macros are used for attaching/detaching mbuf's to each other. This is the way the code retrieves the direct buffer from an indirect one, and vica versa. I think if we want to keep the metadata feature (which I guess is quite important), we need to add a pointer to rte_mbuf, which helps the direct and indirect structs to find each other. Something like:
struct rte_mbuf *attach; /**< Points to the other buffer if this one
is (in)direct. Otherwise NULL. */
What do you think?
Regards,
Zoltan Kiss
_______________________________________________
dev mailing list
[email protected]
http://openvswitch.org/mailman/listinfo/dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Zoltan,
Quoted 30 lines · Zoltan Kiss
Hi,
I've noticed in
lib/netdev-dpdk.cthat__rte_pktmbuf_init()stores the packet metadata right after "structrte_mbuf", and before the buffer data:/* start of buffer is just after mbuf structure */ m->buf_addr = (char *)m + sizeof(struct dp_packet);(struct
dp_packethas therte_mbufas first member if DPDK enabled)However,
lib/librte_mbuf/rte_mbuf.hseems to codify that the buffer should start right after therte_mbuf:/**
* Given thebuf_addrreturns the pointer to corresponding mbuf. */
#defineRTE_MBUF_FROM_BADDR(ba)(((structrte_mbuf*)(ba)) - 1)/**
* Given the pointer to mbuf returns an address where it'sbuf_addr
* should point to.
*/
#defineRTE_MBUF_TO_BADDR(mb)(((structrte_mbuf*)(mb)) + 1)These macros are used for attaching/detaching mbuf's to each other. This is the way the code retrieves the direct buffer from an indirect one, and vica versa. I think if we want to keep the metadata feature (which I guess is quite important), we need to add a pointer to
rte_mbuf, which helps the direct and indirect structs to find each other. Something like:
Quoted 5 lines
struct
rte_mbuf*attach; /**< Points to the other buffer if this
one
is (in)direct. Otherwise NULL. */What do you think?
I've just sent a patch that should fix this issue.
http://dpdk.org/ml/archives/dev/2015-March/015722.html
Let me know if you have any comment on it.
Regards,
Olivier
A message that is not in the archive (sent off-list or not yet received): [email protected]
Zoltan Kiss
Reviewed-byHi Olivier,
Quoted 38 lines · Olivier MATZ
Hi Zoltan,
Quoted 35 lines · Zoltan Kiss
Hi,
I've noticed in
lib/netdev-dpdk.cthat__rte_pktmbuf_init()stores the packet metadata right after "structrte_mbuf", and before the buffer data:/* start of buffer is just after mbuf structure */ m->buf_addr = (char *)m + sizeof(struct dp_packet);(struct
dp_packethas therte_mbufas first member if DPDK enabled)However,
lib/librte_mbuf/rte_mbuf.hseems to codify that the buffer should start right after therte_mbuf:/**
* Given thebuf_addrreturns the pointer to corresponding mbuf. */
#defineRTE_MBUF_FROM_BADDR(ba)(((structrte_mbuf*)(ba)) - 1)/**
* Given the pointer to mbuf returns an address where it'sbuf_addr
* should point to.
*/
#defineRTE_MBUF_TO_BADDR(mb)(((structrte_mbuf*)(mb)) + 1)These macros are used for attaching/detaching mbuf's to each other. This is the way the code retrieves the direct buffer from an indirect one, and vica versa. I think if we want to keep the metadata feature (which I guess is quite important), we need to add a pointer to
rte_mbuf, which helps the direct and indirect structs to find each other. Something like:struct
rte_mbuf*attach; /**< Points to the other buffer if this
one
is (in)direct. Otherwise NULL. */
Quoted 6 lines
Quoted 1 line
What do you think?
I've just sent a patch that should fix this issue.
http://dpdk.org/ml/archives/dev/2015-March/015722.htmlLet me know if you have any comment on it.
I have some comments for the first patch:
examples/vhost/main.c
Quoted 4 lines
examples/vhost/main.c +0
-0
I've sent in a separate patch for this file, I think it's just easier to ditch the old copy-pasted code, see "[PATCH] examples/vhost: use library routines instead of local copies"
Quoted 14 lines
lib/librte_mbuf/rte_mbuf.h +1
-1@@ -268,7 +268,7 @@ struct rte_mbuf {uint16_t data_len;/**< Amount of data in segment buffer. */ uint32_t pkt_len;/**< Total pkt len: sum of all segments. */ uint16_t vlan_tci;/**< VLAN Tag Control Identifier (CPU order) */ -uint16_t reserved; +uint16_t priv_size;/**< size of the application private data */ union {uint32_t rss;/**< RSS hash result if RSS enabled */ struct {
lib/librte_mbuf/rte_mbuf.h struct rte_mbuf { · line 328
Quoted 10 lines
lib/librte_mbuf/rte_mbuf.h +6
-1@@ -320,15 +320,38 @@ struct rte_mbuf { } __rte_cache_aligned;/** -* Given the buf_addr returns the pointer to corresponding mbuf. +* Return the mbuf owning the given data buffer address. +* +* @param mi +* The pointer to the indirect mbuf. +* @param buffer_addr +* The address of the data buffer of the direct mbuf.
You don't need this parameter, it's mi->buf_addr.
lib/librte_mbuf/rte_mbuf.h static inline void rte_pktmbuf_attach(struct rte_mbuf *mi, struct rte_mbuf *md) · line 771
Quoted 7 lines
lib/librte_mbuf/rte_mbuf.h +1
-1@@ -744,9 +767,11 @@ static inline void rte_pktmbuf_attach(struct rte_mbuf *mi, struct rte_mbuf *md)static inline void rte_pktmbuf_detach (struct rte_mbuf *m) {const struct rte_mempool *mp = m->pool; -void *buf =RTE_MBUF_TO_BADDR (m); +void *buf =rte_mbuf_to_baddr (m);uint32_t buf_len = mp->elt_size -sizeof (*m);
I don't see any reason to keep buf and buf_len, just assign straight to m->buf_addr and *len. Besides that, you need to deduct m->priv_size from buf_len.
lib/librte_mbuf/rte_mbuf.h static inline void rte_pktmbuf_attach(struct rte_mbuf *mi, struct rte_mbuf *md) · line 777
Quoted 7 lines
lib/librte_mbuf/rte_mbuf.h +3
-1- m->buf_physaddr =rte_mempool_virt2phy (mp, m) +sizeof (*m); + + m->buf_physaddr =rte_mempool_virt2phy (mp, m) +sizeof (*m) + + m->priv_size; m->buf_addr = buf; m->buf_len = (uint16_t )buf_len;
The rest of the series looks good,
_______________________________________________
dev mailing list
[email protected]
http://openvswitch.org/mailman/listinfo/dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Zoltan,
Quoted 9 lines · Zoltan Kiss
I have some comments for the first patch:
examples/vhost/main.c
Quoted 4 lines
examples/vhost/main.c +0
-0I've sent in a separate patch for this file, I think it's just easier to ditch the old copy-pasted code, see "[PATCH] examples/vhost: use library routines instead of local copies"
As Konstantin stated in the other thread, this modification is wrong, sorry. I don't see why I changed this code instead of keeping the initial logic. I'll fix that in the v2.
Quoted 9 lines
Quoted 8 lines
/**
- * Given thebuf_addrreturns the pointer to corresponding mbuf.
+ * Return the mbuf owning the given data buffer address.
+ *
+ * @param mi
+ * The pointer to the indirect mbuf.
+ * @parambuffer_addr
+ * The address of the data buffer of the direct mbuf.You don't need this parameter, it's
mi->buf_addr.
That's correct. In this case, I propose to change the name of the function in:
struct rte_mbuf *rte_mbuf_from_indirect(struct rte_mbuf *mi)
Quoted 11 lines
examples/vhost/main.c static inline void rte_pktmbuf_attach(struct · line 772
Quoted 8 lines
examples/vhost/main.c +1
-1@@ -744,9 +767,11 @@ static inline void rte_pktmbuf_attach(struct rte_mbuf *mi, struct rte_mbuf *md) static inline void rte_pktmbuf_detach(struct rte_mbuf *m) { const struct rte_mempool *mp = m->pool; -void *buf =RTE_MBUF_TO_BADDR (m); +void *buf =rte_mbuf_to_baddr (m);uint32_t buf_len = mp->elt_size -sizeof (*m);I don't see any reason to keep buf and
buf_len, just assign straight tom->buf_addrand *len.
Besides that, you need to deductm->priv_sizefrombuf_len.
Agree. I suggest something like:
static inline void rte_pktmbuf_detach(struct rte_mbuf *m)
{
const struct rte_mempool *mp = m->pool;
void *buf = rte_mbuf_to_baddr(m);
unsigned mhdr_size = (char *)buf - (char *)m;
m->buf_addr = buf;
m->buf_physaddr = rte_mempool_virt2phy(mp, m) + mhdr_size;
m->buf_len = (uint16_t)(mp->elt_size - mhdr_size);...
Quoted 3 lines
The rest of the series looks good,
Reviewed-by Zoltan Kiss <[email protected]>
Thank you for your review. I'll send a v2 today.
Regards,
Olivier