Skip to contentSkip to conversation

skb_linearize

DISCUSSION Michael S. Tsirkin 2 participants 4 messages Sep 16, 2012 – Oct 31, 2012

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I notice that dev_hard_start_xmit might invoke __skb_linearize e.g. if device does not support NETIF_F_SG.

This in turn onvokes __pskb_pull_tail, and
documentation of __pskb_pull_tail says:
&sk_buff MUST have reference count of 1.

I am guessing 'reference count' means users in this context, right? IIUC this is because it modifies skb in a way that
isn't safe if anyone else is looking at the skb.

However, I don't see what guarantees that reference count is 1 when dev_hard_start_xmit invokes linearize. In particular it calls dev_queue_xmit_nit which could queue packets on a network tap.

Could someone help me understand please?

Thanks!

Signature · 1 line
MST
#1 lore raw
Quoted 12 lines · Michael S. Tsirkin

I notice that dev_hard_start_xmit might invoke __skb_linearize e.g. if device does not support NETIF_F_SG.

This in turn onvokes __pskb_pull_tail, and
documentation of __pskb_pull_tail says:
&sk_buff MUST have reference count of 1.

I am guessing 'reference count' means users in this context, right? IIUC this is because it modifies skb in a way that
isn't safe if anyone else is looking at the skb.

Quoted 6 lines

However, I don't see what guarantees that reference count is 1 when dev_hard_start_xmit invokes linearize. In particular it calls dev_queue_xmit_nit which could queue packets on a network tap.

Could someone help me understand please?

Reference count here means references to struct sk_buff itself. The header area and data fragments are allowed to be shared.

dev_queue_xmit_nit() clones the skb for each tap, so the reference count on the original skb remains 1.

Ben.

Signature · 3 lines
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#2 lore raw
Quoted 20 lines · Ben Hutchings
Quoted 12 lines · Michael S. Tsirkin

I notice that dev_hard_start_xmit might invoke __skb_linearize e.g. if device does not support NETIF_F_SG.

This in turn onvokes __pskb_pull_tail, and
documentation of __pskb_pull_tail says:
&sk_buff MUST have reference count of 1.

I am guessing 'reference count' means users in this context, right? IIUC this is because it modifies skb in a way that
isn't safe if anyone else is looking at the skb.

Quoted 6 lines

However, I don't see what guarantees that reference count is 1 when dev_hard_start_xmit invokes linearize. In particular it calls dev_queue_xmit_nit which could queue packets on a network tap.

Could someone help me understand please?

Quoted 7 lines

Reference count here means references to struct sk_buff itself. The header area and data fragments are allowed to be shared.

dev_queue_xmit_nit() clones the skb for each tap, so the reference count on the original skb remains 1.

Ben.

Interesting. But don't skb clones share the fragment list? Maybe I misunderstand? If they do it looks like the following race would be possible:

- skb is cloned and queued e.g. at socket receive queue.
dataref becomes 2.
- On CPU 1, skb_copy_datagram_iovec is called on clone 1, is reads nr_frags and sees value > 1.
- On CPU 2, __skb_linearize is now called on clone 2, it modified the skb so nr_frags is now 0, and does put_page for all frags > 1.
- On CPU 1, skb_copy_datagram_iovec will now use the previously read nr_frags > 1 and access a fragment page that was already freed.

What did I miss?

Quoted 4 lines
Signature · 3 lines
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#3 lore raw
Quoted 24 lines · Michael S. Tsirkin
Quoted 23 lines · Ben Hutchings
Quoted 12 lines · Michael S. Tsirkin

I notice that dev_hard_start_xmit might invoke __skb_linearize e.g. if device does not support NETIF_F_SG.

This in turn onvokes __pskb_pull_tail, and
documentation of __pskb_pull_tail says:
&sk_buff MUST have reference count of 1.

I am guessing 'reference count' means users in this context, right? IIUC this is because it modifies skb in a way that
isn't safe if anyone else is looking at the skb.

Quoted 6 lines

However, I don't see what guarantees that reference count is 1 when dev_hard_start_xmit invokes linearize. In particular it calls dev_queue_xmit_nit which could queue packets on a network tap.

Could someone help me understand please?

Reference count here means references to struct sk_buff itself. The header area and data fragments are allowed to be shared.

Quoted 6 lines
Quoted 4 lines

dev_queue_xmit_nit() clones the skb for each tap, so the reference count on the original skb remains 1.

Ben.

Interesting. But don't skb clones share the fragment list?

Yes.

Quoted 12 lines

Maybe I misunderstand? If they do it looks like the following race would be possible:

- skb is cloned and queued e.g. at socket receive queue.
dataref becomes 2.
- On CPU 1, skb_copy_datagram_iovec is called on clone 1, is reads nr_frags and sees value > 1.
- On CPU 2, __skb_linearize is now called on clone 2, it modified the skb so nr_frags is now 0, and does put_page for all frags > 1.
- On CPU 1, skb_copy_datagram_iovec will now use the previously read nr_frags > 1 and access a fragment page that was already freed.

Quoted 1 line

What did I miss?

__skb_linearize() calls __pskb_pull_tail(), which starts with:

	if (eat > 0 || skb_cloned(skb)) {
		if (pskb_expand_head(skb, 0, eat > 0 ? eat + 128 : 0,
				     GFP_ATOMIC))
			return NULL;
	}

pskb_expand_head() will then create a new unshared head area for the skb being linearised, and will add a reference to each fragment page (skb_frag_ref()). __pskb_pull_tail() unreferences the pages later, as you say, but this all cancels out.

(pskb_expand_head() will also cancel zero-copy (skb_orphan_frags() -> skb_copy_ubufs()), which seems like it should be done only after the head area has been unshared. But skb_clone() will also do that, so I don't know how the zero-copy flag would still be set on a cloned skb.)

Ben.

Signature · 3 lines
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.