skb frag: kmap_atomic fixes
Subsystems (3)
v2 cover skb frag: kmap_atomic fixes
From: Willem de Bruijn <[email protected]>
skb frags may be backed by highmem and/or compound pages. Various code calls kmap_atomic to safely access highmem pages. But this needs additional care for compound pages. Fix a few issues:
patch 1 expect kmap mappings with CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP patch 2 fixes kmap_atomic + compound page support in skb_seq_read patch 3 fixes kmap_atomic + compound page support in esp
changes
v1->v2
- in patch 2, increase frag_off size from u16 to u32
Willem de Bruijn (3):
net: support kmap_local forced debugging in skb_frag_foreach
net: compound page support in skb_seq_read
esp: avoid unneeded kmap_atomic call
include/linux/skbuff.h | 3 ++- net/core/skbuff.c | 28 +++++++++++++++++++++++----- net/ipv4/esp4.c | 7 +------ net/ipv6/esp6.c | 7 +------ 4 files changed, 27 insertions(+), 18 deletions(-)
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
PATCH v2 1/3 net: support kmap_local forced debugging in skb_frag_foreach
From: Willem de Bruijn <[email protected]>
Skb frags may be backed by highmem and/or compound pages. Highmem pages need kmap_atomic mappings to access. But kmap_atomic maps a single page, not the entire compound page.
skb_foreach_page iterates over an skb frag, in one step in the common case, page by page only if kmap_atomic must be called for each page. The decision logic is captured in skb_frag_must_loop.
CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP extends kmap from highmem to all pages, to increase code coverage.
Extend skb_frag_must_loop to this new condition.
--- include/linux/skbuff.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/skbuff.h +1 -1
@@ -366,7 +366,7 @@ static inline void skb_frag_size_sub(skb_frag_t *frag, int delta)static inline bool skb_frag_must_loop (struct page *p) {#if defined(CONFIG_HIGHMEM ) -if (PageHighMem (p)) +if (IS_ENABLED (CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP ) ||PageHighMem (p))return true;#endif return false;
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
PATCH v2 2/3 net: compound page support in skb_seq_read
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
skb_seq_read iterates over an skb, returning pointer and length of the next data range with each call.
It relies on kmap_atomic to access highmem pages when needed.
An skb frag may be backed by a compound page, but kmap_atomic maps only a single page. There are not enough kmap slots to always map all pages concurrently.
Instead, if kmap_atomic is needed, iterate over each page.
As this increases the number of calls, avoid this unless needed. The necessary condition is captured in skb_frag_must_loop.
I tried to make the change as obvious as possible. It should be easy to verify that nothing changes if skb_frag_must_loop returns false.
Tested:
On an x86 platform with
CONFIG_HIGHMEM=y
CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP=y
CONFIG_NETFILTER_XT_MATCH_STRING=y
Run
ip link set dev lo mtu 1500
iptables -A OUTPUT -m string --string 'badstring' -algo bm -j ACCEPT
dd if=/dev/urandom of=in bs=1M count=20
nc -l -p 8000 > /dev/null &
nc -w 1 -q 0 localhost 8000 < in---
Changes:
v1->v2
- increase frag_off size to u32
--- include/linux/skbuff.h | 1 + net/core/skbuff.c | 28 +++++++++++++++++++++++----- 2 files changed, 24 insertions(+), 5 deletions(-)
include/linux/skbuff.h +1 -0
@@ -1203,6 +1203,7 @@ struct skb_seq_state {struct sk_buff *root_skb;struct sk_buff *cur_skb;__u8 *frag_data; +__u32 frag_off; }; void skb_prepare_seq_read(struct sk_buff *skb, unsigned int from,
net/core/skbuff.c +23 -5
@@ -3442,6 +3442,7 @@ void skb_prepare_seq_read(struct sk_buff *skb, unsigned int from, st->root_skb = st->cur_skb = skb; st->frag_idx = st->stepped_offset =0 ; st->frag_data =NULL ; + st->frag_off =0 ; }EXPORT_SYMBOL (skb_prepare_seq_read);
@@ -3496,14 +3497,27 @@ unsigned int skb_seq_read(unsigned int consumed, const u8 **data, st->stepped_offset +=skb_headlen (st->cur_skb);while (st->frag_idx <skb_shinfo (st->cur_skb)->nr_frags) { +unsigned int pg_idx, pg_off, pg_sz; + frag = &skb_shinfo (st->cur_skb)->frags[st->frag_idx]; - block_limit =skb_frag_size (frag) + st->stepped_offset; + pg_idx =0 ; + pg_off =skb_frag_off (frag); + pg_sz =skb_frag_size (frag); + +if (skb_frag_must_loop (skb_frag_page (frag))) { + pg_idx = (pg_off + st->frag_off) >>PAGE_SHIFT ; + pg_off =offset_in_page (pg_off + st->frag_off); + pg_sz = min_t(unsigned int, pg_sz - st->frag_off, +PAGE_SIZE - pg_off); + } + + block_limit = pg_sz + st->stepped_offset;if (abs_offset < block_limit) {if (!st->frag_data) - st->frag_data =kmap_atomic (skb_frag_page (frag)); + st->frag_data =kmap_atomic (skb_frag_page (frag) + pg_idx); - *data = (u8 *) st->frag_data +skb_frag_off (frag) + + *data = (u8 *)st->frag_data + pg_off + (abs_offset - st->stepped_offset);return block_limit - abs_offset;
@@ -3514,8 +3528,12 @@ unsigned int skb_seq_read(unsigned int consumed, const u8 **data, st->frag_data =NULL ; } - st->frag_idx++; - st->stepped_offset +=skb_frag_size (frag); + st->stepped_offset += pg_sz; + st->frag_off += pg_sz; +if (st->frag_off ==skb_frag_size (frag)) { + st->frag_off =0 ; + st->frag_idx++; + } }if (st->frag_data) {
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
PATCH v2 3/3 esp: avoid unneeded kmap_atomic call
In reply to Willem de BruijnFrom: Willem de Bruijn <[email protected]>
esp(6)_output_head uses skb_page_frag_refill to allocate a buffer for the esp trailer.
It accesses the page with kmap_atomic to handle highmem. But skb_page_frag_refill can return compound pages, of which kmap_atomic only maps the first underlying page.
skb_page_frag_refill does not return highmem, because flag __GFP_HIGHMEM is not set. ESP uses it in the same manner as TCP. That also does not call kmap_atomic, but directly uses page_address, in skb_copy_to_page_nocache. Do the same for ESP.
This issue has become easier to trigger with recent kmap local debugging feature CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP.
--- net/ipv4/esp4.c | 7 +------ net/ipv6/esp6.c | 7 +------ 2 files changed, 2 insertions(+), 12 deletions(-)
net/ipv4/esp4.c +1 -6
@@ -443,7 +443,6 @@ static int esp_output_encap(struct xfrm_state *x, struct sk_buff *skb,int esp_output_head (struct xfrm_state *x,struct sk_buff *skb,struct esp_info *esp) {u8 *tail; -u8 *vaddr;int nfrags;int esph_offset;struct page *page;
@@ -485,14 +484,10 @@ int esp_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info * page = pfrag->page;get_page (page); - vaddr =kmap_atomic (page); - - tail = vaddr + pfrag->offset; + tail =page_address (page) + pfrag->offset;esp_output_fill_trailer (tail, esp->tfclen, esp->plen, esp->proto); -kunmap_atomic (vaddr); - nfrags =skb_shinfo (skb)->nr_frags; __skb_fill_page_desc(skb, nfrags, page, pfrag->offset,
net/ipv6/esp6.c +1 -6
@@ -478,7 +478,6 @@ static int esp6_output_encap(struct xfrm_state *x, struct sk_buff *skb,int esp6_output_head (struct xfrm_state *x,struct sk_buff *skb,struct esp_info *esp) {u8 *tail; -u8 *vaddr;int nfrags;int esph_offset;struct page *page;
@@ -519,14 +518,10 @@ int esp6_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info page = pfrag->page;get_page (page); - vaddr =kmap_atomic (page); - - tail = vaddr + pfrag->offset; + tail =page_address (page) + pfrag->offset;esp_output_fill_trailer (tail, esp->tfclen, esp->plen, esp->proto); -kunmap_atomic (vaddr); - nfrags =skb_shinfo (skb)->nr_frags; __skb_fill_page_desc(skb, nfrags, page, pfrag->offset,
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
Re: [PATCH net v2 3/3] esp: avoid unneeded kmap_atomic call
Quoted 14 lines · Willem de Bruijn
From: Willem de Bruijn <[email protected]>
esp(6)_output_head uses
skb_page_frag_refillto allocate a buffer for the esp trailer.It accesses the page with
kmap_atomicto handle highmem. Butskb_page_frag_refillcan return compound pages, of whichkmap_atomiconly maps the first underlying page.
skb_page_frag_refilldoes not return highmem, because flag__GFP_HIGHMEMis not set. ESP uses it in the same manner as TCP. That also does not callkmap_atomic, but directly usespage_address, inskb_copy_to_page_nocache. Do the same for ESP.
Quoted 7 lines
This issue has become easier to trigger with recent kmap local debugging feature
CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP.Fixes cac2661c53f3 ("esp4: Avoid skb_cow_data whenever possible")Fixes 03e2a30f6a27 ("esp6: Avoid skb_cow_data whenever possible")Signed-off-by Willem de Bruijn <[email protected]>Cc Steffen Klassert <[email protected]>
As this patchset goes through the net tree:
Thanks!
This series was applied to netdev/net.git (refs/heads/master):
Hello:
This series was applied to netdev/net.git (refs/heads/master):
Quoted 11 lines · On Sat, 9 Jan 2021 17:18:31 -0500 you wrote:
From: Willem de Bruijn <[email protected]>
skb frags may be backed by highmem and/or compound pages. Various code calls
kmap_atomicto safely access highmem pages. But this needs additional care for compound pages. Fix a few issues:patch 1 expect kmap mappings with
CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAPpatch 2 fixeskmap_atomic+ compound page support inskb_seq_readpatch 3 fixeskmap_atomic+ compound page support in esp[...]
Here is the summary with links:
- [net,v2,1/3] net: support kmap_local forced debugging in skb_frag_foreach
https://git.kernel.org/netdev/net/c/29766bcffad0 - [net,v2,2/3] net: compound page support in skb_seq_read https://git.kernel.org/netdev/net/c/97550f6fa592 - [net,v2,3/3] esp: avoid unneeded kmap_atomic call https://git.kernel.org/netdev/net/c/9bd6b629c39e
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html