Skip to contentSkip to conversation

Internal error: Oops from inet_frag_find, when inserting a IP frag into a rhashtable

DISCUSSION [email protected] 2 participants 3 messages May 18, 2021 – May 26, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

We are observing BUG_ON from the __get_vm_area_node when processing the IP fragments in the context of NET_RX.

When the rehashing is in progress and an insertion is attempted, we may end up doing a bucket_table_alloc, which results in BUG_ON if in interrupt context.
Is it the case that the slow path code has to be executed only in the context of rht_deferred_worker and not in any other context? OR are these scenario's not anticipated and is a missed handling?

Please provide your suggestions on proceeding further with this issue.

static struct vm_struct *__get_vm_area_node(unsigned long size,
unsigned long align, unsigned long flags, unsigned long
start,
unsigned long end, int node, gfp_t gfp_mask, const void
*caller)

 {
    struct vmap_area *va;
    struct vm_struct *area;

      BUG_ON(in_interrupt()); --> Point of panic.

Panic stack:

784.185010: <2> (2)[71:kworker/2:1][20210408_17:01:54.754253]@2 Workqueue: events rht_deferred_worker
784.185020: <2> (2)[71:kworker/2:1][20210408_17:01:54.754262]@2
pstate: 00c00005 (nzcv daif +PAN +UAO)
784.185032: <2> (2)[71:kworker/2:1][20210408_17:01:54.754274]@2 pc
: __get_vm_area_node.llvm.17374696036975823682+0x1ac/0x1c8
784.185041: <2> (2)[71:kworker/2:1][20210408_17:01:54.754283]@2 lr
: __vmalloc_node_flags_caller+0xb4/0x170
784.185046: <2> (2)[71:kworker/2:1][20210408_17:01:54.754289]@2 sp
: ffffffc0104135a0
784.185052: <2> (2)[71:kworker/2:1][20210408_17:01:54.754295]@2
x29: ffffffc0104135a0 x28: ffffff82ccbcae50
784.185060: <2> (2)[71:kworker/2:1][20210408_17:01:54.754303]@2
x27: ffffff82eea8d3c0 x26: ffffff800ad044b0
784.185067: <2> (2)[71:kworker/2:1][20210408_17:01:54.754311]@2
x25: fffffffffffffffe x24: fffffffffffffff5
784.185075: <2> (2)[71:kworker/2:1][20210408_17:01:54.754318]@2
x23: 0000000000001000 x22: 0068000000000f13
784.185082: <2> (2)[71:kworker/2:1][20210408_17:01:54.754326]@2
x21: 00000000ffffffff x20: 0000000000008040
784.185090: <2> (2)[71:kworker/2:1][20210408_17:01:54.754333]@2
x19: 0000000000002b20 x18: ffffffc0103ad0c0
784.185097: <2> (2)[71:kworker/2:1][20210408_17:01:54.754340]@2
x17: 00000000e1ba4003 x16: d101500000cf012a
784.185104: <2> (2)[71:kworker/2:1][20210408_17:01:54.754347]@2
x15: 52d39bfeffbfebd0 x14: 3130393837363534
784.185111: <2> (2)[71:kworker/2:1][20210408_17:01:54.754355]@2
x13: 000000003b1ad624 x12: 0000000000000000
784.185118: <2> (2)[71:kworker/2:1][20210408_17:01:54.754362]@2
x11: 000000000000002e x10: 0000000000000600
784.185126: <2> (2)[71:kworker/2:1][20210408_17:01:54.754369]@2 x9
: 00000000002c022a x8 : 0000000000000101
784.185133: <2> (2)[71:kworker/2:1][20210408_17:01:54.754376]@2 x7
: f6da030000000000 x6 : ffffffe14adb3990
784.185140: <2> (2)[71:kworker/2:1][20210408_17:01:54.754384]@2 x5
: 0000000000002b20 x4 : fffffffebfff0000
784.185148: <2> (2)[71:kworker/2:1][20210408_17:01:54.754391]@2 x3
: ffffffc010000000 x2 : 0000000000000022
784.185155: <2> (2)[71:kworker/2:1][20210408_17:01:54.754398]@2 x1
: 0000000000000001 x0 : 0000000000009000
784.185164: <2> (2)[71:kworker/2:1][20210408_17:01:54.754406]@2

Call trace:
  784.185172:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754415]@2  
__get_vm_area_node.llvm.17374696036975823682+0x1ac/0x1c8
   784.185179:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754422]@2  
__vmalloc_node_flags_caller+0xb4/0x170
   784.185189:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754432]@2  
kvmalloc_node+0x40/0xa8
   784.185199:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754442]@2  
rhashtable_insert_rehash+0x84/0x264
   784.185206:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754449]@2  
rhashtable_try_insert+0x3fc/0x478
   784.185213:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754455]@2  
rhashtable_insert_slow+0x34/0x5c
   784.185223:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754466]@2  
__rhashtable_insert_fast+0x368/0x4f0
   784.185234:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754476]@2  
inet_frag_find+0x21c/0x2a8
   784.185244:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754486]@2  
nf_ct_frag6_gather+0x1f4/0x2a0
   784.185252:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754494]@2  
ipv6_defrag+0x58/0x7c
   784.185262:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754504]@2  
nf_hook_slow+0xa8/0x148
   784.185272:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754514]@2  
ipv6_rcv+0x80/0xe4
   784.185282:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754524]@2  
__netif_receive_skb+0x84/0x17c
   784.185290:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754532]@2  
process_backlog+0x15c/0x1b8
   784.185297:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754539]@2  
napi_poll+0x88/0x284
   784.185304:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754547]@2  
net_rx_action+0xbc/0x23c
   784.185314:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754557]@2  
__do_softirq+0x1e8/0x44c
   784.185325:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754567]@2  
__local_bh_enable_ip+0xb8/0xc8
   784.185333:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754576]@2  
local_bh_enable+0x1c/0x28
   784.185340:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754583]@2  
rhashtable_rehash_chain+0x12c/0x1ec
   784.185347:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754590]@2  
rht_deferred_worker+0x13c/0x200
   784.185357:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754600]@2  
process_one_work+0x2cc/0x568
   784.185365:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754607]@2  
worker_thread+0x28c/0x524
   784.185373:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754616]@2  
kthread+0x184/0x194
   784.185381:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754623]@2  
ret_from_fork+0x10/0x18
#1 lore raw
Quoted 9 lines · On Wed, May 19, 2021 at 12:52:27AM +0530, [email protected] wrote:

784.185172: <2> (2)[71:kworker/2:1][20210408_17:01:54.754415]@2

__get_vm_area_node.llvm.17374696036975823682+0x1ac/0x1c8
   784.185179:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754422]@2
__vmalloc_node_flags_caller+0xb4/0x170
   784.185189:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754432]@2
kvmalloc_node+0x40/0xa8
   784.185199:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754442]@2
rhashtable_insert_rehash+0x84/0x264

Something very fishy is going on here.

The code path in rhashtable_insert_rehash cannot possibly trigger vmalloc because it uses GFP_ATOMIC. Is this a pristine upstream kernel or are there patches that may change things?

Cheers,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#2 lore raw
Quoted 11 lines · Herbert Xu
Quoted 9 lines · On Wed, May 19, 2021 at 12:52:27AM +0530, [email protected] wrote:

784.185172: <2> (2)[71:kworker/2:1][20210408_17:01:54.754415]@2

__get_vm_area_node.llvm.17374696036975823682+0x1ac/0x1c8
   784.185179:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754422]@2
__vmalloc_node_flags_caller+0xb4/0x170
   784.185189:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754432]@2
kvmalloc_node+0x40/0xa8
   784.185199:   <2>  (2)[71:kworker/2:1][20210408_17:01:54.754442]@2
rhashtable_insert_rehash+0x84/0x264
Quoted 7 lines

Something very fishy is going on here.

The code path in rhashtable_insert_rehash cannot possibly trigger vmalloc because it uses GFP_ATOMIC. Is this a pristine upstream kernel or are there patches that may change things?

Cheers,

Thanks for the reply Herbert, we have got this crash reported from external folks and I am trying to get the answers for your questions.