mm, something wring in page_lock_anon_vma_read()?

Hi, my system triggers this bug, and the vmcore shows the anon_vma seems be freed. The kernel is RHEL 7.2, and the bug is hard to reproduce, so I don't know if it exists in mainline, any reply is welcome!

[35030.332666] general protection fault: 0000 [#1] SMP
[35030.333016] Modules linked in: veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c rtos_kbox_panic(OE) ipmi_devintf ipmi_si ipmi_msghandler signo_catch(O) cirrus syscopyarea sysfillrect sysimgblt ttm crc32_pclmul ghash_clmulni_intel drm_kms_helper aesni_intel ppdev drm lrw gf128mul parport_pc glue_helper ablk_helper serio_raw cryptd i2c_piix4 parport pcspkr sg floppy i2c_core dm_mod sha512_generic ip_tables sd_mod crc_t10dif crct10dif_generic sr_mod cdrom virtio_console virtio_scsi virtio_net ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci virtio_ring virtio ata_piix libata ext4 mbcache
[35030.333016]  jbd2
[35030.333016] CPU: 3 PID: 48 Comm: kswapd0 Tainted: G           OE  ---- -------   3.10.0-327.36.58.4.x86_64 #1
[35030.333016] Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.8.1-0-g4adadbd-20160826_044443-hghoulaslx112 04/01/2014
[35030.333016] task: ffff8801b2d20000 ti: ffff8801b4c38000 task.ti: ffff8801b4c38000
[35030.333016] RIP: 0010:[<ffffffff810acac5>]  [<ffffffff810acac5>] down_read_trylock+0x5/0x50
[35030.333016] RSP: 0000:ffff8801b4c3ba90  EFLAGS: 00010282
[35030.333016] RAX: 0000000000000000 RBX: ffff8801b3e2a100 RCX: 0000000000000000
[35030.333016] RDX: 0000000000000000 RSI: 0000000000000000 RDI: deb604d497705c5d
[35030.333016] RBP: ffff8801b4c3bab8 R08: ffffea0002c34460 R09: ffff8801b3d7e8a0
[35030.333016] R10: 0000000000000004 R11: fff00000fe000000 R12: ffff8801b3e2a101
[35030.333016] R13: ffffea0002c34440 R14: deb604d497705c5d R15: ffffea0002c34440
[35030.333016] FS:  0000000000000000(0000) GS:ffff8801bed80000(0000) knlGS:0000000000000000
[35030.333016] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[35030.333016] CR2: 000000c422011080 CR3: 0000000001976000 CR4: 00000000001407e0
[35030.333016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[35030.333016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[35030.333016] Stack:
[35030.333016]  ffffffff811b2795 ffffea0002c34440 0000000000000000 000000000000000f
[35030.333016]  0000000000000001 ffff8801b4c3bb30 ffffffff811b2a17 ffff8800a712d640
[35030.333016]  000000000c4229e2 ffff8801b4c3bb80 0000000100000000 000000000c41fe38
[35030.333016] Call Trace:
[35030.333016]  [<ffffffff811b2795>] ? page_lock_anon_vma_read+0x55/0x110
[35030.333016]  [<ffffffff811b2a17>] page_referenced+0x1c7/0x350
[35030.333016]  [<ffffffff8118d9b4>] shrink_active_list+0x1e4/0x400
[35030.333016]  [<ffffffff8118e08d>] shrink_lruvec+0x4bd/0x770
[35030.333016]  [<ffffffff8118e3b6>] shrink_zone+0x76/0x1a0
[35030.333016]  [<ffffffff8118f6cc>] balance_pgdat+0x49c/0x610
[35030.333016]  [<ffffffff8118f9b3>] kswapd+0x173/0x450
[35030.333016]  [<ffffffff810a8a00>] ? wake_up_atomic_t+0x30/0x30
[35030.333016]  [<ffffffff8118f840>] ? balance_pgdat+0x610/0x610
[35030.333016]  [<ffffffff810a79bf>] kthread+0xcf/0xe0
[35030.333016]  [<ffffffff810a78f0>] ? kthread_create_on_node+0x120/0x120
[35030.333016]  [<ffffffff81665bd8>] ret_from_fork+0x58/0x90
[35030.333016]  [<ffffffff810a78f0>] ? kthread_create_on_node+0x120/0x120
[35030.333016] Code: 00 ba ff ff ff ff 48 89 d8 f0 48 0f c1 10 79 05 e8 31 06 27 00 5b 5d c3 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7
[35030.333016] RIP  [<ffffffff810acac5>] down_read_trylock+0x5/0x50
[35030.333016]  RSP <ffff8801b4c3ba90>
[35030.333016] ------------[ cut here ]------------

struct page {
flags = 9007194960298056,
mapping = 0xffff8801b3e2a101,

  {
    {
      index = 34324593617,
      freelist = 0x7fde7bbd1,
      pfmemalloc = 209,
      thp_mmu_gather = {
        counter = -35144751
      },
      pmd_huge_pte = 0x7fde7bbd1
    },
    {
      counters = 8589934592,
      {
        {
          _mapcount = {
            counter = 0
          },
          {
            inuse = 0,
            objects = 0,
            frozen = 0
          },
          units = 0
        },
        _count = {
          counter = 2
        }
      }
    }
  },
  {
    lru = {
      next = 0xdead000000100100,
      prev = 0xdead000000200200
    },
    {
      next = 0xdead000000100100,
      pages = 2097664,
      pobjects = -559087616
    },
    list = {
      next = 0xdead000000100100,
      prev = 0xdead000000200200
    },
    slab_page = 0xdead000000100100
  },
  {
    private = 0,
    ptl = {
      {
        rlock = {
          raw_lock = {
            {
              head_tail = 0,
              tickets = {
                head = 0,
                tail = 0
              }
            }
          }
        }
      }
    },
    slab_cache = 0x0,
    first_page = 0x0

}
}

crash> struct anon_vma 0xffff8801b3e2a100
struct anon_vma {
root = 0xdeb604d497705c55,

  rwsem = {
    count = -8192007903225070328,
    wait_lock = {
      raw_lock = {
        {
          head_tail = 2955503940,
          tickets = {
            head = 26948,
            tail = 45097
          }
        }
      }
    },
    wait_list = {
      next = 0x559f9107c1b47439,
      prev = 0x3de13f709bfa043b
    }
  },
  refcount = {
    counter = -13243516
  },
  rb_root = {
    rb_node = 0x11dd18f9ce0bb2e9

}
}

This address 0xffff8801b3e2a100 can not find in "kmem -S anon_vma"

The page flags is
crash> kmem -g 0x1FFFFF00080048
FLAGS: 1fffff00080048

  PAGE-FLAG        BIT  VALUE
  PG_uptodate        3  0000008
  PG_active          6  0000040
  PG_swapbacked     19  0080000

.

Quoted 4 lines · Xishi Qiu

Hi, my system triggers this bug, and the vmcore shows the anon_vma seems be freed. The kernel is RHEL 7.2, and the bug is hard to reproduce, so I don't know if it exists in mainline, any reply is welcome!

When we alloc anon_vma, we will init the value of anon_vma->root, so can we set anon_vma->root to NULL when calling anon_vma_free -> kmem_cache_free(anon_vma_cachep, anon_vma);

anon_vma_free()

	...
	anon_vma->root = NULL;
	kmem_cache_free(anon_vma_cachep, anon_vma);

I find if we do this above, system boot failed, why?

Thanks,
Xishi Qiu

Quoted 148 lines

[35030.332666] general protection fault: 0000 [#1] SMP
[35030.333016] Modules linked in: veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c rtos_kbox_panic(OE) ipmi_devintf ipmi_si ipmi_msghandler signo_catch(O) cirrus syscopyarea sysfillrect sysimgblt ttm crc32_pclmul ghash_clmulni_intel drm_kms_helper aesni_intel ppdev drm lrw gf128mul parport_pc glue_helper ablk_helper serio_raw cryptd i2c_piix4 parport pcspkr sg floppy i2c_core dm_mod sha512_generic ip_tables sd_mod crc_t10dif crct10dif_generic sr_mod cdrom virtio_console virtio_scsi virtio_net ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci virtio_ring virtio ata_piix libata ext4 mbcache
[35030.333016]  jbd2
[35030.333016] CPU: 3 PID: 48 Comm: kswapd0 Tainted: G           OE  ---- -------   3.10.0-327.36.58.4.x86_64 #1
[35030.333016] Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.8.1-0-g4adadbd-20160826_044443-hghoulaslx112 04/01/2014
[35030.333016] task: ffff8801b2d20000 ti: ffff8801b4c38000 task.ti: ffff8801b4c38000
[35030.333016] RIP: 0010:[<ffffffff810acac5>]  [<ffffffff810acac5>] down_read_trylock+0x5/0x50
[35030.333016] RSP: 0000:ffff8801b4c3ba90  EFLAGS: 00010282
[35030.333016] RAX: 0000000000000000 RBX: ffff8801b3e2a100 RCX: 0000000000000000
[35030.333016] RDX: 0000000000000000 RSI: 0000000000000000 RDI: deb604d497705c5d
[35030.333016] RBP: ffff8801b4c3bab8 R08: ffffea0002c34460 R09: ffff8801b3d7e8a0
[35030.333016] R10: 0000000000000004 R11: fff00000fe000000 R12: ffff8801b3e2a101
[35030.333016] R13: ffffea0002c34440 R14: deb604d497705c5d R15: ffffea0002c34440
[35030.333016] FS:  0000000000000000(0000) GS:ffff8801bed80000(0000) knlGS:0000000000000000
[35030.333016] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[35030.333016] CR2: 000000c422011080 CR3: 0000000001976000 CR4: 00000000001407e0
[35030.333016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[35030.333016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[35030.333016] Stack:
[35030.333016]  ffffffff811b2795 ffffea0002c34440 0000000000000000 000000000000000f
[35030.333016]  0000000000000001 ffff8801b4c3bb30 ffffffff811b2a17 ffff8800a712d640
[35030.333016]  000000000c4229e2 ffff8801b4c3bb80 0000000100000000 000000000c41fe38
[35030.333016] Call Trace:
[35030.333016]  [<ffffffff811b2795>] ? page_lock_anon_vma_read+0x55/0x110
[35030.333016]  [<ffffffff811b2a17>] page_referenced+0x1c7/0x350
[35030.333016]  [<ffffffff8118d9b4>] shrink_active_list+0x1e4/0x400
[35030.333016]  [<ffffffff8118e08d>] shrink_lruvec+0x4bd/0x770
[35030.333016]  [<ffffffff8118e3b6>] shrink_zone+0x76/0x1a0
[35030.333016]  [<ffffffff8118f6cc>] balance_pgdat+0x49c/0x610
[35030.333016]  [<ffffffff8118f9b3>] kswapd+0x173/0x450
[35030.333016]  [<ffffffff810a8a00>] ? wake_up_atomic_t+0x30/0x30
[35030.333016]  [<ffffffff8118f840>] ? balance_pgdat+0x610/0x610
[35030.333016]  [<ffffffff810a79bf>] kthread+0xcf/0xe0
[35030.333016]  [<ffffffff810a78f0>] ? kthread_create_on_node+0x120/0x120
[35030.333016]  [<ffffffff81665bd8>] ret_from_fork+0x58/0x90
[35030.333016]  [<ffffffff810a78f0>] ? kthread_create_on_node+0x120/0x120
[35030.333016] Code: 00 ba ff ff ff ff 48 89 d8 f0 48 0f c1 10 79 05 e8 31 06 27 00 5b 5d c3 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7
[35030.333016] RIP  [<ffffffff810acac5>] down_read_trylock+0x5/0x50
[35030.333016]  RSP <ffff8801b4c3ba90>
[35030.333016] ------------[ cut here ]------------

struct page {
flags = 9007194960298056,
mapping = 0xffff8801b3e2a101,

  {
    {
      index = 34324593617,
      freelist = 0x7fde7bbd1,
      pfmemalloc = 209,
      thp_mmu_gather = {
        counter = -35144751
      },
      pmd_huge_pte = 0x7fde7bbd1
    },
    {
      counters = 8589934592,
      {
        {
          _mapcount = {
            counter = 0
          },
          {
            inuse = 0,
            objects = 0,
            frozen = 0
          },
          units = 0
        },
        _count = {
          counter = 2
        }
      }
    }
  },
  {
    lru = {
      next = 0xdead000000100100,
      prev = 0xdead000000200200
    },
    {
      next = 0xdead000000100100,
      pages = 2097664,
      pobjects = -559087616
    },
    list = {
      next = 0xdead000000100100,
      prev = 0xdead000000200200
    },
    slab_page = 0xdead000000100100
  },
  {
    private = 0,
    ptl = {
      {
        rlock = {
          raw_lock = {
            {
              head_tail = 0,
              tickets = {
                head = 0,
                tail = 0
              }
            }
          }
        }
      }
    },
    slab_cache = 0x0,
    first_page = 0x0

}
}

crash> struct anon_vma 0xffff8801b3e2a100
struct anon_vma {
root = 0xdeb604d497705c55,

  rwsem = {
    count = -8192007903225070328,
    wait_lock = {
      raw_lock = {
        {
          head_tail = 2955503940,
          tickets = {
            head = 26948,
            tail = 45097
          }
        }
      }
    },
    wait_list = {
      next = 0x559f9107c1b47439,
      prev = 0x3de13f709bfa043b
    }
  },
  refcount = {
    counter = -13243516
  },
  rb_root = {
    rb_node = 0x11dd18f9ce0bb2e9

}
}

This address 0xffff8801b3e2a100 can not find in "kmem -S anon_vma"

The page flags is
crash> kmem -g 0x1FFFFF00080048
FLAGS: 1fffff00080048

Quoted 8 lines

  PAGE-FLAG        BIT  VALUE
  PG_uptodate        3  0000008
  PG_active          6  0000040
  PG_swapbacked     19  0080000

.

.

Quoted 11 lines · Xishi Qiu

Quoted 4 lines · Xishi Qiu

Hi, my system triggers this bug, and the vmcore shows the anon_vma seems be freed. The kernel is RHEL 7.2, and the bug is hard to reproduce, so I don't know if it exists in mainline, any reply is welcome!

When we alloc anon_vma, we will init the value of anon_vma->root, so can we set anon_vma->root to NULL when calling anon_vma_free -> kmem_cache_free(anon_vma_cachep, anon_vma);

Quoted 7 lines

anon_vma_free()

	...
	anon_vma->root = NULL;
	kmem_cache_free(anon_vma_cachep, anon_vma);

I find if we do this above, system boot failed, why?

If anon_vma was freed, we should not to access the root_anon_vma, because it maybe also freed(e.g. anon_vma == root_anon_vma), right?

page_lock_anon_vma_read()

	...
	anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON);
	root_anon_vma = ACCESS_ONCE(anon_vma->root);
	if (down_read_trylock(&root_anon_vma->rwsem)) {  // it's not safe
	...
	if (!atomic_inc_not_zero(&anon_vma->refcount)) {  // check anon_vma was not freed
	...
	anon_vma_lock_read(anon_vma);  // it's safe
	...

Quoted 24 lines · Xishi Qiu

Quoted 11 lines · Xishi Qiu

Quoted 4 lines · Xishi Qiu

Hi, my system triggers this bug, and the vmcore shows the anon_vma seems be freed. The kernel is RHEL 7.2, and the bug is hard to reproduce, so I don't know if it exists in mainline, any reply is welcome!

When we alloc anon_vma, we will init the value of anon_vma->root, so can we set anon_vma->root to NULL when calling anon_vma_free -> kmem_cache_free(anon_vma_cachep, anon_vma);

Quoted 7 lines

anon_vma_free()

	...
	anon_vma->root = NULL;
	kmem_cache_free(anon_vma_cachep, anon_vma);

I find if we do this above, system boot failed, why?

If anon_vma was freed, we should not to access the root_anon_vma, because it maybe also freed(e.g. anon_vma == root_anon_vma), right?

page_lock_anon_vma_read()

Quoted 9 lines

	...
	anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON);
	root_anon_vma = ACCESS_ONCE(anon_vma->root);
	if (down_read_trylock(&root_anon_vma->rwsem)) {  // it's not safe
	...
	if (!atomic_inc_not_zero(&anon_vma->refcount)) {  // check anon_vma was not freed
	...
	anon_vma_lock_read(anon_vma);  // it's safe
	...

You're ignoring the rcu_read_lock() on entry to page_lock_anon_vma_read(), and the SLAB_DESTROY_BY_RCU (recently renamed SLAB_TYPESAFE_BY_RCU) nature of the anon_vma_cachep kmem cache. It is not safe to muck with anon_vma-> root in anon_vma_free(), others could still be looking at it.

Hugh

.

Quoted 8 lines · Hugh Dickins

You're ignoring the rcu_read_lock() on entry to page_lock_anon_vma_read(), and the SLAB_DESTROY_BY_RCU (recently renamed SLAB_TYPESAFE_BY_RCU) nature of the anon_vma_cachep kmem cache. It is not safe to muck with anon_vma-> root in anon_vma_free(), others could still be looking at it.

Hugh

Hi Hugh,

Thanks for your reply.

SLAB_DESTROY_BY_RCU will let it call call_rcu() in free_slab(), but if the anon_vma *reuse* by someone again, access root_anon_vma is not safe, right?

e.g. if I clean the root pointer before free it, then access root_anon_vma in page_lock_anon_vma_read() is NULL pointer access, right?

anon_vma_free()

	...
	anon_vma->root = NULL;
	kmem_cache_free(anon_vma_cachep, anon_vma);
	...

Thanks,
Xishi Qiu

Quoted 10 lines · Xishi Qiu

Quoted 8 lines · Hugh Dickins

You're ignoring the rcu_read_lock() on entry to page_lock_anon_vma_read(), and the SLAB_DESTROY_BY_RCU (recently renamed SLAB_TYPESAFE_BY_RCU) nature of the anon_vma_cachep kmem cache. It is not safe to muck with anon_vma-> root in anon_vma_free(), others could still be looking at it.

Hugh

Quoted 6 lines

Hi Hugh,

Thanks for your reply.

SLAB_DESTROY_BY_RCU will let it call call_rcu() in free_slab(), but if the anon_vma *reuse* by someone again, access root_anon_vma is not safe, right?

That is safe, on reuse it is still a struct anon_vma; then the test for !page_mapped(page) will show that it's no longer a reliable anon_vma for this page, so page_lock_anon_vma_read() returns NULL.

But of course, if page->_mapcount has been corrupted or misaccounted, it may think page_mapped(page) when actually page is not mapped, and the anon_vma is not good for it.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2),

Quoted 3 lines

e.g. if I clean the root pointer before free it, then access root_anon_vma in page_lock_anon_vma_read() is NULL pointer access, right?

Yes, cleaning root pointer before free may result in NULL pointer access.

Hugh

Quoted 9 lines

anon_vma_free()

	...
	anon_vma->root = NULL;
	kmem_cache_free(anon_vma_cachep, anon_vma);
	...

Thanks,
Xishi Qiu

.

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Any ideas?

Quoted 8 lines · Xishi Qiu

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Quoted 1 line

Any ideas?

Sorry, no. I assume that _mapcount has been misaccounted, for example a pte mapped in on top of another pte; but cannot begin tell you where in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.

Hugh

.

Quoted 9 lines · Hugh Dickins

Quoted 8 lines · Xishi Qiu

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Quoted 9 lines

Quoted 1 line

Any ideas?

Sorry, no. I assume that _mapcount has been misaccounted, for example a pte mapped in on top of another pte; but cannot begin tell you where in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.

Hugh

.

Hi, Hugh

I find the following message from the dmesg.

[26068.316592] BUG: Bad rss-counter state mm:ffff8800a7de2d80 idx:1 val:1

I can prove that the __mapcount is misaccount. when task is exited. the rmap still exist.

Thanks
zhongjiang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 21 lines · zhong jiang

Quoted 9 lines · Hugh Dickins

Quoted 8 lines · Xishi Qiu

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Quoted 9 lines

Quoted 1 line

Any ideas?

Sorry, no. I assume that _mapcount has been misaccounted, for example a pte mapped in on top of another pte; but cannot begin tell you where in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.

Hugh

.

Hi, Hugh

Quoted 6 lines

I find the following message from the dmesg.

[26068.316592] BUG: Bad rss-counter state mm:ffff8800a7de2d80 idx:1 val:1

I can prove that the __mapcount is misaccount. when task is exited. the rmap still exist.

Check if the kernel in question contains this commit: ad33bb04b2a6 ("mm:
thp: fix SMP race condition between THP page fault and MADV_DONTNEED")

Quoted 9 lines

Thanks
zhongjiang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

.

Quoted 26 lines · Vlastimil Babka

Quoted 25 lines · zhong jiang

Quoted 9 lines · Hugh Dickins

Quoted 8 lines · Xishi Qiu

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Quoted 9 lines

Quoted 1 line

Any ideas?

Sorry, no. I assume that _mapcount has been misaccounted, for example a pte mapped in on top of another pte; but cannot begin tell you where in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.

Hugh

.

Hi, Hugh

I find the following message from the dmesg.

[26068.316592] BUG: Bad rss-counter state mm:ffff8800a7de2d80 idx:1 val:1

Quoted 4 lines

Quoted 2 lines

I can prove that the __mapcount is misaccount. when task is exited. the rmap still exist.

Check if the kernel in question contains this commit: ad33bb04b2a6 ("mm:
thp: fix SMP race condition between THP page fault and MADV_DONTNEED")

HI, Vlastimil

I miss the patch.

when I read the patch. I find the following issue. but I am sure it is right.

if (unlikely(pmd_trans_unstable(pmd)))
return 0;
/*
* A regular pmd is established and it can't morph into a huge pmd
* from under us anymore at this point because we hold the mmap_sem
* read mode and khugepaged takes it in write mode. So now it's
* safe to run pte_offset_map().
*/
pte = pte_offset_map(pmd, address);

after pmd_trans_unstable call, without any protect method. by the comments,
it think the pte_offset_map is safe. before pte_offset_map call, it still may be
unstable. it is possible?

Thanks
zhongjiang

Quoted 12 lines

Quoted 9 lines

Thanks
zhongjiang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 29 lines · zhong jiang

Quoted 28 lines · Vlastimil Babka

Quoted 27 lines · zhong jiang

Quoted 9 lines · Hugh Dickins

Quoted 8 lines · Xishi Qiu

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Quoted 9 lines

Quoted 1 line

Any ideas?

Sorry, no. I assume that _mapcount has been misaccounted, for example a pte mapped in on top of another pte; but cannot begin tell you where in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.

Hugh

.

Hi, Hugh

I find the following message from the dmesg.

[26068.316592] BUG: Bad rss-counter state mm:ffff8800a7de2d80 idx:1 val:1

I can prove that the __mapcount is misaccount. when task is exited. the rmap still exist.

Quoted 5 lines

Quoted 2 lines

Check if the kernel in question contains this commit: ad33bb04b2a6 ("mm:
thp: fix SMP race condition between THP page fault and MADV_DONTNEED")

HI, Vlastimil

I miss the patch.

Try applying it then, there's good chance the error and crash will go away. Even if your workload doesn't actually run any madvise(MADV_DONTNEED).

Quoted 12 lines

when I read the patch. I find the following issue. but I am sure it is right.

if (unlikely(pmd_trans_unstable(pmd)))
return 0;
/*
* A regular pmd is established and it can't morph into a huge pmd
* from under us anymore at this point because we hold the mmap_sem
* read mode and khugepaged takes it in write mode. So now it's
* safe to run pte_offset_map().
*/
pte = pte_offset_map(pmd, address);

Quoted 3 lines

after pmd_trans_unstable call, without any protect method. by the comments,
it think the pte_offset_map is safe. before pte_offset_map call, it still may be
unstable. it is possible?

IIRC it's "unstable" wrt possible none->huge->none transition. But once we've seen it's a regular pmd via pmd_trans_unstable(), we're safe as a transition from regular pmd can't happen.

Quoted 11 lines

Thanks
zhongjiang

Quoted 9 lines

Quoted 9 lines

Thanks
zhongjiang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 9 lines

Quoted 2 lines

.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

.

Quoted 29 lines · zhong jiang

Quoted 28 lines · Vlastimil Babka

Quoted 27 lines · zhong jiang

Quoted 9 lines · Hugh Dickins

Quoted 8 lines · Xishi Qiu

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Quoted 9 lines

Quoted 1 line

Any ideas?

Sorry, no. I assume that _mapcount has been misaccounted, for example a pte mapped in on top of another pte; but cannot begin tell you where in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.

Hugh

.

Hi, Hugh

I find the following message from the dmesg.

[26068.316592] BUG: Bad rss-counter state mm:ffff8800a7de2d80 idx:1 val:1

I can prove that the __mapcount is misaccount. when task is exited. the rmap still exist.

Quoted 5 lines

Quoted 2 lines

Check if the kernel in question contains this commit: ad33bb04b2a6 ("mm:
thp: fix SMP race condition between THP page fault and MADV_DONTNEED")

HI, Vlastimil

I miss the patch.

Try applying it then, there's good chance the error and crash will go away. Even if your workload doesn't actually run any madvise(MADV_DONTNEED).

Quoted 12 lines

when I read the patch. I find the following issue. but I am sure it is right.

if (unlikely(pmd_trans_unstable(pmd)))
return 0;
/*
* A regular pmd is established and it can't morph into a huge pmd
* from under us anymore at this point because we hold the mmap_sem
* read mode and khugepaged takes it in write mode. So now it's
* safe to run pte_offset_map().
*/
pte = pte_offset_map(pmd, address);

Quoted 3 lines

after pmd_trans_unstable call, without any protect method. by the comments,
it think the pte_offset_map is safe. before pte_offset_map call, it still may be
unstable. it is possible?

IIRC it's "unstable" wrt possible none->huge->none transition. But once we've seen it's a regular pmd via pmd_trans_unstable(), we're safe as a transition from regular pmd can't happen.

Quoted 16 lines

Thanks
zhongjiang

Quoted 12 lines

Quoted 9 lines

Thanks
zhongjiang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

.

Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

.

Quoted 36 lines · Vlastimil Babka

Quoted 34 lines · zhong jiang

Quoted 22 lines · Vlastimil Babka

Quoted 21 lines · zhong jiang

Quoted 9 lines · Hugh Dickins

Quoted 8 lines · Xishi Qiu

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Quoted 9 lines

Quoted 1 line

Any ideas?

Sorry, no. I assume that _mapcount has been misaccounted, for example a pte mapped in on top of another pte; but cannot begin tell you where in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.

Hugh

.

Hi, Hugh

Quoted 8 lines

Quoted 6 lines

I find the following message from the dmesg.

[26068.316592] BUG: Bad rss-counter state mm:ffff8800a7de2d80 idx:1 val:1

I can prove that the __mapcount is misaccount. when task is exited. the rmap still exist.

Check if the kernel in question contains this commit: ad33bb04b2a6 ("mm:
thp: fix SMP race condition between THP page fault and MADV_DONTNEED")

HI, Vlastimil

I miss the patch.

Quoted 3 lines

Try applying it then, there's good chance the error and crash will go away. Even if your workload doesn't actually run any madvise(MADV_DONTNEED).

Hi Vlastimil,

I find this error was reported by Kirill as following, right?
https://patchwork.kernel.org/patch/7550401/

The call trace is quite like the same as ours.

Thanks,
Xishi Qiu

Quoted 46 lines

Quoted 12 lines

when I read the patch. I find the following issue. but I am sure it is right.

if (unlikely(pmd_trans_unstable(pmd)))
return 0;
/*
* A regular pmd is established and it can't morph into a huge pmd
* from under us anymore at this point because we hold the mmap_sem
* read mode and khugepaged takes it in write mode. So now it's
* safe to run pte_offset_map().
*/
pte = pte_offset_map(pmd, address);

Quoted 3 lines

after pmd_trans_unstable call, without any protect method. by the comments,
it think the pte_offset_map is safe. before pte_offset_map call, it still may be
unstable. it is possible?

IIRC it's "unstable" wrt possible none->huge->none transition. But once we've seen it's a regular pmd via pmd_trans_unstable(), we're safe as a transition from regular pmd can't happen.

Quoted 16 lines

Thanks
zhongjiang

Quoted 12 lines

Quoted 9 lines

Thanks
zhongjiang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

.

Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

.

Quoted 40 lines · Xishi Qiu

Quoted 38 lines · Vlastimil Babka

Quoted 27 lines · zhong jiang

Quoted 26 lines · Vlastimil Babka

Quoted 25 lines · zhong jiang

Quoted 9 lines · Hugh Dickins

Quoted 8 lines · Xishi Qiu

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Quoted 9 lines

Quoted 1 line

Any ideas?

Sorry, no. I assume that _mapcount has been misaccounted, for example a pte mapped in on top of another pte; but cannot begin tell you where in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.

Hugh

.

Hi, Hugh

I find the following message from the dmesg.

[26068.316592] BUG: Bad rss-counter state mm:ffff8800a7de2d80 idx:1 val:1

Quoted 7 lines

Quoted 4 lines

Quoted 2 lines

I can prove that the __mapcount is misaccount. when task is exited. the rmap still exist.

Check if the kernel in question contains this commit: ad33bb04b2a6 ("mm:
thp: fix SMP race condition between THP page fault and MADV_DONTNEED")

HI, Vlastimil

I miss the patch.

Try applying it then, there's good chance the error and crash will go away. Even if your workload doesn't actually run any madvise(MADV_DONTNEED).

Quoted 4 lines

Hi Vlastimil,

I find this error was reported by Kirill as following, right?
https://patchwork.kernel.org/patch/7550401/

That was reported by Minchan.

Quoted 1 line

The call trace is quite like the same as ours.

In that thread, the error seems just disappeared in the end.

So, did you apply the patch I suggested? Did it help?

Quoted 47 lines

Thanks,
Xishi Qiu

Quoted 42 lines

Quoted 12 lines

when I read the patch. I find the following issue. but I am sure it is right.

if (unlikely(pmd_trans_unstable(pmd)))
return 0;
/*
* A regular pmd is established and it can't morph into a huge pmd
* from under us anymore at this point because we hold the mmap_sem
* read mode and khugepaged takes it in write mode. So now it's
* safe to run pte_offset_map().
*/
pte = pte_offset_map(pmd, address);

Quoted 3 lines

after pmd_trans_unstable call, without any protect method. by the comments,
it think the pte_offset_map is safe. before pte_offset_map call, it still may be
unstable. it is possible?

IIRC it's "unstable" wrt possible none->huge->none transition. But once we've seen it's a regular pmd via pmd_trans_unstable(), we're safe as a transition from regular pmd can't happen.

Quoted 11 lines

Thanks
zhongjiang

Quoted 9 lines

Quoted 9 lines

Thanks
zhongjiang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 9 lines

Quoted 2 lines

.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

.

.

Quoted 38 lines · Xishi Qiu

Quoted 36 lines · Vlastimil Babka

Quoted 34 lines · zhong jiang

Quoted 22 lines · Vlastimil Babka

Quoted 21 lines · zhong jiang

Quoted 9 lines · Hugh Dickins

Quoted 8 lines · Xishi Qiu

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Quoted 9 lines

Quoted 1 line

Any ideas?

Sorry, no. I assume that _mapcount has been misaccounted, for example a pte mapped in on top of another pte; but cannot begin tell you where in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.

Hugh

.

Hi, Hugh

Quoted 8 lines

Quoted 6 lines

I find the following message from the dmesg.

[26068.316592] BUG: Bad rss-counter state mm:ffff8800a7de2d80 idx:1 val:1

I can prove that the __mapcount is misaccount. when task is exited. the rmap still exist.

Check if the kernel in question contains this commit: ad33bb04b2a6 ("mm:
thp: fix SMP race condition between THP page fault and MADV_DONTNEED")

HI, Vlastimil

I miss the patch.

Quoted 8 lines

Quoted 3 lines

Try applying it then, there's good chance the error and crash will go away. Even if your workload doesn't actually run any madvise(MADV_DONTNEED).

Hi Vlastimil,

I find this error was reported by Kirill as following, right?
https://patchwork.kernel.org/patch/7550401/

That was reported by Minchan.

Quoted 1 line

The call trace is quite like the same as ours.

In that thread, the error seems just disappeared in the end.

So, did you apply the patch I suggested? Did it help?

Hi,

Unfortunately, this patch(mm: thp: fix SMP race condition between THP page fault and MADV_DONTNEED) didn't help, I got the panic again.

And I find this error before panic, "[468229.996610] BUG: Bad rss-counter state mm:ffff8806aebc2580 idx:1 val:1"

[468451.702807] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[468451.702861] IP: [<ffffffff810ac089>] down_read_trylock+0x9/0x30
[468451.702900] PGD 12445e067 PUD 11acaa067 PMD 0 
[468451.702931] Oops: 0000 [#1] SMP 
[468451.702953] kbox catch die event.
[468451.703003] collected_len = 1047419, LOG_BUF_LEN_LOCAL = 1048576
[468451.703003] kbox: notify die begin
[468451.703003] kbox: no notify die func register. no need to notify
[468451.703003] do nothing after die!
[468451.703003] Modules linked in: ipt_REJECT macvlan ip_set_hash_ipport vport_vxlan(OVE) xt_statistic xt_physdev xt_nat xt_recent xt_mark xt_comment veth ct_limit(OVE) bum_extract(OVE) policy(OVE) bum(OVE) ip_set nfnetlink openvswitch(OVE) nf_defrag_ipv6 gre ext3 jbd ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack bridge stp llc kboxdriver(O) kbox(O) dm_thin_pool dm_persistent_data crc32_pclmul dm_bio_prison dm_bufio ghash_clmulni_intel libcrc32c aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg parport_pc cirrus virtio_console parport syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core pcspkr ip_tables ext4 jbd2 mbcache sr_mod cdrom ata_generic pata_acpi
[468451.703003]  virtio_net virtio_blk crct10dif_pclmul crct10dif_common ata_piix virtio_pci libata serio_raw virtio_ring crc32c_intel virtio dm_mirror dm_region_hash dm_log dm_mod
[468451.703003] CPU: 6 PID: 21965 Comm: docker-containe Tainted: G           OE  ----V-------   3.10.0-327.53.58.73.x86_64 #1
[468451.703003] Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.8.1-0-g4adadbd-20170107_142945-9_64_246_229 04/01/2014
[468451.703003] task: ffff880692402e00 ti: ffff88018209c000 task.ti: ffff88018209c000
[468451.703003] RIP: 0010:[<ffffffff810ac089>]  [<ffffffff810ac089>] down_read_trylock+0x9/0x30
[468451.703003] RSP: 0018:ffff88018209f8f8  EFLAGS: 00010202
[468451.703003] RAX: 0000000000000000 RBX: ffff880720cd7740 RCX: ffff880720cd7740
[468451.703003] RDX: 0000000000000001 RSI: 0000000000000301 RDI: 0000000000000008
[468451.703003] RBP: ffff88018209f8f8 R08: 00000000c0e0f310 R09: ffff880720cd7740
[468451.703003] R10: ffff88083efd8000 R11: 0000000000000000 R12: ffff880720cd7741
[468451.703003] R13: ffffea000824d100 R14: 0000000000000008 R15: 0000000000000000
[468451.703003] FS:  00007fc0e2a85700(0000) GS:ffff88083ed80000(0000) knlGS:0000000000000000
[468451.703003] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[468451.703003] CR2: 0000000000000008 CR3: 0000000661906000 CR4: 00000000001407e0
[468451.703003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[468451.703003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[468451.703003] Stack:
[468451.703003]  ffff88018209f928 ffffffff811a7eb5 ffffea000824d100 ffff88018209fa90
[468451.703003]  ffffea00082f9680 0000000000000301 ffff88018209f978 ffffffff811a82e1
[468451.703003]  ffffea000824d100 ffff88018209fa00 0000000000000001 ffffea000824d100
[468451.703003] Call Trace:
[468451.703003]  [<ffffffff811a7eb5>] page_lock_anon_vma_read+0x55/0x110
[468451.703003]  [<ffffffff811a82e1>] try_to_unmap_anon+0x21/0x120
[468451.703003]  [<ffffffff811a842d>] try_to_unmap+0x4d/0x60
[468451.712006]  [<ffffffff811cc749>] migrate_pages+0x439/0x790
[468451.712006]  [<ffffffff81193280>] ? __reset_isolation_suitable+0xe0/0xe0
[468451.712006]  [<ffffffff811941f9>] compact_zone+0x299/0x400
[468451.712006]  [<ffffffff81059aff>] ? kvm_clock_get_cycles+0x1f/0x30
[468451.712006]  [<ffffffff811943fc>] compact_zone_order+0x9c/0xf0
[468451.712006]  [<ffffffff811947b1>] try_to_compact_pages+0x121/0x1a0
[468451.712006]  [<ffffffff8163ace6>] __alloc_pages_direct_compact+0xac/0x196
[468451.712006]  [<ffffffff811783e2>] __alloc_pages_nodemask+0xbc2/0xca0
[468451.712006]  [<ffffffff811bcb7a>] alloc_pages_vma+0x9a/0x150
[468451.712006]  [<ffffffff811d1573>] do_huge_pmd_anonymous_page+0x123/0x510
[468451.712006]  [<ffffffff8119bc58>] handle_mm_fault+0x1a8/0xf50

[468451.712006]  [<ffffffff8164b4d6>] __do_page_fault+0x166/0x470
[468451.712006]  [<ffffffff8164b8a3>] trace_do_page_fault+0x43/0x110
[468451.712006]  [<ffffffff8164af79>] do_async_page_fault+0x29/0xe0
[468451.712006]  [<ffffffff81647a38>] async_page_fault+0x28/0x30
[468451.712006] Code: 00 00 00 ba 01 00 00 00 48 89 de e8 12 fe ff ff eb ce 48 c7 c0 f2 ff ff ff eb c5 e8 42 ff fc ff 66 90 0f 1f 44 00 00 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 
[468451.712006] RIP  [<ffffffff810ac089>] down_read_trylock+0x9/0x30
[468451.738667]  RSP <ffff88018209f8f8>
[468451.738667] CR2: 0000000000000008

Quoted 4 lines · Xishi Qiu

Hi,

Unfortunately, this patch(mm: thp: fix SMP race condition between THP page fault and MADV_DONTNEED) didn't help, I got the panic again.

Too bad then. I don't know of any other patch from my own experience being directly related, try to look for similar THP-related race fixes. Did you already check whether disabling THP (set it to "never" under /sys/...) prevents the issue? I forgot.

Quoted 1 line

And I find this error before panic, "[468229.996610] BUG: Bad rss-counter state mm:ffff8806aebc2580 idx:1 val:1"

This likely means that a pte was overwritten to zero, and an anon page had no other reference than this pte, so it became orphaned. Its anon_vma object was freed as the process exited, and eventually overwritten by a new user, so compaction or reclaim looking at it sooner or later makes a bad memory access.

The pte overwriting may be a result of races with multiple threads trying to either read or write within the same page, involving THP zero page. It doesn't have to be MADV_DONTNEED related.

Quoted 45 lines

[468451.702807] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[468451.702861] IP: [<ffffffff810ac089>] down_read_trylock+0x9/0x30
[468451.702900] PGD 12445e067 PUD 11acaa067 PMD 0 
[468451.702931] Oops: 0000 [#1] SMP 
[468451.702953] kbox catch die event.
[468451.703003] collected_len = 1047419, LOG_BUF_LEN_LOCAL = 1048576
[468451.703003] kbox: notify die begin
[468451.703003] kbox: no notify die func register. no need to notify
[468451.703003] do nothing after die!
[468451.703003] Modules linked in: ipt_REJECT macvlan ip_set_hash_ipport vport_vxlan(OVE) xt_statistic xt_physdev xt_nat xt_recent xt_mark xt_comment veth ct_limit(OVE) bum_extract(OVE) policy(OVE) bum(OVE) ip_set nfnetlink openvswitch(OVE) nf_defrag_ipv6 gre ext3 jbd ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack bridge stp llc kboxdriver(O) kbox(O) dm_thin_pool dm_persistent_data crc32_pclmul dm_bio_prison dm_bufio ghash_clmulni_intel libcrc32c aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg parport_pc cirrus virtio_console parport syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core pcspkr ip_tables ext4 jbd2 mbcache sr_mod cdrom ata_generic pata_acpi
[468451.703003]  virtio_net virtio_blk crct10dif_pclmul crct10dif_common ata_piix virtio_pci libata serio_raw virtio_ring crc32c_intel virtio dm_mirror dm_region_hash dm_log dm_mod
[468451.703003] CPU: 6 PID: 21965 Comm: docker-containe Tainted: G           OE  ----V-------   3.10.0-327.53.58.73.x86_64 #1
[468451.703003] Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.8.1-0-g4adadbd-20170107_142945-9_64_246_229 04/01/2014
[468451.703003] task: ffff880692402e00 ti: ffff88018209c000 task.ti: ffff88018209c000
[468451.703003] RIP: 0010:[<ffffffff810ac089>]  [<ffffffff810ac089>] down_read_trylock+0x9/0x30
[468451.703003] RSP: 0018:ffff88018209f8f8  EFLAGS: 00010202
[468451.703003] RAX: 0000000000000000 RBX: ffff880720cd7740 RCX: ffff880720cd7740
[468451.703003] RDX: 0000000000000001 RSI: 0000000000000301 RDI: 0000000000000008
[468451.703003] RBP: ffff88018209f8f8 R08: 00000000c0e0f310 R09: ffff880720cd7740
[468451.703003] R10: ffff88083efd8000 R11: 0000000000000000 R12: ffff880720cd7741
[468451.703003] R13: ffffea000824d100 R14: 0000000000000008 R15: 0000000000000000
[468451.703003] FS:  00007fc0e2a85700(0000) GS:ffff88083ed80000(0000) knlGS:0000000000000000
[468451.703003] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[468451.703003] CR2: 0000000000000008 CR3: 0000000661906000 CR4: 00000000001407e0
[468451.703003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[468451.703003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[468451.703003] Stack:
[468451.703003]  ffff88018209f928 ffffffff811a7eb5 ffffea000824d100 ffff88018209fa90
[468451.703003]  ffffea00082f9680 0000000000000301 ffff88018209f978 ffffffff811a82e1
[468451.703003]  ffffea000824d100 ffff88018209fa00 0000000000000001 ffffea000824d100
[468451.703003] Call Trace:
[468451.703003]  [<ffffffff811a7eb5>] page_lock_anon_vma_read+0x55/0x110
[468451.703003]  [<ffffffff811a82e1>] try_to_unmap_anon+0x21/0x120
[468451.703003]  [<ffffffff811a842d>] try_to_unmap+0x4d/0x60
[468451.712006]  [<ffffffff811cc749>] migrate_pages+0x439/0x790
[468451.712006]  [<ffffffff81193280>] ? __reset_isolation_suitable+0xe0/0xe0
[468451.712006]  [<ffffffff811941f9>] compact_zone+0x299/0x400
[468451.712006]  [<ffffffff81059aff>] ? kvm_clock_get_cycles+0x1f/0x30
[468451.712006]  [<ffffffff811943fc>] compact_zone_order+0x9c/0xf0
[468451.712006]  [<ffffffff811947b1>] try_to_compact_pages+0x121/0x1a0
[468451.712006]  [<ffffffff8163ace6>] __alloc_pages_direct_compact+0xac/0x196
[468451.712006]  [<ffffffff811783e2>] __alloc_pages_nodemask+0xbc2/0xca0
[468451.712006]  [<ffffffff811bcb7a>] alloc_pages_vma+0x9a/0x150
[468451.712006]  [<ffffffff811d1573>] do_huge_pmd_anonymous_page+0x123/0x510
[468451.712006]  [<ffffffff8119bc58>] handle_mm_fault+0x1a8/0xf50

Quoted 11 lines

[468451.712006]  [<ffffffff8164b4d6>] __do_page_fault+0x166/0x470
[468451.712006]  [<ffffffff8164b8a3>] trace_do_page_fault+0x43/0x110
[468451.712006]  [<ffffffff8164af79>] do_async_page_fault+0x29/0xe0
[468451.712006]  [<ffffffff81647a38>] async_page_fault+0x28/0x30
[468451.712006] Code: 00 00 00 ba 01 00 00 00 48 89 de e8 12 fe ff ff eb ce 48 c7 c0 f2 ff ff ff eb c5 e8 42 ff fc ff 66 90 0f 1f 44 00 00 55 48 89 e5 <48> 8b 07 48 89 c2 48 83 c2 01 7e 07 f0 48 0f b1 17 75 f0 48 f7 
[468451.712006] RIP  [<ffffffff810ac089>] down_read_trylock+0x9/0x30
[468451.738667]  RSP <ffff88018209f8f8>
[468451.738667] CR2: 0000000000000008

.

I find two patches from upstream.
887843961c4b4681ee993c36d4997bf4b4aa8253

a9c8e4beeeb64c22b84c803747487857fe424b68

I can't find any relations to the panic from the first one, and the second

one seems triggered from xen, but we use kvm.

but if zap_pte in a fremap fails to drop the anon page that was under memory migration/compaction the exact same thing will happen. Either

Quoted 5 lines · Xishi Qiu

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

Quoted 5 lines

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Any ideas?

Sorry, no. I assume that _mapcount has been misaccounted, for example a pte mapped in on top of another pte; but cannot begin tell you where

in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.

Hugh

.

Quoted 10 lines · Hugh Dickins

Quoted 9 lines · Xishi Qiu

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Quoted 4 lines

Quoted 1 line

Any ideas?

Sorry, no. I assume that _mapcount has been misaccounted, for example a pte mapped in on top of another pte; but cannot begin tell you where

Hi Hugh,

What does "a pte mapped in on top of another pte" mean? Could you give more info?

Thanks,
Xishi Qiu

Quoted 3 lines

in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.

Hugh

Quoted 11 lines · Xishi Qiu

Quoted 10 lines · Hugh Dickins

Quoted 9 lines · Xishi Qiu

Here is a bug report form redhat: https://bugzilla.redhat.com/show_bug.cgi?id=1305620
And I meet the bug too. However it is hard to reproduce, and 624483f3ea82598("mm: rmap: fix use-after-free in __put_anon_vma") is not help.

From the vmcore, it seems that the page is still mapped(_mapcount=0 and _count=2), and the value of mapping is a valid address(mapping = 0xffff8801b3e2a101), but anon_vma has been corrupted.

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

Any ideas?

Sorry, no. I assume that _mapcount has been misaccounted, for example a pte mapped in on top of another pte; but cannot begin tell you where

Hi Hugh,

What does "a pte mapped in on top of another pte" mean? Could you give more info?

I mean, there are various places in mm/memory.c which decide what they intend to do based on orig_pte, then take pte lock, then check that pte_same(pte, orig_pte) before taking it any further. If a pte_same() check were missing (I do not know of any such case), then two racing tasks might install the same pte, one on top of the other - page mapcount being incremented twice, but decremented only once when that pte is finally unmapped later.

Please see similar discussion in the earlier thread at marc.info/?l=linux-mm&m=148222656211837&w=2

Hugh

Quoted 7 lines

Thanks,
Xishi Qiu

Quoted 3 lines

in Red Hat's kernel-3.10.0-229.4.2.el7 that might happen.

Hugh

.

Quoted 8 lines · Hugh Dickins

I mean, there are various places in mm/memory.c which decide what they intend to do based on orig_pte, then take pte lock, then check that pte_same(pte, orig_pte) before taking it any further. If a pte_same() check were missing (I do not know of any such case), then two racing tasks might install the same pte, one on top of the other - page mapcount being incremented twice, but decremented only once when that pte is finally unmapped later.

Hi Hugh,

Do you mean that the ptes from two racing point to the same page? or the two racing point to two pages, but one covers the other later? and the first page maybe alone in the lru list, and it will never be freed when the process exit.

We got this info before crash.

[26068.316592] BUG: Bad rss-counter state mm:ffff8800a7de2d80 idx:1 val:1