memory leak in bpf
Subsystems (4)
syzbot
syzbotHello,
syzbot found the following issue on:
HEAD commit: f01c30de Merge tag 'vfs-5.10-fixes-2' of git://git.kernel... git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=15b9b181500000 kernel config: https://syzkaller.appspot.com/x/.config?x=a3f13716fa0212fd dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=14b78b81500000
IMPORTANT: if you fix the issue, please add the following tag to the commit:
2020/11/14 15:01:05 executed programs: 33
2020/11/14 15:01:11 executed programs: 35
2020/11/14 15:01:17 executed programs: 37
2020/11/14 15:01:22 executed programs: 39
BUG: memory leak
unreferenced object 0xffff8881161c6440 (size 64):
comm "syz-executor.0", pid 8961, jiffies 4295107077 (age 12.370s)
hex dump (first 32 bytes):
80 62 58 04 00 ea ff ff c0 17 37 04 00 ea ff ff .bX.......7.....
80 6f 47 04 00 ea ff ff 40 64 58 04 00 ea ff ff .oG.....@dX.....
backtrace:
[<00000000189a27fd>] kmalloc_node include/linux/slab.h:575 [inline]
[<00000000189a27fd>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]
[<00000000189a27fd>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline]
[<00000000189a27fd>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline]
[<00000000189a27fd>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150
[<000000009e5cec3e>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline]
[<000000009e5cec3e>] map_create kernel/bpf/syscall.c:825 [inline]
[<000000009e5cec3e>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381
[<00000000c513b5d1>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
[<0000000033006ec5>] entry_SYSCALL_64_after_hwframe+0x44/0xa9---
This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].
syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot. syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches
syzbot
syzbotsyzbot has found a reproducer for the following issue on:
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing program
BUG: memory leak
unreferenced object 0xffff88810efccc80 (size 64):
comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s)
hex dump (first 32 bytes):
c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................
c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected].....
backtrace:
[<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline]
[<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline]
[<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline]
[<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150
[<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline]
[<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline]
[<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381
[<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
[<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9Quoted 28 lines · syzbot
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing programBUG: memory leak unreferenced object 0xffff88810efccc80 (size 64): comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) hex dump (first 32 bytes): c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected]..... backtrace: [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]
Quoted 10 lines
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline]
[<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline]
[<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150
[<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline]
[<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline]
[<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381
[<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
[<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9i am pretty sure that this one is a false positive
the problem with reproducer is that it does not terminate all of the child processes that it spawns
i confirmed that it is a false positive by tracing __fput() and bpf_map_release(), i ran reproducer, got kmemleak report, then i manually killed those running leftover processes from reproducer and then both functions were executed and memory was freed
i am marking this one as:
#syz invalid
Quoted 45 lines · Rustam Kovhaev
Quoted 28 lines · syzbot
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing programBUG: memory leak unreferenced object 0xffff88810efccc80 (size 64): comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) hex dump (first 32 bytes): c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected]..... backtrace: [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]Quoted 10 lines
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline] [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline] [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150 [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline] [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline] [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381 [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9i am pretty sure that this one is a false positive
the problem with reproducer is that it does not terminate all of the child processes that it spawns
Quoted 7 lines
i confirmed that it is a false positive by tracing
__fput()andbpf_map_release(), i ran reproducer, got kmemleak report, then i manually killed those running leftover processes from reproducer and then both functions were executed and memory was freedi am marking this one as:
#syz invalid
Hi Rustam,
Thanks for looking into this.
I wonder how/where are these objects referenced? If they are not leaked and referenced somewhere, KMEMLEAK should not report them as leaks.
So even if this is a false positive for BPF, this is a true positive bug and something to fix for KMEMLEAK ;)
And syzbot will probably re-create this bug report soon as this still happens and is not a one-off thing.
Quoted 58 lines · Dmitry Vyukov
Quoted 45 lines · Rustam Kovhaev
Quoted 28 lines · syzbot
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing programBUG: memory leak unreferenced object 0xffff88810efccc80 (size 64): comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) hex dump (first 32 bytes): c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected]..... backtrace: [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]Quoted 10 lines
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline] [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline] [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150 [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline] [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline] [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381 [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9i am pretty sure that this one is a false positive
the problem with reproducer is that it does not terminate all of the child processes that it spawnsQuoted 7 lines
i confirmed that it is a false positive by tracing
__fput()andbpf_map_release(), i ran reproducer, got kmemleak report, then i manually killed those running leftover processes from reproducer and then both functions were executed and memory was freedi am marking this one as:
#syz invalidHi Rustam,
Thanks for looking into this.
Quoted 7 lines
I wonder how/where are these objects referenced? If they are not leaked and referenced somewhere, KMEMLEAK should not report them as leaks.
So even if this is a false positive for BPF, this is a true positive bug and something to fix for KMEMLEAK ;)
And syzbot will probably re-create this bug report soon as this still happens and is not a one-off thing.
hi Dmitry, i haven't thought of it this way, but i guess you are right, it is a kmemleak bug, ideally kmemleak should be aware that there are still running processes holding references to bpf fd/anonymous inodes which in their turn hold references to allocated bpf maps
Quoted 68 lines · Rustam Kovhaev
Quoted 58 lines · Dmitry Vyukov
Quoted 45 lines · Rustam Kovhaev
Quoted 28 lines · syzbot
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing programBUG: memory leak unreferenced object 0xffff88810efccc80 (size 64): comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) hex dump (first 32 bytes): c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected]..... backtrace: [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]Quoted 10 lines
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline] [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline] [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150 [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline] [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline] [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381 [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9i am pretty sure that this one is a false positive
the problem with reproducer is that it does not terminate all of the child processes that it spawnsQuoted 7 lines
i confirmed that it is a false positive by tracing
__fput()andbpf_map_release(), i ran reproducer, got kmemleak report, then i manually killed those running leftover processes from reproducer and then both functions were executed and memory was freedi am marking this one as:
#syz invalidHi Rustam,
Thanks for looking into this.
Quoted 7 lines
I wonder how/where are these objects referenced? If they are not leaked and referenced somewhere, KMEMLEAK should not report them as leaks.
So even if this is a false positive for BPF, this is a true positive bug and something to fix for KMEMLEAK ;)
And syzbot will probably re-create this bug report soon as this still happens and is not a one-off thing.
Quoted 4 lines
hi Dmitry, i haven't thought of it this way, but i guess you are right, it is a kmemleak bug, ideally kmemleak should be aware that there are still running processes holding references to bpf fd/anonymous inodes which in their turn hold references to allocated bpf maps
KMEMLEAK scans whole memory, so if there are pointers to the object anywhere in memory, KMEMLEAK should not report them as leaked. Running processes have no direct effect on KMEMLEAK logic.
So the question is: where are these pointers to these objects? If we answer this, we can check how/why KMEMLEAK misses them. Are they mangled in some way?
Quoted 74 lines · Dmitry Vyukov
Quoted 68 lines · Rustam Kovhaev
Quoted 58 lines · Dmitry Vyukov
Quoted 45 lines · Rustam Kovhaev
Quoted 28 lines · syzbot
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing programBUG: memory leak unreferenced object 0xffff88810efccc80 (size 64): comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) hex dump (first 32 bytes): c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected]..... backtrace: [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]Quoted 10 lines
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline] [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline] [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150 [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline] [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline] [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381 [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9i am pretty sure that this one is a false positive
the problem with reproducer is that it does not terminate all of the child processes that it spawnsQuoted 7 lines
i confirmed that it is a false positive by tracing
__fput()andbpf_map_release(), i ran reproducer, got kmemleak report, then i manually killed those running leftover processes from reproducer and then both functions were executed and memory was freedi am marking this one as:
#syz invalidHi Rustam,
Thanks for looking into this.
Quoted 7 lines
I wonder how/where are these objects referenced? If they are not leaked and referenced somewhere, KMEMLEAK should not report them as leaks.
So even if this is a false positive for BPF, this is a true positive bug and something to fix for KMEMLEAK ;)
And syzbot will probably re-create this bug report soon as this still happens and is not a one-off thing.Quoted 4 lines
hi Dmitry, i haven't thought of it this way, but i guess you are right, it is a kmemleak bug, ideally kmemleak should be aware that there are still running processes holding references to bpf fd/anonymous inodes which in their turn hold references to allocated bpf maps
Quoted 6 lines
KMEMLEAK scans whole memory, so if there are pointers to the object anywhere in memory, KMEMLEAK should not report them as leaked. Running processes have no direct effect on KMEMLEAK logic.
So the question is: where are these pointers to these objects? If we answer this, we can check how/why KMEMLEAK misses them. Are they mangled in some way?
thank you for your comments, they make sense, and indeed, the pointer gets vmaped.
i should have looked into this sooner, becaused syzbot did trigger the issue again, and Andrii had to look into the same bug, sorry about that. if i am understanding this correctly here is what the fix should be:
--- kernel/bpf/ringbuf.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/bpf/ringbuf.c +2 -0
@@ -8,6 +8,7 @@#include <linux/vmalloc.h> #include <linux/wait.h> #include <linux/poll.h> +#include <linux/kmemleak.h> #include <uapi/linux/btf.h> #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)
@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb =vmap (pages, nr_meta_pages +2 * nr_data_pages,VM_ALLOC |VM_USERMAP ,PAGE_KERNEL );if (rb) { +kmemleak_not_leak ((void *) pages); rb->pages = pages; rb->nr_pages = nr_pages;return rb;
Signature · 1 line
2.30.2
Quoted 82 lines · Rustam Kovhaev
Quoted 74 lines · Dmitry Vyukov
Quoted 68 lines · Rustam Kovhaev
Quoted 58 lines · Dmitry Vyukov
Quoted 45 lines · Rustam Kovhaev
Quoted 28 lines · syzbot
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing programBUG: memory leak unreferenced object 0xffff88810efccc80 (size 64): comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) hex dump (first 32 bytes): c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected]..... backtrace: [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]Quoted 10 lines
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline] [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline] [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150 [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline] [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline] [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381 [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9i am pretty sure that this one is a false positive
the problem with reproducer is that it does not terminate all of the child processes that it spawnsQuoted 7 lines
i confirmed that it is a false positive by tracing
__fput()andbpf_map_release(), i ran reproducer, got kmemleak report, then i manually killed those running leftover processes from reproducer and then both functions were executed and memory was freedi am marking this one as:
#syz invalidHi Rustam,
Thanks for looking into this.
Quoted 7 lines
I wonder how/where are these objects referenced? If they are not leaked and referenced somewhere, KMEMLEAK should not report them as leaks.
So even if this is a false positive for BPF, this is a true positive bug and something to fix for KMEMLEAK ;)
And syzbot will probably re-create this bug report soon as this still happens and is not a one-off thing.Quoted 4 lines
hi Dmitry, i haven't thought of it this way, but i guess you are right, it is a kmemleak bug, ideally kmemleak should be aware that there are still running processes holding references to bpf fd/anonymous inodes which in their turn hold references to allocated bpf maps
Quoted 6 lines
KMEMLEAK scans whole memory, so if there are pointers to the object anywhere in memory, KMEMLEAK should not report them as leaked. Running processes have no direct effect on KMEMLEAK logic.
So the question is: where are these pointers to these objects? If we answer this, we can check how/why KMEMLEAK misses them. Are they mangled in some way?
Quoted 4 lines
thank you for your comments, they make sense, and indeed, the pointer gets vmaped.
i should have looked into this sooner, becaused syzbot did trigger the issue again, and Andrii had to look into the same bug, sorry about that.
No worries! I actually forgot about this thread :) Let's leave the link to my today's investigation ([0]) just for completeness.
[0] https://lore.kernel.org/bpf/CAEf4BzYk+dqs+jwu6VKXP-RttcTEGFe+ySTGWT9CRNkagDiJVA@mail.gmail.com/
Quoted 18 lines
if i am understanding this correctly here is what the fix should be:
--- kernel/bpf/ringbuf.c | 2 ++ 1 file changed, 2 insertions(+)kernel/bpf/ringbuf.c +1
-0@@ -8,6 +8,7 @@#include <linux/vmalloc.h> #include <linux/wait.h> #include <linux/poll.h> +#include <linux/kmemleak.h> #include <uapi/linux/btf.h> #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)
kernel/bpf/ringbuf.c static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) · line 109
Quoted 4 lines
kernel/bpf/ringbuf.c +1
-0@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb =vmap (pages, nr_meta_pages +2 * nr_data_pages,VM_ALLOC |VM_USERMAP ,PAGE_KERNEL );if (rb) { +kmemleak_not_leak ((void *) pages);
If that makes kmemleak happy, I have no problems with this. But maybe leave some comment explaining why this is needed at all?
And for my understanding, how vmap changes anything? Those pages are still referenced from rb, which is referenced from some struct file in the system. Sorry if that's a naive question.
Quoted 6 lines
kernel/bpf/ringbuf.c +0
-1rb->pages = pages; rb->nr_pages = nr_pages;return rb; --2.30.2
Quoted 119 lines · Andrii Nakryiko
Quoted 82 lines · Rustam Kovhaev
Quoted 74 lines · Dmitry Vyukov
Quoted 68 lines · Rustam Kovhaev
Quoted 58 lines · Dmitry Vyukov
Quoted 45 lines · Rustam Kovhaev
Quoted 28 lines · syzbot
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing programBUG: memory leak unreferenced object 0xffff88810efccc80 (size 64): comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) hex dump (first 32 bytes): c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected]..... backtrace: [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]Quoted 10 lines
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline] [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline] [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150 [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline] [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline] [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381 [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9i am pretty sure that this one is a false positive
the problem with reproducer is that it does not terminate all of the child processes that it spawnsQuoted 7 lines
i confirmed that it is a false positive by tracing
__fput()andbpf_map_release(), i ran reproducer, got kmemleak report, then i manually killed those running leftover processes from reproducer and then both functions were executed and memory was freedi am marking this one as:
#syz invalidHi Rustam,
Thanks for looking into this.
Quoted 7 lines
I wonder how/where are these objects referenced? If they are not leaked and referenced somewhere, KMEMLEAK should not report them as leaks.
So even if this is a false positive for BPF, this is a true positive bug and something to fix for KMEMLEAK ;)
And syzbot will probably re-create this bug report soon as this still happens and is not a one-off thing.Quoted 4 lines
hi Dmitry, i haven't thought of it this way, but i guess you are right, it is a kmemleak bug, ideally kmemleak should be aware that there are still running processes holding references to bpf fd/anonymous inodes which in their turn hold references to allocated bpf maps
Quoted 6 lines
KMEMLEAK scans whole memory, so if there are pointers to the object anywhere in memory, KMEMLEAK should not report them as leaked. Running processes have no direct effect on KMEMLEAK logic.
So the question is: where are these pointers to these objects? If we answer this, we can check how/why KMEMLEAK misses them. Are they mangled in some way?Quoted 4 lines
thank you for your comments, they make sense, and indeed, the pointer gets vmaped.
i should have looked into this sooner, becaused syzbot did trigger the issue again, and Andrii had to look into the same bug, sorry about that.No worries! I actually forgot about this thread :) Let's leave the link to my today's investigation ([0]) just for completeness.
[0] https://lore.kernel.org/bpf/CAEf4BzYk+dqs+jwu6VKXP-RttcTEGFe+ySTGWT9CRNkagDiJVA@mail.gmail.com/
Quoted 18 lines
if i am understanding this correctly here is what the fix should be:
--- kernel/bpf/ringbuf.c | 2 ++ 1 file changed, 2 insertions(+)kernel/bpf/ringbuf.c +1
-0@@ -8,6 +8,7 @@#include <linux/vmalloc.h> #include <linux/wait.h> #include <linux/poll.h> +#include <linux/kmemleak.h> #include <uapi/linux/btf.h> #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)kernel/bpf/ringbuf.c static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) · line 109
Quoted 4 lines
kernel/bpf/ringbuf.c +1
-0@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb =vmap (pages, nr_meta_pages +2 * nr_data_pages,VM_ALLOC |VM_USERMAP ,PAGE_KERNEL );if (rb) { +kmemleak_not_leak ((void *) pages);If that makes kmemleak happy, I have no problems with this. But maybe leave some comment explaining why this is needed at all?
Quoted 4 lines
And for my understanding, how vmap changes anything? Those pages are still referenced from rb, which is referenced from some struct file in the system. Sorry if that's a naive question.
valid question, it does look like kmemleak should be scanning vmalloc()/vmap() memory, i will research this further
Quoted 120 lines · Rustam Kovhaev
Quoted 119 lines · Andrii Nakryiko
Quoted 82 lines · Rustam Kovhaev
Quoted 74 lines · Dmitry Vyukov
Quoted 68 lines · Rustam Kovhaev
Quoted 58 lines · Dmitry Vyukov
Quoted 45 lines · Rustam Kovhaev
Quoted 28 lines · syzbot
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing programBUG: memory leak unreferenced object 0xffff88810efccc80 (size 64): comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) hex dump (first 32 bytes): c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected]..... backtrace: [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]Quoted 10 lines
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline] [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline] [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150 [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline] [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline] [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381 [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9i am pretty sure that this one is a false positive
the problem with reproducer is that it does not terminate all of the child processes that it spawnsQuoted 7 lines
i confirmed that it is a false positive by tracing
__fput()andbpf_map_release(), i ran reproducer, got kmemleak report, then i manually killed those running leftover processes from reproducer and then both functions were executed and memory was freedi am marking this one as:
#syz invalidHi Rustam,
Thanks for looking into this.
Quoted 7 lines
I wonder how/where are these objects referenced? If they are not leaked and referenced somewhere, KMEMLEAK should not report them as leaks.
So even if this is a false positive for BPF, this is a true positive bug and something to fix for KMEMLEAK ;)
And syzbot will probably re-create this bug report soon as this still happens and is not a one-off thing.Quoted 4 lines
hi Dmitry, i haven't thought of it this way, but i guess you are right, it is a kmemleak bug, ideally kmemleak should be aware that there are still running processes holding references to bpf fd/anonymous inodes which in their turn hold references to allocated bpf maps
Quoted 6 lines
KMEMLEAK scans whole memory, so if there are pointers to the object anywhere in memory, KMEMLEAK should not report them as leaked. Running processes have no direct effect on KMEMLEAK logic.
So the question is: where are these pointers to these objects? If we answer this, we can check how/why KMEMLEAK misses them. Are they mangled in some way?Quoted 4 lines
thank you for your comments, they make sense, and indeed, the pointer gets vmaped.
i should have looked into this sooner, becaused syzbot did trigger the issue again, and Andrii had to look into the same bug, sorry about that.No worries! I actually forgot about this thread :) Let's leave the link to my today's investigation ([0]) just for completeness.
[0] https://lore.kernel.org/bpf/CAEf4BzYk+dqs+jwu6VKXP-RttcTEGFe+ySTGWT9CRNkagDiJVA@mail.gmail.com/
Quoted 18 lines
if i am understanding this correctly here is what the fix should be:
--- kernel/bpf/ringbuf.c | 2 ++ 1 file changed, 2 insertions(+)kernel/bpf/ringbuf.c +1
-0@@ -8,6 +8,7 @@#include <linux/vmalloc.h> #include <linux/wait.h> #include <linux/poll.h> +#include <linux/kmemleak.h> #include <uapi/linux/btf.h> #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)kernel/bpf/ringbuf.c static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) · line 109
Quoted 4 lines
kernel/bpf/ringbuf.c +1
-0@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb =vmap (pages, nr_meta_pages +2 * nr_data_pages,VM_ALLOC |VM_USERMAP ,PAGE_KERNEL );if (rb) { +kmemleak_not_leak ((void *) pages);If that makes kmemleak happy, I have no problems with this. But maybe leave some comment explaining why this is needed at all?
Quoted 6 lines
Quoted 4 lines
And for my understanding, how vmap changes anything? Those pages are still referenced from rb, which is referenced from some struct file in the system. Sorry if that's a naive question.
valid question, it does look like kmemleak should be scanning vmalloc()/vmap() memory, i will research this further
a quick update, i see a problem in kmemleak code, and i have simplified the reproducer by getting rid of a vmap().
i will reach out to maintainer and mm and afterwards i will update this bug, cheers!
Quoted 125 lines · Rustam Kovhaev
Quoted 124 lines · Rustam Kovhaev
Quoted 123 lines · Andrii Nakryiko
Quoted 82 lines · Rustam Kovhaev
Quoted 74 lines · Dmitry Vyukov
Quoted 68 lines · Rustam Kovhaev
Quoted 58 lines · Dmitry Vyukov
Quoted 45 lines · Rustam Kovhaev
Quoted 28 lines · syzbot
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing programBUG: memory leak unreferenced object 0xffff88810efccc80 (size 64): comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) hex dump (first 32 bytes): c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected]..... backtrace: [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]Quoted 10 lines
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline] [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline] [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150 [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline] [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline] [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381 [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9i am pretty sure that this one is a false positive
the problem with reproducer is that it does not terminate all of the child processes that it spawnsQuoted 7 lines
i confirmed that it is a false positive by tracing
__fput()andbpf_map_release(), i ran reproducer, got kmemleak report, then i manually killed those running leftover processes from reproducer and then both functions were executed and memory was freedi am marking this one as:
#syz invalidHi Rustam,
Thanks for looking into this.
Quoted 7 lines
I wonder how/where are these objects referenced? If they are not leaked and referenced somewhere, KMEMLEAK should not report them as leaks.
So even if this is a false positive for BPF, this is a true positive bug and something to fix for KMEMLEAK ;)
And syzbot will probably re-create this bug report soon as this still happens and is not a one-off thing.Quoted 4 lines
hi Dmitry, i haven't thought of it this way, but i guess you are right, it is a kmemleak bug, ideally kmemleak should be aware that there are still running processes holding references to bpf fd/anonymous inodes which in their turn hold references to allocated bpf maps
Quoted 6 lines
KMEMLEAK scans whole memory, so if there are pointers to the object anywhere in memory, KMEMLEAK should not report them as leaked. Running processes have no direct effect on KMEMLEAK logic.
So the question is: where are these pointers to these objects? If we answer this, we can check how/why KMEMLEAK misses them. Are they mangled in some way?Quoted 4 lines
thank you for your comments, they make sense, and indeed, the pointer gets vmaped.
i should have looked into this sooner, becaused syzbot did trigger the issue again, and Andrii had to look into the same bug, sorry about that.No worries! I actually forgot about this thread :) Let's leave the link to my today's investigation ([0]) just for completeness.
[0] https://lore.kernel.org/bpf/CAEf4BzYk+dqs+jwu6VKXP-RttcTEGFe+ySTGWT9CRNkagDiJVA@mail.gmail.com/
Quoted 18 lines
if i am understanding this correctly here is what the fix should be:
--- kernel/bpf/ringbuf.c | 2 ++ 1 file changed, 2 insertions(+)kernel/bpf/ringbuf.c +1
-0@@ -8,6 +8,7 @@#include <linux/vmalloc.h> #include <linux/wait.h> #include <linux/poll.h> +#include <linux/kmemleak.h> #include <uapi/linux/btf.h> #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)kernel/bpf/ringbuf.c static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) · line 109
Quoted 4 lines
kernel/bpf/ringbuf.c +1
-0@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb =vmap (pages, nr_meta_pages +2 * nr_data_pages,VM_ALLOC |VM_USERMAP ,PAGE_KERNEL );if (rb) { +kmemleak_not_leak ((void *) pages);If that makes kmemleak happy, I have no problems with this. But maybe leave some comment explaining why this is needed at all?
And for my understanding, how vmap changes anything? Those pages are still referenced from rb, which is referenced from some struct file in the system. Sorry if that's a naive question.
Quoted 8 lines
Quoted 2 lines
valid question, it does look like kmemleak should be scanning vmalloc()/vmap() memory, i will research this further
a quick update, i see a problem in kmemleak code, and i have simplified the reproducer by getting rid of a
vmap().
i will reach out to maintainer and mm and afterwards i will update this bug, cheers!
Andrii, we have discovered that kmemleak scans struct page, but it does not scan page contents and this is by design. if we allocate some memory with kmalloc(), then allocate page with alloc_page(), and if we put kmalloc pointer somewhere inside that page, kmemleak will report kmalloc pointer as a false positive.
we can instruct kmemleak to scan the memory area by calling kmemleak_alloc()/kmemleak_free() as shown below. if we don't need that memory to be scanned then we can use kmemleak_not_leak().
if we use the former then i guess we need to be careful since we do not want/need to scan the memory that is being used by user-space.
--- kernel/bpf/ringbuf.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/bpf/ringbuf.c +3 -0
@@ -8,6 +8,7 @@#include <linux/vmalloc.h> #include <linux/wait.h> #include <linux/poll.h> +#include <linux/kmemleak.h> #include <uapi/linux/btf.h> #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)
@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb =vmap (pages, nr_meta_pages +2 * nr_data_pages,VM_ALLOC |VM_USERMAP ,PAGE_KERNEL );if (rb) { +kmemleak_alloc (rb,PAGE_SIZE ,1 , flags); rb->pages = pages; rb->nr_pages = nr_pages;return rb;
@@ -184,6 +186,7 @@ static void bpf_ringbuf_free(struct bpf_ringbuf *rb)struct page **pages = rb->pages;int i, nr_pages = rb->nr_pages; +kmemleak_free (rb);vunmap (rb);for (i =0 ; i < nr_pages; i++)__free_page (pages[i]);
Signature · 1 line
2.30.2
Quoted 132 lines · Rustam Kovhaev
Quoted 123 lines
Quoted 123 lines
Quoted 123 lines · Rustam Kovhaev
Quoted 82 lines · Dmitry Vyukov
Quoted 74 lines · Rustam Kovhaev
Quoted 68 lines · Dmitry Vyukov
Quoted 58 lines · Rustam Kovhaev
Quoted 45 lines · syzbot
Quoted 28 lines
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing programBUG: memory leak unreferenced object 0xffff88810efccc80 (size 64): comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) hex dump (first 32 bytes): c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected]..... backtrace: [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]Quoted 10 lines
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline] [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline] [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150 [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline] [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline] [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381 [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9i am pretty sure that this one is a false positive
the problem with reproducer is that it does not terminate all of the child processes that it spawnsQuoted 7 lines
i confirmed that it is a false positive by tracing
__fput()andbpf_map_release(), i ran reproducer, got kmemleak report, then i manually killed those running leftover processes from reproducer and then both functions were executed and memory was freedi am marking this one as:
#syz invalidHi Rustam,
Thanks for looking into this.
Quoted 7 lines
I wonder how/where are these objects referenced? If they are not leaked and referenced somewhere, KMEMLEAK should not report them as leaks.
So even if this is a false positive for BPF, this is a true positive bug and something to fix for KMEMLEAK ;)
And syzbot will probably re-create this bug report soon as this still happens and is not a one-off thing.Quoted 4 lines
hi Dmitry, i haven't thought of it this way, but i guess you are right, it is a kmemleak bug, ideally kmemleak should be aware that there are still running processes holding references to bpf fd/anonymous inodes which in their turn hold references to allocated bpf maps
Quoted 6 lines
KMEMLEAK scans whole memory, so if there are pointers to the object anywhere in memory, KMEMLEAK should not report them as leaked. Running processes have no direct effect on KMEMLEAK logic.
So the question is: where are these pointers to these objects? If we answer this, we can check how/why KMEMLEAK misses them. Are they mangled in some way?Quoted 4 lines
thank you for your comments, they make sense, and indeed, the pointer gets vmaped.
i should have looked into this sooner, becaused syzbot did trigger the issue again, and Andrii had to look into the same bug, sorry about that.No worries! I actually forgot about this thread :) Let's leave the link to my today's investigation ([0]) just for completeness.
[0] https://lore.kernel.org/bpf/CAEf4BzYk+dqs+jwu6VKXP-RttcTEGFe+ySTGWT9CRNkagDiJVA@mail.gmail.com/
Quoted 18 lines
if i am understanding this correctly here is what the fix should be:
--- kernel/bpf/ringbuf.c | 2 ++ 1 file changed, 2 insertions(+)kernel/bpf/ringbuf.c +1
-0@@ -8,6 +8,7 @@#include <linux/vmalloc.h> #include <linux/wait.h> #include <linux/poll.h> +#include <linux/kmemleak.h> #include <uapi/linux/btf.h> #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)kernel/bpf/ringbuf.c static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) · line 109
Quoted 4 lines
kernel/bpf/ringbuf.c +1
-0@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb =vmap (pages, nr_meta_pages +2 * nr_data_pages,VM_ALLOC |VM_USERMAP ,PAGE_KERNEL );if (rb) { +kmemleak_not_leak ((void *) pages);If that makes kmemleak happy, I have no problems with this. But maybe leave some comment explaining why this is needed at all?
And for my understanding, how vmap changes anything? Those pages are still referenced from rb, which is referenced from some struct file in the system. Sorry if that's a naive question.
Quoted 8 lines
Quoted 2 lines
valid question, it does look like kmemleak should be scanning vmalloc()/vmap() memory, i will research this further
a quick update, i see a problem in kmemleak code, and i have simplified the reproducer by getting rid of a
vmap().
i will reach out to maintainer and mm and afterwards i will update this bug, cheers!
Quoted 10 lines
Andrii, we have discovered that kmemleak scans struct page, but it does not scan page contents and this is by design. if we allocate some memory with
kmalloc(), then allocate page withalloc_page(), and if we put kmalloc pointer somewhere inside that page, kmemleak will report kmalloc pointer as a false positive.
we can instruct kmemleak to scan the memory area by calling kmemleak_alloc()/kmemleak_free() as shown below. if we don't need that memory to be scanned then we can usekmemleak_not_leak().
if we use the former then i guess we need to be careful since we do not want/need to scan the memory that is being used by user-space.
Thanks for your heroic digging and persistence on this issue, Rustam!
Quoted 35 lines
--- kernel/bpf/ringbuf.c | 3 +++ 1 file changed, 3 insertions(+)kernel/bpf/ringbuf.c +3
-1@@ -8,6 +8,7 @@#include <linux/vmalloc.h> #include <linux/wait.h> #include <linux/poll.h> +#include <linux/kmemleak.h> #include <uapi/linux/btf.h> #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb =vmap (pages, nr_meta_pages +2 * nr_data_pages,VM_ALLOC |VM_USERMAP ,PAGE_KERNEL );if (rb) { +kmemleak_alloc (rb,PAGE_SIZE ,1 , flags); rb->pages = pages; rb->nr_pages = nr_pages;return rb;@@ -184,6 +186,7 @@ static void bpf_ringbuf_free(struct bpf_ringbuf *rb)struct page **pages = rb->pages;int i, nr_pages = rb->nr_pages; +kmemleak_free (rb);vunmap (rb);for (i =0 ; i < nr_pages; i++)__free_page (pages[i]); --2.30.2
Quoted 144 lines · Dmitry Vyukov
Quoted 132 lines · Rustam Kovhaev
Quoted 123 lines
Quoted 123 lines
Quoted 123 lines · Rustam Kovhaev
Quoted 82 lines · Dmitry Vyukov
Quoted 74 lines · Rustam Kovhaev
Quoted 68 lines · Dmitry Vyukov
Quoted 58 lines · Rustam Kovhaev
Quoted 45 lines · syzbot
Quoted 28 lines
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing programBUG: memory leak unreferenced object 0xffff88810efccc80 (size 64): comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) hex dump (first 32 bytes): c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected]..... backtrace: [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]Quoted 10 lines
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline] [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline] [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150 [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline] [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline] [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381 [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9i am pretty sure that this one is a false positive
the problem with reproducer is that it does not terminate all of the child processes that it spawnsQuoted 7 lines
i confirmed that it is a false positive by tracing
__fput()andbpf_map_release(), i ran reproducer, got kmemleak report, then i manually killed those running leftover processes from reproducer and then both functions were executed and memory was freedi am marking this one as:
#syz invalidHi Rustam,
Thanks for looking into this.
Quoted 7 lines
I wonder how/where are these objects referenced? If they are not leaked and referenced somewhere, KMEMLEAK should not report them as leaks.
So even if this is a false positive for BPF, this is a true positive bug and something to fix for KMEMLEAK ;)
And syzbot will probably re-create this bug report soon as this still happens and is not a one-off thing.Quoted 4 lines
hi Dmitry, i haven't thought of it this way, but i guess you are right, it is a kmemleak bug, ideally kmemleak should be aware that there are still running processes holding references to bpf fd/anonymous inodes which in their turn hold references to allocated bpf maps
Quoted 6 lines
KMEMLEAK scans whole memory, so if there are pointers to the object anywhere in memory, KMEMLEAK should not report them as leaked. Running processes have no direct effect on KMEMLEAK logic.
So the question is: where are these pointers to these objects? If we answer this, we can check how/why KMEMLEAK misses them. Are they mangled in some way?Quoted 4 lines
thank you for your comments, they make sense, and indeed, the pointer gets vmaped.
i should have looked into this sooner, becaused syzbot did trigger the issue again, and Andrii had to look into the same bug, sorry about that.No worries! I actually forgot about this thread :) Let's leave the link to my today's investigation ([0]) just for completeness.
[0] https://lore.kernel.org/bpf/CAEf4BzYk+dqs+jwu6VKXP-RttcTEGFe+ySTGWT9CRNkagDiJVA@mail.gmail.com/
Quoted 18 lines
if i am understanding this correctly here is what the fix should be:
--- kernel/bpf/ringbuf.c | 2 ++ 1 file changed, 2 insertions(+)kernel/bpf/ringbuf.c +1
-0@@ -8,6 +8,7 @@#include <linux/vmalloc.h> #include <linux/wait.h> #include <linux/poll.h> +#include <linux/kmemleak.h> #include <uapi/linux/btf.h> #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)kernel/bpf/ringbuf.c static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) · line 109
Quoted 4 lines
kernel/bpf/ringbuf.c +1
-0@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb =vmap (pages, nr_meta_pages +2 * nr_data_pages,VM_ALLOC |VM_USERMAP ,PAGE_KERNEL );if (rb) { +kmemleak_not_leak ((void *) pages);If that makes kmemleak happy, I have no problems with this. But maybe leave some comment explaining why this is needed at all?
And for my understanding, how vmap changes anything? Those pages are still referenced from rb, which is referenced from some struct file in the system. Sorry if that's a naive question.
Quoted 8 lines
Quoted 2 lines
valid question, it does look like kmemleak should be scanning vmalloc()/vmap() memory, i will research this further
a quick update, i see a problem in kmemleak code, and i have simplified the reproducer by getting rid of a
vmap().
i will reach out to maintainer and mm and afterwards i will update this bug, cheers!Quoted 10 lines
Andrii, we have discovered that kmemleak scans struct page, but it does not scan page contents and this is by design. if we allocate some memory with
kmalloc(), then allocate page withalloc_page(), and if we put kmalloc pointer somewhere inside that page, kmemleak will report kmalloc pointer as a false positive.
we can instruct kmemleak to scan the memory area by calling kmemleak_alloc()/kmemleak_free() as shown below. if we don't need that memory to be scanned then we can usekmemleak_not_leak().
if we use the former then i guess we need to be careful since we do not want/need to scan the memory that is being used by user-space.
Quoted 2 lines
Thanks for your heroic digging and persistence on this issue, Rustam!
thank you for the kind words, Dmitry, and thank you for asking the questions, i learned quite a lot while trying to answer them. i've just sent out a patch for this.
Quoted 35 lines
Quoted 35 lines
--- kernel/bpf/ringbuf.c | 3 +++ 1 file changed, 3 insertions(+)kernel/bpf/ringbuf.c +3
-1@@ -8,6 +8,7 @@#include <linux/vmalloc.h> #include <linux/wait.h> #include <linux/poll.h> +#include <linux/kmemleak.h> #include <uapi/linux/btf.h> #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb =vmap (pages, nr_meta_pages +2 * nr_data_pages,VM_ALLOC |VM_USERMAP ,PAGE_KERNEL );if (rb) { +kmemleak_alloc (rb,PAGE_SIZE ,1 , flags); rb->pages = pages; rb->nr_pages = nr_pages;return rb;@@ -184,6 +186,7 @@ static void bpf_ringbuf_free(struct bpf_ringbuf *rb)struct page **pages = rb->pages;int i, nr_pages = rb->nr_pages; +kmemleak_free (rb);vunmap (rb);for (i =0 ; i < nr_pages; i++)__free_page (pages[i]); --2.30.2
Quoted 136 lines · Rustam Kovhaev
Quoted 125 lines · Rustam Kovhaev
Quoted 124 lines · Rustam Kovhaev
Quoted 123 lines · Andrii Nakryiko
Quoted 82 lines · Rustam Kovhaev
Quoted 74 lines · Dmitry Vyukov
Quoted 68 lines · Rustam Kovhaev
Quoted 58 lines · Dmitry Vyukov
Quoted 45 lines · Rustam Kovhaev
Quoted 28 lines · syzbot
syzbot has found a reproducer for the following issue on:
HEAD commit: a68a0262 mm/madvise: remove racy mm ownership check git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=11facf17500000 kernel config: https://syzkaller.appspot.com/x/.config?x=4305fa9ea70c7a9f dashboard link: https://syzkaller.appspot.com/bug?extid=f3694595248708227d35 compiler: gcc (GCC) 10.1.0-syz 20200507 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=159a9613500000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=11bf7123500000IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by [email protected]Debian GNU/Linux 9 syzkaller ttyS0
Warning: Permanently added '10.128.0.9' (ECDSA) to the list of known hosts. executing program
executing program
executing programBUG: memory leak unreferenced object 0xffff88810efccc80 (size 64): comm "syz-executor334", pid 8460, jiffies 4294945724 (age 13.850s) hex dump (first 32 bytes): c0 cb 14 04 00 ea ff ff c0 c2 11 04 00 ea ff ff ................ c0 56 3f 04 00 ea ff ff 40 18 38 04 00 ea ff ff [email protected]..... backtrace: [<0000000036ae98a7>] kmalloc_node include/linux/slab.h:575 [inline] [<0000000036ae98a7>] bpf_ringbuf_area_alloc kernel/bpf/ringbuf.c:94 [inline]Quoted 10 lines
[<0000000036ae98a7>] bpf_ringbuf_alloc kernel/bpf/ringbuf.c:135 [inline] [<0000000036ae98a7>] ringbuf_map_alloc kernel/bpf/ringbuf.c:183 [inline] [<0000000036ae98a7>] ringbuf_map_alloc+0x1be/0x410 kernel/bpf/ringbuf.c:150 [<00000000d2cb93ae>] find_and_alloc_map kernel/bpf/syscall.c:122 [inline] [<00000000d2cb93ae>] map_create kernel/bpf/syscall.c:825 [inline] [<00000000d2cb93ae>] __do_sys_bpf+0x7d0/0x30a0 kernel/bpf/syscall.c:4381 [<000000008feaf393>] do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46 [<00000000e1f53cfd>] entry_SYSCALL_64_after_hwframe+0x44/0xa9i am pretty sure that this one is a false positive
the problem with reproducer is that it does not terminate all of the child processes that it spawnsQuoted 7 lines
i confirmed that it is a false positive by tracing
__fput()andbpf_map_release(), i ran reproducer, got kmemleak report, then i manually killed those running leftover processes from reproducer and then both functions were executed and memory was freedi am marking this one as:
#syz invalidHi Rustam,
Thanks for looking into this.
Quoted 7 lines
I wonder how/where are these objects referenced? If they are not leaked and referenced somewhere, KMEMLEAK should not report them as leaks.
So even if this is a false positive for BPF, this is a true positive bug and something to fix for KMEMLEAK ;)
And syzbot will probably re-create this bug report soon as this still happens and is not a one-off thing.Quoted 4 lines
hi Dmitry, i haven't thought of it this way, but i guess you are right, it is a kmemleak bug, ideally kmemleak should be aware that there are still running processes holding references to bpf fd/anonymous inodes which in their turn hold references to allocated bpf maps
Quoted 6 lines
KMEMLEAK scans whole memory, so if there are pointers to the object anywhere in memory, KMEMLEAK should not report them as leaked. Running processes have no direct effect on KMEMLEAK logic.
So the question is: where are these pointers to these objects? If we answer this, we can check how/why KMEMLEAK misses them. Are they mangled in some way?Quoted 4 lines
thank you for your comments, they make sense, and indeed, the pointer gets vmaped.
i should have looked into this sooner, becaused syzbot did trigger the issue again, and Andrii had to look into the same bug, sorry about that.No worries! I actually forgot about this thread :) Let's leave the link to my today's investigation ([0]) just for completeness.
[0] https://lore.kernel.org/bpf/CAEf4BzYk+dqs+jwu6VKXP-RttcTEGFe+ySTGWT9CRNkagDiJVA@mail.gmail.com/
Quoted 18 lines
if i am understanding this correctly here is what the fix should be:
--- kernel/bpf/ringbuf.c | 2 ++ 1 file changed, 2 insertions(+)kernel/bpf/ringbuf.c +1
-0@@ -8,6 +8,7 @@#include <linux/vmalloc.h> #include <linux/wait.h> #include <linux/poll.h> +#include <linux/kmemleak.h> #include <uapi/linux/btf.h> #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)kernel/bpf/ringbuf.c static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) · line 109
Quoted 4 lines
kernel/bpf/ringbuf.c +1
-0@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb =vmap (pages, nr_meta_pages +2 * nr_data_pages,VM_ALLOC |VM_USERMAP ,PAGE_KERNEL );if (rb) { +kmemleak_not_leak ((void *) pages);If that makes kmemleak happy, I have no problems with this. But maybe leave some comment explaining why this is needed at all?
And for my understanding, how vmap changes anything? Those pages are still referenced from rb, which is referenced from some struct file in the system. Sorry if that's a naive question.
Quoted 8 lines
Quoted 2 lines
valid question, it does look like kmemleak should be scanning vmalloc()/vmap() memory, i will research this further
a quick update, i see a problem in kmemleak code, and i have simplified the reproducer by getting rid of a
vmap().
i will reach out to maintainer and mm and afterwards i will update this bug, cheers!
Quoted 11 lines
Andrii, we have discovered that kmemleak scans struct page, but it does not scan page contents and this is by design. if we allocate some memory with
kmalloc(), then allocate page withalloc_page(), and if we put kmalloc pointer somewhere inside that page, kmemleak will report kmalloc pointer as a false positive.
we can instruct kmemleak to scan the memory area by calling kmemleak_alloc()/kmemleak_free() as shown below. if we don't need that memory to be scanned then we can usekmemleak_not_leak().
if we use the former then i guess we need to be careful since we do not want/need to scan the memory that is being used by user-space.
Hi, sorry for late reply, I was out for two weeks. Thanks for investigating and sending a patch!
Quoted 35 lines
--- kernel/bpf/ringbuf.c | 3 +++ 1 file changed, 3 insertions(+)kernel/bpf/ringbuf.c +3
-1@@ -8,6 +8,7 @@#include <linux/vmalloc.h> #include <linux/wait.h> #include <linux/poll.h> +#include <linux/kmemleak.h> #include <uapi/linux/btf.h> #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE)@@ -105,6 +106,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) rb =vmap (pages, nr_meta_pages +2 * nr_data_pages,VM_ALLOC |VM_USERMAP ,PAGE_KERNEL );if (rb) { +kmemleak_alloc (rb,PAGE_SIZE ,1 , flags); rb->pages = pages; rb->nr_pages = nr_pages;return rb;@@ -184,6 +186,7 @@ static void bpf_ringbuf_free(struct bpf_ringbuf *rb)struct page **pages = rb->pages;int i, nr_pages = rb->nr_pages; +kmemleak_free (rb);vunmap (rb);for (i =0 ; i < nr_pages; i++)__free_page (pages[i]); --2.30.2