perf failed on multi-cpus system with multi-events specified

Hi,

On a Nehalem-EX system, which has 24 CPUs, perf commands with multi-events specified fail:

[root@intel-s3e36-01 perf]# ./perf record -e syscalls:sys_enter_sched_get_priority_min -e kmem:mm_kernel_pagefault

(quoted diff) +0 -2

-e workqueue:workqueue_execution  -e writeback:writeback_pages_written
-e kmem:mm_kernel_pagefault  -e writeback:writeback_bdi_unregister  -e

syscalls:sys_exit_capset -e syscalls:sys_enter_sched_get_priority_min -e writeback:writeback_exec -e syscalls:sys_enter_vmsplice -e syscalls:sys_exit_mprotect -e block:block_rq_complete -e syscalls:sys_enter_readlinkat -e signal:signal_overflow_fail -e syscalls:sys_exit_sendfile64 -e block:block_bio_backmerge -e syscalls:sys_exit_unlink -e syscalls:sys_exit_keyctl -e jbd2:jbd2_submit_inode_data -e syscalls:sys_enter_tgkill -e syscalls:sys_exit_lremovexattr -e syscalls:sys_exit_munmap -e syscalls:sys_enter_mmap -e syscalls:sys_exit_sched_getparam -e syscalls:sys_exit_getpgid -e syscalls:sys_exit_clock_getres -e block:block_split -e ext4:ext4_mballoc_discard -e task-clock -e syscalls:sys_enter_sendfile64 -e syscalls:sys_enter_ptrace -e syscalls:sys_enter_nanosleep -e syscalls:sys_exit_setdomainname -e writeback:writeback_nowork -e syscalls:sys_exit_mbind -e

ext4:ext4_da_write_begin  -e syscalls:sys_enter_recvfrom  -e
syscalls:sys_enter_write  -e syscalls:sys_exit_getresgid  -e

syscalls:sys_exit_fsync -e syscalls:sys_enter_rt_sigprocmask -e syscalls:sys_enter_getdents -e syscalls:sys_exit_semtimedop -f ls

Error: sys_perf_event_open() syscall returned with -1 (Too many open
files). /bin/dmesg may provide additional information.

Fatal: No CONFIG_PERF_EVENTS=y kernel support configured?

trace-cmd: Too many open files
No memory to alloc tracepoints list
[root@intel-s3e36-01 perf]# ./perf --version
perf version 2.6.38.rc6.166.g493f33
[root@intel-s3e36-01 perf]#

On another latop which has only two CPUs, there is no this problem.

Quoted 39 lines · Han Pingtian

Hi,

On a Nehalem-EX system, which has 24 CPUs, perf commands with multi-events specified fail:

[root@intel-s3e36-01 perf]# ./perf record -e syscalls:sys_enter_sched_get_priority_min -e kmem:mm_kernel_pagefault

(quoted diff) +0 -2

-e workqueue:workqueue_execution  -e writeback:writeback_pages_written
-e kmem:mm_kernel_pagefault  -e writeback:writeback_bdi_unregister  -e

syscalls:sys_exit_capset -e syscalls:sys_enter_sched_get_priority_min -e writeback:writeback_exec -e syscalls:sys_enter_vmsplice -e syscalls:sys_exit_mprotect -e block:block_rq_complete -e syscalls:sys_enter_readlinkat -e signal:signal_overflow_fail -e syscalls:sys_exit_sendfile64 -e block:block_bio_backmerge -e syscalls:sys_exit_unlink -e syscalls:sys_exit_keyctl -e jbd2:jbd2_submit_inode_data -e syscalls:sys_enter_tgkill -e syscalls:sys_exit_lremovexattr -e syscalls:sys_exit_munmap -e syscalls:sys_enter_mmap -e syscalls:sys_exit_sched_getparam -e syscalls:sys_exit_getpgid -e syscalls:sys_exit_clock_getres -e block:block_split -e ext4:ext4_mballoc_discard -e task-clock -e syscalls:sys_enter_sendfile64 -e syscalls:sys_enter_ptrace -e syscalls:sys_enter_nanosleep -e syscalls:sys_exit_setdomainname -e writeback:writeback_nowork -e syscalls:sys_exit_mbind -e

ext4:ext4_da_write_begin  -e syscalls:sys_enter_recvfrom  -e
syscalls:sys_enter_write  -e syscalls:sys_exit_getresgid  -e

syscalls:sys_exit_fsync -e syscalls:sys_enter_rt_sigprocmask -e syscalls:sys_enter_getdents -e syscalls:sys_exit_semtimedop -f ls

Error: sys_perf_event_open() syscall returned with -1 (Too many open
files). /bin/dmesg may provide additional information.

Fatal: No CONFIG_PERF_EVENTS=y kernel support configured?

trace-cmd: Too many open files
No memory to alloc tracepoints list
[root@intel-s3e36-01 perf]# ./perf --version
perf version 2.6.38.rc6.166.g493f33
[root@intel-s3e36-01 perf]#

Quoted 1 line

On another latop which has only two CPUs, there is no this problem.

perf needs nr_cpus * nr_events file descriptors

Try before :

ulimit -n 1000000

Error: sys_perf_event_open() syscall returned with -1 (Too many open
files). /bin/dmesg may provide additional information.

Quoted 2 lines · Han Pingtian

Error: sys_perf_event_open() syscall returned with -1 (Too many open
files). /bin/dmesg may provide additional information.

Well, you tried to use 42 events (quick count), we need to open a fd for each cpu for each event, which results in 1008 fds, which together with some actual file handles might have tickled your fd limit.