perf record: Add support to store data in directory
Jiri Olsa
reviewerv5 cover perf record: Add support to store data in directory
hi,
this patchset adds the --dir option to record command (and all the other record command that overload cmd_record) that allows the data to be stored in directory with multiple data files.
It's next step for multiple threads implementation in record. It's now possible to make directory data via --dir option, like:
$ perf record --dir perf bench sched messaging $ ls -l perf.data total 344 -rw-------. 1 jolsa jolsa 43864 Jan 20 22:26 data.0 -rw-------. 1 jolsa jolsa 30464 Jan 20 22:26 data.1 -rw-------. 1 jolsa jolsa 53816 Jan 20 22:26 data.2 -rw-------. 1 jolsa jolsa 30368 Jan 20 22:26 data.3 -rw-------. 1 jolsa jolsa 40088 Jan 20 22:26 data.4 -rw-------. 1 jolsa jolsa 42592 Jan 20 22:26 data.5 -rw-------. 1 jolsa jolsa 56136 Jan 20 22:26 data.6 -rw-------. 1 jolsa jolsa 25992 Jan 20 22:26 data.7 -rw-------. 1 jolsa jolsa 8832 Jan 20 22:26 header
There's a data file created for every cpu and it's storing data for those cpu maps. The report command will read it transparently, sort it and display as single file data.
There's new DIR_FORMAT feature to describe directory data layout/format. In future we can describe different data files layout according to special needs.
It's possible to transform directory data into standard perf.data file via simple inject command:
$ perf inject -o perf.data.file -i perf.data
The old perf fails over the directory data with following message:
$ perf report incompatible file format (rerun with -v to learn more)
I'm now testing the record threads support, so I'd like to have some agreement on the directory data support before.
v5 changes:
- rebased to latest Arnaldo's perf/core
v4 changes:
- some of the patches already taken
- removing up perf.data directory on perf_data__open error path
v3 changes:
- add rm_rf_perf_data to safely remove file/directory perf data
- allocation fix in perf_data__create_dir
v2 changes:
- rm_rf changes are already accepted with requested changes
- updated doc/man plus adding perf.data-directory-format.txt to describe directory format/layout
- the --switch-output options now works over directory data
- data rollback is not part of this patchset, updated my TODO though ;-)
- added --output-dir option to combine -o and --dir
- added DIR_FORMAT feature to describe directory data
- disabling directory output for aio for now
It's also available in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/dir
thanks,
jirka
---
Jiri Olsa (11):
perf data: Add directory support
perf data: Don't store auxtrace index for directory data file perf data: Add perf_data__update_dir function
perf data: Make perf_data__size to work over directory perf header: Add DIR_FORMAT feature to describe directory data perf session: Add process callback to reader object perf session: Add __perf_session__process_dir_events function perf session: Add path to reader object
perf record: Add --dir option to store data in directory perf record: Add --output-dir option to store data in directory perf record: Describe perf.data directory format
tools/lib/subcmd/parse-options.h | 4 ++++ tools/perf/Documentation/perf-record.txt | 6 ++++++ tools/perf/Documentation/perf.data-directory-format.txt | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/perf/builtin-record.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------- tools/perf/util/data.c | 96 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--- tools/perf/util/data.h | 12 ++++++++---- tools/perf/util/header.c | 44 +++++++++++++++++++++++++++++++++++++++++++- tools/perf/util/header.h | 5 +++++ tools/perf/util/mmap.h | 23 ++++++++++++----------- tools/perf/util/session.c | 125 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------- 10 files changed, 430 insertions(+), 34 deletions(-) create mode 100644 tools/perf/Documentation/perf.data-directory-format.txt
Jiri Olsa
reviewerPATCH v5 1/11 perf data: Add directory support
Adding support to have directory as perf.data.
The caller needs to set 'struct perf_data::is_dir flag and the path will be treated as directory.
The 'struct perf_data::file' is initialized and open as 'path/header' file.
Adding check to direcory interface functions to check on is_dir flag.
--- tools/perf/util/data.c | 49 ++++++++++++++++++++++++++++++++++++++- tools/perf/util/data.h | 6 +++++ tools/perf/util/session.c | 4 ++++ 3 files changed, 58 insertions(+), 1 deletion(-)
tools/perf/util/data.c +48 -1
@@ -34,6 +34,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)struct perf_data_file *files =NULL ;int i, ret =-1 ; +if (WARN_ON (!data->is_dir)) +return -EINVAL ; + files =zalloc (nr *sizeof (*files));if (!files)return -ENOMEM ;
@@ -69,6 +72,9 @@ int perf_data__open_dir(struct perf_data *data)DIR *dir;int nr =0 ; +if (WARN_ON (!data->is_dir)) +return -EINVAL ; + dir =opendir (data->path);if (!dir)return -EINVAL ;
@@ -173,6 +179,16 @@ static int check_backup(struct perf_data *data)return 0 ; } +static bool is_dir (struct perf_data *data) +{ +struct stat st; + +if (stat (data->path, &st)) +return false; + +return (st.st_mode &S_IFMT ) ==S_IFDIR ; +} +static int open_file_read (struct perf_data *data) {struct stat st;
@@ -254,6 +270,30 @@ static int open_file_dup(struct perf_data *data)return open_file (data); } +static int open_dir (struct perf_data *data) +{ +int ret; + +/* +* So far we open only the header, so we +* can read the data version and layout. +*/ +if (asprintf (&data->file.path,"%s/header" , data->path) <0 ) +return -ENOMEM ; + +if (perf_data__is_write (data) && +mkdir (data->path,S_IRWXU ) <0 ) +return -1 ; + + ret =open_file (data); + +/* Cleanup whatever we managed to create so far. */ +if (ret &&perf_data__is_write (data)) +rm_rf_perf_data (data->path); + +return ret; +} +int perf_data__open (struct perf_data *data) {if (check_pipe (data))
@@ -265,11 +305,18 @@ int perf_data__open(struct perf_data *data)if (check_backup (data))return -1 ; -return open_file_dup (data); +if (perf_data__is_read (data)) + data->is_dir =is_dir (data); + +return perf_data__is_dir (data) ? +open_dir (data) :open_file_dup (data); }void perf_data__close (struct perf_data *data) { +if (perf_data__is_dir (data)) +perf_data__close_dir (data); +zfree (&data->file.path);close (data->file.fd); }
tools/perf/util/data.h +6 -0
@@ -19,6 +19,7 @@ struct perf_data {const char *path;struct perf_data_file file;bool is_pipe; +bool is_dir;bool force;enum perf_data_mode mode;
@@ -43,6 +44,11 @@ static inline int perf_data__is_pipe(struct perf_data *data)return data->is_pipe; } +static inline bool perf_data__is_dir (struct perf_data *data) +{ +return data->is_dir; +} +static inline int perf_data__fd (struct perf_data *data) {return data->file.fd;
tools/perf/util/session.c +4 -0
@@ -152,6 +152,10 @@ struct perf_session *perf_session__new(struct perf_data *data, }perf_evlist__init_trace_event_sample_raw (session->evlist); + +/* Open the directory data. */ +if (data->is_dir &&perf_data__open_dir (data)) + goto out_delete; } }else { session->machines.host.env = &perf_env;
Signature · 1 line
2.17.2
Re: [PATCH 01/11] perf data: Add directory support
Em Fri, Mar 08, 2019 at 02:47:35PM +0100, Jiri Olsa escreveu:
Quoted 68 lines
Adding support to have directory as perf.data.
The caller needs to set 'struct
perf_data::is_dirflag and the path will be treated as directory.The 'struct
perf_data::file' is initialized and open as 'path/header' file.Adding check to direcory interface functions to check on
is_dirflag.Signed-off-by Jiri Olsa <[email protected]>--- tools/perf/util/data.c | 49 ++++++++++++++++++++++++++++++++++++++- tools/perf/util/data.h | 6 +++++ tools/perf/util/session.c | 4 ++++ 3 files changed, 58 insertions(+), 1 deletion(-)tools/perf/util/data.c +20
-0@@ -34,6 +34,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)struct perf_data_file *files =NULL ;int i, ret =-1 ; +if (WARN_ON (!data->is_dir)) +return -EINVAL ; + files =zalloc (nr *sizeof (*files));if (!files)return -ENOMEM ;@@ -69,6 +72,9 @@ int perf_data__open_dir(struct perf_data *data)DIR *dir;int nr =0 ; +if (WARN_ON (!data->is_dir)) +return -EINVAL ; + dir =opendir (data->path);if (!dir)return -EINVAL ;@@ -173,6 +179,16 @@ static int check_backup(struct perf_data *data)return 0 ; } +static bool is_dir (struct perf_data *data) +{ +struct stat st; + +if (stat (data->path, &st)) +return false; + +return (st.st_mode &S_IFMT ) ==S_IFDIR ; +} +static int open_file_read (struct perf_data *data) {struct stat st;@@ -254,6 +270,30 @@ static int open_file_dup(struct perf_data *data)return open_file (data); } +static int open_dir (struct perf_data *data) +{ +int ret; +
tools/perf/util/data.c static int open_file_dup(struct perf_data *data) · line 286
Quoted 10 lines
tools/perf/util/data.c +10
-0+/* +* So far we open only the header, so we +* can read the data version and layout. +*/ +if (asprintf (&data->file.path,"%s/header" , data->path) <0 ) +return -ENOMEM ; + +if (perf_data__is_write (data) && +mkdir (data->path,S_IRWXU ) <0 ) +return -1 ;
Please be consistent, either -ESOMETHING or -1, I think returning -1 and letting the caller look at errno should be enough, as asprintf() returns straight away if the malloc it does fail, leaving errno set to ENOMEM.
I'm doing this change here.
Quoted 74 lines
tools/perf/util/data.c +18
-1+ + ret =open_file (data); + +/* Cleanup whatever we managed to create so far. */ +if (ret &&perf_data__is_write (data)) +rm_rf_perf_data (data->path); + +return ret; +} +int perf_data__open (struct perf_data *data) {if (check_pipe (data))@@ -265,11 +305,18 @@ int perf_data__open(struct perf_data *data)if (check_backup (data))return -1 ; -return open_file_dup (data); +if (perf_data__is_read (data)) + data->is_dir =is_dir (data); + +return perf_data__is_dir (data) ? +open_dir (data) :open_file_dup (data); }void perf_data__close (struct perf_data *data) { +if (perf_data__is_dir (data)) +perf_data__close_dir (data); +zfree (&data->file.path);close (data->file.fd); }tools/perf/util/data.h +6
-0@@ -19,6 +19,7 @@ struct perf_data {const char *path;struct perf_data_file file;bool is_pipe; +bool is_dir;bool force;enum perf_data_mode mode;@@ -43,6 +44,11 @@ static inline int perf_data__is_pipe(struct perf_data *data)return data->is_pipe; } +static inline bool perf_data__is_dir (struct perf_data *data) +{ +return data->is_dir; +} +static inline int perf_data__fd (struct perf_data *data) {return data->file.fd;tools/perf/util/session.c +4
-0@@ -152,6 +152,10 @@ struct perf_session *perf_session__new(struct perf_data *data, }perf_evlist__init_trace_event_sample_raw (session->evlist); + +/* Open the directory data. */ +if (data->is_dir &&perf_data__open_dir (data)) + goto out_delete; } }else { session->machines.host.env = &perf_env;Signature · 1 line
2.17.2
Signature · 1 line
- Arnaldo
tip-bot for Jiri Olsa
merge bot[tip:perf/urgent] perf data: Support having perf.data stored as a directory
In reply to Jiri Olsaperf data: Support having perf.data stored as a directory
Commit-ID: ec65def1045e4c7817b7f741a86dadae82877a93 Gitweb: https://git.kernel.org/tip/ec65def1045e4c7817b7f741a86dadae82877a93 Author: Jiri Olsa <[email protected]> AuthorDate: Fri, 8 Mar 2019 14:47:35 +0100 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Mon, 11 Mar 2019 11:56:03 -0300
perf data: Support having perf.data stored as a directory
The caller needs to set 'struct perf_data::is_dir flag and the path will be treated as a directory.
The 'struct perf_data::file' is initialized and open as 'path/header' file.
Add a check to the direcory interface functions to check the is_dir flag.
--- tools/perf/util/data.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++- tools/perf/util/data.h | 6 ++++++ tools/perf/util/session.c | 4 ++++ 3 files changed, 58 insertions(+), 1 deletion(-)
tools/perf/util/data.c +48 -1
@@ -34,6 +34,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)struct perf_data_file *files =NULL ;int i, ret =-1 ; +if (WARN_ON (!data->is_dir)) +return -EINVAL ; + files =zalloc (nr *sizeof (*files));if (!files)return -ENOMEM ;
@@ -69,6 +72,9 @@ int perf_data__open_dir(struct perf_data *data)DIR *dir;int nr =0 ; +if (WARN_ON (!data->is_dir)) +return -EINVAL ; + dir =opendir (data->path);if (!dir)return -EINVAL ;
@@ -173,6 +179,16 @@ static int check_backup(struct perf_data *data)return 0 ; } +static bool is_dir (struct perf_data *data) +{ +struct stat st; + +if (stat (data->path, &st)) +return false; + +return (st.st_mode &S_IFMT ) ==S_IFDIR ; +} +static int open_file_read (struct perf_data *data) {struct stat st;
@@ -254,6 +270,30 @@ static int open_file_dup(struct perf_data *data)return open_file (data); } +static int open_dir (struct perf_data *data) +{ +int ret; + +/* +* So far we open only the header, so we can read the data version and +* layout. +*/ +if (asprintf (&data->file.path,"%s/header" , data->path) <0 ) +return -1 ; + +if (perf_data__is_write (data) && +mkdir (data->path,S_IRWXU ) <0 ) +return -1 ; + + ret =open_file (data); + +/* Cleanup whatever we managed to create so far. */ +if (ret &&perf_data__is_write (data)) +rm_rf_perf_data (data->path); + +return ret; +} +int perf_data__open (struct perf_data *data) {if (check_pipe (data))
@@ -265,11 +305,18 @@ int perf_data__open(struct perf_data *data)if (check_backup (data))return -1 ; -return open_file_dup (data); +if (perf_data__is_read (data)) + data->is_dir =is_dir (data); + +return perf_data__is_dir (data) ? +open_dir (data) :open_file_dup (data); }void perf_data__close (struct perf_data *data) { +if (perf_data__is_dir (data)) +perf_data__close_dir (data); +zfree (&data->file.path);close (data->file.fd); }
tools/perf/util/data.h +6 -0
@@ -19,6 +19,7 @@ struct perf_data {const char *path;struct perf_data_file file;bool is_pipe; +bool is_dir;bool force;enum perf_data_mode mode;
@@ -43,6 +44,11 @@ static inline int perf_data__is_pipe(struct perf_data *data)return data->is_pipe; } +static inline bool perf_data__is_dir (struct perf_data *data) +{ +return data->is_dir; +} +static inline int perf_data__fd (struct perf_data *data) {return data->file.fd;
tools/perf/util/session.c +4 -0
@@ -152,6 +152,10 @@ struct perf_session *perf_session__new(struct perf_data *data, }perf_evlist__init_trace_event_sample_raw (session->evlist); + +/* Open the directory data. */ +if (data->is_dir &&perf_data__open_dir (data)) + goto out_delete; } }else { session->machines.host.env = &perf_env;
Jiri Olsa
reviewerPATCH v5 2/11 perf data: Don't store auxtrace index for directory data file
In reply to Jiri OlsaWe can't store auxtrace index when we store to multiple files, because we keep only offset for it, not the file.
The auxtrace data will be processed correctly in the 'pipe' mode.
--- tools/perf/builtin-record.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/builtin-record.c +1 -1
@@ -392,7 +392,7 @@ static int record__process_auxtrace(struct perf_tool *tool,size_t padding;u8 pad[8 ] = {0 }; -if (!perf_data__is_pipe (data)) { +if (!perf_data__is_pipe (data) && !perf_data__is_dir (data)) {off_t file_offset;int fd =perf_data__fd (data);int err;
Signature · 1 line
2.17.2
tip-bot for Jiri Olsa
merge bot[tip:perf/urgent] perf data: Don't store auxtrace index for directory data file
perf data: Don't store auxtrace index for directory data file
Commit-ID: cd3dd8dd8ff62374d90cb3f2e54b8c94106c7810 Gitweb: https://git.kernel.org/tip/cd3dd8dd8ff62374d90cb3f2e54b8c94106c7810 Author: Jiri Olsa <[email protected]> AuthorDate: Fri, 8 Mar 2019 14:47:36 +0100 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Mon, 11 Mar 2019 11:56:03 -0300
perf data: Don't store auxtrace index for directory data file
We can't store the auxtrace index when we store into multiple files, because we keep only offset for it, not the file.
The auxtrace data will be processed correctly in the 'pipe' mode.
--- tools/perf/builtin-record.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/builtin-record.c +1 -1
@@ -392,7 +392,7 @@ static int record__process_auxtrace(struct perf_tool *tool,size_t padding;u8 pad[8 ] = {0 }; -if (!perf_data__is_pipe (data)) { +if (!perf_data__is_pipe (data) && !perf_data__is_dir (data)) {off_t file_offset;int fd =perf_data__fd (data);int err;
Jiri Olsa
reviewerPATCH v5 3/11 perf data: Add perf_data__update_dir function
In reply to Jiri OlsaAdding perf_data__update_dir function to update size for every file within the perf.data directory.
--- tools/perf/util/data.c | 20 ++++++++++++++++++++ tools/perf/util/data.h | 1 + 2 files changed, 21 insertions(+)
tools/perf/util/data.c +20 -0
@@ -124,6 +124,26 @@ int perf_data__open_dir(struct perf_data *data)return ret; } +int perf_data__update_dir (struct perf_data *data) +{ +int i; + +if (WARN_ON (!data->is_dir)) +return -EINVAL ; + +for (i =0 ; i < data->dir.nr; i++) { +struct perf_data_file *file = &data->dir.files[i]; +struct stat st; + +if (fstat (file->fd, &st)) +return -1 ; + + file->size = st.st_size; + } + +return 0 ; +} +static bool check_pipe (struct perf_data *data) {struct stat st;
tools/perf/util/data.h +1 -0
@@ -79,4 +79,5 @@ int perf_data__switch(struct perf_data *data,int perf_data__create_dir (struct perf_data *data,int nr);int perf_data__open_dir (struct perf_data *data);void perf_data__close_dir (struct perf_data *data); +int perf_data__update_dir (struct perf_data *data);#endif /* __PERF_DATA_H */
Signature · 1 line
2.17.2
tip-bot for Jiri Olsa
merge bot[tip:perf/urgent] perf data: Add perf_data__update_dir() function
perf data: Add perf_data__update_dir() function
Commit-ID: e8be135751f26aa5de63e517d375ecf69e9b20c3 Gitweb: https://git.kernel.org/tip/e8be135751f26aa5de63e517d375ecf69e9b20c3 Author: Jiri Olsa <[email protected]> AuthorDate: Fri, 8 Mar 2019 14:47:37 +0100 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Mon, 11 Mar 2019 11:56:03 -0300
perf data: Add perf_data__update_dir() function
Add perf_data__update_dir() to update the size for every file within the perf.data directory.
--- tools/perf/util/data.c | 20 ++++++++++++++++++++ tools/perf/util/data.h | 1 + 2 files changed, 21 insertions(+)
tools/perf/util/data.c +20 -0
@@ -124,6 +124,26 @@ out_err:return ret; } +int perf_data__update_dir (struct perf_data *data) +{ +int i; + +if (WARN_ON (!data->is_dir)) +return -EINVAL ; + +for (i =0 ; i < data->dir.nr; i++) { +struct perf_data_file *file = &data->dir.files[i]; +struct stat st; + +if (fstat (file->fd, &st)) +return -1 ; + + file->size = st.st_size; + } + +return 0 ; +} +static bool check_pipe (struct perf_data *data) {struct stat st;
tools/perf/util/data.h +1 -0
@@ -79,4 +79,5 @@ int perf_data__switch(struct perf_data *data,int perf_data__create_dir (struct perf_data *data,int nr);int perf_data__open_dir (struct perf_data *data);void perf_data__close_dir (struct perf_data *data); +int perf_data__update_dir (struct perf_data *data);#endif /* __PERF_DATA_H */
Jiri Olsa
reviewerPATCH v5 4/11 perf data: Make perf_data__size to work over directory
In reply to Jiri OlsaMaking perf_data__size to return proper size
for directory data.
--- tools/perf/util/data.c | 17 +++++++++++++++++ tools/perf/util/data.h | 6 +----- 2 files changed, 18 insertions(+), 5 deletions(-)
tools/perf/util/data.c +17 -0
@@ -393,3 +393,20 @@ int perf_data__switch(struct perf_data *data,free (new_filepath);return ret; } + +unsigned long perf_data__size (struct perf_data *data) +{ +u64 size = data->file.size; +int i; + +if (!data->is_dir) +return size; + +for (i =0 ; i < data->dir.nr; i++) { +struct perf_data_file *file = &data->dir.files[i]; + + size += file->size; + } + +return size; +}
tools/perf/util/data.h +1 -5
@@ -54,11 +54,6 @@ static inline int perf_data__fd(struct perf_data *data)return data->file.fd; } -static inline unsigned long perf_data__size (struct perf_data *data) -{ -return data->file.size; -} -int perf_data__open (struct perf_data *data);void perf_data__close (struct perf_data *data); ssize_t perf_data__write(struct perf_data *data,
@@ -80,4 +75,5 @@ int perf_data__create_dir(struct perf_data *data, int nr);int perf_data__open_dir (struct perf_data *data);void perf_data__close_dir (struct perf_data *data);int perf_data__update_dir (struct perf_data *data); +unsigned long perf_data__size (struct perf_data *data);#endif /* __PERF_DATA_H */
Signature · 1 line
2.17.2
Re: [PATCH 04/11] perf data: Make perf_data__size to work over directory
Em Fri, Mar 08, 2019 at 02:47:38PM +0100, Jiri Olsa escreveu:
Quoted 16 lines
Making
perf_data__sizeto return proper size
for directory data.Signed-off-by Jiri Olsa <[email protected]>--- tools/perf/util/data.c | 17 +++++++++++++++++ tools/perf/util/data.h | 6 +----- 2 files changed, 18 insertions(+), 5 deletions(-)tools/perf/util/data.c +0
-0@@ -393,3 +393,20 @@ int perf_data__switch(struct perf_data *data,free (new_filepath);
tools/perf/util/data.c int perf_data__switch(struct perf_data *data, · line 403
Quoted 10 lines
tools/perf/util/data.c +8
-0return ret; } + +unsigned long perf_data__size (struct perf_data *data) +{ +u64 size = data->file.size; +int i; + +if (!data->is_dir) +return size;
So you do the sum over dir.files[i] _plus_ what is in data->file.size, what is in data->file.size? Zero for the dir case? Or some non-per-cpu metadata?
- Arnaldo
Quoted 33 lines
tools/perf/util/data.c +9
-0+ +for (i =0 ; i < data->dir.nr; i++) { +struct perf_data_file *file = &data->dir.files[i]; + + size += file->size; + } + +return size; +}tools/perf/util/data.h +1
-5@@ -54,11 +54,6 @@ static inline int perf_data__fd(struct perf_data *data)return data->file.fd; } -static inline unsigned long perf_data__size (struct perf_data *data) -{ -return data->file.size; -} -int perf_data__open (struct perf_data *data);void perf_data__close (struct perf_data *data); ssize_t perf_data__write(struct perf_data *data,@@ -80,4 +75,5 @@ int perf_data__create_dir(struct perf_data *data, int nr);int perf_data__open_dir (struct perf_data *data);void perf_data__close_dir (struct perf_data *data);int perf_data__update_dir (struct perf_data *data); +unsigned long perf_data__size (struct perf_data *data);#endif /* __PERF_DATA_H */ Signature · 1 line
2.17.2
Signature · 1 line
- Arnaldo
Re: [PATCH 04/11] perf data: Make perf_data__size to work over directory
Quoted 23 lines · Arnaldo Carvalho de Melo
Em Fri, Mar 08, 2019 at 02:47:38PM +0100, Jiri Olsa escreveu:
Quoted 22 lines
Making
perf_data__sizeto return proper size
for directory data.Signed-off-by Jiri Olsa <[email protected]>--- tools/perf/util/data.c | 17 +++++++++++++++++ tools/perf/util/data.h | 6 +----- 2 files changed, 18 insertions(+), 5 deletions(-)tools/perf/util/data.c +4
-0@@ -393,3 +393,20 @@ int perf_data__switch(struct perf_data *data,free (new_filepath);return ret; } + +unsigned long perf_data__size (struct perf_data *data) +{ +u64 size = data->file.size;
Quoted 8 lines
tools/perf/util/data.c int perf_data__switch(struct perf_data *data, · line 403
Quoted 4 lines
tools/perf/util/data.c +4
-0+int i; + +if (!data->is_dir) +return size;So you do the sum over dir.files[i] _plus_ what is in
data->file.size, what is indata->file.size? Zero for the dir case? Or some non-per-cpu metadata?
yes, saw the updated changelong, thanks
jirka
tip-bot for Jiri Olsa
merge bot[tip:perf/urgent] perf data: Make perf_data__size() work over directory
In reply to Jiri Olsaperf data: Make perf_data__size() work over directory
Commit-ID: 29583c17b5ce0bc17fdd80da939f8199a03d9668 Gitweb: https://git.kernel.org/tip/29583c17b5ce0bc17fdd80da939f8199a03d9668 Author: Jiri Olsa <[email protected]> AuthorDate: Fri, 8 Mar 2019 14:47:38 +0100 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Mon, 11 Mar 2019 11:56:03 -0300
perf data: Make perf_data__size() work over directory
Make perf_data__size() return proper size for directory data, summing up all the individual file sizes.
--- tools/perf/util/data.c | 17 +++++++++++++++++ tools/perf/util/data.h | 6 +----- 2 files changed, 18 insertions(+), 5 deletions(-)
tools/perf/util/data.c +17 -0
@@ -393,3 +393,20 @@ out:free (new_filepath);return ret; } + +unsigned long perf_data__size (struct perf_data *data) +{ +u64 size = data->file.size; +int i; + +if (!data->is_dir) +return size; + +for (i =0 ; i < data->dir.nr; i++) { +struct perf_data_file *file = &data->dir.files[i]; + + size += file->size; + } + +return size; +}
tools/perf/util/data.h +1 -5
@@ -54,11 +54,6 @@ static inline int perf_data__fd(struct perf_data *data)return data->file.fd; } -static inline unsigned long perf_data__size (struct perf_data *data) -{ -return data->file.size; -} -int perf_data__open (struct perf_data *data);void perf_data__close (struct perf_data *data); ssize_t perf_data__write(struct perf_data *data,
@@ -80,4 +75,5 @@ int perf_data__create_dir(struct perf_data *data, int nr);int perf_data__open_dir (struct perf_data *data);void perf_data__close_dir (struct perf_data *data);int perf_data__update_dir (struct perf_data *data); +unsigned long perf_data__size (struct perf_data *data);#endif /* __PERF_DATA_H */
Jiri Olsa
reviewerPATCH v5 5/11 perf header: Add DIR_FORMAT feature to describe directory data
In reply to Jiri OlsaThe data files layout is described by HEADER_DIR_FORMAT feature. Currently it holds only version number (1):
uint64_t version;
The current version holds only version value (1) means that data files:
- follow the 'data.*' name format
- contain raw events data in standard perf format as read from kernel (and need to be sorted)
Future versions are expected to describe different data files layout according to special needs.
--- tools/perf/builtin-record.c | 2 ++ tools/perf/util/data.c | 10 +++++++-- tools/perf/util/data.h | 1 + tools/perf/util/header.c | 44 ++++++++++++++++++++++++++++++++++++- tools/perf/util/header.h | 5 +++++ 5 files changed, 59 insertions(+), 3 deletions(-)
tools/perf/builtin-record.c +2 -0
@@ -837,6 +837,8 @@ static void record__init_features(struct record *rec)if (!(rec->opts.use_clockid && rec->opts.clockid_res_ns))perf_header__clear_feat (&session->header,HEADER_CLOCKID ); +perf_header__clear_feat (&session->header,HEADER_DIR_FORMAT ); +perf_header__clear_feat (&session->header,HEADER_STAT ); }
tools/perf/util/data.c +8 -2
@@ -14,6 +14,7 @@#include "data.h" #include "util.h" #include "debug.h" +#include "header.h" static void close_dir (struct perf_data_file *files,int nr) {
@@ -41,8 +42,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)if (!files)return -ENOMEM ; - data->dir.files = files; - data->dir.nr = nr; + data->dir.version =PERF_DIR_VERSION ; + data->dir.files = files; + data->dir.nr = nr;for (i =0 ; i < nr; i++) {struct perf_data_file *file = &files[i];
@@ -75,6 +77,10 @@ int perf_data__open_dir(struct perf_data *data)if (WARN_ON (!data->is_dir))return -EINVAL ; +/* The version is provided by DIR_FORMAT feature. */ +if (WARN_ON (data->dir.version !=PERF_DIR_VERSION )) +return -1 ; + dir =opendir (data->path);if (!dir)return -EINVAL ;
tools/perf/util/data.h +1 -0
@@ -24,6 +24,7 @@ struct perf_data {enum perf_data_mode mode;struct { +u64 version;struct perf_data_file *files;int nr; } dir;
tools/perf/util/header.c +43 -1
68 lines not on this page · raw message
tools/perf/util/header.h +5 -0
@@ -39,6 +39,7 @@ enum {HEADER_SAMPLE_TIME ,HEADER_MEM_TOPOLOGY ,HEADER_CLOCKID , +HEADER_DIR_FORMAT ,HEADER_LAST_FEATURE , HEADER_FEAT_BITS = 256, };
@@ -48,6 +49,10 @@ enum perf_header_version { PERF_HEADER_VERSION_2, }; +enum perf_dir_version { +PERF_DIR_VERSION =1 , +}; +struct perf_file_section {u64 offset;u64 size;
Signature · 1 line
2.17.2
Re: [PATCH 05/11] perf header: Add DIR_FORMAT feature to describe directory data
Em Fri, Mar 08, 2019 at 02:47:39PM +0100, Jiri Olsa escreveu:
Quoted 10 lines
The data files layout is described by
HEADER_DIR_FORMATfeature. Currently it holds only version number (1):uint64_t version;The current version holds only version value (1) means that data files:
- follow the 'data.*' name format
- contain raw events data in standard perf format as read from kernel (and need to be sorted)
and need to be sorted when multiple files are processed, right?
- Arnaldo
Quoted 174 lines
Future versions are expected to describe different data files layout according to special needs.
Signed-off-by Jiri Olsa <[email protected]>--- tools/perf/builtin-record.c | 2 ++ tools/perf/util/data.c | 10 +++++++-- tools/perf/util/data.h | 1 + tools/perf/util/header.c | 44 ++++++++++++++++++++++++++++++++++++- tools/perf/util/header.h | 5 +++++ 5 files changed, 59 insertions(+), 3 deletions(-)tools/perf/builtin-record.c +2
-0@@ -837,6 +837,8 @@ static void record__init_features(struct record *rec)if (!(rec->opts.use_clockid && rec->opts.clockid_res_ns))perf_header__clear_feat (&session->header,HEADER_CLOCKID ); +perf_header__clear_feat (&session->header,HEADER_DIR_FORMAT ); +perf_header__clear_feat (&session->header,HEADER_STAT ); }tools/perf/util/data.c +8
-2@@ -14,6 +14,7 @@#include "data.h" #include "util.h" #include "debug.h" +#include "header.h" static void close_dir (struct perf_data_file *files,int nr) {@@ -41,8 +42,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)if (!files)return -ENOMEM ; - data->dir.files = files; - data->dir.nr = nr; + data->dir.version =PERF_DIR_VERSION ; + data->dir.files = files; + data->dir.nr = nr;for (i =0 ; i < nr; i++) {struct perf_data_file *file = &files[i];@@ -75,6 +77,10 @@ int perf_data__open_dir(struct perf_data *data)if (WARN_ON (!data->is_dir))return -EINVAL ; +/* The version is provided by DIR_FORMAT feature. */ +if (WARN_ON (data->dir.version !=PERF_DIR_VERSION )) +return -1 ; + dir =opendir (data->path);if (!dir)return -EINVAL ;tools/perf/util/data.h +1
-0@@ -24,6 +24,7 @@ struct perf_data {enum perf_data_mode mode;struct { +u64 version;struct perf_data_file *files;int nr; } dir;tools/perf/util/header.c +43
-1@@ -861,6 +861,21 @@ static int write_clockid(struct feat_fd *ff, sizeof(ff->ph->env.clockid_res_ns)); } +static int write_dir_format (struct feat_fd *ff, +struct perf_evlist *evlist__maybe_unused ) +{ +struct perf_session *session; +struct perf_data *data; + + session = container_of(ff->ph, struct perf_session, header); + data = session->data; + +if (WARN_ON (!perf_data__is_dir (data))) +return -1 ; + +return do_write (ff, &data->dir.version,sizeof (data->dir.version)); +} +static int cpu_cache_level__sort (const void *a,const void *b) {struct cpu_cache_level *cache_a = (struct cpu_cache_level *)a;@@ -1341,6 +1356,17 @@ static void print_clockid(struct feat_fd *ff, FILE *fp) ff->ph->env.clockid_res_ns * 1000); } +static void print_dir_format (struct feat_fd *ff,FILE *fp) +{ +struct perf_session *session; +struct perf_data *data; + + session = container_of(ff->ph, struct perf_session, header); + data = session->data; + +fprintf (fp,"# directory data version : %" PRIu64"\n" , data->dir.version); +} +static void free_event_desc (struct perf_evsel *events) {struct perf_evsel *evsel;@@ -2373,6 +2399,21 @@ static int process_clockid(struct feat_fd *ff,return 0 ; } +static int process_dir_format (struct feat_fd *ff, +void *_data__maybe_unused ) +{ +struct perf_session *session; +struct perf_data *data; + + session = container_of(ff->ph, struct perf_session, header); + data = session->data; + +if (WARN_ON (!perf_data__is_dir (data))) +return -1 ; + +return do_read_u64 (ff, &data->dir.version); +} +struct feature_ops {int (*write)(struct feat_fd *ff,struct perf_evlist *evlist);void (*print)(struct feat_fd *ff,FILE *fp);@@ -2432,7 +2473,8 @@ static const struct feature_ops feat_ops[HEADER_LAST_FEATURE] = {FEAT_OPN (CACHE , cache, true),FEAT_OPR (SAMPLE_TIME , sample_time, false),FEAT_OPR (MEM_TOPOLOGY , mem_topology, true), -FEAT_OPR (CLOCKID , clockid, false) +FEAT_OPR (CLOCKID , clockid, false), +FEAT_OPN (DIR_FORMAT , dir_format, false) }; struct header_print_data {tools/perf/util/header.h +5
-0@@ -39,6 +39,7 @@ enum {HEADER_SAMPLE_TIME ,HEADER_MEM_TOPOLOGY ,HEADER_CLOCKID , +HEADER_DIR_FORMAT ,HEADER_LAST_FEATURE , HEADER_FEAT_BITS = 256, };@@ -48,6 +49,10 @@ enum perf_header_version { PERF_HEADER_VERSION_2, }; +enum perf_dir_version { +PERF_DIR_VERSION =1 , +}; +struct perf_file_section {u64 offset;u64 size;Signature · 1 line
2.17.2
Signature · 1 line
- Arnaldo
Re: [PATCH 05/11] perf header: Add DIR_FORMAT feature to describe directory data
Quoted 6 lines · Arnaldo Carvalho de Melo
Em Fri, Mar 08, 2019 at 02:47:39PM +0100, Jiri Olsa escreveu:
Quoted 5 lines
The data files layout is described by
HEADER_DIR_FORMATfeature. Currently it holds only version number (1):uint64_t version;
Quoted 7 lines
Quoted 5 lines
The current version holds only version value (1) means that data files:
- follow the 'data.*' name format
- contain raw events data in standard perf format as read from kernel (and need to be sorted)and need to be sorted when multiple files are processed, right?
correct
jirka
Quoted 181 lines
- Arnaldo
Quoted 174 lines
Future versions are expected to describe different data files layout according to special needs.
Signed-off-by Jiri Olsa <[email protected]>--- tools/perf/builtin-record.c | 2 ++ tools/perf/util/data.c | 10 +++++++-- tools/perf/util/data.h | 1 + tools/perf/util/header.c | 44 ++++++++++++++++++++++++++++++++++++- tools/perf/util/header.h | 5 +++++ 5 files changed, 59 insertions(+), 3 deletions(-)tools/perf/builtin-record.c +2
-0@@ -837,6 +837,8 @@ static void record__init_features(struct record *rec)if (!(rec->opts.use_clockid && rec->opts.clockid_res_ns))perf_header__clear_feat (&session->header,HEADER_CLOCKID ); +perf_header__clear_feat (&session->header,HEADER_DIR_FORMAT ); +perf_header__clear_feat (&session->header,HEADER_STAT ); }tools/perf/util/data.c +8
-2@@ -14,6 +14,7 @@#include "data.h" #include "util.h" #include "debug.h" +#include "header.h" static void close_dir (struct perf_data_file *files,int nr) {@@ -41,8 +42,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)if (!files)return -ENOMEM ; - data->dir.files = files; - data->dir.nr = nr; + data->dir.version =PERF_DIR_VERSION ; + data->dir.files = files; + data->dir.nr = nr;for (i =0 ; i < nr; i++) {struct perf_data_file *file = &files[i];@@ -75,6 +77,10 @@ int perf_data__open_dir(struct perf_data *data)if (WARN_ON (!data->is_dir))return -EINVAL ; +/* The version is provided by DIR_FORMAT feature. */ +if (WARN_ON (data->dir.version !=PERF_DIR_VERSION )) +return -1 ; + dir =opendir (data->path);if (!dir)return -EINVAL ;tools/perf/util/data.h +1
-0@@ -24,6 +24,7 @@ struct perf_data {enum perf_data_mode mode;struct { +u64 version;struct perf_data_file *files;int nr; } dir;tools/perf/util/header.c +43
-1@@ -861,6 +861,21 @@ static int write_clockid(struct feat_fd *ff, sizeof(ff->ph->env.clockid_res_ns)); } +static int write_dir_format (struct feat_fd *ff, +struct perf_evlist *evlist__maybe_unused ) +{ +struct perf_session *session; +struct perf_data *data; + + session = container_of(ff->ph, struct perf_session, header); + data = session->data; + +if (WARN_ON (!perf_data__is_dir (data))) +return -1 ; + +return do_write (ff, &data->dir.version,sizeof (data->dir.version)); +} +static int cpu_cache_level__sort (const void *a,const void *b) {struct cpu_cache_level *cache_a = (struct cpu_cache_level *)a;@@ -1341,6 +1356,17 @@ static void print_clockid(struct feat_fd *ff, FILE *fp) ff->ph->env.clockid_res_ns * 1000); } +static void print_dir_format (struct feat_fd *ff,FILE *fp) +{ +struct perf_session *session; +struct perf_data *data; + + session = container_of(ff->ph, struct perf_session, header); + data = session->data; + +fprintf (fp,"# directory data version : %" PRIu64"\n" , data->dir.version); +} +static void free_event_desc (struct perf_evsel *events) {struct perf_evsel *evsel;@@ -2373,6 +2399,21 @@ static int process_clockid(struct feat_fd *ff,return 0 ; } +static int process_dir_format (struct feat_fd *ff, +void *_data__maybe_unused ) +{ +struct perf_session *session; +struct perf_data *data; + + session = container_of(ff->ph, struct perf_session, header); + data = session->data; + +if (WARN_ON (!perf_data__is_dir (data))) +return -1 ; + +return do_read_u64 (ff, &data->dir.version); +} +struct feature_ops {int (*write)(struct feat_fd *ff,struct perf_evlist *evlist);void (*print)(struct feat_fd *ff,FILE *fp);@@ -2432,7 +2473,8 @@ static const struct feature_ops feat_ops[HEADER_LAST_FEATURE] = {FEAT_OPN (CACHE , cache, true),FEAT_OPR (SAMPLE_TIME , sample_time, false),FEAT_OPR (MEM_TOPOLOGY , mem_topology, true), -FEAT_OPR (CLOCKID , clockid, false) +FEAT_OPR (CLOCKID , clockid, false), +FEAT_OPN (DIR_FORMAT , dir_format, false) }; struct header_print_data {tools/perf/util/header.h +5
-0@@ -39,6 +39,7 @@ enum {HEADER_SAMPLE_TIME ,HEADER_MEM_TOPOLOGY ,HEADER_CLOCKID , +HEADER_DIR_FORMAT ,HEADER_LAST_FEATURE , HEADER_FEAT_BITS = 256, };@@ -48,6 +49,10 @@ enum perf_header_version { PERF_HEADER_VERSION_2, }; +enum perf_dir_version { +PERF_DIR_VERSION =1 , +}; +struct perf_file_section {u64 offset;u64 size;Signature · 1 line
2.17.2Signature · 1 line
- Arnaldo
tip-bot for Jiri Olsa
merge bot[tip:perf/urgent] perf header: Add DIR_FORMAT feature to describe directory data
In reply to Jiri Olsaperf header: Add DIR_FORMAT feature to describe directory data
Commit-ID: 258031c017c353e899902342a25579fc81a34cc1 Gitweb: https://git.kernel.org/tip/258031c017c353e899902342a25579fc81a34cc1 Author: Jiri Olsa <[email protected]> AuthorDate: Fri, 8 Mar 2019 14:47:39 +0100 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Mon, 11 Mar 2019 11:56:03 -0300
perf header: Add DIR_FORMAT feature to describe directory data
The data files layout is described by HEADER_DIR_FORMAT feature. Currently it holds only version number (1):
uint64_t version;
The current version holds only version value (1) means that data files:
- Follow the 'data.*' name format.
- Contain raw events data in standard perf format as read from kernel (and need to be sorted)
Future versions are expected to describe different data files layout according to special needs.
--- tools/perf/builtin-record.c | 2 ++ tools/perf/util/data.c | 10 ++++++++-- tools/perf/util/data.h | 1 + tools/perf/util/header.c | 44 +++++++++++++++++++++++++++++++++++++++++++- tools/perf/util/header.h | 5 +++++ 5 files changed, 59 insertions(+), 3 deletions(-)
tools/perf/builtin-record.c +2 -0
@@ -837,6 +837,8 @@ static void record__init_features(struct record *rec)if (!(rec->opts.use_clockid && rec->opts.clockid_res_ns))perf_header__clear_feat (&session->header,HEADER_CLOCKID ); +perf_header__clear_feat (&session->header,HEADER_DIR_FORMAT ); +perf_header__clear_feat (&session->header,HEADER_STAT ); }
tools/perf/util/data.c +8 -2
@@ -14,6 +14,7 @@#include "data.h" #include "util.h" #include "debug.h" +#include "header.h" static void close_dir (struct perf_data_file *files,int nr) {
@@ -41,8 +42,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)if (!files)return -ENOMEM ; - data->dir.files = files; - data->dir.nr = nr; + data->dir.version =PERF_DIR_VERSION ; + data->dir.files = files; + data->dir.nr = nr;for (i =0 ; i < nr; i++) {struct perf_data_file *file = &files[i];
@@ -75,6 +77,10 @@ int perf_data__open_dir(struct perf_data *data)if (WARN_ON (!data->is_dir))return -EINVAL ; +/* The version is provided by DIR_FORMAT feature. */ +if (WARN_ON (data->dir.version !=PERF_DIR_VERSION )) +return -1 ; + dir =opendir (data->path);if (!dir)return -EINVAL ;
tools/perf/util/data.h +1 -0
@@ -24,6 +24,7 @@ struct perf_data {enum perf_data_mode mode;struct { +u64 version;struct perf_data_file *files;int nr; } dir;
tools/perf/util/header.c +43 -1
68 lines not on this page · raw message
tools/perf/util/header.h +5 -0
@@ -39,6 +39,7 @@ enum {HEADER_SAMPLE_TIME ,HEADER_MEM_TOPOLOGY ,HEADER_CLOCKID , +HEADER_DIR_FORMAT ,HEADER_LAST_FEATURE , HEADER_FEAT_BITS = 256, };
@@ -48,6 +49,10 @@ enum perf_header_version { PERF_HEADER_VERSION_2, }; +enum perf_dir_version { +PERF_DIR_VERSION =1 , +}; +struct perf_file_section {u64 offset;u64 size;
Jiri Olsa
reviewerPATCH v5 6/11 perf session: Add process callback to reader object
In reply to Jiri OlsaAdding callback function to reader object so
callers can process data in different ways.
--- tools/perf/util/session.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)
tools/perf/util/session.c +19 -4
@@ -1847,10 +1847,17 @@ fetch_mmaped_event(struct perf_session *session,#define NUM_MMAPS 128#endif +struct reader ; + +typedef s64 (*reader_cb_t )(struct perf_session *session, +union perf_event *event, +u64 file_offset); +struct reader { -int fd; -u64 data_size; -u64 data_offset; +int fd; +u64 data_size; +u64 data_offset; +reader_cb_t process; }; static int
@@ -1921,7 +1928,7 @@ reader__process_events(struct reader *rd, struct perf_session *session, size = event->header.size;if (size <sizeof (struct perf_event_header ) || - (skip =perf_session__process_event (session, event, file_pos)) <0 ) { + (skip = rd->process (session, event, file_pos)) <0 ) {pr_err ("%#" PRIx64" [%#x]: failed to process type: %d\n" , file_offset + head, event->header.size, event->header.type);
@@ -1947,12 +1954,20 @@ reader__process_events(struct reader *rd, struct perf_session *session,return err; } +static s64 process_simple (struct perf_session *session, +union perf_event *event, +u64 file_offset) +{ +return perf_session__process_event (session, event, file_offset); +} +static int __perf_session__process_events (struct perf_session *session) {struct reader rd = { .fd =perf_data__fd (session->data), .data_size = session->header.data_size, .data_offset = session->header.data_offset, + .process = process_simple, };struct ordered_events *oe = &session->ordered_events;struct perf_tool *tool = session->tool;
Signature · 1 line
2.17.2
tip-bot for Jiri Olsa
merge bot[tip:perf/urgent] perf session: Add process callback to reader object
perf session: Add process callback to reader object
Commit-ID: e51f806198306a8ad7ae6e34d1af0716ef73da80 Gitweb: https://git.kernel.org/tip/e51f806198306a8ad7ae6e34d1af0716ef73da80 Author: Jiri Olsa <[email protected]> AuthorDate: Fri, 8 Mar 2019 14:47:40 +0100 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Mon, 11 Mar 2019 11:56:03 -0300
perf session: Add process callback to reader object
Adding callback function to reader object so callers can process data in different ways.
--- tools/perf/util/session.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)
tools/perf/util/session.c +19 -4
@@ -1847,10 +1847,17 @@ fetch_mmaped_event(struct perf_session *session,#define NUM_MMAPS 128#endif +struct reader ; + +typedef s64 (*reader_cb_t )(struct perf_session *session, +union perf_event *event, +u64 file_offset); +struct reader { -int fd; -u64 data_size; -u64 data_offset; +int fd; +u64 data_size; +u64 data_offset; +reader_cb_t process; }; static int
@@ -1921,7 +1928,7 @@ more: size = event->header.size;if (size <sizeof (struct perf_event_header ) || - (skip =perf_session__process_event (session, event, file_pos)) <0 ) { + (skip = rd->process (session, event, file_pos)) <0 ) {pr_err ("%#" PRIx64" [%#x]: failed to process type: %d\n" , file_offset + head, event->header.size, event->header.type);
@@ -1947,12 +1954,20 @@ out:return err; } +static s64 process_simple (struct perf_session *session, +union perf_event *event, +u64 file_offset) +{ +return perf_session__process_event (session, event, file_offset); +} +static int __perf_session__process_events (struct perf_session *session) {struct reader rd = { .fd =perf_data__fd (session->data), .data_size = session->header.data_size, .data_offset = session->header.data_offset, + .process = process_simple, };struct ordered_events *oe = &session->ordered_events;struct perf_tool *tool = session->tool;
Jiri Olsa
reviewerPATCH v5 7/11 perf session: Add __perf_session__process_dir_events function
In reply to Jiri OlsaAdding __perf_session__process_dir_events function to process events over the directory data.
All directory events are pushed into sessions ordered data and flushed for processing.
--- tools/perf/util/session.c | 88 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 2 deletions(-)
tools/perf/util/session.c +86 -2
@@ -1876,8 +1876,6 @@ reader__process_events(struct reader *rd, struct perf_session *session, file_offset = page_offset; head = rd->data_offset - page_offset; -ui_progress__init_size (prog, data_size,"Processing events..." ); - data_size += rd->data_offset; mmap_size =MMAP_SIZE ;
@@ -2006,6 +2004,89 @@ static int __perf_session__process_events(struct perf_session *session)return err; } +static s64 process_index (struct perf_session *session, +union perf_event *event, +u64 file_offset) +{ +struct perf_evlist *evlist = session->evlist; +u64 timestamp; +s64 ret; + +if (session->header.needs_swap) +event_swap (event,perf_evlist__sample_id_all (evlist)); + +if (event->header.type >=PERF_RECORD_HEADER_MAX ) +return -EINVAL ; + +events_stats__inc (&evlist->stats, event->header.type); + +if (event->header.type >=PERF_RECORD_USER_TYPE_START ) +return perf_session__process_user_event (session, event, file_offset); + + ret =perf_evlist__parse_sample_timestamp (evlist, event, ×tamp); +if (ret) +return ret; + +return ordered_events__queue (&session->ordered_events, event, + timestamp, file_offset); +} + +static int __perf_session__process_dir_events (struct perf_session *session) +{ +struct perf_data *data = session->data; +struct perf_tool *tool = session->tool; +struct reader rd = { + .fd =perf_data__fd (session->data), + .data_size = session->header.data_size, + .data_offset = session->header.data_offset, + .process = process_simple, + }; +int i, ret =0 ; +struct ui_progress prog; +u64 total_size =perf_data__size (session->data); + +perf_tool__fill_defaults (tool); + +ui_progress__init_size (&prog, total_size,"Processing events..." ); + +/* Read data from the header file.. */ + ret =reader__process_events (&rd, session, &prog); +if (ret) + goto out_err; + +/* ... and continue with data files. */ +for (i =0 ; i < data->dir.nr ; i++) { +struct perf_data_file *file = &data->dir.files[i]; + +if (file->size ==0 ) +continue ; + + rd = (struct reader ) { + .fd = file->fd, + .data_size = file->size, + .data_offset =0 , + .process = process_index, + }; + + ret =reader__process_events (&rd, session, &prog); +if (ret) + goto out_err; + } + + ret =ordered_events__flush (&session->ordered_events,OE_FLUSH__FINAL ); + +out_err: +if (!tool->no_warn) +perf_session__warn_about_errors (session); + +/* +* We may switching perf.data output, make ordered_events +* reusable. +*/ +ordered_events__reinit (&session->ordered_events); +return ret; +} +int perf_session__process_events (struct perf_session *session) {if (perf_session__register_idle_thread (session) <0 )
@@ -2014,6 +2095,9 @@ int perf_session__process_events(struct perf_session *session)if (perf_data__is_pipe (session->data))return __perf_session__process_pipe_events (session); +if (perf_data__is_dir (session->data)) +return __perf_session__process_dir_events (session); +return __perf_session__process_events (session); }
Signature · 1 line
2.17.2
Re: [PATCH 07/11] perf session: Add __perf_session__process_dir_events function
Em Fri, Mar 08, 2019 at 02:47:41PM +0100, Jiri Olsa escreveu:
Quoted 86 lines
Adding
__perf_session__process_dir_eventsfunction to process events over the directory data.All directory events are pushed into sessions ordered data and flushed for processing.
Signed-off-by Jiri Olsa <[email protected]>--- tools/perf/util/session.c | 88 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 2 deletions(-)tools/perf/util/session.c +57
-2@@ -1876,8 +1876,6 @@ reader__process_events(struct reader *rd, struct perf_session *session, file_offset = page_offset; head = rd->data_offset - page_offset; -ui_progress__init_size (prog, data_size,"Processing events..." ); - data_size += rd->data_offset; mmap_size =MMAP_SIZE ;@@ -2006,6 +2004,89 @@ static int __perf_session__process_events(struct perf_session *session)return err; } +static s64 process_index (struct perf_session *session, +union perf_event *event, +u64 file_offset) +{ +struct perf_evlist *evlist = session->evlist; +u64 timestamp; +s64 ret; + +if (session->header.needs_swap) +event_swap (event,perf_evlist__sample_id_all (evlist)); + +if (event->header.type >=PERF_RECORD_HEADER_MAX ) +return -EINVAL ; + +events_stats__inc (&evlist->stats, event->header.type); + +if (event->header.type >=PERF_RECORD_USER_TYPE_START ) +return perf_session__process_user_event (session, event, file_offset); + + ret =perf_evlist__parse_sample_timestamp (evlist, event, ×tamp); +if (ret) +return ret; + +return ordered_events__queue (&session->ordered_events, event, + timestamp, file_offset); +} + +static int __perf_session__process_dir_events (struct perf_session *session) +{ +struct perf_data *data = session->data; +struct perf_tool *tool = session->tool; +struct reader rd = { + .fd =perf_data__fd (session->data), + .data_size = session->header.data_size, + .data_offset = session->header.data_offset, + .process = process_simple, + }; +int i, ret =0 ; +struct ui_progress prog; +u64 total_size =perf_data__size (session->data); + +perf_tool__fill_defaults (tool); + +ui_progress__init_size (&prog, total_size,"Processing events..." ); + +/* Read data from the header file.. */ + ret =reader__process_events (&rd, session, &prog); +if (ret) + goto out_err; + +/* ... and continue with data files. */ +for (i =0 ; i < data->dir.nr ; i++) { +struct perf_data_file *file = &data->dir.files[i]; + +if (file->size ==0 ) +continue ; +
tools/perf/util/session.c static int __perf_session__process_events(struct perf_session *session) · line 2073
Quoted 10 lines
tools/perf/util/session.c +10
-0+ rd = (struct reader ) { + .fd = file->fd, + .data_size = file->size, + .data_offset =0 , + .process = process_index, + }; + + ret =reader__process_events (&rd, session, &prog); +if (ret) + goto out_err;
Don't we have to have some handling of PERF_RECORD_FINISHED_ROUND here? I.e. what happens if we fill th ordered events with just the contents of, say, the first CPU and then have those events flushed and processed before we start even looking at the events in the other CPUs?
I think some detailed explanation of what happens here is in need, no?
- Arnaldo
Quoted 31 lines
tools/perf/util/session.c +19
-0+ } + + ret =ordered_events__flush (&session->ordered_events,OE_FLUSH__FINAL ); + +out_err: +if (!tool->no_warn) +perf_session__warn_about_errors (session); + +/* +* We may switching perf.data output, make ordered_events +* reusable. +*/ +ordered_events__reinit (&session->ordered_events); +return ret; +} +int perf_session__process_events (struct perf_session *session) {if (perf_session__register_idle_thread (session) <0 )@@ -2014,6 +2095,9 @@ int perf_session__process_events(struct perf_session *session)if (perf_data__is_pipe (session->data))return __perf_session__process_pipe_events (session); +if (perf_data__is_dir (session->data)) +return __perf_session__process_dir_events (session); +return __perf_session__process_events (session); }Signature · 1 line
2.17.2
Signature · 1 line
- Arnaldo
Re: [PATCH 07/11] perf session: Add __perf_session__process_dir_events function
Em Fri, Mar 08, 2019 at 03:38:05PM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 47 lines
Em Fri, Mar 08, 2019 at 02:47:41PM +0100, Jiri Olsa escreveu:
Quoted 30 lines
tools/perf/util/session.c +30
-0+static int __perf_session__process_dir_events (struct perf_session *session) +{ +struct perf_data *data = session->data; +struct perf_tool *tool = session->tool; +struct reader rd = { + .fd =perf_data__fd (session->data), + .data_size = session->header.data_size, + .data_offset = session->header.data_offset, + .process = process_simple, + }; +int i, ret =0 ; +struct ui_progress prog; +u64 total_size =perf_data__size (session->data); + +perf_tool__fill_defaults (tool); + +ui_progress__init_size (&prog, total_size,"Processing events..." ); + +/* Read data from the header file.. */ + ret =reader__process_events (&rd, session, &prog); +if (ret) + goto out_err; + +/* ... and continue with data files. */ +for (i =0 ; i < data->dir.nr ; i++) { +struct perf_data_file *file = &data->dir.files[i]; + +if (file->size ==0 ) +continue ; +tools/perf/util/session.c static int __perf_session__process_events(struct perf_session *session) · line 2073
Quoted 10 lines
tools/perf/util/session.c +10
-0+ rd = (struct reader ) { + .fd = file->fd, + .data_size = file->size, + .data_offset =0 , + .process = process_index, + }; + + ret =reader__process_events (&rd, session, &prog); +if (ret) + goto out_err;Don't we have to have some handling of
PERF_RECORD_FINISHED_ROUNDhere? I.e. what happens if we fill th ordered events with just the contents of, say, the first CPU and then have those events flushed and processed before we start even looking at the events in the other CPUs?
Quoted 1 line
I think some detailed explanation of what happens here is in need, no?
Ah, I've pushed what I have merged so far to acme/perf/core on git.kernel.org, please continue from there.
- Arnaldo
Re: [PATCH 07/11] perf session: Add __perf_session__process_dir_events function
In reply to Arnaldo Carvalho de MeloOn Fri, Mar 08, 2019 at 03:38:05PM -0300, Arnaldo Carvalho de Melo wrote:
SNIP
Quoted 19 lines
Quoted 19 lines
tools/perf/util/session.c +19
-0+/* Read data from the header file.. */ + ret =reader__process_events (&rd, session, &prog); +if (ret) + goto out_err; + +/* ... and continue with data files. */ +for (i =0 ; i < data->dir.nr ; i++) { +struct perf_data_file *file = &data->dir.files[i]; + +if (file->size ==0 ) +continue ; + + rd = (struct reader ) { + .fd = file->fd, + .data_size = file->size, + .data_offset =0 , + .process = process_index, + }; +
Quoted 8 lines
tools/perf/util/session.c static int __perf_session__process_events(struct perf_session *session) · line 2055
Quoted 3 lines
tools/perf/util/session.c +3
-0+ ret =reader__process_events (&rd, session, &prog); +if (ret) + goto out_err;Don't we have to have some handling of
PERF_RECORD_FINISHED_ROUNDhere? I.e. what happens if we fill th ordered events with just the contents of, say, the first CPU and then have those events flushed and processed before we start even looking at the events in the other CPUs?
PERF_RECORD_FINISHED_ROUND is disabled for directory data, because each file contains sorted cpu data, so we only merge them together
Quoted 2 lines
I think some detailed explanation of what happens here is in need, no?
right, I'll put the comment for the function
jirka
Jiri Olsa
reviewerPATCH v5 8/11 perf session: Add path to reader object
In reply to Jiri OlsaAdding path to reader object, so we can display file the processing fails for (string in [] brackets).
$ perf report --stdio 0x5e0 [perf.data/data.3] [0xa200]: failed to process type: -1577027574
--- tools/perf/util/session.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
tools/perf/util/session.c +8 -2
@@ -1858,6 +1858,7 @@ struct reader {u64 data_size;u64 data_offset;reader_cb_t process; +char *path; }; static int
@@ -1872,6 +1873,8 @@ reader__process_events(struct reader *rd, struct perf_session *session,union perf_event *event;s64 skip; +pr_debug ("reader processing %s\n" , rd->path); + page_offset = page_size * (rd->data_offset / page_size); file_offset = page_offset; head = rd->data_offset - page_offset;
@@ -1927,8 +1930,8 @@ reader__process_events(struct reader *rd, struct perf_session *session,if (size <sizeof (struct perf_event_header ) || (skip = rd->process (session, event, file_pos)) <0 ) { -pr_err ("%#" PRIx64" [%#x]: failed to process type: %d\n" , - file_offset + head, event->header.size, +pr_err ("%#" PRIx64" [%s] [%#x]: failed to process type: %d\n" , + file_offset + head, rd->path, event->header.size, event->header.type); err = -EINVAL ; goto out;
@@ -1966,6 +1969,7 @@ static int __perf_session__process_events(struct perf_session *session) .data_size = session->header.data_size, .data_offset = session->header.data_offset, .process = process_simple, + .path = session->data->file.path, };struct ordered_events *oe = &session->ordered_events;struct perf_tool *tool = session->tool;
@@ -2040,6 +2044,7 @@ static int __perf_session__process_dir_events(struct perf_session *session) .data_size = session->header.data_size, .data_offset = session->header.data_offset, .process = process_simple, + .path = session->data->file.path, };int i, ret =0 ;struct ui_progress prog;
@@ -2066,6 +2071,7 @@ static int __perf_session__process_dir_events(struct perf_session *session) .data_size = file->size, .data_offset = 0, .process = process_index, + .path = file->path, }; ret =reader__process_events (&rd, session, &prog);
Signature · 1 line
2.17.2
Jiri Olsa
reviewerPATCH v5 9/11 perf record: Add --dir option to store data in directory
In reply to Jiri OlsaAdding --dir option to store data in directory. It's next step for multiple threads in record. It's now possible to make directory data via --dir option, like:
$ perf record --dir perf bench sched messaging $ ls -l perf.data total 344 -rw-------. 1 jolsa jolsa 43864 Jan 20 22:26 data.0 -rw-------. 1 jolsa jolsa 30464 Jan 20 22:26 data.1 -rw-------. 1 jolsa jolsa 53816 Jan 20 22:26 data.2 -rw-------. 1 jolsa jolsa 30368 Jan 20 22:26 data.3 -rw-------. 1 jolsa jolsa 40088 Jan 20 22:26 data.4 -rw-------. 1 jolsa jolsa 42592 Jan 20 22:26 data.5 -rw-------. 1 jolsa jolsa 56136 Jan 20 22:26 data.6 -rw-------. 1 jolsa jolsa 25992 Jan 20 22:26 data.7 -rw-------. 1 jolsa jolsa 8832 Jan 20 22:26 header
There's a data file created for every cpu and it's storing data for those cpu maps.
It's possible to transform directory data into standard perf.data file via following inject command:
$ perf inject -o perf.data.file -i perf.data
The --dir option enabled DIR_FORMAT feature to be stored in header file to indicate the directory layout.
Don't allow to use --dir with --aio yet. It needs to be investigated first.
--- tools/perf/Documentation/perf-record.txt | 3 + tools/perf/builtin-record.c | 80 ++++++++++++++++++++++-- tools/perf/util/mmap.h | 23 +++---- 3 files changed, 90 insertions(+), 16 deletions(-)
tools/perf/Documentation/perf-record.txt +3 -0
@@ -524,6 +524,9 @@ config terms. For example: 'cycles/overwrite/' and 'instructions/no-overwrite/'. Implies --tail-synthesize. +--dir:: +Store data into directory with one data file for cpu. + SEE ALSO -------- linkperf:perf-stat[1], linkperf:perf-list[1]
tools/perf/builtin-record.c +75 -5
156 lines not on this page · raw message
tools/perf/util/mmap.h +12 -11
@@ -19,17 +19,18 @@ struct aiocb;* @refcnt - e.g. code using PERF_EVENT_IOC_SET_OUTPUT to share this */ struct perf_mmap { -void *base; -int mask; -int fd; -int cpu; -refcount_t refcnt; -u64 prev; -u64 start; -u64 end; -bool overwrite; -struct auxtrace_mmap auxtrace_mmap; -char event_copy[PERF_SAMPLE_MAX_SIZE ]__aligned(8) ; +void *base; +int mask; +int fd; +int cpu; +refcount_t refcnt; +u64 prev; +u64 start; +u64 end; +bool overwrite; +struct auxtrace_mmap auxtrace_mmap; +struct perf_data_file *file; +char event_copy[PERF_SAMPLE_MAX_SIZE ]__aligned(8) ; #ifdef HAVE_AIO_SUPPORT struct { void **data;
Signature · 1 line
2.17.2
Jiri Olsa
reviewerPATCH v5 10/11 perf record: Add --output-dir option to store data in directory
In reply to Jiri OlsaAdding --output-dir option to mimic -o and --dir options. following commands do the same:
$ perf record -o perf.dir.data --dir ... $ perf record --output-dir perf.dir.data ...
User cannot use both -o and output-dir together,
error is displayed.
--- tools/lib/subcmd/parse-options.h | 4 ++++ tools/perf/Documentation/perf-record.txt | 3 +++ tools/perf/builtin-record.c | 13 +++++++++++-- 3 files changed, 18 insertions(+), 2 deletions(-)
tools/lib/subcmd/parse-options.h +4 -0
@@ -146,6 +146,10 @@ struct option { .value = check_vtype(v, const char **), .argh = (a), .help = (h), \ .flags = PARSE_OPT_OPTARG, .defval = (intptr_t)(d), \ .set = check_vtype(os, bool *)} +#define OPT_STRING_SET(s, l, v, os, a, h) \ + { .type = OPTION_STRING, .short_name = (s), .long_name = (l), \ + .value = check_vtype(v, const char **), .argh = (a), .help = (h), \ + .set = check_vtype(os, bool *)} #define OPT_STRING_NOEMPTY(s, l, v, a, h) { .type = OPTION_STRING, .short_name = (s), .long_name = (l), .value = check_vtype(v, const char **), .argh = (a), .help = (h), .flags = PARSE_OPT_NOEMPTY} #define OPT_DATE(s, l, v, h) \ { .type = OPTION_CALLBACK, .short_name = (s), .long_name = (l), .value = (v), .argh = "time", .help = (h), .callback = parse_opt_approxidate_cb }
tools/perf/Documentation/perf-record.txt +3 -0
@@ -527,6 +527,9 @@ Implies --tail-synthesize. --dir:: Store data into directory with one data file for cpu. +--output-dir:: +Same as --dir option, can't be used together with -o option. + SEE ALSO -------- linkperf:perf-stat[1], linkperf:perf-list[1]
tools/perf/builtin-record.c +11 -2
@@ -80,6 +80,7 @@ struct record {bool buildid_all;bool timestamp_filename;bool timestamp_boundary; +bool output_is_file;struct switch_output switch_output;unsigned long long samples;cpu_set_t affinity_mask;
@@ -1921,8 +1922,10 @@ static struct option __record_options[] = {OPT_STRING ('C' ,"cpu" , &record.opts.target.cpu_list,"cpu" ,"list of cpus to monitor" ),OPT_U64 ('c' ,"count" , &record.opts.user_interval,"event period to sample" ), -OPT_STRING ('o' ,"output" , &record.data.path,"file" , -"output file name" ), +OPT_STRING_SET ('o' ,"output" , &record.data.path, &record.output_is_file, +"file" ,"output file name" ), +OPT_STRING_SET (0 ,"output-dir" , &record.data.path, &record.data.is_dir, +"file" ,"output directory name" ),OPT_BOOLEAN_SET ('i' ,"no-inherit" , &record.opts.no_inherit, &record.opts.no_inherit_set, "child tasks do not inherit counters"),
@@ -2101,6 +2104,12 @@ int cmd_record(int argc, const char **argv) "cgroup monitoring only available in system-wide mode"); } + +if (perf_data__is_dir (&rec->data) && record.output_is_file) { +ui__error ("cannot use both -o and --output-dir\n" ); +return -EINVAL ; + } +if (rec->opts.record_switch_events && !perf_can_record_switch_events ()) {ui__error ("kernel does not support recording context switch events\n" );
Signature · 1 line
2.17.2
Jiri Olsa
reviewerPATCH v5 11/11 perf record: Describe perf.data directory format
In reply to Jiri OlsaAdding perf.data-directory-format.txt to describe the directory data layout.
--- .../perf.data-directory-format.txt | 54 +++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 tools/perf/Documentation/perf.data-directory-format.txt
tools/perf/Documentation/perf.data-directory-format.txt new file +54 -0
@@ -0,0 +1,54 @@ +perf.data directory format + +DISCLAIMER This is not ABI yet and is subject to possible change + in following versions of perf. We will remove this + disclaimer once the directory format soaks in. + + +This document describes the on-disk perf.data format, generated +by perf record with --dir option and consumed by the other perf +tools. + +The directory perf.data is created by perf record command by +using the --dir option: + + $ perf record --dir perf bench sched messaging + $ ls -l perf.data + total 344 + -rw-------. 1 jolsa jolsa 43864 Jan 20 22:26 data.0 + -rw-------. 1 jolsa jolsa 30464 Jan 20 22:26 data.1 + -rw-------. 1 jolsa jolsa 53816 Jan 20 22:26 data.2 + -rw-------. 1 jolsa jolsa 30368 Jan 20 22:26 data.3 + -rw-------. 1 jolsa jolsa 40088 Jan 20 22:26 data.4 + -rw-------. 1 jolsa jolsa 42592 Jan 20 22:26 data.5 + -rw-------. 1 jolsa jolsa 56136 Jan 20 22:26 data.6 + -rw-------. 1 jolsa jolsa 25992 Jan 20 22:26 data.7 + -rw-------. 1 jolsa jolsa 8832 Jan 20 22:26 header + +The header file keeps the standard perf.data file header, +and the data.* files keep data. + +header file +----------- +The header file following the standard format describe in +Documentation/perf.data-file-format doc. Including its data +portion that is used to store manually synthesized events. + +data file +--------- +The data files layout is described by HEADER_DIR_FORMAT feature. +Currently it holds only version number (1): + + HEADER_DIR_FORMAT = 24 + + struct { + uint64_t version; + } + +The current only only version value 1 means that data files: + - follow the 'data.*' format + - contain raw events data in standard perf format as read + from kernel (and need to be sorted) + +Future versions are expected to describe different data files +layout according to special needs.
Signature · 1 line
2.17.2