Tmf Ctf Parser Scalability issues (1/3)
Hello all,
I was looking at the CTF parser in the TMF project of Linux Tools. I have come up with three points where scalability will be an issue. I am sending 3 emails, each one describing one of the issues so we can aggregate them more cohesively. First a primer, CTF is a file format that has traces written in packets, the packets are parts of files in streams, the streams are files in a directory.
Issue 1: Number of open files
CTF can have an unlimited amount of stream files. The parser opens all of them simultaneously. This means if you have say over 1024 or whatever your file limit is, files opened simultaneously, you will hit a wall. An example of this would be the new Xeon Phi boards that have something like 255 cores, if you have more than 4 channels per core, you're going to hit this issue.
Proposed solution:
I envision fixing them by setting up a pool of files and when the limit is hit, swapping files in and out, incurring a performance hit but still allowing the program to run. Any objections?
----- Original Message -----
Quoted 21 lines
From: "Matthew Khouzam" <[email protected]>
To: [email protected], "Linux Tools developer discussions" <[email protected]>
Sent: Tuesday, November 26, 2013 11:44:46 AM
Subject: [lttng-dev] Tmf Ctf Parser Scalability issues (1/3)Hello all,
I was looking at the CTF parser in the TMF project of Linux Tools. I have come up with three points where scalability will be an issue. I am sending 3 emails, each one describing one of the issues so we can aggregate them more cohesively. First a primer, CTF is a file format that has traces written in packets, the packets are parts of files in streams, the streams are files in a directory.Issue 1: Number of open files
CTF can have an unlimited amount of stream files. The parser opens all of them simultaneously. This means if you have say over 1024 or whatever your file limit is, files opened simultaneously, you will hit a wall. An example of this would be the new Xeon Phi boards that have something like 255 cores, if you have more than 4 channels per core, you're going to hit this issue.
Quoted 4 lines
Proposed solution:
I envision fixing them by setting up a pool of files and when the limit is hit, swapping files in and out, incurring a performance hit but still allowing the program to run. Any objections?
No objection. I would think a LRU list of most recently used files would be a good data structure for this. I already described this in a bug against babeltrace: http://bugs.lttng.org/issues/649
Thanks!
Mathieu
Quoted 6 lines
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com