Skip to contentSkip to conversation

Tmf Ctf Parser Scalability issues (1/3)

DISCUSSION Matthew Khouzam 2 participants 2 messages Nov 26, 2013

lore
Share & tools
lore
#0 lore raw

Hello all,
I was looking at the CTF parser in the TMF project of Linux Tools. I have come up with three points where scalability will be an issue. I am sending 3 emails, each one describing one of the issues so we can aggregate them more cohesively. First a primer, CTF is a file format that has traces written in packets, the packets are parts of files in streams, the streams are files in a directory.

Issue 1: Number of open files
CTF can have an unlimited amount of stream files. The parser opens all of them simultaneously. This means if you have say over 1024 or whatever your file limit is, files opened simultaneously, you will hit a wall. An example of this would be the new Xeon Phi boards that have something like 255 cores, if you have more than 4 channels per core, you're going to hit this issue.

Proposed solution:
I envision fixing them by setting up a pool of files and when the limit is hit, swapping files in and out, incurring a performance hit but still allowing the program to run. Any objections?

#1 lore raw

----- Original Message -----

Quoted 21 lines

From: "Matthew Khouzam" <[email protected]>
To: [email protected], "Linux Tools developer discussions" <[email protected]>
Sent: Tuesday, November 26, 2013 11:44:46 AM
Subject: [lttng-dev] Tmf Ctf Parser Scalability issues (1/3)

Hello all,
I was looking at the CTF parser in the TMF project of Linux Tools. I have come up with three points where scalability will be an issue. I am sending 3 emails, each one describing one of the issues so we can aggregate them more cohesively. First a primer, CTF is a file format that has traces written in packets, the packets are parts of files in streams, the streams are files in a directory.

Issue 1: Number of open files
CTF can have an unlimited amount of stream files. The parser opens all of them simultaneously. This means if you have say over 1024 or whatever your file limit is, files opened simultaneously, you will hit a wall. An example of this would be the new Xeon Phi boards that have something like 255 cores, if you have more than 4 channels per core, you're going to hit this issue.

Quoted 4 lines

Proposed solution:
I envision fixing them by setting up a pool of files and when the limit is hit, swapping files in and out, incurring a performance hit but still allowing the program to run. Any objections?

No objection. I would think a LRU list of most recently used files would be a good data structure for this. I already described this in a bug against babeltrace: http://bugs.lttng.org/issues/649

Thanks!

Mathieu

Quoted 6 lines

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com