Skip to contentSkip to conversation

Tmf Ctf Parser Scalability issues (3/3)

DISCUSSION Matthew Khouzam 2 participants 2 messages Nov 26, 2013

lore
Share & tools
lore
#0 lore raw

Hello all,
I was looking at the CTF parser in the TMF project of Linux Tools. I have come up with three points where scalability will be an issue. I am sending 3 emails, each one describing one of the issues so we can aggregate them more cohesively. First a primer, CTF is a file format that has traces written in packets, the packets are parts of files in streams, the streams are files in a directory.

Issue 3: Event size
As hinted in the previous email, it can definitely be a possibility that traces can have events that are too large to fit into memory. These events could also make reading a packet difficult, since the packet could have a said event in the middle, and therefore, we could end up with a packet that cannot be read properly with the sliding window memory map, unless corrective actions are taken.

Proposed solution:
Would anyone do this? I see it as a longer term problem and as a known limitation, unless someone says "Hey, I want 8gb events!".

#1 lore raw

----- Original Message -----

Quoted 21 lines

From: "Matthew Khouzam" <[email protected]>
To: [email protected], "Linux Tools developer discussions" <[email protected]>
Sent: Tuesday, November 26, 2013 11:44:51 AM
Subject: [lttng-dev] Tmf Ctf Parser Scalability issues (3/3)

Hello all,
I was looking at the CTF parser in the TMF project of Linux Tools. I have come up with three points where scalability will be an issue. I am sending 3 emails, each one describing one of the issues so we can aggregate them more cohesively. First a primer, CTF is a file format that has traces written in packets, the packets are parts of files in streams, the streams are files in a directory.

Issue 3: Event size
As hinted in the previous email, it can definitely be a possibility that traces can have events that are too large to fit into memory. These events could also make reading a packet difficult, since the packet could have a said event in the middle, and therefore, we could end up with a packet that cannot be read properly with the sliding window memory map, unless corrective actions are taken.

Quoted 3 lines

Proposed solution:
Would anyone do this? I see it as a longer term problem and as a known limitation, unless someone says "Hey, I want 8gb events!".

I'm tempted to say that we should put that limitation on individual non-compound field, rather than events. Therefore, you could not have such a large string, nor such a large integer, but you could very well have a very large array of integers within an event which would be 8gb in size.

Thanks,

Mathieu

Quoted 6 lines

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com