capture file timestamping
Hi all,
[Simon brought up a related topic a few years ago]
We have a field for TSFT/mactime, which is defined as follows:
Value in microseconds of the MAC's 64-bit 802.11 Time Synchronization Function timer when the first bit of the MPDU arrived at the MAC. For received frames only.
This is great, but not always feasible. Particularly when operating within multiple virtual interfaces and trying to record data from the hardware at the same time, the TSFT is either unobtainable or practically useless.
Similarly, in actual monitor mode, there might not be a TSF timer - just a base hardware timer that is essentially free-running (though with the same frequency.)
As Simon also found, the reference might not be the "first bit of the MPDU" (which I'd also say is actually somewhat vague with OFDM symbols), but something else entirely.
There's also something that I'm not sure about - in 5/10 MHz channels, as I understand this is often achieved by running hardware timers at different frequencies - does that affect the timestamps hardware reports here?
I'm not sure we'll be able to capture everything in a single value, but we could add a field that has a "type" that can be extended in the future.
Say, for example, we add a new field like this:
Structure: u64 timestamp, u32 flags
Alignment: 8
We'd probably need flags for
* TSF timer
* first bit/symbol of MPDU
(these two allow reproducing the existing TSFT field functionality, which might not be necessary, but each could still be useful)
* 32-bit only [our hardware only has an internal 32-bit timestamp]
* end-of-frame [maybe define this better?]
We could also go into the various other places of the frame where a timestamp can be taken - our hardware does it when it detects energy on the air.
The most common use for this would really be looking at monitor files, so for me personally simply having a more generic "timestamp" field that doesn't specify much of its semantics would be sufficient - however, I believe that to be very dangerous as somebody's semantics will eventually get hard-coded into wireshark and then it's fixed anyway ...
Anyone want to offer any thoughts?
johannes
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · Johannes Berg
Hi all,
[Simon brought up a related topic a few years ago]
We have a field for TSFT/mactime, which is defined as follows:
Quoted 8 lines
Value in microseconds of the MAC's 64-bit 802.11 Time Synchronization Function timer when the first bit of the MPDU arrived at the MAC. For received frames only.
This is great, but not always feasible. Particularly when operating within multiple virtual interfaces and trying to record data from the hardware at the same time, the TSFT is either unobtainable or practically useless.
Can you explain why it is unobtainable or practically useless?
Quoted 3 lines
Similarly, in actual monitor mode, there might not be a TSF timer - just a base hardware timer that is essentially free-running (though with the same frequency.)
I agree that it's not quite right to call it TSF time when there is not actually any synchronization of stations.
Quoted 3 lines
As Simon also found, the reference might not be the "first bit of the MPDU" (which I'd also say is actually somewhat vague with OFDM symbols), but something else entirely.
Hmm. What's the reference for beacons and probe responses?
Quoted 4 lines
There's also something that I'm not sure about - in 5/10 MHz channels, as I understand this is often achieved by running hardware timers at different frequencies - does that affect the timestamps hardware reports here?
Is it fair to say that you are generally concerned that not all hardware can stamp received frames with something like a TSF time, but some hardware can stamp frames with a time that is the "moral equivalent" for certain purposes?
What are the use cases for the timestamps? My use cases were these:
I. Measuring nanosecond intervals between several consecutive frames
A. Using the measurements to verify settings of device registers
B. Deriving from several intervals the meters distance
between two 802.11 stations
II. Using the timestamps to place graphic representations of frames
on a timeline
I can certainly imagine someone using hardware timestamps to put frames back into the order they were received if the order was scrambled by, for example, a multiprocessing network stack.
Dave
Signature · 2 lines
David Young [email protected] Urbana, IL (217) 721-9981
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 7 lines · David Young
Quoted 5 lines
This is great, but not always feasible. Particularly when operating within multiple virtual interfaces and trying to record data from the
hardware at the same time, the TSFT is either unobtainable or practically useless.Can you explain why it is unobtainable or practically useless?
If you have two stations, for example, you need to have synchronisation with the two different APs, so you can have two vastly different TSF values. Maybe your device gives those to you, or maybe not. It may make sense for some analysis to look at that, but if you need to look at timing inside the device to be evident then this is useless.
Quoted 6 lines
Quoted 3 lines
Similarly, in actual monitor mode, there might not be a TSF timer - just a base hardware timer that is essentially free-running (though with the same frequency.)
I agree that it's not quite right to call it TSF time when there is not actually any synchronization of stations.
It's not really a major defect though.
Quoted 5 lines
Quoted 3 lines
As Simon also found, the reference might not be the "first bit of the MPDU" (which I'd also say is actually somewhat vague with OFDM symbols), but something else entirely.
Hmm. What's the reference for beacons and probe responses?
First symbol containing the first bit of the timestamp field, I believe. Haven't double-checked now.
Quoted 4 lines
Is it fair to say that you are generally concerned that not all hardware can stamp received frames with something like a TSF time, but some hardware can stamp frames with a time that is the "moral equivalent" for certain purposes?
Yeah, you could put it that way.
Quoted 3 lines
What are the use cases for the timestamps? My use cases were these:
I. Measuring nanosecond intervals between several consecutive frames
I don't think I have ever seen nanoseconds, and we don't currently allow that resolution. However, it's a good point - any new field should probably take resolution into account.
Quoted 1 line
A. Using the measurements to verify settings of device registers
This could be done I guess.
Quoted 2 lines
B. Deriving from several intervals the meters distance
between two 802.11 stations
Possible, but I'm not sure it's all that feasible.
Quoted 2 lines
II. Using the timestamps to place graphic representations of frames
on a timeline
This is one.
One I ran into recently is that I needed to check the relative timings of packets to verify the backoff function was working correctly, which is really just very similar to your I-A case.
Quoted 3 lines
I can certainly imagine someone using hardware timestamps to put frames back into the order they were received if the order was scrambled by, for example, a multiprocessing network stack.
The timestamps added by the capture (e.g. tcpdump) are also notoriously unreliable since there are processing delays - even without reordering.
I was trying to be a bit more generic, but in my case it mostly boils down to this:
a) the timestamps recorded by (e.g.) tcpdump in the pcap packet headers are unreliable
b) the timestamps there are at the end of the frame, beginning is much more useful
c) the timer I'm getting is an internal hardware timer and I can't easily derive either a TSF-based value, nor easily put it at the first bit/symbol of the MPDU (since it's earlier) [*]
johannes
[*] it's actually also somewhat unreliable, due to the reference it's taken at, but that's something I can generally live with, it's in the order of a few microseconds, not the milliseconds I sometimes see with tcpdump
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 2 lines · Johannes Berg
The timestamps added by the capture (e.g. tcpdump) are also notoriously unreliable since there are processing delays - even without reordering.
If you have tcpdump 4.6 or later, with libpcap 1.6 or later, on Linux, you should have the --time-stamp-type and --list-time-stamp-types options.
If so, what does
tcpdump --list-time-stamp-types
print?
If it prints "adapter" or "adapter_unsynced", you might want to try those with the --time-stamp-type option, as those time stamp types mean that the time stamp will come from the adapter rather than from Linux. ("unsynced" means that the time stamps aren't synchronized with the host's clock.)
That won't address
Quoted 2 lines
b) the timestamps there are at the end of the frame, beginning is much more useful
unless that's how the adapter time-stamps the packet and won't address
Quoted 3 lines
c) the timer I'm getting is an internal hardware timer and I can't easily derive either a TSF-based value, nor easily put it at the first bit/symbol of the MPDU (since it's earlier) [*]
unless it's based on the TSF (which it probably won't be, as it should, at least in principle, be Epoch time, although I think Linux may have, at some point, changed adapter time stamps in a fashion making them not useful as libpcap time stamps).
(I.e., the "host time stamps are applied at times not well correlated with actual network events" problem is not best solved with radiotap, it's best solved by libpcap and the capture mechanism atop which it runs, but "I want a time stamp that comes from the TSF timers" is one best solved with radiotap.)
A message that is not in the archive (sent off-list or not yet received): 4E4FE63E-B636-4808-BBFF-6D2CE4EAFB2F-FrUbXkNCsVf2fBVCVOL8/[email protected]
Quoted 9 lines · Guy Harris
If you have tcpdump 4.6 or later, with libpcap 1.6 or later, on Linux, you should have the --time-stamp-type and --list-time-stamp -types options.
If so, what does
> tcpdump --list-time-stamp-types
Quoted 7 lines
print?
If it prints "adapter" or "
adapter_unsynced", you might want to try those with the --time-stamp-type option, as those time stamp types mean that the time stamp will come from the adapter rather than from Linux. ("unsynced" means that the time stamps aren't synchronized with the host's clock.)
It does have "adapter_unsynced", but wouldn't the adapter also have to support it? Ah, yes, if I do
tcpdump -i wlan0 --list-time-stamp-types
it no longer shows it.
Although perhaps I could possibly convince the driver to report this timestamp somehow, rather than changing radiotap.
Quoted 4 lines
unless it's based on the TSF (which it probably won't be, as it should, at least in principle, be Epoch time, although I think Linux may have, at some point, changed adapter time stamps in a fashion making them not useful as libpcap time stamps).
That, however, I almost certainly cannot do - I don't see how I'd synchronize from the adapter time (a free-running 32-bit counter at microseconds resolution) to the host time.
johannes
Quoted 10 lines · Johannes Berg
Quoted 3 lines
II. Using the timestamps to place graphic representations of frames
on a timelineThis is one.
One I ran into recently is that I needed to check the relative timings of packets to verify the backoff function was working correctly, which is really just very similar to your I-A case.
This is exactly what I was doing when I ran into the current TSFT field limitations a few years ago. GPL code is here:
https://github.com/parc-wifi/wireshark
I'm working on getting this updated for 11ac now, so revisiting some of the same issues.
The HT and VHT headers in radiotap are incomplete, and would be much nicer if they could just include the signal field from the plcp (perhaps with a mask to indicate parts that are missing). This would be a better approach for 60GHz I think.
I'd like to display the L-SIG txop protection on the timeline as well.
Simon
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · Simon Barber
This is exactly what I was doing when I ran into the current TSFT field limitations a few years ago.
Right, I vaguely remembered that, hence the CC :)
Quoted 6 lines
GPL code is here:
https://github.com/parc-wifi/wireshark
I'm working on getting this updated for 11ac now, so revisiting some of the same issues.
Ah, that's really nice! I really needed something like this the other day, will check it out.
Quoted 4 lines
The HT and VHT headers in radiotap are incomplete, and would be much nicer if they could just include the signal field from the plcp (perhaps with a mask to indicate parts that are missing). This would be a better approach for 60GHz I think.
Yeah, can't really object to that, even if we might not be able to provide all the data for all the hardware, and might have to re -assemble the field from the available parsed data.
As far as our device is concerned we might perhaps be able to include this data in firmware's monitor mode.
Quoted 3 lines
I'd like to display the L-SIG txop protection on the timeline as well.
Good idea.
As far as the timestamp is concerned though, what would you suggest?
Also, what timestamp would be best for a tool like yours? I'm assuming ours, which is the signal detection timestamp and inherently vague by perhaps a dozen microseconds won't really work that well.
johannes