staging/lustre: tracefile: document seconds overflow
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Dilger
PATCH 4/73 staging/lustre: tracefile: document seconds overflow
Quoted 10 lines · green at linuxhacker.ru
From: Arnd Bergmann <[email protected]>
The lustre tracefile has a timestamp defined as
__u32 ph_sec; __u64 ph_usec;which seems completely backwards, as the microsecond portion of a time stamp will always fit into a
__u32value, while the second portion will overflow in 2038 or 2106 (in case of unsigned seconds).
This is indeed confusing, but the original reason for such a thing is lost in the mists of antiquity.
Quoted 2 lines
Changing this would unfortunately change the format in an incompatible way, breaking all existing user space tools that access the data.
It seems that it would be possible to fix this declaration to be compatible with current usage, since these fields are always written in host-endian order:
#ifdef(__LITTLE_ENDIAN)
__u32 ph_sec;
__u32 ph_usec;
__u32 ph_sec_hi;
#else
__u32 ph_sec;
__u32 ph_sec_hi;
__u32 ph_usec;#endif
ph_sec_hi will always be zero today because microseconds don't grow so large, and we have 22 years to ensure existing user tools are updated to handle this in a similar manner.
Another option would be to mark in ph_flags whether the ph_sec field is a __u32 or __u64 and that would allow us to migrate over to a "normal" field ordering gradually (these debug logs are only useful for a short time anyway).
It would probably also make sense to change to use ph_nsec at this point as well, since that avoids the division (which might be noticeable in itself) and gives us better time resolution.
Cheers, Andreas
Quoted 58 lines
This uses
ktime_get_real_ts64()to replace the insufficientdo_gettimeofday()and then truncates the seconds portion to an u32 type, along with comments to explain the result.A possible alternative would be the use of
ktime_get_ts64()to read a monotonic timestamp that never overflows, but this would trigger a check in user space 'hdr->ph_sec< (1 << 30)' that attempts to ensure that the values are within a reasonable range.Signed-off-by Arnd Bergmann <[email protected]>Signed-off-by Oleg Drokin <[email protected]>---
drivers/staging/lustre/include/linux/libcfs/libcfs_debug.h| 1 +drivers/staging/lustre/lustre/libcfs/linux/linux-tracefile.c| 10
++++++----2 files changed, 7 insertions(+), 4 deletions(-)drivers/staging/lustre/include/linux/libcfs/libcfs_debug.h +1
-0@@ -73,6 +73,7 @@ struct ptldebug_header {__u32 ph_mask;__u16 ph_cpu_id;__u16 ph_type; +/* time_t overflow in 2106 */ __u32 ph_sec;__u64 ph_usec;__u32 ph_stack;drivers/staging/lustre/lustre/libcfs/linux/linux-tracefile.c +6
-4@@ -191,16 +191,18 @@ cfs_set_ptldebug_header(struct ptldebug_header *header, struct libcfs_debug_msg_data *msgdata, unsigned long stack) { -struct timeval tv; +struct timespec64 ts; -do_gettimeofday (&tv); +ktime_get_real_ts64 (&ts); header->ph_subsys = msgdata->msg_subsys; header->ph_mask = msgdata->msg_mask; header->ph_cpu_id =smp_processor_id (); header->ph_type =cfs_trace_buf_idx_get (); - header->ph_sec = (__u32 )tv.tv_sec; - header->ph_usec = tv.tv_usec; +/* y2038 safe since all user space treats this as unsigned, but +* will overflow in 2106 */ + header->ph_sec = (u32 )ts.tv_sec; + header->ph_usec = ts.tv_nsec /NSEC_PER_USEC ; header->ph_stack = stack; header->ph_pid = current->pid;
Quoted 5 lines
header->ph_line_num = msgdata->msg_line;Signature · 1 line
2.1.0
Cheers, Andreas
Signature · 4 lines
Andreas Dilger Lustre Software Architect Intel High Performance Data Division
Drokin
PATCH 4/73 staging/lustre: tracefile: document seconds overflow
Quoted 39 lines · Andreas
Quoted 10 lines · green at linuxhacker.ru
From: Arnd Bergmann <[email protected]>
The lustre tracefile has a timestamp defined as
__u32 ph_sec; __u64 ph_usec;which seems completely backwards, as the microsecond portion of a time stamp will always fit into a
__u32value, while the second portion will overflow in 2038 or 2106 (in case of unsigned seconds).This is indeed confusing, but the original reason for such a thing is lost in the mists of antiquity.
Quoted 2 lines
Changing this would unfortunately change the format in an incompatible way, breaking all existing user space tools that access the data.
It seems that it would be possible to fix this declaration to be compatible with current usage, since these fields are always written in host-endian order:
#ifdef(__LITTLE_ENDIAN) __u32 ph_sec; __u32 ph_usec; __u32 ph_sec_hi; #else __u32 ph_sec; __u32 ph_sec_hi; __u32 ph_usec;#endif
ph_sec_hiwill always be zero today because microseconds don't grow so large, and we have 22 years to ensure existing user tools are updated to handle this in a similar manner.
Quoted 8 lines
Another option would be to mark in
ph_flagswhether theph_secfield is a__u32or__u64and that would allow us to migrate over to a "normal" field ordering gradually (these debug logs are only useful for a short time anyway).It would probably also make sense to change to use
ph_nsecat this point as well, since that avoids the division (which might be noticeable in itself) and gives us better time resolution.
I suggest we start with tools first.
Once we do have the tools that understand this greatly deployed, then we can change the kernel code, as otherwise it's going to be quite a mess. And we still do have quite a bit of time until year 2104, so I don't feel there's a big rush here.
Bye,
Oleg
PATCH 4/73 staging/lustre: tracefile: document seconds overflow
In reply to DilgerQuoted 22 lines · Andreas
Quoted 2 lines
Changing this would unfortunately change the format in an incompatible way, breaking all existing user space tools that access the data.
It seems that it would be possible to fix this declaration to be compatible with current usage, since these fields are always written in host-endian order:
#ifdef(__LITTLE_ENDIAN) __u32 ph_sec; __u32 ph_usec; __u32 ph_sec_hi; #else __u32 ph_sec; __u32 ph_sec_hi; __u32 ph_usec;#endif
ph_sec_hiwill always be zero today because microseconds don't grow so large, and we have 22 years to ensure existing user tools are updated to handle this in a similar manner.
Quoted 8 lines
Another option would be to mark in
ph_flagswhether theph_secfield is a__u32or__u64and that would allow us to migrate over to a "normal" field ordering gradually (these debug logs are only useful for a short time anyway).It would probably also make sense to change to use
ph_nsecat this point as well, since that avoids the division (which might be noticeable in itself) and gives us better time resolution.
See the discussion for the previous version I sent. Unfortunately, there is no generic __LITTLE_ENDIAN macro that we can use in both kernel and user space, so we'd either have to introduce that for all architectures first, or use the approach I described earlier:
ph_usec = (now.tv_nsec / NSEC_PER_SEC) | (now.tv_sec & 0xffffffff00000000ull);
which Oleg said was a bit too ugly.
Arnd
A message that is not in the archive (sent off-list or not yet received): [email protected]
Dilger
PATCH 30/73 staging/lustre: use 64-bit times for request times
Quoted 34 lines · green at linuxhacker.ru
From: Arnd Bergmann <[email protected]>
All request timestamps and deadlines in lustre are recorded in
time_tand timeval units, which overflow in 2038 on 32-bit systems.In this patch, I'm converting them to
time64_tand timespec64, respectively. Unfortunately, this makes a relatively large patch, but I could not find an obvious way to split it up some more without breaking atomicity of the change.Also unfortunately, this introduces two instances of
div_u64_rem()in the request path, which can be slow on 32-bit architectures. This can probably be avoided by a larger restructuring of the code, but it is unlikely that lustre is used in performance critical setups on 32-bit architectures, so it seems better to optimize for correctness rather than speed here.Signed-off-by Arnd Bergmann <[email protected]>Signed-off-by Oleg Drokin <[email protected]>---
.../staging/lustre/lustre/include/lustre_import.h | 2 +- drivers/staging/lustre/lustre/include/lustre_net.h | 22 ++-- drivers/staging/lustre/lustre/mdc/mdc_locks.c | 2 +- drivers/staging/lustre/lustre/mdc/mdc_reint.c | 2 +- drivers/staging/lustre/lustre/osc/osc_request.c | 4 +- drivers/staging/lustre/lustre/ptlrpc/client.c | 48 ++++----- drivers/staging/lustre/lustre/ptlrpc/events.c | 6 +- drivers/staging/lustre/lustre/ptlrpc/import.c | 4 +- .../staging/lustre/lustre/ptlrpc/lproc_ptlrpc.c | 8 +- drivers/staging/lustre/lustre/ptlrpc/niobuf.c | 10 +- .../staging/lustre/lustre/ptlrpc/pack_generic.c | 6 +- drivers/staging/lustre/lustre/ptlrpc/service.c | 115 +++++++++++---------- 12 files changed, 118 insertions(+), 111 deletions(-)
[snip]
Quoted 12 lines
drivers/staging/lustre/lustre/ptlrpc/events.c +1
-1@@ -246,7 +246,7 @@ static void ptlrpc_req_add_history(structptlrpc_service_part *svcpt, struct ptlrpc_request *req) { __u64 sec = req->rq_arrival_time.tv_sec; -__u32 usec = req->rq_arrival_time.tv_usec >>4 ;/* usec / 16 */ +__u32 usec = req->rq_arrival_time.tv_nsec /NSEC_PER_USEC /16 ;/* usec
Quoted 1 line
/ 16 */
This could just be written like:
__u32 usec = req->rq_arrival_time.tv_nsec >> 14 /* nsec / 16384 */;
since the main point of this calculation is to get a number that fits into a 16-bit field to provide ordering for items in a trace log. It doesn't have to be exactly "nsec / 16000", and it avoids the division.
drivers/staging/lustre/lustre/ptlrpc/service.c static int ptlrpc_at_add_timed(struct · line 1195
Quoted 12 lines
drivers/staging/lustre/lustre/ptlrpc/service.c +1
-1@@ -1191,7 +1191,7 @@ static int ptlrpc_at_add_timed(struct ptlrpc_request *req) spin_lock(&svcpt->scp_at_lock);LASSERT (list_empty (&req->rq_timed_list)); - index = (unsigned long )req->rq_deadline % array->paa_size; +div_u64_rem (req->rq_deadline, array->paa_size, &index);
Since this is just a round-robin index that advances once per second, it doesn't matter at all whether the calculation is computed on the 64-bit seconds or on the 32-bit seconds, so there isn't any need for the more expensive div_u64_rem() call here at all. It is fine to just truncate the seconds and then do the modulus on the 32-bit value.
drivers/staging/lustre/lustre/ptlrpc/service.c static int ptlrpc_at_check_timed(struct · line 1425
Quoted 7 lines
drivers/staging/lustre/lustre/ptlrpc/service.c +1
-1@@ -1421,7 +1421,7 @@ static int ptlrpc_at_check_timed(structptlrpc_service_part *svcpt) server will take. Send early replies to everyone expiring soon. */ INIT_LIST_HEAD (&work_list); deadline =-1 ; - index = (unsigned long )array->paa_deadline % array->paa_size; +div_u64_rem (array->paa_deadline, array->paa_size, &index);
Same here.
Cheers, Andreas
Signature · 4 lines
Andreas Dilger Lustre Software Architect Intel High Performance Data Division
PATCH 30/73 staging/lustre: use 64-bit times for request times
Quoted 18 lines · Andreas
Quoted 12 lines · green at linuxhacker.ru
drivers/staging/lustre/lustre/ptlrpc/events.c +1
-1@@ -246,7 +246,7 @@ static void ptlrpc_req_add_history(structptlrpc_service_part *svcpt, struct ptlrpc_request *req) { __u64 sec = req->rq_arrival_time.tv_sec; -__u32 usec = req->rq_arrival_time.tv_usec >>4 ;/* usec / 16 */ +__u32 usec = req->rq_arrival_time.tv_nsec /NSEC_PER_USEC /16 ;/* usec Quoted 1 line
/ 16 */
This could just be written like:
Quoted 5 lines
__u32 usec = req->rq_arrival_time.tv_nsec >> 14 /* nsec / 16384 */;since the main point of this calculation is to get a number that fits into a 16-bit field to provide ordering for items in a trace log. It doesn't have to be exactly "nsec / 16000", and it avoids the division.
Ok, that wasn't clear from the original code, so I just moved the division from the do_gettimeofday() here to keep the data unchanged.
With your change, the
new_seq = (sec << REQS_SEC_SHIFT) |
(usec << REQS_USEC_SHIFT) |
(svcpt->scp_cpt < 0 ? 0 : svcpt->scp_cpt);calculation will get forward jumps once a second, but I guess that doesn't matter if it's only used for sequencing.
The part that I had not noticed here is the y2106 overflow in the sequence number. If we change the logic, we should probably give a few more bits to the seconds, as well, or use monotonic time.
Quoted 19 lines
drivers/staging/lustre/lustre/ptlrpc/service.c static int ptlrpc_at_add_timed(struct · line 1195
Quoted 12 lines
drivers/staging/lustre/lustre/ptlrpc/service.c +1
-1@@ -1191,7 +1191,7 @@ static int ptlrpc_at_add_timed(struct ptlrpc_request *req) spin_lock(&svcpt->scp_at_lock);LASSERT (list_empty (&req->rq_timed_list)); - index = (unsigned long )req->rq_deadline % array->paa_size; +div_u64_rem (req->rq_deadline, array->paa_size, &index);Since this is just a round-robin index that advances once per second, it doesn't matter at all whether the calculation is computed on the 64-bit seconds or on the 32-bit seconds, so there isn't any need for the more expensive
div_u64_rem()call here at all. It is fine to just truncate the seconds and then do the modulus on the 32-bit value.
Quoted 9 lines
drivers/staging/lustre/lustre/ptlrpc/service.c static int ptlrpc_at_check_timed(struct · line 1425
Quoted 7 lines
drivers/staging/lustre/lustre/ptlrpc/service.c +1
-1@@ -1421,7 +1421,7 @@ static int ptlrpc_at_check_timed(structptlrpc_service_part *svcpt) server will take. Send early replies to everyone expiring soon. */ INIT_LIST_HEAD (&work_list); deadline =-1 ; - index = (unsigned long )array->paa_deadline % array->paa_size; +div_u64_rem (array->paa_deadline, array->paa_size, &index);Same here.
I went back and forth on these. Initially I did just what you suggest here and added a (u32) cast on the deadline fields, but I could not convince myself that the backwards jump in 2038 is harmless. For all I can tell, array->paa_size is not normally a power-of-two number, so (0xffffffff % array->paa_size) and (0 % array->paa_size) are not neighboring indices. If you are sure that the index can be allowed to jump in 2038, we should do the simpler math and add a comment.
Arnd
Dilger
PATCH 30/73 staging/lustre: use 64-bit times for request times
Quoted 71 lines · lustre-devel on behalf of Arnd Bergmann
Quoted 17 lines · Andreas
drivers/staging/lustre/lustre/ptlrpc/events.c static void ptlrpc_req_add_history(struct · line 250
Quoted 12 lines · green at linuxhacker.ru
drivers/staging/lustre/lustre/ptlrpc/events.c +1
-1@@ -246,7 +246,7 @@ static void ptlrpc_req_add_history(structptlrpc_service_part *svcpt, struct ptlrpc_request *req) { __u64 sec = req->rq_arrival_time.tv_sec; -__u32 usec = req->rq_arrival_time.tv_usec >>4 ;/* usec / 16 */ +__u32 usec = req->rq_arrival_time.tv_nsec /NSEC_PER_USEC /16 ;/* usec
Quoted 1 line
/ 16 */
Quoted 7 lines
This could just be written like:
__u32 usec = req->rq_arrival_time.tv_nsec >> 14 /* nsec / 16384 */;since the main point of this calculation is to get a number that fits into a 16-bit field to provide ordering for items in a trace log. It doesn't have to be exactly "nsec / 16000", and it avoids the division.
Ok, that wasn't clear from the original code, so I just moved the division from the
do_gettimeofday()here to keep the data unchanged.With your change, the
new_seq= (sec <<REQS_SEC_SHIFT) |(usec << REQS_USEC_SHIFT) | (svcpt->scp_cpt < 0 ? 0 : svcpt->scp_cpt);calculation will get forward jumps once a second, but I guess that doesn't matter if it's only used for sequencing.
The part that I had not noticed here is the y2106 overflow in the sequence number. If we change the logic, we should probably give a few more bits to the seconds, as well, or use monotonic time.
Quoted 19 lines
drivers/staging/lustre/lustre/ptlrpc/service.c static int ptlrpc_at_add_timed(struct · line 1195
Quoted 12 lines
drivers/staging/lustre/lustre/ptlrpc/service.c +1
-1@@ -1191,7 +1191,7 @@ static int ptlrpc_at_add_timed(struct ptlrpc_request *req) spin_lock(&svcpt->scp_at_lock);LASSERT (list_empty (&req->rq_timed_list)); - index = (unsigned long )req->rq_deadline % array->paa_size; +div_u64_rem (req->rq_deadline, array->paa_size, &index);Since this is just a round-robin index that advances once per second, it doesn't matter at all whether the calculation is computed on the 64-bit seconds or on the 32-bit seconds, so there isn't any need for the more expensive
div_u64_rem()call here at all. It is fine to just truncate the seconds and then do the modulus on the 32-bit value.Quoted 9 lines
drivers/staging/lustre/lustre/ptlrpc/service.c static int ptlrpc_at_check_timed(struct · line 1425
Quoted 7 lines
drivers/staging/lustre/lustre/ptlrpc/service.c +1
-1@@ -1421,7 +1421,7 @@ static int ptlrpc_at_check_timed(structptlrpc_service_part *svcpt) server will take. Send early replies to everyone expiring soon. */ INIT_LIST_HEAD (&work_list); deadline =-1 ; - index = (unsigned long )array->paa_deadline % array->paa_size; +div_u64_rem (array->paa_deadline, array->paa_size, &index);Same here.
Quoted 7 lines
I went back and forth on these. Initially I did just what you suggest here and added a (u32) cast on the deadline fields, but I could not convince myself that the backwards jump in 2038 is harmless. For all I can tell,
array->paa_sizeis not normally a power-of-two number, so (0xffffffff%array->paa_size) and (0 %array->paa_size) are not neighboring indices. If you are sure that the index can be allowed to jump in 2038, we should do the simpler math and add a comment.
The jump should be largely harmless. This array is tracking the maximum RPC service time over the past paa_size interval to avoid sending spurious RPC retries if the server is already heavily loaded. If these values are not expired in 100% accurate order the worst thing that can happen is that some clients may resend their RPCs unnecessarily under heavy load.
Cheers, Andreas
Signature · 4 lines
Andreas Dilger Lustre Software Architect Intel High Performance Data Division