Skip to contentSkip to conversation

staging/lustre: tracefile: document seconds overflow

PATCH New Dilger 2 participants 6 messages Sep 28, 2015 3 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

Dilger

#2 lore raw

PATCH 4/73 staging/lustre: tracefile: document seconds overflow

Quoted 10 lines · green at linuxhacker.ru

From: Arnd Bergmann <[email protected]>

The lustre tracefile has a timestamp defined as

      __u32 ph_sec;
      __u64 ph_usec;

which seems completely backwards, as the microsecond portion of a time stamp will always fit into a __u32 value, while the second portion will overflow in 2038 or 2106 (in case of unsigned seconds).

This is indeed confusing, but the original reason for such a thing is lost in the mists of antiquity.

Quoted 2 lines

Changing this would unfortunately change the format in an incompatible way, breaking all existing user space tools that access the data.

It seems that it would be possible to fix this declaration to be compatible with current usage, since these fields are always written in host-endian order:

#ifdef(__LITTLE_ENDIAN)
	__u32 ph_sec;
	__u32 ph_usec;
        __u32 ph_sec_hi;
#else
	__u32 ph_sec;
        __u32 ph_sec_hi;
        __u32 ph_usec;

#endif

ph_sec_hi will always be zero today because microseconds don't grow so large, and we have 22 years to ensure existing user tools are updated to handle this in a similar manner.

Another option would be to mark in ph_flags whether the ph_sec field is a __u32 or __u64 and that would allow us to migrate over to a "normal" field ordering gradually (these debug logs are only useful for a short time anyway).

It would probably also make sense to change to use ph_nsec at this point as well, since that avoids the division (which might be noticeable in itself) and gives us better time resolution.

Cheers, Andreas

Quoted 58 lines

This uses ktime_get_real_ts64() to replace the insufficient do_gettimeofday() and then truncates the seconds portion to an u32 type, along with comments to explain the result.

A possible alternative would be the use of ktime_get_ts64() to read a monotonic timestamp that never overflows, but this would trigger a check in user space 'hdr->ph_sec < (1 << 30)' that attempts to ensure that the values are within a reasonable range.

---
drivers/staging/lustre/include/linux/libcfs/libcfs_debug.h | 1 +
drivers/staging/lustre/lustre/libcfs/linux/linux-tracefile.c | 10
++++++----

2 files changed, 7 insertions(+), 4 deletions(-)
drivers/staging/lustre/include/linux/libcfs/libcfs_debug.h +1 -0
index a3aa644..a1787bb 100644
@@ -73,6 +73,7 @@ struct ptldebug_header {
 	__u32 ph_mask;
 	__u16 ph_cpu_id;
 	__u16 ph_type;
+	/* time_t overflow in 2106 */
 	__u32 ph_sec;
 	__u64 ph_usec;
 	__u32 ph_stack;
drivers/staging/lustre/lustre/libcfs/linux/linux-tracefile.c +6 -4
index 87d8449..64a136c 100644
@@ -191,16 +191,18 @@ cfs_set_ptldebug_header(struct ptldebug_header
 *header,
 			struct libcfs_debug_msg_data *msgdata,
 			unsigned long stack)
 {
-	struct timeval tv;
+	struct timespec64 ts;
 
-	do_gettimeofday(&tv);
+	ktime_get_real_ts64(&ts);
 
 	header->ph_subsys = msgdata->msg_subsys;
 	header->ph_mask = msgdata->msg_mask;
 	header->ph_cpu_id = smp_processor_id();
 	header->ph_type = cfs_trace_buf_idx_get();
-	header->ph_sec = (__u32)tv.tv_sec;
-	header->ph_usec = tv.tv_usec;
+	/* y2038 safe since all user space treats this as unsigned, but
+	 * will overflow in 2106 */
+	header->ph_sec = (u32)ts.tv_sec;
+	header->ph_usec = ts.tv_nsec / NSEC_PER_USEC;
 	header->ph_stack = stack;
 	header->ph_pid = current->pid;
Quoted 5 lines
	header->ph_line_num = msgdata->msg_line;
Signature · 1 line
2.1.0

Cheers, Andreas

Signature · 4 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division
#3 lore raw

PATCH 4/73 staging/lustre: tracefile: document seconds overflow

Quoted 39 lines · Andreas
Quoted 10 lines · green at linuxhacker.ru

From: Arnd Bergmann <[email protected]>

The lustre tracefile has a timestamp defined as

     __u32 ph_sec;
     __u64 ph_usec;

which seems completely backwards, as the microsecond portion of a time stamp will always fit into a __u32 value, while the second portion will overflow in 2038 or 2106 (in case of unsigned seconds).

This is indeed confusing, but the original reason for such a thing is lost in the mists of antiquity.

Quoted 2 lines

Changing this would unfortunately change the format in an incompatible way, breaking all existing user space tools that access the data.

It seems that it would be possible to fix this declaration to be compatible with current usage, since these fields are always written in host-endian order:

#ifdef(__LITTLE_ENDIAN)
	__u32 ph_sec;
	__u32 ph_usec;
       __u32 ph_sec_hi;
#else
	__u32 ph_sec;
       __u32 ph_sec_hi;
       __u32 ph_usec;

#endif

ph_sec_hi will always be zero today because microseconds don't grow so large, and we have 22 years to ensure existing user tools are updated to handle this in a similar manner.

Quoted 8 lines

Another option would be to mark in ph_flags whether the ph_sec field is a __u32 or __u64 and that would allow us to migrate over to a "normal" field ordering gradually (these debug logs are only useful for a short time anyway).

It would probably also make sense to change to use ph_nsec at this point as well, since that avoids the division (which might be noticeable in itself) and gives us better time resolution.

I suggest we start with tools first.
Once we do have the tools that understand this greatly deployed, then we can change the kernel code, as otherwise it's going to be quite a mess. And we still do have quite a bit of time until year 2104, so I don't feel there's a big rush here.

Bye,
Oleg

#4 lore raw

PATCH 4/73 staging/lustre: tracefile: document seconds overflow

In reply to Dilger
Quoted 22 lines · Andreas
Quoted 2 lines

Changing this would unfortunately change the format in an incompatible way, breaking all existing user space tools that access the data.

It seems that it would be possible to fix this declaration to be compatible with current usage, since these fields are always written in host-endian order:

#ifdef(__LITTLE_ENDIAN)
	__u32 ph_sec;
	__u32 ph_usec;
        __u32 ph_sec_hi;
#else
	__u32 ph_sec;
        __u32 ph_sec_hi;
        __u32 ph_usec;

#endif

ph_sec_hi will always be zero today because microseconds don't grow so large, and we have 22 years to ensure existing user tools are updated to handle this in a similar manner.

Quoted 8 lines

Another option would be to mark in ph_flags whether the ph_sec field is a __u32 or __u64 and that would allow us to migrate over to a "normal" field ordering gradually (these debug logs are only useful for a short time anyway).

It would probably also make sense to change to use ph_nsec at this point as well, since that avoids the division (which might be noticeable in itself) and gives us better time resolution.

See the discussion for the previous version I sent. Unfortunately, there is no generic __LITTLE_ENDIAN macro that we can use in both kernel and user space, so we'd either have to introduce that for all architectures first, or use the approach I described earlier:

	ph_usec = (now.tv_nsec / NSEC_PER_SEC) | (now.tv_sec & 0xffffffff00000000ull);

which Oleg said was a bit too ugly.

Arnd

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

PATCH 30/73 staging/lustre: use 64-bit times for request times

Quoted 34 lines · green at linuxhacker.ru

From: Arnd Bergmann <[email protected]>

All request timestamps and deadlines in lustre are recorded in time_t and timeval units, which overflow in 2038 on 32-bit systems.

In this patch, I'm converting them to time64_t and timespec64, respectively. Unfortunately, this makes a relatively large patch, but I could not find an obvious way to split it up some more without breaking atomicity of the change.

Also unfortunately, this introduces two instances of div_u64_rem() in the request path, which can be slow on 32-bit architectures. This can probably be avoided by a larger restructuring of the code, but it is unlikely that lustre is used in performance critical setups on 32-bit architectures, so it seems better to optimize for correctness rather than speed here.

---

.../staging/lustre/lustre/include/lustre_import.h  |   2 +-
drivers/staging/lustre/lustre/include/lustre_net.h |  22 ++--
drivers/staging/lustre/lustre/mdc/mdc_locks.c      |   2 +-
drivers/staging/lustre/lustre/mdc/mdc_reint.c      |   2 +-
drivers/staging/lustre/lustre/osc/osc_request.c    |   4 +-
drivers/staging/lustre/lustre/ptlrpc/client.c      |  48 ++++-----
drivers/staging/lustre/lustre/ptlrpc/events.c      |   6 +-
drivers/staging/lustre/lustre/ptlrpc/import.c      |   4 +-
.../staging/lustre/lustre/ptlrpc/lproc_ptlrpc.c    |   8 +-
drivers/staging/lustre/lustre/ptlrpc/niobuf.c      |  10 +-
.../staging/lustre/lustre/ptlrpc/pack_generic.c    |   6 +-
drivers/staging/lustre/lustre/ptlrpc/service.c     | 115
+++++++++++----------
12 files changed, 118 insertions(+), 111 deletions(-)

[snip]

Quoted 12 lines
drivers/staging/lustre/lustre/ptlrpc/events.c +1 -1
index 53f6b62..afd869b 100644
@@ -246,7 +246,7 @@ static void ptlrpc_req_add_history(struct
 ptlrpc_service_part *svcpt,
 				   struct ptlrpc_request *req)
 {
 	__u64 sec = req->rq_arrival_time.tv_sec;
-	__u32 usec = req->rq_arrival_time.tv_usec >> 4; /* usec / 16 */
+	__u32 usec = req->rq_arrival_time.tv_nsec / NSEC_PER_USEC / 16; /* usec
Quoted 1 line

/ 16 */

This could just be written like:

	__u32 usec = req->rq_arrival_time.tv_nsec >> 14 /* nsec / 16384 */;

since the main point of this calculation is to get a number that fits into a 16-bit field to provide ordering for items in a trace log. It doesn't have to be exactly "nsec / 16000", and it avoids the division.

drivers/staging/lustre/lustre/ptlrpc/service.c static int ptlrpc_at_add_timed(struct · line 1195

Quoted 12 lines
drivers/staging/lustre/lustre/ptlrpc/service.c +1 -1
index 40de622..28f57d7 100644
@@ -1191,7 +1191,7 @@ static int ptlrpc_at_add_timed(struct
 ptlrpc_request *req)
 	spin_lock(&svcpt->scp_at_lock);
 	LASSERT(list_empty(&req->rq_timed_list));
 
-	index = (unsigned long)req->rq_deadline % array->paa_size;
+	div_u64_rem(req->rq_deadline, array->paa_size, &index);

Since this is just a round-robin index that advances once per second, it doesn't matter at all whether the calculation is computed on the 64-bit seconds or on the 32-bit seconds, so there isn't any need for the more expensive div_u64_rem() call here at all. It is fine to just truncate the seconds and then do the modulus on the 32-bit value.

drivers/staging/lustre/lustre/ptlrpc/service.c static int ptlrpc_at_check_timed(struct · line 1425

Quoted 7 lines
drivers/staging/lustre/lustre/ptlrpc/service.c +1 -1
@@ -1421,7 +1421,7 @@ static int ptlrpc_at_check_timed(struct
 ptlrpc_service_part *svcpt)
 	   server will take. Send early replies to everyone expiring soon. */
 	INIT_LIST_HEAD(&work_list);
 	deadline = -1;
-	index = (unsigned long)array->paa_deadline % array->paa_size;
+	div_u64_rem(array->paa_deadline, array->paa_size, &index);

Same here.

Cheers, Andreas

Signature · 4 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division
#7 lore raw

PATCH 30/73 staging/lustre: use 64-bit times for request times

Quoted 18 lines · Andreas
Quoted 12 lines · green at linuxhacker.ru
drivers/staging/lustre/lustre/ptlrpc/events.c +1 -1
index 53f6b62..afd869b 100644
@@ -246,7 +246,7 @@ static void ptlrpc_req_add_history(struct
 ptlrpc_service_part *svcpt,
 				   struct ptlrpc_request *req)
 {
 	__u64 sec = req->rq_arrival_time.tv_sec;
-	__u32 usec = req->rq_arrival_time.tv_usec >> 4; /* usec / 16 */
+	__u32 usec = req->rq_arrival_time.tv_nsec / NSEC_PER_USEC / 16; /* usec
Quoted 1 line

/ 16 */

This could just be written like:

Quoted 5 lines
	__u32 usec = req->rq_arrival_time.tv_nsec >> 14 /* nsec / 16384 */;

since the main point of this calculation is to get a number that fits into a 16-bit field to provide ordering for items in a trace log. It doesn't have to be exactly "nsec / 16000", and it avoids the division.

Ok, that wasn't clear from the original code, so I just moved the division from the do_gettimeofday() here to keep the data unchanged.

With your change, the

new_seq = (sec << REQS_SEC_SHIFT) |

                  (usec << REQS_USEC_SHIFT) |
                  (svcpt->scp_cpt < 0 ? 0 : svcpt->scp_cpt);

calculation will get forward jumps once a second, but I guess that doesn't matter if it's only used for sequencing.

The part that I had not noticed here is the y2106 overflow in the sequence number. If we change the logic, we should probably give a few more bits to the seconds, as well, or use monotonic time.

Quoted 19 lines

drivers/staging/lustre/lustre/ptlrpc/service.c static int ptlrpc_at_add_timed(struct · line 1195

Quoted 12 lines
drivers/staging/lustre/lustre/ptlrpc/service.c +1 -1
index 40de622..28f57d7 100644
@@ -1191,7 +1191,7 @@ static int ptlrpc_at_add_timed(struct
 ptlrpc_request *req)
 	spin_lock(&svcpt->scp_at_lock);
 	LASSERT(list_empty(&req->rq_timed_list));
 
-	index = (unsigned long)req->rq_deadline % array->paa_size;
+	div_u64_rem(req->rq_deadline, array->paa_size, &index);

Since this is just a round-robin index that advances once per second, it doesn't matter at all whether the calculation is computed on the 64-bit seconds or on the 32-bit seconds, so there isn't any need for the more expensive div_u64_rem() call here at all. It is fine to just truncate the seconds and then do the modulus on the 32-bit value.

Quoted 9 lines

drivers/staging/lustre/lustre/ptlrpc/service.c static int ptlrpc_at_check_timed(struct · line 1425

Quoted 7 lines
drivers/staging/lustre/lustre/ptlrpc/service.c +1 -1
@@ -1421,7 +1421,7 @@ static int ptlrpc_at_check_timed(struct
 ptlrpc_service_part *svcpt)
 	   server will take. Send early replies to everyone expiring soon. */
 	INIT_LIST_HEAD(&work_list);
 	deadline = -1;
-	index = (unsigned long)array->paa_deadline % array->paa_size;
+	div_u64_rem(array->paa_deadline, array->paa_size, &index);

Same here.

I went back and forth on these. Initially I did just what you suggest here and added a (u32) cast on the deadline fields, but I could not convince myself that the backwards jump in 2038 is harmless. For all I can tell, array->paa_size is not normally a power-of-two number, so (0xffffffff % array->paa_size) and (0 % array->paa_size) are not neighboring indices. If you are sure that the index can be allowed to jump in 2038, we should do the simpler math and add a comment.

Arnd

#8 lore raw

PATCH 30/73 staging/lustre: use 64-bit times for request times

Quoted 71 lines · lustre-devel on behalf of Arnd Bergmann
Quoted 17 lines · Andreas

drivers/staging/lustre/lustre/ptlrpc/events.c static void ptlrpc_req_add_history(struct · line 250

Quoted 12 lines · green at linuxhacker.ru
drivers/staging/lustre/lustre/ptlrpc/events.c +1 -1
index 53f6b62..afd869b 100644
@@ -246,7 +246,7 @@ static void ptlrpc_req_add_history(struct
 ptlrpc_service_part *svcpt,
 				   struct ptlrpc_request *req)
 {
 	__u64 sec = req->rq_arrival_time.tv_sec;
-	__u32 usec = req->rq_arrival_time.tv_usec >> 4; /* usec / 16 */
+	__u32 usec = req->rq_arrival_time.tv_nsec / NSEC_PER_USEC / 16; /*

usec

Quoted 1 line

/ 16 */

Quoted 7 lines

This could just be written like:

	__u32 usec = req->rq_arrival_time.tv_nsec >> 14 /* nsec / 16384 */;

since the main point of this calculation is to get a number that fits into a 16-bit field to provide ordering for items in a trace log. It doesn't have to be exactly "nsec / 16000", and it avoids the division.

Ok, that wasn't clear from the original code, so I just moved the division from the do_gettimeofday() here to keep the data unchanged.

With your change, the

new_seq = (sec << REQS_SEC_SHIFT) |

                 (usec << REQS_USEC_SHIFT) |
                 (svcpt->scp_cpt < 0 ? 0 : svcpt->scp_cpt);

calculation will get forward jumps once a second, but I guess that doesn't matter if it's only used for sequencing.

The part that I had not noticed here is the y2106 overflow in the sequence number. If we change the logic, we should probably give a few more bits to the seconds, as well, or use monotonic time.

Quoted 19 lines

drivers/staging/lustre/lustre/ptlrpc/service.c static int ptlrpc_at_add_timed(struct · line 1195

Quoted 12 lines
drivers/staging/lustre/lustre/ptlrpc/service.c +1 -1
index 40de622..28f57d7 100644
@@ -1191,7 +1191,7 @@ static int ptlrpc_at_add_timed(struct
 ptlrpc_request *req)
 	spin_lock(&svcpt->scp_at_lock);
 	LASSERT(list_empty(&req->rq_timed_list));
 
-	index = (unsigned long)req->rq_deadline % array->paa_size;
+	div_u64_rem(req->rq_deadline, array->paa_size, &index);

Since this is just a round-robin index that advances once per second, it doesn't matter at all whether the calculation is computed on the 64-bit seconds or on the 32-bit seconds, so there isn't any need for the more expensive div_u64_rem() call here at all. It is fine to just truncate the seconds and then do the modulus on the 32-bit value.

Quoted 9 lines

drivers/staging/lustre/lustre/ptlrpc/service.c static int ptlrpc_at_check_timed(struct · line 1425

Quoted 7 lines
drivers/staging/lustre/lustre/ptlrpc/service.c +1 -1
@@ -1421,7 +1421,7 @@ static int ptlrpc_at_check_timed(struct
 ptlrpc_service_part *svcpt)
 	   server will take. Send early replies to everyone expiring soon. */
 	INIT_LIST_HEAD(&work_list);
 	deadline = -1;
-	index = (unsigned long)array->paa_deadline % array->paa_size;
+	div_u64_rem(array->paa_deadline, array->paa_size, &index);

Same here.

Quoted 7 lines

I went back and forth on these. Initially I did just what you suggest here and added a (u32) cast on the deadline fields, but I could not convince myself that the backwards jump in 2038 is harmless. For all I can tell, array->paa_size is not normally a power-of-two number, so (0xffffffff % array->paa_size) and (0 % array->paa_size) are not neighboring indices. If you are sure that the index can be allowed to jump in 2038, we should do the simpler math and add a comment.

The jump should be largely harmless. This array is tracking the maximum RPC service time over the past paa_size interval to avoid sending spurious RPC retries if the server is already heavily loaded. If these values are not expired in 100% accurate order the worst thing that can happen is that some clients may resend their RPCs unnecessarily under heavy load.

Cheers, Andreas

Signature · 4 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division