rseq: minor optimizations
cover rseq: minor optimizations
From: Eric Dumazet <[email protected]>
rseq is a heavy user of copy to/from user data in fast paths. This series tries to reduce the cost.
Eric Dumazet (3):
rseq: optimize rseq_update_cpu_id()
rseq: remove redundant access_ok()
rseq: optimise for 64bit arches
kernel/rseq.c | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
PATCH 1/3 rseq: optimize rseq_update_cpu_id()
From: Eric Dumazet <[email protected]>
Two put_user() in rseq_update_cpu_id() are replaced by a pair of unsafe_put_user() with appropriate surroundings.
This removes one stac/clac pair on x86 in fast path.
--- kernel/rseq.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
kernel/rseq.c +11 -4
@@ -84,13 +84,20 @@static int rseq_update_cpu_id (struct task_struct *t) {u32 cpu_id =raw_smp_processor_id (); +struct rseq *r = t->rseq; -if (put_user (cpu_id, &t->rseq->cpu_id_start)) -return -EFAULT ; -if (put_user (cpu_id, &t->rseq->cpu_id)) -return -EFAULT ; +if (!user_write_access_begin (r,sizeof (*r))) + goto efault; +unsafe_put_user (cpu_id, &r->cpu_id_start, efault_end); +unsafe_put_user (cpu_id, &r->cpu_id, efault_end); +user_write_access_end ();trace_rseq_update (t);return 0 ; + +efault_end: +user_write_access_end (); +efault: +return -EFAULT ; }static int rseq_reset_rseq_cpu_id (struct task_struct *t)
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
Mathieu Desnoyers
maintainerRe: [PATCH 1/3] rseq: optimize rseq_update_cpu_id()
Quoted 25 lines · ----- On Apr 13, 2021, at 3:36 AM, Eric Dumazet [email protected] wrote:
From: Eric Dumazet <[email protected]>
Two
put_user()inrseq_update_cpu_id()are replaced by a pair ofunsafe_put_user()with appropriate surroundings.This removes one stac/clac pair on x86 in fast path.
Signed-off-by Eric Dumazet <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc Peter Zijlstra <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Boqun Feng <[email protected]>Cc Arjun Roy <[email protected]>Cc Ingo Molnar <[email protected]>---
kernel/rseq.c| 15 +++++++++++----1 file changed, 11 insertions(+), 4 deletions(-)kernel/rseq.c +0
-0index
a4f86a9d6937cdfa2f13d1dcc9be863c1943d06f..d2689ccbb132c0fc8ec0924008771e5ee1ca855e100644
kernel/rseq.c line 87
Quoted 4 lines
kernel/rseq.c +1
-0@@ -84,13 +84,20 @@static int rseq_update_cpu_id (struct task_struct *t) {u32 cpu_id =raw_smp_processor_id (); +struct rseq *r = t->rseq;
AFAIU the variable above should be a struct rseq __user *.
Elsewhere in the file we use "rseq" rather than "r" for struct rseq __user * variable name, it would be better to keep the naming consistent across the file if possible.
Thanks,
Mathieu
Quoted 22 lines
kernel/rseq.c +10
-5-if (put_user (cpu_id, &t->rseq->cpu_id_start)) -return -EFAULT ; -if (put_user (cpu_id, &t->rseq->cpu_id)) -return -EFAULT ; +if (!user_write_access_begin (r,sizeof (*r))) + goto efault; +unsafe_put_user (cpu_id, &r->cpu_id_start, efault_end); +unsafe_put_user (cpu_id, &r->cpu_id, efault_end); +user_write_access_end ();trace_rseq_update (t);return 0 ; + +efault_end: +user_write_access_end (); +efault: +return -EFAULT ; }static int rseq_reset_rseq_cpu_id (struct task_struct *t) --2.31.1.295.g9ea45b61b8-goog
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [PATCH 1/3] rseq: optimize rseq_update_cpu_id()
Quoted 35 lines · Mathieu Desnoyers
Quoted 25 lines · ----- On Apr 13, 2021, at 3:36 AM, Eric Dumazet [email protected] wrote:
From: Eric Dumazet <[email protected]>
Two
put_user()inrseq_update_cpu_id()are replaced by a pair ofunsafe_put_user()with appropriate surroundings.This removes one stac/clac pair on x86 in fast path.
Signed-off-by Eric Dumazet <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc Peter Zijlstra <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Boqun Feng <[email protected]>Cc Arjun Roy <[email protected]>Cc Ingo Molnar <[email protected]>---
kernel/rseq.c| 15 +++++++++++----1 file changed, 11 insertions(+), 4 deletions(-)kernel/rseq.c +0
-0index
a4f86a9d6937cdfa2f13d1dcc9be863c1943d06f..d2689ccbb132c0fc8ec0924008771e5ee1ca855e100644kernel/rseq.c line 87
Quoted 4 lines
kernel/rseq.c +1
-0@@ -84,13 +84,20 @@static int rseq_update_cpu_id (struct task_struct *t) {u32 cpu_id =raw_smp_processor_id (); +struct rseq *r = t->rseq;AFAIU the variable above should be a struct rseq
__user*.
Quoted 3 lines
Elsewhere in the file we use "rseq" rather than "r" for struct rseq
__user* variable name, it would be better to keep the naming consistent across the file if possible.
Absolutely, thanks for the feedback.
PATCH 2/3 rseq: remove redundant access_ok()
In reply to Eric DumazetFrom: Eric Dumazet <[email protected]>
After commit 8f2817701492 ("rseq: Use get_user/put_user rather than __get_user/__put_user") we no longer need
an access_ok() call from __rseq_handle_notify_resume()
--- kernel/rseq.c | 2 -- 1 file changed, 2 deletions(-)
kernel/rseq.c +0 -2
@@ -273,8 +273,6 @@ void __rseq_handle_notify_resume(struct ksignal *ksig, struct pt_regs *regs)if (unlikely (t->flags &PF_EXITING ))return ; -if (unlikely (!access_ok (t->rseq,sizeof (*t->rseq)))) - goto error; ret =rseq_ip_fixup (regs);if (unlikely (ret <0 )) goto error;
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
Mathieu Desnoyers
maintainerRe: [PATCH 2/3] rseq: remove redundant access_ok()
Quoted 5 lines · ----- On Apr 13, 2021, at 3:36 AM, Eric Dumazet [email protected] wrote:
From: Eric Dumazet <[email protected]>
After commit
8f2817701492("rseq: Use get_user/put_user rather than __get_user/__put_user") we no longer need
anaccess_ok()call from__rseq_handle_notify_resume()
While we are doing that, should we also remove the access_ok() check in rseq_syscall() ? It look like it can also be removed for the exact same reason outlined here.
Thanks,
Mathieu
Quoted 30 lines
Signed-off-by Eric Dumazet <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc Peter Zijlstra <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Boqun Feng <[email protected]>Cc Arjun Roy <[email protected]>Cc Ingo Molnar <[email protected]>---
kernel/rseq.c| 2 --1 file changed, 2 deletions(-)kernel/rseq.c +0
-0index
d2689ccbb132c0fc8ec0924008771e5ee1ca855e..57344f9abb43905c7dd2b6081205ff508d963e1e100644kernel/rseq.c +0
-2@@ -273,8 +273,6 @@ void __rseq_handle_notify_resume(struct ksignal *ksig, struct pt_regs *regs)if (unlikely (t->flags &PF_EXITING ))return ; -if (unlikely (!access_ok (t->rseq,sizeof (*t->rseq)))) - goto error; ret =rseq_ip_fixup (regs);if (unlikely (ret <0 ))goto error;--
2.31.1.295.g9ea45b61b8-goog
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [PATCH 2/3] rseq: remove redundant access_ok()
Quoted 5 lines · Mathieu Desnoyers
Quoted 2 lines · ----- On Apr 13, 2021, at 3:36 AM, Eric Dumazet [email protected] wrote:
From: Eric Dumazet <[email protected]>
Quoted 7 lines
Quoted 3 lines
After commit
8f2817701492("rseq: Use get_user/put_user rather than __get_user/__put_user") we no longer need
anaccess_ok()call from__rseq_handle_notify_resume()While we are doing that, should we also remove the
access_ok()check inrseq_syscall()? It look like it can also be removed for the exact same reason outlined here.
Yes, good idea.
I was focusing in __rseq_handle_notify_resume() paths butrseq_syscall() can get the same.
Quoted 39 lines
Thanks,
Mathieu
Quoted 28 lines
Signed-off-by Eric Dumazet <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc Peter Zijlstra <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Boqun Feng <[email protected]>Cc Arjun Roy <[email protected]>Cc Ingo Molnar <[email protected]>---
kernel/rseq.c| 2 --1 file changed, 2 deletions(-)kernel/rseq.c +0
-0index
d2689ccbb132c0fc8ec0924008771e5ee1ca855e..57344f9abb43905c7dd2b6081205ff508d963e1e100644kernel/rseq.c +0
-2@@ -273,8 +273,6 @@ void __rseq_handle_notify_resume(struct ksignal *ksig, struct pt_regs *regs)if (unlikely (t->flags &PF_EXITING ))return ; -if (unlikely (!access_ok (t->rseq,sizeof (*t->rseq)))) - goto error; ret =rseq_ip_fixup (regs);if (unlikely (ret <0 ))goto error;Quoted 2 lines
--
2.31.1.295.g9ea45b61b8-goog--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
PATCH 3/3 rseq: optimise for 64bit arches
In reply to Eric DumazetFrom: Eric Dumazet <[email protected]>
Commit ec9c82e03a74 ("rseq: uapi: Declare rseq_cs field as union, update includes") added regressions for our servers.
Using copy_from_user() and clear_user() for 64bit values on 64bit arches is suboptimal.
We might revisit this patch once all 32bit arches support get_user() and/or put_user() for 8 bytes values.
--- kernel/rseq.c | 9 +++++++++ 1 file changed, 9 insertions(+)
kernel/rseq.c +9 -0
@@ -127,8 +127,13 @@ static int rseq_get_rseq_cs(struct task_struct *t, struct rseq_cs *rseq_cs)u32 sig;int ret; +#ifdef CONFIG_64BIT +if (get_user (ptr, &t->rseq->rseq_cs.ptr64)) +return -EFAULT ; +#else if (copy_from_user (&ptr, &t->rseq->rseq_cs.ptr64,sizeof (ptr)))return -EFAULT ; +#endif if (!ptr) {memset (rseq_cs,0 ,sizeof (*rseq_cs));return 0 ;
@@ -211,9 +216,13 @@ static int clear_rseq_cs(struct task_struct *t)* * Set rseq_cs to NULL. */ +#ifdef CONFIG_64BIT +return put_user (0ULL , &t->rseq->rseq_cs.ptr64); +#else if (clear_user (&t->rseq->rseq_cs.ptr64,sizeof (t->rseq->rseq_cs.ptr64)))return -EFAULT ;return 0 ; +#endif }/*
Signature · 1 line
2.31.1.295.g9ea45b61b8-goog
Peter Zijlstra
maintainerRe: [PATCH 3/3] rseq: optimise for 64bit arches
Quoted 10 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Commit
ec9c82e03a74("rseq: uapi: Declarerseq_csfield as union, update includes") added regressions for our servers.Using
copy_from_user()andclear_user()for 64bit values on 64bit arches is suboptimal.We might revisit this patch once all 32bit arches support
get_user()and/orput_user()for 8 bytes values.
Argh, what a mess :/ afaict only nios32 lacks put_user_8, but get_user_8 is missing in a fair number of archs.
That said; 32bit archs never have to actually set the top bits in that word, so they _could_ only set the low 32 bits. That works provided userspace itself keeps the high bits clear.
So I suppose that if we're going to #ifdef this, we might as well do the whole thing.
Mathieu; did I forget a reason why this cannot work?
kernel/rseq.c +8 -3
@@ -115,20 +115,25 @@ static int rseq_reset_rseq_cpu_id(struct task_struct *t)static int rseq_get_rseq_cs (struct task_struct *t,struct rseq_cs *rseq_cs) {struct rseq_cs __user *urseq_cs; -u64 ptr; +unsigned long ptr;u32 __user *usig;u32 sig;int ret; -if (copy_from_user (&ptr, &t->rseq->rseq_cs.ptr64,sizeof (ptr))) +#ifdef CONFIG_64BIT +if (get_user (ptr, &t->rseq->rseq_cs.ptr64))return -EFAULT ; +#else +if (get_user (ptr, &t->rseq->rseq_cs.ptr32)) +return -EFAULT ; +#endif if (!ptr) {memset (rseq_cs,0 ,sizeof (*rseq_cs));return 0 ; }if (ptr >=TASK_SIZE )return -EINVAL ; - urseq_cs = (struct rseq_cs __user *)(unsigned long )ptr; + urseq_cs = (struct rseq_cs __user *)ptr;if (copy_from_user (rseq_cs, urseq_cs,sizeof (*rseq_cs)))return -EFAULT ;
RE: [PATCH 3/3] rseq: optimise for 64bit arches
From: Peter Zijlstra
Quoted 14 lines
Sent: 13 April 2021 10:10
Quoted 10 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Commit
ec9c82e03a74("rseq: uapi: Declarerseq_csfield as union, update includes") added regressions for our servers.Using
copy_from_user()andclear_user()for 64bit values on 64bit arches is suboptimal.We might revisit this patch once all 32bit arches support
get_user()and/orput_user()for 8 bytes values.
Quoted 6 lines
Argh, what a mess :/ afaict only nios32 lacks
put_user_8, butget_user_8is missing in a fair number of archs.That said; 32bit archs never have to actually set the top bits in that word, so they _could_ only set the low 32 bits. That works provided userspace itself keeps the high bits clear.
Does that work for 32bit BE ?
David
Quoted 28 lines
So I suppose that if we're going to #ifdef this, we might as well do the whole thing.
Mathieu; did I forget a reason why this cannot work?
kernel/rseq.c +7
-2@@ -115,20 +115,25 @@ static int rseq_reset_rseq_cpu_id(struct task_struct *t)static int rseq_get_rseq_cs (struct task_struct *t,struct rseq_cs *rseq_cs) {struct rseq_cs __user *urseq_cs; -u64 ptr; +unsigned long ptr;u32 __user *usig;u32 sig;int ret; -if (copy_from_user (&ptr, &t->rseq->rseq_cs.ptr64,sizeof (ptr))) +#ifdef CONFIG_64BIT +if (get_user (ptr, &t->rseq->rseq_cs.ptr64))return -EFAULT ; +#else +if (get_user (ptr, &t->rseq->rseq_cs.ptr32)) +return -EFAULT ; +#endif if (!ptr) {
kernel/rseq.c static int rseq_reset_rseq_cpu_id(struct task_struct *t) · line 139
Quoted 10 lines
kernel/rseq.c +1
-1memset (rseq_cs,0 ,sizeof (*rseq_cs));return 0 ; }if (ptr >=TASK_SIZE )return -EINVAL ; - urseq_cs = (struct rseq_cs __user *)(unsigned long )ptr; + urseq_cs = (struct rseq_cs __user *)ptr;if (copy_from_user (rseq_cs, urseq_cs,sizeof (*rseq_cs)))return -EFAULT ;
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Mathieu Desnoyers
maintainerRe: [PATCH 3/3] rseq: optimise for 64bit arches
Quoted 15 lines · ----- On Apr 13, 2021, at 6:36 AM, David Laight [email protected] wrote:
From: Peter Zijlstra
Quoted 14 lines
Sent: 13 April 2021 10:10
Quoted 10 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Commit
ec9c82e03a74("rseq: uapi: Declarerseq_csfield as union, update includes") added regressions for our servers.Using
copy_from_user()andclear_user()for 64bit values on 64bit arches is suboptimal.We might revisit this patch once all 32bit arches support
get_user()and/orput_user()for 8 bytes values.
Quoted 8 lines
Quoted 6 lines
Argh, what a mess :/ afaict only nios32 lacks
put_user_8, butget_user_8is missing in a fair number of archs.That said; 32bit archs never have to actually set the top bits in that word, so they _could_ only set the low 32 bits. That works provided userspace itself keeps the high bits clear.
Does that work for 32bit BE ?
Yes, because uapi/linux/rseq.h defines the ptr32 as:
#if (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || defined(__BIG_ENDIAN)
__u32 padding; /* Initialized to zero. */
__u32 ptr32;
#else /* LITTLE */
__u32 ptr32;
__u32 padding; /* Initialized to zero. */
#endif /* ENDIAN */
which takes care of BE vs LE.
Quoted 7 lines
David
Quoted 4 lines
So I suppose that if we're going to #ifdef this, we might as well do the whole thing.
Mathieu; did I forget a reason why this cannot work?
The only difference it brings on 32-bit is that the truncation of high bits will be done before the following validation:
if (!ptr) {
memset(rseq_cs, 0, sizeof(*rseq_cs));
return 0;
}
if (ptr >= TASK_SIZE)
return -EINVAL;The question is whether we really want to issue a segmentation fault if 32-bit user-space has set non-zero high bits, or if silently ignoring those high bits is acceptable.
Nits below:
Quoted 11 lines
kernel/rseq.c static int rseq_reset_rseq_cpu_id(struct task_struct *t) · line 118
Quoted 11 lines
kernel/rseq.c +1
-1@@ -115,20 +115,25 @@ static int rseq_reset_rseq_cpu_id(struct task_struct *t)static int rseq_get_rseq_cs (struct task_struct *t,struct rseq_cs *rseq_cs) {struct rseq_cs __user *urseq_cs; -u64 ptr; +unsigned long ptr;
I am always reluctant to use long/unsigned long type as type for the get/put_user
(x) argument, because it hides the cast deep within architecture-specific macros. I understand that in this specific case it happens that on 64-bit archs we end up casting a u64 to unsigned long (same size), and on 32-bit archs we end up casting a u32 to unsigned long (also same size), so there is no practical concern about type promotion and sign-extension, but I think it would be better to have something explicit, e.g.:
#ifdef CONFIG_64BIT
static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp, struct rseq_cs *rseq_cs)
{
u64 ptr;
if (get_user(ptr, &rseq_cs->ptr64))
return -EFAULT;
*ptrp = (struct rseq_cs __user *)ptr;
return 0;
}
#else
static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp, struct rseq_cs *rseq_cs)
{
u32 ptr;
if (get_user(ptr, &rseq_cs->ptr.ptr32))
return -EFAULT;
*ptrp = (struct rseq_cs __user *)ptr;
return 0;
}#endif
And use those helpers to get the ptr value.
Quoted 10 lines
kernel/rseq.c static int rseq_reset_rseq_cpu_id(struct task_struct *t) · line 127
Quoted 10 lines
kernel/rseq.c +4
-1u32 __user *usig;u32 sig;int ret; -if (copy_from_user (&ptr, &t->rseq->rseq_cs.ptr64,sizeof (ptr))) +#ifdef CONFIG_64BIT +if (get_user (ptr, &t->rseq->rseq_cs.ptr64))return -EFAULT ; +#else +if (get_user (ptr, &t->rseq->rseq_cs.ptr32))
Note that this is also not right. It should be &t->rseq->rseq_cs.ptr.ptr32.
Thanks,
Mathieu
Quoted 18 lines
kernel/rseq.c static int rseq_reset_rseq_cpu_id(struct task_struct *t) · line 139
Quoted 13 lines
kernel/rseq.c +3
-1+return -EFAULT ; +#endif if (!ptr) {memset (rseq_cs,0 ,sizeof (*rseq_cs));return 0 ; }if (ptr >=TASK_SIZE )return -EINVAL ; - urseq_cs = (struct rseq_cs __user *)(unsigned long )ptr; + urseq_cs = (struct rseq_cs __user *)ptr;if (copy_from_user (rseq_cs, urseq_cs,sizeof (*rseq_cs)))return -EFAULT ;-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
RE: [PATCH 3/3] rseq: optimise for 64bit arches
From: Mathieu Desnoyers <[email protected]>
Quoted 1 line
Sent: 13 April 2021 15:22
...
Quoted 17 lines
Quoted 6 lines
David
Quoted 4 lines
So I suppose that if we're going to #ifdef this, we might as well do the whole thing.
Mathieu; did I forget a reason why this cannot work?
The only difference it brings on 32-bit is that the truncation of high bits will be done before the following validation:
if (!ptr) { memset(rseq_cs, 0, sizeof(*rseq_cs)); return 0; } if (ptr >= TASK_SIZE) return -EINVAL;
Quoted 3 lines
The question is whether we really want to issue a segmentation fault if 32-bit user-space has set non-zero high bits, or if silently ignoring those high bits is acceptable.
Well, 32bit programs running on 64bit kernels better zero the 'padding'.
It is a shame that the 32bit compilers don't have an attribute(64bit) for pointers.
Quoted 8 lines
...
I am always reluctant to use long/unsigned long type as type for the get/put_user
(x) argument, because it hides the cast deep within architecture-specific macros. I understand that in this specific case it happens that on 64-bit archs we end up casting a u64 to unsigned long (same size), and on 32-bit archs we end up casting a u32 to unsigned long (also same size), so there is no practical concern about type promotion and sign-extension, but I think it would be better to have something explicit, e.g.:
If the ABI passes small structures in registers (most modern ones) then user pointers probably ought to be encapsulated in a structure. Possibly as a pointer to an unknown structure.
That would force all the required type checking. Unfortunately the fallout/churn would be massive.
Quoted 20 lines
#ifdef
CONFIG_64BIT
static intrseq_get_cs_ptr(struct rseq_cs __user **uptrp, struct rseq_cs *rseq_cs){ u64 ptr; if (get_user(ptr, &rseq_cs->ptr64)) return -EFAULT; *ptrp = (struct rseq_cs __user *)ptr; return 0; } #else static int rseq_get_cs_ptr(struct rseq_cs __user **uptrp, struct rseq_cs *rseq_cs) { u32 ptr; if (get_user(ptr, &rseq_cs->ptr.ptr32)) return -EFAULT; *ptrp = (struct rseq_cs __user *)ptr; return 0; }
Quoted 1 line
#endif
Hmmm... too much replication.
You could do:
#ifdef CONFIG_64BIT
#define PTR_TYPE u64
#define PTR_FLD ptr64
#else
#define PTR_TYPE u32
#define PTR_FLD ptr32
#endif
Then have one copy of the code and the #undefs.
...
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Mathieu Desnoyers
maintainerRe: [PATCH 3/3] rseq: optimise for 64bit arches
----- On Apr 13, 2021, at 11:06 AM, David Laight [email protected] wrote:
[...]
Quoted 11 lines
Hmmm... too much replication.
You could do:
#ifdefCONFIG_64BIT
#definePTR_TYPEu64
#definePTR_FLDptr64
#else
#definePTR_TYPEu32
#definePTR_FLDptr32
#endif
Quoted 1 line
Then have one copy of the code and the #undefs.
Good point, agreed.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com